FastVideo UniServe: Production serving for FastH3 on NVIDIA Blackwell GPUs at 1.8x lower latency and 1.45x higher throughput

TL;DR

FastVideo UniServe is a serving engine for FastH3 8-Step text-to-video-with-audio generation (more models to come!), built for production serving with our partners Nuva Lab and Reactor:

• Production Ready on Day 0. UniServe is serving production API traffic on Reactor and Nuva Lab!
• Faster than real time. On eight NVIDIA GB200 GPUs, a 5-second 1344×768 clip with audio from a 1K-token prompt arrives as a finished MP4 in 3.1 s and a 10-second clip in 6.2 s (5.3 s with NVFP4).
• Throughput. About 11,000 videos a day on eight NVIDIA GB200 GPUs and 3,400 on eight RTX PRO 6000 GPUs for a mix of 5- to 15-second clips: 6.4 and 21.2 GPU-seconds per second of generated video.
• Fastest open-source serving. 1.2–1.8× lower median latency for a 10-second clip with a 1K-token prompt, and 15–45% higher throughput under load, than FastVideo, vLLM-Omni and SGLang.
• Hardware. Verified on NVIDIA GB200, RTX PRO 6000 Blackwell and H200. An NVFP4 PTQ checkpoint runs on Blackwell GPUs for lower latency and higher throughput with minimal quality drop.

FastH3 turns a text prompt into a video with a synchronized soundtrack in eight denoising steps. Each request passes through heterogeneous components: a text encoder and refiner, a long-sequence multimodal diffusion transformer (DiT), video and audio VAE decoders, and CPU-side video encoding and muxing. They differ in whether they run on the GPU or the CPU, in how they parallelize and in arithmetic intensity, so a faster attention kernel speeds up only one part of a request.

UniServe therefore optimizes the whole request path. It cuts the communication and memory traffic around sparse attention in the DiT, batches and fuses the VAE decoder, and encodes video on the CPU while the GPUs are still decoding.

Results at a glance

Unless noted otherwise, all systems run the FastH3 8-Step V2 checkpoint with BF16 text encoding and denoising and FP16 video-decoder projection weights. Latency is measured end to end: from request submission until the client receives the last byte of an MP4 containing both video and audio.

Latency. For a 10-second clip with a 1K-token prompt, one request at a time, UniServe’s median latency is about 1.3× lower than the baselines’ on four GB200 GPUs, 1.3–1.8× lower on eight GB200 GPUs across two nodes and 1.2–1.4× lower on eight RTX PRO 6000 GPUs. On eight GB200 GPUs the median request finishes faster than real time at every clip length and prompt size: a 5-second clip with a 1K-token prompt in 3.1 s, a 10-second clip with a 1K-token prompt in 6.2 s and a 15-second clip with a 10K-token prompt in 14.1 s.

Throughput. Under load, UniServe’s best throughput is 24–33% above the baselines’ on four GB200 GPUs, 15–22% above on eight GB200 GPUs and 21–45% above on eight RTX PRO 6000 GPUs. For a synthetic workload that mixes 5-, 10- and 15-second clips, that is about 6,100 videos a day on four GB200 GPUs, 11,000 on eight and 3,400 on eight RTX PRO 6000 GPUs in BF16, or 5.8, 6.4 and 21.2 GPU-seconds per second of generated video.

NVFP4. FastVideo’s NVFP4 checkpoint lowers UniServe’s median latency by a further 10–15% and raises its throughput by 15–19%; NVFP4 explains what it quantizes and what it changes. The methodology and full results describe the workload and configurations and list every measured point.

Background: the FastH3 workload

Each request runs a Qwen3-VL-based text encoder, a two-layer text refiner, eight forward passes of a 50-layer DiT, and separate video and audio VAE decoders. The DiT attends over a joint sequence of text, video and audio tokens.

Video sparse attention (VSA) works on 64-token tiles. Text and audio queries attend to every key. Video queries attend to all text and audio keys and to the highest-scoring 20% of video key tiles (rounded up), plus a coarse branch over mean-pooled tiles whose output a learned gate adds to the fine output. UniServe implements this pattern in full.

At 1344 × 768 and 24 fps, a 10-second clip with a 1K-token prompt is about 74K tokens (video, audio and text). A 15-second clip with a 10K-token prompt is about 119K. At these lengths, every extra full-sequence intermediate tensor costs substantial memory traffic.

Architecture

UniServe separates the control plane, the numerical code and the execution resources. A Rust server and engine handle the HTTP API, admission control, component placement and batch scheduling. Python GPU workers run the model and own every execution resource: request buffers, CUDA streams, transfers and CUDA graphs.

The example deployment for four GB200 GPUs runs the text encoder with four-way tensor parallelism (TP4) and the DiT with four-way Ulysses sequence parallelism (SP4). The video and audio decoders distribute temporal units (the groups of latent frames the VAE decodes together) across the same four GPUs. Placing each component separately matters because the text encoder, the long-sequence DiT and the tiled VAE decoder parallelize along different axes: raising the DiT’s sequence-parallel degree speeds up only part of the request path.

Each request’s conditioning occupies a preallocated request slot, and its latents are double-buffered in two banks of pages: a denoising step reads the current latent from one bank and writes the next latent into the other, and the runtime commits the result when the batch completes. CUDA graphs are captured at startup for each shape bucket and address request slots and latent pages through device-side index tensors, so every request in the same bucket replays the same graphs on its own data.

The diffusion runner allocates its workspace once, for the largest supported shape, and smaller shapes run on views of that allocation. Constants and step graphs stay resident in worker-owned memory pools, so serving a request neither allocates a new workspace nor captures a new graph. The runner orders the calls that share scratch memory; the worker owns collectives and resource lifetimes.

Precompute schedule-dependent values

Precompute AdaLN modulation

Each DiT block projects the timestep embedding into per-modality shift, scale and gate vectors (AdaLN modulation). These depend only on the weights and the timestep, not on the prompt or the latent. Since the checkpoint fixes its eight timesteps, UniServe computes every modulation vector at load time and keeps only the results.

Across 50 layers, eight steps and separate video and audio timesteps, the BF16 table is about 148 MiB. The projection weights it replaces occupy about 24 GiB in the checkpoint. UniServe streams those weights through the precomputation at load time and frees them afterwards.

Run the text refiner once

The two-layer text refiner depends only on the encoded prompt, not on the timestep. UniServe runs it once per request and reuses its output across all eight DiT forwards.

Head-sharded projections for Ulysses

Standard Ulysses sequence parallelism starts with each GPU holding a shard of the sequence. In FastVideo’s Ulysses path, each GPU applies the full, replicated Q, K, V and gate projections to its local tokens; an all-to-all then switches from sequence sharding to head sharding, so attention sees the full sequence for the local heads. A second all-to-all switches back to sequence sharding before the output projection.

UniServe changes the first half of this data flow. It shards the Q/K/V/gate weights by attention head, all-gathers the hidden states, which are narrower than the projected Q/K/V/gate tensors, and runs one merged projection for its 14 of the 56 heads over the full sequence. Local rows are projected immediately. Remote rows arrive through chunked asynchronous all-gathers, and each chunk enters the projection and attention preprocessing as soon as it lands while later chunks are still in flight.

For a padded sequence length $S$, hidden size $H=5376$ and attention width $A=56 \times 128=7168$, each GPU receives the following number of BF16 elements in the input collective:

With SP4, the hidden-state all-gather moves 75% of the data of the projected all-to-all, the ratio H/A. The output all-to-all is unchanged.

Head sharding also reduces weight memory. The 200 Q/K/V/gate matrices occupy about 14.4 GiB per GPU when replicated and 3.6 GiB per GPU when sharded four ways. The total projection FLOPs are unchanged: the work is repartitioned from local tokens × all heads to all tokens × local heads.

Kernel fusion in the DiT

Fused attention preprocessing

An unfused path runs Q/K RMSNorm, RoPE, gathering the valid rows, filling the padding, tile pooling and layout transposes as separate operations, and each materialized full-sequence tensor adds another round trip through memory.

UniServe’s preprocessing kernel performs the learned Q/K RMSNorm, partial RoPE, tile pooling over valid tokens, tile packing and gate copying in a single pass per projected chunk. It writes the fine-attention inputs and the FP32 pooled statistics directly into the buffers the attention kernels read. Padding is handled inside this kernel, so it never becomes an input to later computation. Top-k tile selection and the fine and coarse attention kernels remain separate.

The kernel runs once per chunk rather than once per transformer block, so preprocessing overlaps communication: the all-gathers run on a separate communication stream, and while the compute stream projects and prepares one chunk, later chunks are still in flight. The wider fusion also avoids the intermediate full-sequence tensors that an unfused RMSNorm / RoPE / pack / pool / transpose path would materialize.

Fused output combination

The fine sparse-attention output and the gated coarse output combine as:

UniServe’s combination kernel evaluates this expression and writes each row directly into its position in the send buffer of the output all-to-all. This fuses the combination with the layout conversion, avoiding a separately materialized combined tensor followed by a permute-and-copy. The all-to-all then restores the sequence sharding that the output projection and residual path consume.

This all-to-all is chunked and asynchronous as well. The combination kernel fills the send buffer one row interval at a time, and each interval’s all-to-all starts on the communication stream as soon as its rows are written. The output projection, gated residual and MLP then consume completed intervals while later intervals are still in flight.

Fused modulation, normalization and residuals

Each block applies AdaLN modulation around RMSNorm and a gated residual update. Schematically, the attention input is RMSNorm(x) * (1 + scale_attn) + shift_attn. After attention, the residual becomes r = x + gate_attn * attention_output, and the MLP input is the modulated RMSNorm of r.

UniServe fuses these steps. The pre-attention kernel looks up each token’s modulation row, selected by its modality and the current timestep, while it normalizes, and saves the residual input in the same pass as chunks arrive. The post-attention kernel produces both the gated residual and the modulated, normalized MLP input. A fused gate/up projection with SwiGLU feeds the down projection, followed by the MLP’s gated residual update. In the BF16 path, finished chunks flow into the next stage without first concatenating the whole shard.

A batched, streaming decode path

Batch spatial tiles, distribute temporal units

The video VAE decoder reconstructs each temporal unit from overlapping spatial tiles. At 1344 × 768 the tile grid is 4 × 7: 28 tiles with a 256-pixel minimum tile size and a 64-pixel overlap. A straightforward implementation loops over the tiles and calls the decoder once per tile. Parallelizing the DiT leaves this loop untouched: in the naive baseline’s trace, all video decoder kernels still run on GPU 0.

UniServe concatenates a unit’s 28 tiles along the batch dimension and decodes them in one call, then restores the grid and blends the overlaps. Tiles stay independent samples in the batch. The worker distributes temporal units across the four GPUs.

For the 10-second sample’s 14 temporal units, the traces show 14 decoder calls in UniServe, versus 392 per-tile calls in FastVideo and SGLang. Each batched call amortizes the host-side launch work over a whole unit.

Fuse the VAE decoder and replay it as a CUDA graph

Batching combines with kernel fusion inside the VAE decoder. UniServe’s video decoder transformer fuses Q/K normalization with RoPE, fuses the scaled attention residual with the following RMSNorm, and defers each MLP residual addition into the next layer’s normalization; the last layer folds its pending addition into the output LayerNorm. The normalization reads the FP32 residual sum, so FP16 projection weights do not round the residual stream to FP16. SwiGLU and the final unpatchify step, which turns tokens back into pixels, have their own fused kernels.

The worker captures the batched decode as a CUDA graph and replays it for each temporal unit. Capturing the whole batched decode removes the host round-trips of launching thousands of small kernels per request.

The video decode spans about 1.1 s on the GPUs in UniServe and 4.8 s in FastVideo. UniServe keeps the GPU busy for almost that entire span; FastVideo spends much of its 4.8 s waiting on the host.

Encode while the GPU is still decoding

A video request is not finished when its pixels are reconstructed. UniServe copies each decoded RGB unit to host memory and schedules its encoding independently of the remaining GPU decoding. On the same host, the encoder reads the published host buffer directly instead of making another copy. Four CPU encoder processes encode units with libx264 while the GPUs continue decoding. The audio decoder produces PCM, which the host encodes to AAC. The muxer appends the already encoded video packets with adjusted timestamps and adds the audio track, without decoding or re-encoding any video.

The trace shows the overlap: encoding starts while later units are still decoding, and the complete MP4 arrives about 100 ms after the last video kernel.

vLLM-Omni offers a similar option, preencode_mp4, which encodes each decoded chunk on the worker while later chunks decode; the vLLM-Omni benchmark results below use it. FastVideo and SGLang encode the MP4 after decoding finishes.

Where the time goes

The following traces use the same 10 s/1K prompt and seed on four GB200 GPUs, one request at a time, after warmup.

Decode covers video decoding, audio decoding and the final reconstruction steps; it excludes the GPU frame conversion performed by the output encoder. The unprofiled column is a warmed request before tracing starts.

NVFP4

The optimizations above change where and when the same computation runs. Precision is a separate choice, set per component when the server starts. The default preset, quality, runs the DiT and text encoder in BF16 and the video-decoder projections in FP16; the BF16 results in this post use it. balanced stores the video decoder’s projections in NVFP4. performance also runs the DiT MLP in FP8, and maximum runs it in NVFP4 with an FP8 text encoder; attention stays in BF16 in every preset.

To go faster still, we also publish an NVFP4¹ post-training-quantized (PTQ) checkpoint, FastVideo/FastVideo-FastH3-8-Step-V2-NVFP4, which UniServe serves natively.

The checkpoint quantizes 150 DiT MLP projections (gate, up and down in each of the 50 layers) and 252 video-VAE decoder projections (Q/K/V, attention output and the three MLP projections in each of 36 layers). The text encoder and the DiT’s attention projections stay in BF16. UniServe loads the packed weights as they are stored, two E2M1 values per byte plus the block scales, and quantizes activations on the fly against each module’s calibrated global scale from the checkpoint, so every chunk of rows uses the same scale no matter how the sequence is split. The packed values and block scales go directly into a block-scaled FP4 GEMM that produces BF16 outputs.

On a 10 s/1K request, DiT time on GPU 0 falls from 10.2 s with BF16 to 8.5 s, and video decoder time from 1.2 s to 0.8 s. End to end, NVFP4 lowers UniServe’s median latency by about 10–15% and raises its throughput by 15–19% across the three hardware configurations; see NVFP4 against BF16.

The clips compare UniServe’s BF16 (left) and NVFP4 (right) outputs for the same benchmark requests, 10 seconds with 1K-token prompts, from the eight-RTX PRO 6000 latency runs.

Serving with NVIDIA Dynamo

UniServe also runs as an experimental backend for NVIDIA Dynamo. Dynamo’s frontend serves the OpenAI-style /v1/videos endpoint and handles service discovery and request routing.

uniserve-dynamo-worker embeds UniServe’s scheduler in a Dynamo worker and serves FastH3 through Dynamo’s /v1/videos endpoint. The Dynamo quickstart shows how to run it on four GB200 GPUs.

Deploy UniServe

Install UniServe as in its README, then choose a model and GPUs to get the deployment file and the commands that serve it. The serving guide covers every other option.

Benchmark methodology and full results

This section describes how we measured and lists every result behind the charts.

Baselines

We compare UniServe against two kinds of baselines.

Naive. To show what UniServe’s optimizations contribute, we build a naive baseline from the pinned FastVideo implementation. It already includes the standard optimizations: Ulysses SP4 denoising, TP4 text encoding, resident weights, the native VSA kernel, one text-refiner pass per request and FP16 VAE decoder projections. FastVideo’s optional fusions, torch.compile, AdaLN precomputation and parallel VAE decoding are disabled.

Tuned. FastVideo, vLLM-Omni and SGLang each run a fast configuration of the same checkpoint: native SM100a VSA on GB200, resident GPU weights, and the same libx264 encode settings. Setup lists the per-system settings. Elsewhere in this post, “the baselines” means these three tuned systems.

Setup

Hardware. (1) 4 × GB200 GPUs in one node; (2) 8 × GB200 GPUs across two nodes connected by multi-node NVLink (MNNVL); (3) 8 × RTX PRO 6000 Blackwell Server Edition GPUs in one node.

Systems and checkpoints. Every system serves the BF16 checkpoint FastVideo/FastVideo-FastH3-8-Step-V2 while only UniServe also serves the NVFP4 checkpoint FastVideo/FastVideo-FastH3-8-Step-V2-NVFP4. All systems run the full eight-step schedule, the checkpoint’s sparse attention, and the full video and audio decoders. Compilation, fused kernels, parallelism and overlapping media processing are in scope.

Supported configurations. On GB200, vLLM-Omni and SGLang use the native SM100a VSA kernel. † vLLM-Omni cannot run one replica across two hosts, so it has no eight-GB200 latency result. NVFP4 is UniServe-only: the other systems do not load this checkpoint natively.

Layouts. We tuned the layouts once and fixed them for every system:

Layouts. r is the number of replicas and s the DiT’s sequence-parallel degree; the text encoder is tensor-parallel across each replica’s GPUs.

Baseline configuration. Within a layout, every system keeps DiT weights resident on the GPU, with no offloading to host memory and no tensor-parallel sharding. Every system also runs the text encoder tensor-parallel across its replica, keeps the video VAE decoder projections in FP16, does not load components that text-to-video does not use, and encodes H.264 with libx264 ultrafast. The settings that differ:

Per-system settings that differ within each layout.

Workload. Six scene families, each with a 1K-token and a 10K-token prompt, at 5-, 10- and 15-second clip lengths. Prompts, seeds and execution order are the same for every system.

Metrics. Latency is 72 requests at concurrency 1 (six scenes × two prompt lengths × three durations × two seeds). We report p50 / p90 / p95 over all 72 and the median of the 12 requests of each shape. Throughput is 32 requests per concurrency level (C = 1, 2, 4, 8, and 16 on eight GPUs), counted as valid MP4s per second from the first submission to the last response.

Latency

Four GB200 GPUs. Across the 72-request workload, UniServe BF16 has a median end-to-end latency of 14.24 s, compared with 17.80 s for SGLang, 18.17 s for FastVideo and 18.83 s for vLLM-Omni. Across the six shapes, SGLang’s median is 1.18 to 1.30 times UniServe’s, FastVideo’s 1.23 to 1.33 times and vLLM-Omni’s 1.28 to 1.37 times.

Eight GB200 GPUs across two nodes. With one eight-GPU replica spanning both nodes, UniServe BF16 has a median latency of 7.51 s, 47% below its four-GPU median, compared with 9.79 s for SGLang and 13.02 s for FastVideo. SGLang’s per-shape medians are 1.24 to 1.35 times UniServe’s and FastVideo’s 1.52 to 1.88 times.

Eight RTX PRO 6000 GPUs. UniServe BF16 has a median latency of 35.57 s, compared with 43.38 s for SGLang, 49.76 s for FastVideo and 51.35 s for vLLM-Omni. The margin grows with duration and prompt length. SGLang’s median is 0.96 times UniServe’s for 5-second videos with 1K-token prompts, where SGLang is 0.85 s faster, and 1.30 times for 15-second videos with 10K-token prompts; FastVideo’s ratio ranges from 1.15 to 1.45 and vLLM-Omni’s from 1.12 to 1.55. No system uses an SM100a kernel on this hardware: UniServe runs FlashInfer’s SM120 block-sparse attention kernel, and the others run their Triton VSA kernels.

Throughput

On four GB200 GPUs with one replica, every system’s throughput is essentially flat across concurrency: higher concurrency adds queueing latency without adding throughput. At each system’s best concurrency, UniServe BF16’s throughput is 24.0% above SGLang’s, 32.6% above vLLM-Omni’s and 32.7% above FastVideo’s. With two replicas on eight GPUs, C=1 leaves one replica idle and both are busy from C=2. On two GB200 nodes, UniServe BF16 reaches 0.1283 videos/s, 14.9% above SGLang’s best (0.1117 videos/s at C=8), 20.3% above FastVideo’s (0.1067 videos/s at C=4) and 22.0% above vLLM-Omni’s (0.1052 videos/s at C=8). On the RTX PRO 6000, UniServe BF16 reaches 0.0389 videos/s, 20.8% above SGLang’s best (0.0322 videos/s at C=4), 39.9% above FastVideo’s (0.0278 videos/s at C=4) and 45.4% above vLLM-Omni’s (0.0267 videos/s at C=4). Each cell below gives throughput in valid videos per second, followed by p50 / p95 end-to-end latency in seconds, for one 32-request run.

NVFP4 against BF16

UniServe BF16 and NVFP4 in the same layouts. NVFP4 is a separate precision tier; see NVFP4.

Meet the Team

Yi Sun
Will Lin
Hao Zhang

Acknowledgments

FastVideo FastH3 builds on MiniMax H3. We thank the MiniMax team for releasing its weights and code.

We thank Nuva Lab and Reactor for their collaboration, feedback and insights during UniServe’s development, and Nuva Lab for access to the production hardware used in our benchmarks.

We thank Guan Luo, Qi Wang, Ryan McCormick, Harry Kim, and the rest of the NVIDIA Dynamo team for their support in integrating Dynamo with UniServe, bringing UniServe closer to a production-grade serving engine.

We also thank NVIDIA FastGen (Julius Berner, Chao Liu, Arash Vahdat), and NVIDIA Enterprise Products (Pengcheng Li, Cliff Woolley) team for collaborating on FastH3.

Finally, we are extremely grateful to the vLLM project, NVIDIA and MBZUAI for their continued sponsorship and support of FastVideo.