Slack

FastVideo is open-sourcing FastH3 Omni, a distilled MiniMax H3 model for reference-guided video and audio. Give it a prompt plus images, video clips and a voice reference to generate a finished video with synchronized sound.

Model weights · UniServe deployment · Local AI with FastVideo

ComfyUI support is available. Our local AI blog post and release are planned for tomorrow, October 21.

This entire video was generated with FastH3 Omni on FastVideo.

TL;DR

  • Eight transformer evaluations, down from Base’s 50, with image, video and audio references.
  • 5.6–11.9× lower end-to-end latency across 27 matched workloads on eight GB200 GPUs. This compares full serving recipes, using Base’s lossless numerical policy; methodology.
  • Real-time Generation on FastVideo UniServe: On particular FastH3 Omni tasks, FastVideo UniServe achieves real time generation speeds for 768p videos for 5s, 10s, and 15s clips.
  • Local AI. Run on DGX Spark or RTX 5090 with FastVideo, using the NVFP4 QAD weights and device recipes.
  • See and hear the outputs below. Change the references or clip length, then play the matched examples together. Reference inputs are expanded; open the full prompt when you need it.
  • Partners. Thanks to Nuva Lab, Reactor, NVIDIA FastGen Team, and NVIDIA Enterprise products for their collaboration and support.

Matched reference inputs and outputs

Choose a reference mix and clip length to compare Base, FastH3 Omni and FastH3 Omni with the 36-layer VAE. All examples use seed 0 and matched inputs; these are qualitative comparisons, not evidence of output parity.

8×GB200 · seed 0

Playback starts when you press play. Choose one soundtrack with Listen to.

1344×768 · 24 FPS · stereo audio. Downloads preserve the generated MP4s. Latencies are three-seed means; the displayed videos use seed 0.

The 36-layer VAE is shown for quality comparison; its latency is not included in the timing data.

Results at a glance

Across 27 matched workloads, FastH3 Omni delivers 5.6–11.9× lower end-to-end latency than Base on eight GB200 GPUs. Timings include reference encoding, generation, decoding and the finished MP4, at a 1344×768 target canvas.

Both lanes use the original video VAE. Base uses SGLang’s quality="lossless" policy, not its stricter exact policy. This compares complete model and serving recipes; see the benchmark methodology.

Select a row to watch its matched outputs.

Base · 50 evaluations FastH3 Omni · 8 evaluations
1 image 6.25× Image + voice 6.15× 3 images 5.88× 9 images 5.63× 1 short clip 6.98× 1 long clip 8.39× 3 clips 7.87× Image + video + voice 7.21× 9 images + 3 clips 5.88×

Mean latency, including audio and MP4 delivery. Both lanes use the original VAE.

Explore all 27 timings and their examples
1344×768, 24 FPS, eight GB200 GPUs. Generation time in seconds, mean ± sample standard deviation; lower is better. Target is the requested output length.
ReferencesTargetBaseFastH3 OmniBase / FastH3Prompt, inputs & outputs
1 image5 s35.44 ± 0.065.67 ± 0.026.25×View example ↓
1 image10 s78.53 ± 0.019.37 ± 0.018.38×View example ↓
1 image15 s139.69 ± 0.1214.14 ± 0.019.88×View example ↓
Image + voice5 s35.92 ± 0.075.84 ± 0.016.15×View example ↓
Image + voice10 s79.92 ± 0.069.71 ± 0.158.23×View example ↓
Image + voice15 s140.91 ± 0.0614.44 ± 0.009.76×View example ↓
3 images5 s73.09 ± 0.9812.42 ± 0.015.88×View example ↓
3 images10 s131.25 ± 0.0219.15 ± 0.066.85×View example ↓
3 images15 s206.88 ± 0.1426.53 ± 0.017.80×View example ↓
9 images5 s121.61 ± 0.0521.59 ± 0.015.63×View example ↓
9 images10 s193.95 ± 0.0530.80 ± 0.016.30×View example ↓
9 images15 s282.81 ± 0.0541.57 ± 0.016.80×View example ↓
1 short clip5 s36.06 ± 0.135.17 ± 0.016.98×View example ↓
1 short clip10 s79.73 ± 0.028.06 ± 0.019.89×View example ↓
1 short clip15 s140.29 ± 0.1711.76 ± 0.0111.93×View example ↓
1 long clip5 s69.93 ± 0.008.34 ± 0.008.39×View example ↓
1 long clip10 s139.27 ± 0.0213.51 ± 0.0310.31×View example ↓
1 long clip15 s215.75 ± 0.1718.41 ± 0.0411.72×View example ↓
3 clips5 s74.35 ± 0.099.45 ± 0.017.87×View example ↓
3 clips10 s132.62 ± 0.0613.55 ± 0.049.79×View example ↓
3 clips15 s206.44 ± 0.2118.38 ± 0.0111.23×View example ↓
Image + video + voice5 s94.14 ± 0.0713.06 ± 0.007.21×View example ↓
Image + video + voice10 s172.37 ± 0.1019.75 ± 0.008.73×View example ↓
Image + video + voice15 s257.71 ± 0.0626.71 ± 0.019.65×View example ↓
9 images + 3 clips5 s219.90 ± 0.1437.38 ± 0.035.88×View example ↓
9 images + 3 clips10 s314.16 ± 0.1348.38 ± 0.046.49×View example ↓
9 images + 3 clips15 s422.20 ± 0.2259.77 ± 0.057.06×View example ↓

Download the timing CSV or full benchmark JSON.

PDD + DMD

FastH3 Omni combines Parallel Decoding Distillation (PDD) with Distribution Matching Distillation (DMD). PDD predicts several denoising updates per transformer evaluation; DMD helps train the completed video and audio output. Reference conditioning is retained throughout generation. For the method and its illustrations, see the PDD paper.

  • PDD: many updates per forward. The student’s output heads are widened from one velocity prediction to a 32-interval fine time grid. At inference those heads run as eight fused blocks of four, so one transformer evaluation advances four intervals and the full trajectory takes eight evaluations instead of Base’s 50. A teacher-supervised head loss keeps every head’s update consistent with the frozen Base model.
  • DMD: match the finished output. On top of PDD, a trainable critic models the distribution of the student’s samples while the frozen Base teacher supplies the target; the student is updated to close the gap between them on the completed video and audio, not just on each individual step. Critic and student updates alternate, with several critic updates per student update.
  • References stay clean conditioning. Reference images, videos and audio are never noised: they condition the student, teacher and critic in every forward, and only the target video and audio enter the noising and the losses.
  • Sparse throughout. The student trains and samples with Video Sparse Attention on 128-row tiles, so the sparsity used for serving is the sparsity it was trained with.

Huge thanks to the NVIDIA FastGen team (Julius Berner, Chao Liu and Arash Vahdat), who introduced PDD and whose DMD2 framework and H3 reference work this model builds on. FastVideo’s PDD and DMPDD trainers adapt the PDD objectives from the FastGen repository to FastVideo’s MiniMax H3 model, sequence-parallel and checkpoint interfaces.

TrainingAmount
GPUs64 NVIDIA B300
Training steps3,500
Estimated computeApproximately 7,800 GPU-hours

The training code is in FastVideo: see the FastH3 Omni DMPDD training guide.

Local AI with FastVideo

FastH3 Omni also runs locally on NVIDIA DGX Spark and RTX 5090 through FastVideo. Start with the FastVideo cookbook and FastH3 Omni recipe builder, then follow the local inference guide for model setup and reference inputs.

DeviceWeights and inference recipe
DGX Spark (GB10, 128 GB unified memory)NVFP4 QAD weights · spark recipe
RTX 5090 (32 GB)NVFP4 QAD weights · rtx-5090 recipe

QAD means quantization-aware distillation: these weights are trained for NVFP4 execution on Blackwell. Install FastVideo with the DGX Spark setup guide or NVIDIA GPU setup guide. Both device profiles default to 480p; Spark keeps the model components resident, while the RTX 5090 profile offloads the text encoder and requires a precomputed AdaLN table. The local inference guide covers preparing that table, composing the model directory and selecting the decoder. Keep the recipe’s attention and precision settings.

For FP8, use the FastH3 Omni FP8 weights and the guide’s FP8 device profile; the RTX 4090 recipe shows layerwise offload for a device without FP4 support. Its text encoder must also match the hardware: the bundled NVFP4 encoder needs Blackwell, so other GPUs need compatible BF16 encoder components. NVFP4 QAD and FP8 are separate numerical variants; their outputs can differ from the BF16 release.

Optional NVFP4 VAE decoders provide a 36-layer quality tier and an 8-layer fast tier. Follow the guide’s --decoder setup; these local configurations have their own resolution, memory and precision choices, so the eight-GB200 timings above do not describe their speed.

Deploy FastH3 Omni with UniServe

Find the checkpoints in the FastH3 Omni weights collection.

The walkthrough below serves the same eight-forward FastH3 Omni student used in the gallery, with its original video VAE. It starts with one image and one audio reference on four GB200 GPUs, then shows the eight-GPU, two-host configuration for the larger reference mixes. The model’s task is ref2va: reference-conditioned video and audio.

Open the complete deployment walkthrough

Install UniServe and apply the large-index correction below for large reference mixes. For other hardware and deployment layouts, see the UniServe FastH3 guide.

1. Install UniServe and its GPU dependencies

Use Linux with the CUDA 13 development toolkit, a compatible NVIDIA driver, Python 3.12, and a C++ compiler. The commands below assume Ubuntu 24.04. Install FFmpeg and ffprobe as well: reference media is processed on the server. Check nvidia-smi and nvcc --version before building; a CUDA runtime image alone does not supply the compiler.

sudo apt-get update
sudo apt-get install -y \
  build-essential pkg-config cmake git curl ca-certificates \
  python3.12 python3.12-dev python3.12-venv ffmpeg

curl -LsSf https://astral.sh/uv/install.sh | sh
curl -sSf https://sh.rustup.rs | sh -s -- \
  -y --profile minimal --default-toolchain stable

export CUDA_HOME=/usr/local/cuda
export PATH="$HOME/.local/bin:$HOME/.cargo/bin:$CUDA_HOME/bin:$PATH"
export MAX_JOBS=2

git clone https://github.com/hao-ai-lab/uniserve.git
cd uniserve

export OMNI_BLOG="https://haoailab.com/blogs/fastvideo-omniref-update/"
curl --fail --location \
  "${OMNI_BLOG}deploy/uniserve-wide-index.patch" \
  --output omni-wide-index.patch
git apply --check omni-wide-index.patch
git apply omni-wide-index.patch

uv sync --locked --python /usr/bin/python3.12 --extra gpu
source .venv/bin/activate
uniserve serve --help
ffprobe -version

uv sync builds the Rust server, host launcher, worker IPC extension and native kernels, and installs the locked GPU providers. Keep the gpu extra when syncing this environment again. Building on a machine without visible GPUs also requires TORCH_CUDA_ARCH_LIST for the deployment hardware; for GB200/B200, use 10.0. The first model startup may compile additional kernels and prepare request layouts before reporting ready.

The patch widens activation indices before multiplying a block offset. Without it, large reference mixes can overflow signed 32-bit indexing and trigger an illegal GPU memory access. It changes indexing, not model weights, precision, reference geometry or the denoising schedule. Apply it on both hosts of a two-host deployment before installation.

2. Download the model weights

Keep the shell in the UniServe checkout. Authenticate with a Hugging Face account that has access to the model when required, then download it:

hf auth login

export OMNI_MODEL="$PWD/models/FastH3-Omni-8-Step-V1"
hf download FastVideo/FastVideo-FastH3-Omni-8-Step-V1 \
  --local-dir "$OMNI_MODEL"

The download supplies transformer_ref, the schedulers and fastvideo_inference.json. UniServe automatically downloads the required text encoder, tokenizer, processor and video/audio VAEs into the Hugging Face cache when they are missing. Keep that cache writable and allow network access during the first startup. Use the native UniServe-compatible model directory, including fastvideo_inference.json.

4. Start the reference-conditioned server

In the same terminal, select four local GPUs and launch the reference deployment:

CUDA_VISIBLE_DEVICES=0,1,2,3 NCCL_NVLS_ENABLE=0 \
uniserve serve "$OMNI_MODEL" \
  --workers configs/minimax_h3/ulysses4-reference.json \
  --served-model-name FastH3-Omni \
  --host 0.0.0.0 --port 8000 \
  --max-running-requests 2 \
  --max-video-seconds 5 \
  --video-resolutions 768p --video-aspect-ratios 16:9 \
  --max-model-len 8192 --video-text-capacities 8192 \
  --max-condition-rows 49152 \
  --mm-processor-kwargs '{"enlarge_references":true}' \
  --dtype bfloat16 --quantization-config '{"mode":"quality"}' \
  --mem-fraction-static 0.70 \
  --media-directory "$OMNI_MEDIA"

ulysses4-reference.json places transformer_ref, the text and reference encoders, and the video/audio decoders across the four GPUs. The CPU worker reads media and produces the finished MP4. enlarge_references matches the released reference sizing used for this article. The quality precision preset retains the original decoder; it is separate from SGLang’s request-quality policy discussed in the methodology.

This quickstart admits 768p landscape outputs up to five seconds, with bounded prompt and reference storage. Its capacities cover the image-plus-voice example, not every gallery composition. The checkpoint supplies the 32-interval PDD schedule, eight transformer forwards, 128-row VSA tiles and trained reference policy automatically. Do not add diffusion-step, CFG or SGLang-specific quality fields to a UniServe request.

5. Generate and download video with audio

Leave the server running. In a second terminal, return to the same checkout, activate the environment, and wait for readiness:

source .venv/bin/activate

curl --fail-with-body http://127.0.0.1:8000/health
curl --fail-with-body http://127.0.0.1:8000/v1/models
curl --fail-with-body http://127.0.0.1:8000/v1/capabilities \
  | python -m json.tool

Continue when /health succeeds. The capabilities response should advertise ref2va, the served canvases and duration limit, condition capacity and the checkpoint’s schedule. Startup preparation is excluded from the latency figures above; there is no fixed startup-time estimate for this walkthrough.

curl --fail-with-body \
  http://127.0.0.1:8000/v1/videos/sync \
  -H 'Content-Type: application/json' \
  --data-binary @omni-request.json \
  --output omni-output.mp4

ffprobe -v error \
  -show_entries stream=codec_name,width,height,r_frame_rate,sample_rate,channels \
  -of json omni-output.mp4

The response is the completed MP4, with 1344×768 H.264 video at 24 FPS and stereo 32-kHz AAC audio. A requested five seconds produces 124 frames under the model’s frame-alignment rule. Open the file locally to inspect both the video and soundtrack.

For your own input, keep the same request structure and change prompt, conditions and seed. Each condition has type, role: "reference" and uri; the prompt names the ordered references as <Picture 1>, <Video 1> and <Audio 1>. The model field must match FastH3-Omni. Requests use target.short_edge, target.aspect_ratio and target.duration_seconds; /v1/capabilities gives the limits of the running deployment.

For a job that should be polled, send the same JSON to POST /v1/videos, then inspect GET /v1/videos/{id} and download GET /v1/videos/{id}/content after its status becomes completed. Keep the server running until the job’s output has been downloaded.

6. Serve the larger mixes on eight GB200 GPUs

For four GPUs on each of two hosts, install the same UniServe source and environment on both, with identical absolute paths for the checkout, Python environment, model and Hugging Face cache. Share or replicate those model/cache directories and the media directory at the same paths. Use a network that connects the workers’ NCCL traffic and host-launcher connections.

Stop the four-GPU server before replacing it. On the head host, run from the same UniServe checkout with the paths from the quickstart:

source .venv/bin/activate
export OMNI_MODEL="$PWD/models/FastH3-Omni-8-Step-V1"
export OMNI_MEDIA="$PWD/omni-media"

CUDA_VISIBLE_DEVICES=0,1,2,3 NCCL_NVLS_ENABLE=0 \
uniserve serve "$OMNI_MODEL" \
  --workers configs/minimax_h3/ulysses8-reference-two-node.json \
  --host-identity rank-0 \
  --served-model-name FastH3-Omni \
  --host 0.0.0.0 --port 8000 \
  --max-running-requests 2 \
  --max-video-seconds 15 \
  --video-resolutions 768p --video-aspect-ratios 16:9 \
  --max-model-len 48128 \
  --video-text-capacities 4096,7168,8192,10240,14336,15360,22528,37888 \
  --max-condition-rows 79104 \
  --mm-processor-kwargs '{"enlarge_references":true}' \
  --dtype bfloat16 --quantization-config '{"mode":"quality"}' \
  --mem-fraction-static 0.70 \
  --media-directory "$OMNI_MEDIA"

The head logs awaiting a launcher with an address such as 0.0.0.0:PORT. Use the port in that log, not the HTTP port 8000. On the second host, activate the identical environment and connect using the head’s reachable IP or hostname:

source .venv/bin/activate
CUDA_VISIBLE_DEVICES=0,1,2,3 NCCL_NVLS_ENABLE=0 \
uniserve-host --head HEAD_HOST:PORT --host-identity rank-1

Replace HEAD_HOST:PORT with that head address and launcher port. Once the head becomes healthy, the same request endpoint works. The larger capacities accommodate this article’s nine reference compositions at requested lengths of 5, 10 and 15 seconds, including nine square images plus three two-second clips. They are capacities for these specific inputs, not a promise that every nine-image/three-video request fits.

For example, in another terminal in the head’s checkout, use the same helper to prepare the largest gallery profile:

source .venv/bin/activate
export OMNI_MEDIA="$PWD/omni-media"
export OMNI_BLOG="https://haoailab.com/blogs/fastvideo-omniref-update/"

python prepare_omni_request.py \
  --blog-url "$OMNI_BLOG" \
  --composition full --seconds 15 --seed 0 \
  --media-directory "$OMNI_MEDIA" \
  --output omni-full-15s.json

curl --fail-with-body \
  http://127.0.0.1:8000/v1/videos/sync \
  -H 'Content-Type: application/json' \
  --data-binary @omni-full-15s.json \
  --output omni-full-15s.mp4

The helper also accepts image, images3, images9, clip, video, clips3 and mixed. Keep FFmpeg and ffprobe on PATH, or set --ffmpeg /absolute/path/to/ffmpeg --ffprobe /absolute/path/to/ffprobe on the server. Keep the original VAE for the timing lane in this article. The gallery’s trained 36-layer decoder requires a separately composed checkpoint; selecting a generic quantization preset does not install that decoder.

Deployment checks
SymptomWhat to check
A checkpoint download returns 401 or 404Confirm the exact repository name, the release’s availability and the account used by hf auth login.
A component is missing at startupCheck network access, Hugging Face authentication and write access to the cache so UniServe can download the required files.
A reference file is refusedIts absolute path must exist under --media-directory on the server. Keep FFmpeg and ffprobe available to that process.
Startup runs out of memory or a request exceeds a boundUse the appropriate deployment and explicit duration, prompt and condition capacities. Larger or longer references need more storage; increasing a bound does not create GPU memory.
A two-host head waits for a launcherMatch rank-0/rank-1, connect to the logged launcher port, and check that both hosts use the same source, environment and absolute paths.

For a container deployment, follow the UniServe Docker build instructions, include FFmpeg in the image, and mount the model, Hugging Face cache and reference directory at the paths the server uses. Worker IPC also needs sufficient shared memory and pinned-memory access. The native installation above is the complete walkthrough for this article.

Benchmark methodology

Nine reference compositions × three clip lengths (5, 10 and 15 seconds), measured on eight GB200 GPUs. Each profile uses one warmup and three measured seeds; the gallery consistently shows seed 0.

Read measurement settings and numerical policy
  • Hardware and scope. Eight NVIDIA GB200 GPUs across two four-GPU trays. GB200 is our proxy for B200; B200 was not separately measured. These results describe the eight-GPU cohort.
  • Matched inputs. Nine reference compositions × three requested lengths, all at 1344×768 and 24 FPS. The aligned outputs contain 124, 243 or 362 frames. Images use the released enlarged-reference mode with a 2048-pixel short edge. Reference clips retain their real soundtracks and standalone audio remains a real input.
  • Measurement. One request at a time, one warmup with seed 42 for each profile, then measured seeds 0, 1 and 2. The tables report arithmetic mean client end-to-end time until the complete MP4 is available; the downloadable data also includes sample standard deviations. Model loading and server startup are excluded. No latency is inferred from missing cells or memory failures, and no timing is normalized to another workload.
  • Model and runtime. Base is MiniMax H3 Ref2VA with 50 denoising evaluations; FastH3 Omni uses the 32-interval PDD student with eight fused evaluations and VSA. The Base lane uses Ulysses 4 × Ring 2 with the native FlashAttention transformer backend; the student uses eight-way Ulysses. Conditioning caches are disabled. Model weights, attention patterns and serving implementations differ, so this compares complete model/runtime recipes, not an isolated distillation or kernel ablation.
  • Base numerical policy. These Base measurements use SGLang’s quality="lossless" policy with original unquantized weights, dense attention and the model’s mixed-precision rules. Its DiT normalization/modulation and VAE fusions can change intermediate rounding. They are not measurements of SGLang’s stricter quality="exact" policy, and reference-exact or perceptual equivalence has not been established. The original model is not uniformly BF16: it retains FP32-sensitive DiT operations and the released VAE’s mixed-precision execution. The displayed latencies have not been converted to an exact-policy estimate.
  • Reading the examples. The timing statistics use three seeds; the gallery consistently uses seed 0. A shared seed does not imply numerical or perceptual output parity across different models or runtimes. The examples support direct inspection and are not a quantitative quality evaluation.

Get started

Download the model weights from the FastH3 Omni collection, then follow the UniServe setup to generate video and audio from your own references.

Acknowledgements

FastH3 builds on MiniMax H3. We thank the MiniMax team for releasing its weights and code.

We thank the NVIDIA Enterprise Products team (Pengcheng Li and Cliff Woolley) for the Video Sparse Attention kernel used by FastH3. We thank the NVIDIA FastGen team (Julius Berner, Chao Liu and Arash Vahdat) for the DMD2 framework and H3 reference work, and for introducing Parallel Decoding Distillation.

We thank Nuva Lab for bringing production-grounded creative video workflows to FastH3, and the upstream serving, kernel and open-source communities whose implementations make these experiments possible. We also acknowledge Reactor for supporting FastH3 Omni, and the vLLM project, NVIDIA and MBZUAI for their support of FastVideo.