--- slug: inference/fasth3_omniref_mlx title: FastH3 OmniRef (Ref2VA) on Apple Silicon MLX --- FastH3 OmniRef is a reference-conditioned PDD student: a prompt plus ordered image/video/audio references become one video+audio clip in the export's fused transformer forwards. The Apple Silicon runtime runs it natively through MLX, the same way it runs the text-to-video-audio FastH3 checkpoints. The CUDA counterpart is [FastH3 OmniRef inference](/FastVideo/inference/fasth3_omniref); it shares this page's packed layout and export contract, so a clip is built from the same request on either runtime. ## What you need 1. The **FastH3 OmniRef PDD export** [`FastVideo/FastVideo-FastH3-Omni-8-Step-V1`](https://huggingface.co/FastVideo/FastVideo-FastH3-Omni-8-Step-V1) — `transformer_ref/`, the two scheduler configs and `fastvideo_inference.json`. Use this BF16 release: the MLX converter needs BF16 weights, so the packed NVFP4 CUDA export cannot be converted. 2. A **base MiniMax-H3 snapshot** — `text_encoder/`, `tokenizer/`, `processor/`, `vae/`, `audio_vae/`. The export does not carry these. 3. An **MLX conversion of `transformer_ref`**. The runtime reads a packed MLX checkpoint, not the diffusers shards. There is no FP4 on Apple Silicon, so convert to INT8, INT6 or INT4 (the same weight-only formats the T2VA MLX path uses), or to dense BF16 on a large-memory Mac (see below). Compose the first two into one directory of symlinks and convert the DiT: ```bash python examples/inference/basic/mlx_fasth3_omniref.py --compose-only \ --export ./FastVideo-FastH3-Omni-8-Step-V1 \ --base-model-path ./MiniMax-H3 \ --composed-dir ./FastH3-OmniRef-composed python scripts/checkpoint_conversion/convert_minimax_h3_mlx.py \ --model-root ./FastH3-OmniRef-composed/transformer_ref \ --out ~/models/FastH3-OmniRef-MLX --formats int8 --include-vsa ``` The composed directory is not throwaway: the runtime reads the contract (`fastvideo_inference.json`) from it to resolve the trained step ladder, and the VAEs and text encoder from it for every request. ## DiT formats, memory and GEMM path `convert_minimax_h3_mlx.py --formats` takes `int8`, `int6`, `int4` and `bf16`. `bf16` is a dense, unquantized checkpoint: the same layout as the quantized formats (AdaLN table baked, its projection weights dropped, VSA gates kept with `--include-vsa`), with every matrix stored as BF16. Conversion needs roughly the output size in free memory while it saves. | Format | DiT body in memory (approx.) | Use it on | |--------|------------------------------|-----------| | `int6` | ~17 GB | 36 GB and up | | `int8` | ~20 GB | 48 GB and up | | `bf16` | ~38 GB, plus activations | 96 GB and up | Add the text encoder, VAEs and activations on top of the DiT figure. On a 96 GB-or-larger Mac, prefer `bf16`: dequantizing INT weights once at load and keeping them resident is the same footprint, so the BF16 checkpoint is the way to hold dense weights. Apple M5 accelerates dense BF16 and FP16 matmul on the GPU neural accelerators, which can make the dense path faster than any quantized one; whether it does on your machine is a measurement, not a given. Do not load a BF16 DiT on a Mac that cannot hold it. Quantized checkpoints pick a linear path per layer. `--quant-gemm` (or `FASTVIDEO_MLX_QUANT_GEMM`) selects it; a `bf16` checkpoint ignores it: - `auto` (default): dequantize + dense GEMM for wide inputs, subject to `FASTVIDEO_MLX_DQ_GEMM`, and `mx.quantized_matmul` otherwise. - `dequant`: dequantize + dense GEMM for every wide input, even if `FASTVIDEO_MLX_DQ_GEMM=0`. - `qmm`: always `mx.quantized_matmul`. The same flag is on the example: `--quant-gemm {auto,dequant,qmm}`. ### Opt-in M5 attention backend `--vsa-impl sdpa` evaluates the trained VSA tile mask through fused MLX SDPA, sharing each attention head's K/V across its query tiles. On M5, the built-in matrix operations can be faster than the custom SIMD kernel even though they visit masked keys. The default `auto` selection is unchanged. This backend keeps the reference resolution, selected tiles, ragged-row masks, and compression gate. Accumulation differs from the SIMD backend, so outputs are not bit-identical. It supports tiles 64/128 with head dimension 128 and raises if fused SDPA is unavailable. Prefix K/V reuse is not supported; do not combine it with `--reuse-prefix-kv`. Query tiles are processed in chunks of at most 128, capped at 256 MiB per boolean-mask chunk with two chunks in flight. `--vsa-impl nax` is a separate opt-in backend for Apple M5 GPUs. It uses Metal TensorOps to visit only the selected key blocks and their valid rows, avoiding the dense mask and masked-key traversal of the SDPA backend. Dense prefix queries and the trained compression branch keep their existing implementations. The selected reference tokens, output resolution and denoising steps are unchanged. The NAX backend uses private helper headers from the installed MLX package; it accepts only the header hashes validated with MLX 0.32.4. Changed headers need numerical and generation validation before their hashes are accepted. Missing headers, unsupported GPUs and compilation failures raise errors rather than silently selecting another backend. Tiles 64/128 and head dimension 128 are supported. Prefix K/V reuse is not supported. Floating-point accumulation differs from both SIMD and SDPA, so the generated clip can differ even with the same seed. `auto` remains unchanged. ## Generate ```bash python examples/inference/basic/mlx_fasth3_omniref.py \ --model-root ./FastH3-OmniRef-composed \ --mlx-checkpoint ~/models/FastH3-OmniRef-MLX/int8 \ --image reference.png --audio voice.wav \ --prompt "a person on a sofa, lit by a warm lamp, speaking to camera" \ --output-path ~/fasth3_outputs/omniref.mp4 ``` References are **ordered**: pass them in the order the model should read them. `--video dance.mp4 --image outfit.png --audio voice.wav` reads the video first, then the image, then the audio. At least one image or video reference is required; audio references accompany them. `--steps` defaults to the contract's `transformer_forwards`: one transformer forward per fused block of the export's trained fine grid, exactly as on CUDA. Each step fuses that block's widened output heads into their integration- weighted mean, so the heads never all run at once. Passing a different count is rejected, because the partition is the checkpoint's. The prompt is cached under `--prompt-cache-dir` **for text-only requests**; a reference-conditioned request re-encodes, because the vision tokens are part of the conditioning. ## Recommended recipe: INT6, 480p, 5 s This is the end-to-end flow the cookbook entry uses. It reads no base video VAE weights: the LightVAE encoder handles references and the FastH3 NVFP4 decoder renders the clip. ```bash # BF16 release, the NVFP4 text-encoder stack, and the standalone VAE files. hf download FastVideo/FastVideo-FastH3-Omni-8-Step-V1 \ --local-dir ./FastVideo-FastH3-Omni-8-Step-V1 hf download FastVideo/FastH3-OmniRef-NVFP4-QAD \ --include 'text_encoder/*' 'tokenizer/*' 'processor/*' 'vae/config.json' 'audio_vae/*' \ --local-dir ./FastH3-OmniRef-NVFP4-QAD hf download lightx2v/LightVAE lightvae-lite-h3-encoder.safetensors --local-dir ./LightVAE hf download FastVideo/FastH3-NVFP4-VAE-Decoder \ --include 'quality-36L/decoder-nvfp4.safetensors' 'fast-8L/decoder-nvfp4.safetensors' \ --local-dir ./FastH3-NVFP4-VAE-Decoder # Compose, convert the DiT to INT6 with VSA gates, and cache the NVFP4 text encoder. python examples/inference/basic/mlx_fasth3_omniref.py --compose-only \ --export ./FastVideo-FastH3-Omni-8-Step-V1 \ --base-model-path ./FastH3-OmniRef-NVFP4-QAD \ --composed-dir ./FastH3-OmniRef-composed python scripts/checkpoint_conversion/convert_minimax_h3_mlx.py \ --model-root ./FastH3-OmniRef-composed/transformer_ref \ --out ./FastH3-OmniRef-MLX --formats int6 --include-vsa \ --nvfp4-conditioner-root ./FastH3-OmniRef-NVFP4-QAD/text_encoder ln -sfn "$PWD/FastH3-OmniRef-MLX/nvfp4-encoder" ./FastH3-OmniRef-composed/text_encoder python examples/inference/basic/mlx_fasth3_omniref.py \ --model-root ./FastH3-OmniRef-composed \ --mlx-checkpoint ./FastH3-OmniRef-MLX/int6 \ --image assets/images/couple.jpg \ --prompt "A person on a sofa, lit by a warm lamp, speaking to camera." \ --vae-encoder ./LightVAE/lightvae-lite-h3-encoder.safetensors \ --vae-decoder ./FastH3-NVFP4-VAE-Decoder/quality-36L/decoder-nvfp4.safetensors \ --vae-dtype bf16 --mlx-cache-limit-gib 8 --metal-wired-limit-gib 42 \ --height 480 --width 832 --num-frames 124 --seed 2026 \ --output-path ./outputs/fasth3_omniref_mlx.mp4 ``` - `fast-8L/decoder-nvfp4.safetensors` is the smaller, faster decoder tier; `quality-36L` is the higher-quality one. - Set `--metal-wired-limit-gib` to about two thirds of the Mac's memory (42 for 64 GB, 24 for 36 GB). It does not add memory. - 480p at 124 frames is roughly five seconds at 24 fps. - Memory: the INT6 DiT takes about 17 GB. Peak memory measured on an M5 Max 64 GB was about 27-31 GB. The recipe is validated on an M4 Max 36 GB and an M5 Max 64 GB. ## What this path does and does not do The MLX runtime runs the same packed layout, the same reference conditioning and the same contract ladder as the CUDA path. Where it differs: - **No NVFP4.** The MLX DiT is an INT8/INT6/INT4 weight-only conversion, and the text encoder and VAEs are the base snapshot's rather than the NVFP4 ones the CUDA consumer stack prefers. An NVFP4 text encoder still works through the resident MLX encoder path if you have one. - **VSA follows the contract.** With VSA on (the default for a VSA contract), the trained tile size, sparsity and `p2_multi_region` reference policy with its `vsa_ref_keep_rate` are read from `fastvideo_inference.json`. VSA needs the checkpoint converted with `--include-vsa`, like the T2VA path; without it the run falls back to dense attention with a warning. - **The prefix split has no MLX equivalent.** The CUDA consumer route runs dense prefix query tiles at 8-bit Q/K to keep conditioning out of FP4 rounding; there is no FP4 kernel to split away from on Apple Silicon. - **Noise draws are not bit-identical.** The seeded reference-posterior draw and the target noise use `mx.random`, so the same request gives the same clip on a given machine but not the same clip as the CUDA path. - **The VAE encoders load only for reference requests.** They are needed to encode the references and are released after, which is why a reference-conditioned request carries a transient VAE-sized allocation. - **No fast temporal mode.** `--fast` stretches the generated video clock, and the reference-prefixed packing does not stretch with it. A reference request that asks for both is rejected rather than quietly stretched at one end. Not yet ported, and so absent rather than approximated: FL2VA keyframes through the MLX pipeline (`--image` on `mlx_fasth3.py`), two-pass refinement, and `--video-decode-backend taeh3` for reference requests (the reference encode needs the real video VAE). ## Video VAE without the base weights `--vae-decoder` and `--vae-encoder` swap in standalone video VAE files, so the base `vae/` shards are never read (its `config.json` is still read, unless the encoder is a LightVAE file, which embeds the config and latent statistics). - Decoder: FastH3 `quality-36L/decoder-nvfp4.safetensors` (1.4 GB, loads as MLX NVFP4 without requantizing), `decoder.safetensors` (bf16 masters; add `--vae-decoder-quantization nvfp4|mxfp8|int8|int4`), or a LightVAE decoder. - Encoder: LightVAE `lightvae-lite-h3-encoder.safetensors` (block-pruned). - Use `--vae-dtype bf16` for the decoder and `fp16` for the encoder; fp32 is several times slower and larger. Decode needs the tiled path (the default): the decoder was trained on 256-pixel tiles. ## When something goes wrong - **`A reference-conditioned request needs the video VAE encoder`** — you passed `--video-decode-backend taeh3`. Use `h3-vae`. - **`not a Ref2VA contract`** — the export is a T2VA one; the PDD ladder and the reference layout come from a `model_type: ref2va` contract. - **`vsa_tile_size ... trained with`** — `--vsa-tile-size` differs from the contract's trained tile. Leave it unset. - **`VSA-H3 cannot run with this packed layout`** — the tile size does not fit the reference prefix. Drop `--vsa` to run dense.