FastH3 OmniRef (Ref2VA) on Apple Silicon MLX

FastH3 OmniRef is a reference-conditioned PDD student: a prompt plus ordered image/video/audio references become one video+audio clip in the export’s fused transformer forwards. The Apple Silicon runtime runs it natively through MLX, the same way it runs the text-to-video-audio FastH3 checkpoints.

The CUDA counterpart is FastH3 OmniRef inference; it shares this page’s packed layout and export contract, so a clip is built from the same request on either runtime.

What you need

1. The FastH3 OmniRef PDD export FastVideo/FastVideo-FastH3-Omni-8-Step-V1 — transformer_ref/, the two scheduler configs and fastvideo_inference.json. Use this BF16 release: the MLX converter needs BF16 weights, so the packed NVFP4 CUDA export cannot be converted.
2. A base MiniMax-H3 snapshot — text_encoder/, tokenizer/, processor/, vae/, audio_vae/. The export does not carry these.
3. An MLX conversion of transformer_ref. The runtime reads a packed MLX checkpoint, not the diffusers shards. There is no FP4 on Apple Silicon, so convert to INT8, INT6 or INT4 (the same weight-only formats the T2VA MLX path uses), or to dense BF16 on a large-memory Mac (see below).

Compose the first two into one directory of symlinks and convert the DiT:

python examples/inference/basic/mlx_fasth3_omniref.py --compose-only \
  --export ./FastVideo-FastH3-Omni-8-Step-V1 \
  --base-model-path ./MiniMax-H3 \
  --composed-dir ./FastH3-OmniRef-composed

python scripts/checkpoint_conversion/convert_minimax_h3_mlx.py \
  --model-root ./FastH3-OmniRef-composed/transformer_ref \
  --out ~/models/FastH3-OmniRef-MLX --formats int8 --include-vsa

The composed directory is not throwaway: the runtime reads the contract (fastvideo_inference.json) from it to resolve the trained step ladder, and the VAEs and text encoder from it for every request.

DiT formats, memory and GEMM path

convert_minimax_h3_mlx.py --formats takes int8, int6, int4 and bf16. bf16 is a dense, unquantized checkpoint: the same layout as the quantized formats (AdaLN table baked, its projection weights dropped, VSA gates kept with --include-vsa), with every matrix stored as BF16. Conversion needs roughly the output size in free memory while it saves.

Add the text encoder, VAEs and activations on top of the DiT figure. On a 96 GB-or-larger Mac, prefer bf16: dequantizing INT weights once at load and keeping them resident is the same footprint, so the BF16 checkpoint is the way to hold dense weights. Apple M5 accelerates dense BF16 and FP16 matmul on the GPU neural accelerators, which can make the dense path faster than any quantized one; whether it does on your machine is a measurement, not a given. Do not load a BF16 DiT on a Mac that cannot hold it.

Quantized checkpoints pick a linear path per layer. --quant-gemm (or FASTVIDEO_MLX_QUANT_GEMM) selects it; a bf16 checkpoint ignores it:

• auto (default): dequantize + dense GEMM for wide inputs, subject to FASTVIDEO_MLX_DQ_GEMM, and mx.quantized_matmul otherwise.
• dequant: dequantize + dense GEMM for every wide input, even if FASTVIDEO_MLX_DQ_GEMM=0.
• qmm: always mx.quantized_matmul.

The same flag is on the example: --quant-gemm {auto,dequant,qmm}.

Opt-in M5 attention backend

--vsa-impl sdpa evaluates the trained VSA tile mask through fused MLX SDPA, sharing each attention head’s K/V across its query tiles. On M5, the built-in matrix operations can be faster than the custom SIMD kernel even though they visit masked keys. The default auto selection is unchanged.

This backend keeps the reference resolution, selected tiles, ragged-row masks, and compression gate. Accumulation differs from the SIMD backend, so outputs are not bit-identical. It supports tiles 64/128 with head dimension 128 and raises if fused SDPA is unavailable. Prefix K/V reuse is not supported; do not combine it with --reuse-prefix-kv. Query tiles are processed in chunks of at most 128, capped at 256 MiB per boolean-mask chunk with two chunks in flight.

--vsa-impl nax is a separate opt-in backend for Apple M5 GPUs. It uses Metal TensorOps to visit only the selected key blocks and their valid rows, avoiding the dense mask and masked-key traversal of the SDPA backend. Dense prefix queries and the trained compression branch keep their existing implementations. The selected reference tokens, output resolution and denoising steps are unchanged.

The NAX backend uses private helper headers from the installed MLX package; it accepts only the header hashes validated with MLX 0.32.4. Changed headers need numerical and generation validation before their hashes are accepted. Missing headers, unsupported GPUs and compilation failures raise errors rather than silently selecting another backend. Tiles 64/128 and head dimension 128 are supported. Prefix K/V reuse is not supported. Floating-point accumulation differs from both SIMD and SDPA, so the generated clip can differ even with the same seed. auto remains unchanged.

Generate

python examples/inference/basic/mlx_fasth3_omniref.py \
  --model-root ./FastH3-OmniRef-composed \
  --mlx-checkpoint ~/models/FastH3-OmniRef-MLX/int8 \
  --image reference.png --audio voice.wav \
  --prompt "a person on a sofa, lit by a warm lamp, speaking to camera" \
  --output-path ~/fasth3_outputs/omniref.mp4

References are ordered: pass them in the order the model should read them. --video dance.mp4 --image outfit.png --audio voice.wav reads the video first, then the image, then the audio. At least one image or video reference is required; audio references accompany them.

--steps defaults to the contract’s transformer_forwards: one transformer forward per fused block of the export’s trained fine grid, exactly as on CUDA. Each step fuses that block’s widened output heads into their integration- weighted mean, so the heads never all run at once. Passing a different count is rejected, because the partition is the checkpoint’s.

The prompt is cached under --prompt-cache-dir for text-only requests; a reference-conditioned request re-encodes, because the vision tokens are part of the conditioning.

Recommended recipe: INT6, 480p, 5 s

This is the end-to-end flow the cookbook entry uses. It reads no base video VAE weights: the LightVAE encoder handles references and the FastH3 NVFP4 decoder renders the clip.

# BF16 release, the NVFP4 text-encoder stack, and the standalone VAE files.
hf download FastVideo/FastVideo-FastH3-Omni-8-Step-V1 \
  --local-dir ./FastVideo-FastH3-Omni-8-Step-V1
hf download FastVideo/FastH3-OmniRef-NVFP4-QAD \
  --include 'text_encoder/*' 'tokenizer/*' 'processor/*' 'vae/config.json' 'audio_vae/*' \
  --local-dir ./FastH3-OmniRef-NVFP4-QAD
hf download lightx2v/LightVAE lightvae-lite-h3-encoder.safetensors --local-dir ./LightVAE
hf download FastVideo/FastH3-NVFP4-VAE-Decoder \
  --include 'quality-36L/decoder-nvfp4.safetensors' 'fast-8L/decoder-nvfp4.safetensors' \
  --local-dir ./FastH3-NVFP4-VAE-Decoder

# Compose, convert the DiT to INT6 with VSA gates, and cache the NVFP4 text encoder.
python examples/inference/basic/mlx_fasth3_omniref.py --compose-only \
  --export ./FastVideo-FastH3-Omni-8-Step-V1 \
  --base-model-path ./FastH3-OmniRef-NVFP4-QAD \
  --composed-dir ./FastH3-OmniRef-composed
python scripts/checkpoint_conversion/convert_minimax_h3_mlx.py \
  --model-root ./FastH3-OmniRef-composed/transformer_ref \
  --out ./FastH3-OmniRef-MLX --formats int6 --include-vsa \
  --nvfp4-conditioner-root ./FastH3-OmniRef-NVFP4-QAD/text_encoder
ln -sfn "$PWD/FastH3-OmniRef-MLX/nvfp4-encoder" ./FastH3-OmniRef-composed/text_encoder

python examples/inference/basic/mlx_fasth3_omniref.py \
  --model-root ./FastH3-OmniRef-composed \
  --mlx-checkpoint ./FastH3-OmniRef-MLX/int6 \
  --image assets/images/couple.jpg \
  --prompt "A person on a sofa, lit by a warm lamp, speaking to camera." \
  --vae-encoder ./LightVAE/lightvae-lite-h3-encoder.safetensors \
  --vae-decoder ./FastH3-NVFP4-VAE-Decoder/quality-36L/decoder-nvfp4.safetensors \
  --vae-dtype bf16 --mlx-cache-limit-gib 8 --metal-wired-limit-gib 42 \
  --height 480 --width 832 --num-frames 124 --seed 2026 \
  --output-path ./outputs/fasth3_omniref_mlx.mp4

• fast-8L/decoder-nvfp4.safetensors is the smaller, faster decoder tier; quality-36L is the higher-quality one.
• Set --metal-wired-limit-gib to about two thirds of the Mac’s memory (42 for 64 GB, 24 for 36 GB). It does not add memory.
• 480p at 124 frames is roughly five seconds at 24 fps.
• Memory: the INT6 DiT takes about 17 GB. Peak memory measured on an M5 Max 64 GB was about 27-31 GB. The recipe is validated on an M4 Max 36 GB and an M5 Max 64 GB.

What this path does and does not do

The MLX runtime runs the same packed layout, the same reference conditioning and the same contract ladder as the CUDA path. Where it differs:

• No NVFP4. The MLX DiT is an INT8/INT6/INT4 weight-only conversion, and the text encoder and VAEs are the base snapshot’s rather than the NVFP4 ones the CUDA consumer stack prefers. An NVFP4 text encoder still works through the resident MLX encoder path if you have one.
• VSA follows the contract. With VSA on (the default for a VSA contract), the trained tile size, sparsity and p2_multi_region reference policy with its vsa_ref_keep_rate are read from fastvideo_inference.json. VSA needs the checkpoint converted with --include-vsa, like the T2VA path; without it the run falls back to dense attention with a warning.
• The prefix split has no MLX equivalent. The CUDA consumer route runs dense prefix query tiles at 8-bit Q/K to keep conditioning out of FP4 rounding; there is no FP4 kernel to split away from on Apple Silicon.
• Noise draws are not bit-identical. The seeded reference-posterior draw and the target noise use mx.random, so the same request gives the same clip on a given machine but not the same clip as the CUDA path.
• The VAE encoders load only for reference requests. They are needed to encode the references and are released after, which is why a reference-conditioned request carries a transient VAE-sized allocation.
• No fast temporal mode. --fast stretches the generated video clock, and the reference-prefixed packing does not stretch with it. A reference request that asks for both is rejected rather than quietly stretched at one end.

Not yet ported, and so absent rather than approximated: FL2VA keyframes through the MLX pipeline (--image on mlx_fasth3.py), two-pass refinement, and --video-decode-backend taeh3 for reference requests (the reference encode needs the real video VAE).

Video VAE without the base weights

--vae-decoder and --vae-encoder swap in standalone video VAE files, so the base vae/ shards are never read (its config.json is still read, unless the encoder is a LightVAE file, which embeds the config and latent statistics).

• Decoder: FastH3 quality-36L/decoder-nvfp4.safetensors (1.4 GB, loads as MLX NVFP4 without requantizing), decoder.safetensors (bf16 masters; add --vae-decoder-quantization nvfp4|mxfp8|int8|int4), or a LightVAE decoder.
• Encoder: LightVAE lightvae-lite-h3-encoder.safetensors (block-pruned).
• Use --vae-dtype bf16 for the decoder and fp16 for the encoder; fp32 is several times slower and larger. Decode needs the tiled path (the default): the decoder was trained on 256-pixel tiles.

When something goes wrong

• A reference-conditioned request needs the video VAE encoder — you passed --video-decode-backend taeh3. Use h3-vae.
• not a Ref2VA contract — the export is a T2VA one; the PDD ladder and the reference layout come from a model_type: ref2va contract.
• vsa_tile_size ... trained with — --vsa-tile-size differs from the contract’s trained tile. Leave it unset.
• VSA-H3 cannot run with this packed layout — the tile size does not fit the reference prefix. Drop --vsa to run dense.