minimax_h3_checkpoint_nvfp4 ¶

Serialized NVFP4 execution for the MiniMax-H3 Qwen3-VL encoder.

The bf16 conditioner is 48 GB resident and sets the single-GB10 peak once the DiT runs in FP8. A checkpoint written by scripts/checkpoint_conversion/convert_minimax_h3_text_encoder_nvfp4.py stores every language_model.layers.* linear as three tensors and no weight::

<prefix>.weight_packed        uint8   [out, in // 2]   two E2M1 values per byte
<prefix>.weight_scale         uint8   [out, in // 16]  one E4M3 scale per 16 values,
                                                       FlashInfer 128x4 swizzled layout
<prefix>.weight_global_scale  float32 [1]              (448 * 6) / amax(|W|)

The bytes are exactly what flashinfer.nvfp4_quantize(W, global_scale, sfLayout=SfLayout.layout_128x4) returns, so loading them reproduces the state nvfp4_config.convert_model_to_nvfp4 builds at runtime without ever materializing the bf16 weight. Activations are quantized per call with a unit global scale and multiplied with flashinfer.mm_fp4.

The checkpoint selects this path through config.json; every field is required so a checkpoint from another exporter cannot pass by omission::

"quantization_config": {"quant_method": "nvfp4", "activation_scheme": "dynamic",
                        "fmt": "e2m1", "group_size": 16, "scale_fmt": "e4m3",
                        "scale_layout": "128x4",
                        "modules_to_not_convert": ["model.visual", "lm_head"]}

Whole projection kinds may stay bf16 by listing their suffix in modules_to_not_convert (for example "mlp.down_proj"); those linears keep a plain weight and the unquantized method. Only the seven language projection names are accepted there, so a typo fails at config time.

Single GPU only: packed columns and swizzled scale rows cannot be narrowed per tensor-parallel rank without repacking. Missing tensors are reported by the loader's strict check before the post-load hook runs; the hook adds only the content checks a copied tensor can still fail.

Classes¶

fastvideo.models.encoders.minimax_h3_checkpoint_nvfp4.MiniMaxH3SerializedNVFP4Config ¶

MiniMaxH3SerializedNVFP4Config(bf16_projections: tuple[str, ...] = ())

Bases: QuantizationConfig

Serialized 16-group NVFP4 contract for the H3 text encoder.

The group size and scale layout are fixed by the loader's parameter shapes, so the only state is which projection kinds the checkpoint kept in bf16.

Source code in fastvideo/models/encoders/minimax_h3_checkpoint_nvfp4.py

def __init__(self, bf16_projections: tuple[str, ...] = ()) -> None:
    super().__init__()
    unknown = sorted(set(bf16_projections) - set(LANGUAGE_PROJECTIONS))
    if unknown:
        raise ValueError(f"MiniMax-H3 serialized NVFP4 cannot keep unknown projection kinds {unknown} in bf16; "
                         f"choose from {LANGUAGE_PROJECTIONS}")
    # Projection kinds the checkpoint kept in bf16 in every language layer,
    # e.g. ("mlp.down_proj",). Those linears load a plain weight.
    self.bf16_projections = tuple(bf16_projections)

fastvideo.models.encoders.minimax_h3_checkpoint_nvfp4.MiniMaxH3SerializedNVFP4LinearMethod ¶

Bases: LinearMethodBase

Execute serialized NVFP4 weights without re-quantizing them.

Methods:¶

fastvideo.models.encoders.minimax_h3_checkpoint_nvfp4.MiniMaxH3SerializedNVFP4LinearMethod.process_weights_after_loading ¶

process_weights_after_loading(layer: Module) -> None

Reject a layer the checkpoint did not fill and derive the GEMM multiplier.

Shapes and dtypes are fixed by create_weights and enforced by the weight loader's copy, and the loader reports missing tensors by name before this runs; what remains is content a copied tensor can still get wrong: a non-positive global scale, or a scale tile left at its 0xFF initializer by a direct caller that bypassed the loader.

Source code in fastvideo/models/encoders/minimax_h3_checkpoint_nvfp4.py

def process_weights_after_loading(self, layer: nn.Module) -> None:
    """Reject a layer the checkpoint did not fill and derive the GEMM multiplier.

    Shapes and dtypes are fixed by ``create_weights`` and enforced by the
    weight loader's copy, and the loader reports missing tensors by name
    before this runs; what remains is content a copied tensor can still get
    wrong: a non-positive global scale, or a scale tile left at its 0xFF
    initializer by a direct caller that bypassed the loader.
    """
    weight_scale = layer.weight_scale
    global_scale = float(layer.weight_global_scale.item())
    if not (global_scale > 0) or global_scale == float("inf"):
        raise ValueError("Serialized MiniMax-H3 NVFP4 weight_global_scale was not loaded: "
                         f"it must be a finite positive value, got {global_scale}")
    if bool((weight_scale.view(-1)[:_SCALE_TILE_BYTES] == _UNLOADED_SCALE_BYTE).all()):
        raise ValueError("Serialized MiniMax-H3 NVFP4 weight_scale was not loaded: its first tile is still 0xFF")
    # ``mm_fp4`` folds both global scales into one multiplier. Activations use a
    # unit global scale, so the multiplier is the inverse weight global scale.
    device = weight_scale.device
    layer.register_buffer("_nvfp4_alpha", torch.tensor(1.0 / global_scale, dtype=torch.float32, device=device),
                          persistent=False)
    layer.register_buffer("_nvfp4_x_global_scale", torch.ones((), dtype=torch.float32, device=device),
                          persistent=False)

Functions:¶

fastvideo.models.encoders.minimax_h3_checkpoint_nvfp4.nvfp4_weight_global_scale ¶

nvfp4_weight_global_scale(weight: Tensor) -> Tensor

The per-tensor scale convert_model_to_nvfp4 uses: E4M3 max times E2M1 max over amax.

Unlike the runtime converter this refuses NaN or infinite weights instead of mapping them to zero, because a checkpoint written from them would be wrong forever.

Source code in fastvideo/models/encoders/minimax_h3_checkpoint_nvfp4.py

def nvfp4_weight_global_scale(weight: torch.Tensor) -> torch.Tensor:
    """The per-tensor scale ``convert_model_to_nvfp4`` uses: E4M3 max times E2M1 max over amax.

    Unlike the runtime converter this refuses NaN or infinite weights instead of
    mapping them to zero, because a checkpoint written from them would be wrong
    forever.
    """
    amax = weight.float().abs().max()
    if not torch.isfinite(amax) or amax <= 0:
        raise ValueError("MiniMax-H3 NVFP4 global scale needs a finite, non-zero weight amax")
    scale = ((_E4M3_MAX * _E2M1_MAX) / amax).to(torch.float32)
    if not torch.isfinite(scale):
        raise ValueError(f"MiniMax-H3 NVFP4 global scale overflowed float32 for weight amax {amax.item():.3e}")
    return scale

fastvideo.models.encoders.minimax_h3_checkpoint_nvfp4.serialized_nvfp4_quantization_config ¶

serialized_nvfp4_quantization_config(*, keep_bf16: tuple[str, ...] | list[str] = (), producer: dict[str, Any] | None = None) -> dict[str, Any]

The config.json quantization_config the converter writes and from_config accepts.

keep_bf16 lists projection kinds from LANGUAGE_PROJECTIONS that stay bf16 in every language layer; they are appended to modules_to_not_convert.

Source code in fastvideo/models/encoders/minimax_h3_checkpoint_nvfp4.py

def serialized_nvfp4_quantization_config(
    *,
    keep_bf16: tuple[str, ...] | list[str] = (),
    producer: dict[str, Any] | None = None,
) -> dict[str, Any]:
    """The ``config.json`` ``quantization_config`` the converter writes and ``from_config`` accepts.

    ``keep_bf16`` lists projection kinds from ``LANGUAGE_PROJECTIONS`` that stay
    bf16 in every language layer; they are appended to ``modules_to_not_convert``.
    """
    unknown = sorted(set(keep_bf16) - set(LANGUAGE_PROJECTIONS))
    if unknown:
        raise ValueError(f"keep_bf16 names unknown projection kinds {unknown}; choose from {LANGUAGE_PROJECTIONS}")
    config: dict[str, Any] = {
        "quant_method": "nvfp4",
        "activation_scheme": "dynamic",
        "fmt": "e2m1",
        "group_size": NVFP4_GROUP_SIZE,
        "scale_fmt": "e4m3",
        "scale_layout": NVFP4_SCALE_LAYOUT,
        "modules_to_not_convert": ["model.visual", "lm_head", *keep_bf16],
    }
    if producer:
        config["producer"] = dict(producer)
    return config