attn_qat_infer ¶

Classes¶

fastvideo.attention.backends.attn_qat_infer.AttnQatInferImpl ¶

AttnQatInferImpl(num_heads: int, head_size: int, causal: bool, softmax_scale: float, num_kv_heads: int | None = None, prefix: str = '', **extra_impl_args)

Bases: AttentionImpl[AttentionMetadata]

Source code in fastvideo/attention/backends/attn_qat_infer.py

def __init__(
    self,
    num_heads: int,
    head_size: int,
    causal: bool,
    softmax_scale: float,
    num_kv_heads: int | None = None,
    prefix: str = "",
    **extra_impl_args,
) -> None:
    self.causal = causal
    self.softmax_scale = softmax_scale
    dropout_p = extra_impl_args.get("dropout_p", 0.0)
    if dropout_p > 0:
        raise NotImplementedError(f"attn_qat_infer does not support dropout (got dropout_p={dropout_p}). "
                                  "The QAT inference kernel applies no stochastic dropout.")
    # Kernel resolution is per-forward, not per-construction: callers
    # (the validation swap, backend selection) gate on
    # is_attn_qat_infer_available() first, and constructing an impl on a
    # host without the kernel must stay legal (pre-existing contract the
    # validation-swap test pins).
    _log_receipt_once()

Functions:¶

fastvideo.attention.backends.attn_qat_infer.attn_qat_infer_receipt ¶

attn_qat_infer_receipt() -> str

One-line receipt of the resolution decision (arch + kernel + quant knobs), for the selection log and for tooling. The FA4 knobs are the repo's tuned defaults passed through verbatim: qk_mode=nvfp4 (per-16 E4M3 SFs), pv_mode=bf16 -- see flash_attn/cute/README.md in the kernel repo.

Source code in fastvideo/attention/backends/attn_qat_infer.py

def attn_qat_infer_receipt() -> str:
    """One-line receipt of the resolution decision (arch + kernel + quant
    knobs), for the selection log and for tooling. The FA4 knobs are the
    repo's tuned defaults passed through verbatim: qk_mode=nvfp4
    (per-16 E4M3 SFs), pv_mode=bf16 -- see flash_attn/cute/README.md in
    the kernel repo."""
    cap = _active_capability()
    arch = f"sm_{cap[0]}{cap[1]}" if cap is not None else "no-cuda"
    kernel = _resolved_kernel()
    if kernel == "cutlass_sm12x":
        return f"arch={arch} kernel=fastvideo-kernel-cutlass scheme=sage3-fp4-sm120"
    if kernel == "fa4_fp4":
        return (f"arch={arch} kernel=flash-attention-fp4 qk_mode=nvfp4(per-16-e4m3-sf) "
                f"pv_mode=bf16 train_sim_mismatch=measured")
    supported = "sm_120a/sm_121a via fastvideo-kernel build.sh; sm_100a/sm_103a via flash-attention-fp4"
    if cap is not None and cap in _FA4_FP4_CAPABILITIES:
        return f"arch={arch} kernel=none (flash_attn.cute not importable -- {_FA4_INSTALL_HINT})"
    return f"arch={arch} kernel=none (supported: {supported})"

fastvideo.attention.backends.attn_qat_infer.is_attn_qat_infer_available ¶

is_attn_qat_infer_available() -> bool

True only when the active device has a built ATTN_QAT_INFER kernel.

The import check alone is not sufficient: CUDA 13 wheel builds can carry the sm_12x extension on any host (e.g. H100, GB200), where the import succeeds, backend selection picks this backend, and the first kernel call then fails with an unsupported-capability error instead of ever reaching the documented FlashAttention fallback in fastvideo.platforms.cuda. Gating on the active device's capability keeps that fallback working on every unsupported GPU, while sm_100a/sm_103a now resolve to the FP4 FA4 kernel (#1221).

Source code in fastvideo/attention/backends/attn_qat_infer.py

def is_attn_qat_infer_available() -> bool:
    """True only when the active device has a built ATTN_QAT_INFER kernel.

    The import check alone is not sufficient: CUDA 13 wheel builds can
    carry the sm_12x extension on any host (e.g. H100, GB200), where the
    import succeeds, backend selection picks this backend, and the first
    kernel call then fails with an unsupported-capability error instead of
    ever reaching the documented FlashAttention fallback in
    fastvideo.platforms.cuda. Gating on the active device's capability
    keeps that fallback working on every unsupported GPU, while
    sm_100a/sm_103a now resolve to the FP4 FA4 kernel (#1221).
    """
    return _resolved_kernel() is not None