Skip to content

mmaudio_clip

Native split DFN5B CLIP encoders used by MMAudio.

MMAudio uses one OpenCLIP checkpoint in two different ways: normalized projected image embeddings and normalized token-wise text hidden states. The classes here reuse FastVideo's native CLIP transformer while exposing those two contracts as independently loadable pipeline components.

Classes

fastvideo.models.encoders.mmaudio_clip.MMAudioDFNCLIPTextEncoder 

MMAudioDFNCLIPTextEncoder(config: MMAudioDFNCLIPTextConfig)

Bases: CLIPTextModel

Return normalized CLIP hidden states for every text token.

Source code in fastvideo/models/encoders/mmaudio_clip.py 
32
33
def __init__(self, config: MMAudioDFNCLIPTextConfig) -> None:
    super().__init__(config)

Methods:

fastvideo.models.encoders.mmaudio_clip.MMAudioDFNCLIPTextEncoder.load_weights 
load_weights(weights: Iterable[tuple[str, Tensor]]) -> set[str]

Load either split OpenCLIP Q/K/V or converted fused QKV weights.

Source code in fastvideo/models/encoders/mmaudio_clip.py 
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
def load_weights(
    self,
    weights: Iterable[tuple[str, torch.Tensor]],
) -> set[str]:
    """Load either split OpenCLIP Q/K/V or converted fused QKV weights."""
    params = dict(self.named_parameters())
    loaded: set[str] = set()
    for name, tensor in weights:
        if name in params:
            parameter = params[name]
            loader = getattr(parameter, "weight_loader", default_weight_loader)
            loader(parameter, tensor)
            loaded.add(name)
            continue
        for param_name, weight_name, shard_id in self.config.arch_config.stacked_params_mapping:
            if weight_name not in name:
                continue
            target_name = name.replace(weight_name, param_name)
            if target_name not in params:
                continue
            parameter = params[target_name]
            parameter.weight_loader(parameter, tensor, shard_id)
            loaded.add(target_name)
            break
    return loaded

fastvideo.models.encoders.mmaudio_clip.MMAudioDFNCLIPVisionEncoder 

MMAudioDFNCLIPVisionEncoder(config: MMAudioDFNCLIPVisionConfig)

Bases: CLIPVisionModel

Return normalized projected CLS embeddings for individual frames.

Source code in fastvideo/models/encoders/mmaudio_clip.py 
114
115
116
def __init__(self, config: MMAudioDFNCLIPVisionConfig) -> None:
    super().__init__(config)
    self.visual_projection = nn.Linear(config.hidden_size, config.projection_dim, bias=False)

Functions: