Skip to content

stages

MMAudio-specific feature stages for the shared FastVideo V2A workflow.

Classes

fastvideo.pipelines.preprocess.mmaudio.stages.MMAudioFeatureExtractionStage 

MMAudioFeatureExtractionStage(*, audio_vae: Module, mel_converter: Module, image_encoder: Module, sync_encoder: Module, text_encoder: Module, tokenizer: Any)

Bases: PipelineStage

Extract VAE posterior statistics plus CLIP/Synchformer features.

Source code in fastvideo/pipelines/preprocess/mmaudio/stages.py 
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
def __init__(
    self,
    *,
    audio_vae: torch.nn.Module,
    mel_converter: torch.nn.Module,
    image_encoder: torch.nn.Module,
    sync_encoder: torch.nn.Module,
    text_encoder: torch.nn.Module,
    tokenizer: Any,
) -> None:
    super().__init__()
    self.audio_vae = audio_vae.eval()
    self.mel_converter = mel_converter.eval()
    self.image_encoder = image_encoder.eval()
    self.sync_encoder = sync_encoder.eval()
    self.text_encoder = text_encoder.eval()
    self.tokenizer = tokenizer

Functions: