stable_audio_pipeline

Stable Audio Open 1.0 pipeline (T2A + A2A + RePaint inpainting).

Components are loaded via the standard ComposedPipelineBase.load_modules against the FastVideo-curated Diffusers-format repo FastVideo/stable-audio-open-1.0-Diffusers (produced by scripts/checkpoint_conversion/stable_audio_to_diffusers.py). The DiT is a BaseDiT subclass loaded by TransformerLoader; the VAE is loaded by VAELoader; the multi-conditioner (T5 + NumberConditioners) is loaded by ConditionerLoader (a Stable Audio-specific addition).

Stages:

InputValidationStage
  → StableAudioConditioningStage      (T5 + NumberConditioner -> cross-attn + global cond, with CFG)
  → StableAudioLatentPreparationStage (initial Gaussian noise; encodes A2A / inpaint refs)
  → StableAudioDenoisingStage         (k-diffusion `dpmpp-3m-sde` over the DiT)
  → StableAudioDecodingStage          (OobleckVAE -> waveform)

`fastvideo.pipelines.basic.stable_audio.stable_audio_pipeline.FastVideoArgs`

FastVideoArgs(model_path: str, mode: ExecutionMode = ExecutionMode.INFERENCE, workload_type: WorkloadType = WorkloadType.T2V, distributed_executor_backend: str = 'mp', enable_streaming_ipc_queues: bool = False, ray_placement_group: PlacementGroup | None = None, ray_runtime_env: RuntimeEnv | None = None, inference_mode: bool = True, trust_remote_code: bool = False, revision: str | None = None, num_gpus: int = 1, tp_size: int = -1, sp_size: int = -1, hsdp_replicate_dim: int = 1, hsdp_shard_dim: int = -1, dist_timeout: int | None = None, pipeline_config: PipelineConfig = PipelineConfig(), preprocess_config: PreprocessConfig | None = None, lora_path: str | None = None, lora_nickname: str = 'default', lora_strength: float = 1.0, lora_target_modules: list[str] | None = None, output_type: str = 'pil', attention_backend: str | None = None, dit_cpu_offload: bool = True, use_fsdp_inference: bool = False, dit_layerwise_offload: bool = True, text_encoder_cpu_offload: bool = True, image_encoder_cpu_offload: bool = True, vae_cpu_offload: bool = True, pin_cpu_memory: bool = True, h3_sequential_load: bool | None = None, h3_encoder_split: bool | None = None, h3_encoder_nodes: int = -1, h3_encoder_workers: int = 0, video_decode_backend: str = 'h3-vae', taeh3_checkpoint: str | None = None, taeh3_chunk_size: int = 5, h3_vae_nvfp4_decoder: str | None = None, lazy_module_load: bool | None = None, vae_parallel_decode: bool = False, vae_parallel_encode: bool = False, vae_parallel_decode_strategy: str | None = None, enable_torch_compile: bool = False, regional_compile: bool = False, enable_torch_compile_text_encoder: bool = False, enable_torch_compile_vae: bool = False, enable_torch_compile_audio_vae: bool = False, torch_compile_kwargs: dict[str, Any] = dict(), torch_compile_kwargs_dit: dict[str, Any] = dict(), torch_compile_kwargs_text_encoder: dict[str, Any] = dict(), torch_compile_kwargs_vae: dict[str, Any] = dict(), torch_compile_kwargs_audio_vae: dict[str, Any] = dict(), inference_torch_compile: bool = False, h3_vsa_fp4: dict[str, Any] = dict(), disable_autocast: bool = False, VSA_sparsity: float | None = None, VSA_tile_size: int | None = None, moba_config_path: str | None = None, moba_config: dict[str, Any] = dict(), master_port: int | None = None, enable_stage_verification: bool = True, prompt_txt: str | None = None, ltx2_vae_tiling: bool | None = None, ltx2_vae_spatial_tile_size_in_pixels: int | None = None, ltx2_vae_spatial_tile_overlap_in_pixels: int | None = None, ltx2_vae_temporal_tile_size_in_frames: int | None = None, ltx2_vae_temporal_tile_overlap_in_frames: int | None = None, ltx2_initial_latent_path: str | None = None, ltx2_audio_latent_path: str | None = None, refine_enabled: bool | None = None, refine_upsampler_path: str | None = None, refine_transformer_path: str | None = None, refine_lora_path: str | None = None, refine_num_inference_steps: int | None = None, refine_guidance_scale: float | None = None, refine_add_noise: bool | None = None, refine_noise_path: str | None = None, refine_audio_noise_path: str | None = None, ltx2_refine_enabled: bool = False, ltx2_refine_upsampler_path: str | None = None, ltx2_refine_transformer_path: str | None = None, ltx2_refine_lora_path: str | None = None, ltx2_refine_num_inference_steps: int = 3, ltx2_refine_guidance_scale: float = 1.0, ltx2_refine_add_noise: bool = True, ltx2_refine_noise_path: str | None = None, ltx2_refine_audio_noise_path: str | None = None, ltx2_legacy_native_noise_order: bool = False, ltx2_use_distilled_sigmas: bool = True, model_paths: dict[str, str] = dict(), model_loaded: dict[str, bool] = (lambda: {'transformer': True, 'vae': True, 'upsampler': True})(), override_text_encoder_safetensors: str | None = None, override_text_encoder_quant: QuantizationMethods = None, transformer_quant: Any | None = None, override_transformer_cls_name: str | None = None, init_weights_from_safetensors: str = '', init_weights_from_safetensors_2: str = '', override_pipeline_cls_name: str | None = None, boundary_ratio: float | None = 0.875)

`fastvideo.pipelines.basic.stable_audio.stable_audio_pipeline.FastVideoArgs.is_output_rank`

is_output_rank: bool

Whether this process may materialize user-facing outputs.

This is runtime state assigned by an SPMD executor, not a public configuration field, so dataclass serialization and typed config parsing deliberately ignore it.

`fastvideo.pipelines.basic.stable_audio.stable_audio_pipeline.FastVideoArgs.add_cli_args`

add_cli_args(parser: FlexibleArgumentParser) -> FlexibleArgumentParser

`fastvideo.pipelines.basic.stable_audio.stable_audio_pipeline.FastVideoArgs.from_cli_args`

from_cli_args(args: argparse.Namespace) -> FastVideoArgs

`fastvideo.pipelines.basic.stable_audio.stable_audio_pipeline.FastVideoArgs.from_kwargs`

from_kwargs(**kwargs: Any) -> FastVideoArgs

`fastvideo.pipelines.basic.stable_audio.stable_audio_pipeline.FastVideoArgs.check_fastvideo_args`

check_fastvideo_args() -> None

Validate inference arguments for consistency

`fastvideo.pipelines.basic.stable_audio.stable_audio_pipeline.FastVideoArgs.finalize_device_offload_policy`

finalize_device_offload_policy(device_id: int = 0) -> bool

Apply device-local memory policy, then resolve incompatible modes.

`fastvideo.pipelines.basic.stable_audio.stable_audio_pipeline.FastVideoArgs.disable_offload_on_unified_memory`

disable_offload_on_unified_memory(device_id: int = 0, *, offload_flag: str | None = None) -> bool

Disable host offload after a worker has selected its device.

CUDA's unified-memory probe reads runtime device properties and may initialize a CUDA context. Callers must therefore use this only inside a device-owning process, after selecting and binding device_id. Returning the classification lets direct component-loader callers apply the same policy to explicit per-call overrides. When offload_flag is given, the return value says whether this policy covers that component role.

`fastvideo.pipelines.basic.stable_audio.stable_audio_pipeline.init_logger`

init_logger(name: str) -> _FastvideoLogger

The main purpose of this function is to ensure that loggers are retrieved in such a way that we can be sure the root fastvideo logger has already been configured.

`fastvideo.pipelines.basic.stable_audio.stable_audio_pipeline.StableAudioConditioningStage`

StableAudioConditioningStage(conditioner)

Run the conditioner over the prompt + duration and stash the DiT-ready (cross_attn_cond, cross_attn_mask, global_embed) triple on batch.extra (plus the negative-prompt triple when CFG is on).

`fastvideo.pipelines.basic.stable_audio.stable_audio_pipeline.StableAudioConditioningStage.verify_input`

verify_input(batch, fastvideo_args)

`fastvideo.pipelines.basic.stable_audio.stable_audio_pipeline.StableAudioConditioningStage.verify_output`

verify_output(batch, fastvideo_args)

`fastvideo.pipelines.basic.stable_audio.stable_audio_pipeline.StableAudioConditioningStage.forward`

forward(batch: ForwardBatch, fastvideo_args: FastVideoArgs) -> ForwardBatch

`fastvideo.pipelines.basic.stable_audio.stable_audio_pipeline.StableAudioDecodingStage`

StableAudioDecodingStage(vae)

Decode latent → audio waveform + slice to [start, end].

`fastvideo.pipelines.basic.stable_audio.stable_audio_pipeline.StableAudioDecodingStage.verify_input`

verify_input(batch, fastvideo_args)

`fastvideo.pipelines.basic.stable_audio.stable_audio_pipeline.StableAudioDecodingStage.verify_output`

verify_output(batch, fastvideo_args)

`fastvideo.pipelines.basic.stable_audio.stable_audio_pipeline.StableAudioDecodingStage.forward`

forward(batch: ForwardBatch, fastvideo_args: FastVideoArgs) -> ForwardBatch

`fastvideo.pipelines.basic.stable_audio.stable_audio_pipeline.StableAudioDenoisingStage`

StableAudioDenoisingStage(transformer)

k-diffusion dpmpp-3m-sde sampling loop.

`fastvideo.pipelines.basic.stable_audio.stable_audio_pipeline.StableAudioDenoisingStage.verify_input`

verify_input(batch, fastvideo_args)

`fastvideo.pipelines.basic.stable_audio.stable_audio_pipeline.StableAudioDenoisingStage.verify_output`

verify_output(batch, fastvideo_args)

`fastvideo.pipelines.basic.stable_audio.stable_audio_pipeline.StableAudioDenoisingStage.forward`

forward(batch: ForwardBatch, fastvideo_args: FastVideoArgs) -> ForwardBatch

`fastvideo.pipelines.basic.stable_audio.stable_audio_pipeline.StableAudioLatentPreparationStage`

StableAudioLatentPreparationStage(io_channels: int = 64, sample_size: int = 2097152, vae = None, sample_rate: int = 44100, audio_channels: int = 2)

`fastvideo.pipelines.basic.stable_audio.stable_audio_pipeline.StableAudioLatentPreparationStage.verify_input`

verify_input(batch, fastvideo_args)

`fastvideo.pipelines.basic.stable_audio.stable_audio_pipeline.StableAudioLatentPreparationStage.verify_output`

verify_output(batch, fastvideo_args)

`fastvideo.pipelines.basic.stable_audio.stable_audio_pipeline.StableAudioLatentPreparationStage.forward`

forward(batch: ForwardBatch, fastvideo_args: FastVideoArgs) -> ForwardBatch

`fastvideo.pipelines.basic.stable_audio.stable_audio_pipeline.ComposedPipelineBase`

ComposedPipelineBase(model_path: str, fastvideo_args: FastVideoArgs | TrainingArgs, required_config_modules: list[str] | None = None, loaded_modules: dict[str, torch.nn.Module] | None = None)

Base class for pipelines composed of multiple stages.

This class provides the framework for creating pipelines by composing multiple stages together. Each stage is responsible for a specific part of the diffusion process, and the pipeline orchestrates the execution of these stages.

Initialize the pipeline. After init, the pipeline should be ready to use. The pipeline should be stateless and not hold any batch state.

`fastvideo.pipelines.basic.stable_audio.stable_audio_pipeline.ComposedPipelineBase.get_hf_download_component_dirs`

get_hf_download_component_dirs() -> tuple[str, ...] | None

Return component directories for an opt-in partial Hub download.

`fastvideo.pipelines.basic.stable_audio.stable_audio_pipeline.ComposedPipelineBase.get_hf_download_allow_patterns`

get_hf_download_allow_patterns() -> list[str] | None

Return Hub patterns for the manifest and selected components.

`fastvideo.pipelines.basic.stable_audio.stable_audio_pipeline.ComposedPipelineBase.set_trainable`

set_trainable() -> None

`fastvideo.pipelines.basic.stable_audio.stable_audio_pipeline.ComposedPipelineBase.post_init`

post_init() -> None

`fastvideo.pipelines.basic.stable_audio.stable_audio_pipeline.ComposedPipelineBase.initialize_training_pipeline`

initialize_training_pipeline(training_args: TrainingArgs)

`fastvideo.pipelines.basic.stable_audio.stable_audio_pipeline.ComposedPipelineBase.initialize_validation_pipeline`

initialize_validation_pipeline(training_args: TrainingArgs)

`fastvideo.pipelines.basic.stable_audio.stable_audio_pipeline.ComposedPipelineBase.from_pretrained`

from_pretrained(model_path: str, device: str | None = None, torch_dtype: torch.dtype | None = None, pipeline_config: str | PipelineConfig | None = None, args: argparse.Namespace | None = None, required_config_modules: list[str] | None = None, loaded_modules: dict[str, torch.nn.Module] | None = None, **kwargs) -> ComposedPipelineBase

Load a pipeline from a pretrained model. loaded_modules: Optional[Dict[str, torch.nn.Module]] = None, If provided, loaded_modules will be used instead of loading from config/pretrained weights.

`fastvideo.pipelines.basic.stable_audio.stable_audio_pipeline.ComposedPipelineBase.get_module`

get_module(module_name: str, default_value: Any = None) -> Any

`fastvideo.pipelines.basic.stable_audio.stable_audio_pipeline.ComposedPipelineBase.add_module`

add_module(module_name: str, module: Any)

`fastvideo.pipelines.basic.stable_audio.stable_audio_pipeline.ComposedPipelineBase.required_config_modules`

required_config_modules: list[str]

List of modules that are required by the pipeline. The names should match the diffusers directory and model_index.json file. These modules will be loaded using the PipelineComponentLoader and made available in the modules dictionary. Access these modules using the get_module method.

class ConcretePipeline(ComposedPipelineBase): _required_config_modules = ["vae", "text_encoder", "transformer", "scheduler", "tokenizer"]

@property
def required_config_modules(self):
    return self._required_config_modules

`fastvideo.pipelines.basic.stable_audio.stable_audio_pipeline.ComposedPipelineBase.stages`

stages: list[PipelineStage]

List of stages in the pipeline.

`fastvideo.pipelines.basic.stable_audio.stable_audio_pipeline.ComposedPipelineBase.create_pipeline_stages`

create_pipeline_stages(fastvideo_args: FastVideoArgs)

Create the inference pipeline stages.

`fastvideo.pipelines.basic.stable_audio.stable_audio_pipeline.ComposedPipelineBase.create_training_stages`

create_training_stages(training_args: TrainingArgs)

Create the training pipeline stages.

`fastvideo.pipelines.basic.stable_audio.stable_audio_pipeline.ComposedPipelineBase.initialize_pipeline`

initialize_pipeline(fastvideo_args: FastVideoArgs)

Initialize the pipeline.

`fastvideo.pipelines.basic.stable_audio.stable_audio_pipeline.ComposedPipelineBase.load_modules`

load_modules(fastvideo_args: FastVideoArgs, loaded_modules: dict[str, torch.nn.Module] | None = None) -> dict[str, Any]

Load the modules from the config. loaded_modules: Optional[Dict[str, torch.nn.Module]] = None, If provided, loaded_modules will be used instead of loading from config/pretrained weights.

`fastvideo.pipelines.basic.stable_audio.stable_audio_pipeline.ComposedPipelineBase.add_stage`

add_stage(stage_name: str, stage: PipelineStage)

`fastvideo.pipelines.basic.stable_audio.stable_audio_pipeline.ComposedPipelineBase.forward`

forward(batch: ForwardBatch, fastvideo_args: FastVideoArgs) -> ForwardBatch

Generate a video or image using the pipeline.

Parameters

batch: ForwardBatch: The batch to generate from.
fastvideo_args: FastVideoArgs: The inference arguments.

Returns: ForwardBatch: The batch with the generated video or image.

`fastvideo.pipelines.basic.stable_audio.stable_audio_pipeline.ComposedPipelineBase.train`

train() -> None

`fastvideo.pipelines.basic.stable_audio.stable_audio_pipeline.ComposedPipelineBase.close`

close() -> None

`fastvideo.pipelines.basic.stable_audio.stable_audio_pipeline.InputValidationStage`

InputValidationStage()

Stage for validating and preparing inputs for diffusion pipelines.

This stage validates that all required inputs are present and properly formatted before proceeding with the diffusion process.

`fastvideo.pipelines.basic.stable_audio.stable_audio_pipeline.InputValidationStage.forward`

forward(batch: ForwardBatch, fastvideo_args: FastVideoArgs) -> ForwardBatch

Validate and prepare inputs.

Parameters

batch: ForwardBatch: The current batch information.
fastvideo_args: FastVideoArgs: The inference arguments.

Returns

ForwardBatch: The validated batch information.

`fastvideo.pipelines.basic.stable_audio.stable_audio_pipeline.InputValidationStage.verify_input`

verify_input(batch: ForwardBatch, fastvideo_args: FastVideoArgs) -> VerificationResult

Verify input validation stage inputs.

`fastvideo.pipelines.basic.stable_audio.stable_audio_pipeline.InputValidationStage.verify_output`

verify_output(batch: ForwardBatch, fastvideo_args: FastVideoArgs) -> VerificationResult

Verify input validation stage outputs.

`fastvideo.pipelines.basic.stable_audio.stable_audio_pipeline.StableAudioPipeline`

StableAudioPipeline(model_path: str, fastvideo_args: FastVideoArgs | TrainingArgs, required_config_modules: list[str] | None = None, loaded_modules: dict[str, torch.nn.Module] | None = None)

Stable Audio Open 1.0 pipeline.

Mode is kwargs-driven on generate_video():

Text-to-audio (default) -- prompt=..., audio_end_in_s=...
Audio-to-audio variation -- add init_audio=ref (and optionally init_noise_level, lower = closer to reference)
RePaint inpainting / outpainting -- add inpaint_audio=ref and inpaint_mask (1-D, 1 = keep / 0 = regenerate)

See examples/inference/basic/basic_stable_audio*.py for runnable examples of each mode.

Source

class StableAudioPipeline(ComposedPipelineBase):
    """Stable Audio Open 1.0 pipeline.
Mode is kwargs-driven on `generate_video()`:

  * Text-to-audio (default) -- `prompt=...`, `audio_end_in_s=...`
  * Audio-to-audio variation -- add `init_audio=ref` (and optionally
    `init_noise_level`, lower = closer to reference)
  * RePaint inpainting / outpainting -- add `inpaint_audio=ref` and
    `inpaint_mask` (1-D, 1 = keep / 0 = regenerate)

See `examples/inference/basic/basic_stable_audio*.py` for runnable
examples of each mode.
&quot;&quot;&quot;

_required_config_modules = &#91;
    &quot;vae&quot;,
    &quot;transformer&quot;,
    &quot;conditioner&quot;,
&#93;

def initialize_pipeline(self, fastvideo_args: FastVideoArgs) -&gt; None:
    &quot;&quot;&quot;Apply Stable Audio&#x27;s process-global numerics overrides BEFORE
    the standard component loaders run (TF32 off for A2A renoise
    determinism).&quot;&quot;&quot;
    _disable_tf32_for_stable_audio()

def create_pipeline_stages(self, fastvideo_args: FastVideoArgs) -&gt; None:
    pc = fastvideo_args.pipeline_config

    self.add_stage(stage_name=&quot;input_validation_stage&quot;, stage=InputValidationStage())

    self.add_stage(
        stage_name=&quot;conditioning_stage&quot;,
        stage=StableAudioConditioningStage(conditioner=self.get_module(&quot;conditioner&quot;)),
    )

    self.add_stage(
        stage_name=&quot;latent_preparation_stage&quot;,
        stage=StableAudioLatentPreparationStage(
            io_channels=64,
            # Per-variant training window: 2,097,152 (~47.5s) for
            # SA-1.0; 524,288 (~11.9s) for SA-small. Pulled from the
            # pipeline config so each variant gets its own latent
            # length.
            sample_size=pc.sample_size,
            vae=self.get_module(&quot;vae&quot;),
            sample_rate=pc.sampling_rate,
            audio_channels=pc.audio_channels,
        ),
    )

    self.add_stage(
        stage_name=&quot;denoising_stage&quot;,
        stage=StableAudioDenoisingStage(transformer=self.get_module(&quot;transformer&quot;)),
    )

    self.add_stage(
        stage_name=&quot;decoding_stage&quot;,
        stage=StableAudioDecodingStage(vae=self.get_module(&quot;vae&quot;)),
    )</code></pre></details>
fastvideo.pipelines.basic.stable_audio.stable_audio_pipeline.StableAudioPipeline.initialize_pipeline
initialize_pipeline(fastvideo_args: FastVideoArgs) -> None
Apply Stable Audio's process-global numerics overrides BEFORE
the standard component loaders run (TF32 off for A2A renoise
determinism).
Source
def initialize_pipeline(self, fastvideo_args: FastVideoArgs) -> None:
    """Apply Stable Audio's process-global numerics overrides BEFORE
    the standard component loaders run (TF32 off for A2A renoise
    determinism)."""
    _disable_tf32_for_stable_audio()
fastvideo.pipelines.basic.stable_audio.stable_audio_pipeline.StableAudioPipeline.create_pipeline_stages
create_pipeline_stages(fastvideo_args: FastVideoArgs) -> None
Source
def create_pipeline_stages(self, fastvideo_args: FastVideoArgs) -> None:
    pc = fastvideo_args.pipeline_config
self.add_stage(stage_name=&quot;input_validation_stage&quot;, stage=InputValidationStage())

self.add_stage(
    stage_name=&quot;conditioning_stage&quot;,
    stage=StableAudioConditioningStage(conditioner=self.get_module(&quot;conditioner&quot;)),
)

self.add_stage(
    stage_name=&quot;latent_preparation_stage&quot;,
    stage=StableAudioLatentPreparationStage(
        io_channels=64,
        # Per-variant training window: 2,097,152 (~47.5s) for
        # SA-1.0; 524,288 (~11.9s) for SA-small. Pulled from the
        # pipeline config so each variant gets its own latent
        # length.
        sample_size=pc.sample_size,
        vae=self.get_module(&quot;vae&quot;),
        sample_rate=pc.sampling_rate,
        audio_channels=pc.audio_channels,
    ),
)

self.add_stage(
    stage_name=&quot;denoising_stage&quot;,
    stage=StableAudioDenoisingStage(transformer=self.get_module(&quot;transformer&quot;)),
)

self.add_stage(
    stage_name=&quot;decoding_stage&quot;,
    stage=StableAudioDecodingStage(vae=self.get_module(&quot;vae&quot;)),
)</code></pre></details>