Testing in FastVideo¶

This guide explains how to add and run tests in FastVideo. The testing suite is divided into several categories to ensure correctness across components, training workflows, and inference quality.

Test Types¶

Unit Tests: Located in fastvideo/tests/dataset, fastvideo/tests/entrypoints, and fastvideo/tests/workflow. These test individual functions and classes.
Component Tests: Located in fastvideo/tests/encoders, fastvideo/tests/transformers, and fastvideo/tests/vaes. These verify the loading and basic functionality of model components.
SSIM Tests: Located in fastvideo/tests/ssim. These are regression tests that compare generated videos against reference videos using the Structural Similarity Index Measure (SSIM) to detect quality degradation.
Training Tests: Located in fastvideo/tests/training. These validate training loops, loss calculations, and specific training techniques like LoRA, Distillation, and VSA.
Inference Tests: Located in fastvideo/tests/inference. These test specialized inference pipelines and optimizations (e.g., VSA, V-MoBA).

For now, we will focus on SSIM Tests.

SSIM Tests¶

SSIM tests are located in fastvideo/tests/ssim. These tests generate videos using specific models and parameters, and compare them against reference videos to ensure that changes in the codebase do not degrade generation quality or alter the output unexpectedly.

Note

If you are adding an SSIM test, this serves as a safeguard. Any future code changes that break or cause errors with the specific arguments and configurations you defined will trigger a failure. Therefore, it is important to include multiple settings and arguments that cover the core features of your new pipeline to ensure robust regression testing.

Directory Structure¶

fastvideo/tests/ssim/
├── reference_videos/
│   ├── default/
│   │   └── <GPU>_reference_videos/
│   │       ├── <Model_Name>/
│   │       │   ├── <Backend>/        # e.g., FLASH_ATTN, TORCH_SDPA
│   │       │   │   └── <Video_File>
│   └── full_quality/
│       └── <GPU>_reference_videos/
├── test_causal_similarity.py
├── test_wan_t2v_similarity.py
├── test_wan_i2v_similarity.py
├── reference_videos_cli.py
└── ...

Adding a New SSIM Test¶

To add a new SSIM test, follow these steps:

Create or Update a Test File: Prefer model-specific files (for example test_wan_t2v_similarity.py) and create a new one when testing a distinct model or pipeline.
Define Model Parameters: Define the configuration for the model you want to test. This includes model path, dimensions, inference steps, and other generation parameters. Note: Consider using lower num_inference_steps or reduced resolution (e.g., 480p instead of 720p) to keep test execution time reasonable, provided it doesn't compromise the test's ability to detect regression.
```
MY_MODEL_PARAMS = {
    "num_gpus": 1,
    "model_path": "organization/model-name",
    "height": 480,
    "width": 832,
    "num_frames": 45,
    "num_inference_steps": 20,
    # ... other parameters
}
```

Implement the Test Function:

Use pytest.mark.parametrize to run the test with different prompts, backends, and models.
Set the attention backend environment variable.
Initialize the VideoGenerator.
Generate the video.
Compare the generated video with the reference video using compute_video_ssim_torchvision.

Example structure:

@pytest.mark.parametrize("prompt", TEST_PROMPTS)
@pytest.mark.parametrize("ATTENTION_BACKEND", ["FLASH_ATTN"])
def test_my_model_similarity(prompt, ATTENTION_BACKEND):
    # Setup output directories
    # ...

    # Initialize Generator
    generator = VideoGenerator.from_pretrained(...)
    generator.generate_video(prompt, ...)

    # Compare with Reference
    ssim_values = compute_video_ssim_torchvision(
        reference_path, generated_path, use_ms_ssim=True
    )
    assert ssim_values[0] >= 0.98  # Threshold

Reference Videos:
- When running the test for the first time (or when updating the reference), the test will fail because the reference video is missing. The generated video will be saved in fastvideo/tests/ssim/generated_videos/<quality-tier>/<GPU>_reference_videos.
- Inspect the generated video to ensure it meets quality expectations.
- Move the generated video to the appropriate quality/GPU reference folder: fastvideo/tests/ssim/reference_videos/<quality-tier>/<GPU>_reference_videos/<Model>/<Backend>/.
- You can use the helper CLI to copy generated videos into a reference folder: python fastvideo/tests/ssim/reference_videos_cli.py copy-local --quality-tier default --reference-dir fastvideo/tests/ssim/reference_videos/default/L40S_reference_videos
- Upload/download can target both quality tiers and specific GPU folders: python fastvideo/tests/ssim/reference_videos_cli.py upload --quality-tier all python fastvideo/tests/ssim/reference_videos_cli.py download --quality-tier full_quality --device-folder H200_reference_videos

Running Tests Locally¶

To run the SSIM tests locally:

pytest fastvideo/tests/ssim/ -vs

Ensure you have the necessary GPUs available as defined in your test parameters.

CI Integration¶

FastVideo uses Modal for running tests in a CI environment. The workflow scripts are located in fastvideo/tests/modal/.

Buildkite Pipeline¶

Tests are orchestrated by Buildkite (.buildkite/pipeline.yml) and executed on Modal GPU instances. The pipeline runs in two modes:

Fastcheck — runs on every PR push, path-filtered. Only tests for the components you changed are triggered. Tests run in parallel.

Full Suite — runs when a PR enters the Merge Queue (or when triggered manually via /test full). Covers SSIM regression, training, distillation, inference, and performance.

`pr_test.py`¶

The main entry point for CI tests is fastvideo/tests/modal/pr_test.py. This script defines Modal functions that execute the pytest suites on specific hardware (e.g., L40S, H100).

If you add a new test that requires:

Different GPU Hardware: You may need to change the @app.function(gpu=...) decorator.
Longer Execution Time: Increase the timeout parameter.
New Environment Variables/Secrets: Add them to secrets=[...] or the image environment. For example, if your model is gated on Hugging Face, ensure HF_API_KEY is passed.

For SSIM tests, use fastvideo/tests/modal/ssim_test.py:

python -m modal run fastvideo/tests/modal/ssim_test.py::run_ssim_tests

Target specific SSIM files/models:

python -m modal run fastvideo/tests/modal/ssim_test.py::run_ssim_tests \
  --test-files test_wan_t2v_similarity.py \
  --model-ids Wan2.1-T2V-1.3B-Diffusers

If HF token env vars are not set (HF_API_KEY / HUGGINGFACE_HUB_TOKEN / HF_TOKEN), the local entrypoint fails fast. To export raw generated_videos from Modal to the shared volume:

python -m modal run fastvideo/tests/modal/ssim_test.py::run_ssim_tests \
  --sync-generated-to-volume

The raw export path is quality-tiered:

default params: ssim_generated_videos/default/<subdir>/generated_videos
full-quality params: ssim_generated_videos/full_quality/<subdir>/generated_videos

The printed modal volume get command also downloads into a quality-specific local directory under ./generated_videos_modal/<quality-tier>.

To turn downloaded Modal outputs into local reference videos, use the matching quality tier with copy-local, for example:

python fastvideo/tests/ssim/reference_videos_cli.py copy-local \
  --quality-tier full_quality \
  --generated-dir ./generated_videos_modal/full_quality/L40S_reference_videos \
  --device-folder L40S_reference_videos

Workflow Scripts¶

The shell script that triggers tests in CI is .buildkite/scripts/pr_test.sh. If you add a new test category (e.g., a new folder outside of ssim), you will need to:

Add a new function in fastvideo/tests/modal/pr_test.py.
Add a new case in .buildkite/scripts/pr_test.sh to handle the new test type.

Note

If you are a maintainer, update the workflow script in Buildkite after merging. Otherwise, ask a maintainer for help.

Triggering Tests via Slash Commands¶

Maintainers and contributors with write access can trigger individual test suites directly from a PR comment. The workflow reacts with a 🚀 emoji to confirm the command was received.

/test ssim              # SSIM regression tests
/test training          # Training pipeline tests
/test lora-training     # LoRA training tests
/test lora-inference    # LoRA inference tests
/test distillation      # DMD distillation tests
/test self-forcing      # Self-Forcing tests
/test vsa               # VSA training tests
/test vmoba             # VMoBA inference tests
/test performance       # Performance benchmarks
/test api               # API server integration tests
/test encoder           # Encoder component tests (Fastcheck)
/test vae               # VAE component tests (Fastcheck)
/test transformer       # Transformer / DiT tests (Fastcheck)
/test kernel            # CUDA kernel tests (Fastcheck)
/test unit              # Unit tests (Fastcheck)
/test full              # Entire Full Suite
/test fastcheck         # Entire Fastcheck suite

See CI Architecture for the complete reference.