CI/CD Architecture¶

This is the canonical reference for FastVideo's CI/CD system. Contributor-facing PR steps live in Pull Requests, and test-authoring guidance lives in Testing.

Overview¶

FastVideo splits validation across GitHub Actions, Buildkite, Slinky Slurm, and Mergify:

PR opened or updated
  |
  |-- Tier 1: pre-commit
  |     GitHub Actions on ubuntu-latest
  |     style, lint, type, spelling, Markdown, workflow syntax, filenames
  |
  |-- Tier 2: Fastcheck
  |     Buildkite schedules six lanes on the Slinky Slurm cluster
  |     encoder, VAE, transformer, kernel, unit, DreamVerse
  |
  |-- /merge or ready label
        |
        `-- Tier 3: Change-aware merge gate
              trusted base-branch planner classifies the complete PR diff
              Buildkite adds only relevant integration, quality, training,
              API, or performance lanes on Slinky Slurm
              |
              pass -> Mergify squash-merges when all merge conditions pass
              fail -> fix, push, and re-run

  |-- /test full
        `-- Explicit all-20-lane diagnostic run

  `-- weekly schedule on main
        `-- Complete four-GPU SSIM matrix

CI is not one monolithic job:

• GitHub Actions owns pre-commit, slash-command handling, aggregate status updates, docs deployment, image builds, package publishing, and community automations.
• Buildkite owns the GPU test graph, statuses, and trusted dispatch control plane. Its agent runs on the Slurm login plane; it does not execute test payloads.
• Slinky Slurm is the only active CI compute backend. A host-owned dispatcher leases GPUs from a persistent four-GPU allocation and runs each lane in an isolated Enroot container at the immutable PR SHA.
• Mergify owns merge protection, labeling, and the final squash merge.

The old files under fastvideo/tests/modal/ are retained as dormant manual rollback code. .buildkite/scripts/pr_test.sh rejects Buildkite invocations, and no pipeline or slash-command route calls Modal.

Three Buildkite entry pipelines share the validated graph:

Each entry pipeline starts with the same trusted pipeline-upload job on the ci-runner queue. The ci pipeline's incoming GitHub webhook is disabled; otherwise it would duplicate the automatic pr-fastcheck build. API and scheduled builds continue to work with webhook processing disabled.

CI Tiers¶

Tier 1: Pre-commit¶

The workflow runs .pre-commit-config.yaml with --hook-stage manual.

Run the pre-commit command above to reproduce failures locally. Do not bypass the project hook chain by calling individual tools directly unless you are debugging a hook implementation.

Tier 2: Fastcheck¶

Fastcheck always schedules these six lanes: encoder, VAE, transformer, custom kernels, unit tests, and DreamVerse. Static steps replace the former host-side path-filter plugin: the login plane never checks out or executes PR code.

Tier 3: Change-Aware Merge Gate¶

Fastcheck is the universal six-lane baseline. The merge gate does not repeat those jobs: it classifies every changed path and adds only the relevant lanes from the fourteen-lane integration set below. Selected jobs are hard gates; there are no soft-fail hardware lanes. A documentation-only PR can therefore finish its merge build after the trusted uploader, while model-family changes typically add focused golden-gate and SSIM files and a training-only change adds only its owning training lane.

.github/scripts/plan_merge_ci.py is the canonical path policy. It runs from the immutable base SHA under pull_request_target; PR code is never executed on the GitHub runner. The changed-file list includes both sides of renames. An API failure, truncated response, empty list, unknown build input, or unknown source path fails closed to all fourteen integration lanes.

A ready-labeled PR does not hit Buildkite immediately: ci-trigger-full-suite.yml first runs .github/scripts/gate_full_suite.sh, which waits for the cheap Tier-1 checks (pre-commit, docs build) on the PR head. A red cheap check blocks the suite (fail closed; the next push re-arms it), while a GitHub outage or a >25 min wait lets it run anyway (fail open). /test full bypasses the gate.

The complete static graph remains available through /test full; path selection never deletes or dynamically invents a Buildkite step.

Integration steps depend on golden-gate. A failed selected golden prevents the expensive downstream jobs from starting; /test full still selects all twenty lanes. A condition-skipped golden satisfies the dependency, so direct lane reruns, scheduled SSIM, and training-only merge plans keep their existing meaning. This follows Buildkite's conditional dependency rules. No dependency allows failures. The trusted uploader accepts either the old complete graph or the complete golden-first graph during rollout, and rejects partial or arbitrary dependency changes.

The six automatic Fastcheck lanes are unchanged. Within VAE and transformer lanes, small Wan goldens run before independent component parity. Wan paths select matching VAE, dense/trajectory, or causal-cache goldens; shared Wan config and pipeline wiring select all four. Shared runtime changes continue to select broader coverage. The existing block references retain their exact environment identity; new tensor gates distinguish the effective FA2/FA4 switch and VAE gates do not depend on an unused attention backend.

Golden-gate and SSIM selections are basenames, not arbitrary pytest arguments. The private host checks the comma-separated allowlist before staging, and the container checks it again before invoking pytest. Shared quality-harness changes still run the complete owning lane. The full SSIM matrix also runs on main every Sunday at 05:00 UTC through ci-scheduled-ssim.yml; /test ssim and /test full remain available for deliberate complete runs.

The four Buildkite workers may accept multiple jobs concurrently. The agent-owned lease broker packs their requested GPU counts onto one persistent four-GPU Slurm allocation and waits when capacity is full. A four-GPU lane such as SSIM or vanilla training owns the whole tray; two two-GPU lanes or up to four one-GPU lanes can overlap without sharing devices. SSIM and vanilla training also share the fastvideo/slinky/whole-tray Buildkite concurrency group. That keeps the second whole-tray lane in Buildkite instead of consuming an agent and its command timeout while the first lane waits for four free GPUs. Because packed Enroot containers share the node network namespace, each GPU lease receives its own 100-port rendezvous range. Tests preserve the runner-assigned MASTER_PORT, and parallel SSIM tasks use distinct offsets inside that range.

See Performance Benchmarks for the performance lane's thresholds, rolling baseline, artifacts, and reseeding process.

/merge Request Flow¶

The Buildkite agent and Slurm have deliberately separate responsibilities:

/merge PR comment
  -> GitHub verifies write permission and refreshes the `ready` label
  -> base-branch `ci-trigger-full-suite` workflow fetches the PR file list,
     computes MERGE_TEST_PLAN plus focused golden/SSIM basenames, and gates on
     cheap checks
  -> sends the PR SHA to pipeline `ci` with TEST_SCOPE=merge and FULL_SUITE=true
  -> trusted `pipeline-upload` job on queue `ci-runner`
       fetch exact-SHA .buildkite/pipeline.yml
       normalize + validate the complete static 20-lane policy and conditions
       upload static Buildkite steps
  -> each accepted step reaches the host policy hook
       validate org/repo/SHA/ref/step key/command/scope/timeout
       skip checkout on the login plane
       stage a mode-0600 request on Lustre
       lease 1-4 GPUs and attach an `srun` step to the Slinky tray
  -> Enroot worker container
       clone and verify the exact PR SHA
       install the lane's project extras and cached kernel
       run the repository-owned lane script
       write numeric exit status and approved artifacts
  -> trusted host returns that status to Buildkite
  -> Buildkite publishes `full-suite-passed` only when every selected lane passes

The pipeline uploader and dispatcher run on the Slurm login plane, but those are control-plane operations only. Python tests, model loading, CUDA kernels, Node/Playwright checks, inference, training, SSIM generation, and performance benchmarks all execute in Slurm allocations.

PR-controlled values never become host commands. The host policy accepts only the pinned pipeline uploader or a known lane tuple. It rejects plugins, artifact globs, shell injection variables, non-immutable commits, and unknown commands before checkout. Hugging Face credentials are added only for lanes that declare them, passed through a mode-0600 request file, and removed before the PR payload starts. Active training lanes keep W&B offline and do not stage a W&B credential. The ARM64 image includes the pinned FA4 CuTe overlay validated on GB200. SSIM opts into FA4 to preserve its reference-video numerics; lanes with FA2 baselines keep FASTVIDEO_FA4=0. Performance artifacts are relayed afterward by the trusted host from an allowlisted directory and extension set.

Slash Commands¶

Slash commands are handled by .github/workflows/ci-slash-commands.yml. Repository write permission is required.

Valid direct test names:

The temporary <name>-ci spellings remain accepted as compatibility aliases; they select the same Slurm lane and do not identify a second backend.

When a direct test completes successfully, Buildkite posts direct-test-completed. .github/workflows/ci-aggregate-status.yml then reads the latest Buildkite statuses for the commit and updates fastcheck-passed or full-suite-passed if all jobs in that group are green.

Buildkite label emojis define the status namespace used by that aggregation: :microscope: is reserved for the six Fastcheck lanes, while Full-Suite-only lanes use :test_tube: or :bar_chart:. Each active lane has exactly one label and therefore one status context.

Merge Protection¶

Mergify enforces these conditions before it squash-merges to main:

The final merge action is a squash merge. Mergify also labels conflicting PRs with needs-rebase and removes that label after conflicts are resolved.

Title Tags And Labels¶

PR title tags are the source for type:* labels and are required by merge protection.

Scope labels are inferred from changed files.

Process labels:

Slurm Lane Entrypoints¶

Every active test selection lives in .buildkite/scripts/unit_test.sh or a focused .buildkite/scripts/lanes/<lane>.sh. The private, agent-owned lane table binds each internal *_ci type to that script, its GPU count, wall-clock limit, dependency extras, kernel-build policy, secrets, and artifacts. The internal suffix is an implementation detail; there is only one active backend.

SSIM uses fastvideo/tests/ssim/ci_runner.py inside a single four-GPU lease. It discovers REQUIRED_GPUS and *_MODEL_TO_PARAMS with AST parsing, then packs independent pytest subprocesses across the visible GPUs with fail-fast termination. Performance writes reports to a host-mounted artifact directory; the host uploads only .md, .html, .json, and .csv files after the container exits.

The Modal launchers remain in the repository for manual rollback archaeology, but they are not CI entrypoints. pr_test.sh rejects Buildkite calls and needs FASTVIDEO_ENABLE_LEGACY_MODAL_CI=1 even for a local manual invocation.

If you add a new CI test category:

1. Add an executable .buildkite/scripts/lanes/<lane>.sh containing the test payload only.
2. Add the static Buildkite step in .buildkite/pipeline.yml, its source/test ownership in .github/scripts/plan_merge_ci.py, and the /test mapping in .github/workflows/ci-slash-commands.yml.
3. Extend fastvideo/tests/contract/test_ci_test_collection.py, test_merge_ci_plan.py, and the trusted private runner's lane table plus uploader policy in the same rollout.
4. Validate on the target GB200 hardware before making the lane a merge gate.
5. Document the lane here and link any domain-specific authoring guide.

CD And Release Workflows¶

Documentation¶

.github/workflows/infra-docs.yml builds documentation for PRs that touch docs/**, mkdocs.yml, requirements-mkdocs.txt, or the workflow itself. On pushes to main, it also deploys the built site to GitHub Pages.

The docs job:

1. Installs the pinned MkDocs dependencies.
2. Runs mkdocs build; the native MkDocs hook generates example pages before the build.
3. Runs python scripts/check_docs_links.py against the generated documentation.
4. Uploads the Pages artifact and deploys only from main.

Docker Images¶

.github/workflows/infra-build-image.yml supports manual workflow_dispatch runs and automatically rebuilds the CUDA matrix when a repository-controlled image input changes on main in the canonical repository. Those inputs include the CUDA Dockerfile and reusable workflow, dependency metadata, Docker context policy, fastvideo-kernel/**, and the kernel artifact metadata/key helper. Manual runs let maintainers choose which image families to build. The fastvideo-dev matrix builds Python 3.12 images for CUDA 12.6 and CUDA 13 on native amd64 and arm64 runners, then publishes one multi-platform manifest per CUDA version. CUDA 12.6 owns the py3.12-latest and global latest tags, as well as the explicit py3.12-cuda12.6.3-latest alias. CUDA 13 is published under the explicit py3.12-cuda13.0.0-latest tag. This publication policy does not change the unparameterized docker/Dockerfile build defaults, which remain CUDA 13 and cu130.

Published development images carry architecture-specific kernel wheels under /opt/fastvideo-kernel-prebuilt. The Slurm worker selects the exact source, ABI, and GPU-architecture match, so normal lanes reuse the trusted artifact while kernel-changing PRs still build locally. Once a kernel or artifact-key change reaches main, the image workflow republishes the matching artifact before later jobs consume the updated image pin.

The same workflow publishes a single-architecture ARM64, CUDA 13, SM100 image for the self-hosted CI runner under the py3.12-cuda13.0.0-sm100-{latest,sha-*} tags. It carries the matching prebuilt kernel wheel so runner jobs can validate and install the exact source and ABI match instead of recompiling it in every lane.

The optional Dreamverse matrix builds backend and UI images for CUDA 12.6 and CUDA 13 on amd64. Dreamverse remains amd64-only because its FA4 dependency stack is not yet validated on ARM64.

The reusable implementation lives in .github/workflows/_template-build-image.yml.

Package Publishing¶

Community Automations¶

File Reference¶