HunyuanVideo 1.5 T2V Training¶

This guide walks through training HunyuanVideo 1.5 (480p, text-to-video) in FastVideo's modular training stack: preprocessing one clip, running a single-GPU overfit to verify the pipeline end to end, and scaling out to full-parameter multi-GPU fine-tuning.

Preprocessing runs preprocess_hunyuan15_overfit.py, which encodes both text streams (Qwen2.5-VL and the ByT5 glyph encoder) into the dual-text parquet schema the training plugin reads. Training itself only needs the parquet that step produces.

What you need¶

Install as documented in the repository README:

UV_TORCH_BACKEND=cu126 uv pip install -e ".[dev]"   # cu130 on CUDA 13
python -c "import torch; print(torch.__version__, torch.cuda.is_available())"

fastvideo-kernel pulls in a CUTLASS submodule. If the build fails with cutlass/cutlass.h: No such file or directory, run git submodule update --init --recursive — or skip the kernel package entirely, since this guide uses the PyTorch SDPA attention backend.

1. Prepare a clip¶

The preprocess script reads a caption manifest plus a videos/ subdirectory:

data/hunyuan15_overfit/
├── videos2caption.json
└── videos/
    └── robot_pouring.mp4

The repository already ships a suitable clip (5.8s, 16fps, 1280x704):

mkdir -p data/hunyuan15_overfit/videos
cp assets/videos/robot_pouring.mp4 data/hunyuan15_overfit/videos/

cat > data/hunyuan15_overfit/videos2caption.json <<'EOF'
[
  {
    "path": "robot_pouring.mp4",
    "cap": ["A robotic arm carefully pours liquid from a bottle into a glass on a kitchen counter, steady camera, soft indoor lighting."]
  }
]
EOF

Clips must supply at least 81 frames at 16fps (≈5.1s) and be a single continuous shot; the script resizes to 480x832.

2. Preprocess¶

CUDA_VISIBLE_DEVICES=0 python fastvideo/pipelines/preprocess/preprocess_hunyuan15_overfit.py

This encodes the video with the HY1.5 VAE and the caption with both text encoders, then writes a parquet shard plus a validation prompt file. Expected output:

[Qwen 1/1] (29, 3584)          # caption tokens x Qwen hidden size
[ByT5 1/1] (0, 1472)           # zero-length: the caption has no quoted glyph text
[VAE  1/1] (32, 21, 30, 52)    # 32 channels, 21 latent frames, 480/16 x 832/16
Wrote 1 records to data/hunyuan15_overfit_preprocessed/data_00000.parquet

The script reads data/hunyuan15_overfit and writes to data/hunyuan15_overfit_preprocessed. To use other directories, set FASTVIDEO_TEST_HUNYUAN15_OVERFIT_DATA_DIR and FASTVIDEO_TEST_HUNYUAN15_OVERFIT_OUTPUT_DIR.

A (0, 1472) ByT5 shape is normal and expected: ByT5 only receives the glyph text extracted from quotes in the caption, so most captions produce zero tokens. The training plugin trims and forwards that zero-length stream unchanged.

Note that the parquet stores raw latent_dist.mode() outputs — the training side multiplies by the VAE scaling_factor (1.03682) in normalize_dit_input("hunyuan15", ...).

3. Overfit on a single GPU¶

export WANDB_MODE=offline      # or `wandb login` for cloud tracking
bash examples/train/run.sh examples/train/configs/overfit_hunyuan15_t2v.yaml

The config trains LoRA adapters on the attention projections, which keeps the run at roughly 30GB. Loss is logged through the configured tracker, not to stdout.

For a quick "does it run at all" check, cap the step count:

bash examples/train/run.sh examples/train/configs/overfit_hunyuan15_t2v.yaml \
  --training.loop.max_train_steps 50

Run long jobs under tmux or screen. A dropped SSH session takes the training process with it, and a checkpoint interrupted mid-write is not resumable (the DCP .metadata file is written last).

Reading the loss¶

Flow-matching loss is noisy: every step samples a random timestep, and the denoising task is far easier at low noise levels than at high ones. Two adjacent steps can differ by an order of magnitude with no bearing on whether the model is learning. Compare segment means rather than individual steps:

RUN=$(ls -dt outputs/hunyuan15_overfit/tracker/wandb/offline-run-* | head -1)
strings -n 1 $RUN/run-*.wandb | grep -A 2 -x "total_loss" \
  | grep -E "^[0-9]+\.[0-9]" | uniq > /tmp/losses.txt

python3 -c "
import re
vals = [float(m.group(1)) for line in open('/tmp/losses.txt')
        if (m := re.match(r'^([0-9]+\.[0-9]+)', line))]
n = len(vals)
for i in range(6):
    seg = vals[i * n // 6:(i + 1) * n // 6]
    print(f'segment {i + 1}/6: {sum(seg) / len(seg):.4f}')
"

A steadily falling segment mean means the data → forward → loss → backward → optimizer chain is numerically sound. A flat series, or NaNs, points at a real problem — stop and investigate rather than burning GPU hours.

4. Scale out¶

examples/train/configs/fine_tuning/hunyuan15/t2v.yaml runs full-parameter training across 8 GPUs with FSDP. Full-parameter training does not fit one 80GB card:

It reads data/hunyuan15_preprocessed (and its validation_prompts.json), not the overfit directory above — either preprocess into that path (set FASTVIDEO_TEST_HUNYUAN15_OVERFIT_OUTPUT_DIR=data/hunyuan15_preprocessed) or pass --training.data.data_path and --callbacks.validation.dataset_file.

That subtotal is independent of sequence length, so shortening the clip (--training.data.num_latent_t) barely helps — measured peak memory moved by 0.07GB going from 21 latent frames down to 9. Options that do move the needle:

• LoRA (what the overfit config uses) — gradients and optimizer state shrink to the adapter parameters, ~30GB total.
• FSDP sharding across GPUs — the 66.8GB subtotal divides by the number of shards.
• A memory-light optimizer — SGD with momentum keeps a single state tensor instead of two. torch.optim.Adafactor is smaller still, but is not DTensor-safe: its in-place .square_() fails under FSDP2 with aten.pow_.Scalar: in-place operations that require placement changes are not supported.

Set training.dit_precision: bf16 for any single-device run. The default is fp32, which doubles weights, gradients, and optimizer state.

Troubleshooting¶

Notes on other platforms¶

The stack runs on aarch64 + Blackwell (DGX Spark, GB10, CUDA 13.0): preprocessing and training both complete there. Unified memory makes checkpoint loading much faster than on a discrete GPU — 17GB of weights load in about 1.5s versus 30s over PCIe — while each training step runs roughly 1.5x slower than an A100 80GB. nvidia-smi reports Memory-Usage: Not Supported on that platform; use free -m instead, since GPU allocations come out of the same pool as system memory.