Skip to content
FastVideo
Esc
↑↓navigate↵open⌘Jpreview
On this page

MiniMax H3 Ref2VA synthetic training fixture

Source: examples/training/finetune/minimax-h3/synthetic

This directory demonstrates the raw manifest accepted by the MiniMax H3 Ref2VA preprocessor. The checked-in media is deterministic, repository-owned synthetic content: a geometric animation, a still geometric reference, and mathematically generated tones. It contains no people or third-party media.

Raw manifest contract

The preprocessor accepts JSON, JSONL, or a JSON object containing a data array. Every sample has this shape:

{
  "schema_version": "minimax_h3_ref2va_raw_v1",
  "id": "unique-sample-id",
  "target": {"video": "relative/or/absolute/target.mp4"},
  "caption": "The target description.",
  "references": [
    {"type": "image", "image": "reference.png"},
    {"type": "video", "video": "silent-reference.mp4"},
    {"type": "video_audio", "video": "reference-with-sound.mp4", "audio": "reference-with-sound.mp4"},
    {"type": "audio", "audio": "reference.wav"}
  ]
}

Relative media paths resolve from the manifest directory. video is always a silent visual reference even when its container has an audio stream; video_audio explicitly uses both streams and requires both video and audio fields. When both fields name the same container, its own audio stream is used as the soundtrack. A sample supports at most 12 total references: 9 images, 3 videos, and 3 standalone audio references. A non-empty reference list must include at least one image or video. FL2VA training is not part of this workflow.

Preprocessing encodes target video/audio latents, dynamic Qwen3-VL hidden states and tags, ordered visual anchors at clean-time 0.999, and clean audio anchors. It writes one validated Parquet row per shard. The video VAE, audio VAE, and Qwen3-VL stack are loaded sequentially on one GPU; plan for the full MiniMax H3 component checkpoint and substantial preprocessing time even for a small manifest.

Fixture generation and provenance

generate_fixture.py is the complete source for all three checked-in media files. It uses fixed formulas and codec settings; it requires NumPy, Pillow, and PyAV from the FastVideo environment. The visual/audio content is source reproducible, but MP4/AAC container bytes are not claimed to be bit-identical across PyAV, FFmpeg, or libx264 versions. PROVENANCE.json records the exact generation library versions used for the checked-in bytes. Regenerate or verify them with:

python examples/training/finetune/minimax-h3/synthetic/generate_fixture.py
python examples/training/finetune/minimax-h3/synthetic/generate_fixture.py --verify

--verify checks the checked-in bytes against the recorded hashes; regeneration updates both the media and provenance for the active codec environment. PROVENANCE.json records the exact generation parameters and SHA-256 digest of each media file. The source and generated artifacts are licensed under the repository’s Apache-2.0 license. They have no external source, third-party rights, likeness, voice, consent, or redistribution dependency.

Configurable setup

The setup script verifies the checked-in fixture before manifest validation and accepts all machine-specific inputs through environment variables:

MODEL_DIR=/path/to/MiniMax-H3 \
MANIFEST=/path/to/train.jsonl \
OUTPUT_DIR=/path/to/preprocessed-ref2va \
PYTHON_BIN=/path/to/python \
CUDA_VISIBLE_DEVICES=0 \
bash examples/train/setup_minimax_h3_ref2va_synthetic_single_sample.sh

ENV_FILE=/path/to/optional.env may be supplied when the selected checkpoint requires environment-based authentication. It is never required for local inputs. Existing non-empty output is refused by default. Set REPLACE_EXISTING=1 to stage and validate a replacement; the previous dataset is moved to a sibling .backup-* directory rather than deleted.

The example training configs describe a 64-rank topology with SP=8 and HSDP 8x8. Adjust paths and distributed dimensions to the actual cluster. A modular training checkpoint is DCP state, not an inference directory. Export a checkpoint with:

python -m fastvideo.train.entrypoint.dcp_to_diffusers \
  --checkpoint /path/to/checkpoint-400 \
  --output-dir /path/to/minimax-h3-export \
  --verify

Full Ref2VA tuning replaces transformer_ref/. H3 LoRA export merges the adapter into native transformer weights and writes the same component used by the public T2VA or Ref2VA inference pipeline. Standalone H3 adapter export and runtime adapter selection are not supported by this workflow.

Export gathers the complete transformer state on CPU while the live training model still exists. --verify releases that training graph before strict reload, but it does not make the gather streaming. For the roughly 62 GiB H3 transformer, use a host with comfortably more than the combined live-model, gathered-state, and runtime working set; a 121 GiB unified-memory GB10 is not a validated full-export target.

Additional Files

??? note “PROVENANCE.json”

```json
{
  "copyright": "FastVideo contributors",
  "description": "Deterministic geometric animation and mathematically synthesized tones.",
  "generation_environment": {
    "audio_encoder": "aac",
    "libav": {
      "libavcodec": "62.28.102",
      "libavdevice": "62.3.102",
      "libavfilter": "11.14.102",
      "libavformat": "62.12.102",
      "libavutil": "60.26.102",
      "libswresample": "6.3.102",
      "libswscale": "9.5.102"
    },
    "numpy": "2.5.2",
    "pillow": "11.3.0",
    "pyav": "18.1.0",
    "video_encoder": "libx264"
  },
  "generator": "generate_fixture.py",
  "human_subjects": false,
  "license": "Apache-2.0",
  "parameters": {
    "fps": 24,
    "height": 128,
    "num_frames": 124,
    "sample_rate": 32000,
    "width": 224
  },
  "schema_version": "fastvideo_synthetic_fixture_v1",
  "sha256": {
    "refs/audios/synthetic_tones.wav": "20478230383ea1dc09cb06f84a737d04afc797e93e094d4909427b557e9579f5",
    "refs/images/geometric_reference.png": "3e5a5783e3673c03e9fd14fa6e8b97c7364c5385ae3da3a87fe60a27f8fb04db",
    "targets/geometric_motion.mp4": "058d8a11d151c0959c6ba56c4e56e82aa7a77952f7be35589c5d96458bcf4f71"
  },
  "third_party_content": false
}

```

??? note “generate_fixture.py”

```py
# SPDX-License-Identifier: Apache-2.0
"""Generate the repository-owned MiniMax H3 Ref2VA synthetic fixture."""

from __future__ import annotations

import argparse
import hashlib
import json
import math
from pathlib import Path
import wave

import av
import numpy as np
from PIL import Image, ImageDraw, __version__ as PILLOW_VERSION

ROOT = Path(__file__).resolve().parent
VIDEO_PATH = ROOT / "targets" / "geometric_motion.mp4"
IMAGE_PATH = ROOT / "refs" / "images" / "geometric_reference.png"
AUDIO_PATH = ROOT / "refs" / "audios" / "synthetic_tones.wav"
PROVENANCE_PATH = ROOT / "PROVENANCE.json"

WIDTH = 224
HEIGHT = 128
FPS = 24
NUM_FRAMES = 124
SAMPLE_RATE = 32_000
NUM_SAMPLES = math.ceil(NUM_FRAMES * SAMPLE_RATE / FPS)
MEDIA_PATHS = (VIDEO_PATH, IMAGE_PATH, AUDIO_PATH)


def _pixels(frame_index: int) -> np.ndarray:
    x = np.arange(WIDTH, dtype=np.uint16)[None, :]
    y = np.arange(HEIGHT, dtype=np.uint16)[:, None]
    phase = frame_index * 3
    pixels = np.empty((HEIGHT, WIDTH, 3), dtype=np.uint8)
    pixels[..., 0] = ((x + phase) % 256).astype(np.uint8)
    pixels[..., 1] = ((2 * y + phase) % 256).astype(np.uint8)
    pixels[..., 2] = (((x // 2) + (y // 2) + phase) % 256).astype(np.uint8)

    image = Image.fromarray(pixels)
    draw = ImageDraw.Draw(image)
    for grid_x in range(0, WIDTH, 32):
        draw.line((grid_x, 0, grid_x, HEIGHT - 1), fill=(235, 240, 255), width=1)
    for grid_y in range(0, HEIGHT, 32):
        draw.line((0, grid_y, WIDTH - 1, grid_y), fill=(235, 240, 255), width=1)
    center_x = 20 + (frame_index * 3) % (WIDTH - 40)
    center_y = HEIGHT // 2 + round(22 * math.sin(frame_index * 2 * math.pi / NUM_FRAMES))
    draw.ellipse((center_x - 12, center_y - 12, center_x + 12, center_y + 12),
                 fill=(20, 70, 220), outline=(255, 255, 255), width=2)
    return np.asarray(image)


def _stereo_samples() -> np.ndarray:
    sample = np.arange(NUM_SAMPLES, dtype=np.float64)
    seconds = sample / SAMPLE_RATE
    phase = 2 * np.pi * (220.0 * seconds + 0.5 * 110.0 * seconds**2 / (NUM_SAMPLES / SAMPLE_RATE))
    left = 0.24 * np.sin(phase) + 0.08 * np.sin(2 * np.pi * 440.0 * seconds)
    right = 0.24 * np.sin(phase + np.pi / 3) + 0.08 * np.sin(2 * np.pi * 330.0 * seconds)
    return np.round(np.stack((left, right), axis=1) * 32767).astype(np.int16)


def _write_image() -> None:
    image = Image.fromarray(_pixels(0))
    image.save(IMAGE_PATH, format="PNG", compress_level=9, optimize=False)


def _write_audio(samples: np.ndarray) -> None:
    with wave.open(str(AUDIO_PATH), "wb") as output:
        output.setnchannels(2)
        output.setsampwidth(2)
        output.setframerate(SAMPLE_RATE)
        output.writeframes(samples.astype("<i2", copy=False).tobytes())


def _write_video(samples: np.ndarray) -> None:
    with av.open(str(VIDEO_PATH), "w") as container:
        container.metadata.clear()
        video_stream = container.add_stream("libx264", rate=FPS, options={"preset": "slow", "crf": "18"})
        video_stream.width = WIDTH
        video_stream.height = HEIGHT
        video_stream.pix_fmt = "yuv420p"
        video_stream.codec_context.thread_count = 1
        audio_stream = container.add_stream("aac", rate=SAMPLE_RATE)
        audio_stream.codec_context.layout = "stereo"

        for frame_index in range(NUM_FRAMES):
            frame = av.VideoFrame.from_ndarray(_pixels(frame_index), format="rgb24")
            frame.pts = frame_index
            container.mux(video_stream.encode(frame))
        container.mux(video_stream.encode())

        interleaved = samples.reshape(1, -1)
        audio_frame = av.AudioFrame.from_ndarray(interleaved, format="s16", layout="stereo")
        audio_frame.sample_rate = SAMPLE_RATE
        resampler = av.audio.resampler.AudioResampler(format="fltp", layout="stereo", rate=SAMPLE_RATE)
        pts = 0
        for frame in resampler.resample(audio_frame):
            frame.pts = pts
            pts += frame.samples
            container.mux(audio_stream.encode(frame))
        container.mux(audio_stream.encode())


def _sha256(path: Path) -> str:
    digest = hashlib.sha256()
    with path.open("rb") as handle:
        for chunk in iter(lambda: handle.read(1024 * 1024), b""):
            digest.update(chunk)
    return digest.hexdigest()


def _relative(path: Path) -> str:
    return path.relative_to(ROOT).as_posix()


def generate() -> None:
    for path in MEDIA_PATHS:
        path.parent.mkdir(parents=True, exist_ok=True)
    samples = _stereo_samples()
    _write_image()
    _write_audio(samples)
    _write_video(samples)
    provenance = {
        "schema_version": "fastvideo_synthetic_fixture_v1",
        "generator": "generate_fixture.py",
        "license": "Apache-2.0",
        "copyright": "FastVideo contributors",
        "third_party_content": False,
        "human_subjects": False,
        "description": "Deterministic geometric animation and mathematically synthesized tones.",
        "generation_environment": {
            "numpy": np.__version__,
            "pillow": PILLOW_VERSION,
            "pyav": av.__version__,
            "libav": {name: ".".join(str(part) for part in version)
                      for name, version in sorted(av.library_versions.items())},
            "video_encoder": "libx264",
            "audio_encoder": "aac",
        },
        "parameters": {
            "width": WIDTH,
            "height": HEIGHT,
            "fps": FPS,
            "num_frames": NUM_FRAMES,
            "sample_rate": SAMPLE_RATE,
        },
        "sha256": {_relative(path): _sha256(path) for path in MEDIA_PATHS},
    }
    PROVENANCE_PATH.write_text(json.dumps(provenance, indent=2, sort_keys=True) + "\n", encoding="utf-8")


def verify() -> None:
    provenance = json.loads(PROVENANCE_PATH.read_text(encoding="utf-8"))
    expected = provenance.get("sha256", {})
    actual = {_relative(path): _sha256(path) for path in MEDIA_PATHS}
    if actual != expected:
        raise SystemExit(f"Synthetic fixture hash mismatch:\nexpected={expected}\nactual={actual}")
    print(json.dumps(actual, indent=2, sort_keys=True))


def main() -> None:
    parser = argparse.ArgumentParser(description=__doc__)
    parser.add_argument("--verify", action="store_true", help="Verify checked-in media against PROVENANCE.json")
    args = parser.parse_args()
    if args.verify:
        verify()
    else:
        generate()
        verify()


if __name__ == "__main__":
    main()

```

??? note “train.jsonl”

```jsonl
{"schema_version":"minimax_h3_ref2va_raw_v1","id":"synthetic_geometric_train","target":{"video":"targets/geometric_motion.mp4"},"caption":"A blue circle moves across a colorful geometric grid while synthetic tones sweep gently in pitch.","references":[{"type":"image","image":"refs/images/geometric_reference.png"},{"type":"audio","audio":"refs/audios/synthetic_tones.wav"}]}

```

??? note “validation.json”

```json
{
  "data": [
    {
      "schema_version": "minimax_h3_ref2va_raw_v1",
      "id": "synthetic_geometric_validation",
      "target": {"video": "targets/geometric_motion.mp4"},
      "caption": "A blue circle moves across a colorful geometric grid while synthetic tones sweep gently in pitch.",
      "references": [
        {"type": "image", "image": "refs/images/geometric_reference.png"},
        {"type": "audio", "audio": "refs/audios/synthetic_tones.wav"}
      ]
    }
  ]
}

```

Was this page helpful?