Preprocess MiniMax-H3 reference-conditioned training data

MiniMax-H3 DMD/PDD training consumes precomputed target video/audio latents, Qwen3-VL embeddings and ordered reference conditions. The packed training adapter uses pyarrow_schema_ref2va: clean normalized visual reference rows, clean audio rows, and the reference geometry needed to reconstruct the packed sequence. The adapter applies visual-reference noise augmentation during batch preparation using the stored reference_noise_seed.

This differs from the existing Ref2VA SFT minimax_h3_ref2va_precomputed_v1 format, which stores already augmented ref_visual_anchor tensors. Keep using preprocess_minimax_h3_ref2va for that format. Do not rename or directly combine shards from these two schemas.

Input

Provide your own local MiniMax-H3 checkpoint with vae, audio_vae, tokenizer, processor, text_encoder and transformer_ref components. Encoding runs on one CUDA device and releases each large component before loading the next. The trainer subsequently reads the encoded tensors without loading the encoders.

The input accepts JSON or JSONL using the public raw Ref2VA schema. Paths are relative to the manifest. Each row needs a target video with synchronized audio, a caption, and at least one image or video reference. This example contains no bundled media:

{
  "schema_version": "minimax_h3_ref2va_raw_v1",
  "id": "example-1",
  "target": {"video": "media/target.mp4"},
  "caption": "The person walks through the scene.",
  "references": [
    {"type": "image", "image": "media/person.png"},
    {"type": "audio", "audio": "media/voice.wav"}
  ],
  "target_geometry": {"width": 832, "height": 480, "num_frames": 124}
}

References remain in their declared order. Supported entries are image (with image), silent video (with video and optional fps), audio (with audio and optional sample_rate), and video_audio (with both video and audio, plus optional rates). Images/videos follow the existing inference reference sizing and presentation rules. The target canvas is independent of the reference geometry.

target_geometry is optional and overrides the command’s defaults for that row. Width and height must be positive multiples of 32. The frame count must be 17 * n + 5, for a nonnegative integer n. Targets are resampled to 24 fps, trimmed to that count and resized to the explicitly requested canvas using Lanczos interpolation. Target stereo audio is resampled to 32 kHz and trimmed to the same duration. Short video or audio inputs fail instead of being extended. Choose canvases with the desired aspect ratio; resizing does not preserve a mismatched source aspect ratio.

Encode

python -m fastvideo.pipelines.preprocess.preprocess_minimax_h3_packed_ref2va \
  --manifest data/my_ref2va/train.jsonl \
  --model-path data/models/MiniMax-H3 \
  --output-dir data/my_ref2va/encoded \
  --width 1344 --height 768 --num-frames 124 \
  --max-presentation-tokens 65536

Add --validate-manifest-only to check paths, reference declarations and geometry without loading models. Actual encoding requires CUDA and the normal MiniMax-H3 preprocessing dependencies. This single-process command does not launch a training job or manage a cluster scheduler.

The command preserves the complete multimodal presentation. The presentation limit rejects an oversized sample; it never truncates text or reference tokens. Each row is checked for target shape, finite tensors and successful Parquet roundtrip. All output is staged and published after every row passes; a nonempty output directory is refused. Use a fresh directory for a changed dataset.

Output has one row per shard under directories such as bucket=832x480-124f/data_00000000.parquet. Preserve these bucket names: native sampling uses them to schedule matching target shapes across data-parallel workers. Metadata keeps the original order and geometry of every reference, text/vision token tags and the fixed reference noise seed. Video targets are [24, T, H/16, W/16]; audio targets are [2, 32, T_audio]; clean reference rows are [N_visual, 96] and [N_audio, 32]. Reference audio rows include both stereo channels in channel-major order.

Connect to training

For the packed reference-conditioned DMD/PDD adapter, set:

training:
  data:
    data_path: data/my_ref2va/encoded
    preprocessed_data_type: ref2va
    train_batch_size: 1
    native_shape_bucketing: true
    max_presentation_tokens: 65536
    require_existing_parquet_cache: false

The first normal dataset initialization builds its file index under map_style_cache. For later runs, require_existing_parquet_cache: true requires a valid existing index and fails instead of rescanning. This index is a pickle file: use only an index you trust and regenerate it after moving or changing the dataset. Index paths must remain inside the dataset root.

Native sampling groups exact target shapes into global microbatches. Rare shape buckets repeat rows within that bucket to fill a global microbatch; the sampler reports the repeats. This preserves each bucket’s presence but can change its relative frequency when buckets are small. SP ranks share the same local rows.

Optional fixed_sequence_capacities selects the smallest fitting padded joint sequence length from positive multiples of 256. The length includes target video/audio, text and reference tokens. Oversized rows are rejected during metadata admission; capacities never crop a document. Capacity padding is separate from cross-sample packing, which the Ref2VA adapter does not support. See reference-conditioned DMPDD training for model roles, optimization settings and the training command.