Skip to content

preprocess_kandinsky5_overfit

Preprocess Kandinsky5 overfit data into parquet format.

Encodes videos with Kandinsky5's VAE (shared with HunyuanVideo) and captions with dual text encoders (Qwen/Reason1 + CLIP) into the t2v parquet schema expected by the training framework.

Unlike Hunyuan's version of this script, text encoders are loaded via FastVideo's own TextEncoderLoader/VAELoader (matching Kandinsky5Model.ensure_negative_conditioning) rather than raw transformers classes -- Kandinsky5's Qwen/Reason1 encoder is a native FastVideo wrapper around Qwen2.5-VL's language model, not a plain AutoModel.from_pretrained load.

The CLIP pooled projection ([768]) is zero-padded into a row and prepended to the Qwen sequence embeddings, then stored as a single [seq+1, dim] tensor in the existing text_embedding parquet field -- the same trick preprocess_hunyuan_overfit.py uses for LLaMA+CLIP. No schema change and no separate attention-mask field are needed: the standard collator derives text_attention_mask from padding against the stored (unpadded) row count, so the prepended pooled row is automatically counted as valid.

Usage

CUDA_VISIBLE_DEVICES=0 python -m fastvideo.pipelines.preprocess.preprocess_kandinsky5_overfit

Input/output roots default to data/kandinsky5_overfit / data/kandinsky5_overfit_preprocessed and can be overridden with the KANDINSKY5_OVERFIT_DATA_DIR / KANDINSKY5_OVERFIT_OUTPUT_DIR env vars (used by the nightly e2e test to keep its disposable roots separate from a user's real dataset at the defaults).

Classes

Functions:

fastvideo.pipelines.preprocess.preprocess_kandinsky5_overfit.get_caption 

get_caption(item: dict) -> str

Extract a caption from one videos2caption.json entry.

The documented schema (docs/training/data_preprocess.md) stores cap as a plain string; some producers (e.g. this repo's own e2e test fixtures, mirroring preprocess_hunyuan_overfit.py) instead store a non-empty list of caption variants and use the first one. Indexing unconditionally with item["cap"][0] silently takes the first character of a string caption instead of erroring, so accept and validate both forms explicitly here.

Source code in fastvideo/pipelines/preprocess/preprocess_kandinsky5_overfit.py 
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
def get_caption(item: dict) -> str:
    """Extract a caption from one ``videos2caption.json`` entry.

    The documented schema (``docs/training/data_preprocess.md``) stores
    ``cap`` as a plain string; some producers (e.g. this repo's own e2e
    test fixtures, mirroring ``preprocess_hunyuan_overfit.py``) instead
    store a non-empty list of caption variants and use the first one.
    Indexing unconditionally with ``item["cap"][0]`` silently takes the
    first *character* of a string caption instead of erroring, so accept
    and validate both forms explicitly here.
    """
    cap = item.get("cap")
    if isinstance(cap, str):
        if not cap:
            raise ValueError(f"Empty 'cap' string for entry: {item}")
        return cap
    if isinstance(cap, list):
        if not cap or not isinstance(cap[0], str) or not cap[0]:
            raise ValueError(f"'cap' list must be non-empty with a non-empty first string, got: {item}")
        return cap[0]
    raise ValueError(f"'cap' must be a string or a list of strings, got {type(cap).__name__}: {item}")

fastvideo.pipelines.preprocess.preprocess_kandinsky5_overfit.load_video 

load_video(path: str, num_frames: int, height: int, width: int) -> Tensor

Load video as [1, C, T, H, W] in [-1, 1], resized to (height, width).

Source clips are frequently at native resolution (e.g. 1920x1080); encoding that directly through the VAE (instead of at the target training resolution) uses far more memory than intended and can OOM.

Source code in fastvideo/pipelines/preprocess/preprocess_kandinsky5_overfit.py 
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
def load_video(path: str, num_frames: int, height: int, width: int) -> torch.Tensor:
    """Load video as [1, C, T, H, W] in [-1, 1], resized to (height, width).

    Source clips are frequently at native resolution (e.g. 1920x1080);
    encoding that directly through the VAE (instead of at the target
    training resolution) uses far more memory than intended and can OOM.
    """
    cap = cv2.VideoCapture(path)
    frames: list[np.ndarray] = []
    while len(frames) < num_frames:
        ret, frame = cap.read()
        if not ret:
            break
        frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
        if frame.shape[0] != height or frame.shape[1] != width:
            frame = cv2.resize(frame, (width, height), interpolation=cv2.INTER_AREA)
        frames.append(frame)
    cap.release()

    if not frames:
        # Without this, the repeat-last-frame fill below raises an opaque
        # IndexError on frames[-1]; cv2.VideoCapture doesn't raise on a
        # missing/corrupt file, it just decodes nothing.
        raise ValueError(f"Could not decode any frames from video {path!r} -- "
                         "the file is missing, empty, or not readable by OpenCV.")

    if len(frames) < num_frames:
        # Repeat last frame to fill
        while len(frames) < num_frames:
            frames.append(frames[-1])

    frames = frames[:num_frames]
    video = np.stack(frames, axis=0)
    video = torch.from_numpy(video).float()
    video = video / 127.5 - 1.0  # [0,255] -> [-1,1]
    video = video.permute(3, 0, 1, 2).unsqueeze(0)  # [1,C,T,H,W]
    return video