datasets

Prompt-corpus datasets for end-to-end benchmark evaluation.

Public API mirrors :mod:fastvideo.eval (metrics side):

from fastvideo.eval.datasets import (
    PromptDataset, Sample,
    register_dataset, get_dataset, list_datasets,
)

A dataset is an iterable of plain dicts (one per sample). Built-in datasets self-register at import time. To add one, drop a module into this package that subclasses :class:PromptDataset and decorates with @register_dataset("name") — auto-discovery picks it up.

fastvideo.eval.datasets.PromptDataset

PromptDataset()

Iterable corpus of sample dicts. Subclasses populate self._rows.

fastvideo.eval.datasets.PromptDataset.by_dimension

by_dimension() -> dict[str, list[dict]]

Group samples by dimension. A multi-dim sample appears under each.

fastvideo.eval.datasets.Sample

Sample()

Documented schema for a row yielded by :class:PromptDataset.

Only prompt is required. Extra keys beyond these are forwarded to the runner's eval-kwargs builder verbatim, so action-conditioned or audio-bearing benchmarks can add their own fields without changing the base class.

fastvideo.eval.datasets.get_dataset

get_dataset(name: str, **kwargs: Any) -> BasePromptDataset

Instantiate a registered dataset by name.

fastvideo.eval.datasets.list_datasets

list_datasets() -> list[str]

Return sorted list of all registered dataset names.

fastvideo.eval.datasets.register_dataset

register_dataset(name: str)

Decorator to register a prompt-dataset class.

Usage::

@register_dataset("vbench")
class VBenchPromptDataset(BasePromptDataset):
    ...

fastvideo.eval.datasets.VBenchPromptDataset

VBenchPromptDataset(dimensions: list[str] | str = 'all', full_info_path: str | Path | None = None)

VBench prompts filtered by evaluation dimension.

Parameters

dimensions: list[str] | str

List of dimension names, or "all". Unknown dimensions raise ValueError.

full_info_path: str | Path | None

Optional override for VBench_full_info.json; defaults to autodetection.

A prompt that belongs to several requested dimensions is yielded once; its dimensions list carries all matches so the scorer can route.