Skip to content

test_registry

Smoke tests for the metric registry surface.

These exercise the public fastvideo.eval API only: list_metrics, get_metric, and the group-resolution logic that create_evaluator(metrics="vbench") uses.

Functions

fastvideo.tests.eval.test_registry.test_create_evaluator_resolves_group_prefix 

test_create_evaluator_resolves_group_prefix()

metrics="<group>" should expand to every <group>.* sub-metric.

Use the physics_iq group because it has multiple sub-metrics and none of them load model weights — the group-resolution behavior is what we're testing, not metric setup.

Source code in fastvideo/tests/eval/test_registry.py 
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
def test_create_evaluator_resolves_group_prefix():
    """``metrics="<group>"`` should expand to every ``<group>.*`` sub-metric.

    Use the ``physics_iq`` group because it has multiple sub-metrics and
    none of them load model weights — the group-resolution behavior is
    what we're testing, not metric setup."""
    ev = create_evaluator(metrics="physics_iq", device="cpu")
    try:
        names = ev.metric_names
        assert all(n.startswith("physics_iq.") for n in names)
        # Don't pin an exact count — sub-metrics may be added later.
        assert "physics_iq.spatial_iou" in names
        assert "physics_iq.spatiotemporal_iou" in names
        assert "physics_iq.mse" in names
    finally:
        ev.shutdown()