Skip to content

test_evaluator_single

End-to-end tests for single-replica eval through the public API.

Runs the lightweight pixel-space metrics — common.psnr and common.ssim — under both shapes that real callers use:

  • one-shot evaluate(video=..., reference=...) (the helper in fastvideo.eval.api);
  • a long-lived Evaluator, called once per sample;
  • a long-lived Evaluator, called with a list of sample dicts to fan out (samples=[...]).

GPU-only metrics live in separate test modules / classes; everything here runs on CPU so the suite stays cheap to invoke.

Classes

Functions

fastvideo.tests.eval.test_evaluator_single.gen_ref 

gen_ref()

Reproducible (gen, ref) pair shaped (T, C, H, W).

Source code in fastvideo/tests/eval/test_evaluator_single.py 
27
28
29
30
31
32
33
@pytest.fixture
def gen_ref():
    """Reproducible (gen, ref) pair shaped (T, C, H, W)."""
    torch.manual_seed(0)
    gen = torch.rand(_T, _C, _H, _W)
    ref = torch.rand(_T, _C, _H, _W)
    return gen, ref

fastvideo.tests.eval.test_evaluator_single.test_evaluator_accepts_legacy_5d_input 

test_evaluator_accepts_legacy_5d_input(evaluator, gen_ref)

Callers that still pass (1, T, C, H, W) should get unwrapped.

Source code in fastvideo/tests/eval/test_evaluator_single.py 
85
86
87
88
89
def test_evaluator_accepts_legacy_5d_input(evaluator, gen_ref):
    """Callers that still pass ``(1, T, C, H, W)`` should get unwrapped."""
    gen, ref = gen_ref
    out = evaluator.evaluate(video=gen.unsqueeze(0), reference=ref.unsqueeze(0))
    _assert_well_formed(out["common.psnr"], "common.psnr")

fastvideo.tests.eval.test_evaluator_single.test_evaluator_psnr_identical_videos_is_high 

test_evaluator_psnr_identical_videos_is_high(evaluator, gen_ref)

PSNR(x, x) is unbounded above; with our clamp it caps near 100 dB.

Source code in fastvideo/tests/eval/test_evaluator_single.py 
100
101
102
103
104
105
def test_evaluator_psnr_identical_videos_is_high(evaluator, gen_ref):
    """PSNR(x, x) is unbounded above; with our clamp it caps near 100 dB."""
    gen, _ = gen_ref
    out = evaluator.evaluate(video=gen, reference=gen)
    assert out["common.psnr"].score > 50.0
    assert out["common.ssim"].score == pytest.approx(1.0, abs=1e-5)

fastvideo.tests.eval.test_evaluator_single.test_evaluator_samples_list_preserves_input_order 

test_evaluator_samples_list_preserves_input_order(evaluator)

When samples=[...] is passed, results must come back per sample.

Source code in fastvideo/tests/eval/test_evaluator_single.py 
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
def test_evaluator_samples_list_preserves_input_order(evaluator):
    """When ``samples=[...]`` is passed, results must come back per sample."""
    torch.manual_seed(1)
    samples = []
    for i in range(4):
        # Vary the reference enough that scores differ across rows.
        gen = torch.rand(_T, _C, _H, _W)
        ref = gen + 0.01 * (i + 1) * torch.rand_like(gen)
        samples.append({"video": gen, "reference": ref})

    out = evaluator.evaluate(samples=samples)
    assert isinstance(out, list)
    assert len(out) == len(samples)
    for row in out:
        assert set(row.keys()) == {"common.psnr", "common.ssim"}
        _assert_well_formed(row["common.psnr"], "common.psnr")

    # Re-running should give bit-identical scores (no nondeterministic
    # scheduling effects under single-GPU dispatch).
    out2 = evaluator.evaluate(samples=samples)
    for a, b in zip(out, out2):
        assert a["common.psnr"].score == pytest.approx(b["common.psnr"].score)