Skip to content
FastVideo
Esc
↑↓navigate↵open⌘Jpreview
On this page

metric

VBench Imaging Quality — MUSIQ-based per-frame technical quality.

Uses MUSIQ (Multi-Scale Image Quality) from pyiqa. Frames are resized so the longer side is at most 512px. Score = mean(MUSIQ_scores) / 100.

fastvideo.eval.metrics.vbench.imaging_quality.metric.BaseMetric

BaseMetric()

Abstract base class for all eval metrics.

Two execution shapes:

  • Per-sample (is_set_metric=False, default) — implement :meth:compute. The Evaluator calls it once per input sample and returns one :class:MetricResult per sample.

  • Set-vs-set (is_set_metric=True) — implement :meth:accumulate (called once per sample to buffer features) and :meth:finalize (called once after all samples to compute the corpus-level result). Use :meth:reset to clear buffers and :meth:merge_from to fold multi-GPU per-worker state together.

Optionally override :meth:setup to eagerly load models. Metrics that chunk along the time dim for memory hardcode their own chunk size in __init__ (see optical_flow for the canonical example). Eval always processes one video per :meth:Evaluator.evaluate call; compute / accumulate receive a single sample, not a batch.

fastvideo.eval.metrics.vbench.imaging_quality.metric.BaseMetric.to

to(device: str | torch.device) -> BaseMetric

Move metric (and its internal models) to device.

fastvideo.eval.metrics.vbench.imaging_quality.metric.BaseMetric.setup

setup() -> None

Eagerly load models. Called once by :class:EvalWorker.

Default is a no-op; metrics with no eager state (pixel math, closed-form ops) inherit this. Override only if your metric needs to load weights.

fastvideo.eval.metrics.vbench.imaging_quality.metric.BaseMetric.compute

compute(sample: dict) -> MetricResult

Per-sample metrics: compute the score for one sample.

sample["video"] is (T, C, H, W) float in [0, 1]. sample["reference"] (if used) has the same shape. Return self._skip(sample, reason) for missing inputs.

fastvideo.eval.metrics.vbench.imaging_quality.metric.BaseMetric.reset

reset() -> None

Clear accumulator state at the start of each evaluate() call.

fastvideo.eval.metrics.vbench.imaging_quality.metric.BaseMetric.accumulate

accumulate(sample: dict) -> None

Buffer per-sample features for a corpus-level metric.

fastvideo.eval.metrics.vbench.imaging_quality.metric.BaseMetric.finalize

finalize() -> MetricResult

Compute the corpus-level result from buffered state.

fastvideo.eval.metrics.vbench.imaging_quality.metric.BaseMetric.merge_from

merge_from(other: BaseMetric) -> None

Multi-GPU: fold another worker's accumulator state into this one.

fastvideo.eval.metrics.vbench.imaging_quality.metric.register

register(name: str)

Decorator to register a metric class.

Usage::

@register("ssim")
class SSIMMetric(BaseMetric):
    ...

fastvideo.eval.metrics.vbench.imaging_quality.metric.MetricResult

MetricResult(name: str, score: float | None, details: dict[str, Any] = dict())

Standard result container returned by all metrics.

score is None when the metric was skipped (e.g. missing required input). Check details["skipped"] for the reason.

fastvideo.eval.metrics.vbench.imaging_quality.metric.ImagingQualityMetric

ImagingQualityMetric()
Source
@register("vbench.imaging_quality")
class ImagingQualityMetric(BaseMetric):

name = "vbench.imaging_quality"
requires_reference = False
higher_is_better = True
needs_gpu = True
dependencies = ["pyiqa"]

def __init__(self) -> None:
    super().__init__()
    self._model: Any = None

def to(self, device):
    super().to(device)
    if self._model is not None:
        self._model = self._model.to(self.device)
    return self

def setup(self) -> None:
    if self._model is not None:
        return
    import pyiqa
    self._model = pyiqa.create_metric("musiq-spaq", device=self.device)
    self._model.eval()

@torch.no_grad()
def compute(self, sample: dict) -> MetricResult:
    video = sample["video"]  # (T, C, H, W)
    T, _, H, W = video.shape

    if max(H, W) > 512:
        scale = 512.0 / max(H, W)
        new_h, new_w = int(H * scale), int(W * scale)
    else:
        new_h, new_w = H, W

    frames = video.to(self.device)
    if (new_h, new_w) != (H, W):
        # antialias=False matches VBench's imaging_quality.transform
        frames = resize(frames, [new_h, new_w], antialias=False)

    chunk = self._chunk_size or 32
    chunks: list[torch.Tensor] = []
    for i in range(0, T, chunk):
        scores = self._model(frames[i:i + chunk])
        chunks.append(scores.squeeze(-1))
    per_frame = torch.cat(chunks, dim=0)  # (T,)
    return MetricResult(
        name=self.name,
        score=float(per_frame.mean().item()) / 100.0,
        details={"per_frame_raw": per_frame.tolist()},
    )</code></pre></details>


fastvideo.eval.metrics.vbench.imaging_quality.metric.ImagingQualityMetric.to


to(device)


Source
def to(self, device):
    super().to(device)
    if self._model is not None:
        self._model = self._model.to(self.device)
    return self


fastvideo.eval.metrics.vbench.imaging_quality.metric.ImagingQualityMetric.setup


setup() -> None


Source
def setup(self) -> None:
    if self._model is not None:
        return
    import pyiqa
    self._model = pyiqa.create_metric("musiq-spaq", device=self.device)
    self._model.eval()


fastvideo.eval.metrics.vbench.imaging_quality.metric.ImagingQualityMetric.compute


compute(sample: dict) -> MetricResult


Source
@torch.no_grad()
def compute(self, sample: dict) -> MetricResult:
    video = sample["video"]  # (T, C, H, W)
    T, _, H, W = video.shape

if max(H, W) &gt; 512:
    scale = 512.0 / max(H, W)
    new_h, new_w = int(H * scale), int(W * scale)
else:
    new_h, new_w = H, W

frames = video.to(self.device)
if (new_h, new_w) != (H, W):
    # antialias=False matches VBench&#x27;s imaging_quality.transform
    frames = resize(frames, &#91;new_h, new_w&#93;, antialias=False)

chunk = self._chunk_size or 32
chunks: list&#91;torch.Tensor&#93; = &#91;&#93;
for i in range(0, T, chunk):
    scores = self._model(frames&#91;i:i + chunk&#93;)
    chunks.append(scores.squeeze(-1))
per_frame = torch.cat(chunks, dim=0)  # (T,)
return MetricResult(
    name=self.name,
    score=float(per_frame.mean().item()) / 100.0,
    details=&#123;&quot;per_frame_raw&quot;: per_frame.tolist()&#125;,
)</code></pre></details>

Was this page helpful?