Skip to content
FastVideo
Esc
↑↓navigate↵open⌘Jpreview
On this page

metric

AudioBox Aesthetics (CE, CU, PC, PQ).

Thin wrapper around Meta's audiobox_aesthetics predictor. Returns four per-clip dimensions (CE — Content Enjoyment, CU — Content Usefulness, PC — Production Complexity, PQ — Production Quality); score exposes PQ, the dimension V2A papers typically report on. The remaining three are surfaced under details.

The earlier Verse-Bench combined score (CE + CU + PQ + (11 − PC)) / 4 is non-standard and is deliberately not used.

fastvideo.eval.metrics.audio.audiobox_aesthetics.metric.BaseMetric

BaseMetric()

Abstract base class for all eval metrics.

Two execution shapes:

  • Per-sample (is_set_metric=False, default) — implement :meth:compute. The Evaluator calls it once per input sample and returns one :class:MetricResult per sample.

  • Set-vs-set (is_set_metric=True) — implement :meth:accumulate (called once per sample to buffer features) and :meth:finalize (called once after all samples to compute the corpus-level result). Use :meth:reset to clear buffers and :meth:merge_from to fold multi-GPU per-worker state together.

Optionally override :meth:setup to eagerly load models. Metrics that chunk along the time dim for memory hardcode their own chunk size in __init__ (see optical_flow for the canonical example). Eval always processes one video per :meth:Evaluator.evaluate call; compute / accumulate receive a single sample, not a batch.

fastvideo.eval.metrics.audio.audiobox_aesthetics.metric.BaseMetric.to

to(device: str | torch.device) -> BaseMetric

Move metric (and its internal models) to device.

fastvideo.eval.metrics.audio.audiobox_aesthetics.metric.BaseMetric.setup

setup() -> None

Eagerly load models. Called once by :class:EvalWorker.

Default is a no-op; metrics with no eager state (pixel math, closed-form ops) inherit this. Override only if your metric needs to load weights.

fastvideo.eval.metrics.audio.audiobox_aesthetics.metric.BaseMetric.compute

compute(sample: dict) -> MetricResult

Per-sample metrics: compute the score for one sample.

sample["video"] is (T, C, H, W) float in [0, 1]. sample["reference"] (if used) has the same shape. Return self._skip(sample, reason) for missing inputs.

fastvideo.eval.metrics.audio.audiobox_aesthetics.metric.BaseMetric.reset

reset() -> None

Clear accumulator state at the start of each evaluate() call.

fastvideo.eval.metrics.audio.audiobox_aesthetics.metric.BaseMetric.accumulate

accumulate(sample: dict) -> None

Buffer per-sample features for a corpus-level metric.

fastvideo.eval.metrics.audio.audiobox_aesthetics.metric.BaseMetric.finalize

finalize() -> MetricResult

Compute the corpus-level result from buffered state.

fastvideo.eval.metrics.audio.audiobox_aesthetics.metric.BaseMetric.merge_from

merge_from(other: BaseMetric) -> None

Multi-GPU: fold another worker's accumulator state into this one.

fastvideo.eval.metrics.audio.audiobox_aesthetics.metric.register

register(name: str)

Decorator to register a metric class.

Usage::

@register("ssim")
class SSIMMetric(BaseMetric):
    ...

fastvideo.eval.metrics.audio.audiobox_aesthetics.metric.MetricResult

MetricResult(name: str, score: float | None, details: dict[str, Any] = dict())

Standard result container returned by all metrics.

score is None when the metric was skipped (e.g. missing required input). Check details["skipped"] for the reason.

fastvideo.eval.metrics.audio.audiobox_aesthetics.metric.AudioBoxAestheticsMetric

AudioBoxAestheticsMetric()

AudioBox Aesthetics: PQ as the primary score, CE/CU/PC/PQ in details.

Source
@register("audio.audiobox_aesthetics")
class AudioBoxAestheticsMetric(BaseMetric):
    """AudioBox Aesthetics: PQ as the primary score, CE/CU/PC/PQ in details."""

name = "audio.audiobox_aesthetics"
requires_reference = False
higher_is_better = True
needs_gpu = True
is_set_metric = False
dependencies = ["audiobox_aesthetics"]

def __init__(self) -> None:
    super().__init__()
    self._predictor: Any = None

def to(self, device):
    super().to(device)
    if self._predictor is not None:
        self._predictor.model.to(self.device)
        self._predictor.device = self.device
    return self

def setup(self) -> None:
    if self._predictor is not None:
        return
    from audiobox_aesthetics.infer import initialize_predictor
    predictor = initialize_predictor()
    # Upstream's setup_model() always lands on default CUDA (cuda:0).
    # Re-pin onto this worker's device so multi-GPU eval actually parallelizes.
    predictor.model.to(self.device)
    predictor.device = self.device
    self._predictor = predictor

@torch.no_grad()
def compute(self, sample: dict) -> MetricResult:
    from pathlib import Path

    if self._predictor is None:
        self.setup()

    audio_path = sample.get("audio")
    if audio_path is None:
        return self._skip(sample, "missing 'audio'")
    if not Path(audio_path).exists():
        return self._skip(sample, f"audio file not found: {audio_path}")

    try:
        score = self._predictor.forward([{"path": audio_path}])[0]
    except Exception as exc:  # pragma: no cover — upstream raises vary
        return self._skip(sample, f"audiobox predictor failed: {type(exc).__name__}: {exc}")
    try:
        return MetricResult(
            name=self.name,
            score=float(score["PQ"]),
            details={
                "CE": float(score["CE"]),
                "CU": float(score["CU"]),
                "PC": float(score["PC"]),
                "PQ": float(score["PQ"]),
            },
        )
    except (KeyError, TypeError) as exc:
        return self._skip(sample, f&quot;audiobox returned unexpected shape: &#123;exc&#125;&quot;)</code></pre></details>


fastvideo.eval.metrics.audio.audiobox_aesthetics.metric.AudioBoxAestheticsMetric.to


to(device)


Source
def to(self, device):
    super().to(device)
    if self._predictor is not None:
        self._predictor.model.to(self.device)
        self._predictor.device = self.device
    return self


fastvideo.eval.metrics.audio.audiobox_aesthetics.metric.AudioBoxAestheticsMetric.setup


setup() -> None


Source
def setup(self) -> None:
    if self._predictor is not None:
        return
    from audiobox_aesthetics.infer import initialize_predictor
    predictor = initialize_predictor()
    # Upstream's setup_model() always lands on default CUDA (cuda:0).
    # Re-pin onto this worker's device so multi-GPU eval actually parallelizes.
    predictor.model.to(self.device)
    predictor.device = self.device
    self._predictor = predictor


fastvideo.eval.metrics.audio.audiobox_aesthetics.metric.AudioBoxAestheticsMetric.compute


compute(sample: dict) -> MetricResult


Source
@torch.no_grad()
def compute(self, sample: dict) -> MetricResult:
    from pathlib import Path

if self._predictor is None:
    self.setup()

audio_path = sample.get(&quot;audio&quot;)
if audio_path is None:
    return self._skip(sample, &quot;missing &#x27;audio&#x27;&quot;)
if not Path(audio_path).exists():
    return self._skip(sample, f&quot;audio file not found: &#123;audio_path&#125;&quot;)

try:
    score = self._predictor.forward(&#91;&#123;&quot;path&quot;: audio_path&#125;&#93;)&#91;0&#93;
except Exception as exc:  # pragma: no cover — upstream raises vary
    return self._skip(sample, f&quot;audiobox predictor failed: &#123;type(exc).__name__&#125;: &#123;exc&#125;&quot;)
try:
    return MetricResult(
        name=self.name,
        score=float(score&#91;&quot;PQ&quot;&#93;),
        details=&#123;
            &quot;CE&quot;: float(score&#91;&quot;CE&quot;&#93;),
            &quot;CU&quot;: float(score&#91;&quot;CU&quot;&#93;),
            &quot;PC&quot;: float(score&#91;&quot;PC&quot;&#93;),
            &quot;PQ&quot;: float(score&#91;&quot;PQ&quot;&#93;),
        &#125;,
    )
except (KeyError, TypeError) as exc:
    return self._skip(sample, f&quot;audiobox returned unexpected shape: &#123;exc&#125;&quot;)</code></pre></details>

Was this page helpful?