metric
CLAP Score (CS) — text-audio cosine similarity.
Uses HuggingFace transformers.ClapModel with the
laion/clap-htsat-fused checkpoint, the closest HF mirror of the
630k-audioset-fusion-best.pt weights that
hkchengrex/av-benchmark and the V2A literature use. The fully
byte-exact comparison still requires the laion_clap pip package
(which pins numpy<2 and forces a fastvideo-wide downgrade — so
we don't depend on it). Numbers from this metric are in the same
ballpark as av-benchmark's LAION_CLAP.
fastvideo.eval.metrics.audio.clap_score.metric.BaseMetric
BaseMetric()Abstract base class for all eval metrics.
Two execution shapes:
-
Per-sample (
is_set_metric=False, default) — implement :meth:compute. The Evaluator calls it once per input sample and returns one :class:MetricResultper sample. -
Set-vs-set (
is_set_metric=True) — implement :meth:accumulate(called once per sample to buffer features) and :meth:finalize(called once after all samples to compute the corpus-level result). Use :meth:resetto clear buffers and :meth:merge_fromto fold multi-GPU per-worker state together.
Optionally override :meth:setup to eagerly load models. Metrics
that chunk along the time dim for memory hardcode their own chunk
size in __init__ (see optical_flow for the canonical
example). Eval always processes one video per
:meth:Evaluator.evaluate call; compute / accumulate
receive a single sample, not a batch.
fastvideo.eval.metrics.audio.clap_score.metric.BaseMetric.to
to(device: str | torch.device) -> BaseMetricMove metric (and its internal models) to device.
fastvideo.eval.metrics.audio.clap_score.metric.BaseMetric.setup
setup() -> NoneEagerly load models. Called once by :class:EvalWorker.
Default is a no-op; metrics with no eager state (pixel math, closed-form ops) inherit this. Override only if your metric needs to load weights.
fastvideo.eval.metrics.audio.clap_score.metric.BaseMetric.compute
compute(sample: dict) -> MetricResultPer-sample metrics: compute the score for one sample.
sample["video"] is (T, C, H, W) float in [0, 1].
sample["reference"] (if used) has the same shape. Return
self._skip(sample, reason) for missing inputs.
fastvideo.eval.metrics.audio.clap_score.metric.BaseMetric.reset
reset() -> NoneClear accumulator state at the start of each evaluate() call.
fastvideo.eval.metrics.audio.clap_score.metric.BaseMetric.accumulate
accumulate(sample: dict) -> NoneBuffer per-sample features for a corpus-level metric.
fastvideo.eval.metrics.audio.clap_score.metric.BaseMetric.finalize
finalize() -> MetricResultCompute the corpus-level result from buffered state.
fastvideo.eval.metrics.audio.clap_score.metric.BaseMetric.merge_from
merge_from(other: BaseMetric) -> NoneMulti-GPU: fold another worker's accumulator state into this one.
fastvideo.eval.metrics.audio.clap_score.metric.register
register(name: str)Decorator to register a metric class.
Usage::
@register("ssim")
class SSIMMetric(BaseMetric):
...
fastvideo.eval.metrics.audio.clap_score.metric.MetricResult
MetricResult(name: str, score: float | None, details: dict[str, Any] = dict())Standard result container returned by all metrics.
score is None when the metric was skipped (e.g. missing
required input). Check details["skipped"] for the reason.
fastvideo.eval.metrics.audio.clap_score.metric.ClapScoreMetric
ClapScoreMetric(model_name: str = DEFAULT_CLAP_REPO)CLAP score — text-audio cosine similarity via HF CLAP.
Source
@register("audio.clap_score")
class ClapScoreMetric(BaseMetric):
"""CLAP score — text-audio cosine similarity via HF CLAP."""
name = "audio.clap_score"
requires_reference = False
higher_is_better = True
needs_gpu = True
is_set_metric = False
dependencies = ["librosa", "pyloudnorm"]
def __init__(self, model_name: str = DEFAULT_CLAP_REPO) -> None:
super().__init__()
self._model_name = model_name
self._model: Any = None
self._processor: Any = None
def to(self, device):
super().to(device)
if self._model is not None:
self._model = self._model.to(self.device)
return self
def setup(self) -> None:
if self._model is not None:
return
from transformers import ClapModel, ClapProcessor
self._processor = ClapProcessor.from_pretrained(self._model_name)
self._model = ClapModel.from_pretrained(self._model_name).to(self.device)
self._model.eval()
def _load_audio(self, audio_path: str):
import librosa
import pyloudnorm as pyln
audio, _ = librosa.load(audio_path, sr=CLAP_SAMPLE_RATE, mono=True)
return pyln.normalize.peak(audio, -1.0)
@staticmethod
def _projected_features(features) -> torch.Tensor:
# transformers >= 5 returns a ModelOutput whose pooler_output holds the
# (L2-normalized) projected CLAP features; 4.x returned the projected
# tensor directly. Cosine similarity is scale-invariant, so both yield
# the same score.
return features.pooler_output if hasattr(features, "pooler_output") else features
def _audio_emb(self, audio_path: str) -> torch.Tensor:
audio = self._load_audio(audio_path)
inputs = self._processor(audio=audio, sampling_rate=CLAP_SAMPLE_RATE, return_tensors="pt")
inputs = {k: v.to(self.device) for k, v in inputs.items()}
return self._projected_features(self._model.get_audio_features(**inputs))
def _text_emb(self, text: str) -> torch.Tensor:
inputs = self._processor(text=[text], return_tensors="pt", padding=True)
inputs = {k: v.to(self.device) for k, v in inputs.items()}
return self._projected_features(self._model.get_text_features(**inputs))
@torch.no_grad()
def compute(self, sample: dict) -> MetricResult:
if self._model is None:
self.setup()
audio = sample.get("audio")
text = sample.get("text_prompt")
if audio is None or text is None:
missing = [k for k, v in (("audio", audio), ("text_prompt", text)) if v is None]
return self._skip(sample, f"missing {', '.join(repr(k) for k in missing)}")
audio_emb = self._audio_emb(audio)
text_emb = self._text_emb(text)
score = F.cosine_similarity(audio_emb, text_emb, dim=1, eps=1e-8)[0].cpu().item()
return MetricResult(name=self.name, score=score, details={})</code></pre></details>
fastvideo.eval.metrics.audio.clap_score.metric.ClapScoreMetric.to
to(device)
Source
def to(self, device):
super().to(device)
if self._model is not None:
self._model = self._model.to(self.device)
return self
fastvideo.eval.metrics.audio.clap_score.metric.ClapScoreMetric.setup
setup() -> None
Source
def setup(self) -> None:
if self._model is not None:
return
from transformers import ClapModel, ClapProcessor
self._processor = ClapProcessor.from_pretrained(self._model_name)
self._model = ClapModel.from_pretrained(self._model_name).to(self.device)
self._model.eval()
fastvideo.eval.metrics.audio.clap_score.metric.ClapScoreMetric.compute
compute(sample: dict) -> MetricResult
Source
@torch.no_grad()
def compute(self, sample: dict) -> MetricResult:
if self._model is None:
self.setup()
audio = sample.get("audio")
text = sample.get("text_prompt")
if audio is None or text is None:
missing = [k for k, v in (("audio", audio), ("text_prompt", text)) if v is None]
return self._skip(sample, f"missing {', '.join(repr(k) for k in missing)}")
audio_emb = self._audio_emb(audio)
text_emb = self._text_emb(text)
score = F.cosine_similarity(audio_emb, text_emb, dim=1, eps=1e-8)[0].cpu().item()
return MetricResult(name=self.name, score=score, details={})</code></pre></details>