CLAP Score (CS) — text-audio cosine similarity.

Uses HuggingFace transformers.ClapModel with the laion/clap-htsat-fused checkpoint, the closest HF mirror of the 630k-audioset-fusion-best.pt weights that hkchengrex/av-benchmark and the V2A literature use. The fully byte-exact comparison still requires the laion_clap pip package (which pins numpy<2 and forces a fastvideo-wide downgrade — so we don't depend on it). Numbers from this metric are in the same ballpark as av-benchmark's LAION_CLAP.

`fastvideo.eval.metrics.audio.clap_score.metric.BaseMetric`

BaseMetric()

Abstract base class for all eval metrics.

Two execution shapes:

Per-sample (is_set_metric=False, default) — implement :meth:compute. The Evaluator calls it once per input sample and returns one :class:MetricResult per sample.
Set-vs-set (is_set_metric=True) — implement :meth:accumulate (called once per sample to buffer features) and :meth:finalize (called once after all samples to compute the corpus-level result). Use :meth:reset to clear buffers and :meth:merge_from to fold multi-GPU per-worker state together.

Optionally override :meth:setup to eagerly load models. Metrics that chunk along the time dim for memory hardcode their own chunk size in __init__ (see optical_flow for the canonical example). Eval always processes one video per :meth:Evaluator.evaluate call; compute / accumulate receive a single sample, not a batch.

`fastvideo.eval.metrics.audio.clap_score.metric.BaseMetric.to`

to(device: str | torch.device) -> BaseMetric

Move metric (and its internal models) to device.

`fastvideo.eval.metrics.audio.clap_score.metric.BaseMetric.setup`

setup() -> None

Eagerly load models. Called once by :class:EvalWorker.

Default is a no-op; metrics with no eager state (pixel math, closed-form ops) inherit this. Override only if your metric needs to load weights.

`fastvideo.eval.metrics.audio.clap_score.metric.BaseMetric.compute`

compute(sample: dict) -> MetricResult

Per-sample metrics: compute the score for one sample.

sample["video"] is (T, C, H, W) float in [0, 1]. sample["reference"] (if used) has the same shape. Return self._skip(sample, reason) for missing inputs.

`fastvideo.eval.metrics.audio.clap_score.metric.BaseMetric.reset`

reset() -> None

Clear accumulator state at the start of each evaluate() call.

`fastvideo.eval.metrics.audio.clap_score.metric.BaseMetric.accumulate`

accumulate(sample: dict) -> None

Buffer per-sample features for a corpus-level metric.

`fastvideo.eval.metrics.audio.clap_score.metric.BaseMetric.finalize`

finalize() -> MetricResult

Compute the corpus-level result from buffered state.

`fastvideo.eval.metrics.audio.clap_score.metric.BaseMetric.merge_from`

merge_from(other: BaseMetric) -> None

Multi-GPU: fold another worker's accumulator state into this one.

`fastvideo.eval.metrics.audio.clap_score.metric.register`

register(name: str)

Decorator to register a metric class.

Usage::

@register("ssim")
class SSIMMetric(BaseMetric):
    ...

`fastvideo.eval.metrics.audio.clap_score.metric.MetricResult`

MetricResult(name: str, score: float | None, details: dict[str, Any] = dict())

Standard result container returned by all metrics.

score is None when the metric was skipped (e.g. missing required input). Check details["skipped"] for the reason.

`fastvideo.eval.metrics.audio.clap_score.metric.ClapScoreMetric`

ClapScoreMetric(model_name: str = DEFAULT_CLAP_REPO)

CLAP score — text-audio cosine similarity via HF CLAP.

Source

@register("audio.clap_score")
class ClapScoreMetric(BaseMetric):
    """CLAP score — text-audio cosine similarity via HF CLAP."""

    name = "audio.clap_score"
    requires_reference = False
    higher_is_better = True
    needs_gpu = True
    is_set_metric = False
    dependencies = ["librosa", "pyloudnorm"]

    def __init__(self, model_name: str = DEFAULT_CLAP_REPO) -> None:
        super().__init__()
        self._model_name = model_name
        self._model: Any = None
        self._processor: Any = None

    def to(self, device):
        super().to(device)
        if self._model is not None:
            self._model = self._model.to(self.device)
        return self

    def setup(self) -> None:
        if self._model is not None:
            return
        from transformers import ClapModel, ClapProcessor
        self._processor = ClapProcessor.from_pretrained(self._model_name)
        self._model = ClapModel.from_pretrained(self._model_name).to(self.device)
        self._model.eval()

    def _load_audio(self, audio_path: str):
        import librosa
        import pyloudnorm as pyln
        audio, _ = librosa.load(audio_path, sr=CLAP_SAMPLE_RATE, mono=True)
        return pyln.normalize.peak(audio, -1.0)

    @staticmethod
    def _projected_features(features) -> torch.Tensor:
        # transformers >= 5 returns a ModelOutput whose pooler_output holds the
        # (L2-normalized) projected CLAP features; 4.x returned the projected
        # tensor directly. Cosine similarity is scale-invariant, so both yield
        # the same score.
        return features.pooler_output if hasattr(features, "pooler_output") else features

    def _audio_emb(self, audio_path: str) -> torch.Tensor:
        audio = self._load_audio(audio_path)
        inputs = self._processor(audio=audio, sampling_rate=CLAP_SAMPLE_RATE, return_tensors="pt")
        inputs = {k: v.to(self.device) for k, v in inputs.items()}
        return self._projected_features(self._model.get_audio_features(**inputs))

    def _text_emb(self, text: str) -> torch.Tensor:
        inputs = self._processor(text=[text], return_tensors="pt", padding=True)
        inputs = {k: v.to(self.device) for k, v in inputs.items()}
        return self._projected_features(self._model.get_text_features(**inputs))

    @torch.no_grad()
    def compute(self, sample: dict) -> MetricResult:
        if self._model is None:
            self.setup()

        audio = sample.get("audio")
        text = sample.get("text_prompt")
        if audio is None or text is None:
            missing = [k for k, v in (("audio", audio), ("text_prompt", text)) if v is None]
            return self._skip(sample, f"missing {', '.join(repr(k) for k in missing)}")

        audio_emb = self._audio_emb(audio)
        text_emb = self._text_emb(text)
        score = F.cosine_similarity(audio_emb, text_emb, dim=1, eps=1e-8)[0].cpu().item()
        return MetricResult(name=self.name, score=score, details={})

`fastvideo.eval.metrics.audio.clap_score.metric.ClapScoreMetric.to`

to(device)

Source

def to(self, device):
    super().to(device)
    if self._model is not None:
        self._model = self._model.to(self.device)
    return self

`fastvideo.eval.metrics.audio.clap_score.metric.ClapScoreMetric.setup`

setup() -> None

Source

def setup(self) -> None:
    if self._model is not None:
        return
    from transformers import ClapModel, ClapProcessor
    self._processor = ClapProcessor.from_pretrained(self._model_name)
    self._model = ClapModel.from_pretrained(self._model_name).to(self.device)
    self._model.eval()

`fastvideo.eval.metrics.audio.clap_score.metric.ClapScoreMetric.compute`

compute(sample: dict) -> MetricResult

Source

@torch.no_grad()
def compute(self, sample: dict) -> MetricResult:
    if self._model is None:
        self.setup()

    audio = sample.get("audio")
    text = sample.get("text_prompt")
    if audio is None or text is None:
        missing = [k for k, v in (("audio", audio), ("text_prompt", text)) if v is None]
        return self._skip(sample, f"missing {', '.join(repr(k) for k in missing)}")

    audio_emb = self._audio_emb(audio)
    text_emb = self._text_emb(text)
    score = F.cosine_similarity(audio_emb, text_emb, dim=1, eps=1e-8)[0].cpu().item()
    return MetricResult(name=self.name, score=score, details={})