Skip to content
FastVideo
Esc
↑↓navigate↵open⌘Jpreview
On this page

metric

VBench Overall Consistency — ViCLIP text-video alignment.

Encodes 8 sampled video frames via ViCLIP vision encoder and a text prompt via ViCLIP text encoder, then computes cosine similarity.

fastvideo.eval.metrics.vbench.overall_consistency.metric.BaseMetric

BaseMetric()

Abstract base class for all eval metrics.

Two execution shapes:

  • Per-sample (is_set_metric=False, default) — implement :meth:compute. The Evaluator calls it once per input sample and returns one :class:MetricResult per sample.

  • Set-vs-set (is_set_metric=True) — implement :meth:accumulate (called once per sample to buffer features) and :meth:finalize (called once after all samples to compute the corpus-level result). Use :meth:reset to clear buffers and :meth:merge_from to fold multi-GPU per-worker state together.

Optionally override :meth:setup to eagerly load models. Metrics that chunk along the time dim for memory hardcode their own chunk size in __init__ (see optical_flow for the canonical example). Eval always processes one video per :meth:Evaluator.evaluate call; compute / accumulate receive a single sample, not a batch.

fastvideo.eval.metrics.vbench.overall_consistency.metric.BaseMetric.to

to(device: str | torch.device) -> BaseMetric

Move metric (and its internal models) to device.

fastvideo.eval.metrics.vbench.overall_consistency.metric.BaseMetric.setup

setup() -> None

Eagerly load models. Called once by :class:EvalWorker.

Default is a no-op; metrics with no eager state (pixel math, closed-form ops) inherit this. Override only if your metric needs to load weights.

fastvideo.eval.metrics.vbench.overall_consistency.metric.BaseMetric.compute

compute(sample: dict) -> MetricResult

Per-sample metrics: compute the score for one sample.

sample["video"] is (T, C, H, W) float in [0, 1]. sample["reference"] (if used) has the same shape. Return self._skip(sample, reason) for missing inputs.

fastvideo.eval.metrics.vbench.overall_consistency.metric.BaseMetric.reset

reset() -> None

Clear accumulator state at the start of each evaluate() call.

fastvideo.eval.metrics.vbench.overall_consistency.metric.BaseMetric.accumulate

accumulate(sample: dict) -> None

Buffer per-sample features for a corpus-level metric.

fastvideo.eval.metrics.vbench.overall_consistency.metric.BaseMetric.finalize

finalize() -> MetricResult

Compute the corpus-level result from buffered state.

fastvideo.eval.metrics.vbench.overall_consistency.metric.BaseMetric.merge_from

merge_from(other: BaseMetric) -> None

Multi-GPU: fold another worker's accumulator state into this one.

fastvideo.eval.metrics.vbench.overall_consistency.metric.register

register(name: str)

Decorator to register a metric class.

Usage::

@register("ssim")
class SSIMMetric(BaseMetric):
    ...

fastvideo.eval.metrics.vbench.overall_consistency.metric.MetricResult

MetricResult(name: str, score: float | None, details: dict[str, Any] = dict())

Standard result container returned by all metrics.

score is None when the metric was skipped (e.g. missing required input). Check details["skipped"] for the reason.

fastvideo.eval.metrics.vbench.overall_consistency.metric.extract_frames

extract_frames(video: torch.Tensor, n_frames: int | None = None) -> torch.Tensor

Uniformly sample n_frames from a (T, C, H, W) video tensor.

fastvideo.eval.metrics.vbench.overall_consistency.metric.OverallConsistencyMetric

OverallConsistencyMetric()
Source
@register("vbench.overall_consistency")
class OverallConsistencyMetric(BaseMetric):

name = "vbench.overall_consistency"
requires_reference = False
higher_is_better = True
needs_gpu = True
dependencies = ["timm", "einops", "clip"]
backbone = "viclip"

def __init__(self) -> None:
    super().__init__()
    self._model: Any = None
    self._tokenizer: Any = None

def to(self, device):
    super().to(device)
    if self._model is not None:
        self._model = self._model.to(self.device)
    return self

def setup(self) -> None:
    if self._model is not None:
        return
    from vbench.third_party.ViCLIP.viclip import ViCLIP
    from vbench.third_party.ViCLIP.simple_tokenizer import SimpleTokenizer

    # ViCLIP's tokenizer reuses OpenAI CLIP's BPE vocab. The file is
    # bundled with the ``openai-clip`` pip package (an ``[eval]`` extra)
    # — no separate download is needed; the model loader only handles
    # the actual .pth weights.
    from clip.simple_tokenizer import default_bpe
    self._tokenizer = SimpleTokenizer(default_bpe())

    from fastvideo.eval.models import ensure_checkpoint
    ckpt = ensure_checkpoint(
        "ViClip-InternVid-10M-FLT.pth",
        source="OpenGVLab/VBench_Used_Models",
        filename="ViClip-InternVid-10M-FLT.pth",
    )

    self._model = ViCLIP(tokenizer=self._tokenizer, pretrain=ckpt)
    self._model.to(self.device)
    self._model.eval()

@torch.no_grad()
def compute(self, sample: dict) -> MetricResult:
    video = sample["video"]  # (T, C, H, W)
    text_prompt = sample.get("text_prompt")
    if text_prompt is None:
        return self._skip(sample, "missing text_prompt")

    frames = _clip_transform(extract_frames(video, 8))  # (8, C, H, W)
    clip_in = frames.unsqueeze(0).to(self.device)  # (1, 8, C, H, W)

    vid_feat = self._model.encode_vision(clip_in, test=True).float()
    vid_feat = F.normalize(vid_feat, dim=-1, p=2)  # (1, D)

    text_feat = self._model.encode_text(text_prompt).float()
    text_feat = F.normalize(text_feat, dim=-1, p=2)
    score = float((vid_feat @ text_feat.T)[0][0].cpu())
    return MetricResult(name=self.name, score=score, details=&#123;&#125;)</code></pre></details>


fastvideo.eval.metrics.vbench.overall_consistency.metric.OverallConsistencyMetric.to


to(device)


Source
def to(self, device):
    super().to(device)
    if self._model is not None:
        self._model = self._model.to(self.device)
    return self


fastvideo.eval.metrics.vbench.overall_consistency.metric.OverallConsistencyMetric.setup


setup() -> None


Source
def setup(self) -> None:
    if self._model is not None:
        return
    from vbench.third_party.ViCLIP.viclip import ViCLIP
    from vbench.third_party.ViCLIP.simple_tokenizer import SimpleTokenizer

# ViCLIP&#x27;s tokenizer reuses OpenAI CLIP&#x27;s BPE vocab. The file is
# bundled with the ``openai-clip`` pip package (an ``&#91;eval&#93;`` extra)
# — no separate download is needed; the model loader only handles
# the actual .pth weights.
from clip.simple_tokenizer import default_bpe
self._tokenizer = SimpleTokenizer(default_bpe())

from fastvideo.eval.models import ensure_checkpoint
ckpt = ensure_checkpoint(
    &quot;ViClip-InternVid-10M-FLT.pth&quot;,
    source=&quot;OpenGVLab/VBench_Used_Models&quot;,
    filename=&quot;ViClip-InternVid-10M-FLT.pth&quot;,
)

self._model = ViCLIP(tokenizer=self._tokenizer, pretrain=ckpt)
self._model.to(self.device)
self._model.eval()</code></pre></details>


fastvideo.eval.metrics.vbench.overall_consistency.metric.OverallConsistencyMetric.compute


compute(sample: dict) -> MetricResult


Source
@torch.no_grad()
def compute(self, sample: dict) -> MetricResult:
    video = sample["video"]  # (T, C, H, W)
    text_prompt = sample.get("text_prompt")
    if text_prompt is None:
        return self._skip(sample, "missing text_prompt")

frames = _clip_transform(extract_frames(video, 8))  # (8, C, H, W)
clip_in = frames.unsqueeze(0).to(self.device)  # (1, 8, C, H, W)

vid_feat = self._model.encode_vision(clip_in, test=True).float()
vid_feat = F.normalize(vid_feat, dim=-1, p=2)  # (1, D)

text_feat = self._model.encode_text(text_prompt).float()
text_feat = F.normalize(text_feat, dim=-1, p=2)
score = float((vid_feat @ text_feat.T)&#91;0&#93;&#91;0&#93;.cpu())
return MetricResult(name=self.name, score=score, details=&#123;&#125;)</code></pre></details>

Was this page helpful?