metric
VBench Multiple Objects — GRiT detection for dual-object presence.
Checks if BOTH target objects are detected in each of 16 sampled frames. Score = matching_frames / total_frames.
fastvideo.eval.metrics.vbench.multiple_objects.metric.BaseMetric
BaseMetric()Abstract base class for all eval metrics.
Two execution shapes:
-
Per-sample (
is_set_metric=False, default) — implement :meth:compute. The Evaluator calls it once per input sample and returns one :class:MetricResultper sample. -
Set-vs-set (
is_set_metric=True) — implement :meth:accumulate(called once per sample to buffer features) and :meth:finalize(called once after all samples to compute the corpus-level result). Use :meth:resetto clear buffers and :meth:merge_fromto fold multi-GPU per-worker state together.
Optionally override :meth:setup to eagerly load models. Metrics
that chunk along the time dim for memory hardcode their own chunk
size in __init__ (see optical_flow for the canonical
example). Eval always processes one video per
:meth:Evaluator.evaluate call; compute / accumulate
receive a single sample, not a batch.
fastvideo.eval.metrics.vbench.multiple_objects.metric.BaseMetric.to
to(device: str | torch.device) -> BaseMetricMove metric (and its internal models) to device.
fastvideo.eval.metrics.vbench.multiple_objects.metric.BaseMetric.setup
setup() -> NoneEagerly load models. Called once by :class:EvalWorker.
Default is a no-op; metrics with no eager state (pixel math, closed-form ops) inherit this. Override only if your metric needs to load weights.
fastvideo.eval.metrics.vbench.multiple_objects.metric.BaseMetric.compute
compute(sample: dict) -> MetricResultPer-sample metrics: compute the score for one sample.
sample["video"] is (T, C, H, W) float in [0, 1].
sample["reference"] (if used) has the same shape. Return
self._skip(sample, reason) for missing inputs.
fastvideo.eval.metrics.vbench.multiple_objects.metric.BaseMetric.reset
reset() -> NoneClear accumulator state at the start of each evaluate() call.
fastvideo.eval.metrics.vbench.multiple_objects.metric.BaseMetric.accumulate
accumulate(sample: dict) -> NoneBuffer per-sample features for a corpus-level metric.
fastvideo.eval.metrics.vbench.multiple_objects.metric.BaseMetric.finalize
finalize() -> MetricResultCompute the corpus-level result from buffered state.
fastvideo.eval.metrics.vbench.multiple_objects.metric.BaseMetric.merge_from
merge_from(other: BaseMetric) -> NoneMulti-GPU: fold another worker's accumulator state into this one.
fastvideo.eval.metrics.vbench.multiple_objects.metric.register
register(name: str)Decorator to register a metric class.
Usage::
@register("ssim")
class SSIMMetric(BaseMetric):
...
fastvideo.eval.metrics.vbench.multiple_objects.metric.MetricResult
MetricResult(name: str, score: float | None, details: dict[str, Any] = dict())Standard result container returned by all metrics.
score is None when the metric was skipped (e.g. missing
required input). Check details["skipped"] for the reason.
fastvideo.eval.metrics.vbench.multiple_objects.metric.MultipleObjectsMetric
MultipleObjectsMetric()Source
@register("vbench.multiple_objects")
class MultipleObjectsMetric(BaseMetric):
name = "vbench.multiple_objects"
requires_reference = False
higher_is_better = True
needs_gpu = True
dependencies = ["detectron2"]
def __init__(self) -> None:
super().__init__()
self._model = None
def setup(self) -> None:
if self._model is not None:
return
from fastvideo.eval.metrics.vbench._grit_helper import load_grit_model
# VBench's multiple_objects uses ObjectDet head
self._model = load_grit_model(self.device, task="ObjectDet")
@torch.no_grad()
def compute(self, sample: dict) -> MetricResult:
from fastvideo.eval.metrics.vbench._grit_helper import prepare_frames, detect_frames
video = sample["video"] # (T, C, H, W)
aux = sample.get("auxiliary_info") or {}
if "object" not in aux:
return self._skip(sample, "missing 'object' in auxiliary_info")
object_info = aux["object"]
if " and " not in object_info:
# multiple_objects expects "<a> and <b>"; single objects are
# the object_class metric's territory — skip this row.
return self._skip(sample, "'object' lacks ' and ' separator")
key_a, key_b = [k.strip() for k in object_info.split(" and ")]
frames_np = prepare_frames(video)
preds = detect_frames(self._model, frames_np)
matching = 0
for frame_pred in preds:
try:
obj_set = set(frame_pred[0][2]) if frame_pred else set()
except (IndexError, TypeError):
obj_set = set()
if key_a in obj_set and key_b in obj_set:
matching += 1
total = len(preds)
score = matching / total if total > 0 else 0.0
return MetricResult(
name=self.name,
score=float(score),
details={
"matching_frames": matching,
"total_frames": total
},
)</code></pre></details>
fastvideo.eval.metrics.vbench.multiple_objects.metric.MultipleObjectsMetric.setup
setup() -> None
Source
def setup(self) -> None:
if self._model is not None:
return
from fastvideo.eval.metrics.vbench._grit_helper import load_grit_model
# VBench's multiple_objects uses ObjectDet head
self._model = load_grit_model(self.device, task="ObjectDet")
fastvideo.eval.metrics.vbench.multiple_objects.metric.MultipleObjectsMetric.compute
compute(sample: dict) -> MetricResult
Source
@torch.no_grad()
def compute(self, sample: dict) -> MetricResult:
from fastvideo.eval.metrics.vbench._grit_helper import prepare_frames, detect_frames
video = sample["video"] # (T, C, H, W)
aux = sample.get("auxiliary_info") or {}
if "object" not in aux:
return self._skip(sample, "missing 'object' in auxiliary_info")
object_info = aux["object"]
if " and " not in object_info:
# multiple_objects expects "<a> and <b>"; single objects are
# the object_class metric's territory — skip this row.
return self._skip(sample, "'object' lacks ' and ' separator")
key_a, key_b = [k.strip() for k in object_info.split(" and ")]
frames_np = prepare_frames(video)
preds = detect_frames(self._model, frames_np)
matching = 0
for frame_pred in preds:
try:
obj_set = set(frame_pred[0][2]) if frame_pred else set()
except (IndexError, TypeError):
obj_set = set()
if key_a in obj_set and key_b in obj_set:
matching += 1
total = len(preds)
score = matching / total if total > 0 else 0.0
return MetricResult(
name=self.name,
score=float(score),
details={
"matching_frames": matching,
"total_frames": total
},
)</code></pre></details>