VBench Multiple Objects — GRiT detection for dual-object presence.

Checks if BOTH target objects are detected in each of 16 sampled frames. Score = matching_frames / total_frames.

`fastvideo.eval.metrics.vbench.multiple_objects.metric.BaseMetric`

BaseMetric()

Abstract base class for all eval metrics.

Two execution shapes:

Per-sample (is_set_metric=False, default) — implement :meth:compute. The Evaluator calls it once per input sample and returns one :class:MetricResult per sample.
Set-vs-set (is_set_metric=True) — implement :meth:accumulate (called once per sample to buffer features) and :meth:finalize (called once after all samples to compute the corpus-level result). Use :meth:reset to clear buffers and :meth:merge_from to fold multi-GPU per-worker state together.

Optionally override :meth:setup to eagerly load models. Metrics that chunk along the time dim for memory hardcode their own chunk size in __init__ (see optical_flow for the canonical example). Eval always processes one video per :meth:Evaluator.evaluate call; compute / accumulate receive a single sample, not a batch.

`fastvideo.eval.metrics.vbench.multiple_objects.metric.BaseMetric.to`

to(device: str | torch.device) -> BaseMetric

Move metric (and its internal models) to device.

`fastvideo.eval.metrics.vbench.multiple_objects.metric.BaseMetric.setup`

setup() -> None

Eagerly load models. Called once by :class:EvalWorker.

Default is a no-op; metrics with no eager state (pixel math, closed-form ops) inherit this. Override only if your metric needs to load weights.

`fastvideo.eval.metrics.vbench.multiple_objects.metric.BaseMetric.compute`

compute(sample: dict) -> MetricResult

Per-sample metrics: compute the score for one sample.

sample["video"] is (T, C, H, W) float in [0, 1]. sample["reference"] (if used) has the same shape. Return self._skip(sample, reason) for missing inputs.

`fastvideo.eval.metrics.vbench.multiple_objects.metric.BaseMetric.reset`

reset() -> None

Clear accumulator state at the start of each evaluate() call.

`fastvideo.eval.metrics.vbench.multiple_objects.metric.BaseMetric.accumulate`

accumulate(sample: dict) -> None

Buffer per-sample features for a corpus-level metric.

`fastvideo.eval.metrics.vbench.multiple_objects.metric.BaseMetric.finalize`

finalize() -> MetricResult

Compute the corpus-level result from buffered state.

`fastvideo.eval.metrics.vbench.multiple_objects.metric.BaseMetric.merge_from`

merge_from(other: BaseMetric) -> None

Multi-GPU: fold another worker's accumulator state into this one.

`fastvideo.eval.metrics.vbench.multiple_objects.metric.register`

register(name: str)

Decorator to register a metric class.

Usage::

@register("ssim")
class SSIMMetric(BaseMetric):
    ...

`fastvideo.eval.metrics.vbench.multiple_objects.metric.MetricResult`

MetricResult(name: str, score: float | None, details: dict[str, Any] = dict())

Standard result container returned by all metrics.

score is None when the metric was skipped (e.g. missing required input). Check details["skipped"] for the reason.

`fastvideo.eval.metrics.vbench.multiple_objects.metric.MultipleObjectsMetric`

MultipleObjectsMetric()

Source

@register("vbench.multiple_objects")
class MultipleObjectsMetric(BaseMetric):

    name = "vbench.multiple_objects"
    requires_reference = False
    higher_is_better = True
    needs_gpu = True
    dependencies = ["detectron2"]

    def __init__(self) -> None:
        super().__init__()
        self._model = None

    def setup(self) -> None:
        if self._model is not None:
            return
        from fastvideo.eval.metrics.vbench._grit_helper import load_grit_model
        # VBench's multiple_objects uses ObjectDet head
        self._model = load_grit_model(self.device, task="ObjectDet")

    @torch.no_grad()
    def compute(self, sample: dict) -> MetricResult:
        from fastvideo.eval.metrics.vbench._grit_helper import prepare_frames, detect_frames

        video = sample["video"]  # (T, C, H, W)
        aux = sample.get("auxiliary_info") or {}
        if "object" not in aux:
            return self._skip(sample, "missing 'object' in auxiliary_info")

        object_info = aux["object"]
        if " and " not in object_info:
            # multiple_objects expects "<a> and <b>"; single objects are
            # the object_class metric's territory — skip this row.
            return self._skip(sample, "'object' lacks ' and ' separator")
        key_a, key_b = [k.strip() for k in object_info.split(" and ")]

        frames_np = prepare_frames(video)
        preds = detect_frames(self._model, frames_np)

        matching = 0
        for frame_pred in preds:
            try:
                obj_set = set(frame_pred[0][2]) if frame_pred else set()
            except (IndexError, TypeError):
                obj_set = set()
            if key_a in obj_set and key_b in obj_set:
                matching += 1

        total = len(preds)
        score = matching / total if total > 0 else 0.0
        return MetricResult(
            name=self.name,
            score=float(score),
            details={
                "matching_frames": matching,
                "total_frames": total
            },
        )

`fastvideo.eval.metrics.vbench.multiple_objects.metric.MultipleObjectsMetric.setup`

setup() -> None

Source

def setup(self) -> None:
    if self._model is not None:
        return
    from fastvideo.eval.metrics.vbench._grit_helper import load_grit_model
    # VBench's multiple_objects uses ObjectDet head
    self._model = load_grit_model(self.device, task="ObjectDet")

`fastvideo.eval.metrics.vbench.multiple_objects.metric.MultipleObjectsMetric.compute`

compute(sample: dict) -> MetricResult

Source

@torch.no_grad()
def compute(self, sample: dict) -> MetricResult:
    from fastvideo.eval.metrics.vbench._grit_helper import prepare_frames, detect_frames

    video = sample["video"]  # (T, C, H, W)
    aux = sample.get("auxiliary_info") or {}
    if "object" not in aux:
        return self._skip(sample, "missing 'object' in auxiliary_info")

    object_info = aux["object"]
    if " and " not in object_info:
        # multiple_objects expects "<a> and <b>"; single objects are
        # the object_class metric's territory — skip this row.
        return self._skip(sample, "'object' lacks ' and ' separator")
    key_a, key_b = [k.strip() for k in object_info.split(" and ")]

    frames_np = prepare_frames(video)
    preds = detect_frames(self._model, frames_np)

    matching = 0
    for frame_pred in preds:
        try:
            obj_set = set(frame_pred[0][2]) if frame_pred else set()
        except (IndexError, TypeError):
            obj_set = set()
        if key_a in obj_set and key_b in obj_set:
            matching += 1

    total = len(preds)
    score = matching / total if total > 0 else 0.0
    return MetricResult(
        name=self.name,
        score=float(score),
        details={
            "matching_frames": matching,
            "total_frames": total
        },
    )