Skip to content
FastVideo
Esc
↑↓navigate↵open⌘Jpreview
On this page

metric

VBench Spatial Relationship — GRiT detection + bbox position scoring.

Detects two target objects via GRiT and checks if their bounding boxes satisfy the expected spatial relationship (left/right/above/below).

fastvideo.eval.metrics.vbench.spatial_relationship.metric.BaseMetric

BaseMetric()

Abstract base class for all eval metrics.

Two execution shapes:

  • Per-sample (is_set_metric=False, default) — implement :meth:compute. The Evaluator calls it once per input sample and returns one :class:MetricResult per sample.

  • Set-vs-set (is_set_metric=True) — implement :meth:accumulate (called once per sample to buffer features) and :meth:finalize (called once after all samples to compute the corpus-level result). Use :meth:reset to clear buffers and :meth:merge_from to fold multi-GPU per-worker state together.

Optionally override :meth:setup to eagerly load models. Metrics that chunk along the time dim for memory hardcode their own chunk size in __init__ (see optical_flow for the canonical example). Eval always processes one video per :meth:Evaluator.evaluate call; compute / accumulate receive a single sample, not a batch.

fastvideo.eval.metrics.vbench.spatial_relationship.metric.BaseMetric.to

to(device: str | torch.device) -> BaseMetric

Move metric (and its internal models) to device.

fastvideo.eval.metrics.vbench.spatial_relationship.metric.BaseMetric.setup

setup() -> None

Eagerly load models. Called once by :class:EvalWorker.

Default is a no-op; metrics with no eager state (pixel math, closed-form ops) inherit this. Override only if your metric needs to load weights.

fastvideo.eval.metrics.vbench.spatial_relationship.metric.BaseMetric.compute

compute(sample: dict) -> MetricResult

Per-sample metrics: compute the score for one sample.

sample["video"] is (T, C, H, W) float in [0, 1]. sample["reference"] (if used) has the same shape. Return self._skip(sample, reason) for missing inputs.

fastvideo.eval.metrics.vbench.spatial_relationship.metric.BaseMetric.reset

reset() -> None

Clear accumulator state at the start of each evaluate() call.

fastvideo.eval.metrics.vbench.spatial_relationship.metric.BaseMetric.accumulate

accumulate(sample: dict) -> None

Buffer per-sample features for a corpus-level metric.

fastvideo.eval.metrics.vbench.spatial_relationship.metric.BaseMetric.finalize

finalize() -> MetricResult

Compute the corpus-level result from buffered state.

fastvideo.eval.metrics.vbench.spatial_relationship.metric.BaseMetric.merge_from

merge_from(other: BaseMetric) -> None

Multi-GPU: fold another worker's accumulator state into this one.

fastvideo.eval.metrics.vbench.spatial_relationship.metric.register

register(name: str)

Decorator to register a metric class.

Usage::

@register("ssim")
class SSIMMetric(BaseMetric):
    ...

fastvideo.eval.metrics.vbench.spatial_relationship.metric.MetricResult

MetricResult(name: str, score: float | None, details: dict[str, Any] = dict())

Standard result container returned by all metrics.

score is None when the metric was skipped (e.g. missing required input). Check details["skipped"] for the reason.

fastvideo.eval.metrics.vbench.spatial_relationship.metric.SpatialRelationshipMetric

SpatialRelationshipMetric()
Source
@register("vbench.spatial_relationship")
class SpatialRelationshipMetric(BaseMetric):

name = "vbench.spatial_relationship"
requires_reference = False
higher_is_better = True
needs_gpu = True
dependencies = ["detectron2"]

def __init__(self) -> None:
    super().__init__()
    self._model: Any = None

def setup(self) -> None:
    if self._model is not None:
        return
    from fastvideo.eval.metrics.vbench._grit_helper import load_grit_model
    # VBench's spatial_relationship uses ObjectDet head and matches
    # pred[0] against class names like "person"/"grass"
    self._model = load_grit_model(self.device, task="ObjectDet")

@torch.no_grad()
def compute(self, sample: dict) -> MetricResult:
    from fastvideo.eval.metrics.vbench._grit_helper import prepare_frames

    video = sample["video"]  # (T, C, H, W)
    aux = sample.get("auxiliary_info") or {}
    if "spatial_relationship" not in aux:
        return self._skip(sample, "missing 'spatial_relationship' in auxiliary_info")

    sp_info = aux["spatial_relationship"]
    try:
        key_a = sp_info["object_a"]
        key_b = sp_info["object_b"]
        relation = sp_info["relationship"]
    except (KeyError, TypeError):
        return self._skip(sample, "spatial_relationship missing object_a/object_b/relationship")

    frames_np = prepare_frames(video)

    preds = []
    for frame in frames_np:
        ret = self._model.run_caption_tensor(frame)
        frame_dets = []
        if len(ret[0]) > 0:
            for info in ret[0]:
                frame_dets.append([info[0], info[1]])  # (caption, bbox)
        preds.append(frame_dets)

    # Score each frame (matching VBench's check_generate).
    frame_scores: list[float] = []
    for frame_pred in preds:
        obj_bboxes = [item[1] for item in frame_pred if item[0] == key_a or item[0] == key_b]

        cur_scores = [0.0]
        for i in range(len(obj_bboxes) - 1):
            for j in range(i + 1, len(obj_bboxes)):
                cur_scores.append(_get_position_score(
                    relation,
                    obj_bboxes[i],
                    obj_bboxes[j],
                ))
        frame_scores.append(max(cur_scores))

    score = float(np.mean(frame_scores)) if frame_scores else 0.0
    return MetricResult(
        name=self.name,
        score=score,
        details={"per_frame": frame_scores},
    )</code></pre></details>


fastvideo.eval.metrics.vbench.spatial_relationship.metric.SpatialRelationshipMetric.setup


setup() -> None


Source
def setup(self) -> None:
    if self._model is not None:
        return
    from fastvideo.eval.metrics.vbench._grit_helper import load_grit_model
    # VBench's spatial_relationship uses ObjectDet head and matches
    # pred[0] against class names like "person"/"grass"
    self._model = load_grit_model(self.device, task="ObjectDet")


fastvideo.eval.metrics.vbench.spatial_relationship.metric.SpatialRelationshipMetric.compute


compute(sample: dict) -> MetricResult


Source
@torch.no_grad()
def compute(self, sample: dict) -> MetricResult:
    from fastvideo.eval.metrics.vbench._grit_helper import prepare_frames

video = sample&#91;&quot;video&quot;&#93;  # (T, C, H, W)
aux = sample.get(&quot;auxiliary_info&quot;) or &#123;&#125;
if &quot;spatial_relationship&quot; not in aux:
    return self._skip(sample, &quot;missing &#x27;spatial_relationship&#x27; in auxiliary_info&quot;)

sp_info = aux&#91;&quot;spatial_relationship&quot;&#93;
try:
    key_a = sp_info&#91;&quot;object_a&quot;&#93;
    key_b = sp_info&#91;&quot;object_b&quot;&#93;
    relation = sp_info&#91;&quot;relationship&quot;&#93;
except (KeyError, TypeError):
    return self._skip(sample, &quot;spatial_relationship missing object_a/object_b/relationship&quot;)

frames_np = prepare_frames(video)

preds = &#91;&#93;
for frame in frames_np:
    ret = self._model.run_caption_tensor(frame)
    frame_dets = &#91;&#93;
    if len(ret&#91;0&#93;) &gt; 0:
        for info in ret&#91;0&#93;:
            frame_dets.append(&#91;info&#91;0&#93;, info&#91;1&#93;&#93;)  # (caption, bbox)
    preds.append(frame_dets)

# Score each frame (matching VBench&#x27;s check_generate).
frame_scores: list&#91;float&#93; = &#91;&#93;
for frame_pred in preds:
    obj_bboxes = &#91;item&#91;1&#93; for item in frame_pred if item&#91;0&#93; == key_a or item&#91;0&#93; == key_b&#93;

    cur_scores = &#91;0.0&#93;
    for i in range(len(obj_bboxes) - 1):
        for j in range(i + 1, len(obj_bboxes)):
            cur_scores.append(_get_position_score(
                relation,
                obj_bboxes&#91;i&#93;,
                obj_bboxes&#91;j&#93;,
            ))
    frame_scores.append(max(cur_scores))

score = float(np.mean(frame_scores)) if frame_scores else 0.0
return MetricResult(
    name=self.name,
    score=score,
    details=&#123;&quot;per_frame&quot;: frame_scores&#125;,
)</code></pre></details>

Was this page helpful?