PaperlayerView exact revision on arXiv

Source-linked AI summary

VANTAGE-Bench: Evaluating the Infrastructure AI Gap in Vision-Language Models

Zaid Pervaiz Bhat, Nimra Nayyar, Arihant Jain, Lap Fung Chan, John Suchanek, Yu Wang, Varun Praveen, Tomasz Kornuta, Vidya Nariyambut Murali

arXiv:2609.09396v1cs.CVcs.AI

TL;DR

VLM evaluation has largely centered on subject-centric consumer video, leaving Infrastructure AI in fixed-camera operational settings undermeasured. VANTAGE-Bench addresses this gap with an eight-task, multi-domain benchmark spanning four reasoning pillars and a single-pass tracking protocol. Across 17 zero-shot models, the deficit is concentrated in specific capabilities rather than uniform, while temporal tasks remain weakest and tracking separates from specialist systems over longer horizons.

  • Problem

    VLM benchmarks emphasize action-oriented Embodied AI and subject-centric consumer video, leaving fixed-camera Infrastructure AI insufficiently evaluated.

  • Method

    VANTAGE-Bench evaluates eight task formulations across four reasoning pillars and three operational domains using 3,346 expert-annotated media assets, including single-pass trajectory tracking.

  • Results

    The shortfall relative to consumer-centric benchmarks is concentrated: event verification, referring expressions, and temporal localization decline, while video question answering remains within 5.3 points of VideoMME and spatial pointing can exceed BLINK.

  • Takeaways & Limitations

    Infrastructure AI capability is task-specific rather than uniformly weak, with persistent absolute weakness in temporal tasks and widening tracking separation as horizons extend.

  • Takeaways & Limitations

    SOT uses synthetic warehouse sequences, so its conclusions remain limited by that distribution and lack real-data validation.

Abstract

from arXiv · show

As Vision-Language Models (VLMs) advance toward physical deployment, the focus has remained on action-oriented Embodied AI evaluated on subject-centric consumer video. This overlooks a pervasive class of Physical AI: Infrastructure AI, which relies on fixed cameras for open-loop insights like safety monitoring and operational logging. We introduce VANTAGE-Bench, a benchmark measuring this "Infrastructure AI Gap." It spans three operational domains (Logistics, Transportation, and Smart Spaces), unifies image and video evaluation across semantic, spatial, temporal, and spatio-temporal capabilities, and moves beyond multiple-choice to eight task formulations including dense captioning and spatio-temporal grounding. It adds a single-pass trajectory protocol for Single Object Tracking and, to our knowledge, the first such evaluation on fixed-camera infrastructure video, scored against specialist trackers. Annotation spans three regimes over 3,346 media assets: 3,342 video-task annotations, 4,281 image-grounding annotations, and 27,404 detection boxes. Evaluating 17 models zero-shot, we find the shortfall relative to consumer-centric benchmarks is concentrated, not general. Event verification, referring expressions, and temporal localization fall roughly 9 to 24 points at every model scale, while video question answering stays within 5.3 points of VideoMME and 2D spatial pointing shows no shortfall against BLINK. The temporal pillar is weakest in absolute terms: no system exceeds 55.7 mIoU on temporal localization or 37.3 SODA_c on dense video captioning. On tracking, frontier models come within roughly 5 points of specialist trackers over short horizons but separate as the horizon extends. Open-weight models lead 2D object localization outright, so neither scale nor proprietary access explains the pattern. Data, evaluation harness, and leaderboard: https://vantage-bench.org/

1. Introduction

VANTAGE-Bench addresses the Infrastructure AI Gap: VLMs trained and evaluated largely on subject-centric consumer video may struggle with dense, fixed-camera infrastructure scenes. It introduces a multi-task benchmark designed to measure this gap beyond multiple-choice evaluation.

  • Infrastructure AI uses fixed cameras for open-loop safety monitoring, operational logging, and large-scale spatial reasoning, unlike the subject-centric video dominating VLM research.Frontier models are predominantly trained on internet-crawled data biased toward standard photographic perspectives.
  • The Infrastructure AI Gap is the discrepancy between performance on general video benchmarks and reliability in dense, fixed-camera infrastructure environments.
  • VANTAGE-Bench evaluates eight task formulations across Semantic, Spatial, Temporal, and Spatio-Temporal Understanding for Infrastructure AI.It is designed to move beyond the single multiple-choice format used by comprehensive video suites.
  • 17-model zero-shot evaluation finds a localized rather than uniform deficit: event verification, referring expressions, and temporal localization decline, while video question answering and spatial pointing remain comparatively strong.Video question answering stays within 5.3 points of VideoMME, while 2D spatial pointing shows an 8.3-point surplus against BLINK at 32B.
  • The benchmark contributes three deployment domains, format-diverse tasks, a single-pass tracking protocol, and expert annotations across 3,346 media assets.Its annotations include video-task, image-grounding, and detection-box regimes, while several tasks require outputs without a candidate set.

2. Related Work

Prior benchmarks separate embodied, surveillance, domain-specific, and tracking settings, often relying on idealized imagery or multiple-choice outputs. VANTAGE-Bench combines fixed-infrastructure perspectives, natural-language grounding, multiple operational domains, and single-pass VLM tracking.

  • General video benchmarks rely on cinematic footage and MCQ formats that provide implicit answer priors, limiting tests of open-ended physical grounding.Infrastructure AI requires outputs such as precise bounding boxes and explicit temporal boundaries.
  • Existing physical-AI datasets are fragmented between moving-camera embodied perception and fixed-camera surveillance without natural-language interfaces.VANTAGE-Bench bridges these divides by pairing infrastructure perspectives with language-based evaluation across Transportation, Warehouse, and Smart Spaces.
  • Standard spatial-grounding datasets use well-lit, centered, distinct objects rather than dense multi-instance scenes viewed from elevated, oblique infrastructure cameras.The benchmark targets disambiguation among repeated objects such as pallets or vehicles.
  • Specialist trackers reach approximately 80 Success AUC on the benchmark’s tracklets, establishing a reference level for VLM tracking.
  • The benchmark contains 3,346 media assets across 3,342 video-task annotations, 4,281 image-grounding annotations, and 27,404 detection boxes.
  • VANTAGE-Bench decomposes joint grounding into language-to-space, language-to-time, and visual correspondence over time through Referring Expressions, Temporal Localization, and SOT.Its SOT protocol removes language to isolate visual correspondence and uses single-pass trajectory generation rather than rolling memory updates.

3. Dataset and Benchmark Construction

VANTAGE-Bench combines expert-annotated infrastructure footage with limited high-fidelity simulation across eight tasks and four reasoning pillars. Its construction emphasizes ecological validity, format authenticity, and scalable geometry-grounded annotation.

  • The dataset comprises 3,346 media assets across eight tasks, using static, elevated, wide-angle infrastructure cameras alongside limited simulated environments.Infrastructure viewpoints include mounting heights of 8 to 20 feet and viewing angles of 30 to 60 degrees from vertical.
  • The benchmark is designed around the disconnect between cinematic and MCQ evaluation priors and operational Infrastructure AI requirements.Its design principles target ecological validity and format authenticity in dense, multi-instance scenes.
  • VANTAGE-Bench organizes evaluation into Semantic, Spatial, Temporal, and Spatio-Temporal reasoning pillars.
  • Semantic Understanding: Semantic tasks comprise Event Verification and Video Question Answering for operational and multi-step reasoning over untrimmed sequences.
  • Spatial Understanding: Spatial tasks test referring-expression disambiguation, coordinate selection, and class-level object localization.
  • Temporal and Spatio-Temporal Understanding: Temporal tasks require exact action boundaries and generative chronological event description, while SOT predicts a target trajectory across a full sequence.
  • Annotations are predominantly expert-generated, with approximately 20% of VQA and Temporal splits using high-fidelity synthetic data for rare safety-critical anomalies.The synthetic component addresses severe long-tail events that are otherwise difficult to collect ethically and privately.
  • The Pointing Data Engine derives questions and answer options deterministically from human-verified box coordinates, while the VLM only rewrites the question linguistically.Same-class distractors preserve difficulty while keeping the answer key tied to expert geometry.

4. Evaluation Framework and Empirical Results

VANTAGE-Bench evaluates 17 VLMs zero-shot across operational visual-intelligence tasks and shows uneven capability across semantic, spatial, temporal, and spatio-temporal reasoning. The Infrastructure AI shortfall is concentrated in particular capabilities, with temporal understanding weakest and scaling effects task-specific.

  • Evaluation setup: 17 VLMs are evaluated zero-shot across four operational visual-intelligence pillars and eight tasks.The evaluation spans models from 2B to frontier scale, including open-weight and proprietary systems.
  • Semantic Understanding: 82.00 Macro F1 is the best Event Verification score, while GPT-5.6 Sol leads Video Question Answering at 78.08.Event Verification separates models more sharply than VQA.
  • Spatial Understanding: 86.97 is the best 2D Object Localization score, achieved by Cosmos3-Super ahead of the best proprietary model at 81.57.Qwen3.5-9B also leads 2D Referring Expressions at 77.16, ahead of the best proprietary score of 75.78.
  • Temporal Understanding: 55.71 mIoU and 37.28 SODA_c are the best Temporal Localization and Dense Video Captioning scores, making temporal understanding the weakest pillar in absolute terms.In dense captioning, models describe content semantically but struggle to localize it in time.
  • Operational deployment gap: Event verification, referring expressions, and temporal localization drop roughly 9 to 24 points against consumer references, while VQA stays within 5.3 points and pointing reaches +8.3 at 32B.The deltas run in both directions, locating the shortfall in specific capabilities rather than overall difficulty.
  • Returns to scale: Temporal Localization improves by an average of 7.5 points with scale, but Qwen3.5 regresses on four of eight tasks despite tripling in size.Scaling therefore produces task-specific rather than pillar-wide returns.
  • Temporal lag: No model exceeds 55.71 mIoU on Temporal Localization or 37.28 SODA_c on Dense Video Captioning across the 17-model suite.Qwen3-VL-32B scores 61.2 mIoU on Charades-STA versus 46.8 on the operational footage evaluated here.
  • Training-data effects: Physical-AI training data raises tracking by 26.1 points but dense captioning by only 1.8 across a shared-base comparison.The same sequence improves object localization by 14.2 and event verification by 9.5, while temporal localization moves 3.7.

5. Qualitative Analysis: Failure Modes of Infrastructure AI

The qualitative analysis identifies failures that separate fluent recognition from reliable spatial-temporal grounding. Models can reason well in single frames yet lose temporal persistence, with vulnerabilities also varying by deployment domain.

  • Semantic illusion: Models often recognize what happens but fail to resolve when it occurs, producing fluent yet weakly grounded predictions.Smaller models additionally show causal blindness in Event Verification by rejecting negatives while missing true events.
  • Spatial versus spatio-temporal breakdown: Strong single-frame spatial reasoning does not ensure temporal persistence, as weaker models initialize correctly but lose object identity over longer horizons.The observed tracking failures are tracking freeze and target drift.
  • Domain-specific vulnerabilities: Cosmos3-Super VQA accuracy falls from 75.73 in Warehouse and 71.51 in Transportation to 55.38 in Smart Spaces.Smart Spaces contains less structured human activity, while dense captioning follows a different domain ordering.
  • Frontier comparison: The distinctive frontier advantage is temporal persistence rather than spatial grounding, while proprietary access provides no across-the-board advantage.The two newest frontier releases lead most tasks by fairly uniform margins, with tracking as the large outlier.

6. Discussion and Limitations

VANTAGE-Bench establishes a baseline for Infrastructure AI evaluation while documenting scope, annotation, and deployment limitations. Its coverage remains constrained by sampled camera geometries, geography, task distributions, quality-assurance design, and zero-shot evaluation settings.

  • VANTAGE-Bench establishes a baseline for Infrastructure AI evaluation.
  • Scope limitations: Camera coverage excludes eye-level retail and low-angle industrial viewpoints because each deployment type requires separate licensing agreements.
  • Scope limitations: Real-world footage comes primarily from two U.S. municipalities, so geographic coverage is weighted toward North America.
  • Data and annotation limitations: Detection annotations average 43.6 boxes per image, reflecting dense infrastructure scenes rather than broad sampling.
  • Data and annotation limitations: Quality-assurance acceptance rates measure expert work passing sequential review, not agreement between independent annotators.
  • Data and annotation limitations: The 2D Spatial Pointing rewriting stage does not currently verify that rewritten questions preserve unique spatial cues in dense scenes.
  • Scope limitations: Single Object Tracking uses synthetic warehouse sequences, so conclusions remain limited to that distribution and do not yet separate tracking failure from synthetic-domain mismatch.
  • Evaluation limitations: The evaluation is strictly zero-shot, omits systematic extreme-weather and lighting coverage, and reports intervals covering item sampling rather than prompt or decoding variation.

7. Conclusion

VANTAGE-Bench evaluates VLMs in open-loop Infrastructure AI settings through an eight-task taxonomy spanning four reasoning pillars and 3,346 expert-annotated media assets. The results show uneven performance: deficits concentrate in event verification, referring expressions, and temporal grounding, while spatial localization and pointing remain comparatively strong.

  • VANTAGE-Bench evaluates open-loop Infrastructure AI with eight tasks spanning semantic, spatial, temporal, and spatio-temporal reasoning over 3,346 expert-annotated media assets.
  • Open-weight models lead 2D Object Localization outright, while the deficit against consumer-centric benchmarks concentrates in event verification, referring expressions, and temporal grounding.
  • Returns to scale are task-specific rather than pillar-specific, and pointing becomes a surplus at 32B.

A. Data Statistics and Benchmark Comparison

VANTAGE-Bench compares its annotation regimes with established benchmarks and formalizes each task as an input-to-structured-output mapping with a primary metric. Its video and image-grounding annotation counts are comparable to or larger than the cited counterparts, while dense detection reflects infrastructure-scene structure.

  • Table 4 compares benchmark scale, modality, and reasoning-pillar coverage, reporting VANTAGE-Bench annotations by regime elsewhere.
  • Video tasks contain 3,342 annotations versus VideoMME’s 2,700, across 854 versus 900 videos, while image grounding contains 4,281 versus BLINK’s 1,906 test examples.
  • Each VANTAGE-Bench task maps specified inputs to a structured output and reports a primary evaluation metric.
  • The formal input spaces include images, videos, queries, and initial boxes, while outputs include temporal segments, captions, candidate selections, and coordinate sets.
  • SODA_c measures narrative quality through temporal overlap and BERTScore similarity, Success AUC evaluates tracking across IoU thresholds, and Macro F1 addresses majority-class bias in Event Verification.

C.1. Data Adaptation for SOT and RefEx

VANTAGE-Bench adapts elevated-perspective RefDrone imagery for dense referring expressions and synthetic warehouse sequences for single-object tracking. Its analyses expose a separation between linguistic fluency, temporal grounding, event calibration, and persistent tracking.

  • 2D Referring Expressions: RefEx uses elevated RefDrone imagery to test dense disambiguation between identically colored vehicles using relative spatial relationships.
  • Single Object Tracking: Single Object Tracking adapts 17 synthetic warehouse scenes containing workers and robotic profiles, with sequences sampled at 8, 16, and 32 frames after an initial visual anchor.
  • Failure analyses: Dense video captioning separates semantically coherent descriptions from temporal localization, with low joint SODA_c indicating temporal smearing across events.
  • Failure analyses: Event Verification can produce asymmetric sensitivity and specificity profiles, which Macro F1 alone conceals.
  • Failure analyses: Qwen3.5-27B scores 85.46 on 2D Object Localization and 76.35 on 2D Referring Expressions but 24.40 on tracking, showing that spatial grounding does not imply temporal persistence.

D.3. Tracking Anchors and Horizon Analysis

The tracking evaluation compares VLMs with specialist trackers across increasing horizons and finds that short-horizon parity gives way to persistent degradation. This pattern occurs alongside domain variation, proprietary/open-weight asymmetries, and an aggregate score that weights pillars equally.

  • Tracking anchors: Approximately 80 Success AUC is achieved by specialist trackers and remains stable as the tracking horizon grows.The specialists establish the reference level using the same 200 tracklets, frames, and Success-AUC scorer as the VLMs.
  • Horizon analysis: At 8 frames, frontier VLMs come within roughly 5 points of specialist trackers, but their gap widens as the horizon extends.The reported frontier differences are −4.1 for GPT-5.6 Sol and −9.7 for Gemini 3.6 Flash from 8 to 32 frames.
  • Open-weight behavior: 45.3 versus 13.8 AUC at 32 frames shows Cosmos3-Super remains above the static floor, while weaker open-weight models finish near it at 18.0 and 15.7.Long-horizon degradation is broad, but convergence toward the static baseline is confined to weaker open-weight models.
  • Freeze-rate interpretation: Freeze rate must be interpreted jointly with AUC because unchanged boxes can be correct for stationary targets; the static-box floor is 99% throughout.Specialist freeze rates rise from 16–18% at 8 frames to 27–31% at 16 and 32 frames.
  • Operational domains: 55.38 VQA accuracy in Smart Spaces versus 75.73 in Warehouse indicates that semantic performance varies across operational domains.The paper attributes the Smart Spaces weakness to ambiguous human behaviour and the absence of structured, task-driven interactions.
  • Model comparisons: Open-weight models lead 2D Object Localization at 86.97 versus 81.57, so proprietary access alone does not confer an across-the-board advantage.They also lead 2D Referring Expressions by 1.38 points, while frontier models lead tracking by 11.3 points.

F.1. Compute Resources and Frameworks

The evaluation combines open-weight inference on university and cloud compute with API access to proprietary models, using a customized VLMEvalKit harness. All systems use deterministic greedy decoding with controlled visual and token budgets.

  • Compute resources: Open-weight models were evaluated on Clemson University’s Palmetto 2 cluster and additional cloud instances, while proprietary models were accessed through public APIs.The cluster configuration included seven nodes, each with eight NVIDIA H100 80GB GPUs.
  • Frameworks: The evaluation harness extends VLMEvalKit with custom scripts for SODA_c, temporal IoU, and spatial metrics.These extensions support the benchmark’s task-specific evaluation beyond standard multi-task evaluation.
  • Inference settings: Greedy decoding uses temperature = 0 with chain-of-thought disabled, while sampling and pixel budgets vary by model.The reported FPS ranges reflect model-specific sampling rates.

G.1. Item-Level Confidence Intervals

The paper estimates uncertainty primarily through item-level bootstrap intervals because deterministic decoding makes run-to-run variance small. Interval precision differs substantially by task, and seed variance cannot substitute for item-sampling uncertainty.

  • Bootstrap procedure: 95% percentile bootstrap confidence intervals resample test items 10,000 times for each model–task cell.For tracking, the resampling unit is the object tracklet rather than the individual frame because frames within a tracklet are strongly correlated.
  • Task resolution: Referring expressions and object localization have median half-widths of ±1.00 and ±1.27, while event verification reaches ±7.47.The wider event-verification interval reflects its smaller item pool and binary scoring.
  • Seed variance: All five tested tasks have seed standard deviations below 0.5 points, but seed runs do not measure the dominant uncertainty from item sampling.Item-level bootstrap intervals are therefore reported separately from decoding-randomness estimates.
Loading 2609.09396v1…