PaperlayerView exact revision on arXiv

Source-linked AI summary

AnyGroundBench: A Specialized-Domain Benchmark for Video Grounding in Vision-Language Models

Rintaro Otsubo, Ryo Fujii, Reina Ishikawa, Taiki Kanaya, Kanta Sawafuji, Hiroki Kajita, Shigeki Sakai, Hideo Saito, Ryo Hachiuma

arXiv:2607.02269v1cs.CVcs.AI

TL;DR

Existing STVG benchmarks rarely test adaptation to specialized domains, leaving VLM performance on uncommon visual concepts and complex dynamics insufficiently evaluated. AnyGroundBench addresses this gap across five domains, and evaluations of 15 VLMs show specialized-domain grounding remains highly challenging, with ICL gains limited and inconsistent.

  • Problem

    Existing STVG benchmarks focus mainly on everyday scenarios and lack standardized evaluation of VLM adaptation to uncommon, specialized domains.

  • Method

    AnyGroundBench evaluates zero-shot generalization and adaptation across five specialized domains using videos, queries, spatio-temporal boxes, and domain-specific training subsets.

  • Results

    Evaluations of 15 VLMs show specialized-domain STVG remains highly challenging, with spatial localization fragile and ICL gains limited and inconsistent.

  • Takeaways & Limitations

    Specialized-domain VLM grounding remains inadequate, while temporal grounding is often more tractable than full spatio-temporal grounding.

  • Takeaways & Limitations

    AnyGroundBench centers on single-query grounding and does not test open-ended expert QA, causal reasoning, or multi-object relational understanding.

Abstract

from arXiv · show

Vision-Language Models (VLMs) have demonstrated immense promise in Spatio-Temporal Video Grounding (STVG). However, current evaluation protocols are largely confined to zero-shot assessments on general, daily-life benchmarks. This creates a critical disconnect from real-world applications in specialized fields, where models inevitably encounter rare visual concepts and complex spatio-temporal dynamics. Since exhaustive pre-training across infinite data distributions is infeasible, the ability to adapt to novel domains is essential. To bridge this gap, we introduce AnyGroundBench, a domain-adaptation benchmark designed to shift the STVG evaluation paradigm from static zero-shot testing to rigorous domain adaptation. Targeting five specialized domains (animal, industry, sports, surgery, and public security), AnyGroundBench pairs newly captured videos such as expert-annotated mouse behaviors with established datasets, unifying them through dense, high-fidelity spatio-temporal annotations. Crucially, the benchmark provides dedicated training subsets to systematically measure domain adaptability. We extensively evaluate 15 state-of-the-art VLMs, assessing their zero-shot generalization and In-Context Learning (ICL) capabilities under practical computational constraints. Ultimately, our findings reveal that current models fail in both zero-shot and ICL-based adaptation when confronted with specialized domains, exposing critical flaws in spatio-temporal reasoning that future research must address.

1 Introduction

AnyGroundBench addresses the mismatch between everyday, zero-shot STVG benchmarks and specialized real-world domains by evaluating domain adaptation. Its evaluation of 15 VLMs reveals failures in spatial grounding and unstable ICL adaptation.

  • Motivation: Established STVG benchmarks mostly cover everyday scenarios and general objects, despite specialized applications requiring localization of rare visual concepts and intricate spatio-temporal dynamics.The passage gives mouse-scratching localization for dermatology or neurology research as an example.
  • Motivation: Zero-shot evaluation cannot fully measure adaptation because models cannot pre-learn the near-infinite data distributions arising across novel real-world domains and tasks.The paper identifies both missing specialized-domain benchmarks and overreliance on zero-shot protocols as barriers.
  • Benchmark: AnyGroundBench shifts STVG evaluation toward specialized-domain adaptation and provides 2,040 videos with natural-language queries and corresponding spatio-temporal bounding boxes.The benchmark is designed to bridge gaps in domain coverage and adaptability evaluation.
  • Evaluation: The study evaluates 15 VLMs and separates STVG into Spatial Video Grounding and Temporal Video Grounding to identify where spatio-temporal reasoning fails.The evaluation includes open-source and closed-source proprietary models with diverse architectures and sizes.
  • Findings: Current VLMs show limited specialized-domain grounding: proprietary models remain impractical, while open-source models completely collapse and lack fundamental spatial reasoning.This finding is presented as one of three critical conclusions about VLM grounding capability in specialized domains.
  • Findings: Spatial grounding is the primary bottleneck: temporal grounding shows promise under loose thresholds, but practical spatio-temporal performance collapses at vIoU@0.5.The collapse is attributed to severe limitations in Spatial Video Grounding.
  • Findings: Inference-time ICL adaptation is inconsistent: few-shot demonstrations can improve temporal localization but often reduce grounding accuracy depending on the model and domain.The result motivates development of a robust adaptation approach.

2 Related Work

Prior STVG benchmarks and VLM evaluations primarily target generic, daily-life domains and zero-shot performance. AnyGroundBench addresses this gap by targeting specialized fields and explicitly evaluating domain adaptability, including through ICL.

  • Benchmark for Spatio-Temporal Video Grounding: Traditional STVG benchmarks such as VidSTG and HC-STVG focus mainly on generic objects and human-centric activities.VidSTG is built upon VidOR, while HC-STVG is sourced from AVA and YouTube.
  • Benchmark for Spatio-Temporal Video Grounding: Existing benchmarks remain confined to common daily-life domains, leaving specialized real-world scenarios underrepresented.This limitation stems from reliance on general internet platforms and existing generic datasets.
  • Benchmark for Spatio-Temporal Video Grounding: AnyGroundBench targets specialized fields and combines newly captured expert-curated videos with established domain data.It is presented as a benchmark designed to address the specialized-domain gap in existing evaluations.
  • Spatio-Temporal Video Grounding via VLMs: VLM-based STVG evaluations rely almost exclusively on zero-shot performance on general-domain benchmarks.The passage frames this as a limitation of current evaluation protocols rather than of VLM video understanding broadly.
  • Spatio-Temporal Video Grounding via VLMs: AnyGroundBench is introduced as the first standardized benchmark explicitly measuring domain adaptability, including through In-Context Learning (ICL).The motivation is that models cannot be pre-trained on every conceivable domain and therefore must adapt rapidly to unseen specialized data.

3 AnyGroundBench

AnyGroundBench is a domain-adaptation benchmark for spatio-temporal video grounding across five specialized, real-world domains, with per-domain training subsets supporting adaptation evaluation. It unifies three grounding tasks and high-fidelity annotations while testing challenging behaviors, interactions, procedures, and incidents across balanced domain data.

  • Benchmark scope: AnyGroundBench evaluates STVG capability and adaptation across animal, industry, sports, surgery, and public security, providing training subsets for each domain.The benchmark is designed around specialized domains with distinct challenges and high real-world relevance.
  • Task formulation: The benchmark defines STVG, SVG, and TVG over video-query pairs, using one VLM whose task is selected solely by a task-specific system prompt.STVG predicts spatial and temporal localization, SVG isolates per-frame spatial localization in trimmed videos, and TVG predicts temporal boundaries.
  • Adaptation protocol: AnyGroundBench supports comparable adaptation methods by consuming domain training data, including retrieval-based demonstrations, parameter updates, or full fine-tuning.The benchmark is agnostic to the adaptation operator; the reference experiments instantiate it as backpropagation-free m-shot ICL with retrieved in-domain demonstrations.
  • Domain composition: The datasets cover fine-grained mouse behaviors, professional tool interactions, tactical sports plays, open and laparoscopic surgery, and urban surveillance and traffic incidents.These collections combine established datasets with newly curated or captured datasets, including Mouse Scratching and American Football.
  • Dataset statistics: 3,522 queries are evenly distributed across five domains, with average videos of 47.13s and target segments of 7.51s, or 16% of video length.The statistics indicate a balanced testbed requiring precise temporal localization within relatively long videos.

4 Experiments

Experiments evaluate 15 VLMs on STVG, TVG, and SVG using zero-shot and 2-shot ICL settings across specialized domains. Results show that specialized-domain grounding remains difficult, with limited and task-dependent benefits from ICL and strong sensitivity to temporal duration and object size.

  • Experimental Setup: The evaluation benchmarks proprietary, open-source specialized, and open-source general-purpose VLMs on STVG, TVG, and SVG using vIoU@0.3, tIoU@0.3, and sIoU@0.3.The study compares zero-shot performance with 2-shot ICL, retrieving two relevant training examples by similarity.
  • Main Results: STVG remains highly challenging, while ICL gains are limited and inconsistent across specialized domains.Gemini-3.1-Pro improves from 7.69 to 11.8 on Industry, but simple inference-time adaptation remains insufficient.
  • Impact of the Number of Demonstrations: STVG rises only from 10.5 zero-shot to 11.6 with 2-shot and 11.5 with 4-shot ICL, while average SVG consistently declines as demonstrations increase.Increasing demonstrations primarily benefits TVG, indicating that additional examples do not uniformly improve grounding.
  • Effectiveness of In-Context Selection Strategies: Retrieval strategy is task-dependent: text+video retrieval performs best on averaged TVG and STVG, whereas random retrieval is strongest for averaged SVG.The compared strategies include random, text-only, video-only, and text+video retrieval.
  • Sensitivity to Temporal and Spatial Scales: Temporal and spatial scale strongly affect grounding: TVG rises from 6.82 for events shorter than 1 second to 34.0 for events longer than 3 seconds, while SVG rises from 2.61 for small objects to 18.8 for large objects.Across shared spatial tertile bins, STVG also rises from 0.43 to 4.43, showing greater sensitivity in full spatio-temporal grounding.

5 Conclusion · Appendix

AnyGroundBench is a domain-specialized STVG benchmark spanning five specialized domains with training and evaluation subsets for assessing VLM zero-shot grounding and adaptation. Experiments show that specialized-domain STVG remains highly challenging, while its public expert-annotated release supports research on data-efficient domain adaptation.

  • 5 Conclusion: AnyGroundBench targets spatio-temporal video grounding in vision-language models across five specialized domains.The domains include surgery, industry, and animal behavior.
  • 5 Conclusion: The benchmark provides separate training and evaluation subsets for each domain.This design enables assessment of both zero-shot grounding and adaptation capabilities.
  • 5 Conclusion: AnyGroundBench enables evaluation of VLM zero-shot grounding and domain-adaptation capabilities.Its structure shifts evaluation beyond static zero-shot assessment by including dedicated training data.
  • 5 Conclusion: 15 VLM experiments show that specialized-domain STVG remains highly challenging.The conclusion reports results across fifteen evaluated vision-language models.
  • 5 Conclusion: Even the strongest proprietary model lacks accurate zero-shot grounding capability in specialized domains.This finding indicates that leading proprietary systems still struggle with specialized-domain grounding.
  • 5 Conclusion: Retrieval-based in-context learning provides only limited adaptation according to the reported experiments.The supplied conclusion passage truncates immediately after describing retrieval-based ICL as limited.
  • 5 Conclusion: The public release standardizes expert-annotated data in a unified format across five specialized domains.The benchmark is intended to support evaluation under limited training-data scenarios.
  • 5 Conclusion: AnyGroundBench is intended to encourage robust domain-adaptation techniques for video grounding with limited training data.The broader-impacts statement frames the benchmark as infrastructure for future research on specialized-domain adaptation.

A Implementation and Inference Details … B Prompts

The paper specifies model-aware inference, preprocessing, and few-shot retrieval configurations, then provides family-specific prompts and output formats for temporal, spatial, and spatio-temporal grounding. Model outputs are parsed and normalized into a unified representation for evaluation.

  • A.1 Inference Configuration: Inference generally follows each model or API’s defaults, with Qwen3.5 run without thinking mode to preserve timestamp and bounding-box format compliance.The exception was motivated by preliminary trials in which free-form reasoning consumed most output tokens and impaired required-format adherence.
  • A.1 Inference Configuration: Video preprocessing samples most models at 1 fps with a maximum of 120 frames and resizes sampled frames to 512 pixels on the longer side.GPT models use the same 1-fps and 120-frame constraints for longer videos, while LLaVA-ST uses a fixed 100-frame budget.
  • A.2 In-Context Learning Setup: Few-shot evaluation uses m = 2 demonstrations retrieved from the domain-specific training split with a hybrid visual-textual similarity score.The score combines visual and textual cosine similarities from InternVideo2 and SentenceBERT embeddings.
  • A.3 Computational Requirements: Local open-source experiments use RTX 5090, RTX PRO 5000 Blackwell, and A100 GPU servers, whereas GPT and Gemini experiments use official APIs.The listed GPUs provide 32GB, 48GB, and 80GB memory capacities, respectively.
  • B Prompts: The evaluation prompts cover open-source, proprietary, and specialist LLaVA-ST models, with family-specific temporal identifiers, box orders, and coordinate scales.The paper provides prompts for Qwen3-VL, Qwen3.5, InternVL-3, InternVL-3.5, Eagle 2.5, Gemini variants, GPT models, and LLaVA-ST.
  • B.1 Temporal Grounding: Temporal grounding prompts request time ranges using seconds, frame indexes, or model-family-specific timestamp formats, without explanatory text.Gemini requests [start, end] time ranges, GPT requests index ranges, and general open-source prompts request timestamps in seconds.
  • B.2 Spatial Grounding: Spatial grounding prompts request bounding boxes while varying coordinate order, normalization, frame or timestamp identifiers, and JSON output across model families.Gemini uses [y_min, x_min, y_max, x_max] in [0, 1000], GPT uses [x_min, y_min, x_max, y_max] normalized to 0.000–1.000, and other templates differ accordingly.
  • B.3 Spatio-Temporal Grounding: Spatio-temporal prompts jointly request timestamps and spatial boxes, with Gemini, GPT, general open-source, and LLaVA-ST templates specifying different identifiers and formats.All outputs are subsequently parsed and converted into a unified internal representation before evaluation.

C Details of Constituent Datasets

AnyGroundBench comprises ten datasets across five specialized domains, unified through dataset-specific curation and annotation enhancements for spatio-temporal video grounding. These resources cover fine-grained animal behaviors, industrial interactions, sports actions, surgical procedures, and public-security anomalies.

  • Benchmark composition: AnyGroundBench comprises ten datasets spanning animal, industry, sports, surgery, and public security, with constituent-specific curation and annotation enhancements.The benchmark combines newly curated datasets with established resources adapted for STVG evaluation.
  • Animal: Animal Kingdom covers 850 species across 50 hours, with temporal grounding labels extended by per-frame spatial boxes to create STVG annotations.The benchmark uses its temporal grounding subset rather than its other annotation types.
  • Animal: Mouse Scratching is a surgeon-curated clinical benchmark using synchronized four-view videos to localize fine-grained, high-frequency behaviors under motion blur and foggy conditions.It targets behaviors such as hind-paw scratching versus forepaw grooming and supports research on atopic dermatitis models.
  • Industry and sports: Industrial and sports datasets derive STVG targets from dense human-object interaction or action annotations, testing tool trajectories, high-velocity motion, and concurrent multi-agent actions.MECCANO and ENIGMA-51 provide industrial interaction-based targets, while MultiSports covers 66 action classes across four sports.
  • Surgery and public security: American Football, surgical, and public-security datasets add expert-curated tactical plays, context-linked instrument queries, and spatial annotations for anomalous entities or traffic participants.American Football contains 123 multi-view play sequences; EgoSurgery and CholecTrack20 enrich surgical data with textual queries, while UCA and DoTA support anomaly grounding.

D Data Collection and Annotation Details for Newly Captured Datasets … E Licenses and Redistribution Constraints

AnyGroundBench’s newly captured Mouse Scratching and American Football datasets use domain-specific queries and dense spatio-temporal annotations designed for unified STVG evaluation. Existing-dataset components remain governed by source licenses, while the newly captured datasets are released under CC BY-NC-SA 4.0.

  • D Data Collection and Annotation Details for Newly Captured Datasets: The two newly captured datasets were annotated with dense spatio-temporal tubes and domain-faithful query descriptions for unified STVG benchmarking.The annotation objective was to preserve domain-specific event definitions while maintaining benchmark usability.
  • D.1 Mouse Scratching: Mouse Scratching videos used four synchronized GoPro views to reduce viewpoint ambiguity when distinguishing subtle face and forelimb behaviors.The setup included two front views, a side view, and a top-down spatial view; grooming involves forepaw-directed face touching, whereas scratching involves hind-paw-directed touching.
  • D.1 Mouse Scratching: Mouse Scratching spatial annotations combined manual first-frame mouse boxes in CVAT with SAM2 tracking over temporally clipped segments.The manually specified first-frame bounding box served as the pixel-coordinate input prompt, and prediction logits were binarized at threshold 0.
  • D.2 American Football: American Football videos came from university-level amateur games recorded with handheld cameras, including spectators and other non-player distractors.An experienced American football player helped construct natural textual queries for the dataset’s recording context.
  • D.2 American Football: American Football spatial annotations tracked kick and pass targets from manually boxed first frames using Ultralytics SAM3VideoPredictor.Tracking operated on temporally clipped videos, and predicted spatial extents were extracted from pixels with mask values greater than zero.
  • E Licenses and Redistribution Constraints: Curated components derived from existing datasets retain their source licenses, and the release provides metadata and derived annotations rather than redistributing original videos.Users must obtain source videos from official providers and comply with the corresponding terms of use.
  • E Licenses and Redistribution Constraints: Mouse Scratching and American Football are newly captured datasets released under CC BY-NC-SA 4.0.Other components have separate access or licensing conditions, including Animal Kingdom5 via a dataset-use questionnaire and MultiSports8 under CC BY-NC 4.0.

F Main Results under Stricter Evaluation Metrics · G Further Analysis

The benchmark supplements its main evaluation with stricter metrics across spatio-temporal, temporal, and spatial video grounding. Results are reported in Tables 7–9 for STVG, TVG, and SVG, respectively.

  • F Main Results under Stricter Evaluation Metrics: For STVG, evaluation uses mtIoU, mvIoU, and vIoU@0.5 under stricter IoU thresholds.These metrics follow prior STVG evaluation practices.
  • F Main Results under Stricter Evaluation Metrics: For TVG, the reported metrics are tIoU@0.5, tIoU@0.7, and mtIoU.These metrics follow established TVG conventions.
  • F Main Results under Stricter Evaluation Metrics: For SVG, the evaluation reports msIoU and sIoU@0.5.These metrics extend the supplementary evaluation to spatial video grounding.
  • F Main Results under Stricter Evaluation Metrics: Table 7 provides supplementary results for STVG under the stricter evaluation metrics.The table complements the benchmark’s main evaluation results.
  • F Main Results under Stricter Evaluation Metrics: Table 8 provides supplementary results for TVG under the stricter evaluation metrics.The table complements the benchmark’s main evaluation results.
  • F Main Results under Stricter Evaluation Metrics: Table 9 provides supplementary results for SVG under the stricter evaluation metrics.The table complements the benchmark’s main evaluation results.

G.1 Observation from Main Table

AnyGroundBench reveals sharply domain-dependent performance: Animal is easiest for zero-shot STVG, while Surgery and Sports are hardest, and Public Security is exceptional on TVG. Full STVG remains unresolved, with SVG and TVG substantially stronger but inference-time adaptation unreliable.

  • Domain-wise Performance Patterns: Animal is easiest for zero-shot STVG, whereas Surgery and Sports are most difficult; Public Security is exceptional on TVG, while Sports remains weak across tasks.ICL benefits Surgery and Public Security most, especially on TVG, whereas Animal shows little or negative improvement; Sports also remains low for the strongest proprietary models.
  • Spatio-Temporal Video Grounding: Gemini-3.1-Pro reaches only 22.8 on Public Security and 16.5 on Animal, dropping to 1.22 on Sports for zero-shot STVG.The strongest proprietary model therefore does not achieve consistently strong STVG performance across specialized domains.
  • Spatio-Temporal Video Grounding: ICL is unreliable for STVG: GPT-5.1 improves from 4.8 to 9.63 on Public Security, while Gemini-3.1-Pro and Qwen3.5-9B decline on Animal.Gemini-3.1-Pro drops from 16.5 to 12.7, and Qwen3.5-9B drops from 4.45 to 2.54, motivating diagnosis through TVG and SVG.
  • Spatial Video Grounding: Gemini-3.1-Pro’s SVG scores exceed STVG scores on Animal (16.5 to 70.7), Industry (7.69 to 41.4), Surgery (4.16 to 26.1), and Public Security (22.8 to 52.0).The gaps indicate that full spatio-temporal grounding often fails even when the target can be localized within a trimmed temporal window.
  • Temporal Video Grounding: 37.5 on Animal, 39.6 on Industry, 37.4 on Sports, 37.9 on Surgery, and 69.4 on Public Security are the strongest zero-shot TVG scores.TVG is more tractable than STVG, and ICL helps more consistently: GPT-5.1 rises from 22.6 to 41.7 on Surgery, while Gemini-3-Flash rises from 66.8 to 78.4 on Public Security.

G.2 Threshold Sensitivity Analysis · G.3 Prompt Sensitivity to Bounding-Box Coordinate Order

Threshold sensitivity analysis shows that stricter IoU requirements sharply expose failures in precise spatio-temporal grounding across models and domains. Bounding-box coordinate order also substantially affects spatial and tube metrics, while temporal localization remains comparatively stable.

  • G.2 Threshold Sensitivity Analysis: STVG accuracy decreases sharply as IoU thresholds increase, with Gemini-3.1-Pro strongest across most thresholds but approaching zero at high thresholds.TVG degrades more gradually than STVG and SVG, indicating comparatively more robust temporal localization.
  • G.2 Threshold Sensitivity Analysis: Animal and Public Security remain strongest under loose thresholds, whereas Sports and Surgery are consistently difficult for Gemini-3.1-Pro.All domains nevertheless show steep STVG declines as thresholds increase.
  • G.2 Threshold Sensitivity Analysis: The spatial-video-grounding results table reports msIoU and sIoU@0.5 for zero-shot and 2-shot ICL settings, with colored scores marking changes relative to baseline.The text-only baseline uses Gemini-3.1-Pro without video inputs.
  • G.2 Threshold Sensitivity Analysis: STVG failures across domains reflect a broader lack of precise spatio-temporal overlap rather than ranking artifacts at a single threshold.The threshold analysis therefore distinguishes coarse success from precise tube grounding.
  • G.3 Prompt Sensitivity to Bounding-Box Coordinate Order: Both prompt variants obtain much higher spatial and volume scores when predicted boxes are parsed as yxyx rather than xyxy.Table 10 compares zero-shot outputs from both prompt variants under both coordinate-order assumptions.
  • G.3 Prompt Sensitivity to Bounding-Box Coordinate Order: 9.77 mvIoU is achieved by the corrected yxyx prompt under yxyx parsing, compared with 3.25 under xyxy parsing.The older xyxy prompt similarly improves from 3.31 to 9.72 when parsed as yxyx.
  • G.3 Prompt Sensitivity to Bounding-Box Coordinate Order: Gemini-3.1-Pro tends to follow its native [y_min, x_min, y_max, x_max] box_2d convention even when prompted for xyxy.Wrong x/y parsing collapses spatial overlap, tube precision, and tube recall, while temporal metrics remain nearly unchanged.

H Limitations and Future Work

AnyGroundBench is limited to single-query grounding in short videos, leaving open-ended expert QA, multi-object relations, and long-form video search untested. Future extensions should broaden task settings and support longer-video retrieval with adaptive temporal reasoning.

  • Task Scope: The benchmark targets single-query grounding from one object tube, excluding open-ended expert QA, causal reasoning, and multi-object relational understanding.Extensions to multi-query, multi-object, and instruction-following settings are proposed for broader specialized video workflows.
  • Future Work: Future work should extend AnyGroundBench to multi-query, multi-object, instruction-following, and longer-video settings.These extensions address broader workflow understanding and long-form retrieval beyond the current benchmark scope.
  • Video Length: With average videos of 47.13s and target segments of 7.51s, AnyGroundBench does not evaluate long-form video search.Longer-video extensions would require coarse-to-fine retrieval, adaptive frame selection, and memory over sparse events.
Loading 2607.02269v1…