PaperlayerView exact revision on arXiv

Source-linked AI summary

Event-STVG: Spatio-Temporal Video Grounding for Long-Form Event Reasoning

Yizhe Zhang, Xujia Wang, Songqi Tang, Xujie Tang, Leo Jiahao Jiang, Jiacheng Wang, Bowen Qu, JinBo Li, Qin Wang, Ji Qi, Lei Hou, Bin Xu, Juanzi Li

arXiv:2610.22444v1cs.CV

TL;DR

Existing STVG benchmarks largely center on instance-specific descriptions, leaving long-form event reasoning less directly evaluated. Event-STVG introduces event-occurrence and role grounding across a multi-domain benchmark; models struggle across longer contexts and domains, with the best model reaching only 5.4% m-vIoU at 10 minutes.

  • Problem

    Existing STVG benchmarks largely use instance-specific descriptions, while long-form reasoning is increasingly emphasized in MLLMs.

    Source: §1 · p. 2

  • Method

    Event-STVG asks models to resolve an event occurrence, infer its semantic-role fillers, and jointly ground the event and participants across 4,016 annotated occurrences in eight domains.

    Source: §1 · p. 2

  • Results

    5.4% m-vIoU is the best model result at 10 minutes; analyses also report repeated-event confusion, spatial role-grounding difficulty, and rare complete grounding.

    Source: §1 · p. 3, §4.2 · p. 6, §5 · p. 9

  • Takeaways & Limitations

    Event-STVG exposes challenges in jointly reasoning about event timing and structured participants across space and time.

    Source: §1 · pp. 2–3

Abstract

from arXiv · show

Spatio-Temporal Video Grounding (STVG) aims to jointly localize a target in space and time given a natural-language query. Despite continued advances, existing STVG benchmarks remain largely centered on localizing targets through instance-specific descriptions, leaving them largely decoupled from long-form reasoning increasingly emphasized in multimodal large language models (MLLMs). In this paper, we introduce Event-STVG, a task that shifts STVG from locating what is described toward event-level reasoning by unifying long-form context, repeated-event reasoning, event localization, role inference, and multi-target grounding. To facilitate research on Event-STVG, we construct a large-scale benchmark comprising 4,016 annotated event occurrences across soccer and seven additional domains. Experiments with state-of-the-art MLLMs reveal substantial difficulty, with the strongest model achieving only 5.4% m-vIoU at the 10-minute horizon. Across domains, we consistently observe confusion among repeated same-type events, spatial grounding bottlenecks, and rare success in jointly grounding all required roles. Event-STVG establishes a new challenge for visual understanding in MLLMs, calling for future models that can jointly reason about complex events across space and time. Code and annotations are available at https://github.com/THU-KEG/EventSTVG

1 INTRODUCTION

Existing STVG benchmarks primarily test matching detailed descriptions to visual instances, with limited demands on long-form reasoning. Event-STVG reframes grounding as event-level reasoning and evaluates it with a benchmark and systematic model analyses.

1 INTRODUCTION

  • Existing STVG benchmarks primarily test whether models can match detailed linguistic descriptions to visual instances, placing limited demands on long-form reasoning.This formulation remains suited to precise referent localization.Source: §1 · pp. 1–2
  • Event-STVG requires models to identify an event occurrence, infer its role fillers, and jointly ground the event interval and participants.The benchmark comprises 4,016 annotated occurrences across eight domains.Source: §1 · p. 2
  • Systematic analyses identify recurring failures in event-instance selection, spatial role grounding, and complete event grounding.These patterns are reported across models and domains.Source: §1 · p. 3

2 RELATED WORK

Prior STVG work extends localization to multiple targets, compositional queries, specialized domains, and longer videos, but remains largely description-guided. Related event and video-QA benchmarks address parts of the problem without jointly grounding an event and its role fillers.

Spatio-Temporal Video Grounding

  • STVG extensions cover multiple targets (Ji et al., 2024), compositional queries, specialized domains (Otsubo et al., 2026; Xu et al., 2025), and minute-scale contexts (Gu et al., 2026), yet remain largely description-guided.Earlier benchmarks ground language-referred entities in short videos (Tang et al., 2022).Source: §2 · p. 3

Structured and Grounded Event Understanding

  • Event-understanding benchmarks either omit event temporal grounding or do not jointly ground event timing and role fillers spatially.VidEvent and NEST localize events and structure arguments without spatially grounding role fillers; MultiVENT-G separates temporal and spatial retrieval (Asgarov et al., 2026).Source: §2 · p. 3

Video QA with Spatio-Temporal Evidence

Related benchmark scope

  • Event-STVG’s construction and composition cover major soccer events and seven additional domains, with diverse event types and semantic roles.The Table 1 caption identifies comparison dimensions including temporal/spatial grounding, joint grounding, event-role grounding, annotation, and duration.Source: Table 1 · p. 3, Figure 3 · p. 4

3 EVENT-STVG

Event-STVG queries specify an event type, occurrence index, and required roles, asking for the event interval and each role’s spatial trajectory. Its benchmark combines human-verified annotations across soccer and seven other domains, with repeated events and multiple roles common in the data.

3.1 TASK FORMULATION

  • Given an event type, ordinal index, and required roles, Event-STVG asks for the indexed occurrence’s temporal interval and each role’s spatial trajectory.Queries omit participant descriptions and interaction patterns, so models must infer how the occurrence is realized from video.Source: §3.1 · p. 4

3.2 BENCHMARK CONSTRUCTION

  • Soccer annotation uses candidate retrieval and tracking assistance, while annotators set event boundaries and verify or correct role trajectories; other domains use human-drawn or assigned boxes.Contexts are built from oracle windows through 10-minute settings for soccer, and minute-scale videos for the other domains; approximately 10% of data is cross-annotated, with mean temporal IoU 0.9039 and mean spatial IoU 0.9161.Source: §3.2 · p. 5

3.3 BENCHMARK STATISTICS

  • The benchmark contains 4,016 events, 26 event types, 30 roles, 7,323 event–role pairs, and 59,381 boxes across eight domains.Multiple roles occur in 58.91% of instances; repeated same-type events rise from 13.68% of one-minute soccer contexts to 68.69% at ten minutes, and appear in 99.40% of remaining-domain queries.Source: §3.3 · p. 5

4 EXPERIMENT

Event-STVG remains difficult from short to long contexts and across domains, with joint grounding weakest at the 10-minute horizon. Error analysis identifies a hierarchy of occurrence selection, spatial role grounding, and complete multi-role composition bottlenecks.

4.1 SETTINGS

  • The evaluation covers ten vision-language models, oracle and 1-minute soccer settings, 3-, 5-, and 10-minute subsets, and 100 cross-domain videos.Metrics include m-tIoU for event intervals and m-vIoU for role trajectories; missing or invalid predictions score zero.Source: §4.1 · pp. 5–6

4.2 MAIN RESULTS

  • At 1 minute, six of nine general-purpose models score below 1% m-vIoU, while the STVG specialist PTD reaches only 1.72%.Even with ground-truth event intervals, role grounding remains difficult; GPT leads temporal localization, while Gemini is stronger in joint grounding.Source: Table 2 · p. 6, §4.2 · p. 6
  • At 10 minutes, the best m-vIoU is only 5.40%, and no model dominates temporal localization and joint grounding.GPT retains the strongest temporal localization, while Gemini is substantially stronger in joint grounding.Source: §4.2 · p. 6, Table 3 · p. 7
  • Across domains, mean m-vIoU ranges from 21.3% in the best-performing domain to 0.69% in the lowest, with no domain near saturation.Daily work and entertainment are comparatively easier than domains dominated by rapid multi-agent interactions.Source: Table 4 · p. 7, §4.2 · p. 7

4.3 ERROR ANALYSIS

  • The analysis organizes failures into event-instance resolution, spatial role grounding, and multi-role composition, increasingly demanding levels of Event-STVG.The first is especially pronounced in sports with repeated same-type events; the latter two remain prominent across domains.Source: §4.3 · p. 7
  • Longer soccer inputs increase confusion with another valid same-type occurrence; for later basketball targets, average confusion reaches 41.81%.Among predictions that hit the target at every duration, tIoU does not significantly decline as videos lengthen.Source: §4.3 · pp. 7–8, Figure 4 · p. 8
  • Across domains, the spatial grounding gap averages 71.12%, corresponding to only 28.88% mean spatial IoU within temporally overlapping frames.A model may identify event timing yet assign boxes to the wrong entity, produce inaccurate role trajectories, or fail to track a role.Source: §4.3 · p. 8, Figure 5 · p. 8
  • Complete multi-role grounding succeeds for only 0.73% of events on average across domains, even though models may recover individual participants.This bottleneck persists after the correct occurrence and its temporal extent have been identified.Source: Figure 5 · p. 8, §4.3 · p. 9

5 CONCLUSION

Event-STVG reframes spatio-temporal grounding as structured event reasoning and introduces a benchmark of 4,016 occurrences across eight domains. Current models remain far from solving the task, with analyses identifying challenges in repeated-event resolution, spatial role grounding, and multi-role composition.

5 CONCLUSION

  • Event-STVG requires resolving repeated event occurrences, inferring semantic roles, and jointly grounding participants in space and time across 4,016 annotated events in eight domains.The strongest model achieves only 5.4% m-vIoU with 10-minute context.Source: §5 · p. 9
  • Longer contexts hinder event-instance resolution, temporal localization does not ensure accurate spatial grounding, and complete grounding becomes harder as semantic roles multiply.The authors suggest verifiable grounding signals may provide fine-grained feedback for reinforcement-learning-based post-training.Source: §5 · p. 9
  • Generative AI assisted with literature research, writing, and coding, but not with generating or annotating benchmark ground truth; authors verified AI-assisted outputs.Code and annotations are stated to be released.Source: REPRODUCIBILITY STATEMENT · p. 9

A EVALUATION METRICS

The evaluation defines temporal and multi-role spatial grounding scores, then prompts models to return an event window and role trajectories in a structured format. The appendix specifies paired-bootstrap reporting for comparisons.

A EVALUATION METRICS

  • m-vIoU averages role-level tube IoU equally within each event and events equally overall; missing or invalid predictions contribute zero.vIoU@0.3 and vIoU@0.5 count events whose event-level vIoU exceeds the respective threshold.Source: A EVALUATION METRICS · p. 14

B PROMPT TEMPLATES AND OUTPUT FORMAT

  • Prompts specify the event type, ordinal occurrence, temporal boundaries, and roles, and request a temporal window plus role boxes at 0.5-second intervals.The soccer example requests the last passer, shooter, and goalkeeper; Figure 6 shows examples across eight domains.Source: B PROMPT TEMPLATES AND OUTPUT FORMAT · p. 15, Figure 6 · p. 16

C EVALUATION PROTOCOL AND IMPLEMENTATION DETAILS

  • Paired comparisons use 10,000 bootstrap resamples with a fixed seed of 21 and two-sided 95% percentile confidence intervals.Metrics are reported as percentages unless otherwise stated.Source: C EVALUATION PROTOCOL AND… · p. 17

C.1 EVALUATION SETTINGS

The evaluation separates oracle role-grounding tests from end-to-end event retrieval and uses longer contexts on a fixed subset. Model-specific input interfaces are retained where possible, while a common parser and scoring rules handle predictions.

C.1 EVALUATION SETTINGS

  • The soccer benchmark evaluates 3,516 events in oracle and 1-minute settings, then tests a fixed stratified 300-event subset at 3, 5, and 10 minutes.Oracle supplies the ground-truth event clip to isolate spatial role grounding; the 1-minute setting requires occurrence identification, interval prediction, and role grounding.Source: §C.1 · p. 17
  • The independent multidomain extension contains 500 queries over 100 videos, using the same task formulation with domain-specific event and role definitions.Role scores are averaged equally within each query, then queries are averaged; distinct subsets are reported separately rather than pooled.Source: §C.1 · p. 17

C.2 MODEL INPUTS AND INFERENCE CONFIGURATION

  • Models use their supported video interfaces while keeping the semantic task specification fixed whenever possible.For GPT-5.6 Sol, soccer videos are converted to ordered JPEG frames at 2 FPS, with up to 1,200 frames for the 10-minute context.Source: §C.2 · p. 17

C.3 PARSING AND SCORING RULES

  • A common frozen parser scores complete JSON predictions without repairing malformed responses, while failed requests and unparseable outputs remain in the denominator and receive zero for affected components.Temporal and spatial components are evaluated independently where possible; joint vIoU penalizes missing boxes, spatial errors, and temporal over-extension.Source: §C.3 · p. 18

C.4 REPEATABILITY UNDER DEFAULT INFERENCE SETTINGS

Repeated runs show that hosted-model results can vary, while PTD is repeatable in the evaluated configuration. PTD’s role-by-role interface omits the ordinal event index and full boundary specification, which may matter when same-type events recur in the 1-minute setting.

C.4 REPEATABILITY UNDER DEFAULT INFERENCE SETTINGS

  • −3.024 pp: Qwen3.8-Max changes from 47.726% to 44.702% m-tIoU between R0 and R2, with a 95% bootstrap interval of [−5.600, −0.439].The Qwen models show relatively small Oracle variation, but their 1-minute results are not identical across runs.Source: §C.4 · p. 18, Table C.2 · p. 18
  • GLM-4.6V’s larger variation is primarily associated with service-side input failures: Oracle runs contain 0, 135, and 152 API errors, while the common successfully parsed subset varies from 12.521% to 13.439% Oracle m-vIoU.Failures remain in the end-to-end statistics rather than being attributed solely to stochastic decoding variation.Source: Table C.2 · p. 18, §C.4 · p. 19
  • PTD produces identical outputs across all three runs under the fixed 64-frame greedy configuration, establishing repeatability for that evaluated configuration and subset.Each role is inferred independently using the same 64 uniformly sampled frames.Source: §C.4 · p. 19, §C.5 · p. 19

C.5 PTD BASELINE AND INTERFACE ADAPTATION

  • PTD’s native role-specific interface omits the ordinal event index and full event-boundary specification, and decomposes multi-role queries into independent requests.The authors note this may affect 1-minute evaluation when several same-type events occur; it is immaterial in Oracle, where the target event is already cropped.Source: §C.5 · p. 19

C.6 CONTROLLED COMPARISON WITH THE PTD-STYLE QUERY AND OUTPUT ADAPTATION

The PTD-style query/output adaptation reduces Qwen’s 1-minute temporal and end-to-end grounding on repeated-event queries, while the GPT visual-detail setting shows no detected 1-minute performance reduction and enables complete 10-minute input. Additional analysis finds that longer contexts primarily impair selection of the requested occurrence.

C.6 CONTROLLED COMPARISON WITH THE PTD-STYLE QUERY AND OUTPUT ADAPTATION

  • At 1 minute, PTD-style adaptation lowers Qwen m-tIoU from 53.022% to 31.341% and m-vIoU from 4.214% to 2.184%, while Oracle m-vIoU changes from 9.448% to 8.532%.The paired Oracle difference is −0.917 pp (95% CI [−3.491, +1.910]); the temporal and m-vIoU differences are −21.682 pp and −2.031 pp, respectively.Source: §C.6 · p. 20
  • Under the PTD-style adaptation, PTD reaches 13.040% Oracle m-vIoU versus 8.532% for Qwen3.8, while 1-minute m-vIoU is similar at 2.104% versus 2.184%.At 1 minute, m-tIoU is 16.165% for PTD and 31.341% for Qwen3.8.Source: §C.6 · p. 20
  • The Qwen/PTD comparison jointly changes query content, role-wise versus joint output, representation, and visual sampling and decoding, so differences cannot be attributed solely to model capability.Qwen’s native video API and PTD’s 64-frame input and special-token output remain different.Source: §C.6 · p. 20

C.7 EFFECT OF GPT VISUAL detail=low

  • On a paired 1-minute subset, omitting detail=low showed no evidence of reduced performance; confidence intervals quantify uncertainty across sampled queries, not repeated decoding variance.The paired differences (omit minus low) were +0.176 pp m-tIoU (95% CI [−1.905, +2.224]), −0.190 pp m-vIoU ([−0.619, +0.249]), and −0.564 pp m-vIoU-Int ([−1.229, +0.086]).Source: §C.7 · p. 20
  • For 10-minute input, detail=low processes all 1,200 frames sampled at 2 FPS, whereas omitting the setting exceeds the tested provider context limit unless the input is shortened.The successful 500-frame omitted-detail request covers only the beginning of the video, motivating detail=low to preserve complete long-video coverage.Source: §C.7 · p. 20

D.1.1 SOCCER

  • From 1 to 10 minutes, target hit rates fall by 21.05–44.98 percentage points and same-type mismatch rates rise by 13.40–19.62 points; tIoU does not consistently decline among queries hit at every duration.The analysis covers 209 soccer queries with another qualified same-type occurrence in at least one context setting; all six hit-rate and mismatch changes remain significant after correction.Source: §D.1.1 · p. 21

D.1.2 MULTIDOMAIN EVALUATION

Multidomain results show that repeated-event selection remains difficult across all seven evaluated domains, while successful temporal retrieval does not ensure spatial role grounding. Complete grounding of all required roles is especially rare, even when temporal localization is strong.

D.1.2 MULTIDOMAIN EVALUATION

  • Across 500 multidomain queries, 106 / 104 / 88 predictions better match another same-type occurrence than the requested target, with such errors appearing in all seven evaluated domains.Later basketball occurrences also show lower mean tIoU and higher wrong-instance candidate rates, although later ordinal status covaries with video position, event difficulty, and available visual evidence.Source: §D.1.2 · p. 21

D.2 TEMPORAL RETRIEVAL DOES NOT ENSURE SPATIAL ROLE GROUNDING

  • Among multidomain queries hit temporally, mean GapSG is 67.52% / 71.42% / 74.43%, and remains 67.59% / 69.12% / 73.54% on the 206 queries jointly hit by all models.On 30 jointly hit League of Legends queries, mean gaps reach 84.43% / 95.73% / 98.41%.Source: §D.2 · p. 21
  • Even when all three models achieve tIoU > 0.5 on 26 multidomain multi-role queries, mean spatial gaps remain 78.13% / 91.07% / 88.08%, separating temporal retrieval from role grounding.A model may locate when an event occurs yet fail to identify who fills its semantic roles.Source: §D.2 · p. 22

D.3 INCOMPLETE MULTI-ROLE COMPOSITION

  • On 274 multidomain multi-role queries, complete-success rates are only 1.46% / 0.73% / 0.00%, and none of the 52 three-role queries achieves complete success for any model.In soccer, even with the ground-truth interval, only 22.35% / 3.35% / 3.91% of 179 two- or three-role queries succeed on every role.Source: §D.3 · p. 22

D.4 ROBUSTNESS CHECKS

Robustness checks support the repeated-event and multi-role findings under alternative diagnostic criteria, while the interpretation treats the observed failure paths as complementary rather than independent. The accompanying construction and annotation procedures define event windows, role trajectories, and occurrence ordinals using video-verifiable evidence.

D.4 ROBUSTNESS CHECKS

  • Soccer instance-mismatch increases remain significant under stricter definitions, and target misses overlap another same-type occurrence more often than a random-position reference: 23.91% / 54.93% / 38.05% versus 2.86% / 2.31% / 3.50%.Temporal degradation persists among queries with valid predicted windows at every duration, and complete role coverage remains harder than partial coverage across success thresholds.Source: §D.4 · p. 22

D.5 INTERPRETATION

  • The three error paths are complementary diagnostics, not independent probabilistic factors; GapSG combines several spatial-error sources and should not be assigned to one internal mechanism.The paper describes a cascade from selecting a repeated-event occurrence, to grounding its role fillers, to jointly localizing all required roles.Source: §D.5 · p. 22

E.1 ANNOTATION OVERVIEW

  • A semantic role is defined by its relation to the target event, and its entity identity must remain consistent across frames; unlocalizable frames are marked invisible rather than assigned to a nearby person.For example, “last passer” means the player who directly passes to the shooter for the target shot, not any player making a visible pass nearby.Source: §E.1 · p. 23

E.2.3 CONTEXT AND ORDINAL VERIFICATION

  • Occurrence ordinals are assigned within query-specific event sequences: the target is the n-th occurrence when exactly n−1 qualifying events precede it.A real-world event can count separately in different semantic-category sequences, and a verified qualifying occurrence may contribute even if it is not retained as a target annotation.Source: §E.2.3 · p. 25

E.3.1 TEMPORAL ANNOTATION

  • Soccer event intervals are manually set using event-specific semantics rather than inherited source timestamps, and all 3,516 final soccer targets have valid mappings to the full-match timeline.Some source events are split when Event-STVG semantics distinguish their component actions, such as a foul and the subsequent free kick.Source: §E.3.1 · p. 26

E.4 MULTIDOMAIN ANNOTATION

  • Across domains, annotation retains the event–role abstraction while adapting how spatial evidence is collected, including direct box drawing, existing trajectories, or frame-exact image sequences.Domain-specific rules distinguish the entities and event stages that count as targets, such as killer and victim in League of Legends kills and predator and prey in animal events.Source: §E.4 · p. 27, §E.4.2 · p. 27

E.5 QUALITY-CONTROL PROTOCOL

  • Automatic checks validate structure and media alignment, while human review checks semantic roles, identity consistency, event boundaries, and occurrence ordinals; these are complementary forms of quality evidence.The paired re-annotation measures consistency under the existing workflow, not independent double-blind annotation.Source: §E.5.3 · p. 28, §E.5.4 · p. 28, §E.5.5 · p. 28

E.6 ANNOTATION AGREEMENT AND FINAL AUDITS

The appendix reports paired annotation consistency and construction-level audits, then uses a sampled end-to-end review to check whether frozen multidomain instances jointly preserve temporal localization, spatial grounding, and occurrence ordinals.

E.6 ANNOTATION AGREEMENT AND FINAL AUDITS

  • Paired soccer annotations achieve mean temporal tIoU 0.904 and median 1.000; mean box IoU is 0.945 for jointly visible role–frame pairs and 0.919 when one-sided missing boxes count as zero.The latter evaluation includes 2.70% of role–frame pairs with a box on only one side.Source: §E.6 · p. 29
  • The multidomain end-to-end audit passes all 50 sampled instances, a 100% sample-level pass rate, while explicitly not re-estimating temporal or spatial annotation agreement.The audit jointly checks temporal localization, spatial role localization, and target ordinal correctness in the final video and query.Source: §E.6 · p. 30
  • Structural checks, paired re-annotation, operational guidelines, and end-to-end human review provide complementary quality evidence rather than substitutes for complete independent re-annotation.The multidomain media audit records a maximum source-to-output temporal mapping discrepancy of approximately 0.017 s.Source: §E.6 · p. 30

Related papers

Loading 2610.22444v1…