Source-linked AI summary
SVI-Bench: A Dynamic Microworld for Strategic Video Intelligence
Yulu Pan, Han Yi, Seongsu Ha, Md Mohaiminul Islam, Benjamin Zhang, Lorenzo Torresani, Gedas Bertasius
TL;DR
SVI-Bench addresses the lack of benchmarks for strategic video intelligence by evaluating perception, reasoning, simulation, and agency in real-world multi-agent video. It integrates multimodal sports data into nine tasks and finds competent perceptual performance but substantial degradation on higher-level reasoning, planning, and evidence integration.
Problem
Current video-language models can describe visible events but cannot reliably explain their causes, simulate alternatives, or recommend actions.
Method
SVI-Bench integrates five data sources through a data engine and evaluates the SVI stack across nine tasks organized into four progressive pillars.
Results
Models perform competently on perception but degrade substantially on higher-level reasoning tasks, with the strongest models reaching only 54% even given perfect visual information on the agentic task.
Takeaways & Limitations
The benchmark indicates that strategic video intelligence remains limited beyond perception, particularly in reasoning, planning, and multimodal evidence integration.
Takeaways & Limitations
SVI-Bench uses team sports as a controlled proxy, so sports-specific broadcast conventions and fixed rules may not transfer beyond sports.
Abstract
from arXiv · showhide
True video intelligence demands more than recognizing what is visible: it requires reasoning about why events unfold, predicting what would change under different conditions, and deciding what to do next. We refer to this progression, from perception through causal reasoning and simulation to strategic planning, as Strategic Video Intelligence (SVI). No existing benchmark evaluates this capability stack: in-the-wild videos lack verifiable ground truth for causal and strategic questions, while synthetic environments sacrifice the complexity of real multi-agent systems. To bridge this gap, we introduce SVI-Bench, a large-scale benchmark that leverages team sports as a dynamic microworld, combining the complexity of real-world multi-agent interaction (10-22 agents making coordinated decisions under adversarial pressure) with the verifiability of explicit rules and definitive outcomes. SVI-Bench comprises approximately 35K hours of broadcast video, 15M annotated actions, 15K hours of expert commentary, 23K game reports, and 103K structured statistical records across basketball, soccer, and hockey, all constructed via a data engine that transforms raw game data into a dense, cross-referenced corpus. We organize evaluation into 9 tasks spanning a progressive four-pillar hierarchy: Dynamic Scene Understanding, Causal Reasoning, Strategic Simulation, and Agentic Synthesis. Evaluating strong multimodal and agentic baselines, we find a capability cliff: models perform competently on perceptual tasks, achieving approximately 74% on fine-grained action QA, but degrade sharply at each successive cognitive level. Agentic tasks prove hardest: the strongest model achieves only 5% accuracy when required to autonomously gather and integrate evidence across a corpus of 1.8M clips.
1 Introduction
SVI-Bench frames Strategic Video Intelligence as an integrated progression from perception through causal reasoning, simulation, and agentic synthesis. It addresses the lack of benchmarks combining real multi-agent complexity with verifiable causal and strategic ground truth.
- Motivation: Strategic Video Intelligence integrates perception, causal reasoning, simulation, and agentic synthesis into a progressive cognitive stack.The stack spans parsing agents and actions, explaining outcomes, generating futures, and integrating multimodal evidence.
- Motivation: Existing benchmarks cover perception or temporal reasoning, but none evaluates the full stack from perception to agency.Synthetic datasets offer verifiable causal ground truth without real-world multi-agent complexity, while real-world benchmarks lack verifiable causal or strategic ground truth.
- Motivation: 10 to 22 players execute coordinated decisions under adversarial pressure, while scores, turnovers, and wins provide clear outcome labels.These properties make team sports a dynamic microworld for measuring long-horizon causality and strategic reasoning.
- Benchmark: SVI-Bench contains ∼35K hours of broadcast video, ∼15M annotated actions, ∼15K hours of expert commentary, ∼23K game reports, and ∼103K statistical records across three sports.Its data engine integrates these sources and organizes 9 tasks across four pillars.
- Contributions: The benchmark formalizes the full perception-to-agency stack with 9 tasks and reference methods that analyze where and why performance degrades.The four pillars are Dynamic Scene Understanding, Causal Reasoning, Strategic Simulation, and Agentic Synthesis.
2 Related Work
Prior video benchmarks largely emphasize perceptual or temporal reasoning, while causal benchmarks trade real-world richness for simplified settings. SVI-Bench is positioned as combining real multi-agent video, multimodal cross-referencing, and evaluation through strategic agency.
- Video Benchmarks: Early video benchmarks target action recognition, temporal localization, and language-grounded reasoning over short clips.Long-video benchmarks extend temporal scope but lack verifiable causal ground truth.
- Benchmark Positioning: SVI-Bench combines large-scale real-world multi-agent video with cross-referenced multimodal data and evaluation spanning perception through strategic agency.This combination distinguishes it from existing benchmark categories summarized in Table 1.
- Causal and Counterfactual Reasoning: Synthetic causal and counterfactual datasets provide verifiable ground truth but simplify scenes to single objects or low-complexity worlds.Real-video causal QA attempts causal reasoning but remains limited in scope.
3 Data Engine
SVI-Bench uses a data engine to transform grounded game sources into a dense, cross-referenced corpus. The pipeline aligns modalities, resolves entities, generates task instances, and applies automated and human quality control.
- Pipeline: Five raw sources are transformed into a cross-referenced corpus through temporal alignment, cross-modal entity resolution, LLM-based instance generation, and quality control.The four stages connect video, events, commentary, reports, and statistics into evaluable instances.
- Design Principles: The engine is built on primary human- or league-derived evidence, while LLMs scale task-instance generation from grounded sources.Manual verification is applied to a representative subset of every task.
- Corpus: SVI-Bench spans basketball, soccer, and hockey, selected for complementary team sizes, pacing, spatial scales, camera dynamics, and strategic structures.The corpus includes broadcast video, play-by-play logs, commentary, reports, and statistical records aligned through shared game and player identifiers.
- Pipeline: Temporal alignment uses game-clock timestamps and textual cues to link video clips with corresponding events, commentary, and statistical context.Entity resolution then organizes shared players, teams, and events into identity graphs containing relationships and attributes.
- Evaluation Structure: Nine tasks are organized into a four-pillar hierarchy from perception through causal reasoning and simulation to agency.Table 2 summarizes task coverage across basketball, hockey, and soccer and indicates available training splits.
4 The SVI-Bench Evaluation Suite
The evaluation suite contains 9 tasks across four pillars, beginning with structured perception of dense multi-agent scenes and extending toward causal and strategic capabilities. Foundational tasks test detailed action, identity, spatial, temporal, and contextual understanding.
- Suite Overview: Nine tasks are organized into four pillars spanning perception, causal reasoning, simulation, and agency.The hierarchy is designed to evaluate progressively higher cognitive capabilities.
- T1: Structured Play Description: T1 requires 10-second captions covering actions, player identities, spatial positioning, and game context for scenes with 10+ coordinated agents.Unlike standard captioning, it requires parallel sub-actions and higher-order game-state context.
- Dynamic Scene Understanding: The Dynamic Scene Understanding pillar evaluates structured play description, fine-grained action QA, and compositional video retrieval.These tasks establish the perceptual foundation for higher-level reasoning.
- T1: Structured Play Description: 1.61/5.00 is GPT-5.2’s average T1 score, compared with 1.67 for Gemini 3 Flash and 2.17 for fine-tuned LLaVA-Video-7B.Models perform relatively better on spatial understanding (2.74) and temporal understanding (2.72) than identity recognition (1.11) and causality/outcome reasoning (1.82).
- T2: Fine-Grained Action QA: T2 tests precise multi-agent interactions across action recognition, temporal ordering, play analysis, spatial relationships, player identification, and OCR.It uses 10-second clips, five candidate answers, and 31 question types across three sports.
- T2: Fine-Grained Action QA: 73.91% is fine-tuned LLaVA-Video-7B’s T2 accuracy, compared with 58.75% for Gemini 3 Flash and 52.91% for GPT-5.2.Player identification is most challenging, while action recognition is easier; sport-experienced humans reach 75.78% overall.
T3: Compositional Video Retrieval
T3 evaluates fine-grained compositional video retrieval by ranking one relevant video among 5,000 distractors, revealing substantial difficulty when candidates share similar visual content.
- T3 asks models to rank one positive video and 5,000 negatives using natural-language queries specifying entity, dynamics, context, and spatiotemporal structure.
- Queries are generated from ground-truth video attributes, paraphrased by an LLM, and paired with hard negatives to create challenging distractors.
- Recall@K is reported, with R@1 as the primary metric.
- 3.0% aggregate R@1 and 13.3% R@10 show that fine-grained retrieval remains difficult at scale.Increasing near-duplicate distractors nearly halves R@100, underscoring the challenge of distinguishing visually similar compositions.
T4: Strategic Reasoning QA
T4 tests strategic reasoning over full games, requiring models to explain causes and tactical intent rather than recognize isolated events. Models score near 2/5, while T5 forecasting remains below 45% even with fine-tuning.
- T4 uses 55–150-minute full-game videos and open-ended questions about strategic errors, tactical execution, and momentum shifts.
- 1,000 questions from expert commentaries and game reports cover 825 unique games across basketball, soccer, and hockey.
- An LLM judge scores T4 responses from 0–5 for factual consistency and reasoning coherence, with a 0.12-point mean human–judge difference.
- 2.17/5 is Gemini 3.1 Pro’s strongest overall T4 score, while GPT-5.2 leads basketball at 1.99.
- 44.82% is the fine-tuned Qwen3-VL-8B accuracy on T5, yet frontier GPT-5.2 confidence exceeds accuracy by 28 points.
T6: Long-Form Narrative Synthesis
T6 asks models to synthesize hour-scale visual narratives, exposing a sharp gap between factual correctness and coverage of salient events. Strategic simulation extends this challenge to generating alternative futures.
- T6: Long-Form Narrative Synthesis: T6 requires approximately 500-word reports grounded entirely in visual evidence from 55–150-minute games, covering events, performances, and strategy.
- T6: Long-Form Narrative Synthesis: Ten report templates per sport include single-game analysis and cross-game synthesis, with references generated from play-by-play logs, box scores, and journalist reports.
- T6: Long-Form Narrative Synthesis: T6 evaluation measures factual accuracy, saliency, and writing quality using LLM-based judging.
- T6: Long-Form Narrative Synthesis: 73.01% factual accuracy contrasts with 7.33% saliency, while most models exceed 4.5/5 in writing quality.
- T6: Long-Form Narrative Synthesis: T6 models reach 73.0% factual accuracy but only 7.33% saliency, indicating that causal and strategic reasoning remains a central bottleneck.
- 4.3 Pillar 3: Strategic Simulation (T7–T8): Strategic simulation evaluates whether models can generate alternative futures from short clips rather than merely explain observed events.
- 4.3 Pillar 3: Strategic Simulation (T7–T8): T7 generates scenes following prescribed player trajectories, whereas T8 generates goal-directed actions toward specified outcomes.
T7: Motion-Conditioned Generation
T7 evaluates whether models can generate coherent multi-agent video from initial scenes, erased-player backgrounds, and prescribed trajectories. The reference method outperforms off-the-shelf baselines, but substantial trajectory deviations remain.
- T7 conditions generation on an initial frame, player-removed background video, and time-aligned bounding-box trajectories for every player.
- Quality filtering removes instances with unstable tracking, severe occlusion, or visible inpainting artifacts.
- Video mIoU measures spatiotemporal trajectory alignment, while temporal feature similarity measures visual consistency across corresponding player regions.
- The reference Wan 2.1 model is fine-tuned on SVI-Bench with structured inputs, while ATI and MagicMotion are evaluated off the shelf.
- 0.513 basketball Video mIoU exceeds MagicMotion’s 0.466 and ATI’s 0.397; soccer reaches 0.611 versus 0.544 and 0.402.Temporal feature similarity follows the same ordering in both sports.
- T7’s trajectory-following setting differs from T8’s goal-conditioned setting, which replaces exact trajectories with textual goals and endpoint constraints.
4.4 Pillar 4: Agentic Synthesis (T9)
Pillar 4 evaluates whether agents can autonomously search heterogeneous video, report, and statistical sources, then integrate evidence to answer complex strategic queries. T9 is the hardest capability, with current systems achieving very low accuracy at corpus-scale multimodal reasoning.
- Task formulation: T9 requires agents to plan multi-step retrieval, gather multimodal evidence, and reason over it to produce a final answer.The agent searches video clips, game reports, and statistical records using document and video databases.
- Task formulation: T9 operates across ∼1.8M clips and ∼33K documents spanning three sports, requiring looping, backtracking, conditional branching, and numerical aggregation.Questions use seed facts and multi-hop narrative constraints to make relevant events hard to find but easy to verify.
- Evaluation and findings: 4.6% accuracy is achieved by GPT-5.2 across the three sports, compared with 2.1% for Qwen3-Omni-30B.GPT-5.2 makes roughly 21 tool calls per question, while smaller Qwen models make only 3–8 and often terminate early.
- Oracle analysis: Oracle textual descriptions produce the largest gains on T9, indicating that strategic reasoning, forecasting, saliency judgment, and multi-step planning remain distinct bottlenecks.The oracle experiment replaces video with ground-truth textual descriptions of game events.
- Evaluation and findings: Current agents cannot reliably plan retrieval, gather evidence, and reason over a corpus at this scale.The result identifies corpus-scale evidence integration as a major agentic bottleneck.
5 Cross-Task Analysis
Across tasks, models perform much better at perception than at reasoning, simulation, and planning. Oracle inputs and human comparisons show that higher-level failures are not explained by perception alone and that the evaluated tasks are solvable.
- 5.1 The Performance Cliff: Performance drops consistently from perception to agentic synthesis across models, and perception-level fine-tuning gains do not carry to higher pillars.The normalized pillar comparison indicates stronger dynamic-scene perception than reasoning, simulation, or planning.
- 5.2 Oracle Experiments: Oracle access raises T9 accuracy from 4.6% to 54.0%, while gains on T4 and T5 are smaller.Oracle mode replaces video with ground-truth textual descriptions derived from play-by-play logs.
- 5.2 Oracle Experiments: Oracle results indicate that strategic reasoning, forecasting, saliency judgment, and multi-step planning each limit performance independently.The reported changes vary sharply across T4, T5, T6, and T9 rather than identifying one universal bottleneck.
- 5.3 Human Studies: 73.9% versus 75.8% shows near-human model performance on T2 perception, whereas T4 reaches 2.17/5 versus 4.2/5 and T5 reaches 44.8% versus 58.9%.The comparisons are reported as best-model versus human performance for perception, strategic reasoning, and forecasting.
- 5.3 Human Studies: Humans calibrate confidence to accuracy, but GPT-5.2 shows a 28-point gap between average confidence and accuracy.Human accuracy rises across confidence levels, while GPT-5.2 reports similar confidence on correct and incorrect answers.
6 Conclusion
SVI-Bench shows that models handle perception more competently than higher-level reasoning, planning, and evidence integration. The benchmark is a sports-based proxy whose domain-specific conventions and LLM judging limit claims beyond its evaluated setting.
- Conclusion: Across 9 tasks and four pillars, models perform competently on perception but degrade substantially on higher-level reasoning tasks.The benchmark covers real-world multi-agent video from perception through agency.
- Conclusion: 54% is reached by the strongest models on the agentic task even with perfect visual information, extending the challenge beyond perception.The remaining challenge is reported in reasoning, planning, and evidence integration.
- Scope and limitations: SVI-Bench is a team-sports microworld and does not claim cross-domain generalization.Broadcast conventions and fixed rules may not transfer beyond sports.
- Scope and limitations: LLM judges are used for several tasks and, despite human-agreement validation, may retain some bias.This is an evaluation limitation rather than a reported model-performance result.
- Future directions: Future work targets visual grounding for long-form reasoning, explicit multi-agent dynamics, and multimodal agents that reason across corpora at scale.These directions correspond to gaps revealed in T4–T9.