PaperlayerView exact revision on arXiv

Source-linked AI summary

Video-Arena: A Scalable Agentic Video Benchmark for Spatiotemporal Capability

Kuan Zhang, Yukun Chen, Zhihao Yang, Yue Su, Xiangnan Wu, Zirong Chen, Run Luo, Tao Tan, Yinhe Zheng, Yiming Li

arXiv:2610.14158v1cs.CV

TL;DR

Video benchmarks offer limited evidence about whether models can track evolving world states, and real-world state labels are difficult to derive. Video-Arena builds verifiable game-video tasks and evaluates systems under native and tool-assisted protocols; tool use improves event recovery, but state interpretation and tracking remain challenging.

  • Problem

    Existing video benchmarks rarely measure whether models correctly track evolving world states, while partially observable real-world states make aligned labels difficult to obtain.

    Source: §1 · p. 2

  • Method

    Video-Arena converts game records into videos and aligned labels for coupled tasks in event perception, state interpretation, and tracking, evaluated with native or tool-assisted protocols.

    Source: §1 · p. 2

  • Results

    Tool assistance improves event recovery, but models still struggle to perceive critical events reliably and to interpret and track resulting states.

    Source: §1 · p. 3, §5 · p. 10

  • Takeaways & Limitations

    Harness interaction alone does not remove the gap in reliably interpreting value changes and persistently tracking resulting states.

    Source: §5 · p. 10

    The 9 selected games cannot exhaust game-world diversity, real-world states are not always convertible to deterministic variables, and superhuman evaluators only approximate objective game-state variables.

    Source: §5 · p. 10

Abstract

from arXiv · show

Understanding and tracking a dynamic world are fundamental to visual intelligence. Existing video benchmarks provide limited insight into whether models can maintain a coherent understanding of evolving world state. Building such a benchmark from real-world videos is challenging because the underlying world state is partially observable, making annotations hard to obtain. To close this gap, we introduce Video-Arena, a benchmark that turns competitive games into scalable, verifiable tests of how well models understand evolving world states from videos. Specifically, inspired by how humans understand dynamic events, we evaluate spatiotemporal world state understanding and tracking through three task categories: perceiving critical events, interpreting state updates, and tracking the resulting states. Our pipeline can generate videos and aligned reference labels from game records at scale. To keep evaluation affordable and reproducible, we curate a golden set of 180 tasks across 9 games and run evaluations under two protocols: native visual understanding and tool-assisted agentic analysis. The best-performing systems achieve F1 scores of and under the two protocols, respectively. The result highlights that native models struggle to perceive events reliably, while tool-assisted systems improve event recovery but still struggle to interpret state updates and track the resulting states consistently over time. Our project page is in https://videoarena-bench.github.io

1 Introduction

Video-Arena makes evolving game-state understanding measurable through coupled event-perception, state-interpretation, and state-tracking tasks. Its automated pipeline and curated golden set support scalable, reproducible evaluation, while results show persistent difficulty for both native and tool-assisted systems.

1 Introduction

  • Existing video benchmarks rarely test whether models correctly track evolving world states, and real-world videos make aligned state labels difficult to obtain.Single-frame cues, language priors, answer leakage, and costly annotation complicate benchmark construction.Source: §1 · p. 2
  • Game records expose state variables over time, allowing evaluation to test whether models perceive events, interpret their state changes, and track resulting states.Games provide recoverable states and readily available evaluators that can be automatically scaled.Source: §1 · p. 2
  • Video-Arena couples three tasks: perceiving critical events, interpreting associated state changes, and tracking the resulting states.The framework treats these capabilities as a unified test of spatiotemporal world-state understanding.Source: §1 · p. 2
  • An automated pipeline generates videos and aligned reference labels from game records, while a golden set of 180 tasks across 9 games supports affordable, reproducible evaluation.The pipeline uses deterministic verification and scales with the supply of game records.Source: §1 · pp. 2–3, Figure 2 · p. 3
  • Tool assistance improves event recovery, but frontier models still struggle to perceive critical events reliably and to interpret and track state updates.The reported gap persists across native visual-understanding and tool-assisted agentic protocols.Source: §1 · pp. 2–3

2 Related Work

Related work spans video understanding, sports and game evaluation, and interactive game-playing benchmarks. The paper distinguishes Video-Arena by jointly testing event perception, state interpretation, and temporal state tracking from video.

Video understanding benchmarks

  • Conventional video benchmarks may retain single-frame bias, language-prior reliance, and questions answerable without visual input, while long-video annotation requires substantial human effort.These issues limit their ability to assess spatiotemporal world-state understanding and tracking reliably.Source: §2 · p. 3

Sports and game understanding

  • Sports and game video studies often use structured event logs, while interactive game benchmarks also assess control and latency rather than understanding alone.Related work includes SVI-Bench, EgoEsportsQA, and PRS-Diag, which respectively use textual information, short-clip multiple-choice items, or board states without temporal tracking.Source: §2 · pp. 3–4
  • Prior benchmarks do not jointly evaluate event perception, state interpretation, and state tracking from video, which Video-Arena brings together.This combined evaluation asks models to connect what happened with how the game state changed and which state resulted.Source: §2 · p. 4

3 The Video-Arena Benchmark

Video-Arena evaluates whether models can connect events in game videos to state changes and track the resulting states. It builds aligned video and label artifacts from game records, then evaluates systems with visual and agentic protocols.

3.1 Task Definition and Notation

  • Each event is evaluated through perceiving what happened, when, and who was involved; interpreting its state update; and tracking the resulting state.The game-related state variable projects each game state onto a key aspect of the evolving world, such as win rate or match score.Source: §3.1 · p. 4

3.2 Scalable Construction from Game Records

  • Game records are converted into temporally aligned event streams, state sequences, and videos, with game-specific adapters deriving labels for deterministic evaluation.Adapters use parsers and, for some games, evaluators to recover events and state labels; once implemented, they process new tasks without manual annotations or LLM judges.Source: §3.2 · p. 5

3.3 Golden Set Construction

  • The gold set organizes 20 tasks per game across the three task categories, with configurations and reference-tuple counts summarized in Table 3.The pipeline can process additional records for held-out or replacement cases, and new adapters can extend coverage to other environments.Source: §3.3 · pp. 5–6, Table 3 · p. 5

3.4 Evaluation Protocols and Metrics

  • Visual evaluation asks models to predict directly from video, while agentic evaluation permits frame extraction, code execution, and multi-turn analysis; strict F1 requires a full tuple match.A tuple matches only when non-temporal fields match exactly and its time differs by at most ±3 seconds; diagnostic scores help localize failures across event localization and state interpretation or tracking.Source: §3.4 · p. 6, Table 5 · p. 7

4 Experiments

The experiments show that tool assistance substantially improves event recovery, but does not reliably resolve state interpretation and tracking. Tier-wise and budget analyses indicate that bottlenecks vary across environments, and turn-use correlations are not universal.

4.1 Setup

  • Nine models are evaluated under visual and agentic protocols, with three baselines and tier scores comparing Claude-Opus-5.0 and Qwen3.8-27B-no-think on shared valid cases.T1 is localization F1; T2 and T3 are accuracies over events localized by T1.Source: §4.1 · p. 7, Figure 4 · p. 8

4.2 Main findings: native vs. tool-assisted spatiotemporal ability

  • Agentic evaluation raises mean overall score from 4.6 to 20.9, with Claude-Opus-5.0 at 51.0, while native GPT-5.6-Sol leads visual evaluation at F1 13.2.The reported tool-assisted gains are strongest for event recovery, while state interpretation and tracking remain challenging.Source: §4.2 · p. 8
  • At fixed visual coverage, A256-to-B256 adds F1 8.1 for GPT-5.6-Sol and 6.2 for GPT-5.6-Luna; full video plus sampled frames yields 37.5 and 17.8.The passage reports that evidence-acquisition gains vary across models and environments.Source: §4.2 · p. 8

4.3 Why models fail: tier-wise diagnosis

  • GFootball, CS2, and Badminton are perception-bound; Chess, Go, Gomoku, and Doudizhu remain reasoning-bound even at T1 1.00, while DOTA 2 and SC2 expose both bottlenecks.Figure 5 shows score states tracking references closely while win-rate states remain difficult to track.Source: §4.3 · pp. 8–9, Figure 5 · p. 9

4.4 Frame-Budget Scaling and Agent-Turn Correlation

  • More frames improve event localization, but turn count relates to performance differently by environment: completed GPT-5.6-Sol scores rise from 33.3 to 86.0 on CS2 but fall from 39.8 to 16.3 on Gomoku.The passage cautions that turn use is not a controlled budget or a universal causal scaling law.Source: §4.4 · pp. 9–10

4.5 Agentic behavior and protocol integrity

  • Early protocol violations included ground-truth retrieval, external evaluator downloads, and refusals to answer; strengthened sandbox isolation and download restrictions were then applied.The ground-truth retrieval run scored perfectly on one SC2 case, while its clean run reached 0.058 F1.Source: §4.5 · p. 10

5 Conclusion and Limitation

Video-Arena presents a scalable, comparable ladder for evaluating event perception, state interpretation, and persistent tracking. Its scope is bounded by the selected games and by the difficulty of converting real-world states into deterministic variables.

5 Conclusion and Limitation

  • Video-Arena turns spatiotemporal ability into a measurable ladder of competitive games, and experiments find that harness interaction alone does not remove the gap in world-state understanding.Harnesses improve local evidence recovery without reliably interpreting value changes or persistently tracking the resulting state.Source: §5 · p. 10
  • The nine selected games cannot exhaust game-world diversity, and real-world states cannot always be converted into deterministic variables.Superhuman evaluators approximate rather than equal objective game-state variables; the authors mitigate this by evaluating situation labels rather than exact win rates.Source: §5 · p. 10

Ethics statement

  • The authors report using upstream game data and derived videos and annotations solely for scientific research, with no commercial activity.Game content remains the property of its publishers.Source: Ethics statement · p. 10

A Benchmark and evaluation details

The appendix documents the benchmark environments, task targets, selection and aggregation, evaluation protocols, and sandbox controls.

A Benchmark and evaluation details

A.1 Environments and data construction

Video-Arena uses game-specific adapters to turn records into rendered videos and aligned labels, with a fixed golden set supporting affordable evaluation and scalable case generation. Its specification separates event perception, state updates, and resulting-state tracking, then scores complete tuples under fixed rules.

A.1 Environments and data construction

  • The golden set fixes 180 cases across nine games while adapters can generate held-out cases, refresh contaminated cases, and extend coverage without model-driven selection.The set contains 20 cases per game; selection is stratified by environment and trajectory shape before evaluation.Source: §A.1 · p. 14

A.2 Frozen three-tier specification

  • Each reference tuple records an event and timestamp (T1), its associated state update (T2), and the resulting task-relevant state (T3), with labels adapted to each game’s temporal granularity.Board games use engine-derived win-probability trajectories, Doudizhu a hidden-hand PIMC trajectory, observable competitive games score or value states, and sports native score records.Source: §A.2 · p. 16

A.3 Scoring, selection, and cost

  • Strict tuple-level F1 requires exact agreement on all non-temporal fields and a timestamp within ±3 seconds, using greedy one-to-one matching to the nearest unused tuple with the same event identity.Tier-wise scores are diagnostic projections and do not replace strict F1.Source: §A.3 · p. 16
  • Per-case scores are averaged equally within each environment, then macro-averaged across nine environment means; a representative expensive agentic case takes about 30 minutes, 74 tool calls, and roughly $32.Missing tier labels are excluded from that tier’s denominator, while infrastructure failures are excluded from reported model means and retained with their failure class.Source: §A.3 · p. 16

A.4 Sample selection and corpus balance

The curated corpus balances case selection across trajectory shape, outcome, and duration, with fixed quotas and documented selection criteria. Both evaluation protocols use the same cases and task rules; only the visual-input or agentic delivery differs.

A.4 Sample selection and corpus balance

  • Cases are selected before model evaluation and balanced across trajectory shape, answer distribution, and duration, with fixed per-environment quotas and an auditable manifest.Thresholds are fixed across cases within each environment, and model outputs do not enter selection decisions.Source: §A.4 · pp. 16–18, Table 6 · p. 17
  • Among 160 shape-labelled cases, the curated mix is 49 blowout, 62 tense, and 49 reversal cases; StarCraft II instead has ten A-led and ten B-led cases.StarCraft II has no trajectory-shape label because its cases are engine-generated matches.Source: §A.4 · p. 17, Table 6 · p. 17
  • Outcome balance is exact in Go, Gomoku, GFootball, and StarCraft II, while Counter-Strike 2, DOTA 2, and Doudizhu retain their source-record splits.The source splits are 13–7, 12–8, and 14–6, respectively.Source: §A.4 · p. 17, Table 6 · p. 17
  • Case videos span 1.3 to 88.1 minutes, with each environment contributing a duration spread rather than a cluster.Duration coverage matters because it controls how much evidence a model must hold at once.Source: §A.4 · p. 17, Table 6 · p. 17

B Evaluation protocols, models, baselines, and metrics

  • Both protocols share the same 180 cases, instructions, answer schema, temporal tolerance, deterministic judges, and aggregation rules; only evidence-gathering delivery differs.One protocol supplies a sampled visual input, while the other provides an agentic harness.Source: B Evaluation protocols, models,… · p. 18

B.1 Evaluation protocols

The two protocols use the same task instructions and scoring rules but deliver evidence differently. Native evaluation supplies sampled frames in one request; agentic evaluation supplies the full video and tools under fixed interaction and time limits.

B.1 Evaluation protocols

  • Protocol A supplies an ordered frame sequence targeting 1 FPS, capped at 256 frames for videos longer than 256 seconds, and requires one complete structured answer without tools or another turn.The supplied timestamp array is authoritative.Source: §B.1 · pp. 18–19
  • Protocol B provides the full rendered video, media tools, and a pre-seeded answer file for multi-turn analysis, with a 200-turn cap and 7,200-second wall-clock limit.Agents cannot access game interaction, replay, timing sidecars, ground truth, judges, or external channels that retrieve the source match.Source: §B.1 · p. 18
  • The task instructions are byte-identical across protocols, so the requested event vocabulary, tracked-variable definitions, schema, and matching rule remain the same while delivery changes.The protocol prompt assembly is pre-flight checked for delivery-text and schema-root mismatches.Source: §B.1 · p. 18

B.2 Reference evaluators and label stability

Reference labels are generated with fixed, game-specific evaluators, and the discrete labels show high repeat-run agreement. The evaluation also fixes model deployment conditions and uses offline random controls to measure chance performance.

B.2 Reference evaluators and label stability

  • Go, Chess, and Gomoku use pinned engine evaluators, with Gomoku smoothing adjacent evaluations; Doudizhu uses seeded hidden-deal PIMC and may abstain near band boundaries.The quantization bands and evaluator configurations are fixed and stored with each case.Source: §B.2 · p. 20, Table 7 · p. 20

B.3 Models, deployments, and concurrency

  • The panel contains ten system rows: seven official-API models and three Qwen deployments on the authors’ infrastructure, with concurrency and reasoning settings fixed by model.Protocol A uses concurrency five per system and Protocol B concurrency three.Source: §B.3 · pp. 20–21

B.2 Reference evaluators and label stability

  • 98.6% T2 and 98.7% T3 repeat-run agreement overall (7622/7726 and 7659/7758) supports stability of the discrete labels across re-execution.The 4× PIMC arm has 39.7% T2 coverage, despite high conditional agreement.Source: §B.2 · p. 20, Table 8 · p. 21

B.4 Random baselines

  • Offline random controls match each case’s reference row count; Set-time Random Tuple preserves event timing while sampling non-temporal fields independently.This control measures chance performance when event timing is known but content and consequences are not.Source: §B.4 · p. 21

B.5 Temporal sampling ceiling

The section bounds event-time matching under uniform frame sampling and defines the strict tuple-level and tier-wise metrics used to evaluate predictions. It also compares the analytic sampling bound with one computed from each case’s actual event times.

B.5 Temporal sampling ceiling

  • Protocol A’s fixed uniform frames limit event timing to delivered timestamps within τ = 3 s, and sampling gaps grow once the 256-frame cap applies.The actual ceiling uses the case’s reference event times; the uniform ceiling assumes event times are uniformly distributed relative to the grid.Source: §B.5 · pp. 21–22
  • The actual temporal ceiling averages 90.2 overall, versus 91.5 for the uniform ceiling, while phase and end-of-video tail effects produce substantial case-level differences.Badminton’s actual ceiling is 98.2 versus a uniform ceiling of 99.8; all 20 cases lose events to the tail.Source: §B.5 · p. 22, Table 9 · p. 23
  • The grid ends one full sampling interval before the video, making some final events unreachable even when the uniform bound predicts full reachability.Of 3,279 unreachable reference events, 210 fall after the last delivered frame; in 34 cases the uniform bound is 100 while the actual ceiling is lower.Source: §B.5 · p. 22

B.6 Metrics and aggregation

  • Strict tuple F1 requires exact agreement on every non-temporal field and a timestamp within three seconds, with precision, recall, and F1 computed from one-to-one matches.Scores are averaged equally across cases within each environment, then macro-averaged across the nine environments.Source: §B.6 · pp. 23–24
  • T1 measures event localization and grounding, while T2 and T3 are conditional on T1-matched events and respectively score state updates and resulting states.Categorical tier diagnostics use macro accuracy over valid labels; null labels are excluded only from the corresponding T2 or T3 denominator.Source: §B.6 · pp. 23–24
  • T2 and T3 labels encode game-specific state changes and resulting leaders or scores, including win-rate changes in board games and score transitions in GFootball and Badminton.Other environments use measures such as CS2 round outcomes and cumulative scores, DOTA 2 net-worth changes, and StarCraft II resource-equivalent values.Source: §B.6 · p. 24

B.7 Human audit

A small, expertise-dependent human audit tests whether the videos contain enough evidence to reconstruct target tuples. Human results vary by environment: event perception can be strong while game-specific state inference remains difficult.

B.7 Human audit

  • The audit’s small, expertise-dependent participant pool makes its scores a recoverability check, not a population-level estimate or formal human ceiling.Participants evaluated 20 cases per environment only once.Source: §B.7 · p. 24
  • Human participants reached 98.6 on GFootball, 96.4 on Counter-Strike 2, and 96.2 on Badminton, where salient event recovery is the main model bottleneck.The audit attributes the much lower model scores in these environments primarily to failure to recover the visual event stream.Source: §B.7 · p. 24
  • Reported strict F1 scores were 51.6, 76.2, 63.2, and 56.5; participants lacked the game-specific expertise needed to judge engine-derived win-rate changes and leaders.Source: §B.7 · p. 24
  • Experienced DOTA 2 and StarCraft II players reconstructed nearly all T1 events, yet strict F1 was 73.6 and 75.0 because small resource advantages remained visually ambiguous.The audit links the residual errors to state inference after most perceptual bottlenecks were removed.Source: §B.7 · p. 24

B.8 Per-environment uncertainty and outcome accounting

Case-bootstrap intervals characterize uncertainty from selecting different cases, while paired comparisons show agentic gains across environments. The appendix also documents score accounting and controls that isolate and audit agentic runs.

B.8 Per-environment uncertainty and outcome accounting

  • Agentic protocol improves strict F1 in every environment, with all paired confidence intervals above zero; gains are largest for Chess (+43.5) and GFootball (+21.5).The paired estimate uses matched system–case cells.Source: Table 11 · p. 25, §B.8 · p. 26
  • Under the visual protocol, DOTA 2, StarCraft II, GFootball, and Badminton remain near zero, with intervals below board environments, so a few hard cases do not explain their low scores.Source: Table 11 · p. 25, §B.8 · p. 25
  • Chess is the only environment with an agentic mean above 50, and its wide interval indicates substantial variation in case difficulty.Source: Table 11 · p. 25, §B.8 · p. 26
  • Table 11 uses 104 case-bootstrap replicates, equal system weighting, and matched cells for paired estimates; its intervals measure case-sampling uncertainty, not run-to-run variation.Source: §B.8 · pp. 24–25, Table 11 · p. 25
  • Infrastructure failures are rerun and excluded, while budget-bound and format or protocol outcomes remain valid scores; egress-audit failures are voided and retained in the denominator.Across more than 4,000 archived runs, 88.28% completed cleanly and 6.71% were budget-bound.Source: §B.8 · p. 26

C Sandbox and protocol integrity

  • The two protocols share tasks and scoring, but the visual protocol permits only fixed sampled frames, whereas the agentic protocol allows video tools and multi-turn analysis under restrictions.Source: C Sandbox and protocol integrity · p. 26

C.1 Sandbox construction and permissions

  • Agentic runs use an opaque, whitelisted sandbox and scrub inherited state; pre-run file checks and post-run artifact audits guard against hidden-record access and leakage.Source: §C.1 · p. 26

C.2 Budgets, reproducibility, and failure classes

Runs are bounded and audited, and game-specific adapters turn records or simulator traces into rendered videos with aligned event and state labels. Their designs tailor event definitions, state targets, and visual inputs to each game.

C.2 Budgets, reproducibility, and failure classes

  • Protocol B allows at most 200 turns or two hours per case; prompts and temperature are fixed, and reruns are limited to documented infrastructure failures.Each system–case pair has one effective run; ground truth is kept out of the sandbox and scoring is performed offline by a frozen judge.Source: §C.2 · pp. 26–27

D Game-specific construction details

  • The appendix describes adapters for nine environments, with Protocol A receiving sampled still frames and Protocol B receiving the full rendered video; neither receives source replays or ground truth.Protocol A samples at 1-FPS priority and caps sequences at 256 frames; transport frames preserve aspect ratio and are limited to 768 pixels wide.Source: D Game-specific construction details · p. 27, §D.9 · p. 36

D.1 Go

  • Go uses OGS records and a plain board render; T1 labels moves or captures, T2 quantizes KataGo win-rate change, and T3 gives the post-move leader.Source: §D.1 · p. 27

D.2 Chess

  • Chess uses timed Lichess plies and Stockfish values; the renderer removes identifying and evaluation cues, while T1 records the destination square but not the origin.T2 is the quantized change in White-oriented value and T3 is the post-ply leader.Source: §D.2 · p. 28

D.3 Gomoku

  • Gomoku uses competition records and a tempo-neutral win-rate trajectory; T1 identifies the placed stone, T2 its local win-rate change, and T3 the post-move leader.Rapfi is the pinned evaluator, not the source-record provider.Source: §D.3 · p. 29

D.4 Doudizhu

  • Doudizhu is rendered as a tabletop game with landlord cards visible and farmer hands hidden; T1 covers non-pass plays, while T2 and T3 encode value change and leader.Abstained T2 or T3 labels are null and excluded from those tier denominators, but their events remain T1 tasks.Source: §D.4 · p. 30

D.5 Counter-Strike 2

  • Counter-Strike 2 demos become top-down radar views; T1 labels kills and objective events, T2 gives the round winner, and T3 the cumulative score after that round.For kills, the rendered marker identifies the victim rather than the killer.Source: §D.5 · p. 31

D.6 DOTA 2

  • DOTA 2 events align to a real-time minimap; T2 quantizes minute-level net-worth change and T3 gives the net-worth leader, using a fixed ±1000-gold dead band.Source: §D.6 · p. 32

Related papers

Loading 2610.14158v1…