PaperlayerView exact revision on arXiv

Source-linked AI summary

Benchmarking Visual State Tracking in Multimodal Video Understanding

Sihyun Yu, Nanye Ma, Pinzhi Huang, Hyunseok Lee, Shusheng Yang, June Suk Choi, Ellis Brown, Oscar Michel, Boyang Zheng, Jinwoo Shin, Saining Xie

arXiv:2606.03920v1cs.CV

TL;DR

Existing evaluations leave continuous visual state tracking in MLLMs underexplored, despite its importance for understanding procedural video. The paper introduces VSTAT, a benchmark requiring integration across entire video streams, and finds a large human–MLLM gap driven primarily by visual perception. It further shows that current agentic approaches do not readily resolve these failures.

  • Problem

    Existing video benchmarks often permit answers from keyframes or end states, leaving continuous tracking of evolving states in MLLMs insufficiently evaluated.

  • Method

    The paper introduces VSTAT, comprising 834 video clips and 1,500 questions designed to require continuous perception and state integration across synthetic and real-world videos.

  • Results

    State-of-the-art MLLMs perform far below humans and only modestly above answer-prior baselines, while text-transcription tests indicate visual perception is the primary bottleneck.

  • Takeaways & Limitations

    VSTAT provides a diagnostic benchmark for understanding and improving MLLM visual perception on continuous, real-world video streams.

  • Takeaways & Limitations

    Text transcription is infeasible for many real-world VSTAT videos because descriptions can require substantially more information and exceed video length.

Abstract

from arXiv · show

Understanding a video requires more than recognizing isolated moments, as humans continuously track entities, states, and events over time. This capacity for visual state tracking is fundamental to video understanding, yet remains underexplored in current evaluations of Multimodal Large Language Models (MLLMs). We introduce Visual STAte Tracking benchmark (VSTAT), a video-based benchmark designed to diagnose visual state tracking in MLLMs. VSTAT consists of 834 clips drawn from both synthetic and real-world videos, paired with 1,500 questions that cannot be answered from any single frame or short segment, requiring continuous perception and integration of events across the entire video stream. Despite their strong performance on existing video benchmarks, we find that state-of-the-art MLLMs perform far below humans and only modestly above answer-prior baselines. To analyze this gap, we compare MLLMs' thinking traces with the underlying video stream to understand why and when MLLMs fail on VSTAT. We find that MLLMs reason and track correctly in text, but fail at visually perceiving the events they need to track. Finally, our preliminary evaluation suggests that recent agentic approaches, including MLLM-based video agents and coding agents, do not readily resolve these failures, still falling short on VSTAT.

1. Introduction

Visual state tracking requires continuously integrating evolving events, but existing video benchmarks often permit answers from keyframes or end states. VSTAT exposes this gap and finds that MLLM failures primarily reflect visual perception bottlenecks rather than reasoning failures.

  • Motivation: Existing video benchmarks often allow answers from keyframes, salient moments, or visible end states instead of requiring continuous state tracking.This leaves unclear whether MLLMs can track evolving states throughout a procedure.
  • Benchmark: VSTAT contains procedural videos and questions whose answers require integrating hidden, similar, or distributed events across the entire stream.Its tasks include tracking entities through occlusion and varying state complexity.
  • Findings: State-of-the-art MLLMs perform far below humans and only modestly above answer-prior baselines on VSTAT.The benchmark is designed to reveal this gap in visual state tracking.
  • Failure analysis: Temporally stretching events produces only marginal improvement, suggesting frame subsampling is not the main bottleneck.The experiment tests whether brief events are missed because models sparsely sample video frames.
  • Failure analysis: MLLMs solve manually transcribed frame-and-event descriptions almost perfectly despite struggling with the corresponding videos, implicating visual perception over reasoning.Such transcription is infeasible for most complex VSTAT tasks because it requires substantially more information.
  • Failure analysis: Failure traces reveal problems with event recognition, entity association across frames, and state updates after perceived events.Agentic video and coding frameworks do not readily mitigate these failures, leaving a substantial performance gap.

2. VSTAT: Visual State Tracking Benchmark

VSTAT is constructed to test continuous visual state tracking across diverse synthetic and real-world procedural videos. Its questions avoid visual shortcuts and vary both the tracked element and the structure of the required state.

  • Video curation: VSTAT includes 834 clips and 1,500 questions from Blender simulations, YouTube videos, and recordings, spanning tasks such as counting, word recognition, and shot attribution.The collection includes 450 simulated clips, 304 YouTube clips, and 80 self-recorded clips.
  • Question-answer generation: Each question prevents shortcut answers from sparse keyframes or visible end states, requiring models to maintain and update state across the entire video.The shell game requires continuously tracking a hidden object as cups are shuffled.
  • Question-answer generation: Queries target locations, counts, or attributes and require atomic, sequence, set, or dictionary state structures.The one-video, multiple-question format varies the information required and supports comprehensive analysis of tracking ability.
  • Taxonomy: The benchmark labels each video-question pair by perceptual complexity and state complexity to balance coverage across tracking challenges.Perceptual categories include occlusion, camera motion, homogeneity, symbolic decoding, and multi-entity attribution.

3. Evaluation on VSTAT

VSTAT exposes a large gap between human visual state tracking and current MLLMs, which generally remain near answer-prior baselines. Controlled analyses indicate that failures arise primarily from visual perception of events rather than textual reasoning, and agentic methods do not readily resolve them.

  • Main results: 90.5% human average accuracy far exceeds current MLLMs and the Chance-Level (Frequency) baseline on VSTAT.Only Gemini-3.1 Pro and Gemini-3.0 Flash are modestly above the answer-prior baseline; other models perform even worse.
  • Main results: All open-sourced models perform worse than Chance-Level (Frequency) across frame budgets, with model-size increases yielding usually marginal gains.Molmo2 and InternVL3.5 show slight degradation as model size increases.
  • Main results: Motion-grounded or pixel-tracking training does not substantially improve LLaVA-OV-2 or Molmo2, indicating that VSTAT requires tracking latent states evolving across video.These are the two best-performing open-source models, yet they do not substantially outperform other open-source MLLMs.
  • Main results: Higher thinking levels hurt performance, including Gemini-3.0-Flash falling from 39.8 to 38.8 and Qwen3VL-8B declining from 33.2 to 28.2.Gemini-3.1-Pro changes from 44.4 to 43.9, while InternVL3.5-8B moves from 30.6 to 30.2.
  • Why models fail: Video stretching produces only marginal improvement, ruling out frame-subsampling ambiguity as the primary explanation for the performance gap.The stretched Blender videos make every event fully visible under the 1 FPS sampling rate, yet performance remains limited.
  • Why models fail: Text transcriptions yield perfect performance, whereas video performance reaches random-guess level beyond 10 seconds, identifying visual perception as the main bottleneck.The gap is already substantial on 5-second videos and worsens with longer videos.
  • Failure modes: More than 50% of failures stem from event recognition, while other recurring failures involve entity association and state updates.State-update errors correlate highly with textual reasoning capability and remain relatively limited.
  • Agentic evaluation: Agentic methods remain near chance-level on VSTAT despite strong text-task performance, and coding agents take approximately 30 minutes per question on average.The result further indicates that visual perception remains the primary bottleneck for these approaches.

4. Related Work

The paper situates VSTAT among video MLLM benchmarks, video world models, and concurrent state-tracking evaluations. It distinguishes VSTAT through its broader coverage and lack of explicit visual shortcuts.

  • Video MLLM benchmarks evaluate general understanding, event recognition, knowledge reasoning, temporal grounding, and increasingly long-video comprehension.
  • VSTAT differs from VET-bench by covering substantially more clips and more than two simulated shell-game tasks.VET-bench contains 100 clips, whereas VSTAT is described as an order of magnitude larger.
  • Unlike VideoReasonBench, VSTAT includes real-world videos without explicit visual cues that reveal the underlying events.
  • Video world models generally predict future states from prior states, actions, and observations, whereas VSTAT evaluates visual state tracking from video-question inputs.

5. Conclusion

VSTAT diagnoses visual state tracking in MLLMs and exposes a substantial human–model gap. Its analyses identify visual perception as the primary bottleneck, while existing agentic frameworks do not trivially resolve the failures.

  • VSTAT reveals a substantial gap between human performance and current MLLMs, which only modestly exceed answer-prior baselines.
  • Controlled analyses identify visual perception, rather than textual tracking, as the primary bottleneck on VSTAT.
  • Video agents and coding agents do not trivially resolve the benchmark’s visual state-tracking failures.
  • The authors position VSTAT as a diagnostic tool for improving MLLM visual perception on continuous, real-world video streams.

A. Benchmark Breakdown

VSTAT organizes diverse synthetic and real-world procedural videos into taxonomies of state structures, perceptual challenges, and task types. Its construction varies tracking complexity, video sources, and question requirements.

  • Taxonomy: VSTAT’s taxonomy covers state structures and perceptual challenges used to label diverse queries and tasks.
  • Statistics: The benchmark balances element types and state structures while including perceptual challenges such as action ambiguity and camera motion.
  • Video sources: VSTAT combines Blender-rendered simulated tasks with real-world tasks recorded by the authors or curated from YouTube.
  • Question design: Question designs require different minimum information amounts, including whole-video counts and tracking among visually identical entities.
  • Question design: MCQ distractors represent plausible alternative states caused by common tracking errors rather than unrelated semantic answers.

B. Evaluation Setup Details

The evaluation measures human performance under controlled viewing conditions and compares models against random and frequency-based chance baselines. Accuracy is reported with task-specific threshold-based MRA.

  • Human evaluation: Human participants could replay videos and think freely but provided only one answer per question without seeing ground truth.
  • Baselines: Chance-level evaluation includes random guessing for MCQs and a frequency-based predictor for MCQs and numerical answers.
  • Metrics: Threshold-based MRA uses θ∈{0.5, 0.55, . . . , 0.95} and is computed separately for each question type.

C.1. Results across video sources

The section presents results and failure examples across Blender, recorded, and YouTube video sources, showing that models can reason from transcribed video content while visual perception failures remain evident.

  • Results across video sources: The evaluation reports VSTAT performance decomposed across Blender, Recorded, and YouTube video sources.Scores use the reparsed MRA-with-MCQ metric, with rankings reported separately for proprietary and open-sourced models.
  • Text transcription results: Gemini-3.1 Pro can perform tracking and reasoning near perfectly on full text transcriptions of the rolling die, shell game, and tilt box tasks.The transcription results are shown with reasoning traces in Figures 9–11.
  • Analysis coverage: The analysis samples 70 questions spanning video content, state elements, and question structures to cover the benchmark’s failure-case diversity.Questions include tracking die faces, cup positions, keypad presses, box order, and tile locations.
  • Failure analysis: Additional examples attribute failures to event recognition when models misinterpret reflections, occlusions, or missed object reveals.Examples include mistaking a reflected ball for a goal, inferring an incorrect click under occlusion, and missing the seventh doll’s reveal at 00:50.
  • Failure analysis: Additional examples attribute failures to entity association when camera rotation, box movement, or masked tile movement disrupts tracking.These changes lead to incorrect associations or hallucinated entity identities.

C.4. Comparison between different Thinking Levels

The section compares different thinking levels through examples of event-recognition and entity-association errors, including errors that persist or worsen with higher thinking levels.

  • Event recognition: Higher thinking levels still produce perceptual errors when identifying a made shot and player appearances.The comparison categorizes this example as an event-recognition failure.
  • Tracking example: The low-level trace tracks recurring landmarks and concludes that the ride does not complete a third full rotation before the video ends.It cross-checks the purple dragon sign with a building and estimates completed rotations around 00:25 and 00:50.
  • Entity association: Higher thinking levels miss the dragon sign’s first reappearance during the ride.The example concerns counting complete rounds using recurring visual landmarks.
  • Entity association: Higher thinking levels double count the same espresso cups after a shot change.The comparison categorizes this example as an entity-association failure.

C.5. Agentic framework details

The section evaluates agentic video and coding frameworks under contamination controls and finds failure modes involving event recognition, entity association, and state updates.

  • Evaluation setup: The agentic evaluation uses 39 video-question pairs, selecting one randomly chosen question and video from each category.Accuracy is computed for multiple-choice questions and relative accuracy for numerical answers, with values averaged over questions.
  • Contamination risk: Coding-agent evaluation is vulnerable to contamination because agents can search local files for answers, reaching about 87% without reasoning over video frames.The paper presents this as a contamination example rather than valid visual understanding.
  • Evaluation controls: The evaluation harness isolates each run in a temporary directory containing only a randomized copy of input.mp4 and instruction.txt.The setup also scrubs environment variables, restricts filesystem and network access, closes standard input, and adds prompt-level prohibitions.
  • Evaluation controls: Post-hoc audits found zero successful exploitation attempts across reported runs.Audits checked self-recognition, filesystem walks, network calls, cache access, environment dumps, benchmark metadata, and original filenames.
  • Agent failure modes: Claude Code misidentifies player #14 from a standard Kentucky Derby saddle towel, while AVP fails to track that player throughout the video.The example is categorized as event recognition for Claude Code and entity association for AVP.
  • Agent failure modes: Both agentic methods identify events but oversimplify them, producing incorrect state updates.The example isolates state update as a distinct failure mode from event recognition and entity association.

D. Limitations and Future Directions

The paper identifies limitations in its thinking-trace analysis, restricts VSTAT to current video lengths, and warns that benchmark performance is necessary but not sufficient evidence of progress.

  • Analysis limitations: The analysis relies on textual thinking traces because no established practice exists for interpreting MLLMs’ visual processing.The paper proposes vision-centric analyses of intermediate visual representations as a future direction.
  • Future directions: Future pre-training and post-training methods should directly target the perceptual bottlenecks revealed by VSTAT.The paper focuses here on demonstrating and analyzing failures rather than developing those improvements.
  • Scope boundary: VSTAT excludes extremely long video streams because visual state tracking is already challenging at the benchmark’s current video lengths.The paper identifies full sports matches and hour-level scenarios as future extensions.
  • Broader impact: VSTAT performance is a necessary but not sufficient progress indicator because models may overfit to the benchmark’s specific patterns.The paper recommends complementary evaluation on diverse out-of-distribution settings and existing benchmarks.

E. Compute Usage

The benchmark’s synthetic-video generation and model evaluation used substantial heterogeneous compute resources, while each process remained under four GPU-days.

  • Synthetic videos were generated with an Apple M2 Max chip, 4× NVIDIA GeForce RTX 3090 GPUs, and 4× NVIDIA A100 Tensor Core GPUs.
  • Generating all benchmark videos took less than 4 GPU-days.
  • Evaluation used Google and Anthropic APIs plus 4× NVIDIA A100 GPUs for open-sourced models, taking less than 4 GPU-days overall.
Loading 2606.03920v1…