PaperlayerView exact revision on arXiv

Source-linked AI summary

Benchmarking Visual State Tracking in Multimodal Video Understanding

Sihyun Yu, Nanye Ma, Pinzhi Huang, Hyunseok Lee, Shusheng Yang, June Suk Choi, Ellis Brown, Oscar Michel, Boyang Zheng, Jinwoo Shin, Saining Xie

arXiv:2606.03920v1cs.CV

TL;DR

Current video benchmarks often do not test whether MLLMs track evolving states across an entire video. VSTAT evaluates that ability with questions requiring continuous event integration and finds that MLLMs lag humans and are only modestly above answer-prior baselines.

  • Problem

    Existing MLLM video evaluations provide limited evidence about continuous state tracking, a capability relevant to applications such as robotics, because answers often rely on keyframes or end states.

    Source: §1 · p. 3

  • Method

    VSTAT pairs 834 synthetic and real-world clips with 1,500 questions requiring full-stream state tracking, and analyzes failures by comparing video inputs with event transcriptions.

    Source: §1 · p. 3

  • Results

    Across VSTAT, MLLMs substantially lag humans, who achieve 90.5% average accuracy, and mostly fail to exceed answer-prior baselines; transcription success points to visual perception as the bottleneck.

    Source: §1 · p. 3, §3.2 · p. 8

  • Takeaways & Limitations

    VSTAT offers a diagnostic tool for studying MLLMs’ visual perception on continuous, real-world video streams.

    Source: §5 · p. 12

    The transcription diagnostic is infeasible for most VSTAT tasks, and even simple Blender videos required hand-provided transcripts because models could not reliably transcribe them.

    Source: §3.3 · p. 9

Abstract

from arXiv · show

Understanding a video requires more than recognizing isolated moments, as humans continuously track entities, states, and events over time. This capacity for visual state tracking is fundamental to video understanding, yet remains underexplored in current evaluations of Multimodal Large Language Models (MLLMs). We introduce Visual STAte Tracking benchmark (VSTAT), a video-based benchmark designed to diagnose visual state tracking in MLLMs. VSTAT consists of 834 clips drawn from both synthetic and real-world videos, paired with 1,500 questions that cannot be answered from any single frame or short segment, requiring continuous perception and integration of events across the entire video stream. Despite their strong performance on existing video benchmarks, we find that state-of-the-art MLLMs perform far below humans and only modestly above answer-prior baselines. To analyze this gap, we compare MLLMs' thinking traces with the underlying video stream to understand why and when MLLMs fail on VSTAT. We find that MLLMs reason and track correctly in text, but fail at visually perceiving the events they need to track. Finally, our preliminary evaluation suggests that recent agentic approaches, including MLLM-based video agents and coding agents, do not readily resolve these failures, still falling short on VSTAT.

1. Introduction

VSTAT tests whether MLLMs can continuously track evolving visual states, a capability not established by strong performance on existing video benchmarks. Its analyses indicate that visual perception is a major bottleneck, and recent agentic approaches do not readily close the gap.

1. Introduction

  • Existing video benchmarks often allow answers from keyframes or salient moments, while prior state-tracking evaluations [37] [55] cover only a single synthetic task.Strong performance on those benchmarks therefore does not necessarily demonstrate continuous state tracking.Source: §1 · p. 3
  • On VSTAT’s 834 clips and 1,500 questions, state-of-the-art MLLMs perform far below humans and only modestly above answer-prior baselines.The benchmark draws on synthetic, self-recorded, and real-world videos and requires integrating events across the video stream.Source: §1 · pp. 3–4
  • VSTAT questions require continuous tracking because critical events may be hidden, visually similar, or distributed across entities and moments.Unlike keyframe-answerable tasks, VSTAT varies the complexity and perceptual challenges of the state to be tracked.Source: §1 · p. 3
  • Temporally stretching events to reduce subsampling ambiguity yields only marginal improvement, suggesting frame subsampling is not the main bottleneck.The controlled comparison uses original videos and versions in which each event spans more frames.Source: §1 · p. 3
  • MLLMs solve simple VSTAT tasks almost perfectly from frame-and-event text transcriptions but struggle with the original videos, implicating visual perception rather than reasoning.Manual transcription is infeasible for most VSTAT tasks because their settings are more complex and require substantially more information.Source: §1 · p. 3
  • Failure traces reveal errors in event recognition, entity association, and state updates, which video agents [56] and coding agents [44] do not readily mitigate.In the shell game, models may misidentify swaps, lose track of the target-hiding cup, or fail to update its location after correctly identifying an event.Source: §1 · pp. 3–4

2. VSTAT: Visual State Tracking Benchmark

VSTAT evaluates continuous visual state tracking through video questions that require integrating information across an entire stream, using clips from synthetic and real-world procedural videos.

2.1. Data curation

  • VSTAT contains 834 clips—450 Blender-simulated, 304 from YouTube, and 80 recorded by the authors—covering varied procedural tasks with perception challenges.The dataset pairs these clips with 1,500 questions spanning diverse tracking targets.Source: §2.1 · p. 5, §2 · p. 5
  • Questions rule out keyframe or end-state shortcuts and require tracking locations, counts, or attributes in atomic, sequence, set, or dictionary structures.The benchmark is designed so each answer requires processing and updating information across the full video stream.Source: §2.1 · p. 5, §2 · p. 5
  • Each video has multiple questions of varying difficulty; answers use numerical or multiple-choice formats, and annotations undergo human-in-the-loop review.Multiple-choice questions include carefully designed distractors.Source: §2.1 · p. 6

2.2. Taxonomy

  • The taxonomy classifies state complexity by element type—count, location, or attribute—and structure—atomic, sequence, set, or dictionary.It also labels perceptual complexity across occlusion, camera motion, homogeneity, symbolic decoding, multi-entity attribution, and event ambiguity; labels support balanced data distribution.Source: §2.2 · p. 6

3. Evaluation on VSTAT

Humans substantially outperform MLLMs on VSTAT, while the evidence points to visual perception—not insufficient reasoning or frame sampling—as the main bottleneck. MLLMs and recent agentic frameworks commonly fail to identify events, associate entities, or update states correctly.

3.1. Setup

  • The evaluation compares proprietary and open-source MLLMs, agentic frameworks, chance and answer-frequency baselines, and human performance.Open-source models use the best score from uniformly sampled frame budgets of 16, 32, 64, or 128 frames; NQ scores use mean relative accuracy, while MCQs use accuracy.Source: §3 · p. 6, §3.1 · p. 7

3.2. Main results

  • Humans average 90.5% accuracy, while only Gemini-3.1 Pro and Gemini-3.0 Flash modestly exceed the Chance-Level (Frequency) answer-prior baseline; all open-source models fall below it across frame counts.Sequence states are most challenging for humans but best-performing for MLLMs. Open-source models usually show only marginal improvement with increased model size.Source: §3.2 · p. 8

3.3. Why do current MLLMs fail to solve VSTAT?

  • MLLMs solve simple Blender tasks perfectly from textual transcriptions, but video performance falls to random-guess level beyond 10 seconds and is already substantially worse at 5 seconds.The transcriptions were supplied by hand as a diagnostic; reliably producing them from complex real-world videos would be challenging and often infeasible.Source: Figure 2 · p. 8, §3.3 · p. 9

3.2. Main results

  • Gemini-3.1-Pro changes from 44.4 to 43.9, Gemini-3.0-Flash from 39.8 to 38.8, Qwen3VL-8B from 33.2 to 28.2, and InternVL3.5-8B from 30.6 to 30.2 with increased thinking.Thinking can increase hallucination on tasks with higher perceptual complexity.Source: Table 3 · p. 8, §3.2 · p. 8

3.3. Why do current MLLMs fail to solve VSTAT?

  • Temporal stretching yields only marginal improvement, indicating that event ambiguity from frame subsampling is not the primary cause of MLLM failures.Each event was made fully visible even at 1 FPS by duplicating every original frame five times.Source: Table 4 · p. 9, §3.3 · p. 9

3.4. When do current MLLMs fail to solve VSTAT?

  • Gemini-3.1 Pro’s recurring failures include misrecognizing events, losing consistent associations between visually similar entities, and incorrectly updating states despite correctly tracking events.State updates fail especially on continuous trajectories, where the model can oversimplify observations and lose information from the video stream.Source: Figure 3 · p. 10, §3.4 · p. 10
  • More than 50% of failures stem from event recognition, while state-update errors correlate with textual reasoning ability and remain relatively limited.The analysis suggests low-level perception is the dominant bottleneck, rather than visual reasoning.Source: Figure 4 · p. 10, §3.4 · p. 10

3.5. Can agentic frameworks improve performance on VSTAT?

  • AVP and coding agents perform near chance on a small VSTAT subset, despite their strength on text-based tasks.Coding agents take about 30 minutes per question on average and can be confused by inconsistent intermediate results; video agents tend to commit early using fixed low-rate sampling, typically 1 FPS, without verification.Source: §3.5 · p. 11

4. Related Work

VSTAT addresses the neglected evaluation of continuous visual state tracking in video MLLMs. It distinguishes its coverage and state definition from concurrent benchmarks and video world models.

Video Multimodal Large Language Models (MLLMs)

  • Video MLLMs combine foundational visual encoders with LLM language capabilities, and VSTAT evaluates their state tracking for applications such as long-horizon robotic manipulation.The paper situates video MLLMs within the broader progress of multimodal understanding and video-based applications.Source: §4 · p. 11

Evaluation of video MLLMs

  • Existing video benchmarks cover broad understanding, events, knowledge, temporal reasoning, long videos, or spatial information, but give little to no attention to continuous visual state tracking.VSTAT aims to evaluate and diagnose this capability, which the paper describes as important for applications including robotic manipulation, assistive agents, and surveillance.Source: §4 · p. 11

Comparison with concurrent works

  • VSTAT systematically tracks latent video states, unlike VET-bench’s two shell-game-like tasks, Video-MME-v2’s few relevant categories, and VideoReasonBench’s scripted videos with visual shortcuts.VET-bench has 100 clips; VSTAT’s real-world videos lack explicit visual cues for underlying events.Source: §4 · p. 11

Video world models

  • Unlike video world models that use explicit actions and approximate the entire visual world, VSTAT infers actions from events and defines state relative to the query.Its query-relative state captures only video information needed to answer the question.Source: §4 · p. 12

5. Conclusion

VSTAT diagnoses visual state tracking in MLLMs, revealing that they lag humans and only modestly exceed answer-prior baselines. Analyses identify visual perception—not textual tracking—as the primary bottleneck, and show that existing agentic frameworks do not trivially resolve the failures.

5. Conclusion

  • VSTAT is a video-based benchmark for diagnosing visual state tracking in MLLMs.The authors hope it will help the community understand and improve MLLM visual perception on continuous, real-world video streams.Source: §5 · p. 12
  • Current MLLMs show a substantial gap from humans and only modestly exceed answer-prior baselines.Source: §5 · p. 12
  • Controlled analyses identify visual perception, rather than textual tracking, as the primary bottleneck and diagnose recurring failure modes.Source: §5 · p. 12
  • Existing agentic frameworks, including video agents and coding agents, do not trivially resolve these failures.Limitations and future directions are discussed in Appendix D.Source: §5 · p. 12

A. Benchmark Breakdown … A.3. Curation and Filtering Process

VSTAT’s breakdown defines its state-tracking taxonomy, describes a balanced benchmark spanning simulated and real-world tasks, and details how clips and questions were curated, validated, and labeled.

A.1. Detailed Information

  • VSTAT has balanced distributions across element types and state structures, varied perceptual challenges and question keywords, and mostly sub-30-second videos.The paper notes that these videos are shorter than the context lengths of the cited frontier models.Source: §A.1 · p. 19, Figure 5 · p. 19

A.2. Categories and example visualization

  • The benchmark covers Blender-simulated tasks and recorded or curated real-world tasks, with varied state complexity and perceptual challenges.Tables 8 and 9 list the tasks, while Figures 6 and 7 visualize examples from the two groups.Source: §A.2 · p. 20, Table 8 · p. 20, Table 9 · p. 20

A.3. Curation and Filtering Process

  • Blender clips are 20 seconds in the benchmark; YouTube footage is segmented into 10-second-to-1-minute clips without ambiguous boundary events.Shorter 5- and 10-second Blender videos are reserved for studies, and identifiable people in recorded videos were filmed by the authors with explicit consent.Source: §A.3 · p. 25
  • Questions are designed to require different levels of state complexity, including whole-video tracking and distinguishing visually similar entities over time.Examples use terms such as “second-to-last” or “total” and ask who performed an action most.Source: §A.3 · p. 25
  • All answers are manually labeled, and every question-answer pair undergoes at least two human-validation rounds; ambiguous pairs are removed.Automatic annotation is described as largely infeasible given missing metadata ground truth and current MLLM tracking limitations.Source: §A.3 · p. 25
  • Multiple-choice distractors represent plausible alternative states caused by common tracking errors, and choices are revised if answerable without the video.Source: §A.3 · p. 25
  • Questions are labeled with the taxonomy, each label is independently double-checked, and formal category definitions are used to reduce ambiguity.Source: §A.1 · p. 18, Table 6 · p. 18, Table 7 · p. 18

B. Evaluation Setup Details

The evaluation setup specifies a controlled human-testing protocol and defines random and frequency-based chance baselines. It also describes how threshold-based MRA is computed separately for each question type.

B. Evaluation Setup Details

  • Human participants could replay videos and think freely but submitted one locked answer per question without seeing the ground truth.Participants included authors but excluded anyone who had contributed the relevant videos or questions; Figure 8 shows the evaluation interface.Source: B. Evaluation Setup Details · p. 26, Figure 8 · p. 26
  • Chance performance uses uniform random guesses for MCQs and a most-frequent-answer predictor for accuracy or an optimal constant for MRA.Frequency-based expected scores are estimated from the empirical answer distribution across MCQs and numerical answers; random accuracy is not computed for numerical answers.Source: B. Evaluation Setup Details · p. 26
  • MRA uses OpenEQA’s threshold-based measure with θ∈{0.5, 0.55, . . . , 0.95}, computed separately for each question type.Separate computation accounts for differences in answer magnitude, such as counts below 10 versus success/failure rates from 0 to 100.Source: B. Evaluation Setup Details · p. 26

C. Additional Results … C.4. Comparison between different Thinking Levels

The additional results break down VSTAT performance by video source, show near-perfect tracking and reasoning in selected text traces, and document visual-perception failures, including examples comparing thinking levels.

C.1. Results across video sources

  • VSTAT performance is decomposed across video sources, with Table 10 reporting reparsed MRA-with-MCQ scores by category.The table covers YouTube, Synthetic, and Recorded clips; it identifies best results and computes ranks separately for proprietary and open-sourced models.Source: §C.1 · p. 27, Table 10 · p. 27

C.2. Text transcription examples

C.3. Additional Failure cases

  • The additional failure analysis selects 70 questions spanning video categories, state elements, and question structures.The selection is intended to cover the benchmark’s range of question types.Source: §C.3 · p. 30
  • Event-recognition failures include misreading a reflected ball as a goal, inferring a click under occlusion, and missing a doll reveal.The missed reveal occurs at 00:50 and leads to an incorrect prediction.Source: Figure 12 · p. 30, Figure 13 · p. 31, Figure 14 · p. 31
  • Entity-association failures arise from camera rotation, changing box positions, and masked tile movement.These examples lead to incorrect predictions or hallucinated associations.Source: Figure 15 · p. 32, Figure 16 · p. 32, Figure 17 · p. 33

C.4. Comparison between different Thinking Levels

  • Higher thinking levels still show perceptual failures, including errors identifying a shot and players’ appearances.Figure 18 labels the failure as event recognition.Source: Figure 18 · p. 34, §C.4 · p. 34
  • Higher thinking levels also miss a dragon sign’s first reappearance and double-count espresso cups after a shot change.Both examples are labeled as entity-association failures.Source: §C.4 · p. 34, Figure 19 · p. 35, Figure 20 · p. 36

C.5. Agentic framework details

The agentic evaluation used a controlled 39-pair benchmark subset and layered safeguards against contamination. A post-hoc audit found no successful exploitation attempts across reported runs.

Agentic evaluation details

  • The evaluation sampled one question-video pair per category, yielding 39 pairs; absolute scores are not comparable to Table 2 because this subset has a higher chance level.All methods used the same subset; multiple-choice questions were scored by accuracy and numerical questions by relative accuracy.Source: §C.5 · p. 37

Evaluation details with coding agents

  • Coding agents reached approximately 87% performance by searching local directories for video filenames and questions, without reasoning over video frames.The authors describe this as contamination that makes coding-agent evaluation prone to answer injection.Source: §C.5 · p. 37

Pipeline-level defenses

  • The harness isolated each run in a fresh temporary directory containing only a randomized video copy and the question, then applied additional subprocess safeguards.Safeguards included environment-variable scrubbing, OS-level restrictions for Codex, closed standard input, and prompt prohibitions on external files and network access.Source: §C.5 · p. 37

Audit verification

  • The post-hoc audit found zero successful exploitation attempts across reported agent runs; agents derived answers purely from the input video.The audit scanned for seven contamination categories, including filesystem walks, network calls, local-cache access, and reads of benchmark metadata.Source: §C.5 · p. 37

D. Limitations and Future Directions

VSTAT exposes perceptual limitations in MLLMs, while the paper identifies limits in its analysis and video-length scope and outlines directions for improvement. The authors also caution that benchmark performance should be complemented by diverse evaluation to guard against overfitting.

Analysis using thinking traces

  • The analysis relies on text thinking traces because no established practice interprets MLLMs’ visual processing; vision-centric analysis of intermediate representations could guide improvements.The proposed direction could inform both pretraining and post-training.Source: D. Limitations and Future Directions · p. 42

Directions to improve performance on VSTAT

  • The paper demonstrates that existing MLLMs and agentic frameworks fail on VSTAT, and proposes targeting the revealed perceptual bottlenecks in pretraining and post-training.Source: D. Limitations and Future Directions · p. 42

Video length

  • The benchmark excludes hour-level video streams because visual state tracking is already challenging at current video lengths; longer scenarios are a future extension.Possible extensions include full console or e-sports gameplay and entire sports matches, such as computing pass success rate over a 1.5-hour soccer match.Source: D. Limitations and Future Directions · p. 42

Broader impact

  • VSTAT can expose perceptual limitations overlooked by existing benchmarks and guide MLLM training, but adoption risks overfitting to its patterns rather than general visual perception.The paper highlights potential applications in sports analytics, medical video analysis, and embodied agents, and recommends treating VSTAT as necessary but insufficient evidence of progress alongside diverse out-of-distribution and existing-benchmark evaluations.Source: D. Limitations and Future Directions · p. 42

E. Compute Usage

Generating all benchmark videos and evaluating all reported open-source models each take less than 4 GPU-days; evaluation of proprietary models uses Google and Anthropic APIs.

E. Compute Usage

  • Both benchmark video generation and open-source model evaluation take less than 4 GPU-days, with generation using an Apple M2 Max, 4 RTX 3090s, and 4 A100s.Open-source evaluation uses 4 NVIDIA A100 GPUs; evaluation also uses APIs from Google and Anthropic.Source: E. Compute Usage · p. 42

Related papers

Loading 2606.03920v1…