Source-linked AI summary
Can Vision-Language Models Solve the Shell Game?
Tiedong Liu, Wee Sun Lee
TL;DR
Video VLMs struggle to track visually indistinguishable entities because existing benchmarks can permit appearance-based shortcuts. The paper introduces VET-Bench and SGCoT, finding near-chance baseline performance but accuracy exceeding 90% when explicit object trajectories are generated as intermediate reasoning states.
Problem
Visual entity tracking is a critical VLM bottleneck, while existing shell-game benchmarks can contain appearance cues that allow single-frame re-identification instead of temporal tracking.
Method
The paper introduces VET-Bench with visually identical objects and proposes SGCoT, which generates spatiotemporal object trajectories as explicit intermediate states.
Results
State-of-the-art VLMs perform at or near chance on VET-Bench, while SGCoT achieves state-of-the-art accuracy exceeding 90%.
Takeaways & Limitations
Explicit intermediate trajectories enable VLMs to reliably solve the video shell game end-to-end without external tools.
Takeaways & Limitations
Performance comparisons with the Perception Test are constrained by varying cup counts, swap counts, and appearance shortcuts across its recordings.
Abstract
from arXiv · showhide
Visual entity tracking is an innate cognitive ability in humans, yet it remains a critical bottleneck for Vision-Language Models (VLMs). This deficit is often obscured in existing video benchmarks by visual shortcuts. We introduce VET-Bench, a synthetic diagnostic testbed featuring visually identical objects that necessitate tracking exclusively through spatiotemporal continuity. Our experiments reveal that current state-of-the-art VLMs perform at or near chance level on VET-Bench, exposing a fundamental limitation: an over-reliance on static frame-level features and a failure to maintain entity representations over time. We provide a theoretical analysis drawing connections to the state-tracking problem, proving that fixed-depth transformer-based VLMs are fundamentally limited in tracking indistinguishable objects without intermediate supervision due to expressivity constraints. To address this, we propose Spatiotemporal Grounded Chain-of-Thought (SGCoT): generating object trajectories as explicit intermediate states. Leveraging Molmo2's object tracking ability, we elicit SGCoT reasoning by fine-tuning on synthesized text-only data for alignment. Our method achieves state-of-the-art accuracy exceeding 90% on VET-Bench, demonstrating that VLMs can reliably solve the video shell-game task end-to-end without external tools. Our code and data are available at https://vetbench.github.io .
1 Introduction
The paper identifies visual entity tracking as a bottleneck for video VLMs and introduces VET-Bench to remove appearance-based shortcuts. It shows near-chance performance on tracking-dependent cases, provides an expressivity analysis, and proposes SGCoT, which exceeds 90% accuracy.
- Motivation: Visual entity tracking remains a critical bottleneck for VLMs despite their proficiency in general video understanding.The capability is important for downstream applications including embodied AI and game-playing agents.
- Evidence of the bottleneck: 107 video–question pairs remain after filtering 189 cups-game clips for appearance cues, and Gemini-3-Pro falls from 80% to 36.45%.In the restricted 3-cup, non-shuffling-excluded setting, performance reaches 30.77%, matching the 1/3 random baseline.
- Benchmark: VET-Bench uses visually identical objects to force tracking through motion continuity rather than frame-level appearance cues.The benchmark is designed as a synthetic diagnostic testbed for isolating spatiotemporal perception.
- Evidence of the bottleneck: Current state-of-the-art models perform at or near chance on VET-Bench and struggle to maintain coherent entity representations over time.These findings indicate reliance on static appearance features rather than genuine temporal tracking.
- Theory: Theoretical analysis proves visual entity tracking NC1-complete and links fixed-depth transformer limitations to the need for intermediate computation.Direct-answer training without intermediate supervision fails to learn the shell game even with sufficient training.
- Method and result: SGCoT elicits explicit spatiotemporal object trajectories as intermediate states and achieves state-of-the-art accuracy exceeding 90% on VET-Bench.The approach aligns Molmo2 using synthetic text-only data and solves the shell game end-to-end without external tools.
2 Data Generation
VET-Bench represents videos with initially indexed, visually indistinguishable objects whose induced permutation determines the target’s final position. Its continuity constraint prevents identity ambiguity, while synthetic rendering supports controlled variation and tracking-focused game tasks.
- Task formulation: The task tracks N visually indistinguishable objects from initial indices to their final positions under a global permutation.A target is designated at t = 0 by a visual or linguistic cue, and the objective is to determine its terminal index.
- Task formulation: The continuity condition requires 2d < ∆, preventing identity aliasing during object crossovers.Here d is maximum consecutive-frame displacement and ∆ is minimum object separation; VET-Bench uses 2-second swaps requiring at least 1 FPS.
- Data generation: The synthetic pipeline varies color, material, texture, lighting, and camera viewpoint while enabling unlimited episode generation.This design reduces data leakage, overfitting, and memorization risks associated with fixed real-world datasets.
- Task suite: The task suite includes Cups Game, which tracks a hidden ball beneath identical opaque containers, and Cards Game, which tracks a shuffled face-down card.Both tasks require maintaining object identity through positional transformations.
- Task suite: VET-Bench removes single-frame cues revealing target identity or swap operations, forcing reliance on fine-grained spatiotemporal perception.This design prevents static frame-level shortcuts from solving the task.
3 Experiment
Experiments evaluate diverse video VLMs on VET-Bench and related filtered shell-game data, finding near-chance tracking performance and strong sensitivity to swaps, object counts, and visual shortcuts.
- Results: All evaluated VLMs perform near the random-guessing baseline on VET-Bench across model sizes and reasoning configurations.The benchmark evaluates video-capable proprietary and open-source models using Top-1 accuracy, with random baseline 1/N.
- Failure Modes: Models commonly fail by answering directly, describing shuffling coarsely, or generating coherent but incorrectly grounded swap sequences.Incorrect entity motion or hallucinated swaps propagate into incorrect final predictions.
- Swap Count: Performance drops substantially after one swap and then approaches random guessing, whereas zero-swap episodes are near-perfect for most models.Zero-swap performance can rely on directly observable opening-frame information rather than temporal tracking.
- Object Count: Even with two objects, models do not significantly exceed the random baseline, while accuracy scales approximately as 1/N as object count increases.The two-object case reduces to parity, requiring the model to track whether the number of swaps is even or odd.
- Comparison with the Perception Test: Filtering the Perception Test for visual shortcuts, at least one swap, and the strict three-cup setting leaves 65 videos, where evaluated models collapse to near chance.Gemini-3-Pro falls from 0.80 on the full dataset to 0.31 on the filtered subset, compared with a 0.33 random baseline.
- Comparison with VideoReasonBench: VideoReasonBench overlays arrows that encode swaps in frames, whereas VET-Bench removes such cues and requires tracking through spatiotemporal continuity.This distinction separates state reasoning from solving through static symbolic annotations.
4 Theoretical Analysis
The theoretical analysis formalizes visual entity tracking as a state-tracking problem and connects its complexity to limitations of fixed-depth transformers under a standard circuit-class conjecture.
- Problem Formulation: TRACK_k asks whether the final ordering of k visually indistinguishable objects equals their initial ordering after a video sequence.The formulation represents the video as frames on a grid and the result as a global permutation.
- Complexity Result: For any fixed k ≥5, TRACK_k is NC1-complete via characterization through the word problem for the symmetric group S5.Membership composes frame-to-frame permutations, while hardness reduces adjacent transposition sequences to videos.
- Transformer Limitation: Assuming TC0 ⊊ NC1, fixed-depth transformers are theoretically limited on arbitrary-length TRACK_k instances for k ≥5.The argument uses prior results placing constant-depth transformers within TC0.
- Scope and Shortcuts: Unique visual identifiers can reduce tracking to parallelizable appearance-based search, while indistinguishable objects retain the sequential state-tracking difficulty.For smaller object counts, including k = 2, shortcut solutions may exist because the relevant group structure is simpler.
- Empirical Verification: Direct-answer training on 500 synthetic cups-game videos leaves loss at the random-chance level after 60 epochs, motivating intermediate reasoning states.The experiment uses Qwen2.5-VL-3B-Instruct with 8 FPS sampling and varied shuffle permutations.
5 Spatiotemporal Grounded Chain-of-Thought
The section proposes SGCoT, which uses explicit spatiotemporal object trajectories as intermediate states to improve shell-game question answering. Molmo2’s native tracking is elicited through lightweight text-only alignment, achieving over 90% accuracy, though perception errors can still disrupt trajectories.
- Method: Molmo2’s native tracking capability is repurposed to generate object trajectories as intermediate reasoning states for visual entity tracking QA.The approach addresses near-random direct-answer performance by making spatiotemporal tracking explicit.
- Training: Text-only alignment uses minimal synthesized training samples and a tracking-oriented prompt, avoiding high-FPS video input during training.The method prefixes queries with instructions such as “Track [object]” to elicit trajectory generation.
- Training: The synthesized trajectory is used as SGCoT, while training masks trajectory-token loss and supervises only the final answer.In VET-Bench, the terminal coordinate determines the final left, middle, or right answer.
- Error Analysis: SGCoT errors typically begin with misidentification of visually identical objects, causing abrupt trajectory jumps and incorrect terminal locations.These perception-stage errors subsequently produce incorrect final answers.
- Method: SGCoT explicitly aligns event timing with fine-grained entity locations, producing an unambiguous state representation for long-horizon tracking.Timestamps encode when events occur, while spatial coordinates encode where entities are located.
6 Related Work
Related work covers video benchmarks, grounded reasoning, and entity tracking. The paper positions VET-Bench as a focused test of fine-grained spatiotemporal perception and connects its theory to prior state-tracking results.
- Video Benchmarks for Spatiotemporal Perception: Existing video QA benchmarks often emphasize general understanding and reasoning, while some permit high performance through static visual or language shortcuts.Recent benchmarks have highlighted reliance on single-frame biases or language priors instead of genuine temporal modeling.
- Reasoning with Grounding: Recent grounded-reasoning methods use spatial anchors such as bounding boxes to guide multistep reasoning and improve faithfulness or robustness.The related work includes approaches that inject grounding information directly into chain-of-thought reasoning.
- Entity Tracking: Textual entity-tracking research evaluates whether models maintain entity states, while prior theory establishes NC1-completeness for chess state tracking.This paper extends that state-tracking connection to visual entity tracking.
7 Limitations and Future Work
The evaluation studies a simplified setting where final answers depend only on SGCoT location information. Real-world referring expressions and difficult motion conditions remain outside this setting and motivate future work.
- Scope: VET-Bench’s simplified setting makes the final answer fully determined by location information encoded in the generated SGCoT.Real-world questions may require additional visual evidence from scene frames.
- Future Work: The setting does not cover referring expressions that require visual perception in final frames or across arbitrary frames.Future work could integrate SGCoT with more complex referring expressions.
- Scope: The analysis assumes localization and continuity conditions that keep object swaps unambiguous.Close proximity, overlap, motion blur, and occlusion make practical tracking more difficult.
- Future Work: More challenging tracking scenarios may require models with stronger physical priors or world models.The paper identifies these mechanisms as possible directions for improving robustness.
8 Conclusion
The conclusion identifies visual entity tracking as a fundamental VLM bottleneck and reports that SGCoT-aligned Molmo2 exceeds 90% accuracy on VET-Bench. It frames explicit intermediate reasoning as the paper’s route to solving the shell game.
- Conclusion: Visual entity tracking is identified as a fundamental bottleneck in current VLMs.The conclusion ties this diagnosis to the benchmark and theoretical analysis.
- Conclusion: State-of-the-art models perform near random chance when appearance cues are removed, while SGCoT-aligned Molmo2 exceeds 90% accuracy on VET-Bench.The conclusion contrasts direct tracking difficulty with the reported SGCoT result.
- Conclusion: The theoretical analysis establishes visual entity tracking as NC1-complete and suggests fixed-depth VLMs are limited without chain-of-thought computation.The conclusion also reports that direct-answer training remains unsuccessful despite sufficient training.
A Detailed Audit of the Perception Test for Visual Entity Tracking
The audit identifies appearance and frame-level shortcuts in the Perception Test that can reveal object identity or swap operations without full temporal tracking.
- Transparent cups provide appearance cues that allow object identification without relying exclusively on motion continuity.
- Unedited final frames showing lifted cups reveal the answer and make tracking the shuffle irrelevant.
- VideoReasonBench overlays red arrows on swap operations, supplying frame-level cues absent from the audited shell-game setting.
- The audit uses a filtering process for the identical three-cup setting and reports swap-count distributions alongside Gemini-3-Pro and Gemini-3-Flash correct responses.
B Proof of Theorem 1
The proof establishes that fixed-k visual entity tracking lies in NC1 by computing local correspondences and composing their permutations, then proves NC1-hardness through S5 swap gadgets.
- TRACKk asks whether the final permutation of k indistinguishable objects is the identity, under localization and continuity assumptions.
- Each frame’s uniquely matching successor defines a bijective local permutation computable by a constant-size circuit.
- Balanced-tree composition computes the global permutation with O(log n) depth and polynomial size.
- Checking whether the resulting permutation is the identity yields TRACKk ∈NC1.
- For fixed k ≥5, constant-length gadgets simulate adjacent generators of S5 while stationary remaining objects preserve the continuity conditions.
- Concatenating gadgets maps each generator word to a video whose global permutation is the represented S5 element on five objects and identity elsewhere.
- The gadget construction is a uniform AC0 reduction, proving NC1-hardness and therefore NC1-completeness.
- Theorem 1 concludes that TRACKk is NC1-complete for every fixed k ≥5.
C Quantitative Analysis
The quantitative-analysis section illustrates model successes and failures through response examples, including direct-answer behavior, hallucinated swaps, and SGCoT errors involving identical objects.
- Gemini-3-Pro success cases correctly track all intermediate states in both cups-game and cards-game examples.
- Gemini-3-Pro failures arise when perception errors produce an incorrect final prediction.
- Gemini-3-Flash examples show hallucinated non-existent swaps in zero-swap cases, causing incorrect final predictions.
- Molmo2 responds directly without explicit spatiotemporal tracking and therefore achieves near-random performance.
- The task asks which cup contains the ball at the end of the video.
- Molmo2-SGCoT examples present responses for both the cups game and the cards game.
- Molmo2-SGCoT failures typically involve confusing visually identical objects, producing spatial or temporal jumps and wrong terminal locations.
- Training details state that QLoRA fine-tunes the language model while the vision encoder remains frozen.