Can Vision-Language Models Solve the Shell Game?

• Gemini-3-Pro falls from 80% on the full Perception Test dataset to 36.45% on filtered clips and 30.77% in the 3-cup setting.The 30.77% result is for instances that shuffle objects, matching random guessing at 1/3.Source: §1 · p. 1
• VET-Bench uses visually identical objects to require motion-continuity tracking, and tested state-of-the-art VLMs perform at or near chance.The benchmark is designed to remove frame-level appearance cues that can enable shortcut solutions.Source: §1 · p. 2
• Visual entity tracking is NC1-complete, suggesting fixed-depth transformers need intermediate computation; direct-answer supervision fails to teach the shell game even with sufficient training.The analysis connects entity tracking to the state-tracking problem and attributes the limitation to transformer expressivity constraints.Source: §1 · p. 2
• SGCoT exceeds 90% accuracy on VET-Bench by eliciting explicit object trajectories as intermediate states, enabling end-to-end shell-game solving without external tools.The method leverages Molmo2's object-tracking capabilities and fine-tunes it on synthetic text-only alignment data.Source: §1 · pp. 2–3

2.1 Task Formulation

• Given a target cued at t = 0, the task is to identify its terminal index π(i) after shuffling induces a global permutation of visually indistinguishable objects.Objects are identified by their initial indices.Source: §2.1 · p. 3
• To prevent identity aliasing, each object’s maximum consecutive-frame displacement d must satisfy 2d < ∆, the minimum separation between objects.In VET-Bench, each swap lasts 2 seconds, requiring at least 1 FPS to resolve it without ambiguity.Source: §2.1 · p. 3

2.2 Task Suite

• The three.js pipeline varies rendering and simulation parameters to generate unlimited episodes for Cups, which tracks a hidden ball under swapping containers, and Cards, which tracks a face-down shuffled card.The pipeline controls factors including color, material, texture, lighting, viewpoint, object count, and swap count.Source: §2.2 · p. 3
• No single frame reveals the target’s identity or the swap operation, forcing VLMs to rely exclusively on fine-grained spatiotemporal perception.Source: §2.2 · p. 3

3.2 Results

• All evaluated VLMs perform near the random-guessing baseline on VET-Bench, across model sizes and reasoning configurations.The benchmark uses 50 cups-game and 50 cards-game videos with three objects and five swaps; Figure 2 also reports Molmo2-SGCoT, a fine-tuned model based on Molmo2.Source: Figure 2 · p. 4, §3.2 · p. 4

3.1 Experimental Setup

• VLMs are evaluated by multiple-choice Top-1 Accuracy, with a random baseline of 1/N for tasks involving N objects.Swap operations last 2.0 seconds, with five swaps per episode, so even 1-FPS sampling captures at least two frames per swap.Source: §3.1 · p. 4

3.2 Results

• Failures arise from direct guessing, coarse descriptions that miss swap events, and explicit but visually ungrounded or hallucinated swap sequences.Errors in intermediate steps propagate to the final answer, and symbolic reasoning does not compensate for failures in pixel-level grounding.Source: §3.2 · pp. 4–5

3.3 Swap Count

• Performance drops substantially after just one swap and quickly approaches chance, despite most models achieving near-perfect accuracy on zero-swap episodes.The opening frames reveal the ball’s location in zero-swap cases; Gemini-3-Pro and Gemini-3-Flash sometimes hallucinate swaps, a prompt-sensitive failure also described as event hallucination by Zhang et al. (2024).Source: §3.3 · p. 5

3.4 Object Count

• Even with N = 2, models do not significantly outperform chance, and accuracy scales inversely with object count as N increases.With two objects, the task reduces to determining whether the number of swaps is even or odd.Source: §3.4 · p. 5

3.5 Comparison with the Perception Test

• The Perception Test is less diagnostic because cup and swap counts vary, and distinct or transparent cups can bypass tracking.Zero-swap examples primarily test object permanence, while variable object counts shift the random baseline.Source: §3.5 · pp. 5–6

A Filtered Subset Reveals the Perception Bottleneck. We construct a filtered subset targeting the stan-

• On a filtered Perception Test subset of 65 strict three-cup, non-zero-swap videos, models collapse to near chance; Gemini-3-Pro falls from 0.80 on the full dataset to 0.31, near the 0.33 baseline.The filtering removes visual shortcuts and excludes zero-swap cases; the results align with VET-Bench.Source: A Filtered Subset Reveals the… · p. 6, Table 1 · p. 6

3.6 Comparison with VideoReasonBench

• VideoReasonBench overlays arrows to indicate swaps, enabling static-cue reasoning, while VET-Bench requires spatiotemporal continuity; Gemini-2.5-Pro scores 56% on VideoReasonBench but remains near chance on VET-Bench.The arrows function as symbolic swap tokens rather than requiring inference from the underlying motion.Source: §3.6 · p. 6

Theoretical Analysis

• TRACKk asks whether the final ordering of k visually indistinguishable objects matches their initial ordering, and is NC1-complete for every fixed k ≥5.The proof composes frame-to-frame permutations for membership and reduces from WORDS5 by realizing adjacent transpositions in a video.Source: §4 · pp. 6–7
• Assuming TC0 ⊊ NC1, fixed-depth transformers are theoretically limited on arbitrary-length TRACKk sequences for k ≥5, motivating CoT for visual entity tracking.Prior work places constant-depth transformers within TC0; the analysis connects its NC1-completeness result to the need for CoT (Feng et al., 2023; Li et al., 2024c; Huang et al., 2025).Source: §4 · pp. 6–7
• The hardness result applies specifically to k ≥5; smaller object counts may admit shortcuts, while distinct visual identifiers reduce tracking to parallelizable visual search in AC0.For k = 2, the word problem reduces to parity in TC0, but prior work reports that effective length generalization still requires intermediate supervision or CoT (Wei et al., 2022; Anil et al., 2022; Wies et al., 2023; Kim & Suzuki, 2025).Source: §4 · p. 7
• The analysis suggests visual entity tracking shares complexity with chess-move and narrative-entity tracking, and proposes foveal eye movements as “physical CoT” that updates state across frames.The authors posit this as one way the human visual system overcomes the task’s latent reasoning demands (Merrill et al., 2024).Source: §4 · p. 7

Training with Direct Answer Fails in Practice

• After 60 epochs, direct-answer training loss remained at random-chance levels when Qwen2.5-VL-3B-Instruct trained on 500 synthesized cups-game videos.The authors attribute the failure to difficulty mapping long video-token sequences directly to the final label, leading the model to rely on training-distribution label frequency rather than learn underlying dynamics.Source: §4 · p. 7, Figure 4 · p. 7

Limitations of the Base Model

• Molmo2 (Clark et al., 2026) can output object trajectories at 0.5-second intervals with normalized coordinates, but direct-answer prompts bypass tracking and yield near-random performance.SGCoT repurposes Molmo2’s native tracking capability as intermediate reasoning for question answering.Source: §5 · p. 8

Training for SGCoT Alignment

• Text-only alignment elicits tracking with “Track [object]”; it masks synthesized trajectory tokens and supervises only the final answer, using the terminal coordinate to determine the response.Randomly generated trajectories work, while in-distribution synthesized trajectories cause minimal degradation of tracking ability.Source: §5 · p. 8
• 91% accuracy: SGCoT alignment raises Molmo2 from near-random VET-Bench performance after one epoch over 300 samples, taking 3 minutes on one A100 GPU.The alignment requires no explicit training on VET-Bench videos.Source: §5 · p. 8

Error Analysis

• SGCoT errors typically begin when the model misidentifies visually identical objects, producing abrupt trajectory jumps, incorrect terminal locations, and wrong answers.The failures occur during the trajectory perception stage.Source: §5 · p. 8

Why SGCoT Succeeds and Other CoTs Fail

• SGCoT’s fine-grained timestamps and spatial coordinates give each entity an unambiguous state, avoiding temporal misalignment and underspecified locations that can make generic descriptive CoTs brittle.Coarse time ranges can misalign non-integer motion events and accumulate errors; coarse references such as “left” or “right” underspecify the state.Source: §5 · p. 8

Video Benchmarks for Spatiotemporal Perception

• Existing video QA benchmarks often permit static-cue shortcuts (Lei et al., 2023), while newer temporal benchmarks reveal single-frame and language-prior reliance, motivating VET-Bench’s fine-grained spatiotemporal focus.The cited benchmarks include TempCompass, TemporalBench, E.T. Bench, TVBench, VideoVista, TOMATO, MotionBench, Tuna, Know-Show, EgoTempo, VCR-Bench, MVP, AoTBench, SpookyBench, Timeblind, and NarrativeTrack.Source: §6 · p. 9

Reasoning with Grounding

• VoCoT (Li et al., 2025) anchors objects spatially and GCoT (Wu et al., 2025; Xia et al., 2025) injects boxes into CoT; SGCoT extends grounding temporally for reliable tracking.Grounding abilities developed through targeted training support reasoning intended to improve robustness and reliability (Jiang et al., 2025).Source: §6 · p. 9

Entity Tracking

• Prior textual entity-tracking work evaluates state maintenance; Merrill et al. (2024) show UCI chess tracking is NC1-complete, and this work extends the result to visual tracking, requiring CoT for generalization.MET-Bench (Cohen & Mooney, 2025) evaluates shell-game and chess tracking with image and text, whereas this work targets visual entity tracking in videos.Source: §6 · p. 9

7 Limitations and Future Work

• VET-Bench assumes the final answer is fully determined by location information encoded in generated SGCoT.This limits the study to a simplified setting.Source: §7 · p. 9
• Real-world answers may require visual evidence in final frames, motivating SGCoT integration with complex referring expressions across arbitrary frames.For example, identifying which cup contains the ball from a person’s point of view requires additional visual perception.Source: §7 · p. 9
• The analysis assumes localization and continuity that keep swaps unambiguous, whereas proximity, overlap, motion blur, and occlusion complicate tracking.Handling these scenarios may require models with stronger physical priors or world models.Source: §7 · p. 9

8 Conclusion

• Molmo2 aligned to generate SGCoT before answering exceeds 90% accuracy on VET-Bench, while VLMs fail in direct-answer mode even with sufficient training.The paper presents this as state-of-the-art accuracy on VET-Bench.Source: §8 · p. 9
• VLMs perform near random chance on VET-Bench when appearance cues are removed, exposing visual entity tracking as a fundamental bottleneck.Source: §8 · p. 9
• The analysis establishes visual entity tracking as NC1-complete and suggests fixed-depth VLMs are limited without chain-of-thought.Source: §8 · p. 9

A Detailed Audit of the Perception Test for Visual Entity Tracking

• The audit shows Perception Test examples involving distinct cups and transparent cups.Source: Figure 5 · p. 16, Figure 6 · p. 16
• In videos #7632 and #9896, unedited ending frames show lifted cups and reveal the answer, making shuffle tracking unnecessary.Source: Figure 7 · p. 16
• VideoReasonBench shows frame-level arrows highlighting swaps, whereas VET-Bench lacks those explicit cues.Source: Figure 8 · p. 17
• The audit depicts the filtering process for the identical three-cup setting in the Perception Test test split.Source: Figure 9 · p. 17
• The filtered Perception Test subset’s swap-count distribution is shown alongside correct-response counts for Gemini-3-Pro and Gemini-3-Flash.Source: Figure 10 · p. 17

B Proof of Theorem 1

• TRACKk uses fixed k, H, and W, with object centers recoverable in NC1 and movement bounded by d between frames while same-frame centers remain more than 2d apart.These localization and continuity conditions make tracking well-posed by ensuring unique correspondence between consecutive frames.Source: B Proof of Theorem 1 · p. 18
• Given localization and continuity, TRACKk is in NC1: each frame’s matching is constant-size, and a balanced permutation-composition tree computes the global permutation in O(log n) depth.Checking whether that permutation is the identity also takes constant-size circuitry because k is fixed.Source: B Proof of Theorem 1 · pp. 18–19
• The hardness construction encodes S5 adjacent transpositions as constant-length motion gadgets that preserve continuity; concatenation makes the video permutation identity exactly when the word evaluates to identity.The gadgets realize adjacent swaps on the first five objects and leave the remaining objects fixed.Source: B Proof of Theorem 1 · pp. 19–20
• For every fixed k ≥ 5, TRACKk is NC1-complete, with membership in NC1 and hardness via an AC0 reduction from NC1-complete WORDS5.The reduction replaces each WORDS5 input symbol with a constant-length motion gadget.Source: B Proof of Theorem 1 · p. 20

C Quantitative Analysis

• Gemini-3-Pro’s cups-game examples show both successful tracking of intermediate states and failures caused by perception errors.In the failure case, perception errors lead to an incorrect final prediction.Source: Figure 11 · p. 21
• Gemini-3-Pro’s cards-game examples likewise contrast correct intermediate-state tracking with perception-error failures.The failure case ends in an incorrect final prediction.Source: Figure 12 · p. 22

D Hallucination

• In zero-swap cases, Gemini-3-Pro and Gemini-3-Flash can identify the initial state but hallucinate nonexistent swaps, producing incorrect final predictions.The examples show Gemini-3-Flash hallucinating a long sequence of swaps.Source: Figure 14 · p. 24

E Molmo2-SGCoT

• Molmo2 answers directly without explicit spatiotemporal tracking, resulting in near-random VET-Bench performance.Source: Figure 15 · p. 25
• Molmo2-SGCoT examples show its prompts and responses for both the cups and cards games.Source: Figure 16 · p. 25
• Molmo2-SGCoT failures typically arise when the model confuses visually identical objects, causing abrupt spatial or temporal jumps and incorrect terminal locations.These trajectory errors lead to failures in the final answer.Source: Figure 17 · p. 26

F Training

• Molmo2-SGCoT training applies QLoRA to fine-tune the language model while freezing the vision encoder weights.Source: Table 2 · p. 26