Source-linked AI summary
From Perception to Integration: Revisiting the Internal Dynamics of Reasoning in Vision-Language Models
Rong Yu Xu, Prayag Tiwari, Shaolei Zhang
TL;DR
VLMs handle individual visual judgments more reliably than a Composite question requiring them to be combined, and full reasoning is costly. The paper uses hidden-state readiness detection to stop reasoning early; across MMStar and RealWorldQA, this reduces reasoning tokens while raising average accuracy.
Problem
Across four VLMs, accuracy on separate visual tasks is high, but falls on a Composite question requiring several judgments to be combined; reasoning is costly and can end without an answer.
Source: §1 · pp. 1–2Method
A linear detector uses the current hidden state and its change since the previous checkpoint to predict answer readiness, stopping reasoning when its validation-selected threshold is crossed.
Source: §1 · p. 2Results
Takeaways & Limitations
The development of answer readiness during reasoning can guide when to stop and allocate less reasoning computation.
Source: §6 · p. 9
Abstract
from arXiv · showhide
Vision-language models (VLMs) can answer simple visual questions, but often struggle when one question requires several visual judgments. We study this gap with controlled tasks for feature binding, numerosity, spatial relations, and amodal completion, together with a Composite task that combines them. Matched counterfactual image pairs isolate changes in the visual evidence needed to answer. Across four models, direct answers, hidden-state readouts, and state interventions show that the individual judgments can be made without explicit reasoning and that intervening on the corresponding states can affect the answer. During reasoning, the Composite answer becomes decodable from hidden states and usable from shortened traces, often before the model stops on its own. We train a small detector to predict this readiness and stop reasoning at that point. On MMStar and RealWorldQA, this reduces mean reasoning tokens by 79.1% and 74.5%, while average accuracy rises by 3.13 and 3.30 percentage points, respectively. These findings connect the internal development of answer readiness to a practical rule for allocating reasoning computation.
1 INTRODUCTION
VLMs perform well on separate visual judgments but struggle to combine them in a Composite question, and final answers do not reveal when an answer first becomes possible. The study tests how visual information is represented and used, then uses answer readiness to guide early stopping.
1 INTRODUCTION
- Across four VLMs, separate visual-task accuracy is 87.2–91.4%, but average accuracy is 58.6% on a Composite question requiring judgments to be combined.A correct final answer alone does not reveal when the model first became able to provide it.Source: §1 · p. 1
- Controlled tasks test feature binding, numerosity, spatial relations, and amodal completion, while matched image pairs isolate visual changes that alter the correct answer.Hidden-state readouts and state interventions test what information the model carries and uses; shortened traces test when it can use that information to answer.Source: §1 · pp. 1–2, Figure 1 · p. 2
- Reasoning raises Composite pair-consistent accuracy by 40.8 points on average, but full MMStar traces average 1,650 reasoning tokens and 16.6% of Composite runs reach the generation limit unanswered.The earliest tested point where a shortened trace supports a correct answer is called answer readiness.Source: §1 · p. 2
- Answer-Readiness-Guided Early Stopping uses the current hidden state and its change since the previous checkpoint to predict answerability, stopping when the score crosses a validation-selected threshold.The model then generates its answer.Source: §1 · p. 2
- Across four VLMs on MMStar and RealWorldQA, stopping shortens mean reasoning by 79.1% and 74.5%, respectively, while average accuracy rises by 3.13 and 3.30 points, respectively.Source: §1 · p. 2
2 RELATED WORK
Earlier work measures visual-question answers, studies visual representations within a single forward pass, or stops reasoning using answer confidence or sufficiency. This paper instead measures when a Composite answer becomes usable during reasoning and uses that measurement to decide when to stop.
2 RELATED WORK
- Prior multimodal reasoning and visual-evaluation studies measure final answers, while this work measures when reasoning first makes a Composite answer possible.The measurement provides the target for a detector that decides when to stop.Source: §2 · pp. 2–3
- Prior representation studies focus on single-pass visual information, whereas this analysis tracks how the model uses visual information while generating a reasoning trace.Layer-wise readouts have also been used to separate visual grounding from later processing and answer formatting.Source: §2 · p. 3
- Earlier stopping methods use answer confidence or sufficiency, while this work bases its stopping target on a direct measurement of when the visual answer becomes available.Other work estimates when reasoning helps and how long it should continue, then intervenes on attention heads.Source: §2 · p. 3
3 INTERPRETING COMPOSITE VISUAL REASONING
The section tests what component visual judgments VLMs can make without reasoning and tracks how a Composite answer becomes decodable and usable during reasoning. It also tests whether matched state interventions shift answer scores toward the donor’s answer.
3.1 INTERPRETABILITY ANALYSIS SETUP
- Feature Binding, Numerosity, Spatial Relations, and Amodal Completion test separate judgments; Composite combines them to count red circles on either side among partly hidden shapes.Matched counterfactual pairs change the answer-determining variable while preserving the rest of the scene.Source: §3.1 · p. 3, Figure 2 · p. 4
- Each component task isolates an answer-determining variable, and matched image pairs support interventions by changing that variable while keeping the remaining scene fixed.Feature Binding, Numerosity, and Spatial Relations use separate datasets for direct answers and interventions; Amodal Completion and Composite use paired datasets throughout.Source: §3.1 · p. 3
3.2 VISUAL JUDGMENTS WITHOUT REASONING
- All four models answer each component task above its task-specific guess level without explicit reasoning; they do best on Feature Binding and Spatial Relations.They make more errors on Numerosity and Amodal Completion.Source: Table 1 · p. 4, §3.2 · p. 4
- Image-conditioned readouts decode each task label from several layers and outperform question-only controls through most layers; Amodal Completion controls score higher because candidate options reveal shape information.Numerosity and complete-shape accuracy is high, while Feature Binding and Spatial Relations vary more across layers.Source: Figure 3 · p. 5, §3.2 · p. 5
- Matched donor states move scores toward donor answers more often than control donors in early and middle layers, but less consistently at the final tested block.Some labels remain decodable in late layers even without a consistent intervention advantage, so decodability and measured intervention effects can differ across depth.Source: §3.2 · p. 5
3.3 DEVELOPMENT OF COMPOSITE ANSWER READINESS
- At five checkpoints, separate readouts test answer decodability while truncated traces test whether the model can answer from reasoning so far.The offline trace test uses a fixed answer cue and requires the correct answer for scoring, which is unavailable at deployment.Source: §3.3 · p. 6
- By LATE, hidden-state decodability and shortened-trace answer usability are high for all four models, though accuracy does not always improve from LATE to ANSWER.It stays high for the Qwen models but falls for both Gemma models; correct answers are often available before reasoning ends.Source: §3.3 · pp. 6–7, Figure 4 · p. 7
4 ANSWER-READINESS-GUIDED EARLY STOPPING
Answer-Readiness-Guided Early Stopping trains a detector to predict when a VLM can answer correctly, then stops reasoning when its readiness score crosses a validation-selected threshold. The rule aims to shorten reasoning while keeping accuracy close to full thinking.
4 ANSWER-READINESS-GUIDED EARLY STOPPING
- The detector uses the current hidden state, its change since the previous checkpoint, and elapsed reasoning length to predict whether the model can answer correctly.Training uses fixed checkpoints from recorded traces, where the frozen VLM's answer is labeled for correctness.Source: §4 · p. 7
- At the first checkpoint, the previous state is a zero vector; the change feature uses the same convention in training and online inference.The feature notation is standardized using training data, and only the detector weights and bias are trained.Source: §4 · p. 7
- The detector is trained from checkpoint labels indicating whether the forced answer matches its reference, with feature normalization and class weights based only on training checkpoints.The negative-to-positive label ratio is computed over training checkpoints.Source: §4 · p. 7
- At 256-token checkpoints, reasoning stops when the readiness score reaches a validation-selected threshold that minimizes mean length while keeping accuracy within a set tolerance of full thinking.If the score never crosses the threshold, generation continues until the model stops naturally or reaches its token budget.Source: §4 · p. 7
5 EXPERIMENTS
The experiments compare adaptive stopping with full thinking on MMStar and RealWorldQA using matched evaluation rules and held-out questions. Adaptive stopping reduces reasoning and output length while its accuracy effect varies across models and benchmarks.
5.1 EXPERIMENTAL SETUP
- Adaptive stopping is evaluated against full thinking with the same token budget and answer-extraction rule, using two-fold cross-validation on 1,500 MMStar and 765 RealWorldQA questions.The validation threshold minimizes mean reasoning length while allowing at most 0.5 percentage points less accuracy than full thinking.Source: §5.1 · p. 8
5.2 MAIN RESULTS
- On MMStar, every model uses fewer reasoning tokens with adaptive stopping and average accuracy rises.Gemma-4-12B gains 8.87 points, while Qwen3.5-4B is nearly unchanged at +0.07; total output tokens fall by 23.3–84.9%.Source: §5.2 · p. 8, Table 3 · p. 8, Table 4 · p. 9
- On RealWorldQA, all four models use fewer reasoning tokens and average accuracy rises, although Gemma-4-E4B loses 1.05 points.Total output falls by 19.9–83.7%.Source: Table 3 · p. 8, §5.2 · p. 9, Table 4 · p. 9
- On MMStar, adaptive stopping reduces missing answers and changes correct-answer counts differently across open and closed full-thinking traces.Qwen3.5-4B gains 96 correct answers on open traces and loses 95 on closed traces; Gemma-4-12B gains on both groups.Source: §5.2 · p. 9, Table 5 · p. 9
6 CONCLUSION
The conclusion distinguishes component visual judgments from combining them: the former can be made without explicit reasoning, while the Composite answer becomes usable during reasoning. A detector predicts this readiness and stops reasoning early, reducing reasoning tokens while raising average accuracy across the two benchmarks.
6 CONCLUSION
- VLMs can make component visual judgments without explicit reasoning, but combining them is harder.During reasoning, the Composite answer becomes decodable from hidden states and usable from a shortened trace, often before the model stops on its own.Source: §6 · p. 9
- A small detector predicts when the Composite answer is ready and stops reasoning at that point.Across MMStar and RealWorldQA, stopping reduces reasoning tokens and improves average accuracy, partly by producing answers from traces that otherwise end without one.Source: §6 · p. 9
A DATASETS AND MECHANISTIC ANALYSIS
The appendix summarizes the synthetic datasets and their analysis splits. Primitive splits support linear probing, while the Composite split supports reasoning-trajectory analysis before trace filtering.
A DATASETS AND MECHANISTIC ANALYSIS
- Table 6 summarizes synthetic dataset sizes and analysis splits, with counts reported as images.Primitive splits are used for linear probing; the Composite split is used for reasoning-trajectory analysis before trace filtering.Source: Table 6 · p. 12, A DATASETS AND MECHANISTIC ANALYSIS · p. 12
- Behavioral evaluation uses all 1,000 images per primitive task and the 200-image Composite test set, with paired images kept in the same split.These details describe the dataset and evaluation split scope, not model performance.Source: Table 6 · p. 12
A.1 PRIMITIVE DATASET CONSTRUCTION
The primitive datasets vary answer-determining visual features while controlling other scene details, and matched pairs support counterfactual interventions. Readout and intervention procedures then test what information models carry and whether it affects their answers.
A.1 PRIMITIVE DATASET CONSTRUCTION
- The primitive datasets vary the feature determining the answer while controlling other scene details, and paired images remain in the same split.Intervention pairs keep scene context fixed while changing the answer-determining feature; Amodal Completion pairs change the complete shape under the same occluder.Source: §A.1 · pp. 12–13
A.2 PRIMITIVE READOUTS AND INTERVENTIONS
- The readout and intervention setup keeps the VLM frozen during direct answering and readout fitting, then tests whether image-conditioned states affect the answer.The no-thinking patching protocol scores the next-token choice after an answer cue; matched-random donors share the receiver’s semantic label.Source: §A.2 · p. 13
- Intervention pairs are retained only when both directions are available, both unpatched answers are correct, and original donor and receiver margins differ in both directions.The lower row of Figure 3 averages the two directional indicator differences within each pair.Source: §A.2 · p. 13
A.3 COMPOSITE DATA AND REASONING CHECKPOINTS
The Composite task tests whether models can combine visual judgments using controlled scenes and fixed reasoning-trace checkpoints. Matched image pairs reverse the answer while holding other visual properties constant.
A.3 COMPOSITE DATA AND REASONING CHECKPOINTS
- Reasoning traces are cut at fixed checkpoints; Table 9 defines those checkpoints, with token counts and one-based positions.Validation selects one analysis layer per model, and test samples do not affect that choice.Source: §A.3 · p. 14, Table 9 · p. 14
- Each Composite image shows 8–12 partly hidden red or blue shapes, and the model chooses which side has more red circles.Occlusion ranges from 25% to 60%.Source: §A.3 · p. 14
- The 1,000 images form 500 matched pairs that reverse the answer by changing shape colors while keeping positions, shapes, sizes, rotations, occluders, and overall counts fixed.One hundred pairs are reserved for direct-answer testing; the remaining 400 are split into 320 training and 80 validation pairs for reasoning-trace analysis.Source: §A.3 · p. 14
B.1 DATASETS AND CROSS-VALIDATION
The early-stopping evaluation uses question-level two-fold cross-validation, keeping related checkpoints and questions together. Additional results report each test fold separately from the main text’s combined analysis.
B.1 DATASETS AND CROSS-VALIDATION
- Two-fold cross-validation trains the detector and selects its threshold on one fold, then tests on the other, keeping each question’s checkpoints and shared-image RealWorldQA questions together.All models use the same folds; Table 10 gives the evaluation split counts and structure.Source: §B.1 · p. 14, Table 10 · p. 14
- Pooled accuracy weights questions equally across the two evaluation directions, with one held-out prediction per question in each condition.Source: §B.1 · p. 15
C ADDITIONAL EARLY-STOPPING RESULTS
- Unlike the main text’s combined-fold analysis of answer completion, the additional results report each test fold separately.Source: C ADDITIONAL EARLY-STOPPING RESULTS · p. 15
C.1 PERFORMANCE ACROSS EVALUATION FOLDS
Across held-out folds, reasoning length falls for every model and dataset, while accuracy changes are not uniform between folds.
C.1 PERFORMANCE ACROSS EVALUATION FOLDS
- Reasoning length falls in every model and dataset, but accuracy changes differ between folds.Tables 11 and 12 report per-direction held-out MMStar and RealWorldQA accuracy and mean reasoning-token counts.Source: §C.1 · p. 15, Table 11 · p. 15, Table 12 · p. 15
- MMStar fold results evaluate 750 held-out cases per direction, with accuracy reported as percentages and reasoning length as mean tokens.Source: Table 11 · p. 15
- RealWorldQA fold results evaluate 382 questions in A→B and 383 in B→A, reporting accuracy percentages and mean reasoning-token counts.Source: Table 12 · p. 15