PaperlayerView exact revision on arXiv

Source-linked AI summary

Reason Through the Latent! Making Latent Visual Reasoning Necessary

Suhyeong Park, Junha Jung, Jaewoo Kang

arXiv:2609.06746v1cs.AIcs.CLcs.CVcs.LG

TL;DR

Latent visual states may be informative without being necessary when alternative image-conditioned routes remain available. CVRR preserves pretrained visual competence while requiring recurrent computation as the image-conditioned path to prediction. Across several benchmarks and causal interventions, the method retains strong visual competence and makes predictions sensitive to recurrent, image-conditioned content.

  • Problem

    Latent visual reasoning lacks evidence that its proposed latent computation is causally required for prediction when alternative image-conditioned routes remain available.

  • Method

    CVRR initializes recurrence from the pretrained model's image-conditioned question state, repeatedly re-reads fixed visual evidence, and removes visual states and multimodal caches before decoding.

  • Results

    CVRR retains strong visual reasoning under the strict no-bypass interface across V*, MMVP, BLINK, and MME-RealWorld-Lite, while causal interventions show predictions remain tied to recurrent content.

  • Takeaways & Limitations

    The results distinguish latent states that merely contain visual information from latent computation that is actually used for prediction.

Abstract

from arXiv · show

Latent visual reasoning aims to perform multimodal reasoning through hidden-state computation rather than explicit textual chains of thought. However, visual information being present in a latent state does not imply that the model actually relies on that state when producing its answer, especially when alternative image-conditioned paths remain available. We introduce \textbf{C}ausal \textbf{V}isual \textbf{R}ecurrent \textbf{R}easoning (CVRR), which preserves pretrained visual competence while making recurrent computation the required image-conditioned path to prediction. CVRR initializes recurrence from the question hidden state after the pretrained vision-language model has incorporated the image, then repeatedly updates this state while re-reading the same fixed visual evidence. Before decoding, visual states and the original multimodal KV cache are removed so that only the final recurrent state carries image-conditioned information to the answer. Across the $V^*$, MMVP, BLINK, and MME-RealWorld-Lite benchmarks, CVRR retains strong performance under this strict interface, while compatible latent reasoners fail to recover comparable visual competence even when retrained under the same constraint. Causal interventions further show that predictions remain sensitive to recurrent content when the question is held fixed, and that persistent visual evidence causally revises the recurrent trajectory. These results distinguish latent informativeness from latent computation that is actually used for prediction.

1 INTRODUCTION

Latent visual states can contain useful information without being causally necessary for prediction when alternative image-conditioned routes remain available. CVRR addresses this gap by preserving pretrained visual competence while requiring recurrent latent computation for image-conditioned answers.

  • Motivation: Existing latent visual reasoners can remain accurate after interventions on their latent states when the original multimodal answer context remains available.This distinguishes latent informativeness from behavioral necessity.
  • Method: CVRR initializes recurrence from the image-conditioned question state produced by the pretrained vision-language model, then repeatedly re-reads fixed visual evidence.
  • Results: 81.2% on V*, 52.7% MMVP pair accuracy, and 55.2% on BLINK were achieved under the strict no-bypass interface.Existing latent visual reasoners failed to recover comparable competence under the same constraint.
  • Contribution: CVRR recasts latent-state reliance as a causal path-design problem rather than only an informativeness problem.
  • Results: Ablations and causal interventions show that learned updates and persistent visual re-reading contribute to prediction-relevant visual refinement.

2 PRELIMINARY EXPERIMENTS

Preliminary experiments show that existing latent states may be behaviorally bypassed, while pretrained multimodal question representations already carry most of the available visual competence. These findings motivate starting recurrence from that representation and eliminating alternative image-conditioned answer paths.

  • Representation comparison: The preliminary diagnostics compare latent interventions with state representations including Full MM, Multimodal Q, Text-only Q, and Text+Raw Visual.The representation comparison uses the frozen pretrained Qwen2.5-VL-7B-Instruct before CVRR adaptation.
  • CVRR overview: Figure 2 summarizes CVRR as recurrently updating an image-conditioned question state while repeatedly reading fixed visual evidence and discarding visual states and multimodal caches before decoding.
  • Design requirements: The diagnostics motivate two requirements: preserve visual competence formed by pretraining and make prediction depend on computation through the latent path.CVRR is designed around both requirements.
  • Latent reliance: Several latent visual reasoning methods remain near their clean accuracy after latent-state replacement when the original multimodal answer context is retained.UniVLR changes by at most 0.5 percentage points across blank-image, matched-state, and norm-matched-noise replacements.
  • Representation comparison: A 0.5-point gap between Full MM and Multimodal Q indicates that nearly all available visual competence is already carried by the multimodal question representation at that layer.Adding raw visual features to a text-only state does not recover this competence.

3 METHOD

CVRR begins recurrence at a causally informed multimodal boundary, repeatedly updates question tokens against persistent visual evidence, and removes every alternative image-conditioned route before decoding. The resulting final recurrent state is therefore the sole image-conditioned input to the answer decoder.

  • Causal visual-read boundary: Layer-wise activation patching identifies the recurrent boundary by measuring how visual-token substitutions reduce answer preference across contrastive image-question pairs.The layer preceding the first sustained decline in normalized visual influence is selected as the candidate boundary.
  • Causal visual-read boundary: For Qwen2.5-VL-7B, the procedure selects ℓ*=20, with the subsequent layer f0=L21 initializing recurrence.
  • Persistent visual recurrence: The initial recurrent question state and persistent visual evidence come from the same native multimodal forward pass, preserving pretrained visual computation rather than reconstructing it from raw features.
  • Persistent visual recurrence: Each recurrent transition replaces only the question rows with the previous recurrent state while keeping visual rows fixed, so the state is repeatedly revised against persistent evidence.The shared decoder layer performs visual re-reading through native causal self-attention.
  • Strict causal decoder interface: The strict decoder interface uses image-free text-only caches below the boundary, discards visual rows and original multimodal KV caches, and exposes only hT as image-conditioned information.The backbone and upper decoder remain frozen while shared low-rank parameters are optimized.

4 EXPERIMENTS

The experiments evaluate CVRR and latent reasoning baselines across four visual question-answering benchmarks under strict no-bypass decoding. Results test whether visual competence survives when answers must depend on the latent path.

  • Datasets and metrics: The evaluation uses Visual CoT training data and four VQA benchmarks: V*, MMVP, BLINK, and MME-RealWorld-Lite.V* contains 191 high-resolution examples, while MMVP contains 300 questions over 150 visually similar image pairs.
  • Experimental design: The comparisons retrain compatible latent reasoning baselines with the same data and decoder interface while removing visual states and multimodal prefix caches before decoding.This isolates whether latent methods can preserve visual competence when bypass routes are unavailable.
  • Main results: CVRR reaches 81.2% on V*, 52.7% MMVP pair accuracy, and 55.2% on BLINK overall under the strict no-bypass interface.The central comparison is visual competence after answers are required to depend on the latent path, rather than absolute benchmark rank.
  • Main results: Latent reasoning baselines under the same interface reach at most 39.8% on V*, 2.7% MMVP pair accuracy, 38.8% on BLINK, and 33.1% on MME-RealWorld-Lite.These results show that forcing prediction through a latent state alone does not preserve comparable visual competence.
  • Controls: CVRR matches or exceeds full-multimodal SFT controls on V*, MMVP, and BLINK despite removing the standard multimodal answer path.On MME-RealWorld-Lite, CVRR drops 4.1 points from its pretrained backbone versus a 6.1-point drop for the SFT control.

5 ANALYSIS

The analyses show that native multimodal initialization, learned recurrence, and persistent visual re-reading each contribute to CVRR’s strict-path performance. Causal interventions further show that prediction depends on image-conditioned recurrent content and that the visually relevant regions can change across recurrent steps.

  • What does recurrence add beyond h1?: Replacing native multimodal initialization with the text-only anchor reduces V ∗ accuracy from 81.2% to 37.2% and MMVP pair accuracy from 52.7% to 2.0%.Persistent visual evidence and a retrained transition do not reconstruct the competence integrated into the native multimodal question state.
  • What does recurrence add beyond h1?: Removing visual re-reading reduces V ∗ accuracy by 12.0 points and MMVP pair accuracy by 34.7 points, while removing the learned transition reduces them by 4.7 and 20.0 points.These component changes show that visual access and learned recurrent computation both contribute beyond initialization alone.
  • Causal visual re-reading: Corrupting hT lowers accuracy from 81.2% to 14.0% under matched replacement, 36.1% with blank noise, and 35.6% with norm-matched noise under the strict path.Restoring the original multimodal prefix partially compensates, raising matched-swap and noise accuracy to 17.4% and 44.5%.
  • Causal visual re-reading: Clean visual evidence corrects an incorrect h1 and incorrect evidence degrades a clean h1 when question-to-visual KV connections are enabled, but blocking them largely removes both effects.The crossed-state intervention yields difference-in-differences effects of +14.8 and a corresponding negative effect reported in the analysis.
  • Image-conditioned content of the recurrent state: With the question fixed, clean accuracy is 73.3%, while whole-state and same-question replacements reduce it to 23.6% and 26.7%.The text-only anchor reaches 50.0%, above the 26.7% obtained with content from the competing image.
  • Step-varying visual influence: Causally influential image regions shift across recurrent steps, with consecutive causal maps showing mean cosine similarities of 0.37 and 0.35 across the full V ∗ set.Later transitions qualitatively emphasize objects implicated by the query in the visualized examples.

6 CONCLUSION & LIMITATIONS

CVRR addresses the gap between latent visual information and latent computation that prediction actually uses by preserving pretrained visual competence while requiring recurrent computation as the image-conditioned path. The paper reports strong strict-interface performance and identifies scope boundaries for its visual-read boundary, mechanistic analyses, and persistent-evidence design.

  • Conclusion: CVRR starts from the native multimodal question state, refines it against persistent visual evidence, and removes alternative multimodal answer routes before decoding.Only the final recurrent state remains as the image-conditioned route to prediction.
  • Conclusion: Under the strict interface, CVRR retains strong visual reasoning performance while causal analyses show that predictions follow image-conditioned recurrent state content.The analyses also show that persistent visual evidence revises the recurrent trajectory.
  • Limitations: The visual-read boundary remains backbone-specific, mechanistic analyses mainly use multiple-choice settings, and the design relies on persistent rather than adaptively acquired visual evidence.These scope boundaries motivate extending causal path-necessity analysis to more adaptive and open-ended multimodal reasoning.
  • Context: The paper situates CVRR within work on explicit visual evidence manipulation, latent visual reasoning, and causal tests of whether internal representations influence behavior.These related lines include visual self-reflection, region search, pixel-space operations, and latent replacement or removal experiments.

B.1 CVRR TRAINING AND RECURRENT DEPTH

CVRR initializes recurrence from a native image-conditioned question representation, repeatedly re-reads fixed visual evidence through a shared adapted layer, and discards visual states and multimodal caches before decoding. The training setup freezes the pretrained backbone while adapting the recurrent transition under fixed recurrent-depth and optimization settings.

  • Recurrent depth: The native image-conditioned question representation h1 initializes recurrence, while the visual state v remains fixed throughout recurrent computation.The default setup selects ℓ∗ = 20, reuses layer L21 as the shared transition, and uses T = 4 recurrent steps.
  • Inference interface: Before decoding, visual rows and the original multimodal KV cache are removed, so hT is the sole source of image-conditioned information.The decoder receives hT with image-free text-only prefix caches.
  • Training: The pretrained backbone remains frozen, and only the recurrent transition is adapted with LoRA using rank 32.CVRR training minimizes answer-token cross entropy with β = 0.50 during training and inference.
  • Training: The default training uses AdamW, a 2 × 10^-5 learning rate, cosine decay, 0.05 warmup, gradient clipping at 1.0, three epochs, and an effective global batch size of 128.Training uses bfloat16 precision, seed 0, and four NVIDIA B200 180GB GPUs.
  • Baselines: Strict-interface baseline comparisons retain each compatible method’s architecture, latent mechanism, and method-specific objectives while standardizing Visual CoT data and common SFT settings.The answer interface is modified to remove visual states and multimodal prefix KV caches before decoding.
  • Recurrent computation: Recurrence updates only the question state while repeatedly re-reading fixed visual evidence through a shared adapted decoder layer.The native multimodal forward first initializes the image-conditioned state and persistent visual evidence, after which visual states and multimodal caches are discarded.

D EFFICIENCY AND COMPUTATIONAL COST

CVRR evaluates recurrent depth as a trade-off between accuracy and inference cost, and compares favorably with six latent visual reasoning baselines under a matched four-step budget.

  • Accuracy across recurrent depths: 75.4% at T = 1 rises to 81.2% at T = 4, then falls to 78.5% at T = 6 and recovers to 80.1% at T = 8.Because each depth T ≥ 2 is independently trained, the sweep is not a monotonic test-time scaling law; T = 4 is best among tested settings.
  • CVRR procedure: The training and inference procedure uses a frozen VLM, a shared recurrent layer, and recurrent depth T to produce the answer distribution.The algorithm initializes recurrent processing from multimodal hidden states, preserves visual evidence during recurrence, and updates only the shared adapter during training.
  • CVRR procedure: Strict decoding discards visual states and multimodal or recurrent caches, leaving h_T as the only image-conditioned state available to the answer decoder.The final answer distribution is computed from h_T and the text cache.
  • Accuracy across recurrent depths: From T = 1 to T = 4, latency increases by 9.9%, decoder-side computation by 13.4%, and peak incremental memory by 0.12 GiB.Table 3 measures latency on an NVIDIA B200 over 32 examples, excluding image preprocessing.
  • Comparison with baselines: CVRR has the lowest measured latency, decoder-side computation, and incremental memory among six baselines under the same four-step reasoning budget.Its latency is 1.146 seconds, decoder computation is 53.96 TFLOPs, and incremental memory is 1.16 GiB.

E QUERY-CHANNEL CORRECTIONS ACROSS RECURRENCE

Recurrent updates increasingly modify the question representation across successive visual re-reads, with corrections varying across token positions and hidden channels.

  • Query-channel corrections: 8.9% at R1 →R2, 10.6% at R2 →R3, and 12.0% at R3 →R4 quantify increasing aggregate query correction.The correction is measured as ∥∆q∥2/∥q(0)∥2.
  • Query-channel corrections: Correction magnitudes vary non-uniformly across question-token positions and hidden channels.Two-dimensional maps show the structured variation, while three-dimensional surfaces show its magnitude and concentration.
  • Interpretation: These corrections characterize functional changes induced by recurrence, not attention weights or semantic feature importance.The farthest token-distance regions are not interpreted strongly because fewer examples are available there.

F PROJECTION-WISE FUNCTIONAL CORRECTIONS

Projection-wise analysis shows that recurrent corrections are concentrated in the output projection and increase across transitions, while key and value corrections remain comparatively stable.

  • Projection-wise corrections: Query, gate, and up projections also receive increasing corrections, whereas key and value projections remain comparatively stable.The figures use consistent projection ordering and radial scale across transitions.
  • Projection-wise corrections: 48.3%, 53.0%, and 54.9% are the output projection’s relative corrections across the three recurrent transitions.The output projection receives the largest relative correction throughout recurrence.
  • Interpretation: The correction pattern is consistent with fixed visual evidence being reread while the question representation evolves.Query- and output-side corrections reflect modification in the evolving recurrent state, but no projection is shown to uniquely perform visual reasoning.

G POPULATION STATISTICS FOR SPATIAL VISUAL RE-READING

Population-level statistics over the full V* evaluation set support spatially concentrated causal influence across recurrent transitions, alongside only moderate similarity between consecutive region maps.

  • Population statistics: Causal influence is spatially concentrated across all recurrent transitions, while consecutive region maps show only moderate similarity.The population statistics complement qualitative examples and indicate that step-varying visual rereading is not driven by a small set of selected cases.

H LOCALIZING THE CAUSAL VISUAL-READ INTERFACE

The visual-read interface is localized where question-state sufficiency is already high while residual visual influence remains available for refinement. Boundary sweeps support the 20–22 region, and strict-path results generalize across multimodal backbones.

  • Boundary selection: Early layers retain visual influence but provide insufficient question states, whereas deeper layers increase sufficiency as residual visual influence declines.This identifies a transition region suitable for recurrent visual refinement rather than simply locating where visual mediation begins to decline.
  • Boundary selection: Activation patching selects ℓ∗=20, immediately before recurrent layer L21, where high question-state sufficiency coexists with remaining visual signal.The held-out V∗ sufficiency curve independently confirms that visual competence is already high in this region.
  • Boundary validation: Layers 20 and 22 achieve 80.10% and 78.53% clean accuracy, with matched-swap drops of 66.29 and 65.73 points, respectively.The matched-budget boundary sweep supports the 20–22 region as a functional visual-read interface rather than a downstream-accuracy insertion point.
  • Backbone generalization: InternVL3-9B reaches 75.9% under the strict path versus 75.4% with full multimodal continuation, while recurrent-content replacement causes a 57.1 percentage-point drop.Table 7 reports paired strict–full accuracy differences and intervention-induced drops across multimodal backbones.
Loading 2609.06746v1…