PaperlayerView exact revision on arXiv

Source-linked AI summary

Rethinking Latent Visual Reasoning: Grounding Latent Reasoning in Visual Evidence

Xi Xiao, Tianchen Zhao, Youngeun Kim, Zhuowei Li, Linghan Xu, Jiaye Wu, Zheng Zhang, Xiang Xu, Xuanbai Chen, Farhan Tejani, Jakub Zablocki, Julia Xu, Yifan Xing

arXiv:2609.34563v1cs.CVcs.CLcs.LG

TL;DR

Latent visual reasoning can produce correct answers without reliably preserving the visual evidence needed in its latent trajectory, and answer-level feedback gives no direct signal about what to preserve or where to supervise. ReaLVR adds visual-evidence supervision to free-running trajectories, improving evaluated LVR baselines across model families and scales, including a 63.7% five-task average on Qwen2.5-VL-7B.

  • Problem

    Vanilla LVR does not reliably preserve answer-relevant visual evidence, while answer-level feedback provides no direct signal about which latent positions need supervision or what evidence to preserve.

    Source: §1 · pp. 1–2

  • Method

    ReaLVR trains free-running latent trajectories using correct-versus-wrong answer comparisons to locate supervision and relevant-versus-mismatched visual evidence to specify what to preserve.

    Source: §1 · p. 2

  • Results

    Across tested backbones up to 235B, ReaLVR improves available LVR baselines; it reaches a 63.7% five-task average on Qwen2.5-VL-7B, the highest among evaluated latent-reasoning methods.

    Source: §1 · p. 2, §6 · p. 10

  • Takeaways & Limitations

    Across tested backbones up to 235B, evidence supervision improves available LVR baselines without changing model architecture or inference procedure, supporting the grounding and use of continuous latent states.

    Source: §6 · p. 10

    In vanilla Stage 2, the policy loss does not backpropagate through latent-state generation, leaving free-running latent states without direct visual-evidence supervision.

    Source: §3.2 · p. 4

Abstract

from arXiv · show

Latent visual reasoning (LVR) enables multimodal large language models (MLLMs) to perform intermediate computation in continuous latent tokens rather than expressing every reasoning step in words. However, unlike textual CoT, latent reasoning is not directly observable, making it difficult to supervise what latent tokens learn. In this work, we first conduct a thorough analysis of latent-token behavior and identify a latent evidence-credit gap: latent tokens respond only weakly to image perturbations that alter the correct answer. We hypothesize that this issue stems from the lack of explicit supervision during GRPO training. These findings suggest that a final-answer reward provides too little guidance on what visual evidence to preserve or how credit should be assigned across latent tokens. To bridge this gap, we propose ReaLVR, which brings visual-evidence supervision to the model's own free-running latent trajectories. ReaLVR contrasts correct and model-generated wrong answers to determine where stronger supervision is needed, and relevant and mismatched visual evidence to specify what to preserve. Across three model families, ReaLVR consistently outperforms evaluated LVR baselines, achieving the highest five-task average of 63.7% on Qwen2.5-VL-7B. Crucially, we are the first to scale visual reasoning in latent space, showing that our framework continues to deliver robust improvements at frontier model scales up to 235B. Further analyses show more question-sensitive latent-token positions, stronger alignment with relevant visual regions, and greater fixed-context dependence on the most attended latent tokens.

1 INTRODUCTION

The paper identifies a latent evidence-credit gap: vanilla latent visual reasoning does not reliably preserve answer-relevant visual evidence, and answer-level feedback provides no direct guidance about where supervision is needed or what evidence to preserve. ReaLVR adds supervision to free-running latent trajectories and improves evaluated LVR baselines across model families and scales.

1 INTRODUCTION

  • Vanilla LVR does not reliably preserve answer-relevant visual evidence, while standard GRPO optimizes generated text rather than the free-running latent trajectory.The paper names this missing connection the latent evidence-credit gap.Source: §1 · pp. 1–2
  • ReaLVR trains free-running latent tokens to preserve answer-relevant visual evidence, using answer comparisons to choose where to supervise and visual-evidence contrasts to specify what to preserve.The additional supervision requires no change to model architecture or inference procedure.Source: §1 · p. 2
  • ReaLVR achieves the highest five-task average among evaluated latent-reasoning methods on Qwen2.5-VL-7B at 63.7%, with improvements also reported through Qwen3-VL-235B.At 235B, it improves over LVR-SFT on all three evaluated benchmarks.Source: §1 · p. 2

2 RELATED WORK

Prior work has expanded latent trajectory designs and explored ways to ground multimodal reasoning, but how free-running latent trajectories preserve answer-required visual evidence remains an open question. ReaLVR applies answer-based position weighting and visual-evidence supervision within a differentiable current-model trajectory.

Visual Reasoning in Latent Space

  • Existing LVR methods vary trajectory form, including interleaved, adaptive, shared-workspace, structured, coarse-to-fine, and parallel or multi-hypothesis designs.These methods expand latent computation but leave open how free-running trajectories preserve evidence required for their answers.Source: §2 · pp. 2–3

Visually Grounded Multimodal Reasoning

  • Grounding approaches use regions, visual tools, explicit coordinates, visual targets, or latent-specific policy objectives, while diagnostic studies probe what latent states encode and how they respond to image evidence.ReaLVR weights positions in one differentiable current-model trajectory using correct-versus-wrong answer readout, then applies relevant-versus-mismatched visual supervision.Source: §2 · p. 3

3 PRELIMINARIES

LVR inserts continuous decoder states between image-question input and textual answers, generating latent tokens autoregressively from the available causal prefix. Its two-stage training first uses target-conditioned visual supervision, then applies free-running outcome optimization that does not directly supervise visual evidence in the latent states.

3.1 LATENT VISUAL REASONING

  • LVR inserts continuous decoder states between the image-question input and answer; each latent token depends on the image, question, and earlier latent tokens, not future answer tokens.The generation order is input, latent span, then answer.Source: §3.1 · p. 3

3.2 TWO-STAGE LVR TRAINING

  • Stage 1 trains decoder states under teacher forcing to reconstruct target visual features derived from an optional ROI annotation, alongside next-token prediction.The resulting parameters are held fixed as the Stage 2 reference policy, and target length can differ from free-running latent length.Source: §3.2 · pp. 3–4
  • In Stage 2, GRPO uses group-relative advantages and a text-token KL penalty, but its policy loss scores generated text positions and does not backpropagate through generation of fixed latent states.Thus, free-running latent states receive no direct visual-evidence supervision in this setup.Source: §3.2 · p. 4

4 REALVR: OUTCOME-CONTRASTIVE EVIDENCE CREDIT

ReaLVR supervises the model’s own free-running latent trajectory, contrasting visual evidence to specify what states should preserve and answer readouts to decide where supervision is strongest. Its visual loss updates latent generation while leaving the original inference procedure unchanged.

4 REALVR: OUTCOME-CONTRASTIVE EVIDENCE CREDIT

  • ReaLVR combines answer-contrastive position weighting with relevant-versus-mismatched visual supervision on a differentiable trajectory generated by the current model.The shared latent span is generated before correct and wrong answers are separately teacher-forced.Source: §4 · p. 4, §4.1 · p. 5

4.2 SPECIFY WHAT TO PRESERVE WITH VISUAL CONTRAST

  • Relevant visual prototypes define the evidence to preserve, while mismatched prototypes provide competing content for the visual margin.When an annotation mask is empty, the target becomes the whole image; the vision encoder and connector remain frozen.Source: §4.2 · p. 5

4.3 LOCATE SUPERVISION WITH ANSWER CONTRAST

  • Correct-answer readout minus mean wrong-answer readout assigns selective credit to latent positions with stronger correct-answer attention.The positive contrast retains its magnitude to represent both preferred positions and routing strength.Source: §4.3 · p. 5

4.4 TRAIN WITH READOUT-WEIGHTED VISUAL EVIDENCE

  • Detached weights allocate visual supervision while gradients improve evidence representation, preventing optimization from reducing the weights themselves.A uniform baseline retains supervision when selective contrast is absent, and GRPO rewards and advantages remain in the Stage 2 objective.Source: §4.4 · pp. 5–6

5 EXPERIMENTS

The experiments evaluate ReaLVR across visual-reasoning benchmarks, model sizes, and families, then examine latent-state variation, visual grounding, and answer dependence. ReaLVR outperforms evaluated latent-reasoning baselines in the reported comparisons, with analyses showing stronger evidence readout and answer sensitivity.

5.1 EXPERIMENTAL SETUP

  • The evaluation covers visual discrimination, spatial reasoning, and high-resolution perception benchmarks, comparing five latent-reasoning and non-latent baselines.Reported metrics are task accuracy and, when all five benchmark scores are available, the unweighted five-benchmark mean.Source: §5.1 · p. 6

5.2 MAIN RESULTS

  • 63.7 is ReaLVR’s highest five-task average on Qwen2.5-VL-7B, 0.8 points above ILVR, the strongest competing latent-reasoning row by average accuracy.The evaluation uses five benchmarks and six backbones spanning three model families; Table 1 reports mean ± standard deviation over three random seeds.Source: Table 1 · p. 6, §5.2 · p. 6
  • At 8B and 30B, Qwen3-VL averages are 61.2 and 65.2; at 235B, ReaLVR improves over LVR-SFT on all three evaluated benchmarks.The 235B evaluation covers MMVP, BLINK, and MME-RealWorld, so no five-task mean is reported.Source: §5.2 · pp. 6–7, Table 2 · p. 7

5.3 MECHANISM ANALYSIS: VARIATION, GROUNDING, AND USE

  • ReaLVR strengthens both latent-to-visual-target and answer-to-latent attention links, and target-region enrichment reaches about 2 in middle layers.The corresponding enrichment is near 1.6 for Monet and at or below 1.3 for LVR; correct and incorrect ReaLVR responses have similar curves.Source: Figure 3 · p. 7, §5.3 · pp. 8–9
  • Across four image-edit types, answer-changing evidence changes the correct answer in 81.45%–86.33% of pairs, while LVR changes its prediction in only 5.66%–13.09%.ReaLVR’s mean-pooled latent distance is 0.13–0.34, versus below 0.0005 for LVR.Source: §5.3 · p. 8, Figure 4 · p. 8
  • Replacing ReaLVR’s eight most answer-attended latent tokens lowers correct-answer probability from 0.70 to 0.59 with other states fixed.Figure 6 also shows that latent-state vocabulary readouts do not form an intermediate explanation.Source: §5.3 · p. 9, Figure 6 · p. 9

6 CONCLUSION

A correct final answer does not guarantee that latent tokens preserve the visual evidence needed to produce it. ReaLVR supervises what those tokens preserve and where supervision is most critical, improving evaluated LVR baselines without architecture or inference changes.

6 CONCLUSION

  • Across tested backbones up to 235B, evidence supervision improves available LVR baselines without changing model architectures or inference procedures.The results support visual-evidence supervision as a way to improve the grounding and use of continuous latent states.Source: §6 · p. 10

A TRAINING DETAILS AND HYPERPARAMETERS

The appendix documents a shared two-stage training recipe and supporting implementation and evaluation procedures. It also details how ReaLVR constructs visual-evidence supervision, allocates it across latent tokens, and prevents a local routing shortcut through detachment.

A TRAINING DETAILS AND HYPERPARAMETERS

  • ReaLVR combines target-conditioned visual supervision in Stage 1 with free-running outcome optimization and evidence loss in Stage 2, using shared optimization settings across backbones.Only GPU count, and therefore global batch size, changes with model scale.Source: A TRAINING DETAILS AND HYPERPARAMETERS · p. 16
  • The 235B model uses 200 nodes with 1,600 ROCm-visible GCDs, while the 7B, 8B, and 30B models use 64 nodes with 512 GCDs.Each MI250X accelerator contributes two 64 GB GCDs.Source: A TRAINING DETAILS AND HYPERPARAMETERS · p. 16, Table A.1 · p. 16

C.1 BENCHMARKS AND MODEL COVERAGE

  • The evaluation covers five visual benchmarks and five backbones with complete results; Qwen3-VL-235B is evaluated on three benchmarks without a five-task average.The reported five-benchmark mean is unweighted and computed only for rows with all five scores.Source: §C.1 · p. 18, §C.2 · p. 18

C.3 VISUAL PROTOTYPES

  • Visual prototypes use the same pooling rule for mismatched examples, while frozen vision and connector weights keep prototypes fixed as the evidence loss updates the language model.When ROI annotation is absent or its visual-token mask is empty, the target is the whole image.Source: §C.3 · pp. 18–19

C.4 WRONG-ANSWER CONSTRUCTION

  • Wrong-answer candidates come from the behavior policy and are filtered to remove parse failures, correct answers, candidates without answer-content positions, and duplicates.Each retained candidate is teacher-forced after the same regenerated latent span.Source: §C.4 · p. 19

C.5 DEPENDENCIES AND GRADIENT FLOW

  • Gradients flow through autoregressive latent generation, so a loss at one position can update earlier generation steps and shared parameters rather than only that token.Token weights specify contributions to the loss, not position-restricted updates.Source: §C.5 · p. 19

D.2 UNIFORM ALLOCATION AND BOUNDARY CASES

  • For 0 < η < 1, the allocation combines nonzero uniform weights for every latent token with selective routing; η = 0 and η = 1 give the selective-only and uniform-only endpoints.When no valid wrong-answer comparison exists, η > 0 preserves uniform evidence supervision.Source: §D.2 · p. 19

E.1 LOCAL DERIVATIVE OF THE UNDETACHED OBJECTIVE

  • Without detachment, the evidence-loss gradient can lower a token’s routing weight at an active hinge without improving its visual evidence margin, creating a local optimization shortcut.Detachment preserves the forward weight but removes the evidence loss’s direct gradient through routing weights; weights are recomputed at each update.Source: §E.1 · p. 20, §E.2 · p. 20

F.2 TASK-DEPENDENT BUDGETS

The appendix finds that latent-token budgets work best when matched to task needs, and that ReaLVR’s routing, visual negatives, regeneration, and detachment each contribute to its performance. Diagnostics show increased answer dependence and target-region attention, while also distinguishing those measures from causal credit or answer correctness.

F.2 TASK-DEPENDENT BUDGETS

  • MME-RealWorld peaks at K = 4 and declines 4.2 points by K = 20, while MMVP varies by only 1.0 point across tested lengths without a consistent trend.HR-8K first reaches its maximum at K = 8; BLINK peaks at K = 16, 6.3 points above K = 0.Source: §F.2 · pp. 20–21
  • K = 8 gives the highest mean accuracy across the four budget-ablation tasks; K = 16 is 0.1 points lower, while K = 0 costs 3.0 points.The post-hoc task-level oracle is 1.1 points above K = 8, supporting task-dependent budgets.Source: §F.2 · p. 21, Table F.1 · p. 21

F.3 COMPONENTS ABLATIONS

  • 63.7 is ReaLVR’s five-task average; removing any component leaves performance above LVR-RL, but the full method is best on all five benchmarks.Uniform routing costs 1.3 points on average, no negatives costs 1.6 points, off-policy supervision costs 1.8 points, and removing stop-gradient costs 1.1 points.Source: §F.3 · p. 21, Table F.2 · p. 22

G.1 FIXED-CONTEXT LATENT-TOKEN DEPENDENCE

  • Replacing ReaLVR’s top-k answer-read latent tokens while holding the remaining context fixed lowers correct-answer probability from 0.70 to 0.59, a larger drop than for Monet or either LVR variant.This diagnostic measures local answer dependence on selected latent tokens.Source: §G.1 · p. 22, Figure G.1 · p. 23

G.2 TARGET-REGION ATTENTION ENRICHMENT

  • ReaLVR’s target-region attention rises from near 1 in lower layers to approximately 2 in middle layers and stays enriched through most upper layers, while LVR curves remain at or below approximately 1.3.Correct and incorrect answer curves track closely, showing that target-region attention alone does not determine correctness.Source: §G.2 · p. 23, Figure G.2 · p. 24

G.3 ANSWER-CONDITIONED SALIENCY

  • The answer-conditioned saliency panels show attention and saliency for ReaLVR and LVR-7B in that order, with token-level attribution displayed as query-key matrices and colors normalized within each panel.The appendix includes twelve image-question cases at a common scale.Source: §G.3 · p. 24

G.4 SIMILARITY AND REPRESENTATION GEOMETRY

  • Latent trajectories remain stable and highly similar while occupying fewer principal directions than visual embeddings, but similarity alone does not establish useful visual computation.The appendix compares within-trajectory similarity, stability under perturbations, and effective rank.Source: §G.4 · p. 24, Table G.1 · p. 25, Table G.2 · p. 25

G.5 TARGET ALIGNMENT AND ANSWER READOUT

  • Free-running latent states remain weakly aligned with reconstruction targets as more target vectors increase similarity; fully forced trajectories match by construction and do not measure learned inference-time alignment.Answer-readout attention is associated with correctness, whereas residual injection measures local sensitivity; neither alone assigns causal credit.Source: §G.5 · p. 25, Table G.3 · p. 26, Table G.4 · p. 26

G.6 TASK STRUCTURE AND COUNTERFACTUAL SENSITIVITY

The counterfactual diagnostic compares answer-changing image edits with model responses, and the appendix supplements dataset-level analyses with qualitative image and token visualizations. The examples span varied visual-reasoning tasks and include a case where ReaLVR and LVR-7B answer differently.

G.6 TASK STRUCTURE AND COUNTERFACTUAL SENSITIVITY

  • The counterfactual table reports task-label structure on BLINK (N=697) and sensitivity to synthetic image edits (N=2,048; 512 pairs per edit).LVR distance uses the mean-pooled latent trajectory; final distance uses the final hidden representation.Source: Table G.5 · p. 26, §G.6 · p. 26
  • Image edits hold the question fixed; “Should flip” is the ground-truth answer-change rate, while “Model flip” is the prediction-change rate.The correct answer changes for most edited pairs, but LVR answers change infrequently and mean-pooled trajectories move only slightly.Source: §G.6 · p. 26, Table G.5 · p. 26

H.1 IMAGE SALIENCY AND TOKEN VIEWS

  • The appendix’s qualitative visualizations examine individual generated answers, while dataset-level diagnostics are reported separately in Appendix G.Image overlays use answer-conditioned |attention × gradient| saliency; additional examples include labeled attention and saliency panels.Source: H ADDITIONAL QUALITATIVE RESULTS · p. 26, §H.1 · p. 27

H.2 ATTENTION AND SALIENCY CASES

  • The twelve common-scale examples cover varied scenes, including a cap on a table, objects near appliances, pans beside pots, and a vehicle containing its pilot.In one case asking what touches the bed, ReaLVR answers “teddy bear” and LVR-7B answers “blanket.”Source: §H.2 · p. 27, Figure H.1 · p. 28, Figure H.2 · p. 29

H.3 TASK EXAMPLES ACROSS FIVE BENCHMARKS

The appendix illustrates benchmark evidence demands and diagnostics of latent-token variation, visual attention, question-dependent evidence selection, and answer dependence. Its tests include paired image edits and fixed-context exchanges through the latent span.

H.3 TASK EXAMPLES ACROSS FIVE BENCHMARKS

  • Across five benchmark examples, required evidence ranges from counting and depth comparison to map locations, finding a boat, and subtracting chart values (1,537 − 1,393 = 144).The figure uses official-dataset images, questions, and answers; its robot bubbles show annotated answers, not ReaLVR predictions.Source: §H.3 · p. 32, Figure H.5 · p. 32

H.4 POSITION-WISE LATENT VARIATION

  • 0.07 for ReaLVR, 0.02 for Monet, and 0.01 for each LVR variant is the reported top-token variation gap; ReaLVR concentrates more measured variation at particular latent positions.Figure H.6 separates cross-example variation from variation across questions about the same image.Source: Figure H.6 · p. 33

I.1 VISUAL EVIDENCE AND ANSWER READOUT

  • Figure I.1 compares target-region attention with same-area background windows by decoder layer and latent step (1 means no preference), alongside raw image-token and first-answer-query latent-state attention.The panels show aggregate post-softmax quantities.Source: Figure I.1 · p. 34

I.2 QUESTION-DEPENDENT EVIDENCE SELECTION

  • Figure I.2 uses density maps normalized to a full-image mean of 1 to compare each question’s specified region with its paraphrase; its uniform baseline differs from Figure I.1’s matched-background baseline.The example is HRBench-8K image 798; the boat-and-buildings and animal regions are manually specified.Source: Figure I.2 · p. 35, §I.2 · p. 35

I.3 LATENT SELECTION AND FIXED-CONTEXT INTERVENTION

  • Figure I.3 tests correct-versus-incorrect-answer readout, then compares fixed-context answer-log-probability loss under contrast-, attention-, and random-selected replacements; all strategies coincide when all eight states are replaced.Results average 1,000 BLINK examples on Qwen2.5-VL-7B; the endpoint therefore does not distinguish position-ranking strategies.Source: Figure I.3 · p. 36

J.1 PAIRED COUNTERFACTUAL BEHAVIOR

  • Changed-answer pairs are presented as two-turn dialogues retaining the first exchange; strict correct flips require a correct answer change, while unchanged-answer pairs measure false flips.Parse failures count as incorrect and not as valid prediction changes; parse coverage counts parseable predictions across all view-level predictions.Source: §J.1 · p. 37

J.2 LATENT EXCHANGE AND POSITION SELECTION

  • Fixed-context swaps replace selected recipient latent states with donor states while holding the recipient image, question, markers, and unselected states fixed, testing local answer dependence.The recipient’s own states provide a sham control, and unrelated-image donors test generic perturbation effects. For k=1, 2, or 4, contrast-ranked positions are compared with random positions; replacing all eight states tests whole-span sensitivity, not ranking.Source: §J.2 · p. 37

J.3 MASS-MATCHED WHERE-BY-WHAT ABLATION

The ablation crosses where visual supervision is allocated with which evidence-loss objective is used, while controlling the Stage 2 training setup. The section also notes limits in evidence-target specificity and latent-token budgeting.

J.3 MASS-MATCHED WHERE-BY-WHAT ABLATION

  • The 2×2 diagnostic crosses two choices: allocate visual supervision uniformly or by answer contrast, and use positive-only alignment or positive–negative visual contrast as the evidence loss.Source: §J.3 · p. 37
  • All arms share the Stage 1 checkpoint, examples, GRPO rollout and reward settings, latent length, optimizer, update count, and evidence-loss coefficient.The comparison reports five benchmark accuracies and seed-level macro-averages of strict correct-flip and false-flip rates; active-hinge fractions, gradient norms, and compute are separate checks. Table J.3 reports unit per-example routing mass and seed mean ± standard deviation.Source: §J.3 · p. 37, Table J.3 · p. 39
  • Without region annotations, visual evidence targets are less spatially specific, and inference currently uses a prescribed latent-token budget.Future work could derive finer targets from weak supervision and adapt the budget to each question.Source: §J.3 · p. 38

Related papers

Loading 2609.34563v1…