PaperlayerView exact revision on arXiv

Source-linked AI summary

Vision-aligned Latent Reasoning for Multi-modal Large Language Model

Byungwoo Jeon, Yoonwoo Jeong, Hyunseok Lee, Minsu Cho, Jinwoo Shin

arXiv:2602.04476v2cs.CV

TL;DR

MLLMs struggle with extensive multi-step reasoning as visual information dilutes during long-context generation. VaLR generates vision-aligned latent tokens before each reasoning step and trains them to align with vision-encoder features. On VSI-Bench, it raises Qwen2.5-VL accuracy from 33.0% to 52.9%.

  • Problem

    MLLMs struggle with extensive multi-step reasoning, while visual information progressively dilutes during long-context generation.

    Source: §1 · p. 1, Impact Statement · p. 9

  • Method

    VaLR generates vision-aligned latent tokens before each reasoning step and aligns intermediate MLLM embeddings with vision-encoder features during training.

    Source: §1 · p. 1, §3.1 · p. 3, §3.2 · p. 4

  • Results

    On VSI-Bench, VaLR boosts Qwen2.5-VL accuracy from 33.0% to 52.9%, and improves with longer reasoning while baselines degrade.

    Source: §1 · p. 2

  • Takeaways & Limitations

    VaLR consistently improves performance on benchmarks requiring long or short contexts and exhibits test-time scaling behavior.

    Source: §1 · p. 2, §5 · p. 8

Abstract

from arXiv · show

Despite recent advancements in Multi-modal Large Language Models (MLLMs) on diverse understanding tasks, these models struggle to solve problems which require extensive multi-step reasoning. This is primarily due to the progressive dilution of visual information during long-context generation, which hinders their ability to fully exploit test-time scaling. To address this issue, we introduce Vision-aligned Latent Reasoning (VaLR), a simple, yet effective reasoning framework that dynamically generates vision-aligned latent tokens before each Chain of Thought reasoning step, guiding the model to reason based on perceptual cues in the latent space. Specifically, VaLR is trained to preserve visual knowledge during reasoning by aligning intermediate embeddings of MLLM with those from vision encoders. Empirical results demonstrate that VaLR consistently outperforms existing approaches across a wide range of benchmarks requiring long-context understanding or precise visual perception, while exhibiting test-time scaling behavior not observed in prior MLLMs. In particular, VaLR improves the performance significantly from 33.0% to 52.9% on VSI-Bench, achieving a 19.9%p gain over Qwen2.5-VL.

1. Introduction

MLLMs have advanced on multimodal tasks, but long-context reasoning remains difficult as visual signals attenuate. VaLR inserts vision-aligned latent tokens during reasoning and improves benchmark performance, including on VSI-Bench.

  • Long-context MLLM reasoning is challenged by visual-signal attenuation as generated sequences grow, despite demand for multi-step applications such as computer-use agents and vision-language-action models.Integrating perceptual information into multi-step logical reasoning is identified as a core challenge.Source: §1 · p. 1
  • Prior approaches either strengthen text reasoning or reintroduce visual information, but the latter use static visual features as fixed initial context.The introduction contrasts these approaches with dynamically allocating visual details at each reasoning stage.Source: §1 · p. 1
  • VaLR generates vision-aligned latent tokens before each text-based reasoning step, creating visual checkpoints that keep reasoning grounded in image details.The tokens are supervised to align with dense visual representations correlated with the subsequent reasoning step.Source: §1 · pp. 1–2
  • VaLR uses a two-stage curriculum: general vision-question-answering fine-tuning first, then latent-token alignment with dense image features from vision encoders.The second stage applies representation alignment before every CoT step, using encoders such as DINOv2/v3, CLIP, or SigLIPv2.Source: §1 · p. 2
  • VaLR raises Qwen2.5-VL accuracy on VSI-Bench from 33.0% to 52.9% and improves with longer reasoning while baselines degrade.The introduction also reports superior performance over existing baselines across multiple VQA benchmarks.Source: §1 · p. 2

2. Related Works

Related work covers MLLM architectures and efforts to improve reasoning through CoT, latent reasoning, and visual-token interleaving. Prior encoder-based approaches enrich visual representations, while fixed initial visual context remains a challenge for long reasoning chains.

  • MLLMs commonly integrate visual information using resamplers, Q-Former, or lightweight projectors, yet comprehensive reasoning remains a challenge.The passage contrasts these models' reasoning capabilities with Chain-of-Thought.Source: §2 · p. 2
  • Reasoning research spans CoT prompting and training, search over reasoning chains, latent-space reasoning, visual generation from latent features, and interleaved visual tokens.VaLR aligns latent reasoning tokens with vision-encoder features.Source: §2 · p. 2
  • External-encoder methods use CLIP, DINO, SigLIP, or VGGT features for visual or spatial reasoning, but fixed initial visual context can diminish as reasoning chains lengthen.VaLR instead aligns latent tokens at each reasoning step with vision encoders.Source: §2 · p. 2

3. Vision-aligned Latent Reasoning

VaLR aligns latent reasoning tokens with visual features to preserve visual information during reasoning and support test-time scaling. Its pipeline alternates latent and language generation, trains representation alignment, and uses a two-stage SFT curriculum.

3. Vision-aligned Latent Reasoning

  • VaLR aligns latent reasoning tokens with visual features to prevent visual signal decay and enable effective test-time scaling.Its training pipeline progressively equips MLLMs with latent multi-modal reasoning capabilities.Source: §3 · pp. 2–3

3.1. Latent Reasoning in MLLMs

  • During latent mode, the model uses its previous hidden state for the next prediction, then returns to language generation after a fixed number of latent steps.The model alternates latent tokens, which are not shown as text, with response text tokens.Source: §3.1 · p. 3

3.2. Latent Reasoning with Representation Alignment

  • VaLR is the only method described as consistently improving as reasoning length increases while remaining robust on long-horizon tasks.The analysis reports hallucination rate on MMhalu and accuracy on MathVista, MathVision, and MMVP.Source: Figure 2 · p. 4
  • REPA aligns intermediate MLLM features with pre-trained visual features using patch-wise cosine similarity across latent reasoning stages.At inference, alignment supervision is absent; the model relies on learned visual grounding.Source: §3.2 · p. 4

3.3. Training Pipeline

  • VaLR's two-stage curriculum first applies SFT to CoT visual question-answering data, then trains latent tokens with REPA alignment.Stage 2 extends the training objective with a REPA loss, or a multi-encoder REPA loss when multiple encoders are used.Source: §3.3 · pp. 4–5

4. Experiment

The experiments test VaLR on spatial reasoning, perception, reasoning-length scaling, and alignment and data ablations. The reported results show gains over baselines and monotonic improvement as reasoning length increases.

4.3. Perception Tasks

  • VaLR-M outperforms CoVT by 8.7%p on BLINK and 8.9%p on V∗, and VaLR improves consistently across perception tasks.Ocean-R1’s results vary: 78.0% on V∗, 56.8% on BLINK, and 58.0% on MMVP.Source: Table 2 · p. 6, §4.3 · p. 7

4.1. Experimental Setup

  • Main experiments use Qwen2.5-VL-7B with DINOv3 by default and train both stages on a 450K Chain-of-Thought dataset.The evaluation reports accuracy across VQA benchmarks and uses greedy sampling.Source: §4.1 · p. 6

4.2. 3D Spatial Reasoning Tasks

  • 52.9% is VaLR-M’s state-of-the-art average accuracy on VSI-Bench, versus 34.0% for GPT-4o and 30.5% for Ocean-R1.VaLR-S reaches 41.5% average accuracy, compared with 33.0% for Qwen2.5-VL.Source: §4.2 · p. 6

4.4. Reasoning Length Analysis

  • VaLR improves monotonically across reasoning lengths on all benchmarks, while baselines peak at intermediate lengths and then degrade.On MMVP, Ocean-R1 falls from 62.7% to 56.5% at 300 tokens.Source: §4.4 · p. 7

4.5. Ablation Study and Analysis

  • VaLR remains effective with Qwen’s native vision encoder, external encoders further improve performance, and middle-layer alignment performs best.The tested alignment layers are the 4th, 12th, and 27th layers.Source: Table 3 · p. 6, §4.5 · pp. 7–8, Table 6 · p. 8
  • VaLR improves over the base model across encoder choices; π3 benefits VSI-Bench, 2D encoders benefit perception benchmarks, and all three encoders perform best overall.The single-encoder comparison includes CLIP, DINOv2, DINOv3, and SigLIPv2; larger gains are reported with stronger encoders such as DINOv3.Source: Table 4 · p. 7, Table 5 · p. 7, §4.5 · p. 8
  • >20× faster training to reach comparable performance on V∗ is reported for VaLR-M; both VaLR variants improve as sample size grows, while Vanilla-SFT saturates beyond 200K.The data-scaling comparison covers 10K, 50K, 100K, 200K, and 450K samples across VSI-Bench, BLINK, and V∗.Source: §4.5 · p. 8, Figure 3 · p. 8

5. Conclusion

VaLR generates vision-aligned latent tokens during reasoning. Experiments report test-time scaling and improved performance across benchmarks requiring long or short context.

5. Conclusion

  • VaLR generates vision-aligned latent tokens during the reasoning process.The paper presents the framework as a way to support multimodal reasoning.Source: §5 · p. 8
  • VaLR exhibits test-time scaling behavior and improves performance on benchmarks requiring long or short context.These are the reported experimental conclusions.Source: §5 · p. 8
  • The authors hope VaLR will facilitate future research on reasoning in multimodal large language models.Source: §5 · p. 8

Impact Statement

The statement identifies visual-information dilution during autoregressive generation as a challenge for long-context MLLM reasoning. VaLR injects latent visual checkpoints, and the authors report improved long-context reasoning, test-time scalability, and general VQA performance.

Impact Statement

  • Visual information is diluted during autoregressive text generation, especially in long-context reasoning, constraining MLLM use in VLA and CUA applications.Source: Impact Statement · p. 9
  • VaLR injects a visual checkpoint using a latent token and is reported to improve long-context reasoning, test-time scalability, and general VQA performance.Source: Impact Statement · p. 9
  • The authors connect alleviating long-context reasoning challenges with using MLLMs in complex visual tasks such as robotics with VLA or proactive CUA.Source: Impact Statement · p. 9

A.1. Training Details

Training freezes the vision encoder in both stages, while the language backbone is trained first alone and then with the alignment MLP. Evaluation details specify tools, benchmarks, and API model versions.

A.1. Training Details

  • Stage 1 freezes the vision encoder and trains the language backbone; stage 2 keeps the encoder frozen and trains the backbone jointly with the alignment MLP.Source: §A.1 · p. 14
  • Both stages use the same Chain-of-Thought datasets, and Table 7 lists their hyperparameters.Source: §A.1 · p. 14, Table 7 · p. 14

A.2. Evaluation Details

  • Evaluations use VLMEvalKit and GPT-4o as judge across VSI-Bench, perception benchmarks, and reasoning-length benchmarks; Table 8 summarizes VSI-Bench frame counts.Source: §A.2 · p. 14, Table 8 · p. 14
  • API-based evaluations report openai/gpt-4o-2024-08-06 and Claude/claude-sonnet-4-20250514.Source: §A.2 · p. 14

B. Dataset Construction

The dataset construction adapts interleaved and non-interleaved CoT data for latent reasoning, aligning tokens with images relevant to reasoning steps. VaLR is reported to handle single- and multi-view scenarios and achieve state-of-the-art performance on the multi-view benchmark.

B. Dataset Construction

B.1. Data Statistics

  • The collection contains 450K samples: 125K interleaved CoT samples and 325K non-interleaved samples, including 170K from OneThinker-SFT.The interleaved sources include Zebra-CoT, CogCoM, ReFocus, and Visual-CoT; the non-interleaved data uses GCoT with OneThinker-SFT samples.Source: §B.1 · p. 15

B.2. Non-interleaved CoT Data

  • For non-interleaved data, each example has an image set and CoT steps ending in an answer; single-view data aligns latent tokens with image features before the first reasoning step.Source: §B.2 · p. 15
  • For multi-view non-interleaved data, GPT-4o matches each reasoning step to a target image, whose visual features are used for representation alignment.Source: §B.2 · p. 15

B.3. Interleaved CoT Data

  • For interleaved data, latent reasoning begins where an image appears, and that image is aligned to the following reasoning step.This uses the dataset's explicit visual checkpoints to select the target image.Source: §B.3 · p. 15

C.1. Detailed Analysis in Table 1

  • VaLR applies to both single-view and multi-view scenarios and achieves state-of-the-art performance on the multi-view benchmark, while LVR, CoVT, and Monet collapse there.The paper attributes baseline weaknesses to single-view coverage or limited multi-view extension and reports vulnerable performance on long-term visual-memory tasks.Source: §C.1 · p. 16

C.2. Detailed Analysis in Table 4

The multi-encoder ablation reports the best overall performance when alignment includes all encoders, including Qwen’s native encoder. It evaluates encoder combinations on multi-view and perception benchmarks.

C.2. Detailed Analysis in Table 4

  • Aligning with all encoders, including Qwen’s native encoder, achieves the best overall performance among the reported combinations.Table 9 reports accuracy for VaLR ablations using different combinations of vision encoders.Source: Table 9 · p. 16, §C.2 · p. 16
  • The evaluation covers VSI-Bench for multi-view performance and BLINK, MMVP, MMStar, V∗, and CVBench for perception.The top row is the Qwen2.5-VL-7B baseline, and bold marks the best result within each group.Source: §C.2 · p. 16
  • The encoder set includes geometry-model π3, self-supervised DINOv3 and SigLIPv2, and Qwen’s encoder.The reported findings describe additional alignment with stronger encoders as a potential benefit.Source: §C.2 · p. 16

C.3. Clock-time Reasoning Analysis

The analysis measures reasoning clock-time, compares REPA with using DINOv3 features as input tokens, and visualizes VaLR’s intermediate features. REPA outperforms the input-token approach on three benchmarks without requiring a vision encoder at test time.

C.3. Clock-time Reasoning Analysis

  • Clock-time is measured at batch size 1 on one NVIDIA Tesla A100 GPU using vLLM, for 32-view VSI-Bench and 1-view CVBench scenarios.The analysis compares LVR, Monet, and VaLR.Source: §C.3 · p. 16, Table 10 · p. 16

C.4. REPA vs. Input Tokens for Visual Representations

  • REPA outperforms DINOv3-as-input-token features on VSI-Bench, BLINK, and V∗, without requiring a vision encoder at test time.Figure 4 compares the two visual-representation methods using accuracy on those three benchmarks.Source: Figure 4 · p. 17, §C.4 · p. 17

C.5. Feature Visualization

  • VaLR’s intermediate features are visualized using features extracted from its 12th layer.Figure 5 is identified as a feature visualization.Source: §C.5 · p. 17, Figure 5 · p. 17

C.6. Additional Ablation Study

The additional ablations examine REPA-loss weight λ and latent-token count K. They report best performance at λ = 0.5 and a clear advantage from increasing K.

C.6. Additional Ablation Study

  • The additional ablations are conducted on the aligned model using a single encoder.The encoder is identified by Siméoni et al. (2025).Source: §C.6 · p. 18
  • VaLR achieves its best performance at λ = 0.5, balancing preserved language semantics with visual alignment.The training objective combines standard autoregressive loss with REPA loss weighted by λ.Source: Table 11 · p. 18, §C.6 · p. 18
  • Increasing latent-token count K has a clear advantage; K determines the resolution of vision-aligned features in latent mode.The comparison is reported in Table 12.Source: Table 12 · p. 18, §C.6 · p. 18
Loading 2602.04476v2…