PaperlayerView exact revision on arXiv

Source-linked AI summary

LaViT: Aligning Latent Visual Thoughts for Multi-modal Reasoning

Linquan Wu, Tianxiang Jiang, Yifei Dong, Haoyu Yang, Fengji Zhang, Shichaang Meng, Ai Xuan, Linqi Song, Jacky Keung

arXiv:2601.10129v2cs.CVcs.AI

TL;DR

Multimodal distillation can reproduce teacher outputs without transferring the visual attention needed for grounded reasoning. LaViT aligns latent visual semantics and attention trajectories before text generation, using curriculum sensory gating to limit shortcut learning. Experiments show stronger grounding and reasoning, including a compact 3B model’s competitiveness with larger models.

  • Problem

    Existing distillation can align textual outputs while students diverge from teachers’ visual attention trajectories, leaving a Perception Gap in grounded multimodal reasoning.

  • Method

    LaViT autoregressively reconstructs teacher visual semantics and attention trajectories with latent tokens before textual generation, using Curriculum Sensory Gating to prevent shortcut learning.

  • Results

    LaViT improves visual grounding and multimodal reasoning, with +16.94% on Relative Depth and performance surpassing Qwen2.5-VL-7B on five of seven benchmarks.

  • Takeaways & Limitations

    LaViT-3B rivals or surpasses larger open-source models and GPT-4o on reasoning-intensive benchmarks while transferring both what to see and where to look.

Abstract

from arXiv · show

Current multimodal latent reasoning often relies on external supervision (e.g., auxiliary images), ignoring intrinsic visual attention dynamics. In this work, we identify a critical Perception Gap in distillation: student models frequently mimic a teacher's textual output while attending to fundamentally divergent visual regions, effectively relying on language priors rather than grounded perception. To bridge this, we propose LaViT, a framework that aligns latent visual thoughts rather than static embeddings. LaViT compels the student to autoregressively reconstruct the teacher's visual semantics and attention trajectories prior to text generation, employing a curriculum sensory gating mechanism to prevent shortcut learning. Extensive experiments show that LaViT significantly enhances visual grounding, achieving up to +16.9% gains on complex reasoning tasks and enabling a compact 3B model to outperform larger open-source variants and proprietary models like GPT-4o.

1 Introduction

Existing multimodal latent reasoning often relies on external visual supervision, while distillation can match textual outputs without transferring visual attention. LaViT addresses this Perception Gap by aligning latent visual thoughts and regulating visual access during training.

  • Motivation: Existing latent multimodal reasoning methods commonly use auxiliary images or annotated regions, leaving intrinsic visual attention dynamics unexplored.
  • Motivation: Standard multimodal distillation mainly aligns textual outputs or distributions, assuming that answer reproduction transfers multimodal reasoning ability.
  • Perception Gap: Empirical analysis finds that students can match teacher outputs while their visual attention diverges, producing a Perception Gap between what to say and where to look.
  • LaViT: LaViT autoregressively reconstructs the teacher’s visual semantics and attention trajectories with latent tokens before generating text.
  • LaViT: Curriculum Sensory Gating progressively restricts and relaxes visual input, enforcing an early latent bottleneck while avoiding training–inference mismatch.
  • Results: LaViT-3B achieves up to +5.0% gains on MMVP, improves BLINK, and rivals or surpasses 7B models and GPT-4o.

2 Related Work

Related work spans textual and latent approaches to multimodal reasoning, as well as knowledge distillation methods that align outputs, hidden states, attention, or visual-textual features.

  • Multimodal reasoning: Multimodal reasoning research includes Chain-of-Thought and RL-style optimization, but these approaches often rely on indirect textual proxies rather than intrinsic visual understanding.
  • Multimodal reasoning: Other approaches enable thinking with images through tools, executable programs, and iterative region grounding.
  • Latent reasoning: Latent reasoning replaces discrete token sequences with continuous hidden states and extends to multimodal models through visually supervised latent tokens.
  • Knowledge distillation: Multimodal distillation methods variously match hidden attention distributions and feature maps, align visual and textual token features, or use token selection.

3 Empirical Analysis of Perception Gap

The empirical analysis links focused visual attention with reasoning success and exposes a Perception Gap: textual alignment can remain stable while teacher–student visual attention diverges, especially for fine-grained visual concepts.

  • Research questions: The pilot study asks whether focused visual attention preconditions correct visual reasoning and whether teacher–student visual trajectories diverge despite similar textual outputs.
  • Visual attention: Visual Focusing Score Sfocus measures the attention mass assigned to a target bounding box relative to total image attention.
  • Visual attention: Reasoning accuracy increases monotonically with Sfocus thresholds; correct samples average 18.78% Sfocus versus 9.18% for incorrect samples, a ∼104% relative gap.
  • Visual attention: Samples with negligible Sfocus (< 1%) predominantly produce irrelevant or severely hallucinated responses, indicating unreliable reasoning without active visual grounding.
  • Perception Gap: Teacher–student attention divergence rises with visual semantic reliance, reaching µ ≈1.39 for Attribute tokens versus µ = 1.11 for Functional tokens.
  • Perception Gap: Despite rising attention divergence, hidden-state Cosine Distance remains stable from 0.52 to 0.55, showing textual mimicry without aligned visual attention.
  • Perception Gap: The findings characterize a Perception Gap in which SFT reproduces teacher words but does not transfer the underlying visual trajectory.

4 Method

LaViT distills contextualized visual semantics and attention trajectories into autoregressive latent tokens before text generation. Curriculum Sensory Gating prevents shortcut learning by gradually opening the direct visual path, while dual-stream objectives align semantics and gaze.

  • LaViT-SFT-15K: 15K tuples distill the teacher’s internal cognitive states, filtered for correctness, difficulty, and attention alignment.The dataset uses Qwen2.5-VL-32B supervision and rejects samples with insufficient aggregated attention mass.
  • Teacher Signals: Dynamic Visual Gaze averages teacher cross-attention over layers, heads, and response positions, then applies Min-Max normalization to produce patch-level gaze probabilities.Top-N sparsification retains N = 8 salient visual cues for sparse supervision.
  • Teacher Signals: Vsem uses the teacher’s final-layer image tokens, whose self-attention interaction with the query yields contextualized visual thoughts rather than frozen visual features.Thus, the target reflects how the teacher interprets image content for a specific instruction.
  • Latent Teacher Forcing: Latent Teacher Forcing makes the student generate K continuous latent tokens before its textual response, forming a visual-information bottleneck and cognitive bridge.The complete trajectory is X = [I, Q, V, A], with V supplying visual evidence for A.
  • Curriculum Sensory Gating: Curriculum Sensory Gating gradually increases direct image access during warm-up, then fully opens it to preserve inference topology and avoid distribution shift.Early suppression creates a latent bottleneck; later residual perception restores access to fine-grained visual details.
  • Optimization Objectives: Dual-stream distillation aligns latent semantics with Vsem and latent-originating attention with Atraj, while gating controls response-loss gradients through direct visual features.Trajectory supervision is applied only to the final latent token, and fixed-weight joint training combines next-token, concept, and trajectory losses.

5 Experiment

LaViT consistently improves multimodal benchmark performance, with gains spanning fine-grained perception, complex visual reasoning, and attention quality. Its 3B model also demonstrates focused, stable visual attention and benefits from trajectory alignment, semantic reconstruction, and curriculum sensory gating.

  • Main Results: LaViT achieves consistent improvements across all evaluated benchmarks, covering fine-grained perception, visual reasoning, and multimodal robustness.
  • Cross-Scale Superiority and Efficiency: +16.94% on Relative Depth and +15.67% on Relative Reflectance demonstrate strong gains despite LaViT’s compact 3B scale.LaViT outperforms Qwen2.5-VL-7B on five of seven benchmarks and surpasses LVR-7B on Relative Depth.
  • Advantage in Complex Visual Reasoning: 78.23% on Relative Depth and 32.0% on IQ-Test outperform GPT-4o’s 64.52% and 30.0%, respectively, on these BLINK tasks.LaViT-3B also exceeds LVR-7B by +1.62% on Relative Depth and +3.0% on Relative Reflectance.
  • Fine-Grained Perception and Robustness: 67.33% on MMVP and 54.07% on MMStar exceed reported baselines, supporting improved fine-grained perception and multimodal robustness.MMVP exceeds DMLR’s 61.33% and PAPO’s 50.0%; MMStar exceeds the 50.2% baseline.
  • In-depth Analysis of Attention Dynamics: LaViT reduces mean attention entropy from 4.870 in the Base 3B model to 4.686, approaching the Teacher’s 4.284 and focusing on critical depth markers.The analysis uses entropy to measure visual-focus sharpness and salient regions to characterize active visual processing.
  • In-depth Analysis of Attention Dynamics: LaViT’s attention stability improves to CV = 0.102 versus the Teacher’s 0.392, while Top-N sparsification filters low-confidence attention patterns.The resulting student remains focused on critical regions with less variance than the teacher.
  • Ablation Study: Removing trajectory alignment or semantic reconstruction causes significant performance drops, while removing curriculum gating lowers MMVP accuracy to 59.33%.Single-stage training also underperforms, including 38.81% versus 45.52% on Relative Reflectance.

6 Conclusion

The paper identifies a Perception Gap in multimodal distillation and introduces LaViT to align latent visual thoughts, including visual semantics and attention patterns, before response generation.

  • LaViT addresses the Perception Gap, where students mimic textual outputs without inheriting teachers’ visual attention patterns.
  • LaViT combines White-box Trajectory Distillation with Curriculum Sensory Gating to align latent visual thoughts.
  • LaViT-3B reconstructs teachers’ visual semantics and gaze through latent tokens before generating responses.
  • LaViT-3B significantly outperforms SFT baselines and rivals 7B-scale models on reasoning-intensive BLINK and MMVP benchmarks.

A Implementation and Reproducibility Details

The implementation section describes reproducibility materials and a benchmark design spanning baseline paradigms, model scales, and training configurations.

  • Reproducibility: The authors released code, pretrained models, and the LaViT-SFT-15k dataset in an anonymous repository.
  • Baselines: Evaluation compares LaViT with state-of-the-art MLLMs across three paradigms to isolate latent reasoning, scale, and data-exposure effects.
  • Baselines: Naive-SFT uses the identical LaViT-SFT-15k dataset with standard text-only supervision as a controlled baseline.
  • Baselines: Comparisons include explicit textual or reinforcement-learning approaches such as R1-OneVision and PAPO.
  • Baselines: Latent-space comparisons include LVR on a larger 7B backbone and DMLR with test-time latent optimization.
  • Training configuration: Table 4 reports hyperparameters for the 1000-step training run.

A.3 Computational Cost and Efficiency

LaViT adds modest latent-generation latency but achieves faster text decoding and a competitive total inference time than the compared reasoning frameworks.

  • Measurement setup: The comparison measures clock-time with batch size 1 on a single NVIDIA RTX Pro 6000 96GB GPU.
  • Latency: 91.26 ms is LaViT’s latent-generation overhead, below LVR’s 156.83 ms and DMLR’s 956.62 ms.
  • Latency: 15.61 ms/token is LaViT’s text-generation speed, faster than the base model at 28.83 ms/token and text-only SFT at 42.47 ms/token.
  • Efficiency: LaViT’s compressed continuous latent tokens reduce KV-cache burden during autoregressive decoding.
  • Efficiency: 0.143 seconds is LaViT’s average total inference time on the reported comparison.

A.4 Evaluation Protocols

The evaluation protocol uses VLMEvalKit to standardize comparisons across the reported vision-language benchmarks.

  • All benchmark evaluations use the open-source VLMEvalKit suite.
  • VLMEvalKit standardizes prompt formatting, inference pipelines, and metric calculations across datasets.
  • The protocol is intended to reduce discrepancies caused by differing evaluation implementations.

B.1 Analysis of numbers of continuous latent tokens K

The ablation compares latent-token capacities K ∈ {4, 6, 8} across training steps and identifies K = 4 as the best balance between visual grounding and reasoning.

  • K = 4 yields the superior balance between visual grounding and reasoning capabilities.
  • At 1,000 training steps, K = 4 reaches 67.33 on MMVP and 32.0 on IQ-Test.

B.2 Data Scaling Efficiency

LaViT scales more effectively with unique training data than standard SFT under fixed-iteration training. LaViT improves across the evaluated benchmarks, while standard SFT saturates and can degrade on complex spatial tasks.

  • LaViT consistently improves performance across MMVP, Jigsaw, and Relative Depth as unique training samples increase from 3K to 15K.
  • Standard SFT plateaus at 65.33 on MMVP beyond 12K samples.
  • Under fixed iterations, repeated exposure to limited visual-text pairs causes standard SFT to severely overfit.
  • By 15K samples, LaViT reaches 67.33 on MMVP and mitigates the data-scarcity bottleneck described for standard SFT.

C Training Data Construction

LaViT’s training data combines teacher visual features and attention trajectories with filtering for correctness, difficulty, and spatial alignment. The resulting dataset is curated from diverse multimodal sources to emphasize grounded visual reasoning.

  • Teacher supervision: The distillation data provides 5120-dimensional V-top tensors and compressed attention maps as teacher supervision signals.
  • Teacher supervision: Bilinear interpolation aligns V-top feature-map resolution with attention maps to preserve pixel-level correspondence across varying aspect ratios.
  • Latent supervision: Supervision is applied to the last latent token so preceding tokens must compress and structure visual information through the bottleneck.
  • Filtering: The dataset retains only teacher responses judged strictly correct against ground-truth answers.
  • Filtering: Text-only-solvable instances are discarded to ensure retained examples require genuine multimodal information.
  • Filtering: Samples pass visual-alignment filtering when teacher attention reaches at least τ = 0.20 inside target regions during one reasoning step.
Loading 2601.10129v2…