PaperlayerView exact revision on arXiv

Source-linked AI summary

From Narrow to Panoramic Vision: Attention-Guided Cold-Start Reshapes Multimodal Reasoning

Ruilin Luo, Chufan Shi, Yizhen Zhang, Cheng Yang, Songtao Jiang, Tongkun Guan, Ruizhe Chen, Ruihang Chu, Peng Wang, Mingkun Yang, Yujiu Yang, Junyang Lin, Zhibo Yang

arXiv:2603.03825v1cs.CVcs.AI

TL;DR

Cold-start initialization is underexplored in multimodal reasoning, particularly why multimodal cold-start yields marginal gains despite text-only cold-start improvements. The paper introduces VAS and AVAR to analyze and reshape visual attention, achieving a 7.0% average gain across seven benchmarks.

  • Problem

    The cold-start initialization stage of multimodal reasoning remains underexplored, and the limited gains from multimodal cold-start lack a clear quantitative explanation.

  • Method

    The paper introduces VAS to measure visual-token attention and AVAR to reshape it through visual-anchored synthesis, attention-guided objectives, and reward shaping.

  • Results

    7.0% average gain across 7 multimodal reasoning benchmarks is achieved by AVAR-Thinker over baseline Qwen2.5-VL-7B.

  • Takeaways & Limitations

    Training-free attention modulation yields consistent 1–2% gains, providing causal evidence that visual attention allocation is decisive for multimodal reasoning.

Abstract

from arXiv · show

The cold-start initialization stage plays a pivotal role in training Multimodal Large Reasoning Models (MLRMs), yet its mechanisms remain insufficiently understood. To analyze this stage, we introduce the Visual Attention Score (VAS), an attention-based metric that quantifies how much a model attends to visual tokens. We find that reasoning performance is strongly correlated with VAS (r=0.9616): models with higher VAS achieve substantially stronger multimodal reasoning. Surprisingly, multimodal cold-start fails to elevate VAS, resulting in attention distributions close to the base model, whereas text-only cold-start leads to a clear increase. We term this counter-intuitive phenomenon Lazy Attention Localization. To validate its causal role, we design training-free interventions that directly modulate attention allocation during inference, performance gains of 1$-$2% without any retraining. Building on these insights, we further propose Attention-Guided Visual Anchoring and Reflection (AVAR), a comprehensive cold-start framework that integrates visual-anchored data synthesis, attention-guided objectives, and visual-anchored reward shaping. Applied to Qwen2.5-VL-7B, AVAR achieves an average gain of 7.0% across 7 multimodal reasoning benchmarks. Ablation studies further confirm that each component of AVAR contributes step-wise to the overall gains. The code, data, and models are available at https://github.com/lrlbbzl/Qwen-AVAR.

1 INTRODUCTION

The introduction identifies multimodal cold-start as an ineffective bottleneck and attributes it to insufficient visual attention. It then presents VAS, training-free attention interventions, and AVAR as progressively stronger responses, culminating in substantial benchmark gains.

  • Motivation and Problem: Multimodal cold-start provides only marginal gains, unlike text-only cold-start, because MLRMs fail to leverage multimodal signals efficiently.This bottleneck limits the potential of reinforcement learning for multimodal reasoning.
  • Attention Analysis: VAS quantifies attention to visual tokens, and reasoning performance strongly correlates with it across representative MLRMs (r = 0.9616).Models with higher VAS achieve stronger multimodal reasoning, while models with lower VAS perform worse.
  • Training-Free Interventions: Training-free inference interventions that amplify visual-token attention and reduce redundant system-token focus yield average improvements of 1–2% across Qwen2.5-VL-7B, Revisual-R1-CS, and OVR-CS.The gains occur without retraining and across models with different baseline performance levels.
  • AVAR Framework: AVAR reshapes cold-start attention through visual-anchored data synthesis, attention-guided objectives, and visual-anchored reward shaping.Its three-stage synthesis pipeline embeds visual anchors throughout reasoning and incorporates visual reflection into generated data.
  • Experimental Results: 7.0% is the average gain achieved by AVAR-Thinker over baseline Qwen2.5-VL-7B across 7 multimodal reasoning benchmarks.The strongest improvements are MathVision (+12.2%) and HallusionBench (+8.8%); ablations validate each pipeline component.

2 RELATED WORKS

Related work on multimodal large reasoning models emphasizes cold-start thinking data and reinforcement learning, while studies of attention identify insufficient visual-token focus as a persistent bottleneck. However, existing work has not fully explained how reasoning data produces these effects or investigated multimodal reasoning-with-image data in depth.

  • Multimodal reasoning models: MLRMs target reasoning tasks in multimodal settings, including STEM and perception-related tasks.The cited work frames MLRMs around both STEM problems and perception-related tasks.
  • Cold-start and reinforcement learning: Recent research improves cold-start thinking-data curation and explores reinforcement-learning-based approaches for MLRMs.These efforts span multimodal cold-start data construction and RL-based training methods.
  • Cold-start and reinforcement learning: High-quality unimodal thinking data can substantially improve MLRM reasoning, but prior work does not uncover the mechanism or examine multimodal reasoning-with-image data.The passage identifies a clear empirical effect alongside an unresolved mechanism and an underexplored multimodal-data setting.
  • Visual attention in multimodal models: Attention studies find that inappropriate visual-token allocation remains a bottleneck, with insufficient visual attention and over-reliance on language priors.Yin et al. report that modality fusion occurs predominantly in the middle layers, while models under-attend to visual signals.

3 COLD START RESHAPES ATTENTION ALLOCATION

This section introduces Visual Attention Score (VAS) and shows that stronger visual-token attention predicts better multimodal reasoning. It identifies Lazy Attention Localization: multimodal cold-start leaves visual attention weak, whereas text-only initialization reshapes attention toward stronger visual grounding.

  • Visual Attention Score: VAS measures attention to visual tokens relative to system tokens during multimodal reasoning.Model-level VAS averages attention across heads, layers, and user query tokens.
  • VAS and Reasoning Performance: 0.9616 Pearson correlation links VAS with multimodal reasoning performance.Models with higher visual-token attention consistently achieve stronger reasoning performance.
  • VAS and Reasoning Performance: VAS below 10 identifies Narrow-View Models that devote minimal attention to visual features and consistently underperform.Examples include Qwen2.5-VL-7B-Instruct, R1-OneVision, ThinkLite-VL, and MM-Eureka.
  • Cold-Start Effects: 15–20% higher visual-feature attention is maintained by text-only cold-start models than by multimodal cold-start models.The comparison includes OVR-CS and Revisual-R1-CS versus R1-OneVision and ThinkLite-VL.
  • Cold-Start Effects: Multimodal cold-start leaves Qwen2.5-VL-7B and R1-OneVision with nearly identical, persistently weak visual-token attention, unlike text-only-initialized OVR-CS.This counter-intuitive pattern is termed Lazy Attention Localization and reflects a shift toward stronger visual grounding induced by text-only initialization.

4 TRAINING-FREE ATTENTION ROLE IDENTIFICATION

This section shows that attention allocation can be manipulated directly during inference without retraining, with image-token emphasis improving reasoning while system-token attention is redundant.

  • Training-Free Attention Modulation: Training-free experiments directly manipulate attention weights during inference to test whether attention-allocation gains can be achieved without additional training.The intervention operates across all transformer layers and requires no model retraining or parameter updates.
  • Training-Free Attention Modulation: The method differentially scales image and system token categories by modifying the attention weight matrix during scaled dot-product attention.Enhancement and suppression masks control image and system tokens, with α_img and α_sys determining their relative attention importance.
  • Findings: Emphasizing image features strengthens reasoning capabilities, whereas system-token attention is redundant in reasoning tasks.Training-free attention modification improves performance on MathVista, MathVision, and MathVerse-VO.

5 ATTENTION-GUIDED VISUAL ANCHORING AND REFLECTION (AVAR)

AVAR is a cold-start framework that counters Lazy Attention Localization by reshaping attention allocation through visual-anchored data synthesis, attention-guided training, and visual-anchored reward shaping. Its pipeline sustains visual grounding by integrating explicit visual reflection into reasoning data, optimizing attention to visual tokens, and rewarding visual anchoring during reinforcement learning.

  • Framework overview: AVAR integrates 3 complementary components: visual-anchored reflection data synthesis, attention-guided objectives, and visual-anchored reward shaping.The framework explicitly incorporates the attention-reallocation mechanism identified by training-free interventions into cold-start training.
  • Visual-anchored reflection data synthesis: The data synthesis pipeline uses high-fidelity descriptions, reflection-enhanced reasoning, and explicit visual anchors to keep each reasoning step tied to the image.Gemini 2.5-Pro generates descriptions, Qwen3-235B-A22B produces reflective reasoning, and Qwen3-32B inserts references such as checking the image again.
  • Visual-anchored reflection data synthesis: The synthesized data makes visual anchoring arise naturally throughout reasoning, mirroring panoramic-view models that sustain high visual attention ratios.Visual reflection is embedded throughout the reasoning process rather than added through a caption-then-reason pipeline.
  • Attention-guided training: AVAR combines standard language modeling with image enhancement and system suppression losses to directly optimize attention allocation.The image enhancement loss promotes sustained attention to visual tokens, while system suppression reduces redundant attention to system tokens.
  • Visual-anchored reward shaping: The reinforcement-learning stage adds a visual attention reward based on attention assigned to visual tokens relative to system tokens, alongside accuracy and format signals.The shaped reward encourages correct answers, sustained visual grounding, and compliance with the required output structure, and is optimized with GRPO.

6 EXPERIMENT

AVAR-Thinker improves multimodal reasoning across mathematical, multidisciplinary, and perceptual benchmarks, achieving a 7.0% average gain over Qwen2.5-VL-7B. Ablations show stepwise benefits from visual-anchored data, attention-guided training, and visual-anchored reward shaping, alongside progressively higher VAS.

  • Overall Performance: 7.0% average gain over Qwen2.5-VL-7B, with improvements across mathematical, multidisciplinary, and perceptual reasoning benchmarks.Gains include MathVista +6.5%, MathVision +12.2%, MMMU +5.7%, MMMU-Pro +3.1%, and HallusionBench +8.8%.
  • Overall Performance: AVAR-Thinker establishes a new state of the art among 7B models, surpassing ThinkLite-VL by 3.0% and MM-Eureka by 4.9% on average.It matches Vision-R1 despite not being trained on MathVision and outperforms R1-OneVision and OpenVLThinker.
  • Ablation Study: VARD data synthesis alone yields +1.7%, including MathVision +7.7% and HallusionBench +4.6%, demonstrating the benefit of embedding visual anchors in reasoning chains.VARD consistently outperforms R1-OneVision (+6.4%), OpenVLThinker (+2.9%), and Vision-SR1 (+6.2%) on the same baseline model.
  • Ablation Study: AGTO adds cumulative improvements of +1.6%, with visual enhancement and system suppression losses synergistically reshaping attention distributions.The largest reported gains occur on MathVerse-VO (+2.9%) and MMMU-VAL (+3.1%).
  • Ablation Study: VARS completes the framework and achieves +6.8%, indicating that visual-attention rewards during RL help prevent reversion to text-only reasoning.This is the best performance among the reported ablation configurations.
  • VAS Evolution: VAS rises from 7.5 for Qwen2.5-VL-7B to 10.1 with VARD and 13.8 with attention-guided training, while average performance increases from 49.3% to 52.6%.The reported stages associate higher VAS with progressively improved average performance.

7 CONCLUSION

The work examines cold-start initialization in training multimodal large reasoning models and introduces VAS to quantify reliance on visual tokens. It identifies Lazy Attention Localization, in which conventional multimodal cold-start training fails to enhance visual attention.

  • The study investigates the role of cold-start initialization in training MLRMs.
  • VAS quantifies a model’s reliance on visual tokens and strongly correlates with multimodal reasoning performance.
  • Lazy Attention Localization describes conventional multimodal cold-start training failing to enhance visual attention.

ETHICS STATEMENT · Appendix · A LLM USABLE STATEMENT

The paper reports compliance with research-ethics requirements, describes safeguards for privacy and fairness, and discloses limited LLM assistance that was reviewed by the authors.

  • ETHICS STATEMENT: The study adheres to the ICLR Code of Ethics and involved neither human subjects nor animal experiments.
  • ETHICS STATEMENT: The datasets and models are widely adopted in the research community and contain no personally identifiable information or sensitive content.
  • ETHICS STATEMENT: The authors took deliberate steps to identify and mitigate potential biases in data selection, model training, and evaluation.These measures were intended to ensure fairness and avoid discriminatory outcomes.
  • A LLM USABLE STATEMENT: LLMs supported grammar correction, language refinement, and improvement of textual clarity during preparation of the work.
  • A LLM USABLE STATEMENT: LLMs also assisted with code debugging and synthetically generated small portions of data for preliminary experiments.
  • A LLM USABLE STATEMENT: All LLM-generated content was carefully reviewed, validated, and revised by the authors, who take full responsibility for its accuracy.

B GENERALIZATION EXPERIMENT

AVAR’s generalization experiments, using the same training dataset on Llama-3.1-Vision-Instruct, show robust transfer: each module delivers significant, consistent incremental improvements. Table 5 presents the corresponding ablation study on Llama-3.2-11B-Vision-Instruct.

  • Generalization Experiments: Table 5 reports an ablation study of AVAR’s proposed components on Llama-3.2-11B-Vision-Instruct.
  • Generalization Experiments: AVAR’s individual modules produce significant and consistent incremental improvements when evaluated on Llama-3.1-Vision-Instruct with the same training dataset.These results demonstrate the robust generalizability of the approach.

C DATA CURATION

The paper curates separate Cold-Start and RL datasets from multiple multimodal sources, totaling approximately 30.6K and 17.9K instances, respectively.

  • Cold-Start Dataset: The Cold-Start dataset combines five sources and totals approximately 30.6K instances.It includes R1-ShareVL (∼22.2K), Geo3K (∼2.1K), M3COT (∼3.2K), AlgoPuzzleVQA (∼1.8K), and SOLIDGEO (∼1.3K).
  • RL Dataset: The RL dataset combines four sources and totals approximately 17.9K samples.It includes R1-ShareVL (∼12.1K), Geo3K (∼2.1K), Super-CLEVER (∼2.2K), and AI2D (∼1.5K).

D FINE-GRAINED ATTENTION ANALYSIS OF DIFFERENT MODELS

Fine-grained attention analysis compares visual-feature, user-instruction, and system-prompt allocation across representative models on MathVista and MathVision. Multimodal thinking-data cold starts preserve attention behavior, whereas unimodal thinking-data cold starts reduce system-token attention and increase image focus.

  • Experimental setup: Attention distributions are analyzed across visual features, user instructions, and system prompts for representative models on MathVista and MathVision.Figures 4–5 report MathVista results, while Figure 6 reports MathVision results.
  • MathVista: R1-OneVision and ThinkLite-RL, trained on multimodal thinking data, do not alter attention distribution behavior.This pattern is shown in the MathVista analysis.
  • MathVista: RevisualR1-CS, OVR-CS, and MIMO-VL-CS reduce redundant system-token attention and redirect greater focus toward image information.These models are trained on high-quality unimodal thinking data.
  • MathVision: Vision-R1, trained on multimodal thinking data, fails to elicit the reflective attention mechanism that enhances visual focus on MathVision.The same attention-allocation pattern is observed in the MathVision results.

E EXPERIMENT SETUP … H BASELINE MODEL LIST

The appendix specifies RL hyperparameters, illustrates visual reflection in MathVerse-VO, details prompts for visual transcription and reasoning style transfer, and lists compared models with their repositories.

  • E EXPERIMENT SETUP: RL uses a 1 × 10−6 learning rate and 256 batch size to improve stability and prevent catastrophic forgetting.The setup also applies 3× 10−5 weight decay, KL coefficient 0.0, temperature 0.8, and rollout 8.
  • F CASE STUDY: MathVerse-VO demonstrates how reasonable visual reflection contexts can inspire a visual reflection pattern.
  • G PROMPT ENGINEERING: Prompt engineering uses high-fidelity visual descriptions, reflection-enhanced reasoning, and visual-anchor integration for mathematical and scientific problems.Gemini-2.5-Pro translates visual information, while Qwen3-235B-A22B-Thinking-2507 generates pseudo-multimodal reflection data.
  • G PROMPT ENGINEERING: The visual transcription prompt requires objective, complete recording of image facts to support reasoning and enable accurate reconstruction.It emphasizes acting as a visual scanner and data recorder without omitting relevant details.
  • G PROMPT ENGINEERING: Visual transcription proceeds from central or fundamental elements through iterative deconstruction, then synthesizes the complete scene for problem solving.Each element is described with its type, form, labels, and special visual cues before moving to the next.
  • 2. Full-Process Coverage & Integrity: Style transfer rewrites both Thinking content and Response while preserving every original reasoning step in its original order.The protocol explicitly forbids skipping or merging steps and requires complete dual-part coverage.
  • 3. Multimodal Authenticity: Multimodal enhancement replaces description-dependent wording with visual interaction while preserving reasoning, coordinates, formulas, proportions, variables, and conclusions exactly.Anchors must correspond to the structured image description, and post-transfer checks verify reasoning preservation, completeness, anchor accuracy, and multimodal authenticity.
  • H BASELINE MODEL LIST: Table 7 summarizes the compared models and provides their Hugging Face repositories.
Loading 2603.03825v1…