PaperlayerView exact revision on arXiv

Source-linked AI summary

Look Again, Think Slowly: Enhancing Visual Reflection in Vision-Language Models

Pu Jian, Junhong Wu, Wei Sun, Chen Wang, Shuo Ren, Jiajun Zhang

arXiv:2509.12132v1cs.CVcs.CL

TL;DR

Current visual reasoning models often lose visual attention as responses grow, limiting visual reflection. Reflection-V addresses this with vision-centered cold-start reasoning data and visual-attention-based RL, improving benchmark performance while sustaining reliance on visual information.

  • Problem

    Existing visual reasoning models show limited visual reflection because their attention to visual information declines rapidly during longer responses.

  • Method

    Reflection-V combines interaction-driven vision-centered reasoning data for cold-start learning with a visual attention-based reward model for reinforcement learning.

  • Results

    Reflection-V significantly improves performance across mathematical, multi-disciplinary, and general visual reasoning benchmarks, with a 7B model comparable to or surpassing several larger VLMs.

  • Takeaways & Limitations

    Reflection-V maintains stronger and more consistent visual reliance during reasoning, supporting visual reflection as a basis for improved visual reasoning.

  • Takeaways & Limitations

    Experiments cover only the 3B and 7B scales and relatively limited visual-language dataset categories.

Abstract

from arXiv · show

Recent advances in text-only "slow-thinking" reasoning have prompted efforts to transfer this capability to vision-language models (VLMs), for training visual reasoning models (\textbf{VRMs}). owever, such transfer faces critical challenges: Effective "slow thinking" in VRMs requires \textbf{visual reflection}, the ability to check the reasoning process based on visual information. Through quantitative analysis, we observe that current VRMs exhibit limited visual reflection, as their attention to visual information diminishes rapidly with longer generated responses. To address this challenge, we propose a new VRM \textbf{Reflection-V}, which enhances visual reflection based on reasoning data construction for cold-start and reward design for reinforcement learning (RL). Firstly, we construct vision-centered reasoning data by leveraging an agent that interacts between VLMs and reasoning LLMs, enabling cold-start learning of visual reflection patterns. Secondly, a visual attention based reward model is employed during RL to encourage reasoning based on visual information. Therefore, \textbf{Reflection-V} demonstrates significant improvements across multiple visual reasoning benchmarks. Furthermore, \textbf{Reflection-V} maintains a stronger and more consistent reliance on visual information during visual reasoning, indicating effective enhancement in visual reflection capabilities.

1 Introduction

Existing VRMs often transfer textual reflection into visual reasoning without checking the image, while Reflection-V trains visual reflection through interleaved LLM-VLM reasoning data and visual-attention rewards. Reflection-V improves visual reasoning across multiple benchmark categories and sustains visual reliance during reasoning.

  • Motivation: Existing VRMs can degrade into text-only reflection because transferred reflection patterns are detached from visual inputs.Visual reflection requires actively verifying and refining reasoning against the visual input.
  • Method: Reflection-V constructs cold-start data through an agent that interleaves LLM and VLM reasoning, keeping visual information repeatedly accessible during reasoning.This construction introduces visual reflection patterns into training data.
  • Method: A visual-attention reward in GRPO encourages Reflection-V to consistently attend to visual information during reinforcement learning.The reward is designed to reinforce the visual reflection behavior learned during cold-start training.
  • Results: Reflection-V improves mathematical, multi-disciplinary, and general visual reasoning benchmarks and can match or surpass larger widely used VLMs at 7B scale.The reported comparisons include GPT-4o and InternVL2.5-38B.
  • Results: Compared with the base model, Reflection-V maintains more sustained visual attention and reliance, indicating visual reflection during reasoning.The paper characterizes this sustained reliance as the emergence of a visual-reasoning “aha moment.”

2 VRMs Struggle with Visual Reflection

The analysis measures visual-token attention and visual dependency during generation, finding that both decline sharply as reasoning continues. RL-enhanced models do not resolve this decay and may rely even less on visual tokens than their base models.

  • Visual Attention Weight: Visual-token attention is measured from response tokens to visual tokens as more response tokens are generated.The analysis tracks how visual-token contribution changes during reasoning.
  • Visual Dependency Measure: Visual dependency is estimated by dropping visual tokens after several generated tokens and measuring divergence between subsequent next-token distributions.The divergence uses Hellinger distance between generations with and without visual tokens.
  • Findings: After roughly 300 generated tokens, visual attention falls to 20–30% of its initial level, while visual dependency also declines sharply.The trends are measured on MMMU using layer-wise attention and visual dependency.
  • Findings: OpenVLThinker, an RL-based Qwen2.5-VL model, shows lower visual-token focus and dependence during reasoning than the base model.Thus, widely used RL improves reasoning performance without equipping VRMs with visual reflection in this analysis.

3 Method

The method trains VRMs in two stages: interaction-based reasoning data provides visual reflection patterns, while reinforcement learning adds a visual-attention reward to sustain reliance on visual tokens.

  • 3.1 Reasoning Data with Visual Reflection Construction: LLM-VLM interaction continuously acquires and reuses visual information, creating reasoning data with a visual reflection pattern.The process includes visual requesting, visual responding, and subsequent reasoning-context updates.
  • 3.1 Reasoning Data with Visual Reflection Construction: Post-generation filtering removes samples solved after the first interaction, while cohesion enhancement refines text across interaction rounds.These steps aim to preserve sustained visual reliance and produce coherent reasoning processes.
  • 3 Method: Reflection-V combines cold-start training on interactive visual-reflection data with GRPO using a visual-attention reward.The data construction and reward design target sustained use of visual information during reasoning.
  • 3.2 Visual Attention Based Reward: The visual-attention reward is higher when relatively greater attention to visual tokens remains after several reasoning tokens are generated.The reward uses average response-to-visual-token attention and is calculated from the last layer where visual attention is most significant.
  • 3 Method: Table 1 compares Reflection-V with existing VLMs across visual reasoning benchmarks, marking best and second-best open-source scores.The table spans multiple visual reasoning benchmarks and distinguishes reproduced results and Qwen2.5-7B-series reasoning models.
  • 3.2 Visual Attention Based Reward: The overall GRPO reward is a weighted sum of accuracy, visual-attention, and format rewards, with visual-attention and format coefficients set to 0.5 and 0.1.The visual-attention term preserves visual reliance while the other terms retain accuracy and formatting objectives.

4 Experiments

Reflection-V is evaluated across visual reasoning benchmarks, ablations, and analyses of visual attention and reflection. It improves benchmark performance while sustaining reliance on visual information during longer reasoning.

  • 4.2 Main Result: Reflection-V significantly outperforms Qwen2.5-VL and similarly scaled open-source models across math, multi-disciplinary, and general visual reasoning benchmarks.It also achieves notable gains over existing RL-based visual reasoning models.
  • 4.2 Main Result: Reflection-V-7B surpasses or matches larger VLMs across several benchmarks, including GPT-4o, InternVL-2.5-38B, LLaVA-OneVision-72B, and GPT-4V.The reported comparisons vary by benchmark: it outperforms some models, matches others, and is comparable to InternVL-2.5-38B on MMMU-Pro.
  • 4.2 Main Result: Reflection-V shows significantly fewer visual hallucinations than the Qwen2.5-VL base model, while other VRMs exhibit more severe hallucinations than the base model.The paper links this result to sustained focus on visual information during reasoning.
  • 4.3 Ablation Study: Both cold-start visual-reflection data and visual-attention reward improve VRM performance, with cold-start producing a particularly substantial gain.The reward improvement becomes more pronounced after cold-start initialization.
  • 4.3 Ablation Study: Cold-start data containing visual-reflection patterns outperforms image-caption-based reasoning data, indicating that the gain is tied to incorporating visual reflection patterns rather than only distilling larger-model outputs.The comparison uses the same origin data for the two cold-start constructions.
  • 4.4 Analyses: Reflection-V maintains higher visual-token attention and slower attention decay than OpenVLThinker-7B as generated-token count increases.Its visual dependency also remains consistent during generation, and examples show renewed visual attention during textual “Aha moments.”

5 Related Works

Related work studies visual reasoning models and methods for reducing visual forgetting during long-chain inference. These approaches seek to preserve the influence of visual cues while balancing reasoning performance.

  • Visual Reasoning Models: Large VLMs project non-text inputs into textual representations that language models can process, supporting strong vision understanding.The related-work discussion situates visual reasoning models within this multimodal architecture.
  • Visual Forgetting Alleviation: Visual-forgetting methods such as M3ID amplify image influence and weaken linguistic priors, but the cited discussion notes that this can reduce performance on complex reasoning tasks.The trade-off motivates distinguishing sustained visual reliance from simply suppressing linguistic information.

6 Conclusion

The paper defines visual reflection as verifying and refining reasoning from visual input and introduces a two-stage strategy to strengthen it in VRMs. The strategy improves multiple benchmarks while sustaining visual attention and reliance during reasoning.

  • Visual reflection is defined as actively verifying and refining reasoning based on visual input, addressing a weakness identified in existing VRMs.
  • The proposed two-stage strategy combines LLM-VLM interaction-driven reflective reasoning patterns with visual-attention-based reinforcement learning.
  • The strategy significantly improves performance across multiple benchmarks while maintaining sustained visual attention and reliance during reasoning.

Limitations

The study is limited to 3B and 7B models and a relatively narrow set of visual-language dataset categories, constraining evaluation of broader generalization.

  • Experiments are limited to the 3B and 7B parameter scales because larger-model GRPO experiments were computationally infeasible.
  • The training initialization, reinforcement learning, and evaluation use relatively limited categories of visual-language datasets.
  • Future work will expand dataset coverage across diverse problem types to evaluate generalization more broadly.

A Implementation Details and Hyperparameters

Implementation uses separate requester, responder, and summarizer prompts to elicit visual information, structured reasoning, and coherence-repair behavior during training and evaluation.

  • Training uses supervised fine-tuning with a 1e−5 learning rate, cosine scheduling, BF16 precision, batch size 8, and three epochs, followed by RL with 2048-token limits and KL coefficient 1e−2.
  • Prompt Templates of Visual Requester: The visual requester asks a Vision Expert targeted image questions while receiving the current problem information.Its prompt explicitly gives an example of asking about an image property such as a bird’s color.
  • Prompt Templates of Visual Responder: The visual responder is instructed to describe relevant image information concisely rather than answer yes/no questions directly.
  • Prompt Templates of Summarizer: The summarizer fills gaps in disjointed reasoning and may use reflective connectors such as checking the image again or double-checking.
  • The prompts require explicit thought and final-answer formats, including boxed answers and, in one template, <think> tags.

Prompt Templates of RL Training and Evaluation

The RL-training and evaluation templates require internal reasoning before boxed answers, while the supporting datasets are documented in composition tables.

  • Prompt Templates of RL Training and Evaluation: The reasoning template requires an internal monologue enclosed in <think> tags before the final answer is placed in \boxed{}.
  • Prompt Templates of RL Training and Evaluation: Table 7 documents the dataset composition used to construct visual-reflection reasoning data for cold-start initialization.
  • Prompt Templates of RL Training and Evaluation: Table 8 documents the dataset composition used to conduct GRPO.

C Data Resources

The authors collect multimodal-corpus data for both cold-start reasoning-data construction and GRPO training.

  • C Data Resources: Multimodal-corpus data support two stages: constructing visual-reflection reasoning data for cold-start initialization and training with GRPO.The data uses are summarized in Tables X and Y.

D Supplementary Experiments

Supplementary experiments examine sustained visual attention and visual dependency across additional benchmarks, scaling to InternVL3-14B, and visual-reflection case studies. Reflection-V shows stronger sustained reliance on visual information than the compared baselines and outperforms text-only-reward GRPO in the scaling experiment.

  • Sustained Visual Attention: Reflection-V-7B maintains stronger visual attention than Qwen2.5VL-7B across three additional visual-reasoning benchmarks, with attention tracked over 500 generated tokens.Figure 6 reports last-layer attention from response tokens to visual tokens and emphasizes slower decay for Reflection-V-7B.
  • Visual Dependency: Reflection-V’s visual dependency remains nearly flat across tasks, whereas the baselines show a pronounced downward trend.Figure 7 uses the Visual Dependency Measure, comparing next-token probability distributions before and after visual-token removal.
  • Scaling Experiment: On InternVL3-14B with LoRA and 12 GRPO epochs, the proposed method significantly outperforms GRPO using only textual-output rewards.The scaling experiment evaluates several visual-reasoning benchmarks, with results reported in Table 9.
  • Case Studies: In the case studies, Reflection-V rechecks the image, finds visual information absent from the generated reasoning context, and infers the correct answer.Figures 8–10 present three cases illustrating this visual-reflection behavior.
Loading 2509.12132v1…