Source-linked AI summary
More Thought, Less Accuracy? On the Dual Nature of Reasoning in Vision-Language Models
Xinyu Tian, Shu Zou, Zhaoyuan Yang, Mengqi He, Fabian Waschkowski, Lukas Wesemann, Peter Tu, Jing Zhang
TL;DR
The paper investigates why reasoning does not consistently improve VLMs, finding that longer reasoning can weaken perceptual grounding through visual forgetting. It introduces VAPO, which anchors reasoning in visual evidence, and reports stronger visual reliance and state-of-the-art benchmark performance.
Problem
The paper asks whether reasoning is a consistent performance booster for VLMs despite evidence that multimodal reasoning may produce limited gains and degrade visual grounding.
Method
VAPO is a policy-gradient method that embeds visual anchors and perception rewards to steer reasoning toward visually grounded trajectories.
Results
VAPO-Thinker-7B strengthens visual reliance and achieves average gains of 2 ∼4% over strong baselines on established benchmarks.
Takeaways & Limitations
Longer reasoning may improve logical inference while causing visual forgetting, whereas visually anchored reasoning can preserve visual attention and improve accuracy during later reasoning stages.
Takeaways & Limitations
VAPO depends on the quality of GPT-generated visual claims, which may limit performance and create a potential ceiling for improvement.
Abstract
from arXiv · showhide
Reasoning has emerged as a pivotal capability in Large Language Models (LLMs). Through Reinforcement Learning (RL), typically Group Relative Policy Optimization (GRPO), these models are able to solve complex tasks such as mathematics and code generation. Building on these advances, recent research has sought to extend reasoning to Vision-Language Models (VLMs), yielding promising results across diverse visual tasks. Despite this progress, our study uncovers the dual nature of multimodal reasoning: while it substantially enhances logical inference and facilitates performance on challenging problems, it may gradually impair perceptual grounding, leading to recognition failures on otherwise basic visual questions. Through further analysis, we attribute this phenomenon to visual forgetting, wherein prolonged reasoning causes the model to increasingly disregard visual input. To address this, we propose Vision-Anchored Policy Optimization (VAPO), a simple yet effective method that explicitly steers the reasoning process toward visually grounded trajectories. Our result model, VAPO-Thinker-7B, significantly strengthens the model's reliance on visual information and achieves new state-of-the-art results on a wide range of established benchmarks. Project page: https://xytian1008.github.io/VAPO/
1 INTRODUCTION
The paper asks whether reasoning consistently improves VLM performance and finds a dual effect: early reasoning can help, but prolonged reasoning may weaken visual grounding. It attributes this degradation to visual forgetting and proposes VAPO to keep reasoning visually anchored.
- Longer reasoning can improve logical inference while eventually reducing perceptual accuracy on otherwise straightforward visual questions.The analysis compares direct answering, full reasoning, and early decision to track how reasoning length affects outcomes.
- Over 50% of failure cases are perception errors, and many can be corrected by stopping reasoning earlier.Early decision exposes cases where the model initially has enough visual information but is later misled by extended reasoning.
- Visual forgetting emerges as visual attention declines during reasoning, causing models to rely less on visual input and suffer performance degradation.Visual replay and focus prompts alleviate this degradation, supporting visual forgetting as a central constraint on multimodal reasoning.
- VAPO uses visual anchors to guide reasoning toward visually grounded trajectories and strengthen dependence on visual information.The method is presented as a policy-gradient approach intended to address the behavioral deficiency without the inference overhead of replay or focus prompts.
- Average gains of 2 ∼4% over strong baselines establish VAPO-Thinker-7B as a new state of the art across established benchmarks.
2 RELATED WORK
Prior work extends language-model reasoning to VLMs through supervised and reinforcement-learning approaches, while separate research identifies persistent text bias and visual forgetting. This paper connects extended reasoning with that modality imbalance and proposes VAPO as a training-based remedy.
- VLM reasoning research commonly combines high-quality supervised trajectories for cold-start training with subsequent reinforcement learning, or bypasses that sequence through alternative training strategies.
- Earlier VLMs already exhibited text bias, with responses guided predominantly by textual input while visual cues were often ignored.
- Extended step-by-step explanations can amplify visual forgetting, limiting the benefits of reasoning in multimodal settings.
- VAPO explicitly reinforces perceptual grounding and is reported to achieve new state-of-the-art results across established benchmarks.
3 THE DUAL NATURE OF REASONING
Reasoning initially improves VLM accuracy but can later reverse its gains as models lose perceptual grounding. This decline is linked to decreasing visual attention, while replaying visual input partially restores performance.
- 3.1 NO-FREE-LUNCH DILEMMA: LOGIC VS. PERCEPTION: Reasoning gains plateau and can reverse as reasoning progresses from direct answering to full reasoning, causing accuracy to decline.Early decision enables control over reasoning length and exposes this dual effect.
- 3.1 NO-FREE-LUNCH DILEMMA: LOGIC VS. PERCEPTION: 55.23% of errors are perception errors versus 33.05% logical errors, and 32.35% of perceptual errors are recoverable through early decision.The recoverable cases indicate that prolonged reasoning can mislead models that were initially on the right path.
- 3.1 NO-FREE-LUNCH DILEMMA: LOGIC VS. PERCEPTION: Perception errors dominate failures, especially on vision-heavy benchmarks with high-resolution or perceptually elusive images.MMStar and HallusionBench show particularly high perception-error proportions and recoverability through early decision.
- 3.2 SIDE EFFECT OF REASONING: VISUAL FORGETTING: Visual-token attention declines toward negligible levels during vanilla reasoning, indicating visual forgetting as decisions become driven by prior thoughts.The analysis tracks attention across generation steps to explain the perceptual decline.
- 3.2 SIDE EFFECT OF REASONING: VISUAL FORGETTING: Visual replay and focus prompts temporarily increase visual attention, while visual replay ultimately outperforms vanilla reasoning by around 1.5%.The interventions insert the image or a reminder at approximately four logical positions in the reasoning trajectory.
4 THE PROPOSED METHOD
The method replaces standard multimodal GRPO with VAPO, which inserts visual claims as intermediate perceptual checks during reasoning. Its perception reward emphasizes later reasoning stages and is gated by accuracy to reduce reward hacking.
- 4.1 PRELIMINARY: GRPO compares generated candidate sequences, using normalized group rewards and clipped policy updates to encourage high-quality responses while stabilizing optimization.The reference-policy penalty and clipping bounds constrain policy deviation and support on-policy training.
- 4.2 VISION-ANCHORED POLICY OPTIMIZATION: VAPO replaces multimodal GRPO with a visually grounded policy-optimization approach that inserts correct and incorrect visual claims throughout reasoning.The claims function as visual anchors for assessing perceptual capability at different stages.
- 4.2 VISION-ANCHORED POLICY OPTIMIZATION: VAPO evaluates each sampled claim with a binary correctness judgment against its reference label, producing per-anchor perceptual scores.Claims are sampled from pools of balanced, independent visual statements.
- 4.2 VISION-ANCHORED POLICY OPTIMIZATION: The perception reward aggregates anchor scores with greater weight on later anchors because perceptual capability tends to decline as reasoning progresses.The hyperparameter β controls the emphasis assigned to later anchors.
- 4.2 VISION-ANCHORED POLICY OPTIMIZATION: The final reward adds accuracy, format, and perception terms, but applies the perception reward only when the accuracy reward equals one.This accuracy condition is intended to prevent reward hacking through trivially short reasoning paths.
5 EXPERIMENT AND RESULTS
VAPO-Thinker improves performance across mathematical and general-purpose benchmarks while preserving visual attention and accuracy during extended reasoning. Analyses attribute these gains to visually anchored training, with performance benefits increasing with suitable anchor placement and weighting.
- 5.1 MAIN RESULTS: 3.2%: VAPO-Thinker-7B improves general-purpose benchmark accuracy from 59.9% to 63.1%, establishing a new state of the art among 7B baselines.On mathematical benchmarks, it improves average accuracy by 2%, from 49.1% to 51.1%.
- 5.1 MAIN RESULTS: VAPO remains substantially stronger than baselines augmented with focus prompts or visual replay, indicating that inference-level remedies alone are insufficient.Table 3 compares VAPO with strong baselines equipped with these test-time strategies.
- 5.2 FURTHER DISCUSSION: During reasoning, VAPO sustains higher visual attention and steadily increasing accuracy, whereas the baseline’s accuracy declines sharply in later steps.Figure 5 tracks visual attention and average accuracy across ten benchmarks using early decisions.
- 5.2 FURTHER DISCUSSION: Perception reward rises alongside accuracy during training, indicating progressively stronger visual-input use and visual-claim verification.An initial reward decline is attributed to early emphasis on accuracy and format alignment.
- 5.2 FURTHER DISCUSSION: Average accuracy improves rapidly as anchor count increases and approaches saturation around K = 20, the default configuration.The K = 0 setting reduces to vanilla GRPO training.
- 5.2 FURTHER DISCUSSION: Performance peaks at β = 1.5, concentrating about 50% of total anchor weight on the last 30% of anchors.β controls the relative emphasis placed on later anchors.
6 CONCLUSION
The paper characterizes multimodal reasoning as beneficial for logical inference but potentially harmful to perceptual grounding as reasoning length increases. It identifies visual forgetting and proposes VAPO to restore visual anchoring, achieving state-of-the-art benchmark performance.
- Multimodal reasoning improves logical inference and challenging-problem performance but can impair recognition on basic visual questions as reasoning length grows.
- The paper attributes this decline to visual forgetting, in which prolonged reasoning causes the model to disregard visual information.
- VAPO steers reasoning along trajectories anchored in visual evidence, and VAPO-Thinker-7B establishes a new state of the art across varied benchmarks.
A.1 BENCHMARK STATISTICS
The evaluation spans ten benchmarks covering mathematical, logical, general-purpose, and vision-intensive multimodal tasks. Results use specified test, validation, or full-test splits, with several comparisons against reproduced or leaderboard baselines.
- Benchmark splits: Evaluation uses benchmark-specific subsets or full test sets, including MathVerse Test Mini, MathVista Test Mini, MMMU validation, and full-test evaluations for MathVision and MMStar.
- Baseline settings: Baseline comparisons include GPT-5-Thinking, Gemini-2.5-Pro, InternVL2.5, R1-OneVision, VLAA-Thinker, and Vision-R1, using leaderboard or reproduced results with greedy decoding.
- Baseline settings: VLAA-Thinker is an RL-only 7B model based on Qwen2.5-VL-7B, while R1-OneVision uses a 7B checkpoint trained with SFT followed by RL.
A.3 ERROR ANALYSIS
The error analysis categorizes reasoning failures with GPT-5 and audits their co-occurrence, while the appendix documents visual-claim generation and inference-level remedies used to study visual forgetting. The examples emphasize distinctions between perceptual and logical errors.
- Error categorization: GPT-5 categorizes failure traces into three predefined error types plus Others, and the study quantifies their distribution.The prompt is shown in Figure 8.
- Error co-occurrence: Multiple error types occur in fewer than 20 audited examples, or under 4%, while logical errors preceding perceptual errors are extremely rare.This supports identifying a primary error category for most responses.
- Inference-level remedies: Visual replay periodically reinserts a downsampled image, while focus prompts ask the model to verify against visual input at aligned reasoning positions.
- Visual-claim construction: Visual claims are filtered for length, ambiguity, correctness balance, and redundancy before use in perception-reward training.Claims are dynamically generated and resampled until the target number of valid claims is reached.
A.7 TRAINING CONFIGURATION
The training configuration combines ViRL39K data, visual-anchor scoring during rollouts, and GRPO-based optimization. Ablations examine perception-reward weighting, anchor placement, claim-example augmentation, and related training choices.
- Training setup: VAPO training uses ViRL39K, five sampled responses per example, visual anchors inserted into reasoning traces, and binary claim judgments for perceptual scoring.The default training dataset spans STEM, social, chart, and spatial-relation domains; each anchor evaluates a sampled reasoning prefix against a visual claim.
- Training setup: The configuration uses GRPO loss with ϵ = 0.2 and λ = 1e−2, while VAPO sets K = 20, β = 1.5, and γ = 0.1.Training runs on 8 NVIDIA A100-80G GPUs using verl.
- Claim design: Non-visually-dependent and question-aligned claims perform substantially worse than the default visual-claim setup.The comparison supports using claims that require direct visual verification rather than external knowledge or question-correlated reasoning.
- Perception reward weight: Increasing the perception-reward weight improves average benchmark accuracy up to approximately γ = 0.1, while mathematical tasks favor smaller values.The experiment uses a 5000-example subset rather than the full training set.
B.4 ABLATION ON DATA AUGMENTATION
The ablations distinguish VAPO’s visual-grounding mechanism from generic data augmentation and attention maximization. Results favor perceptual claims as training anchors and show meaningful accuracy gains with modest computational overhead.
- Data augmentation: Visual-claim data augmentation improves vanilla GRPO only marginally from 49.0 to 49.3, whereas VAPO improves performance from 49.0 to 51.0.The authors attribute the difference to visually simple claims and repeated images reducing augmentation diversity.
- Claim variants: Non-visually-dependent and question-aligned claims perform substantially worse than default claims because they weaken direct perceptual verification.The former can be solved without strong perception, while the latter can be inferred from question-related reasoning.
- Attention-based reward: Maximizing visual attention directly provides little or no gain and can degrade performance on some tasks, unlike the perception reward.Attention indicates visual-input contribution but does not directly measure effective visual-feature use.
- Efficiency and cost: VAPO improves accuracy by 4.37% over GRPO with only around 30 minutes of additional training, while DAPO requires approximately 6 extra hours.The comparison uses the same data budget and training epochs.
- Efficiency and cost: The anchor-scoring process is efficient because each visual claim requires only a binary yes-or-no judgment rather than a full reasoning rollout.This single-token evaluation explains the limited training overhead relative to GRPO.
B.10 LIMITATION ANALYSIS
The limitation analysis identifies claim quality, task-specific hyperparameter sensitivity, and single-image evaluation as important boundaries. Additional experiments nevertheless report reliable claims, transfer across base models, and gains beyond the original setting.
- Visual claim quality: Visual-claim quality may limit VAPO because claims must be vision-dependent, verifiable, and correctly labeled.The authors suggest that stronger model-generated or human-annotated claims could raise the performance ceiling.
- Hyperparameter sensitivity: VAPO’s hyperparameters require task-dependent tuning: vision-intensive tasks favor larger β and γ, whereas mathematical reasoning often favors smaller values.The authors identify adaptive parameter selection as future work.
- Single-image setting: The current study focuses exclusively on single-image tasks, leaving multi-image and video reasoning as broader evaluation settings.The paper describes extension to multiple frames or views as future work.
- Claim and label quality: Human evaluation finds approximately 2% claim noise and below 4% labeling error, with model and human assessments closely aligned.The results support the reliability of the generated claims and labels while leaving room for further improvement.
- Other base models: Applying VAPO to Qwen3-VL yields a 3.9% average gain over GRPO, from 64.9% to 68.8%, indicating effectiveness beyond Qwen2.5-VL.The experiment tests transfer to a substantially different reasoning-model architecture.
B.17 DATA DEDUPLICATION CHECK
The appendix checks dataset overlap, gating and anchor-placement choices, baseline comparisons, reasoning length, and multi-image transfer. These analyses report limited overlap concerns, favor random anchors, and show gains without noticeably lengthening reasoning.
- Data deduplication: The appendix reports training–evaluation overlap checks for mathematical and general-purpose benchmarks, alongside full numerical benchmark results and ablation tables.These materials document the deduplication and supplementary evaluation procedures but do not provide overlap values in the supplied passage.
- Anchor placement: Random anchor placement outperforms uniform, front-loaded, and back-loaded alternatives, with later reasoning positions appearing important.Uniform placement is more susceptible to reward hacking, while front-loaded placement performs worse than random and back-loaded placement.
- Reasoning length: The model’s average reasoning length does not differ noticeably from other baselines, indicating that VAPO targets perceptual effectiveness rather than trace length.The paper reports no explicit length regularization.
- Multi-image evaluation: VAPO improves multi-image performance over GRPO by 2.5% on average, from 58.7% to 61.2%, despite training on approximately 2,500 examples.Video-based tasks remain unevaluated in this pilot.
D.4 FULL RESULTS OF AUGMENTED BASELINES
This section provides full numerical results for the main experiments, ablations, and baselines augmented with inference-level test-time remedies.
- Full numerical results for the main experiments are reported in Table 28, corresponding to Tables 1 and 2.
- The ablation study of K is reported numerically in Table 29, corresponding to Fig. 7(A).
- The ablation study of β is reported numerically in Table 30, corresponding to Fig. 7(B).
- Table 31 reports full results for baselines augmented with test-time remedies, corresponding to Table 3.