PaperlayerView exact revision on arXiv

Source-linked AI summary

Perception or Reasoning? Diagnosing and Tackling Reinforcement Learning Bottlenecks in VLMs

Zeyuan Yang, Zhuohao Chen, Jiaben Chen, Yao Zhang, Maohao Shen, Chuang Gan

arXiv:2610.21380v1cs.CV

TL;DR

Visual-task RL gains are less consistent than gains in language-model reasoning, and failures may reflect different bottlenecks across models on the same task. PARA measures recoverable perceptual and reasoning headroom, while GRAD uses group-relative attention distillation to guide weaker rollouts. The paper reports that RL gains track reasoning headroom more positively than perceptual headroom, and that GRAD improves performance across models and benchmarks.

  • Problem

    Existing analyses do not generally quantify how much improvement remains available from perception versus reasoning for a particular model–benchmark pair.

    Source: §1 · p. 1

  • Method

    PARA measures recoverable perceptual and reasoning headroom per model–benchmark pair, and GRAD uses the best rollout in each group as a visual-attention target for weaker rollouts.

    Source: §1 · pp. 1–2

  • Results

    RL gains decrease with perceptual headroom and increase with reasoning-related headroom, while GRAD consistently improves performance across two model scales and seven benchmarks.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The findings suggest that current RL exploits reasoning headroom more effectively than perceptual headroom, and identify visual attention as a more direct interface for optimizing perception.

    Source: §1 · p. 2

Abstract

from arXiv · show

Reinforcement learning has brought strong gains to language model reasoning, but its improvements on visual tasks remain less consistent. One difficulty is that visual task failures can arise from different sources: the same benchmark may be limited by perception for one model and by reasoning for another. We introduce Perception and Reasoning Attribution (PARA), which measures the remaining perceptual and reasoning headroom of each model--benchmark pair by comparing how many current errors are recovered with more accurate visual information versus a stronger reasoner. Across 4 vision-language models and 17 multimodal benchmarks, PARA broadly agrees with the expected characteristics of existing benchmarks while revealing substantial model-specific variation. More importantly, it shows that RL gains become smaller as perceptual headroom increases, but larger when reasoning explains more of the recoverable errors, suggesting that current RL exploits reasoning headroom more effectively than perceptual headroom. Motivated by this gap, we analyze how existing RL objectives affect visual perception and find that reward- and token-level signals influence visual attention only indirectly. We therefore propose GRAD, a group-relative attention distillation objective that uses the best rollout in each group to directly guide the visual attention of weaker rollouts. GRAD consistently improves performance across two model scales and seven benchmarks, and further complements existing RL methods.

1 Introduction

Visual-task failures can reflect perception or reasoning, and the limiting factor can differ by model on the same task. PARA measures recoverable headroom for each model–benchmark pair; the paper finds an RL asymmetry and proposes GRAD to target visual attention directly.

1 Introduction

  • PARA measures how many current errors can be recovered through stronger perception versus stronger reasoning across four VLMs and 17 benchmarks.The measurements reveal substantial model-specific variation that benchmark labels alone cannot predict.Source: §1 · pp. 1–2
  • The same visual task can expose different failures: one model may miss relevant evidence, while another perceives it but fails to reason from it.Source: §1 · p. 1, Figure 1 · p. 2
  • RL gains decrease as perceptual headroom increases but increase with reasoning-related headroom, while reward- and token-level signals affect visual attention only indirectly.The authors identify visual attention as a more direct optimization interface for perception.Source: §1 · p. 2
  • GRAD uses the best on-policy rollout in each group as an attention target for weaker rollouts, improving performance across two model scales and seven benchmarks.It requires neither external teachers nor additional annotations and complements existing RL methods.Source: §1 · p. 2

2 Related Work

Prior VLM post-training work improves downstream performance, but outcome-level feedback does not identify whether errors come from perception or reasoning. This paper positions model–benchmark diagnosis and within-group attention targets as complementary to reward-, token-, and teacher-based approaches.

Post-Training Vision-Language Models

  • Outcome-level feedback in VLM post-training does not identify whether a failure originates from perception or reasoning.Prior methods add perception-specific rewards, grounding signals, process rewards, or curricula; other approaches weight trajectories or tokens or regularize outputs under altered visual evidence.Source: §2 · pp. 2–3

On-Policy Distillation

  • GRAD constructs attention-distillation targets from on-policy rollout groups, unlike prior formulations that often require an expert teacher or extra annotated or privileged data.The paper distinguishes its attention-level target from logit-level on-policy distillation and other attention-supervision methods.Source: §2 · p. 3

3 Diagnosing Perception and Reasoning Bottlenecks in Vision-Language Model RL

PARA diagnoses recoverable perceptual and reasoning headroom for each model–benchmark pair rather than assigning a fixed bottleneck to a benchmark. Its measurements use selective interventions, validate question-blind descriptions as a perception proxy, and reveal model-specific variation.

3 Diagnosing Perception and Reasoning Bottlenecks in Vision-Language Model RL

  • The framework addresses the fact that a model’s visual-task failures may reflect perception, reasoning, or both, and that the dominant limitation can vary across models on one benchmark.Source: §3 · p. 3, §3.1 · p. 3

3.1 PARA: Perception and Reasoning Attribution

  • PARA defines perceptual and reasoning headroom as fractions of current failures recoverable by selectively strengthening visual information or reasoning while retaining the other capability as much as possible.The reasoning intervention uses the target model’s question-blind description and checks that the reference model can answer from the original image before counting a recovered failure.Source: §3.1 · pp. 3–4

3.2 Validating Question-Blind Descriptions as a Perception Proxy.

  • Question-blind descriptions closely track direct-image accuracy, with a fitted slope above 0.90 and R2 = 0.97 across model–benchmark pairs.Mean accuracy decreases range from 0.1 to 3.4 points across the four models, supporting descriptions as a perception proxy for the evaluated VLMs.Source: §3.2 · p. 4

3.3 Bottlenecks Are Model–Benchmark Specific

  • PARA measurements show broad benchmark-level patterns but substantial model variation, including different headroom profiles on MME and greater perceptual than reasoning headroom on MathVista.The results support diagnosing bottlenecks at the model–benchmark level rather than inferring them from benchmark characteristics alone.Source: §3.3 · pp. 4–5, Figure 2 · p. 5

4 Understanding the Perception–Reasoning Asymmetry in Visual RL

The section tests whether diagnosed model–benchmark bottlenecks explain uneven RL gains and examines how existing objectives reach visual attention.

4.1 RL Exploits Reasoning Headroom More Than Perceptual Headroom

  • Across algorithms and model scales, RL gains fall with perceptual headroom and rise with PARABIR, a cross-benchmark directional measure rather than a task-level predictor.PARABIR measures how broadly reasoning intervention covers failures recoverable through either intervention; Fig. 4 presents the trends.Source: Figure 4 · p. 6, §4.1 · p. 6

4.2 Existing RL Objectives Indirectly Optimize Visual Attention

  • Reward-level methods change effective advantage values and token-level methods redistribute them, but neither directly supervises attention; stronger perceptual signals do not systematically favor high-headroom benchmarks.The examined methods alter the strength or allocation of policy-update signals, whose influence on attention propagates through output logits.Source: §4.2 · pp. 6–7

4.3 Visual Attention Provides a Direct Optimization Interface

  • OPAD’s advantage-weighted attention regularizer preserves patterns from successful rollouts and pushes away from unsuccessful ones; increasing λ moves the fitted headroom slope toward flat.The section presents visual attention as a more direct interface for addressing perceptual headroom.Source: §4.3 · p. 7

5 GRAD: Group-Relative Attention Distillation for Perception

GRAD addresses ambiguity in outcome-coupled attention updates by using the best sibling rollout as a positive attention target for weaker rollouts.

5 GRAD: Group-Relative Attention Distillation for Perception

  • Outcome rewards jointly reflect perception and reasoning, so pushing away from a failed rollout’s attention can penalize valid visual perception when reasoning caused the failure.This motivates constructing attention supervision from sibling rollouts instead.Source: §5 · p. 7
  • OPAD changes the perception-dependent trend but yields only moderate overall gains relative to other methods, motivating a more carefully designed attention objective.Source: §5 · p. 7
  • GRAD replaces each rollout’s old-policy attention reference with the best sibling’s visual-attention pattern as a positive target for weaker rollouts.The sibling group shares the same image and prompt.Source: §5 · pp. 7–8
  • GRAD activates supervision only for negative-advantage students, strengthens it as advantage falls, and pulls their attention toward the best sibling rather than away from their own prior attention.Equal-reward groups have no negative-advantage students and are excluded; the target requires no external teacher or additional annotation.Source: §5 · p. 8
  • GRAD aligns attention over visual tokens, renormalizing and pooling across response tokens and heads so sibling distributions remain comparable despite different response lengths.The full training objective combines standard GRPO with GRAD; GRAD needs no external teacher, annotations, extra rollouts, or inference-time changes.Source: §5 · p. 8

6 Experiments

Across seven visual benchmarks, GRAD outperforms the compared baselines and improves both perceptual headroom and compatibility with existing methods. Ablations support the perception-only restriction, best-sibling target, and last-layer supervision choices.

6.1 Experimental Setup

  • Experiments use Vision-SR1-47K for training and evaluate seven benchmarks with their official protocols and metrics.The baselines include GRPO, Vision-SR1, VPPO, OPAD, and DAPO.Source: §6.1 · p. 8

6.2 Experimental Results

  • GRAD achieves the highest average performance for both models, beating GRPO by 2.9 and 2.5 points, respectively.It also exceeds OPAD by 3.3 and 2.2 points, respectively.Source: §6.2 · p. 9, Table 1 · p. 9
  • In representative cases, GRAD shifts attention to a fish’s tail and to Object A rather than irrelevant Object D.These cases provide qualitative evidence for perception improvement through direct visual attention supervision.Source: §6.2 · p. 9, Figure 5 · p. 10
  • GRAD reduces perceptual headroom on five benchmarks, while GRPO does so on only two; GRAD also increases reasoning headroom more on the other four.On MMMU-Pro, PerceptionBench, and ScienceQA, GRAD reverses an increase in perceptual headroom into a reduction.Source: §6.2 · p. 9, Figure 5 · p. 10
  • Adding GRAD improves Vision-SR1, VPPO, and DAPO average scores by 2.1, 0.7, and 1.9 points, respectively.The combinations improve all seven benchmarks for Vision-SR1 and six of seven for VPPO and DAPO.Source: §6.2 · p. 9, Table 2 · p. 9

6.3 Ablation Studies

  • Removing the perception-only restriction lowers Qwen3-VL-2B’s average by 1.5 points, while randomizing the target sibling lowers its score to 45.5.Keeping only tokens attributed to the best sibling’s top 50% attention lowers the average by 1.1 points.Source: §6.3 · p. 10, Table 3 · p. 10
  • Changing λ from 1.0 to 0.2 or 5.0 lowers average performance by about 2 points, and broader layer supervision underperforms last-layer supervision.Supervising the top 16 or all layers leads to substantially larger drops; the authors suggest denser targets are harder to optimize.Source: §6.3 · p. 10

7 Conclusion

The paper measures model–benchmark-specific perceptual and reasoning headroom and examines how each relates to RL post-training gains. It proposes GRAD, which improves performance and complements existing RL methods.

7 Conclusion

  • PARA estimates how many errors are recoverable through better visual information versus better reasoning for each model–benchmark pair.Across diverse models and benchmarks, it reveals substantial model-specific variation and that RL gains decrease with perceptual headroom but increase with reasoning headroom.Source: §7 · p. 10
  • Reward- and token-level objectives affect visual attention only indirectly, motivating direct attention-level supervision.Source: §7 · p. 10
  • GRAD consistently improves performance and remains complementary to existing RL methods.Source: §7 · p. 10

A.1 Full Benchmark Results

The appendix reports full PARA results for 17 benchmarks across four evaluated models. The table reports perceptual and reasoning headroom and their directional bottleneck indices.

A.1 Full Benchmark Results

  • Full PARA results cover 17 benchmarks across Qwen3-VL-2B, Qwen3-VL-4B, Llama-4-Scout-17B, and Gemma-3-27B.Source: §A.1 · p. 16, Table 4 · p. 16
  • The table reports perception headroom (PH), reasoning headroom (RH), and directional bottleneck indices BIP and BIR.Its caption says the complete results support benchmark-level trends alongside distinct patterns across models.Source: Table 4 · p. 16

A.2 Prompt Templates and Evaluation Pipeline

The appendix describes the evaluation sample, prompt templates, and intervention procedures used for PARA, alongside an alternative normalization of the two headrooms.

A.2 Prompt Templates and Evaluation Pipeline

  • PARA evaluation uses 200 examples per benchmark as a computationally constrained estimate, rather than evaluating every example.Evaluation uses exact match on boxed final answers, with multiple-choice formats for nine benchmarks.Source: §A.2 · p. 16
  • Oracle and target descriptions use the same question-blind instruction, and no condition uses a separate system prompt.Source: §A.2 · p. 16, Table 5 · p. 16
  • Representative descriptions capture visible content without adding question-conditioned reasoning or answer information.The appendix also shows a case where a GPT-5.6-Luna description changes Qwen3-VL-4B’s answer from A to correct answer C.Source: §A.2 · pp. 16–17, Table 6 · p. 17
  • For perceptual intervention, an oracle’s question-blind description augments the target model’s image; for reasoning intervention, an oracle answers using the target’s description.Item-level recoveries are unioned across Claude Opus 4.8 and GPT-5.6-Luna.Source: §A.2 · p. 17
  • Descriptions use a 1024-token budget, target answers up to 4096 tokens, and locally served models use greedy decoding.Source: §A.2 · p. 17

A.3 Reasoning Share

  • Reasoning Share (RS) is 0.5 when perceptual and reasoning headroom are equal, and exceeds 0.5 when reasoning headroom is greater.Source: §A.3 · pp. 17–18
  • RS treats the two channels as complementary shares and obscures overlap, whereas BIP and BIR normalize by the union of recoverable errors.The paper uses the bottleneck indices in the main text and reports RS as an alternative that preserves ordering and broad trends.Source: §A.3 · p. 18

B.1 Datasets and Implementation Details

The appendix documents the shared training data, evaluation benchmarks, and implementation settings used to compare post-training methods. It also includes examples of visual descriptions and perception recovery.

B.1 Datasets and Implementation Details

  • All post-training methods use Vision-SR1-47K, approximately 47K image question–answer pairs from 24 open-source datasets spanning multimodal math, commonsense, and visual understanding.The same training split is used across methods to isolate learning-objective effects.Source: §B.1 · p. 18
  • Experiments evaluate seven post-training benchmarks and extend evaluation to ten additional benchmarks for PARA.The listed benchmark descriptions include MME’s perceptual and cognitive tasks, MMStar’s six vision-dependent capabilities, MM-Vet’s integrated skills, and RealWorldQA’s spatial understanding.Source: §B.1 · pp. 18–20
  • The appendix includes representative question-blind descriptions and a Qwen3-VL-4B MMMU example in which a generated description recovers a perception failure.A separate economic-graph example identifies the AD–SRAS intersection at (400, 600), yielding real GDP of 400 billion.Source: Table 8 · p. 19, Table 9 · p. 20, §B.1 · p. 20
  • Models use each benchmark’s official metric and protocol, with MME normalized to 100 points only for the crossbenchmark average.Source: §B.1 · p. 20
  • Qwen3-VL-2B-Instruct and Qwen3-VL-4B-Instruct are fully fine-tuned with binary verifier accuracy rewards, eight rollouts per prompt, and results reported at 270 steps.Training uses a shared 128 global batch size, one PPO epoch per batch, and seed 2026.Source: §B.1 · p. 20
  • GRAD’s student attention is recomputed during the policy forward pass, using response tokens as queries and visual tokens as keys before head averaging and response-wise pooling.Siblings share an image and visual-token layout; padding tokens are excluded.Source: §B.1 · pp. 20–21

B.2 Additional Ablation Results

The appendix examines a grounding-penalty variant of sibling attention alignment and reports coefficient and layer-supervision comparisons.

B.2 Additional Ablation Results

  • The grounding variant derives a visual support set from the best sibling’s pooled attention and penalizes the current rollout’s attention mass outside that set.The support is the smallest sorted prefix of visual tokens whose cumulative probability exceeds threshold γ.Source: §B.2 · p. 21
  • The variant adds βL_ground to sibling alignment and uses the same configurations as GRAD.The support set is sample-specific and drawn from existing visual tokens.Source: §B.2 · p. 21
  • The appendix reports attention-loss coefficient comparisons on Qwen3-VL-2B and comparisons of supervision across the final layer, top 4, 8, or 16 layers, and all layers.Source: Table 11 · p. 21, Table 12 · p. 22

Related papers

Loading 2610.21380v1…