PaperlayerView exact revision on arXiv

Source-linked AI summary

Perception-Aware Policy Optimization for Multimodal Reasoning

Zhenhailong Wang, Xuehang Guo, Sofia Stoica, Haiyang Xu, Hongru Wang, Hyeonjeong Ha, Xiusi Chen, Yangyi Chen, Ming Yan, Fei Huang, Heng Ji

arXiv:2507.06448v5cs.CL

TL;DR

Multimodal RLVR faces a perception bottleneck that standard objectives do not explicitly incentivize models to overcome. PAPO adds an Implicit Perception Loss and Double Entropy Loss to core policy optimization, yielding consistent benchmark gains and fewer perception errors without extra annotations, teacher models, or neural reward models.

  • Problem

    67% of manually examined multimodal reasoning errors stemmed from perception, while existing RLVR objectives do not explicitly incentivize visually grounded responses.

  • Method

    PAPO integrates an Implicit Perception Loss based on original-versus-corrupted visual inputs and Double Entropy Loss regularization into GRPO or DAPO.

  • Results

    PAPO delivers consistent improvements of 4.4%-17.5% across eight multimodal benchmarks and reduces perception-related errors by 30.5%.

  • Takeaways & Limitations

    PAPO provides a visually grounded, drop-in RLVR optimization approach without additional annotations, stronger teacher models, or expensive neural reward models.

  • Takeaways & Limitations

    The work does not explore scaling to larger model sizes or compatibility with other model families, and does not focus on training-efficiency optimization.

Abstract

from arXiv · show

Reinforcement Learning with Verifiable Rewards (RLVR) has proven to be a highly effective strategy for endowing Large Language Models (LLMs) with robust multi-step reasoning abilities. However, its design and optimizations remain tailored to purely textual domains, resulting in suboptimal performance when applied to multimodal reasoning tasks. In particular, we observe that a major source of error in current multimodal reasoning lies in the perception of visual inputs. To address this bottleneck, we propose PAPO, a novel policy gradient algorithm that encourages the model to learn to perceive while learning to reason. Specifically, we introduce the Implicit Perception Loss in the form of a KL divergence term, which can be seamlessly plugged into mainstream RLVR algorithms such as GRPO and DAPO. Notably, PAPO does not rely on additional data curation, reward models, or stronger teacher models. To further enhance the training stability of PAPO, we introduce the Double Entropy Loss, which effectively regularizes the new KL objective without compromising performance. Despite its simplicity, PAPO yields significant overall improvements of 4.4%-17.5% on diverse multimodal benchmarks. The improvements are more pronounced, approaching 8.0%-19.1%, on tasks with high vision dependency. We also observe a substantial reduction of 30.5% in perception errors, indicating improved perceptual capabilities with PAPO. Overall, our work introduces a deeper integration of perception-aware supervision into core learning objectives and lays the groundwork for a new RL framework that encourages visually grounded reasoning. Code and data will be made publicly available for research purposes. Project page: https://mikewangwzhl.github.io/PAPO.

1 INTRODUCTION

Multimodal RLVR has a perception bottleneck that standard objectives do not explicitly address. PAPO integrates perception-aware optimization into RLVR and improves benchmark performance while reducing perception errors.

  • Error bottleneck: 67% of manually examined multimodal reasoning errors stemmed from perception across 200 cases and four benchmarks.The analysis attributes this bottleneck to RLVR objectives lacking incentives for visually grounded responses.
  • Approach: PAPO is a policy-gradient algorithm that encourages models to learn perception while learning reasoning.It is designed as a direct replacement for GRPO or DAPO rather than a data- or reward-level modification.
  • Approach: The Implicit Perception Loss maximizes a KL divergence between policies conditioned on original and corrupted visual inputs.The objective uses the same rollout token sequence to encourage visually grounded responses.
  • Results: 4.4%-17.5% average gains over GRPO and DAPO were observed across eight multimodal reasoning benchmarks.The evaluations used identical training data and reward functions for the comparisons.
  • Results: 30.5% fewer perception-related errors were observed with PAPO, while gains reached 8.0%-19.1% on tasks with higher vision dependency.PAPO also showed early-stage gains beginning around 25 training steps.

2 PRELIMINARY

GRPO removes the value model and estimates advantages from response groups, while DAPO extends GRPO with several policy-optimization modifications. PAPO adds perception-aware and entropy-based objectives to this RLVR setting.

  • GRPO: GRPO removes the value model and estimates advantages through group-based computation.Its objective uses responses sampled from a rollout policy for each visual question instance.
  • GRPO: The token-level advantage is the sequence-level reward normalized across the response group.A rule-based equivalence verifier supplies the response rewards.
  • DAPO: DAPO extends GRPO with Clip-Higher, Dynamic Sampling, and Token-Level Policy Gradient Loss.The paper evaluates PAPO as an addition to both GRPO and DAPO.
  • PAPO: Double Entropy Loss regularization is added to enhance training stability by keeping both entropy values low.The regularizer addresses instability associated with the perception-aware objective.

3 METHOD

PAPO adds perception-aware supervision to RLVR by comparing policy outputs under original and corrupted visual inputs, while entropy regularization stabilizes this objective. The method uses masking-based corruption and can be combined with GRPO or DAPO without additional annotations, teacher models, or reward models.

  • Implicit Perception Loss: PAPO combines RLVR with an Implicit Perception Loss that encourages visually grounded, high-reward responses without additional annotations, teacher models, or neural reward models.The objective is designed as a modification of the optimization objective rather than the data or reward pipeline.
  • Implicit Perception Loss: The perception objective compares output distributions conditioned on original versus corrupted visual inputs, measuring how masking meaningful information changes the model’s predictions.A higher ratio indicates that full visual context contributes more information to the decision.
  • Entropy Regularization: Double Entropy Loss regularizes the theoretically unbounded KL objective by encouraging low entropy in both the original-input and masked-input policies.Rising rollout entropy in both policies is treated as a representative sign of collapse.
  • Objective: The complete PAPOG objective combines the GRPO objective with Implicit Perception Loss, reference KL regularization, and two entropy terms; a DAPO variant is also derived.The weighting coefficients γ, η1, and η2 control the added loss components.
  • Masking Strategy: PAPO constructs corrupted inputs by masking image patches, with random masking selecting patches uniformly and semantic-aware masking selecting salient patches identified by DINOv2.Patch masking is preferred to Gaussian noise because it removes informative semantic content more effectively.

4 EXPERIMENTS

The experiments train and compare PAPO variants with GRPO and DAPO across diverse multimodal reasoning benchmarks using direct RL and exact-match evaluation. PAPO improves performance broadly, learns faster, and remains compatible with complementary rollout modifications.

  • Experimental Setup: PAPO is evaluated against GRPO and DAPO using Qwen2.5-VL and Qwen3-VL models on general and vision-dependent multimodal reasoning tasks.The benchmark suite covers mathematical, geometric, multidisciplinary, logical, and counting problems, with exact match as the evaluation metric.
  • Main Results: 8%-19% gains are observed especially on tasks with high vision dependency, where the input question provides limited visual clues.These gains are reported as averaged relative improvements over the baseline for each task.
  • Training Dynamics: PAPO learns faster from early training stages on both GRPO and DAPO, while PAPOD continues improving after DAPO-7B undergoes later-stage model collapse.The training curves use running averages with a stepping window size of 20.
  • Compatibility: PAPO can be combined with NoisyRollout and delivers additional gains, whereas NoisyRollout alone produces reductions on 4/9 benchmarks.The integration study uses Qwen2.5-VL-3B as the base model.

5 RESULTS

PAPO consistently improves multimodal reasoning over GRPO and DAPO, with larger gains on vision-dependent tasks. Ablations also identify masking and loss-weighting choices, while Double Entropy Loss mitigates KLprcp Hacking and stabilizes training.

  • Main Results: PAPO consistently outperforms GRPO and DAPO across multimodal reasoning tasks, using identical training datasets, rollout spaces, and reward designs.
  • Vision-Dependent Reasoning: 8.0%-19.1% relative gain appears on vision-dependent reasoning tasks, exceeding PAPO’s 4.4%-17.5% overall improvement.
  • Robustness: PAPO remains robust when the reference KL penalty is removed, improving overall performance by 11.2% on 3B models and 4.0% on 7B models.
  • Masking Ablation: Random masking outperforms semantic-aware masking, while masking 0.6 to 0.8 of the image yields the best performance; complete blackening is unfavorable.
  • Regularization: Double Entropy Loss prevents KLprcp Hacking while preserving performance, and entropy on the original sequence is essential for training stability.

6 RELATED WORK

Prior multimodal RLVR work mainly modifies data, rollouts, or rewards, while other approaches use tools to improve perception. PAPO instead argues that visually grounded reasoning requires changes to the core optimization objective.

  • RLVR Modifications: Prior multimodal RLVR approaches primarily enhance data, rollout, and reward components while leaving core optimization objectives largely unchanged.
  • Data-Centric Approaches: Data-centric methods distill textual reasoning traces or select higher-quality samples to improve multimodal training.
  • Rollout Improvements: Rollout improvements diversify visual inputs or use selective sample replay to address limited rollout diversity and vanishing advantages.
  • Reward Enhancements: Reward-based methods add grounding metrics or captioning rewards, but captioning can separate perception from reasoning and miss low-level visual details.
  • Perception as Tool-Using: Tool-using methods improve perception through visual operations such as zooming, but do not directly improve native multimodal perception capabilities.
  • Motivation: PAPO challenges data- and reward-only modifications by integrating visually grounded incentives into the core optimization objective.

7 CONCLUSION AND LIMITATIONS

PAPO encourages LMM reasoning to remain internally grounded in visual inputs while serving as a direct replacement for GRPO or DAPO. The paper also identifies boundaries involving scale, model families, efficiency, and token selection.

  • Conclusion: PAPO encourages reasoning steps in Large Multimodal Models to be internally grounded in visual inputs.
  • Conclusion: PAPO requires no additional annotations, stronger teacher models, or expensive neural reward models, and can replace GRPO or DAPO directly.
  • Limitations: The paper has not explored scaling PAPO to larger models or compatibility with other model families such as InternVL.
  • Limitations: Training efficiency was not optimized despite moderate computational overhead, and finer-grained token selection remains future work.

8 REPRODUCIBILITY STATEMENT

The supplementary material includes anonymous source code and instructions for reproducing training and evaluation, alongside dataset and model implementation details.

  • Reproducibility: Anonymous source code provides training and evaluation instructions for reproducing the paper’s results.
  • Reproducibility: Dataset and model implementation details are provided in §C.
  • Reproducibility: The LLM-as-a-judge prompt used for Figure 13 is provided in Figure 14.

9 ETHICS STATEMENT

The paper reports its ethical scope, datasets, models, training setup, and vision-dependency analysis, alongside qualitative attention comparisons and controlled reference-KL results.

  • 9 ETHICS STATEMENT: The study uses publicly available benchmarks and reports no human subjects, sensitive personal data, private information, or foreseeable ethical risks.
  • C IMPLEMENTATION DETAILS: Training uses ViRL39K, a 38.8K-example multimodal reasoning dataset spanning math, STEM, charts, and social topics without intermediate chain-of-thought annotations.
  • C IMPLEMENTATION DETAILS: Experiments use Qwen2.5-VL-3B, Qwen2.5-VL-7B, and Qwen3-VL-2B-thinking as base models, with PAPOG instantiated from GRPO and PAPOD from DAPO.
  • C IMPLEMENTATION DETAILS: PAPO models use random masking with a default ratio of 0.6, while Double Entropy Loss is essential for higher γ and settings without reference KL.
  • Vision Dependency: The benchmarks are organized by vision dependency, and PAPO’s improvements are most pronounced on high-dependency tasks.
  • Qualitative Analysis: Qualitative attention analysis finds that PAPO increases attention to image tokens and salient objects while reducing attention to empty regions.

F CONTROLLED EXPERIMENTS ON REFERENCE KL REMOVAL WITH GRPO

Controlled experiments remove the reference KL penalty from GRPO to test PAPO’s robustness, while also describing the masking procedures used for the perception-aware objective.

  • Controlled Reference-KL Removal: The controlled setting removes DKL[πθ||πref] from both GRPO and PAPOG without introducing DAPO’s other modifications.
  • Controlled Reference-KL Removal: 11.2% and 4.0% overall improvements are achieved by PAPOG without reference KL on the 3B and 7B models, respectively.
  • Controlled Reference-KL Removal: 13.6% and 7.5% gains over GRPO + No KLref occur on more vision-dependent tasks for the 3B and 7B models, respectively.
  • Masking Strategies: Random masking independently masks image patches, with masking probability p and expected masked-patch fraction p.
  • Masking Strategies: Semantic-aware masking uses a vision encoder’s patch-level attention scores to select the most salient patches for masking.

G.3 FURTHER ANALYSIS OF MASKING STRATEGIES

The masking-strategy analysis compares random and semantic-aware corruption through attention distributions and error categories, finding random masking empirically superior.

  • Masking Comparison: Random masking empirically outperforms semantic-aware masking by balancing informative-content removal against preserving distributed visual information.
  • Attention Pattern Analysis: Attention patterns do not differ significantly between strategies, but random masking generally produces slightly higher normalized attention entropy.
  • Error Analysis: The error analysis samples 30 instances per dataset from Geo3K, Counting, and MathVerse where random masking succeeds and semantic masking fails.
  • Error Analysis: Perception-related errors remain the predominant error source when semantic-masking models underperform random-masking models.

H ADDITIONAL EXPERIMENTS ON REGULARIZING DAPO BASELINE

The experiments show that entropy regularization stabilizes DAPO and PAPO training, while excessive perception-loss weighting can still trigger collapse and degraded generation.

  • DAPO Regularization: Single entropy loss delays DAPO collapse and improves end-task performance but does not fully prevent collapse.
  • DAPO Regularization: PAPOD with Double Entropy Loss consistently outperforms DAPO variants and completely prevents collapse throughout training.
  • KLprcp Hacking: A KLprcp weight of 0.02 can cause irreversible collapse without reference KL, even with Double Entropy Loss; 0.01 is an empirical default.
  • Entropy Coefficients: Double Entropy Loss coefficients of 0.03 and 0.05 prevent hacking behavior, while 0.01 permits later-stage collapse; 0.05 performs best across 9 evaluation benchmarks.
  • Additional Evaluations: PAPO benefits OCRbench-v2 and remains competitive or better on MMLU-Pro with dummy visual inputs, indicating robustness to information-dense and uninformative visual inputs.
  • Collapse Behavior: Collapsed models show approximately 18% lower response relatedness and 8.4 times higher KLprcp variance than the baseline.
  • Perception Errors: In a sample of original GRPO perception errors, 41% are corrected under PAPO, 24% remain perception errors, and 25% shift to other categories.
Loading 2507.06448v5…