Source-linked AI summary
PEAR: Adaptive Advantage Reshaping for Perception and Exploration in Large Vision-Language Models
Junxin Wang, Yingfeng Luo, Xiangchen Song, Weiqiao Shan, Runsong Zhao, Xinyu Liu, Shunjie Xing, Yifu Huo, Dingyang Lin, Qiuyu Gu, Yongyu Mu, Chenglong Wang, Changliang Li, Jingbo Zhu, Tong Xiao
TL;DR
Prior multimodal credit assignment can underweight high-entropy tokens with low visual dependency, despite their role in reasoning exploration. PEAR retains visual dependency as a base and adaptively adds exploration gains at high-entropy positions. It achieves the highest average performance among evaluated baselines on multimodal reasoning benchmarks, with leading results on textual reasoning and general multimodal benchmarks.
Problem
Prior fusion scales token entropy by visual dependency, potentially underweighting high-entropy, low-dependency reasoning positions important for exploring alternative paths.
Source: §1 · p. 1Method
PEAR uses visual dependency as a base, adds exploration gains only at high-entropy positions with larger gains at lower visual dependency, and maps the fused signal to positive token weights.
Source: §7 · pp. 8–9Results
PEAR achieves the highest average performance among evaluated baselines on multimodal reasoning benchmarks, improving the 7B average by 2.61 points over PEPO (50.09 →52.70).
Source: §4.2 · p. 5, §7 · p. 9Takeaways & Limitations
On evaluated Qwen2.5-VL-3B/7B models, PEAR's strong results extend beyond multimodal reasoning to textual reasoning and general multimodal benchmarks.
Source: §4.2 · p. 5, §7 · p. 9Experiments cover Qwen2.5-VL-3B/7B and lack multiple independent runs for every setting, leaving larger-scale performance and run-to-run variability unexamined.
Source: Limitations · p. 9
Abstract
from arXiv · showhide
Effective multimodal reasoning requires a model to reliably utilize visual information while sufficiently exploring diverse reasoning paths. To achieve this goal, prior work combines visual dependency and token entropy to reshape sequence-level advantages, but scaling the contribution of token entropy by visual dependency can leave high-entropy tokens with low visual dependency underweighted, despite their important role in reasoning exploration. To address this limitation, we propose PEAR, an adaptive advantage reshaping method for improving perception and exploration in multimodal reasoning. PEAR retains visual dependency as the base signal and adds an entropy-based exploration gain only at high-entropy positions, with lower visual dependency leading to a larger additional gain. The fused signal is then mapped to positive token-level weights that reshape the sequence-level advantage. Experiments on Qwen2.5-VL-3B/7B show that PEAR achieves the highest average performance among evaluated baselines across multimodal reasoning benchmarks, alongside leading results on textual reasoning and general multimodal benchmarks.
1 Introduction
Multimodal reasoning needs token-level credit assignment that accounts for both visual evidence and exploration. PEAR retains visual dependency as its base and adds larger exploration gains to high-entropy positions with lower visual dependency; it outperforms evaluated baselines across multimodal reasoning benchmarks.
1 Introduction
- PEAR addresses the challenge of assigning token-level credit in multimodal reasoning, where models must use visual evidence throughout reasoning.Earlier approaches use token entropy for text-only reasoning, while multimodal reasoning also requires visual perception.Source: §1 · p. 1
- Prior fusion can underweight high-entropy, low-visual-dependency tokens that connect intermediate conclusions and support exploration of alternative reasoning paths.Examples include “since,” “Therefore,” and “thus.”Source: §1 · p. 1
- PEAR retains visual dependency as the credit-assignment base and adds an exploration gain only at high-entropy positions, with lower visual dependency producing a larger gain.The fused signal becomes positive, response-normalized token weights that reshape sequence-level advantage while preserving its total mass.Source: §1 · pp. 1–2
- PEAR achieves the highest average performance among evaluated baselines on Qwen2.5-VL-3B/7B multimodal reasoning benchmarks, with gains extending to textual reasoning and general multimodal tasks.Compared with prior work, its 7B average improves by 2.61 points, and it outperforms that work on every reported 3B benchmark.Source: §1 · p. 2
2 Preliminary
GRPO uses group-relative rewards to produce a shared sequence-level advantage, while PEAR's preliminary signals estimate each token's visual dependency and exploration potential. These signals are derived from image-conditioned predictions and token probability.
2.1 Group Relative Policy Optimization
- GRPO samples responses for a multimodal input, normalizes their group rewards, and assigns each response a sequence-level advantage; under outcome supervision, all its tokens share that advantage.DAPO extends GRPO with Clip-Higher and a Token-Level Policy Gradient Loss.Source: §2.1 · p. 2
2.2 Visual Dependency and Token Entropy
- Visual dependency compares generated-token probabilities with and without image input, estimates their difference using a low-variance KL estimator, and applies a log transform to compress extreme values.A larger dependency value indicates a greater change in prediction when visual information is removed.Source: §2.2 · pp. 2–3
- Token entropy is approximated by the generated token's negative log-probability under the original visual input, avoiding a full-vocabulary sum at each position.Larger values indicate lower generation probability and serve as a proxy for greater exploration potential.Source: §2.2 · p. 3
3 Method
PEAR combines normalized visual dependency and entropy to assign token-level credit from a shared sequence-level advantage. Its fusion preserves visual dependency as a base while selectively adding exploration gains at high-entropy positions, then converts the fused signal into smooth normalized weights.
3 Method
- PEAR constructs token weights from visual dependency and entropy to reshape shared sequence-level advantage, assigning more credit to tokens with larger weights.Visual dependency remains the base signal while exploration gains are added selectively at high-entropy positions.Source: §3 · p. 3
3.1 Signal Normalization
- Min-Max normalization maps visual dependency and entropy to comparable [0, 1] scales while preserving within-response ordering; both signals are detached from policy-optimization gradients.Normalization is applied separately within each response.Source: §3.1 · p. 3
3.2 Adaptive Signal Fusion
- PEAR filters exploration gains to high-entropy positions, then modulates them by visual dependency so high-entropy tokens with lower dependency can receive a larger added contribution.Directly adding similar gains to high- and low-dependency tokens would leave the distribution biased toward tokens already weighted by the visual base.Source: §3.2 · pp. 3–4
3.3 Advantage Reshaping
- The fused signal becomes positive, response-normalized token weights that preserve inter-token differences while smoothing weight distribution and controlling policy-update scale.These weights enable fine-grained credit assignment according to visual dependency and exploration potential.Source: §3.3 · p. 4
4 Experiments
The experiments evaluate PEAR against reinforcement-learning baselines on multimodal reasoning and generalization benchmarks using Qwen2.5-VL-3B/7B. PEAR leads on multimodal reasoning and also performs strongly on textual and general multimodal tasks.
4.1 Experimental Setup
- PEAR and the primary baselines are evaluated using Qwen2.5-VL-3B/7B-Instruct trained on ViRL39K, which contains questions and final answers.GRPO and DAPO share PEAR’s initial checkpoints and training data; PAPO uses released checkpoints, and PEPO uses reported 3B results and a code-based 7B run.Source: §4.1 · p. 4
- Training uses two epochs, five sampled responses per input, and an accuracy-and-format reward; evaluation covers multimodal reasoning, textual reasoning, and general multimodal benchmarks.Multimodal reasoning uses Avg@8; generalization uses Avg@1 except AIME24, which uses Avg@16.Source: §4.1 · pp. 4–5
4.2 Results
- PEAR achieves the highest average scores at both model scales across eight multimodal reasoning benchmarks, outperforming PEPO by 2.61 points on 7B.The 7B average comparison is 50.09 →52.70; PEAR also improves on every benchmark at both scales versus GRPO.Source: Table 1 · p. 5, §4.2 · p. 5
- PEAR outperforms GRPO on all four textual benchmarks at both scales, with average scores of 41.98 on 3B and 46.21 on 7B.The 3B average is 3.71 points above GRPO; the 7B average is 1.08 points above GRPO.Source: §4.2 · p. 5
- PEAR leads the compared RL methods on general multimodal benchmarks, scoring 66.88 on Qwen2.5-VL-3B and 72.44 on Qwen2.5-VL-7B.Without dedicated training on these tasks, it retains general multimodal capability and achieves modest average gains at both scales.Source: §4.2 · p. 5
5 Analysis
The analysis examines which token types PEAR emphasizes, how its training dynamics compare with baselines, and how its components and computational overhead affect performance. The findings highlight complementary signal roles and sustained exploration during optimization.
5.1 Token Analysis
- Tokens with high visual dependency tend to describe visual entities and geometric attributes, while low-dependency, high-entropy tokens contain more reasoning transitions and intermediate judgments.The word clouds use correct Qwen2.5-VL-3B-Instruct trajectories from MathVerse; font size indicates frequency.Source: §5.1 · p. 6, Figure 3 · p. 6
5.2 Training Dynamics
- PEAR maintains higher token entropy than GRPO while continuing to improve training accuracy; unlike DAPO, its entropy and accuracy trajectories remain stable.The authors attribute this to using entropy to identify exploration-potential positions rather than directly maximizing entropy.Source: §5.2 · p. 6, Figure 4 · p. 6
5.3 Ablation
- Visual Base Only scores 42.52 and High-Entropy Gain Only scores 44.89, both below full PEAR’s 46.25 average, while removing either modulation or filtering also lowers performance.The ablations report averages of 45.54 without visual-dependency modulation and 45.84 without entropy filtering; the authors state that the drops support retaining both components.Source: §5.3 · p. 7
5.4 Computational Overhead
- PEAR adds moderate training overhead, mainly from the image-free pass for visual dependency; its other added computations are almost negligible.Under the reported setup, PEAR’s overall training cost is lower than vanilla DAPO’s, which the authors speculate may relate to reduced resampling during dynamic sampling.Source: §5.4 · pp. 7–8, Table 4 · p. 8
6 Related Work
Related work on multimodal reasoning spans methods focused on visual perception, reasoning, and their joint optimization. The paper places PEAR among token-level approaches that reshape sequence-level advantages using perception and entropy signals.
Visual Perception
- Perception-focused approaches use perceptual supervision, explicit grounding, or perception-guided optimization to improve visual information use.Examples include visual rewards or auxiliary objectives, grounded trajectories or representations, and visual dependency or relevance signals.Source: §6 · p. 8
Reasoning
- Reasoning-focused methods use explicit reasoning supervision, reinforcement learning, or process refinement to improve multimodal reasoning.The cited examples include chain-of-thought trajectories, verifiable-reward optimization, reflection, iterative refinement, and consistency-guided optimization.Source: §6 · p. 8
Joint Perception and Reasoning Optimization
- Joint methods coordinate perception and reasoning through objectives, rewards, or token-level credit assignment, with PEPO reshaping sequence-level advantages using perception and entropy signals.The paper says PEPO cannot fully preserve entropy’s contribution to reasoning exploration; TPAE instead estimates token-level advantages using visual dependency and predictive entropy relative to correct-rollout reference distributions.Source: §6 · p. 8
7 Conclusion
PEAR adaptively combines visual dependency and entropy-based exploration gains to reshape sequence-level advantages. Experiments on Qwen2.5-VL-3B/7B report the highest average performance among evaluated baselines on multimodal reasoning, with leading results on textual reasoning and general multimodal benchmarks.
7 Conclusion
- PEAR retains visual dependency as its base signal and adds entropy-based exploration gains only at high-entropy positions, with larger gains for lower visual dependency.The fused signal becomes positive token-level weights for reshaping sequence-level advantages.Source: §7 · pp. 8–9
- The fused signal is mapped to positive token-level weights that reshape sequence-level advantages.Source: §7 · p. 9
- On Qwen2.5-VL-3B/7B, PEAR achieves the highest average performance among evaluated baselines on multimodal reasoning and leading results on textual reasoning and general multimodal benchmarks.Source: §7 · p. 9
Limitations
The evaluation is limited to Qwen2.5-VL-3B/7B and does not include multiple independent runs with different random seeds for every setting. Performance at larger scales and run-to-run variability therefore remain unexamined.
Limitations
- The experiments evaluate PEAR on Qwen2.5-VL-3B/7B but do not extend to larger models.Source: Limitations · p. 9
- The study could not conduct multiple independent runs with different random seeds for every experimental setting.Source: Limitations · p. 9
- Performance at larger scales and run-to-run variability remain to be examined.Source: Limitations · p. 9
A.1 Complete Training Configuration
The appendix summarizes PEAR’s training setup and the additional stabilization designs used for the 7B model. These designs anneal exploration gains during training and add an entropy regularizer.
A.1 Complete Training Configuration
- Training uses ViRL39K for two epochs, with a reward combining answer accuracy and format compliance at weights 0.9 and 0.1.ViRL39K covers mathematics, science, charts, and social domains.Source: §A.1 · p. 12
- For the 3B PEPO comparison, reported results are used on shared benchmarks; for 7B, the authors train PEPO using its released code and their evaluation protocol.Source: §A.1 · p. 12
A.2 Stabilization Designs for the 7B Model
- The schedule sets ρmax = 1.0 and ρmin = 0.1 across S = 202 training steps, assigning greater exploration gain early and gradually reducing its contribution as optimization proceeds.This prevents the exploration term from exerting an unnecessarily strong influence after the policy has begun to stabilize.Source: §A.2 · p. 12
- The 7B objective adds a lightweight entropy penalty, which empirically improves training stability.The penalty is a regularizer; token entropy is used to estimate exploration potential rather than being directly maximized as an objective.Source: §A.2 · p. 12
A.3 Ablation of the Stabilization Designs
The 7B ablation compares fixed exploration gain, cosine annealing, and a final configuration that also adds an entropy penalty. Average performance improves with each stabilization addition.
A.3 Ablation of the Stabilization Designs
- With fixed ρ = 1 and no entropy penalty, policy entropy can grow rapidly; the next variant adds cosine annealing to control this growth.The final configuration adds a small entropy penalty, while otherwise sharing the same training configuration.Source: §A.3 · pp. 12–13
- Figure 5 plots training reward and entropy loss across optimization steps; faint curves are raw values and solid curves are smoothed trends.Source: Figure 5 · p. 13
- Cosine annealing improves 7B average performance over a fixed exploration coefficient, and adding an entropy penalty provides a further gain.The table compares PEPO results with the ablation variants and the complete PEAR 7B model.Source: Table 6 · p. 13, §A.3 · p. 13
B.1 Multimodal Reasoning Benchmarks
The appendix describes benchmark coverage and evaluation procedures for multimodal reasoning, text-only reasoning, and general multimodal capability. It also documents how the paper’s word-cloud visualization is constructed.
B.1 Multimodal Reasoning Benchmarks
- Eight multimodal benchmarks span visual mathematics, diagram grounding, visual logic, and expert-level evaluation; all use sampled-response accuracy, not pass@8.The benchmarks include Geo3K (Lu et al., 2021), DynaMath (Zou et al., 2025), We-Math, MathVision (Wang et al., 2024a), MathVerse (Zhang et al., 2024), LogicVista (Xiao et al., 2024), and MMMU-Pro (Yue et al., 2025).Source: §B.1 · p. 13
B.2 Textual Reasoning Benchmarks
- Textual reasoning is evaluated on AIME24, MATH-500, BBH, and MMLU-Pro, using Avg@16 for AIME24 and Avg@1 accuracy for the other three.Evaluation is text-only, with benchmark-specific answer extraction and matching procedures.Source: §B.2 · pp. 13–14
B.3 General Multimodal Benchmarks
- General multimodal evaluation covers MMStar, MMMU, MME, and POPE, reporting Avg@1 under their corresponding official protocols.The reported metrics are accuracy for MMStar and MMMU, normalized percentage for MME, and F1 for POPE.Source: §B.3 · p. 14
C Word-Cloud Construction Details
- The MathVerse word cloud selects tokens with normalized visual dependency ≤ 0.1 and normalized entropy > 0.5, yielding 91,680 word occurrences and 4,682 unique words.Word size represents occurrence frequency; the processing is used only to construct the visualization.Source: C Word-Cloud Construction Details · p. 14