Reinforcing Multimodal Reasoning via Token-Level Perception-Grounded Advantage Estimation

1 Introduction

• GRPO and DAPO assign one rollout-level advantage uniformly to every token, while learned process-level reward models require extra training and can be costly and vulnerable to reward hacking.The introduction frames direct, fine-grained estimates from policy behavior as the research question.Source: §1 · pp. 1–2
• Correct chains show sharply lower entropy as visual dependency rises, while incorrect chains retain high entropy despite strong visual dependency.The authors call the latter pattern non-resolving grounding and identify the coupling as a model-intrinsic indicator of reasoning quality.Source: §1 · p. 2
• A 7.64% subset of tokens in incorrect chains are severe outliers from the correct-chain vision-entropy distribution, and human evaluation identifies them as semantic pivots of reasoning errors.The human evaluation sampled 200 outliers.Source: §1 · p. 2
• TPAE estimates dense token-level advantages by scoring vision-entropy alignment with reference distributions from correct rollouts, then uses those scores to modulate rollout-level advantages without an auxiliary reward model.The method is designed for integration with frameworks including GRPO and DAPO.Source: §1 · p. 2
• Across seven benchmarks, TPAE with GRPO or DAPO yields consistent gains, and TPAE-D-Qwen2.5-7B surpasses same-backbone state-of-the-art models.The evaluation spans two base RL algorithms and two MLLM backbones.Source: §1 · p. 2

2 Related Work

• Multimodal reinforcement-learning approaches have focused on data curation, rollout improvements, or reward strategies such as verifiable intermediate steps and external reward models.Related work also situates these methods alongside multimodal alignment and persistent reasoning gaps.Source: §2 · p. 2
• Prior work often treats grounding as post-hoc verification, while TPAE uses token-level visual-dependency and entropy dynamics to modulate coarse rewards.The related-work discussion says token-level visual dependency had also been used as passive filters or heuristic penalties.Source: §2 · p. 2

3.1 Group Relative Policy Optimization

• GRPO estimates advantages from groups of rollouts without a separate value model, but RLVR assigns a final-answer-based sequence advantage uniformly across each rollout’s tokens.DAPO extends GRPO with clip-higher, dynamic sampling, and token-level policy-gradient loss; the paper investigates TPAE with both algorithms.Source: §3.1 · p. 3

3.2.1 Predictive Entropy.

• Predictive entropy measures uncertainty over the next-token distribution, with low entropy indicating concentrated confidence and high entropy indicating multiple plausible next steps.The distribution is generated by the policy conditioned on the image, query, and preceding rollout tokens.Source: §3.2.1 · pp. 3–4

3 Preliminary

• TPAE’s overview uses correct rollouts to form reference distributions of token vision-entropy states, applies trust boundary τ to score distributional alignment, and integrates that score with GRPO’s rollout advantage.The preliminary analysis introduces entropy and visual dependency as token-level metrics for characterizing multimodal generation.Source: §3.2 · p. 3, Figure 3 · p. 4

3.2.2 Visual Dependency.

• Visual dependency measures reliance on the image by comparing the original next-token distribution with one conditioned on a masked, non-informative image.The analysis uses random patch masking at a 60% ratio.Source: §3.2.2 · p. 4

3.3 Analysis of Multimodal Reasoning

• Across 100 ViRL39K questions, correct trajectories averaged 418.07 tokens versus 546.57 for incorrect trajectories, and correct chains concentrated more in high-visual-dependency regions.The analysis used 128 trajectories per question, comprising 5,995 correct and 6,805 incorrect trajectories.Source: §3.3 · p. 4
• Correct trajectories show monotonically decreasing entropy as visual dependency rises, while incorrect trajectories remain at a high entropy plateau.The paper calls the latter pattern non-resolving grounding: image attention does not resolve internal uncertainty.Source: §3.3 · p. 4
• 7.64% of tokens in failed trajectories are extreme vision-entropy outliers, and over 74% of these occupy the grid’s upper-middle and upper-right regions.The outliers deviate by more than three standard deviations from the correct-chain reference center; the grid classifies deviations in entropy and visual dependency.Source: §3.3 · pp. 4–5
• In a human evaluation of 200 divergent tokens, 68.5% were perception errors, 18.8% reasoning errors, and 3.7% benign noise.The authors conclude that distributional divergence is effective for localizing pivotal moments of reasoning collapse.Source: §3.3 · p. 5

4.1 Vision-Entropy Distribution

• Correct rollouts define token-specific reference distributions over visual dependency and entropy, modeled with multivariate Gaussians and covariance shrinkage for sparse samples.The Gaussian choice is motivated by observed unimodal centrality; Ledoit-Wolf shrinkage blends empirical covariance with a diagonal prior, weighted adaptively by sample size.Source: §4.1 · p. 5

4.2 Token Trustworthiness Quantification

• For each token’s vision-entropy state, the method tests alignment with the correct-rollout reference, rejecting the null at 99% confidence when squared Mahalanobis distance exceeds 9.21 (distance threshold 3.03).Token trustworthiness is quantified as the residual distance to the established trust boundary.Source: §4.2 · p. 5

4.3 Token-level Advantage Estimation

• TPAE integrates token trustworthiness into the rollout-level advantage, penalizing only tokens that violate the trust boundary while preserving advantages for compliant tokens.The sigmoid-based transformation provides a dense, bounded penalty for vision-entropy misalignment.Source: §4.3 · p. 6

• Across seven benchmarks, the evaluation compares TPAE-integrated RL algorithms with strong multimodal reasoning baselines and examines training efficiency, accuracy, trust-boundary sensitivity, and failure localization.Source: §5 · p. 6

5.1.1 Models and Baselines.

• The study evaluates Qwen2.5-VL-7B-Instruct and Qwen3-VL-8B-Instruct with GRPO and DAPO, comparing against a suite of leading multimodal reasoning models.Source: §5.1.1 · p. 6, §5.1.3 · p. 6

5.1.2 Dataset, Benchmarks, and Evaluation.

• Training uses ViRL39K, while evaluation spans seven mathematical, geometric, logical, and multidisciplinary benchmarks with exact-match scoring and average accuracy@8.MathVerse evaluation excludes its text-only subset; all models use the same fixed evaluation pipeline at temperature 1.0.Source: §5.1.2 · p. 6

5.1.3 Training Details.

• The four TPAE variants train for two epochs using VerL, with 16 rollouts per question and a trust boundary of 3.03.Training uses a learning rate of 1e-6, batch size 384, and eight NVIDIA Pro6000 96G GPUs.Source: §5.1.3 · p. 6

• 48.71% average accuracy for TPAE-D-Qwen2.5-7B leads the seven-benchmark comparison, 1.39% above Shuffle-R1-7B and 1.12% above VPPO-7B.TPAE-G-Qwen2.5-7B reaches 47.63%; TPAE-G-Qwen3-8B and TPAE-D-Qwen3-8B reach 60.62% and 61.35% average accuracy, respectively.Source: §5.2 · p. 6

5.3.1 Comparison with Base RL Algorithm.

• TPAE accelerates initial convergence and surpasses the peak accuracy of base RL algorithms under identical training configurations.The reported comparison uses the same training dataset, rollout space, and reward functions.Source: §5.3.1 · p. 6, Figure 4 · p. 8
• Table 1 reports average accuracy across seven multimodal reasoning benchmarks using exact-match scoring on verifiable instances.Comparative baselines use the Qwen2.5-VL-7B backbone.Source: Table 1 · p. 7
• 3.74% and 2.50% average-accuracy gains with DAPO, and 3.77% and 2.30% gains with GRPO, are reported for the 7B and 8B variants, respectively.Table 2 compares multimodal reinforcement learning with and without TPAE.Source: §5.3.1 · p. 7, Table 2 · p. 7
• The training-dynamics comparison uses accuracy rewards, with solid lines showing running averages over windows of 20.The figure caption reports faster learning with TPAE across GRPO and DAPO and two model backbones.Source: Figure 4 · p. 8

5.3.2 Sensitivity of Trust Boundary 𝜏.

• In the TPAE-G-Qwen2.5-7B trust-boundary ablation, default τ=3.03 gives highest accuracy on six of seven benchmarks; stringent τ=3.26 (99.5% confidence) averages 47.34%, versus 46.96% for permissive τ=2.45 (95%).Source: Table 3 · p. 7, §5.3.1 · p. 8

5.4 Qualitative Analysis

• High-untrustworthiness tokens cluster at visually hallucinated reasoning steps in the failed trajectory.Dark red highlights identify these tokens in the qualitative analysis.Source: §5.4 · p. 8
• The collapse begins with misidentifying ∠AOB as subtending arc BC rather than arc AB, creating an incorrect association with ∠ADC.The passage also describes misclassifying ∠CBO as an inscribed angle rather than an interior angle of △OBC.Source: §5.4 · p. 8
• The final output syntax, including the boxed terminal, also has elevated untrustworthiness.The passage notes this alongside the earlier geometric reasoning errors.Source: §5.4 · p. 8

6 Conclusion

• The joint distribution of visual dependency and predictive entropy is presented as an indicator of multimodal reasoning collapse.The conclusion describes this as a finding from the preliminary study.Source: §6 · p. 8
• TPAE derives fine-grained advantages by aligning each token’s vision-entropy state with the reference distribution from correct reasoning chains.Source: §6 · p. 8
• The conclusion reports state-of-the-art results across diverse multimodal benchmarks and substantially accelerated convergence during RL.Source: §6 · p. 8