PaperlayerView exact revision on arXiv

Source-linked AI summary

MAPO: Mixed Advantage Policy Optimization

Wenke Huang, Quan Zhang, Yiyang Fang, Jian Liang, Xuankun Rong, Huanjin Yao, Guancheng Wan, Ke Liang, Wenwen He, Mingjun Li, Leszek Rutkowski, Mang Ye, Bo Du, Dacheng Tao

arXiv:2509.18849v3cs.AI

TL;DR

Existing GRPO advantage formulations face advantage reversion and advantage mirror across samples with differing trajectory certainty. MAPO uses Advantage Percent Deviation for high-certainty trajectories and reweights advantages by trajectory certainty; experiments report superior performance across in-domain and out-of-domain aspects.

  • Problem

    Existing GRPO advantage formulations face advantage reversion and advantage mirror, while a fixed formulation does not account for samples with different trajectory certainty.

    Source: §1 · p. 2

  • Method

    MAPO introduces Advantage Percent Deviation for high-certainty trajectories and uses Trajectory Certainty Reweight to adaptively combine advantage formulations by trajectory certainty.

    Source: §1 · p. 2, §5 · p. 9

  • Results

    Experiments across multiple datasets with Qwen2.5-VL-7B demonstrate superior performance in both In-Domain and Out-of-Domain aspects.

    Source: §1 · p. 3

  • Takeaways & Limitations

    MAPO is reported to require no additional model architecture or hyperparameter configuration and to remain compatible with diverse reasoning formats.

    Source: §5 · p. 10

    Experiments are limited by computational constraints to models up to 7B parameters and datasets with a few thousand samples.

    Source: §3.3 · p. 8

Abstract

from arXiv · show

Recent advances in reinforcement learning for foundation models, such as Group Relative Policy Optimization (GRPO), have significantly improved the performance of foundation models on reasoning tasks. Notably, the advantage function serves as a central mechanism in GRPO for ranking the trajectory importance. However, existing explorations encounter both advantage reversion and advantage mirror problems, which hinder the reasonable advantage allocation across different query samples. In this work, we propose an easy but effective GRPO strategy, Mixed Advantage Policy Optimization (MAPO). We reveal that the trajectory appears with different certainty and propose the advantage percent deviation for samples with high-certainty trajectories. Furthermore, we dynamically reweight the advantage function for samples with varying trajectory certainty, thereby adaptively configuring the advantage function to account for sample-specific characteristics. Comparison with related state-of-the-art methods, along with ablation studies on different advantage variants, validates the effectiveness of our approach.

1 INTRODUCTION

The introduction argues that one fixed GRPO advantage formulation does not suit samples with different trajectory certainty, causing advantage reversion and advantage mirror. MAPO combines a high-certainty advantage variant with certainty-based reweighting, and experiments report superior performance across in-domain and out-of-domain settings.

1 INTRODUCTION

  • Advantage reversion gives high-certainty samples more differentiated allocations than low-certainty samples, while advantage mirror treats easy and hard samples indistinguishably.The introduction argues that low-certainty trajectories benefit from stronger correction, whereas high-certainty samples do not require strong penalization.Source: §1 · p. 2
  • Advantage Percent Deviation replaces standard z-score normalization with relative normalization for high-certainty samples.The introduction notes that small rollout-reward variance can make standard deviation normalization numerically unstable and produce an uncontrollable advantage-allocation boundary.Source: §1 · p. 2
  • Trajectory Certainty Reweight dynamically combines advantage formulations according to sample trajectory certainty.The introduction treats trajectories as successes or failures and identifies a 50% success-to-failure ratio as the point of greatest uncertainty.Source: §1 · p. 2
  • Experiments across multiple datasets with Qwen2.5-VL-7B demonstrate superior performance in both In-Domain and Out-of-Domain aspects.The paper also reports ablation studies validating the effectiveness of its mixture advantage strategies.Source: §1 · p. 3

2 RELATED WORKS

The related-work section situates foundation models’ move toward multimodal inputs and reasoning, then reviews approaches to improving reasoning with GRPO. It groups GRPO variants by trajectory diversification, reward refinement, and advantage redesign.

2.1 FOUNDATION MODEL

  • Foundation models have expanded from language-only systems toward multimodal inputs, while reinforcement learning is presented as a post-training route for reasoning beyond supervised fine-tuning.The section describes visual encoders and connector modules as common components of multimodal systems.Source: §2.1 · p. 3

2.2 GROUP RELATIVE POLICY OPTIMIZATION

  • GRPO-related work is grouped into trajectory-diversity methods, reward-formulation refinement, and advantage-estimation redesign; Table 1 labels weakness dimensions including added thinking cost and task-specific adaptation.The cited methods include input perturbation or process polishing, task-specific reward functions, and reformulated or rescaled advantages.Source: §2.2 · pp. 3–4, Table 1 · p. 4

3 METHODOLOGY

MAPO argues that GRPO’s single advantage formulation mishandles samples with differing trajectory certainty. It combines Advantage Percent Deviation for high-certainty samples with certainty-based reweighting across samples.

3.1 PRELIMINARY

  • GRPO samples a group of outputs per prompt, scores them with a reward function, and computes advantages from relative rewards within that group.Its objective also uses a clipped policy-ratio function and a hyperparameter β.Source: §3.1 · pp. 4–5

3.2 PROPOSED METHOD

  • Trajectory outcomes are modeled as Bernoulli successes or failures, with certainty estimated empirically from the sampling group’s successful trajectories.Success requires a correct answer on all reward metrics; the empirical estimate approximates the success probability.Source: §3.2 · p. 5
  • For high-certainty samples, Advantage Percent Deviation measures each reward’s relative deviation from the batch mean instead of using z-score normalization.This is intended to mitigate instability from small standard deviations and mirrored advantage allocations.Source: §3.2 · p. 6
  • Trajectory Certainty Reweight interpolates between variance-sensitive advantages under uncertainty and mean-relative advantages under high certainty.The standard-deviation-based advantage receives weight 1 − λ(p), while the mean-based advantage receives λ(p).Source: §3.2 · p. 6
  • MAPO uses trajectory certainty to adaptively reweight advantage construction, aiming to mitigate existing advantage limitations across samples.The method replaces the original GRPO advantage with its sample-wise construction.Source: §3.2 · p. 6

3.3 DISCUSSION AND LIMITATION

  • Prior advantage variants still leave both advantage reversion and mirror unresolved, while MAPO combines relative deviation for high-certainty samples with certainty-aware reweighting.The discussion reports that MAPO achieves satisfying performance; the cited Figure 4 compares formulations on geometry with Qwen2.5-VL-7B-Instruct and G=12.Source: §3.3 · p. 7, Figure 4 · p. 7
  • MAPO’s gradient analysis indicates larger gradients for harder samples with p < 1/2 and smaller gradients for easier samples with p > 1/2.The analysis simplifies by ignoring clipping and KL regularization and treating reward as Bernoulli.Source: §3.3 · pp. 7–8
  • Experiments cover models up to 7B parameters and datasets of a few thousand samples, and consistently failing rollouts can reduce MAPO to a single-function strategy.The authors also identify more refined reward allocation as an open direction.Source: §3.3 · p. 8

4 EXPERIMENTS

Experiments use Geo3K and EmoSet for training, with in-domain and out-of-domain evaluation. MAPO outperforms Vanilla, GRPO, and DAPO across those settings, while ablations favor trajectory-certainty reweighting over random weighting.

4.1 EXPERIMENTAL SETUP

  • Experiments train Qwen2.5-VL-7B-Instruct on Geo3K and EmoSet, then assess in-domain accuracy and average out-of-domain accuracy across held-out datasets.Source: §4.1 · p. 8

4.3 COMPARISON TO STATE-OF-THE-ARTS

  • With G = 12, MAPO achieves the highest overall accuracies (51.26 on math and 66.77 on emotion) and consistently outperforms Vanilla, GRPO, and DAPO in- and out-of-domain.Source: Table 3 · p. 9, §4.3 · p. 9

4.2 DIAGNOSTIC ANALYSIS

  • Ablations report limited or negative gains from a single advantage replacement, unstable improvements with random weights, and better overall performance with TCR; DAPO fails to complete EmoSet training.Source: §4.2 · p. 9, Figure 5 · p. 9

5 CONCLUSION

MAPO addresses advantage reversion and advantage mirror in GRPO by using trajectory certainty to tailor advantage allocation. The conclusion highlights its architecture and reasoning-format compatibility, lack of additional hyperparameters, and validation across diverse scenarios.

5 CONCLUSION

  • MAPO uncovers trajectory certainty and introduces advantage percent deviation for high-certainty trajectories to address advantage reversion and advantage mirror.It also dynamically reweights advantages according to trajectory certainty, tailoring them to sample-specific characteristics.Source: §5 · pp. 9–10
  • MAPO operates without additional model architectures, is compatible with diverse reasoning formats, and avoids additional hyperparameters through certainty-based reweighting.The conclusion describes these as three advantages of the approach.Source: §5 · p. 10
  • The conclusion reports validation across diverse scenarios and notes MAPO’s potential for broader applications.Source: §5 · p. 10

A NOTATION AND ALGORITHM

The section points readers to a notation table and the proposed method algorithm. The table summarizes key MAPO variables and operations, with definitions tied to their first appearance in the main text.

A NOTATION AND ALGORITHM

B.1 DATASET INTRODUCTION

The experiments use datasets spanning mathematics and emotional tasks, with visualizations of the in-domain and out-of-domain datasets. Geo3K supports geometry problem solving through joint visual-symbolic reasoning, while EmoSet targets visual emotion recognition.

B.1 DATASET INTRODUCTION

  • The experiments use datasets from mathematics and emotional tasks, and Figure 6 visualizes the in-domain and out-of-domain datasets.Source: §B.1 · p. 15, Figure 6 · p. 16
  • Geo3K is designed for geometry problem solving and contains images, text, and formulas for joint visual-symbolic reasoning.Source: §B.1 · p. 16
  • EmoSet is described as a large-scale collection targeting visual emotion recognition.Source: §B.1 · p. 16

B.2 IMPLEMENTATION DETAILS

The implementation uses EasyR1, built on Verl2, with Qwen2.5-VL-7B fine-tuned without freezing its vision tower. Training and validation use specified sampling, optimization, and token-generation settings; Figures 7 and 8 show in-domain and out-of-domain case visualizations.

B.2 IMPLEMENTATION DETAILS

  • Training uses EasyR1, built on Verl2, with Qwen2.5-VL-7B fine-tuned without freezing its vision tower.The rollout batch size is 512 and global batch size is 128.Source: §B.2 · p. 16
  • Training uses temperature 1.0, Top-p 0.99, token-mean policy-loss aggregation, AdamW at 1 × 10−6, and KL coefficient β = 1 × 10−2.Source: §B.2 · p. 16
  • Validation uses temperature 0.5, Top-p 0.95, and a maximum generation length of 2048 tokens.Training runs for T = 20 epochs and E = 80 steps on Geo3K, and T = 15 epochs and E = 60 steps on EmoSet.Source: §B.2 · p. 16
  • Figures 7 and 8 provide in-domain and out-of-domain case visualizations, respectively.Source: Figure 7 · p. 17, Figure 8 · p. 17

B.3 VISUALIZATION ANALYSIS

For high-certainty samples where vanilla Qwen2.5-VL-7B-Instruct answers correctly, GRPO and DAPO degrade on both the in-domain EmoSet and out-of-domain WEBEmo datasets.

B.3 VISUALIZATION ANALYSIS

  • GRPO and DAPO degrade on EmoSet and WEBEmo for high-certainty samples that vanilla Qwen2.5-VL-7B-Instruct answers correctly, suggesting distorted optimization directions reduce performance.EmoSet is in-domain and WEBEmo is out-of-domain.Source: §B.3 · p. 17

C THEORETICAL ANALYSIS

The gradient analysis finds that MAPO amplifies gradients on harder samples and makes smaller updates on easier ones, under a simplified accuracy-reward model that ignores clipping and KL regularization.

C THEORETICAL ANALYSIS

  • The gradient analysis models accuracy rewards as Bernoulli outcomes across G rollouts and ignores clipping and KL regularization.For rollout k, it defines the advantage as Ak = Rk − µ.Source: C THEORETICAL ANALYSIS · p. 18
  • MAPO amplifies gradients on harder samples and makes smaller updates on easier samples than GRPO.The analysis compares gradient behavior across samples with different trajectory certainty.Source: C THEORETICAL ANALYSIS · p. 18

Related papers

Loading 2509.18849v3…