Source-linked AI summary
DiPO: Disentangled Perplexity Policy Optimization for Fine-grained Exploration-Exploitation Trade-Off
Xiaofan Li, Ming Yang, Zhiyuan Ma, Shichao Ma, Jintao Du, Yu Cheng, Weiqiang Wang, Zhizhong Zhang, Xin Tan, Yanyun Qu, Lizhuang Ma, Yuan Xie
TL;DR
RLVR training must manage exploration and exploitation for extremely hard and easy samples, whose extreme groups can lose useful advantage signals. DiPO disentangles perplexity space to identify fine-grained samples and reallocates rewards bidirectionally with minimal verification-reward disruption; experiments report comprehensive superiority on mathematical reasoning and function calling.
Problem
GRPO-based RLVR can produce extreme hard and easy groups with uniform rewards and zero advantages, leaving these groups without exploration or exploitation signals.
Method
DiPO partitions perplexity space using correctness distributions, then applies bidirectional reward reallocation to selected samples while minimizing perturbations to verification rewards.
Results
DiPO achieves comprehensive superiority across mathematical reasoning and function calling tasks, while PSD and BRR together achieve the best ablation performance on 4B and 8B models.
Takeaways & Limitations
Fine-grained exploration and exploitation, implemented through PSD and BRR, enhance LLM performance while supporting more stable policy optimization.
Takeaways & Limitations
PSD may hinder optimization during early RL when perplexity and correctness are not positively correlated, so the method uses advantage judgment before activating it.
Abstract
from arXiv · showhide
Reinforcement Learning with Verifiable Rewards (RLVR) has catalyzed significant advances in the reasoning capabilities of Large Language Models (LLMs). However, effectively managing the exploration and exploitation trade-off remains a critical challenge. In this paper, we fully analyze the exploration and exploitation dilemma of extremely hard and easy samples during the training and propose a new fine-grained trade-off mechanism. Concretely, we introduce a perplexity space disentangling strategy that divides the sample space into distinct exploration (high perplexity) and exploitation (low perplexity) subspaces, thereby mining fine-grained samples requiring exploration-exploitation trade-off. Subsequently, we propose a bidirectional reward allocation mechanism with a minimum impact on verification rewards to implement perplexity-guided exploration and exploitation, enabling more stable policy optimization. Finally, we have evaluated our method on two mainstream tasks: mathematical reasoning and function calling, and experimental results demonstrate the superiority of the proposed method, confirming its effectiveness in enhancing LLM performance by fine-grained exploration-exploitation trade-off.
1 Introduction
The paper identifies two exploration-exploitation dilemmas in GRPO-based RLVR and proposes DiPO, combining perplexity-space disentanglement with bidirectional reward reallocation for fine-grained optimization.
- Motivation: Extreme hard and easy groups often have uniform zero or one rewards, producing zero advantages and depriving GRPO of exploration or exploitation gradients.This creates the paper’s first exploration-exploitation dilemma during training.
- Motivation: Existing PPL-based approaches use coarse partitioning, add hyper-parameters, or directly bias rewards, potentially hindering early optimization or destabilizing training.DiPO is introduced to address these limitations with a fine-grained trade-off mechanism.
- Method: DiPO proposes Perplexity Space Disentangling, which partitions PPL space using perplexity and correctness distributions to enable fine-grained exploration and exploitation.The strategy identifies regions associated with high- and low-perplexity samples.
- Method: DiPO proposes Bidirectional Reward Reallocation to apply exploration and exploitation signals while minimally perturbing the verification-reward distribution.BRR is designed to stabilize policy optimization rather than directly use PPL as a reward bias.
- Evaluation: DiPO achieves superior results on mathematical reasoning and function calling tasks, confirming its effectiveness for enhancing LLM reasoning.The paper evaluates the method on both downstream task categories.
2 Preliminaries and Definitions
The preliminaries define GRPO’s group-relative advantages and DAPO’s role as a baseline, then introduce perplexity as a confidence-related metric linked to exploration and exploitation.
- GRPO: GRPO estimates each sampled response’s advantage by normalizing its verifiable reward relative to the rewards of other responses in the group.The supplied passages describe group sampling and the estimated advantage calculation.
- GRPO: When all sampled rewards are 0 or 1, every advantage becomes 0, indicating groups composed solely of hard or easy samples.Such groups provide no advantage signal for policy optimization.
- DAPO: DAPO extends GRPO, and the paper uses advanced DAPO as its baseline without overlong reward shaping because that shaping can damage model performance in some cases.The passage identifies the baseline choice and its stated implementation condition.
- Perplexity: Perplexity measures how confidently a language model predicts a sample, with low PPL associated with exploitation and high PPL associated with exploration.The paper defines PPL for model-generated responses and interprets its low- and high-value regions accordingly.
3 Methodology
DiPO disentangles perplexity into exploitation and exploration spaces, then reallocates rewards for extreme groups to guide stable, fine-grained policy optimization. Its objective combines the original verification-reward objective with an orthogonal reallocation objective weighted by α.
- Perplexity Space Disentangling: PSD uses online PPL-reward statistics to select a threshold that separates exploitation space (lower PPL) from exploration space (higher PPL).The threshold is accepted only when the two regions show the expected correctness and error advantages, then classification error determines the optimum.
- Perplexity Space Disentangling: The threshold judgment uses conservative 95% confidence-interval boundaries to reduce sampling randomness during online estimation.The PPL queue caches samples from the two most recent batches, and empirical conditional reward probabilities are estimated from it.
- Perplexity Space Disentangling: PSD identifies hard groups in exploitation space and easy groups in exploration space as samples requiring exploration-exploitation trade-offs.These groups are ineffective for their current region because hard samples are treated as likely correct and easy samples as likely erroneous under the disentangled partition.
- Bidirectional Reward Reallocation: BRR assigns reward 1 to the maximum-PPL output in hard groups located in exploitation space, encouraging movement toward higher entropy.If the threshold condition is not met, rewards remain unchanged; the accompanying theorem states that maximum-PPL reward increases average policy entropy.
- Bidirectional Reward Reallocation: BRR assigns a penalty to the maximum-PPL output in easy groups located in exploration space, encouraging movement toward lower entropy.The easy-group procedure reverses the hard-group operation, and the corresponding theorem states that maximum-PPL penalties decrease average policy entropy.
- Policy Optimization: BRR reallocates rewards only for easy and hard groups, keeping normal-group rewards zero so reallocated and verification rewards remain orthogonal.DiPO optimizes JDAPO(θ, R) + α × JDAPO(θ, Rr), with α controlling the reallocation-loss weight.
4 Experiment
DiPO is evaluated on mathematical reasoning and function calling against strong baselines, with ablations and training analyses examining its components and behavior. It achieves the strongest reported results across the evaluated tasks and shows benefits from fine-grained exploration-exploitation control.
- Mathematical Reasoning: DiPO achieves the highest average mathematical-reasoning scores across all three evaluated model scales: 50.55% for Qwen3-4B-Base, 54.79% for Qwen3-8B-Base, and 43.56% for Qwen2.5-7B.The comparison covers ACC/mean@8 across six benchmarks.
- Mathematical Reasoning: On Qwen3-8B-Base, DiPO reaches 35.00% on AIME24 and 27.50% on AIME25, exceeding the second-best average score of 53.90% with an AVG of 54.79%.The reported results highlight performance on challenging AIME benchmarks and the larger model scale.
- Function Calling: DiPO obtains the highest Overall acc on BFCLv3, reaching 55.03% with Qwen2.5-3B-Instruct and 62.51% with Qwen2.5-7B-Instruct.The function-calling comparison uses ToolRL and ToolRL+DAPO as baselines.
- Function Calling: DiPO leads Multi-Turn Acc with 8.62% and 24.50% for the 3B and 7B models, versus ToolRL+DAPO at 8.00% and 19.75%.The corresponding margins are 0.62 and 4.75 percentage points.
- Ablation and Analysis: The combination of PSD and BRR performs best in ablations, while PSD contributes gains of 2.39 and 1.19 points with PPL rewards and 2.99 and 3.88 points with BRR.These gains are reported for the 4B and 8B models, respectively.
- Quantitative Analysis: DiPO separates error samples into higher-PPL regions and correct samples into lower-PPL regions, while maintaining later-stage exploratory improvement where DAPO slows.The analyses use PPL distributions and ACC/mean@8 training curves on AIME24 and AIME25.
5 Conclusion
DiPO addresses the exploration-exploitation trade-off in RL training by disentangling perplexity-guided sample selection and reallocating rewards for hard and easy groups. The paper reports superior results on mathematical reasoning and function calling, while its entropy analysis finds opposite effects for maximum-PPL rewards and penalties.
- Conclusion: DiPO combines Perplexity Space Disentangling with Bidirectional Reward Reallocation to implement fine-grained exploration and exploitation.PSD identifies hard samples requiring exploration and easy samples requiring exploitation; BRR incorporates these signals while minimizing disruption to verification rewards.
- Conclusion: Experiments on mathematical reasoning and function calling demonstrate DiPO’s comprehensive superiority.The reported evaluations support the method’s effectiveness in enhancing LLM reasoning through fine-grained exploration-exploitation trade-offs.
- Conclusion: High-PPL rewards increase average entropy, whereas high-PPL penalties decrease average entropy.The theoretical conclusion links maximum-PPL rewards to ΔH_avg > 0 and maximum-PPL penalties to ΔH_avg < 0.
- Conclusion: The entropy analysis relies on multiple idealized assumptions and provides only an approximate estimate of entropy-change trends.The paper states that the theoretical results require further experimental verification in actual training.
A.2 Experimental verification
The appendix verifies the theoretical relationship between perplexity-based rewards and entropy changes using Qwen3-0.6B trained on DAPO-17K. It also specifies the PSD and BRR procedures used to select and reallocate rewards.
- Experimental verification: Qwen3-0.6B trained on DAPO-17K shows entropy trends consistent with the theoretical proof when maximum-PPL rewards or penalties replace verification rewards.The verification experiment records model entropy under separate maximum-PPL reward and maximum-PPL penalty settings.
- Experimental verification: Figure 5 presents entropy curves for maximum-PPL rewards and maximum-PPL penalties on Qwen3-0.6B trained with DAPO-17K.The figure compares the entropy trajectories induced by the two reward treatments.
- Perplexity Space Disentangling: PSD estimates an optimal threshold τ∗ from conditional correctness probabilities to partition samples into exploration and exploitation subspaces.The algorithm samples outputs, computes verification rewards and perplexities, updates a queue, and selects a threshold minimizing classification error among candidates.
- Bidirectional Reward Reallocation: BRR reallocates rewards only for uniformly hard or uniformly easy groups after comparing their mean perplexity with τ∗.The hard and easy group branches assign exploration- or exploitation-oriented reallocated rewards.
D Detailed Experiment Setup
The experiments use task-specific training configurations for mathematical reasoning and function calling. Mathematical reasoning uses DAPO-17K, while function calling uses TooRL as its baseline and adapts its reward scheme for DiPO.
- Mathematical reasoning: Mathematical reasoning training uses DAPO-17K for 700 steps on 4×8 A100 GPUs.Comparison methods use the configurations in Table 5 and remaining VERL defaults.
- Mathematical reasoning: The mathematical reasoning prompt asks models to think step by step and place the final answer within \boxed{}.The same template distinguishes the question placeholder from the required answer format.
- Function calling: TooRL is the function-calling baseline and introduced GRPO into function calling as an open-source method.Its reward design covers formatting and matching, including tool names, parameter names, and parameter content.
- Function calling: For DiPO adaptation, the maximum-reward sample is treated as correct, while BRR uses rewards 4 and -3 for correct and error reallocations.TooRL’s validation rewards span [−3, 4], with matching-based components.
E.1 Results on Llama3.1-8B-Instruct
On Llama3.1-8B-Instruct evaluated on GSM8K and MATH, DiPO outperforms the compared methods and achieves the strongest reported MATH and overall-average results.
- Results: 56.75% on MATH and 73.39% overall average are DiPO’s best reported results for Llama3.1-8B-Instruct.The paper describes these results as demonstrating consistent effectiveness across model families.
- Results: Entropy loss and CDE perform below the DAPO baseline, especially on MATH, whereas DiPO shows moderate improvement.The comparison is reported across GSM8K and MATH for Llama3.1-8B-Instruct.
E.2 Results of Majority Vote
DiPO achieves the highest average mathematical-reasoning performance across all three evaluated base models, with strong results across six benchmarks.
- E.2 Results of Majority Vote: DiPO achieves the highest average score across six benchmarks for Qwen3-4B-Base, Qwen3-8B-Base, and Qwen2.5-7B.The reported averages are 56.41%, 60.65%, and 49.79%, respectively.
- E.2 Results of Majority Vote: DiPO exceeds CDE by 0.90 percentage points on Qwen3-4B-Base and by 0.95 percentage points on Qwen3-8B-Base.The corresponding DiPO averages are 56.41% versus 55.51% and 60.65% versus 59.70%.
- E.2 Results of Majority Vote: DiPO exceeds DAPO by 0.82 percentage points on Qwen2.5-7B, achieving 49.79% versus 48.97%.It ranks first on AIME24, AMC, and OLY for this model.
- E.2 Results of Majority Vote: DiPO ranks first in 10 of 18 benchmark-model pairs, supporting consistent performance across the evaluated mathematical reasoning settings.It also achieves state-of-the-art results on several individual benchmarks.
E.3 Coefficient Sensitivity Analysis
DiPO remains comparatively stable across a tenfold coefficient change, whereas entropy loss is highly sensitive to its coefficient.
- E.3 Coefficient Sensitivity Analysis: 54.79%: DiPO with coefficient 0.10 improves over the DAPO baseline by 1.56 AVG points on Qwen3-8B-Base.The DAPO baseline is 53.23%.
- E.3 Coefficient Sensitivity Analysis: 1.43 AVG points: DiPO’s performance variation across coefficients 0.10 and 1.00 is limited despite the tenfold coefficient increase.At coefficient 1.00, DiPO maintains an AVG of 53.36%.
- E.3 Coefficient Sensitivity Analysis: 7.90 points: increasing entropy-loss coefficient from 0.001 to 0.01 reduces AVG from 53.90% to 46.00%.The 0.01 setting is 7.23 points below the DAPO baseline.
E.4 Results of Risk Prediction
The risk-prediction evaluation compares DiPO and DAPO on Qwen3-8B using a 4,000-example dataset, with DiPO achieving stronger accuracy, recall, and F1 performance.
- E.4 Results of Risk Prediction: The risk-prediction prompt applies a strict no-risk standard, assigning a risk label when ambiguity or a negative indicator appears.The prompt targets potential crises across product experience, fraud, and regulatory compliance.
- E.4 Results of Risk Prediction: The evaluation is bounded by confidentiality-related replacement of bracketed prompt content with placeholders.This limits visibility into the complete prompt wording.
- E.4 Results of Risk Prediction: 4,000 examples: the risk-prediction dataset contains 3,000 training examples and 1,000 test examples.The comparison evaluates DiPO and DAPO trained on Qwen3-8B.
- E.4 Results of Risk Prediction: 78.37% accuracy, 79.49% recall, and 86.84% F1: DiPO records the highest reported values for these metrics.The results are reported as means over eight independent inferences.
- E.4 Results of Risk Prediction: 95.69% precision: DiPO remains close to DAPO’s 95.96% precision while achieving higher recall.The paper highlights recall as important for minimizing missed critical risks.
E.6 Case Analysis
Case visualizations contrast DiPO and DAPO on extreme groups, showing lower entropy for DiPO’s correct answers and denser high-entropy tokens for its incorrect answers.
- E.6 Case Analysis: Darker colors indicate higher entropy in the DAPO and DiPO case visualizations.The cases were generated at training step 500 on DAPO-17K.
- E.6 Case Analysis: For correct answers, DiPO cases exhibit lower overall entropy than the corresponding DAPO cases.This pattern is presented as evidence of stronger exploitation in correct cases.
- E.6 Case Analysis: For incorrect answers, DiPO cases contain a greater density of high-entropy tokens than DAPO cases.The paper associates this pattern with stronger exploration in incorrect cases.
- E.6 Case Analysis: The examples include mathematical prompts involving a 5x5 chip grid and boxes containing pens and pencils.These prompts illustrate the extreme-group cases shown for DAPO and DiPO.