PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning from Hard Prompts: Difficulty-aware Advantage Amplification in Dynamic Sampling

Siyuan Gan, Yuhan Li, Xiran Wang, Linjian Meng, Boyan Wang, Zhen Zhao, Jing Huo, Lei Bai, Yang Gao

arXiv:2608.27982v1cs.AI

TL;DR

Dynamic Sampling improves DAPO’s training stability but asymmetrically amplifies responses on hard prompts, limiting its use of hard-to-sample correct responses. The paper proposes DAA and integrates it into DAPO as DA3PO; experiments show consistent accuracy gains across seven benchmarks and both model scales, while fixed amplification factors remain a limitation.

  • Problem

    Dynamic Sampling reduces training efficiency because it amplifies incorrect responses more than correct responses on hard prompts, limiting capitalization on hard-to-sample correct responses.

  • Method

    DAA amplifies the advantages of hard-to-sample correct responses on hard prompts, and its integration with DAPO produces DA3PO.

  • Results

    DA3PO yields consistent accuracy enhancements across both model scales and all seven mathematical benchmarks, significantly outperforming GRPO and other classical GRPO variants.

  • Takeaways & Limitations

    DA3PO provides a straightforward way to address Dynamic Sampling’s asymmetric advantage amplification while remaining compatible with GRPO and its variants.

  • Takeaways & Limitations

    DA3PO uses constant λ and fixed difficulty threshold τ, which may not optimally compensate asymmetric amplification across all prompt difficulties during training.

Abstract

from arXiv · show

Decoupled Clip and Dynamic Sampling Policy Optimization (DAPO) is a prominent variant of Group Relative Policy Optimization (GRPO). DAPO introduces several improvements over GRPO. Among these, Dynamic Sampling contributes the most to DAPO's accuracy gains relative to GRPO. To improve accuracy, Dynamic Sampling enhances training stability by eliminating zero policy gradients from zero advantages. Specifically, it avoids such zero gradients by filtering out prompts where sampled responses are either entirely correct or incorrect. However, our theoretical analysis shows that Dynamic Sampling decrease training efficiency as it cannot effectively utilize hard-to-sample correct responses on hard prompts. Formally, it asymmetrically amplifies the advantages of distinct responses to the same prompts. On hard prompts, incorrect responses undergo greater amplification than correct ones. This leads the model to avoid generating the observed incorrect responses rather than capitalizing on the hard-to-sample correct ones on hard prompts, resulting in low training efficiency. To improve training efficiency, we propose Direct Advantage Amplification (DAA), which amplifies the advantages of hard-to-sample correct responses on hard prompts, as obtained by Dynamic Sampling. This ensures that, when Dynamic Sampling is used, these hard-to-sample responses can be effectively capitalized on, implying higher training efficiency. By integrating DAA into DAPO, we obtain Difficulty-aware Advantage Amplification Policy Optimization (DA3PO), which is implemented with fewer than 30 lines of code from DAPO. Experiments show that DA3PO significantly outperforms GRPO and other classical GRPO variants.

1 Introduction

DAPO extends GRPO with several improvements, with Dynamic Sampling contributing most to its accuracy gains by removing zero policy gradients. The paper identifies an efficiency problem in Dynamic Sampling and proposes DAA, yielding DA3PO, which improves accuracy across tested models and benchmarks.

  • DAPO and its motivation: Dynamic Sampling contributes the most to DAPO’s accuracy enhancements over GRPO among its improvements.It improves training stability by eliminating zero policy gradients caused by zero advantages.
  • DAPO and its motivation: Dynamic Sampling filters prompts whose sampled responses are entirely correct or entirely incorrect, avoiding zero advantages and zero policy gradients.The resulting training process keeps prompts with mixed response correctness.
  • Problem with Dynamic Sampling: Dynamic Sampling reduces training efficiency because it cannot effectively capitalize on hard-to-sample correct responses on hard prompts.Its asymmetric advantage amplification favors incorrect responses on hard prompts.
  • Proposed solution: DAA amplifies the advantages of hard-to-sample correct responses on hard prompts obtained through Dynamic Sampling.Integrating DAA into DAPO produces DA3PO in fewer than 30 lines of code.
  • Results: DA3PO yields consistent accuracy enhancements across all tested models and seven mathematical reasoning benchmarks, significantly outperforming GRPO, DAPO, and GSPO.The evaluation covers AIME24, AIME25, AIME26, AMC, Minerva, Olympiad, and MATH.

2 Preliminary

GRPO samples response groups, assigns verifier-based rewards, computes advantages, and updates the policy with a clipped objective. DAPO adds clipping, reward shaping, token-level normalization, and Dynamic Sampling to improve training efficiency and stability.

  • GRPO: GRPO samples a prompt and a group of G responses from policy πθ at each training step.The prompt is sampled from dataset D, and each response has a conditional probability under πθ.
  • GRPO: GRPO uses a behavior policy πθold to define token-level importance-sampling ratios for policy updates.The ratio compares current-policy and behavior-policy token probabilities.
  • GRPO: A rule-based verifier assigns reward 1 to correct responses and −1 otherwise, from which token-level advantages are computed.The policy is updated by maximizing a clipped objective with per-sample loss averaging.
  • Other DAPO improvements: DAPO additionally applies Clip-Higher, overlong reward shaping, and a global token-level normalizer.These components respectively support exploration, discourage overly long responses, and ensure every token contributes equally to the gradient.
  • Dynamic Sampling: DAPO uses Dynamic Sampling to filter out groups with identical rewards, retaining prompts with positive-response rates 0 < p̂ < 1.This avoids zero advantages and zero policy gradients while maintaining a consistent number of prompts.

3 Theory

Dynamic Sampling asymmetrically amplifies response advantages according to prompt difficulty. On hard prompts, it amplifies incorrect-response penalties more than correct-response rewards, reducing training efficiency.

  • Theorem 3.1: Dynamic Sampling induces asymmetric amplification of correct and incorrect response advantages that varies with prompt difficulty.Theorem 3.1 characterizes amplification factors α+ and α− for correct and incorrect responses.
  • Theorem 3.1: On hard prompts, α+ remains close to 1 while α− becomes much larger, disproportionately amplifying incorrect-response penalties.This pattern is observed across the hard-prompt regime p ≪ 0.5.
  • Corollary 3.2: For p < 0.5, γ = α+/α− is strictly less than 1 for every group size G.The ratio equals 1 only at p = 0.5; for p > 0.5, it exceeds 1.
  • Corollary 3.2: At G = 8 and p = 0.1, α+ ≈ 1.00, α− ≈ 1.92, and γ ≈ 0.52.At G = 16 and p = 0.1, the corresponding values are α+ ≈ 1.00, α− ≈ 1.26, and γ ≈ 0.79.
  • Corollary 3.2: Smaller group sizes intensify the asymmetry between incorrect-response penalties and correct-response rewards.The penalty on incorrect responses is amplified substantially more than the reward for correct ones.
  • Implication: As p approaches 0, α− grows unboundedly while α+ remains close to 1, so the model avoids observed incorrect responses without strengthening hard-to-sample correct responses.This behavior reduces training efficiency on hard prompts.

4 Algorithm

DAA modifies Dynamic Sampling by amplifying hard-to-sample correct responses on hard prompts, yielding DA3PO when integrated into DAPO. The resulting method retains a lightweight implementation and is reported to improve training efficiency and accuracy.

  • Direct Advantage Amplification: DAA amplifies the advantages of hard-to-sample correct responses on hard prompts obtained through Dynamic Sampling.The method uses a positive constant to strengthen these responses and address Dynamic Sampling’s asymmetric amplification.
  • Implementation: DA3PO requires fewer than 30 lines of code derived from DAPO, with only marked additions differing from standard DAPO.The implementation changes are described as straightforward and minimal.
  • Direct Advantage Amplification: DAA uses a constant λ > 1 instead of estimating prompt difficulty p, preserving the hard-prompt amplification effect with minimal DAPO modifications.The design avoids tracking the changing true value of γ as the policy updates.
  • DAA Details: DAA activates only for correct responses with Ri > 0 when estimated prompt difficulty satisfies p̂ < τ.Here τ ∈ (0, 1) is the difficulty threshold.
  • DA3PO Objective: DA3PO integrates DAA into the DAPO objective and updates the token-level loss using the DAA advantage Ãi,t.The training procedure retains DAPO’s Dynamic Sampling and modifies the advantage used in optimization.
  • Results and Discussion: DA3PO yields consistent accuracy enhancements across all tested models and benchmarks, outperforming GRPO, DAPO, and GSPO.DAA is also described as compatible with GRPO and its variants because it modifies Dynamic Sampling.

5 Experiments

Experiments evaluate DA3PO across two base-model scales and seven mathematical reasoning benchmarks. DA3PO consistently improves accuracy, broadens correct-response coverage, reshapes training trajectories, and remains robust across amplification factors.

  • Setup: Experiments evaluate DA3PO on Qwen3-4B-Base and Qwen3-8B-Base using seven mathematical reasoning benchmarks.Training uses DAPO-Math-17K, while evaluation reports avg@32 for AIME and AMC and avg@8 for Minerva, Olympiad, and MATH.
  • Main Results: At least 5.02% and 4.95% average-accuracy gains over other GRPO variants occur on Qwen3-4B-Base and Qwen3-8B-Base, respectively.DA3PO also achieves the best performance on the hardest benchmarks.
  • Main Results: Up to 5.00% and 6.25% gains over the best baseline occur on AIME benchmarks for Qwen3-4B-Base and Qwen3-8B-Base, respectively.On MATH, DA3PO still exceeds other variants by 5.33% and 2.93% at the two model scales.
  • Response Coverage: DA3PO broadens the space of correct responses rather than collapsing the policy onto a single solution mode.This conclusion is supported by pass@k results across the evaluated benchmarks.
  • Response Coverage: At least 5.81% and 5.87% average pass@k gains over baselines occur on Qwen3-4B-Base and Qwen3-8B-Base, respectively.DA3PO exceeds all baselines on five of seven benchmarks at both scales, including up to 10.00% and at least 13.33% on AIME24.
  • Training Dynamics: On both model scales, DA3PO remains above GRPO, DAPO, and GSPO throughout AIME24 RL training, not only at the best checkpoint.The validation trajectory therefore reflects performance throughout training rather than only checkpoint selection.
  • Robustness: λ = 2.0 yields the best average accuracy on both model scales: 39.63% on Qwen3-4B-Base and 46.89% on Qwen3-8B-Base.DA3PO outperforms DAPO across λ values 1.5, 2.0, and 2.5; λ = 1.5 already improves over DAPO by 4.42% and 3.38%.

6 Conclusion

The paper identifies asymmetric advantage amplification as an efficiency problem in Dynamic Sampling and proposes DAA to amplify hard-to-sample correct responses. Integrated into DAPO as DA3PO, this approach achieves consistent gains across both model scales and seven mathematical benchmarks.

  • Conclusion: Dynamic Sampling asymmetrically amplifies advantages and reduces training efficiency in DAPO.The paper attributes this issue to weaker capitalization of hard-to-sample correct responses on hard prompts.
  • Conclusion: DAA amplifies advantages for hard-to-sample correct responses on hard prompts, and its integration with DAPO produces DA3PO.DA3PO requires fewer than 30 lines of code derived from DAPO.
  • Conclusion: DA3PO delivers consistent accuracy enhancements across both model scales and all seven evaluated mathematical benchmarks.The paper reports significant outperformance over GRPO and other classical GRPO variants.

Limitations

DA3PO uses a constant λ and fixed difficulty threshold τ to amplify hard-to-sample correct responses. Adaptive factors that adjust to evolving prompt difficulty remain a promising direction.

  • DA3PO uses a constant λ and fixed difficulty threshold τ to amplify hard-to-sample correct responses on hard prompts.The design preserves amplification on hard prompts but may not optimally compensate asymmetric amplification across all prompt difficulties during training.
  • Replacing λ and τ with adaptive factors that dynamically adjust to evolving prompt difficulty is identified as a promising future direction.

A Related Work

Prior work improves GRPO stability and efficiency through objective, entropy, clipping, masking, scheduling, and advantage-reweighting changes. The paper instead reveals Dynamic Sampling’s efficiency cost and proposes DA3PO, which preserves stability while improving accuracy across scales and benchmarks.

  • GRPO instability can accumulate during long-term training, motivating methods that improve the objective function and training stability.
  • DAPO is a representative GRPO variant that introduces four improvements, including Dynamic Sampling, and achieves higher accuracy than GRPO.
  • Prior efficiency methods regulate entropy dynamics through clipping, token masking, temperature scheduling, and advantage reweighting, while largely overlooking Dynamic Sampling’s efficiency impact.
  • DA3PO improves Dynamic Sampling’s training efficiency while preserving its training-stability benefits and significantly improves accuracy across model scales and all benchmarks.

B.1 Proof of Eq. (5)

Conditioned on a correct response, the proof partitions the sample space into all-correct and mixed outcomes, then uses their probabilities and conditional expected advantages to derive Eq. (5).

  • Conditioning on Racc(oi)=1 makes the all-incorrect event impossible, leaving all-correct and mixed-response cases.
  • The probability of all responses being correct is computed by requiring the remaining G−1 responses to be independently correct.
  • When all G responses are correct, identical rewards produce zero variance under z-score normalisation.
  • The proof applies the law of total expectation over the all-correct and mixed-response partition.
  • Substituting the preceding equations and rearranging under p ∈ (0, 1) yields Eq. (5).

B.2 Proof of Eq. (6)

Conditioned on an incorrect response, the proof partitions outcomes into all-incorrect and mixed cases, then uses their probabilities and conditional expected advantages to derive Eq. (6).

  • Conditioning on Racc(oi)=−1 makes the all-correct event impossible, leaving all-incorrect and mixed-response cases.
  • The probability of all responses being incorrect is computed by requiring the remaining G−1 responses to be independently incorrect.
  • When all G responses are incorrect, identical rewards produce zero variance under z-score normalisation.
  • The proof applies the law of total expectation over the all-incorrect and mixed-response partition.
  • Substituting the preceding equations and rearranging under p ∈ (0, 1) yields Eq. (6).
Loading 2608.27982v1…