PaperlayerView exact revision on arXiv

Source-linked AI summary

CPPO: Accelerating the Training of Group Relative Policy Optimization-Based Reasoning Models

Zhihang Lin, Mingbao Lin, Yuan Xie, Rongrong Ji

arXiv:2503.22342v2cs.AI

TL;DR

GRPO-based reasoning-model training is costly because it samples multiple completions and performs group-level computations, while completion count improves accuracy but increases training time multiplicatively. CPPO prunes low-absolute-advantage completions and dynamically reallocates freed capacity, achieving substantial speedups while preserving or sometimes enhancing accuracy.

  • Problem

    GRPO training is computationally expensive because it requires multiple completions per question and additional group-level reward, advantage, and probability-ratio computations.

  • Method

    CPPO computes relative advantages, prunes low-absolute-advantage completions for loss computation, and dynamically allocates freed capacity to improve GPU utilization.

  • Results

    CPPO achieves up to 7.98× speedup on GSM8K and 3.48× on Math while preserving or sometimes enhancing accuracy compared with GRPO.

  • Takeaways & Limitations

    CPPO reduces GRPO-based reasoning-model training cost through selective completion use and improved parallel utilization.

  • Takeaways & Limitations

    CPPO does not reduce completion-generation time and was evaluated only on models smaller than 14B and the Math and GSM8K datasets.

Abstract

from arXiv · show

This paper introduces Completion Pruning Policy Optimization (CPPO) to accelerate the training of reasoning models based on Group Relative Policy Optimization (GRPO). GRPO, while effective, incurs high training costs due to the need to sample multiple completions for each question. Our experiment and theoretical analysis reveal that the number of completions impacts model accuracy yet increases training time multiplicatively, and not all completions contribute equally to policy training -- their contribution depends on their relative advantage. To address these issues, we propose CPPO, which prunes completions with low absolute advantages, significantly reducing the number needed for gradient calculation and updates. Additionally, we introduce a dynamic completion allocation strategy to maximize GPU utilization by incorporating additional questions, further enhancing training efficiency. Experiments show that CPPO achieves up to $7.98\times$ speedup on GSM8K and $3.48\times$ on Math while preserving or even enhancing the accuracy compared to the original GRPO. We release our code at \href{https://github.com/lzhxmu/CPPO}{https://github.com/lzhxmu/CPPO}.

1 Introduction

Reasoning-model reinforcement learning is effective but computationally expensive, especially under GRPO’s multiple-completion design. CPPO prunes low-contribution completions and uses this reduction to accelerate training while preserving or enhancing accuracy.

  • 1 Introduction: GRPO requires multiple completions per question, reward and advantage calculations, and probability-ratio computations, creating substantial training overhead.The overhead limits training efficiency and scalability.
  • 1 Introduction: Completion contributions vary with relative advantage, so pruning low-absolute-advantage completions can reduce the number used for policy updates.CPPO retains completions with high absolute advantage for loss computation.
  • 1 Introduction: CPPO combines completion pruning with dynamic completion allocation to improve training efficiency and GPU utilization.The supplied introduction describes pruning as the core acceleration mechanism; dynamic allocation is not detailed in these passages.
  • 1 Introduction: 7.98× speedup on GSM8K and 3.48× on Math were achieved while preserving or enhancing accuracy compared with original GRPO.Experiments used Qwen-2.5 models across Math and GSM8K datasets.

2 Related Work

Related work develops reasoning models and reinforcement-learning methods, while also seeking more efficient inference. These approaches share the challenge that reasoning and multi-completion training can require substantial computation.

  • Large Scale Reasoning Models: Large language models are being advanced for reasoning tasks including mathematics, coding, and scientific reasoning.The passage references process-based verification and adaptive response-distribution updates as examples of this work.
  • Reinforcement Learning: Reinforcement-learning algorithms for reasoning commonly require multiple completions per question, resulting in substantial computational costs.The passage identifies an urgent need to accelerate reinforcement-learning training.
  • Inference Acceleration for Reasoning Models: Inference-acceleration work selectively compresses or skips less important chain-of-thought tokens to balance efficiency and accuracy.TokenSkip is cited as an example of controllable chain-of-thought compression.
  • Inference Acceleration for Reasoning Models: Figure 1 compares completion number with accuracy and training time on GSM8K using Qwen2.5-1.5B-Instruct.The left panel concerns accuracy and the right panel concerns training time.

3 Method

GRPO training is expensive because it samples many completions and performs forward computation across the policy, reference, and old policy models. CPPO reduces this cost by retaining high-absolute-advantage completions and dynamically filling available GPU capacity with completions from additional questions.

  • GRPO Cost: GRPO samples a group of completions per question, and its forward computation scales by 3× the completion number.For example, 64 completions require 192 forward passes per question across three models.
  • GRPO Cost: More completions improve accuracy but increase training time multiplicatively, producing diminishing performance returns and making simple completion reduction impractical.The GSM8K ablation used Qwen2.5-1.5B-Instruct.
  • Completion Contribution: Completion contributions vary substantially because the advantage-weighted probability ratio directly ties each completion’s policy-training signal to its relative advantage.The policy gradient and KL terms also affect contribution, but low values in any required component can make the overall contribution minimal.
  • Completion Pruning: CPPO computes relative advantages before policy-model forward computation and prunes completions whose absolute advantages fall below a predefined threshold γ.The retained completions are used for the policy, reference, and old policy model forwards and gradient update.
  • Completion Pruning: For multi-GPU training, CPPO retains the k largest-absolute-advantage completions per question on each GPU to reduce imbalance from the bucket effect.The bucket effect occurs when the device processing the largest number of completions bottlenecks overall training efficiency.
  • Dynamic Completion Allocation: Dynamic allocation fills unused device capacity with pruned completions from additional questions, maximizing utilization and enabling more questions per batch with fewer convergence steps.Newly incorporated completions undergo the same pruning process.

4 Experiments

Experiments on GSM8K, MATH, and out-of-distribution benchmarks show that CPPO preserves or improves accuracy while reducing training time through advantage-based pruning and dynamic completion allocation.

  • Experimental Setup: CPPO evaluates Qwen2.5 models on GSM8K and MATH, with additional out-of-distribution evaluation on AMC2023 and AIME2024.Qwen2.5-1.5B-Instruct is used for GSM8K, while Qwen2.5-7B-Instruct is used for MATH.
  • GSM8K Results: 80.01% accuracy at a pruning rate of 87.50% surpasses GRPO’s 77.38% on GSM8K by 2.63%.CPPO achieves comparable or higher accuracy across various pruning rates.
  • GSM8K Results: 7.98× acceleration at a pruning rate of 93.75% demonstrates substantial CPPO training-speed improvement on GSM8K.The gain combines completion pruning with allocation that uses freed memory to process more questions per batch and reduce training steps.
  • MATH Results: 75.95% accuracy and 3.48× faster training at an 87.5% pruning rate outperform GRPO’s 75.26% on MATH without sacrificing accuracy.Evaluation on AMC2023 and AIME2024 also preserves out-of-distribution reasoning ability.
  • Accuracy Analysis: CPPO’s accuracy gains persist when questions per training step are matched, indicating that higher-quality retained completions—not merely larger batches—account for the improvement.CPPO retains the top k completions from a larger group, whereas GRPO directly generates k completions for updating.
  • Accuracy Analysis: Removing low-quality completions generally improves performance, but pruning 93.75% can discard high-quality completions and reduce training effectiveness.High absolute advantage completions provide stronger training signals, while low-quality completions can introduce learning noise.
  • Generalization: CPPO extends to DAPO and Dr.GRPO, where combining the method with these algorithms further improves training speed and accuracy.This supports CPPO’s generalizability across group relative policy optimization-based algorithms.
  • Ablation Study: The completion pruning module improves training efficiency by 1.23×, and completion allocation raises the improvement to 1.65×.The ablation uses Qwen2.5-7B-Instruct on Math.

5 Limitations and Future Work

CPPO’s evaluation is limited by completion-generation cost, model scale, and task coverage; future work targets larger models, more tasks, and faster generation.

  • Limitations: CPPO does not reduce completion-generation time, so its speedup may diminish when generation dominates total training time.Inference acceleration methods are described as orthogonal ways to improve efficiency further.
  • Limitations: Experiments cover models smaller than 14B and the Math and GSM8K datasets because of limited academic GPU resources.The authors plan to evaluate larger models and more tasks.

6 Conclusion

CPPO improves GRPO-based reasoning-model training by pruning completions according to relative advantage and dynamically allocating additional questions to preserve GPU utilization. The method reduces computational overhead while maintaining or sometimes improving accuracy and reports substantial speedups.

  • CPPO selectively prunes completions based on their relative advantages to improve GRPO training efficiency.The method retains completions with higher absolute advantages for policy updates.
  • Dynamic completion allocation complements pruning by using additional questions and GPU parallelism to improve training speed.The implementation samples b/(1 −p) questions based on the pruning rate p before pruning completions.
  • CPPO achieves up to 7.98× speedup on GSM8K and 3.48× on Math while sometimes preserving or enhancing accuracy compared with GRPO.The conclusion presents CPPO as a lower-cost approach to optimizing reasoning-model training.
  • CPPO defines dataset-specific reward components for GSM8K and Math to evaluate completion format and answer correctness.GSM8K distinguishes direct and regularly parsed correct answers, while Math uses direct answer matching.
  • The algorithm samples groups of completions, computes rewards and advantages, retains k = G × (1 −p) high-absolute-advantage completions, and updates the policy.Completion allocation is implemented before pruning because the allocated number can be pre-computed from p.

D Experimental Results on Larger Models

Experiments with Qwen2.5-14B-Instruct show that CPPO scales to larger models, accelerating training without compromising accuracy on the MATH test subset.

  • CPPO accelerates Qwen2.5-14B-Instruct training by up to 2.83× without compromising accuracy.The comparison is conducted on the MATH test subset, with retained completions denoted by k = ⌊G × (1 −P)⌋.

E CPPO Results on GSM8K with Qwen2.5-7B-Instruct

On GSM8K with Qwen2.5-7B-Instruct, CPPO remains robust despite limited baseline headroom, delivering speedup without accuracy loss.

  • 4.67× speedup is achieved by CPPO on GSM8K with Qwen2.5-7B-Instruct without loss of accuracy.The model starts at 83.00% accuracy, limiting potential improvement over the baseline.
  • Qwen2.5-7B-Instruct gains 8.37% from GRPO over the baseline, below the 21.66% gain reported for Qwen2.5-1.5B-Instruct.The paper attributes the difference to GSM8K being relatively easy for the 7B model.

F Experimental Results on Different LLM Backbones

Experiments across Llama backbones indicate that CPPO generalizes beyond Qwen models, accelerating training while maintaining accuracy.

  • CPPO accelerates Llama-model training by up to 3.13× without compromising accuracy.The results are presented for the GSM8K test subset, where retained completions are denoted by k = ⌊G × (1 −P)⌋.
  • The Llama experiments support CPPO’s generalizability across different large-language-model backbones.

G Comparison with Other Reinforcement Learning Algorithms

On the GSM8K test subset, CPPO outperforms REINFORCE++ and PPO without KL divergence while using a reported training time of 5192s. The comparison is conducted with Qwen2.5-1.5B-Instruct on the verl framework, retaining k = ⌊G × (1 −P)⌋ completions after pruning.

  • Results: 78.92% accuracy is achieved by CPPO on the GSM8K test subset, the best result among the compared reinforcement learning algorithms.The experiment trains Qwen2.5-1.5B-Instruct on the GSM8K training subset using the verl framework.
  • Training efficiency: 5192s is CPPO’s reported training time in the GSM8K algorithm comparison.The same comparison reports CPPO as outperforming REINFORCE++ and PPO without KL divergence.
  • Experimental setup: After pruning, the retained completion count is k = ⌊G × (1 −P)⌋.This retention rule is specified for the GSM8K comparison on the verl framework.

H Stability and Convergence

CPPO’s reward curves on GSM8K and Math preserve GRPO’s training stability while showing faster convergence. Case studies further report maintained or improved reasoning ability, including instances where high-pruning CPPO succeeds after GRPO fails.

  • Stability and convergence: CPPO preserves GRPO’s training stability and improves convergence speed on GSM8K and Math reward curves.The curves do not crash or show drastic fluctuations, while exhibiting a clear upward trend toward higher reward values.
  • Case studies: CPPO maintains or improves the trained model’s reasoning ability while pruning completions with low absolute advantages.The case studies report up to 7.98× acceleration on GSM8K.
  • Case studies: In some cases, CPPO with a high pruning rate produces the correct answer when GRPO fails.The case studies compare completions generated by GRPO and CPPO on GSM8K questions.
  • Completion analysis: The completion examples distinguish correct and incorrect formats from correct and incorrect answers.These categories include correct format with incorrect answer and incorrect format with correct answer, illustrating different completion types.
Loading 2503.22342v2…