PaperlayerView exact revision on arXiv

Source-linked AI summary

Prompt Curriculum Learning for Efficient LLM Post-Training

Zhaolin Gao, Joongwon Kim, Wen Sun, Thorsten Joachims, Sid Wang, Richard Yuanzhe Pang, Liang Tan

arXiv:2510.01135v1cs.LGcs.CL

TL;DR

Existing intermediate-difficulty prompt selection relies on costly current-policy rollouts or off-policy reward dictionaries, while batch-size effects on RL convergence remain underexplored. PCL trains a value model online, predicts prompt difficulty with a forward pass, and greedily selects prompts near a target difficulty threshold for on-policy RL. PCL either achieves the highest performance or requires significantly less training time to reach comparable performance.

  • Problem

    Existing intermediate-difficulty prompt selection relies on costly current-policy rollouts or off-policy reward dictionaries, while batch-size effects on RL convergence remain underexplored.

  • Method

    PCL trains a value model online, predicts prompt difficulty with a forward pass, and greedily selects prompts near a target difficulty threshold for on-policy RL.

  • Results

    PCL either achieves the highest performance or requires significantly less training time to reach comparable performance.

  • Takeaways & Limitations

    Intermediate-difficulty prompts near p(x) ≈0.5 provide the strongest gradient signal and sample efficiency, enabling a better performance-efficiency tradeoff for reasoning-focused RL.

  • Takeaways & Limitations

    The authors state that PCL has several limitations and leave avenues for future work.

Abstract

from arXiv · show

We introduce Prompt Curriculum Learning (PCL), a lightweight reinforcement learning (RL) algorithm that selects intermediate-difficulty prompts using a learned value model to post-train language models. Since post-training LLMs via RL remains sensitive to batching and prompt selection strategies, we first conduct a series of systematic experiments where we (1) determine the optimal training batch size that balances generation efficiency and gradient quality and (2) establish the importance of focusing on prompts of intermediate difficulty for the policy. We build upon these results to design PCL, which identifies prompts of intermediate difficulty for the current policy in an on-policy manner by using a value model that is concurrently updated based on the current policy. By focusing on informative prompts that yield high effective ratios, PCL achieves either the highest performance or requires significantly less time to reach comparable performance to its counterparts. Compared to rollout-based filtering methods, PCL avoids costly rollouts and achieves $12.1\times$ and $16.9\times$ faster speed on identifying intermediate-difficulty prompts when training on MATH and DeepScaleR, respectively. We further demonstrate that our value model accurately predicts prompt difficulty and allows PCL to focus on progressively more challenging prompts during RL. Our results present a new methodology that delivers improved tradeoff between upper-bound performance and efficiency for reasoning-focused RL.

1 Introduction

The paper studies how batch configuration and prompt difficulty affect RL convergence, then introduces PCL to select intermediate-difficulty prompts online with a learned value model. PCL either achieves the highest performance or reaches comparable performance in substantially less training time.

  • 1 Introduction: RL enables language models to self-explore and improve iteratively by generating responses online and using rule-based rewards.The paper situates its study in reasoning-focused RL methods such as PPO and GRPO.
  • 1 Introduction: Prior intermediate-prompt methods use costly current-policy rollouts or stale reward dictionaries, while batch-size effects remain underexplored.These approaches respectively incur online-generation overhead or suffer from off-policyness on large datasets.
  • 1 Introduction: An optimal batch size occurs at the transition from sublinear to linear generation-time growth, where convergence speed is maximized.Larger batches reduce gradient noise but eventually increase generation time enough to limit update frequency.
  • 1 Introduction: 12.1× and 16.9× faster prompt filtering on MATH and DeepScaleR, respectively, makes PCL faster than rollout-based filtering.PCL trains a value model online rather than relying on costly filtering rollouts.
  • 1 Introduction: Prompts with approximately 50% success probability yield the highest gradient norms and require fewer samples for informative updates.Very easy or very hard prompts tend to produce vanishing gradient signals, and the finding is validated across models, datasets, and batch configurations.
  • 1 Introduction: PCL either achieves the highest performance or requires substantially less training time to reach comparable performance across models and datasets.The method dynamically selects intermediate-difficulty prompts using a value model.

2 Problem Setup

The setup uses a purely on-policy GRPO objective for binary-reward solution generation and examines how batch configuration affects generation cost and learning signals.

  • 2 Problem Setup: The policy maximizes an on-policy GRPO objective using token-level probability ratios weighted by the solution advantage.The formulation omits KL regularization and standard-deviation-based advantage regularization.
  • 2 Problem Setup: Figure 2 compares training reward over steps and wall-clock time with generation time and test accuracy across batch configurations on MATH and DeepScaleR.Batch configurations are represented by the number of prompts m and generations per prompt n, with log-scaled axes.

3 Preliminary Investigations

The investigations identify an optimal batch-size regime balancing generation efficiency against gradient quality, and show that intermediate-difficulty prompts provide the most informative learning signals. These findings motivate controlling both batch decomposition and prompt difficulty in RL training.

  • 3 Preliminary Investigations: The experiments evaluate batch and prompt-selection effects across MATH and DeepScaleR, multiple benchmarks, four models, and varied training configurations.The setup uses rule-based math verification and synchronous generation-optimization training, with broader robustness results reported across architectures, datasets, context lengths, hardware, engines, and batch configurations.
  • 3.1 Optimal Batch Size: Larger batches converge faster in training steps because reduced gradient noise permits higher learning rates, but generation time eventually scales linearly with batch size.Generation time is initially sublinear because the longest response dominates, then becomes compute-bound as batch size increases.
  • 3.1 Optimal Batch Size: Around 8K samples per batch gives the fastest convergence, whether factorized as (m,n) = (512,16), (256,32), or (128,64); this optimum occurs at the transition from sublinear to linear generation-time scaling.Compared with larger batch sizes, it allows more frequent updates in the same amount of time.
  • 3.2 Optimal Number of Prompts and Generations per Prompt: The controlled downsampling procedure retains prompts near each specified difficulty threshold, validating the comparison of prompt difficulty under fixed total batch size.The ablation keeps m × n = 4096 while varying n from 2 to 128 and avoids reusing selection responses for training.
  • 3.2 Optimal Number of Prompts and Generations per Prompt: For fixed batch size, increasing generations per prompt raises the effective ratio, while p(x) = 0.5 achieves high effective ratios with relatively few generations.At n = 16, p(x) = 0.5 already exceeds the effective ratio of other thresholds even at n = 128.
  • 3.2 Optimal Number of Prompts and Generations per Prompt: Prompts with p(x) = 0.5 yield the highest gradient norms and test accuracy, although accuracy degrades beyond n = 32 at this difficulty.Intermediate difficulty allows smaller n and larger m, combining a high effective ratio with greater prompt diversity.

4 PCL: Prompt Curriculum Learning

PCL addresses the cost of identifying informative prompts during online RL by using a learned value model to select candidates near an intermediate difficulty target. It updates that model from the same training responses, avoiding additional rollouts while keeping value-model overhead low.

  • 4 PCL: Prompt Curriculum Learning: PCL samples km candidate prompts, predicts each prompt’s expected reward with one value-model forward pass, and selects m prompts closest to threshold τ, defaulting to 0.5.The selected prompts then receive n current-policy generations for standard policy-gradient updates.
  • 4 PCL: Prompt Curriculum Learning: The value model is updated online by minimizing prediction error against empirical average rewards from the selected prompts’ generated responses.This uses existing training responses and requires no additional rollouts.
  • 4 PCL: Prompt Curriculum Learning: Table 1 reports MATH and DeepScaleR performance, marking the best and second-best methods and measuring time to the checkpoint with the best average performance.Reported time includes training and generation but excludes validation and checkpointing.
  • 4 PCL: Prompt Curriculum Learning: The value model incurs negligible reported cost because it processes only prompts, with training and inference completed in under 30 seconds per step.The value model is one policy step behind, which the authors consider acceptable because successive policy updates are small.

5 Experiments

PCL combines batch and prompt-selection findings into an on-policy value-model curriculum, achieving strong performance or faster convergence while maintaining intermediate-difficulty training prompts.

  • 5 Experiments: PCL uses a value model to select prompts closest to a target difficulty threshold, enabling on-policy filtering without the rollout overhead of dynamic-sampling methods.The evaluation uses same-sized policy and value models, with τ = 0.5 and k = 4 unless otherwise noted.
  • 5.1 Convergence Comparison: PCL achieves the highest performance across all four MATH models and ranks first or second across six DeepScaleR benchmarks, while often reducing time to comparable performance.DS requires substantially more generation time, while SPEED and GRESO suffer from off-policy estimates; several SPEED runs crashed during training.
  • 5 Experiments: PCL maintains either a higher effective ratio or lower generation time, while DS and SPEED reach an effective ratio of 1 only with 105% and 81.8% higher generation time.On DeepScaleR with Qwen3-8B-Base, PCL’s slightly higher generation time than GRPO and Pre-filter reflects its selection of harder prompts when policy accuracy exceeds 0.5.
  • 5.2 Analysis & Ablation: Pre-filter increasingly trains on easy prompts because prompts previously excluded as difficult are never revisited after becoming intermediate for the improving policy.PCL keeps filtered training reward near 0.5, whereas other methods shift toward easier prompts.
  • 5.2 Analysis & Ablation: The value model reaches explained variance comparable to approximately three rollouts per prompt, while identifying difficulty 12.1× faster on MATH and 16.9× faster on DeepScaleR.With 2,048 candidate prompts, three rollouts require 288 seconds on MATH and 396 seconds on DeepScaleR per step, compared with 23.9 and 23.5 seconds for value-model training and inference.
  • 5.2 Analysis & Ablation: Filtering at τ = 0.5 gives the highest value-model prediction accuracy and remains comparable to no filtering, whereas extreme thresholds degrade accuracy.The paper attributes this pattern to retaining diverse reward outcomes and avoiding the label imbalance produced by selecting only very easy or very hard prompts.
  • 5.2 Analysis & Ablation: PCL progressively selects harder prompts during training despite a fixed τ = 0.5, because previously hard prompts become intermediate as the policy improves.Reference-policy rewards for selected prompts decline over time for PCL and other filtering methods, but remain nearly constant for GRPO and Pre-filter.

6 Related Work

Prior efficient RL work improves objectives, generation costs, or data selection, while PCL combines rollout avoidance with on-policy prompt selection. It uses curriculum learning to target intermediate-difficulty prompts for the current policy.

  • Efficient RL methods reduce model or generation costs through new objectives, while another line improves sample efficiency through data selection.
  • Generation-based filtering methods estimate effective ratios through rollouts, whereas dictionary-based GRESO records historical average rewards and can become off-policy on large datasets.
  • PCL combines direct prompt filtering without costly rollouts with on-policy selection, avoiding the off-policyness of dictionary-based methods.
  • PCL applies curriculum learning by continually focusing on prompts of intermediate difficulty for the current policy.

7 Discussions & Conclusion

The paper identifies intermediate-difficulty prompts and an optimal batch-size regime as key factors for efficient RL post-training, then develops PCL around these findings. It reports broad efficiency benefits while noting assumptions about prompt-level generalization and extensions beyond binary rewards.

  • PCL targets an optimal batch size and prompts with p(x) ≈0.5, which provide high gradient signal and sample efficiency while avoiding wasted rollouts or off-policy selection.
  • PCL either achieves the highest performance or reaches comparable performance in significantly less training time than its counterparts.
  • The value model extends naturally to non-binary scalar rewards by changing its range and the target threshold τ.
  • PCL assumes that training on intermediate-difficulty prompts improves performance on filtered-out prompts, an assumption that may fail outside domains with shared structural similarities.

8 Limitations

The paper’s limitations include restricted training settings, context lengths, and horizons, alongside a purely on-policy setup. Its appendix specifies the objective, datasets, models, rewards, and implementation details used for the study.

  • 8 Limitations: The study is purely on-policy, which simplifies analysis but may reduce generalization to pipelines using off-policy data or replay buffers.
  • 8 Limitations: Experiments use synchronous training, so asynchronous extensions may require value-model and prompt-selection strategies that handle stale policies.
  • 8 Limitations: Experiments are limited to 4,096-token contexts, although longer contexts shift the batch-size transition toward larger batches.
  • 8 Limitations: Training runs last roughly 2–3 days, leaving long-term convergence behavior, especially for larger models and datasets, unresolved.
  • A Problem Setup Details: The setup optimizes expected binary correctness reward with a purely on-policy GRPO variant and token-level policy-gradient objectives.
  • B.2 Model Details: Experiments use Qwen3-1.7B, 4B, and 8B base models plus Llama3.2-3B-it, trained with full parameters on eight A100 GPUs.
  • B.3 Reward Details: Rewards assign +1 to correct answers and 0 to incorrect or over-context generations, without format-based reward terms.

B.4 Evaluation Details

Evaluation covers mathematical reasoning benchmarks across multiple datasets and models, with generation settings and learning rates specified for reproducibility. The appendix also lists the complete experiment configurations for Math and DeepScaleR.

  • B.4 Evaluation Details: Evaluation spans MATH500, Minerva Math, OlympiadBench, AMC 2023, AIME 2024, and AIME 2025, covering standard and competition-level mathematical reasoning.
  • B.4 Evaluation Details: Smaller-scale evaluations report Avg@4 for Minerva Math and Avg@32 for AMC 2023, AIME 2024, and AIME 2025.
  • B.4 Evaluation Details: MATH and DeepScaleR use temperature 0.6 and top_p = 0.95, with top_k = 20 for MATH and disabled for DeepScaleR.
  • B.5 Complete List of Experiments: Learning rates are tuned logarithmically by batch size on Qwen3-8B-Base and transferred to the other models.
  • B.5 Complete List of Experiments: Tables 4–6 provide the complete experiment lists for Math and DeepScaleR.

C.1.1 Results with varying m

The experiments examine how batch decomposition, implementation settings, and prompt difficulty affect RL training efficiency and gradient magnitude. Across these analyses, intermediate-difficulty prompts are motivated as especially informative because they yield larger expected advantages and gradient updates.

  • C.1.1 Results with varying m: Figures 10–15 compare training reward, wall-clock time, generation time per step, and test accuracy across batch configurations, varying model, dataset, n, context length, rollout workers, and inference engine.
  • C.1.2 Results with varying m and n: Figures 10–12 vary batch size through prompts m and generations per prompt n, tracking generation time per step, training reward over steps and wall-clock time, and test accuracy on MATH and DeepScaleR.Figures 10–12 use logarithmic axes; Figures 10 and 11 use n = 16, while Figure 12 uses n = 32 and 64 for Qwen3-4B.
  • C.1.3 Results with a different context length: The appendix tests whether the batch-size and training trends persist under 8192-token context length, one rollout worker with eight GPUs, and SGLang instead of VLLM.These settings are compared against the default 4096-token, eight-worker, VLLM configuration.
  • C.2 Complete Results for Section 3.2: Figures 16 and 17 report reward before and after downsampling together with effective ratio, gradient norm, and test accuracy across p(x) thresholds and batch decompositions.The results are grouped both by p(x) and by the numbers of prompts and generations per prompt.
  • D Connection between p(x) and Gradient Magnitude: Intermediate-difficulty prompts yield the largest expected advantage magnitude and gradient-update upper bound, motivating a curriculum that prioritizes pπθ(x) = 1/2.The derivation connects prompt difficulty to gradient magnitude and training efficiency.

E Experiment Details

The experiments specify baseline algorithms, dataset and evaluation settings, and tuned hyperparameters for comparing GRPO, filtering methods, and PCL. PCL uses a value model with tuned critic learning rate and sampling parameter, while the main experiments match its value-model size to the policy.

  • E.1 Baselines Algorithms: The baseline procedures include GRPO, pre-filtering, dynamic sampling, and SPEED, each constructing prompt batches before updating the policy with the GRPO objective.The algorithms differ in whether prompts are filtered before training, dynamically sampled, or buffered using additional generations.
  • Algorithm 5 SPEED: SPEED screens candidate prompts with initial generations, adds accepted samples to a buffer, and updates the policy from buffered batches.Its procedure generates ninit responses for sampled prompts and uses accepted data to form later training batches.
  • E.2 Dataset, Model, Reward, Evaluation Details: The preliminary investigation adopts the same dataset, model, reward, and evaluation setting used for the main experiments.Additional details are provided in Appendix B.
  • E.3 Hyperparameters: The value-model learning rate is set to 1e-6 because larger rates are more unstable, while k = 4 is selected because larger k values provide only marginal effective-ratio gains beyond 4.The main PCL results use a value model with the same size as the policy, and its accuracy is only mildly affected by k.
  • E.3 Hyperparameters: PCL uses m = 512 and n = 16, τ = 0.5, and a critic learning-rate search over {1e-6, 3e-6, 1e-5} on both MATH and DeepScaleR.The policy learning rates are 8e-6 on MATH and 4e-6 on DeepScaleR.

F Value Model Size Ablation

The value-model ablation evaluates explained-variance prediction across model-size combinations for the policy and value model. Larger value models converge faster, with their early advantage especially pronounced on the larger DeepScaleR dataset.

  • F Value Model Size Ablation: Larger value models converge faster, and on DeepScaleR the smaller model substantially underperforms larger counterparts after 100 training steps.On MATH, all three value models eventually reach similar performance levels.
  • F Value Model Size Ablation: The authors hypothesize that smaller value models may need more training steps to reach comparable accuracy, highlighting larger models' early-training benefit on large-scale data.The ablation measures value prediction using explained variance, with average reward over 16 generations as ground-truth p(x).
Loading 2510.01135v1…