Pass@k Training for Adaptively Balancing Exploration and Exploitation of Large Reasoning Models

1 Introduction

• Pass@1 outcome rewards can penalize useful unsuccessful explorations, pushing policies toward exploitation and potentially a local optimum.Under outcome supervision, an erroneous solution with a correct answer can receive positive reward, while a correct solution with a wrong answer receives negative reward.Source: §1 · p. 2
• Because Pass@k rewards success within k attempts, it can favor diverse candidate solutions over similar samples and thereby encourage broader exploration.The introduction argues that stronger exploration can give models more comprehensive knowledge and robustness.Source: §1 · p. 2
• The authors apply Pass@k rewards to models already trained with Pass@1, then use bootstrap sampling and analytical derivation to make training more efficient and reduce sampling variance.They also report that Pass@k Training naturally prevents decreases in policy-distribution entropy, a metric associated with exploration.Source: §1 · pp. 2–3, Figure 2 · p. 3
• Pass@k Training enhances exploration and Pass@k performance without harming Pass@1 scores, is robust across k, domains, and tasks, and strengthens exploitation through continual training.The authors report that a 7B model surpasses GPT-4o and Claude-3.7 after continual training.Source: §1 · pp. 2–3
• Analytical Pass@k Training directly designs the advantage function, motivating implicit reward design as a way to control optimization without complex theoretical derivations.The proposed directions include focusing on harder problems and improving training efficiency.Source: §1 · p. 3

2.1 Formulation of Reasoning Tasks and Pass@1 Training

• Reasoning tasks use verifier rewards for response correctness, while Pass@1 Training estimates group-relative advantages; experiments use a DAPO variant retaining clip-higher and token-level policy-gradient loss.The verifier reward is binary in the experiments, with Rneg = 0 and Rpos = 1.Source: §2.1 · p. 4

2.2 Pass@k Training

• Pass@k Training continually improves downstream Pass@k while Pass@1 Training stagnates later, and it preserves or can improve Pass@1 performance.The reported trend is based on full sampling; additional training steps or rollouts consistently bring further Pass@k improvement.Source: Figure 3 · p. 5, §2.2 · p. 5

2.3 Efficient Pass@k Training via Bootstrap Sampling

• With equal rollouts bootstrap sampling outperforms full sampling; with equal groups it retains comparable Pass@k using one-fourth the theoretical computational cost and comparable Pass@1.The proposed procedure forms groups by resampling k distinct responses from the rollout set and sums the advantages of groups containing each response.Source: Figure 4 · p. 6, §2.3 · p. 6

2.4 Analytical Derivation of Efficient and Effective Pass@k Training

• Both Pass@k methods outperform Pass@1 Training, while analytical derivation reduces bootstrap fluctuations and supports continued Pass@k improvement as training steps increase.The comparison fixes rollouts at 32; the bootstrap method has a Pass@k decline at 400 steps, whereas analytical derivation removes group-construction sampling.Source: Figure 5 · p. 8, §2.4 · pp. 8–9
• Analytical derivation computes each response’s advantage directly from rollout count, positive and negative response counts, and k, avoiding group construction and its sampling variance.The method derives positive and negative response-relative advantages from the group rewards and applies them directly during advantage estimation.Source: §2.4 · pp. 6, 8, 9

3.1 How does Pass@k Training Compare to Noise Rewards or Entropy Regularization?

• Pass@k Training outperforms Noise Rewards and Entropy Regularization: flipped negative-response rewards progressively degrade Pass@1 and Pass@k, while entropy regularization can collapse or destabilize training.Higher entropy-regularization coefficients risk collapse; smaller coefficients still do not outperform Pass@k Training and may reduce performance as training continues.Source: §3.1 · pp. 9–10

3.2 Does Pass@k Training Really Improve the Exploration Ability of LLMs?

• Pass@k Training promotes diverse answers when the model is uncertain, maintains higher policy entropy than Pass@1 Training, and can improve exploration and exploitation together.Pass@1 Training instead converges to low entropy, while Pass@k Training’s entropy rises from step 200.Source: §3.2 · p. 11

3.3 What is the Generalization Ability of LLMs After Pass@k Training?

• Pass@k Training outperforms Pass@1 Training on in-domain and out-of-domain generalization, with both methods improving capacity over the base model.The passage attributes Pass@k Training’s stronger out-of-domain performance to better exploration, whereas Pass@1 Training encourages conservative behavior.Source: Table 1 · p. 11, §3.3 · p. 11

3.4 How does the Value of k Affect Pass@k Training?

• Pass@k Training remains effective across k values 4, 8, and 16, though larger k slows improvement; increasing the learning rate can mitigate this efficiency loss.For k=8 with N=32, a learning rate of 4 × 10−6 makes Pass@8 training’s efficiency exceed Pass@4 training’s.Source: §3.4 · pp. 11–12, Figure 8 · p. 12

3.5 Can the Benefits from Pass@k Training Be Transferred to Pass@1 Performance?

• Pass@k Training’s benefits transfer to Pass@1 across model scales, dense and MoE architectures, model families, and natural-language and multimodal tasks.Continued Pass@1 Training after Pass@k Training can improve reasoning performance; the reported examples include Qwen2.5 7B and 32B models and Seed1.5-VL-Small.Source: Table 2 · p. 13, Table 3 · p. 13, §3.5 · p. 13

4.1.1 Analysis Based on Advantage Value Curves

• Pass@k’s sum of absolute advantages peaks on harder problems and declines for easy ones, whereas Pass@1’s curve is symmetric around its peak.Pass@1’s maximum sum is higher, but the paper says that maximum is not the critical factor; its peak occurs at 50% accuracy, compared with 25% for Pass@8, suggesting Pass@k emphasizes harder, previously unsolved problems rather than medium-difficulty or mastered ones.Source: §4.1.1 · pp. 14–15

4.1.2 Analysis Based on Model Performance

• Reducing optimization on easy questions helps prevent local optima, while shifting the advantage curve’s peak toward harder questions improves training efficiency.Setting easy-question rewards to zero only delays a performance plateau; these conclusions are preliminary and require validation across tasks and scenarios.Source: §4.1.2 · pp. 15–16

4.2.1 Exceeding Pass@k Training

• Exceeding Pass@k Training shifts the advantage curve’s peak to 1/32 correctness and improves early Pass@k performance, but downstream Pass@1 gains progress more slowly.The method applies a transformation to the advantage function; its excessive emphasis on difficult problems motivates adapting advantage computation to the model’s current state.Source: Figure 11 · p. 16, §4.2.1 · pp. 16–17

4.2 RLVR Training Through Implicit Reward Design

• Implicit reward design directly shapes advantage functions from optimization objectives, offering a way to control RLVR optimization when analytical reward derivations are difficult.The section presents this as a preliminary exploration of potential directions for future research.Source: §4.2 · p. 16, §4.2.3 · p. 18

4.2.2 Combination of Pass@1 and Pass@k Training

• Combination Training consistently outperforms standard Pass@k Training on both Pass@1 and Pass@8 for Qwen models on Enigmata.It gives Pass@1 advantages greater weight at low correctness and Pass@k advantages greater weight at high correctness, balancing training efficiency with reduced overfitting to mastered problems.Source: Figure 12 · p. 17, §4.2.2 · p. 17

4.2.3 Adaptive Training based on Policy Entropy

• Adaptive Training improves both Pass@1 and Pass@K performance, outperforming Pass@1 Training and Pass@K Training.It uses policy entropy to classify problems: the top half by average entropy receive Pass@1 advantages to exploit prior exploration, while the rest receive Pass@K advantages to encourage exploration.Source: Figure 13 · p. 18, §4.2.3 · p. 18

5.1 Reinforcement Learning with Verifiable Rewards

• RLVR has enabled complex reasoning and other capabilities, yet PPO and GRPO face instability, model collapse, and reward noise; prior Pass@k training [45] had not established its connection to exploration.This paper applies Pass@k in GRPO and variants through three approaches and derives analytical advantage values to examine exploration and exploitation during training.Source: §5.1 · pp. 18–19

5.2 Effective Exploration in Test-time Scaling

• Test-time scaling depends on exploration, but base-model limits can constrain it; prior approaches adjust sampling, use self- or external verification, and modify training to preserve exploration.Training approaches include focusing on critical tokens [47], adding regularization [28], selecting useful sampled experience [37] [59], and incorporating entropy into advantage estimation [12].Source: §5.2 · p. 19

6 Conclusion

• Pass@k Training uses Pass@k rewards to enhance diverse-output exploration, which in turn improves exploitation and aims to raise overall model performance.The authors frame exploration and exploitation as capabilities that can mutually improve through this training approach.Source: §6 · p. 19
• The method adds bootstrap sampling and an analytically derived advantage function to make Pass@k Training more efficient and effective.The paper also poses five research questions to investigate why the method works and what benefits it brings.Source: §6 · p. 19
• Advantage-value curves suggest that the argmax and the trend of the sum of absolute advantage η contribute to Pass@k Training’s success.The authors describe this analysis as an implicit reward-design perspective.Source: §6 · p. 19
• Customized advantage functions showed preliminary effectiveness, which the authors identify as a promising direction for future RLVR research.This exploration was inspired by Pass@k Training’s effectiveness.Source: §6 · p. 19

A.1 Details of Downstream Tasks

• The Maze task represents mazes as text and asks models to reason before selecting an up, down, left, or right action.Training uses 9 × 9, 11 × 11, 13 × 13, and 15 × 15 mazes; testing includes those sizes plus 7 × 7, 17 × 17, 19 × 19, and 21 × 21 mazes, with strict deduplication.Source: §A.1 · p. 25
• Downstream evaluations also include Enigmata’s synthetic puzzles, image-based MathVision math problems, and MMMU’s college-level multimodal reasoning tasks.Enigmata spans 36 puzzle categories across seven primary categories; MathVision contains 3,040 competition problems, while MMMU covers six academic subjects and varied image types.Source: §A.1 · p. 25

Training

• Qwen2.5-7B-Instruct and Qwen2.5-32B-Instruct are trained through DAPO using clip-higher and token-level policy-gradient loss.The setup uses ε_low = 0.2 and ε_high = 0.28, and removes other optimizations.Source: §A.2 · p. 25
• Training assigns reward 1 to verified responses and 0 otherwise, without KL or entropy regularization.The policy learning rate is 1 × 10−6 with 10 warmup steps; prompt batch size is 128, mini-batch size is 32, and rollout times are 32.Source: §A.2 · pp. 25–26

Evaluation

• Evaluation uses Temperature 1.0 and Top_P 0.95, sampling 32 Maze-task responses or 8 responses for other tasks to compute Pass@1 and Pass@k.Source: §A.2 · p. 26

B Details of Analytical Derivation

• The derivation covers the average and standard deviation of group reward and the response-relative advantage.It provides details of the procedure mentioned in Section 2.4.Source: B Details of Analytical Derivation · p. 26

B.1 Derivation of the Average of Group Reward

• Figure 15 presents an example of the Enigmata task.Source: Figure 15 · p. 27

B.2 Derivation of the Standard Deviation of Group Reward

• Figure 16 presents an example of the MathVision task.Source: Figure 16 · p. 28
• Figure 17 presents examples of MMMU tasks.Source: Figure 17 · p. 28

C Pseudo Code for Pass@k Training

• Full sampling partitions responses into k-sized groups, computes each group’s Pass@k reward, and discards redundant instances.The procedure then computes group-relative advantages from the groups’ mean and standard deviation and assigns each group’s advantage to its responses.Source: C Pseudo Code for Pass@k Training · p. 29
• Bootstrap sampling randomly draws k responses per group, computes group rewards, and derives response-relative advantages using Eq. 6.The pseudocode obtains group-relative advantages using the groups’ average reward and standard deviation.Source: C Pseudo Code for Pass@k Training · p. 29

D Curves of Advantage Function

• Figure 18 presents advantage-function curves for Pass@k Training without easy problems, Pass@k Training with combination, exceeding Pass@k Training, and combination training.The supplied passages identify the approaches shown but do not report curve values or comparisons.Source: D Curves of Advantage Function · p. 29

E Experiments on Various LLMs and Tasks

• The experiments are framed as evaluating Pass@k Training across AIME 2024, AIME 2025, OlymMATH, and Enigmata.The passage identifies the first three as mathematical tasks and Enigmata as a synthetic puzzle task, but gives no performance results.Source: E Experiments on Various LLMs and Tasks · p. 29
• Analytical-derivation pseudocode computes group-reward averages and standard deviations, then derives separate advantages for positive and negative responses before assigning advantages to instances.The steps use Eqs. 11, 12, 14, and 15, respectively.Source: E Experiments on Various LLMs and Tasks · p. 30

E.1 Pass@k Training on Mathematical Tasks

• Experiments trained LLaMA3.2-3B-Instruct, LLaMA3.1-8B-Instruct, and DeepSeek-R1-Distill-Qwen 1.5B and 7B models on mathematical tasks using prior work’s training data [9].The models are identified as LLaMA and DeepSeek-R1-Distill-Qwen [13]. The experiment follows Appendix A.2 settings.Source: §E.1 · p. 30
• LLaMA runs used maximum prompt and response lengths of 2048 and 6144, while DeepSeek-R1-Distill-Qwen used a response length of 10240.Evaluation followed Appendix A.2 settings.Source: §E.1 · p. 30
• Table 5 reports Pass@1 and Pass@k performance and includes two repetitions of Pass@k Training followed by Pass@1 Training.The table labels this repeated process “(P@k T. + P@1 T.) × 2”; P@k T. denotes analytically derived Pass@k Training.Source: Table 5 · p. 30, §E.1 · p. 30

E.2 Pass@k Training on Enigmata Task

• Experiments train LLaMA3.2-3B-Instruct and LLaMA3.1-8B-Instruct with maximum prompt and response lengths of 4096 tokens, following Appendix A.2 settings for training and evaluation.Source: §E.2 · p. 31
• Table 6 reports Enigmata Pass@1/Pass@k performance for LLaMA models trained with different RLVR approaches, including Pass@1 Training and analytically derived Pass@k Training.“P@1 T.” and “P@k T.” denote Pass@1 Training and Pass@k Training with analytical derivation, respectively.Source: Table 6 · p. 31

• Without RLVR training, solving the Maze task is very difficult, so the backbone model’s performance is not reported.Source: §E.3 · p. 31
• Table 7 covers Qwen2.5-7b-Instruct Pass@1 and Pass@k performance across Maze sizes for Pass@1 Training and analytically derived Pass@k Training.Its caption defines FS as full sampling, BS as bootstrap sampling, and AD as analytical derivation.Source: Table 7 · p. 31
• Figure 18 plots advantage-function curves for N_rollout = 32.Source: Figure 18 · p. 32