Pass@k Training for Adaptively Balancing Exploration and Exploitation of Large Reasoning Models

Zhipeng Chen; Xiaobo Qin; Youbin Wu; Yue Ling; Qinghao Ye; Wayne Xin Zhao; Guang Shi

Source-linked AI summary

Pass@k Training for Adaptively Balancing Exploration and Exploitation of Large Reasoning Models

Zhipeng Chen, Xiaobo Qin, Youbin Wu, Yue Ling, Qinghao Ye, Wayne Xin Zhao, Guang Shi

arXiv:2508.10751v1Aug 14, 2025cs.LGcs.AIcs.CL

TL;DR

RLVR commonly uses Pass@1, leaving the relationship between reward design and LLM exploration underexplored. The paper introduces Pass@k Training and analytical advantage derivation, reporting improved exploration and Pass@k performance without harming Pass@1, while motivating implicit reward design for RLVR.

Problem
The connection between Pass@k and LLM exploration ability in RLVR remains largely overlooked, although Pass@1 Training can impair exploration and create an exploration–exploitation imbalance.
Method
The paper trains a Pass@1-trained model with Pass@k rewards, then improves the procedure using bootstrap sampling and an analytical derivation of the advantage function.
Results
Pass@k Training enhances exploration, improves Pass@k performance, maintains Pass@1 scores, and is reported as robust across k values, domains, and tasks.
Takeaways & Limitations
Exploration and exploitation can mutually enhance each other, while directly designing advantage functions offers a promising direction for future RLVR development.
Takeaways & Limitations
The conclusions about advantage factors are preliminary and require more comprehensive experiments tailored to specific tasks and scenarios.

Abstract

from arXiv · show

Reinforcement learning with verifiable rewards (RLVR), which typically adopts Pass@1 as the reward, has faced the issues in balancing exploration and exploitation, causing policies to prefer conservative actions, converging to a local optimum. Identifying an appropriate reward metric is therefore crucial. Regarding the prior work, although Pass@k has been used in evaluation, its connection to LLM exploration ability in RLVR remains largely overlooked. To investigate this, we first use Pass@k as the reward to train the policy model (i.e., $\textbf{Pass@k Training}$), and observe the improvement on its exploration ability. Next, we derive an analytical solution for the advantage of Pass@k Training, leading to an efficient and effective process. Building on this, our analysis reveals that exploration and exploitation are not inherently conflicting objectives, while they can mutually enhance each other. Moreover, Pass@k Training with analytical derivation essentially involves directly designing the advantage function. Inspired by this, we preliminarily explore the advantage design for RLVR, showing promising results and highlighting a potential future direction.

1 Introduction

The paper proposes Pass@k Training to address impaired exploration under Pass@1 Training, then develops more efficient variants and analyzes broader implications for RLVR advantage design.

Motivation: Pass@1 Training can penalize unsuccessful explorations containing correct ideas, creating an imbalance between exploration and exploitation.Outcome-level supervision may reward erroneous solutions with correct answers while penalizing correct solutions with wrong answers.
Pass@k Training: Pass@k Training uses the Pass@k metric as the reward to train a model that has already undergone Pass@1 Training.The approach is intended to tolerate multiple incorrect responses while encouraging diverse candidate solutions.
Findings: Pass@k Training improves Pass@k performance without harming Pass@1 scores and significantly enhances LLM exploration ability.The paper reports this comparison against Pass@1 Training.
Training Procedure: Full sampling, bootstrap sampling, and analytical derivation form three progressive Pass@k Training enhancements.Bootstrap sampling improves training efficiency, while analytical derivation mitigates sampling variance as its theoretical asymptotic form.
Findings: Pass@k Training is robust to different k values and generalizable across domains and tasks, while enhanced exploration can improve exploitation through continual training.The authors report that this practical value enabled a 7B LLM to surpass GPT-4o and Claude-3.7.
Future Direction: Analytical Pass@k Training directly designs the advantage function and motivates implicit reward design as a future RLVR direction.The paper reports that implicit reward design may provide finer-grained optimization control, including emphasis on harder problems or training efficiency.

2 Pass@k as Reward in RLVR Training

This section introduces Pass@k as an RLVR reward to improve exploration while preserving Pass@1 performance, then develops bootstrap sampling and analytical derivation for more efficient and stable training.

2.2 Pass@k Training: Pass@k is defined as the expected maximum verifier reward among k sampled responses and is incorporated as an RLVR reward.The verifier scores each response, and the group receives the maximum reward among its k responses.
2.2 Pass@k Training: Pass@k Training continually improves downstream Pass@k performance without compromising Pass@1 performance, unlike Pass@1 Training’s later stagnation.More training steps or rollouts consistently improve Pass@k, while Pass@1 performance is maintained or gains.
2.3 Efficient Pass@k Training via Bootstrap Sampling: Bootstrap sampling reuses N_rollout responses to construct multiple k-response groups, assigning each response the summed advantages of groups containing it.Responses may appear in multiple groups, reducing the need for additional rollouts while preserving group-based advantage estimation.
2.3 Efficient Pass@k Training via Bootstrap Sampling: With the same number of rollouts, bootstrap sampling outperforms full sampling; with the same number of groups, it uses one-fourth of the theoretical computational cost with comparable performance.The reported efficiency is attributed to more groups and lower variance in the sampled reward distribution.
2.4 Analytical Derivation of Efficient and Effective Pass@k Training: Analytical derivation computes response-relative advantages directly from N_rollout, N_pos, N_neg, and k, eliminating bootstrap-sampling variance and full-sampling overhead.The derived advantages are applied after rollout to optimize model parameters and support continued exploration improvement as training scales.

3 Balancing Exploration and Exploitation with Pass@k Training

Pass@k Training is evaluated as an approach for balancing exploration and exploitation in RLVR. Across comparisons, it preserves or improves Pass@1 performance while strengthening exploration, generalization, and robustness.

3.1 How does Pass@k Training Compare to Noise Rewards or Entropy Regularization?: Pass@k Training outperforms Noise Rewards and Entropy Regularization in sustaining improvement during training.Randomly flipping negative-response rewards can degrade performance, while entropy regularization introduces trade-offs and may fail to achieve continuous improvement.
3.2 Does Pass@k Training Really Improve the Exploration Ability of LLMs?: Pass@k Training maintains greater answer diversity and higher policy entropy than Pass@1 Training, indicating stronger exploration.Pass@1 Training converges toward low entropy and similar answers, whereas Pass@k Training keeps entropy relatively high and can increase it during later training.
3.2 Does Pass@k Training Really Improve the Exploration Ability of LLMs?: Exploration and exploitation can mutually improve under Pass@k Training rather than functioning as inherently conflicting objectives.The reported results connect continued exploration with improved performance while retaining exploitation through correct responses.
3.4 How does the Value of k Affect Pass@k Training?: Pass@k Training is robust to different k values, although increasing k reduces optimization efficiency; enlarging the learning rate can address this decline.The reported process remains stable and effective across k choices despite shorter optimization steps at larger k.
3.5 Can the Benefits from Pass@k Training Be Transferred to Pass@1 Performance?: Continual Pass@1 Training after Pass@k Training improves Pass@1 performance across model scales, architectures, families, and downstream task forms.The reported settings include 7B and 32B models, dense and MoE architectures, Qwen and Seed families, and natural-language and multimodal tasks.

4 Generalizing Pass@k Training via Implicit Reward Design

The section interprets Pass@k Training through advantage-function design, identifying how its magnitude, peak location, and trend shape optimization. Experiments suggest that reducing emphasis on easy problems and adapting advantages can improve efficiency and both Pass@1 and Pass@k performance.

Advantage analysis: Pass@k Training’s sum of absolute advantage peaks at 25% accuracy, whereas Pass@1 Training peaks at 50%, indicating stronger emphasis on harder problems.This focus guides optimization toward previously unsolved or difficult problems rather than already mastered ones.
Advantage analysis: Pass@k Training gradually reduces optimization strength as correctness exceeds 60%, while Pass@1 Training does not show the same trend.The declining strength on easy problems is associated with continued training progress and avoiding local optima.
Model-performance analysis: Removing optimization for easy questions can improve rewards and test performance initially, but performance eventually plateaus, so zeroing their rewards is insufficient.The comparison suggests that gradual reduction is more effective than simply eliminating easy-problem optimization.
Model-performance analysis: Shifting the η argmax toward harder problems enables faster escape from local optima and better performance with fewer training steps.The experiments attribute the efficiency gain to assigning greater optimization strength to harder problems.
Adaptive advantage design: The authors conclude that implicit reward design can control optimization without complex derivations, including harder-problem emphasis and simultaneous exploration–exploitation improvement.They characterize the initial advantage-design findings as promising but preliminary.

5 Related Work

Related work positions RLVR as a successful framework for reasoning, while noting persistent training challenges and the importance of exploration for test-time scaling. Existing methods address instability and related issues, but exploration remains constrained by the base model.

RLVR: RLVR trains LLMs with verifiable outcome rewards and has improved reasoning capabilities on complex tasks.The framework supports learning from generated responses under outcome-level supervision.
RLVR: Popular RLVR algorithms such as PPO and GRPO still face training instability, model collapse, and reward noise.Prior work proposes optimization changes to mitigate these challenges.
Test-time scaling: Effective exploration is necessary for test-time scaling, but existing work reports that exploration ability is limited by the corresponding base model.This limitation can hinder continued performance scaling at inference time.

6 Conclusion

The conclusion presents Pass@k Training as an RLVR method intended to improve exploration and exploitation together, then extends the idea to implicit reward design. The paper reports promising results while describing customized advantage design as preliminary future work.

Conclusion: Pass@k Training uses Pass@k as the RLVR reward to enhance diverse-output exploration while maintaining or improving exploitation capability.The method is intended to enable mutual improvement between exploration and exploitation.
Conclusion: Bootstrap sampling and analytical advantage derivation make Pass@k Training more efficient and effective.The conclusion frames these mechanisms as optimizations of the training procedure.
Conclusion: Implicit reward design analyzes advantage curves and uses the η argmax and trend as factors for designing customized advantage functions.The authors report remarkable effectiveness but characterize this exploration as relatively preliminary.
Evaluation tasks: The evaluation covers Maze, Enigmata, MathVision, and MMMU, with Maze represented as text-based n × n grids and the other benchmarks spanning puzzles, multimodal mathematics, and academic reasoning.The supplied passages describe the benchmark settings but do not provide their numerical results here.

A.2 Implementation Details

The implementation trains Qwen2.5 instruction models with DAPO using a streamlined token-level policy-gradient setup. Evaluation samples multiple responses per question to compute Pass@1 and Pass@k scores across task settings.

Training: Experiments use Qwen2.5-7B-Instruct and Qwen2.5-32B-Instruct as backbone models, trained through DAPO.The setup retains clip-higher and token-level policy-gradient loss while removing other optimizations.
Training: The learning rate is 1 × 10^-6 with 10 warmup steps, using prompt batch size 128 and mini-batch size 32.The passage also reports rollout times of 32, though the sentence is truncated after introducing that setting.
Maze task: Maze prompts ask models to solve text-described grids using the actions U, D, L, and R, with symbols defining starts, destinations, walkable areas, and walls.The example figure illustrates a Maze Task, while the supplied table passage provides only its caption.
Reward: Verified responses receive reward 1 and other responses reward 0, with no KL or entropy regularization.This defines the binary reward scheme used in training.
Evaluation: Evaluation uses temperature 1.0 and top-p 0.95, sampling 32 Maze responses and 8 responses for other tasks to compute Pass@1 and Pass@k.The sampling counts differ by task family.

B Details of Analytical Derivation

This section presents the analytical derivation procedure for Pass@k Training, covering group-reward statistics and response-relative advantages.

The derivation computes the average of the group reward.
It computes the standard deviation of the group reward.
It derives the response-relative advantage from the group-reward statistics.

B.1 Derivation of the Average of Group Reward

This section describes tasks in which a function must be inferred from preceding input-output examples and applied to a final input list.

The task requires applying a function to the final input list to generate an output list.
Preceding inputs and outputs serve as examples for identifying the function.
All example outputs are generated using the same function.
The response format requests a list of numbers enclosed in a code block.
The examples include Enigmata, MathVision, and MMMU tasks.
The task prompt asks which number should replace the question mark.

C Pseudo Code for Pass@k Training

The pseudo code implements Pass@k Training through full sampling, bootstrap sampling, and analytical derivation, producing estimated response advantages from rollout rewards.

Pass@k Training is specified with full sampling, bootstrap sampling, and analytical derivation procedures.
Full sampling separates the rollout rewards into groups of k instances and discards redundant instances.
Group rewards are computed before average reward, standard deviation, and group-relative advantage estimation.
Response-relative advantages are assigned from group-relative advantages or computed separately for positive and negative responses.
The procedures take rollout rewards, the rollout count, and k as inputs, and output estimated response advantages.
Bootstrap sampling constructs groups by randomly sampling k instances from the rollout rewards.
Figure 18 compares advantage-function curves for variants including removing easy problems, combining approaches, and combination training.
The experiments evaluate Pass@k Training on mathematical tasks and the synthetic Enigmata puzzle task.

E.1 Pass@k Training on Mathematical Tasks

This section evaluates Pass@k Training on mathematical tasks using LLaMA and DeepSeek-R1-Distill-Qwen models, with performance reported across different RLVR approaches.

The experiments train LLaMA3.2-3B-Instruct, LLaMA3.1-8B-Instruct, and 1.5B and 7B DeepSeek-R1-Distill-Qwen models.
Table 5 reports Pass@1 and Pass@k performance for models trained with different RLVR approaches.
Pass@k Training with analytical derivation is denoted “P@k T.”, while Pass@1 Training is denoted “P@1 T.”.
The repeated procedure “(P@k T. + P@1 T.) × 2” means alternating Pass@k Training followed by Pass@1 Training twice.

E.2 Pass@k Training on Enigmata Task

This section evaluates Pass@k Training with analytical derivation on LLaMA models using Enigmata Pass@1 and Pass@k performance.

The experiments use LLaMA3.2-3B-Instruct and LLaMA3.1-8B-Instruct under the stated Pass@k Training evaluation settings.The maximum prompt and response lengths are both 4096.
Evaluation follows the settings described in Appendix A.2.
Table 6 reports Enigmata Pass@1 and Pass@k performance for LLaMA models trained with different RLVR approaches.“P@1 T.” denotes Pass@1 Training, while “P@k T.” denotes Pass@k Training with analytical derivation.

E.3 Pass@k Training on Maze Task

This section presents Pass@k Training results for Qwen2.5-7B-Instruct across Maze sizes, with Pass@1 and Pass@k performance compared across approaches.

Table 7 reports Pass@1 and Pass@k performance across various Maze sizes.The backbone model is not reported because solving Maze is difficult without RLVR training.
The comparison includes Pass@1 Training and Pass@k Training with analytical derivation.
Table 7 distinguishes full sampling, bootstrap sampling, and analytical derivation as training approaches or variants.“FS”, “BS”, and “AD” denote full sampling, bootstrap sampling, and analytical derivation, respectively.
Figure 18 shows the curves of the advantage function when N_rollout = 32.