Source-linked AI summary
Prompt Interference in LLM Post-Training: Why Pass@k Optimization Can Degrade Pass@1
Anas Barakat, Souradip Chakraborty, Khushbu Pahwa, Amrit Singh Bedi
TL;DR
Pass@k optimization can improve multi-sample performance while degrading pass@1, a trade-off that matters when single-shot accuracy remains operationally important. The paper characterizes when prompt interference and pass@k’s reweighting produce conflicting population gradients, and concludes that this mechanism can explain pass@1 degradation under pass@k updates.
Problem
It is not well understood why or when pass@k optimization can hurt pass@1, despite the operational importance of single-shot performance.
Source: §1 · p. 2Method
The paper analyzes prompt interference and characterizes how pass@k’s implicit reweighting of low-success prompts can make population pass@k and pass@1 gradients conflict.
Source: §1 · pp. 2–3Results
Pass@k reweighting can amplify negatively interfering prompts, producing gradient conflict that permits pass@k improvement alongside pass@1 degradation.
Source: §1 · pp. 2–3Takeaways & Limitations
Prompt interference offers a lens for diagnosing post-training objectives that redistribute optimization effort across prompts and for considering interference-aware mitigation strategies.
Source: §6 · p. 10The one-step pass@1 degradation result assumes standard policy-parameter regularity, including bounded expected score and Hessian norms.
Source: §4.2 · p. 7
Abstract
from arXiv · showhide
Pass@$k$ is a widely used performance metric for verifiable large language model tasks, including mathematical reasoning, code generation, and short-answer reasoning. It defines success if any of $k$ independently sampled solutions passes a verifier. This multi-sample inference metric has motivated inference-aware fine-tuning methods that directly optimize pass@$k$. However, prior work reports a trade-off: pass@$k$ improves while pass@1 degrades under such methods. This trade-off is practically important because pass@1 often remains a hard operational constraint due to latency and cost budgets, imperfect verifier coverage, and the need for a reliable single-shot fallback. We study the origin of this trade-off and provide a theoretical characterization of when pass@$k$ policy optimization can reduce pass@1 through gradient conflict induced by prompt interference. This interference arises because shared model parameters couple prompts, so improving performance on some prompts can adversely affect others. We show that pass@$k$ policy gradients can conflict with pass@1 gradients because pass@$k$ optimization implicitly reweights prompts toward low-success prompts; when these prompts are negatively interfering, their upweighting can rotate the pass@$k$ update direction away from the pass@1 direction. We illustrate our findings with large language model experiments on verifiable mathematical reasoning and coding tasks.
1 Introduction
The paper asks when pass@k optimization can degrade pass@1 despite prior gains in multi-sample performance. It explains the trade-off through prompt interference and pass@k-specific prompt reweighting, then characterizes when the resulting population gradients conflict.
1 Introduction
- Pass@1 can fall as pass@k rises, a trade-off reported in prior work that matters when single-shot accuracy is operationally required [29] [32].Latency and cost budgets, incomplete verifier coverage, and the need for a single-shot fallback make pass@1 important in deployment.Source: §1 · p. 2
- Pass@k optimization reweights prompts toward those with lower success probability, which can amplify negatively interfering prompts and rotate its population gradient against pass@1.The paper distinguishes this population-level conflict from per-prompt alignment: per-prompt pass@k and pass@1 gradients are positively collinear.Source: §1 · pp. 2–3
- Prompt interference captures when improving one prompt tends to reduce another’s success, linking shared-parameter coupling to the pass@k–pass@1 trade-off.The paper relates this phenomenon to multi-task gradient conflict while emphasizing that the objectives are aligned for each fixed prompt [26] [34].Source: §1 · p. 3
- The authors give sufficient conditions for gradient conflict and show that increasing k encourages conflict under some conditions on relative success probabilities.They also prove simultaneous pass@1 decrease and pass@k increase for one-step updates satisfying an explicit stepsize condition, under a standard policy-parameter regularity assumption.Source: §1 · p. 3
- Large language model experiments illustrate the findings on mathematical reasoning and code generation tasks.Figure 1 schematically depicts how reweighting a negatively interfering prompt can make the population pass@k and pass@1 gradients conflict.Source: Figure 1 · p. 2, §1 · p. 3
2 Preliminaries
The section defines pass@k as success on at least one of k independent responses and gives its policy gradient. The gradient weights prompt-wise success gradients more heavily for prompts with lower success probability.
2 Preliminaries
- Pass@k is the probability that at least one of k independent responses is correct, expressed as Jk(θ) = Ex∼D[fk(pθ(x))] for fk(p) = 1 − (1 − p)^k.The objective is increasing in k and bounded above by min(1, fk(J1(θ))).Source: §2 · pp. 3–4
- The pass@k policy gradient averages prompt-wise success gradients weighted by wk(p) = k(1 − p)^(k−1).The resulting prompt weight is larger for low-success prompts and downweights high-success prompts.Source: §2 · p. 4
- For each prompt, pass@k and pass@1 gradients are positively collinear, but their expectations over prompts need not be.This difference motivates analyzing how prompt-wise gradients interact across the shared prompt distribution.Source: §2 · p. 4
3 Prompt Interference
Prompt interference describes how shared policy parameters can make an update that helps one prompt hurt another. The section formalizes this with similarities between prompt-wise pass@1 gradients.
3 Prompt Interference
- Shared policy parameters couple prompts, so an update can improve success on some prompts while degrading it on others.The paper calls this negative prompt interference.Source: §3 · p. 4
- The prompt gradient similarity kernel compares pass@1 gradients to indicate whether improving one prompt tends to help or hurt another.Its interpretation follows from the first-order change in the other prompt’s success probability under an update along the first prompt’s gradient.Source: §3 · pp. 4–5
- Two prompts interfere negatively when their gradient similarity is negative, so increasing success on one tends to decrease success on the other.The section’s toy example illustrates how negative interference, amplified by pass@k reweighting, can contribute to the pass@k–pass@1 trade-off.Source: §3 · p. 5
4 Pass@k and Pass@1 Gradients Conflict
The section explains how pass@k’s reweighting of prompts can make its population gradient conflict with pass@1 despite per-prompt alignment. It characterizes when this conflict arises and proves that it can produce a pass@1 decrease under pass@k updates.
4 Pass@k and Pass@1 Gradients Conflict
- Pass@k upweights low-success prompts, so negative interference among them can rotate its population gradient away from the pass@1 gradient.This prompt reweighting is the proposed population-level explanation for the pass@k–pass@1 trade-off.Source: §4 · p. 5
4.1 Gradient conflict: Reweighting meets interference
- Gradient conflict occurs exactly when average agreement under the pass@k-induced prompt distribution is negative.That distribution places higher mass on prompts with lower success probability; negative agreement means a prompt’s pass@1 gradient is anti-aligned with population pass@1 improvement.Source: §4.1 · pp. 5–6
4.2 Sufficient conditions and threshold in k
- Conflict occurs when weighted contributions from negatively interfering prompts dominate; under the stated success-probability separation, larger negative-region mass lowers the threshold k.The separation assumption treats negatively interfering prompts as harder on average, and the authors note it may not always hold.Source: §4.2 · p. 7
4.3 Pass@1 degradation under pass@k updates
- Under gradient conflict, pass@k updates can provably decrease pass@1 while increasing pass@k for sufficiently small steps.The analysis also gives diagnostics based on low-success prompt interference, the choice of k, and prompt-subset alignment.Source: §4.2 · p. 7, §4.3 · p. 8
- Inspect whether low-success prompts are aligned with population pass@1 improvement, because datasets with equal average pass@1 can behave differently.The practical guidance also cautions that larger k strengthens reweighting and can amplify conflict when these prompts interfere negatively.Source: §4.3 · p. 8
5 Experiments
Experiments show that pass@k reweighting can turn positive unweighted pass@1 gradient alignment negative, and an end-to-end pass@8 run improves pass@8 while reducing pass@1. The results are consistent with conflict arising when low-success, negatively interfering prompts receive greater weight.
Experiments
- The gradient-conflict test used filtered MATH prompts and two reasoning models, with further code-generation evidence reported for HumanEval and MBPP.The MATH experiments used DeepSeek-R1-Distill-Llama-8B and DeepSeek-R1-Distill-Qwen-7B; pass@1 gradients were computed at the final hidden layer.Source: §5 · p. 8
- In Figure 4, weighted agreement shifted from +2.80 × 10−3 to −1.12 × 10−3 for Llama-8B and from +2.97 × 10−2 to −2.74 × 10−1 for Qwen-7B.The corresponding inner products were −0.613 and −181, respectively.Source: Figure 4 · p. 9, §5 · p. 10
- Hard prompts had negative agreement scores while easy prompts had positive scores, and pass@k assigned hard prompts weights exceeding easy-prompt weights by more than 10^28:1.The reweighting shifted the population gradient from positive unweighted agreement to negative weighted agreement, consistent with the proposed conflict mechanism.Source: §5 · pp. 9–10
- A separate Qwen2.5-Math-1.5B-Instruct experiment evaluated pass@8 policy optimization on 500 held-out MATH problems across five independent training runs.The model was trained for 500 steps with binary correctness rewards, and gradient-interference diagnostics were recomputed at each checkpoint.Source: §5 · p. 10
- Pass@8 policy optimization increased pass@8 by +1.65 ± 0.49 percentage points and decreased pass@1 by 2.40 ± 0.49 percentage points relative to initialization.The population-gradient cosine became negative early and remained negative as pass@1 degraded; results are mean ±95% confidence interval across five runs.Source: Table 1 · p. 10
6 Conclusion and Future Work
The paper explains pass@1 degradation through prompt interference under pass@k reweighting. It presents prompt interference as a lens for diagnosing and addressing objective-induced trade-offs in post-training.
Conclusion and Future Work
- Pass@k reweighting can amplify negatively interfering prompts, producing conflict with pass@1 when shared model parameters couple prompts.Improving some prompts can adversely affect others.Source: §6 · p. 10
- Prompt interference offers a lens for post-training objectives that redistribute optimization effort across prompts, with practical diagnostics and interference-aware mitigation strategies suggested.Source: §6 · p. 10
- The paper suggests that future post-training methods could exploit prompt-distribution structure rather than treating prompts independently.Source: §6 · p. 10
A Related Work
Related work develops pass@k estimators and policy-gradient methods, and studies per-prompt gradient alignment and interference. This paper adds a characterization of when population-level gradients can conflict and pass@1 can degrade.
Related Work
- Prior work introduced pass@k evaluation [6] [17], unbiased estimation [6], and variance-reduced policy-gradient methods [29] [32].Mahdavi et al. also derived pass@k gradient expressions and proposed gradient reweighting with a tunable parameter.Source: A Related Work · p. 15
- Although Yu [35] found per-prompt pass@k and pass@1 gradients positively collinear, this paper shows their expected gradients over prompts need not be positively collinear.Source: A Related Work · p. 15
- Related approaches include a soft-maximum objective for trading off pass@k and pass@1 [15] and SimKO's redistribution of updates across top-K token candidates [23].Source: A Related Work · p. 15
- Qu et al. [25] study interference as an exploration and solvability bottleneck, while this paper provides a complementary theoretical account of pass@1 degradation under pass@k optimization.The paper formalizes prompt interference and gives conditions for gradient conflict [25].Source: A Related Work · pp. 15–16
- Qu et al. [25] propose POPE to mitigate ray interference, whereas this paper focuses on theoretical understanding of the pass@k versus pass@1 trade-off.The authors suggest their insights may clarify when such interventions are necessary and effective.Source: A Related Work · p. 16
B Limitations
The limitations qualify the theory’s local and assumption-dependent guarantees, the scope and sampling uncertainty of the experiments, and the need to test proposed mitigations. A toy contextual-bandit example illustrates how overlapping easy and hard prompts can negatively interfere.
B Limitations
- The gradient-conflict characterization is local to a fixed policy, so it does not establish monotonic pass@1 degradation during stochastic, multi-step training.As policy-dependent weights and prompt agreement scores evolve, gradient alignment may change and pass@1 may stabilize or recover; success-probability separation and smoothness are needed for specific guarantees, not the core population-gradient identity.Source: B Limitations · p. 16
- The original MATH experiments use filtered hard/easy subsets and reduced-space diagnostics, while the end-to-end study uses 32 responses per prompt and may have substantial Monte Carlo variance.The end-to-end experiment evaluates all 500 held-out MATH-500 prompts and computes diagnostics with respect to trainable LoRA parameters.Source: B Limitations · p. 16
- The illustrative empirical study is limited to one 1.5B model, one mathematical-reasoning benchmark, and k = 8, leaving broader frequency and strength across settings to future evaluation.The authors also identify testing and comparing the suggested mitigation strategies in full LLM post-training as future work.Source: B Limitations · p. 16
C.1 Setting and illustration of negative prompt interference
- In the toy setting, easy and hard prompts have overlapping Gaussian features and share a two-action softmax policy, allowing similar prompt representations with different correct actions.For prompts in the overlap, opposite per-prompt pass@1 gradients make them negatively interfering; Figure 5 visualizes the cosine-kernel pattern for prompt subsamples.Source: §C.1 · pp. 16–17, Figure 5 · p. 17
C.2 Tradeoff: pass@k ascent can decrease pass@1
A two-prompt toy example shows how pass@k reweighting can make its gradient oppose pass@1 and decrease single-shot performance. The discussion identifies negative correlation between hardness weights and agreement scores, together with sufficient variance, as a condition for conflict.
C.2 Tradeoff: pass@k ascent can decrease pass@1
- The toy prompts have nearly identical representations but opposite labels, yielding almost antiparallel per-prompt pass@1 gradients and negative interference.Their representation cosine is approximately 0.98, while their gradient cosine is approximately −0.98.Source: §C.2 · p. 17
- Pass@1 falls from 0.48 to 0.46 while pass@10 rises from 0.83 to 0.95 after a pass@10 gradient step in the toy example.The pass@10 and pass@1 gradients have cosine similarity −0.77; the low-success hard prompt is upweighted while the easy prompt is downweighted.Source: §C.2 · p. 17
D Discussion of the covariance condition in Proposition 4.1
- Conflict requires strong negative correlation between hardness weights and agreement scores, as well as sufficient variance in both.When either the weights or agreement scores barely vary, the conditions for a strong focus shift or meaningful conflict structure weaken.Source: D Discussion of the covariance… · p. 18
E A kernel view of gradient conflict beyond k = 1
The kernel view separates where pass@k objectives place weight from whether prompts interfere, and shows that negative interference is necessary for gradient conflict. The accompanying proof sections give a weighted-agreement characterization, sufficient-condition bounds, a k-dependent weight analysis, and pass@k smoothness.
E A kernel view of gradient conflict beyond k = 1
- For pass@k and pass@m, gradient alignment depends on both their prompt weights and the sign pattern of the prompt-interference kernel; conflict occurs when emphasized regions have negative cross-similarity.If pairwise prompt similarities are nonnegative almost everywhere, all pass@k objectives are mutually aligned.Source: E A kernel view of gradient conflict… · pp. 18–19
F.1 Proof of Proposition 4.1: Gradients conflict characterization
- The proof characterizes pass@k–pass@1 conflict by whether the pass@k-weighted average of prompt agreement scores is negative.The cited derivation equates the gradient inner product with the expectation of the pass@k weight times the agreement score.Source: §F.1 · p. 19
F.2 Proof of Corollary 4.4: Dominating negatively interfering prompts
- A sufficient conflict condition follows when the aggregate weight on strongly negatively interfering prompts dominates the contribution from the remaining prompts.The proof defines a bound δ(θ) from these two aggregate contributions and obtains a negative gradient inner product when δ(θ) > 0.Source: §F.2 · p. 20
F.3 Proof of Proposition 4.5: Influence of k
- Under success-probability separation, the proof bounds pass@k weights differently for low-success prompts and the remaining prompts.For pθ(x) ≤ ε the weight is at least k(1 − ε)^(k−1), while for pθ(x) ≥ δ it is at most k(1 − δ)^(k−1).Source: §F.3 · p. 20
F.4 Lemma F.1: Smoothness of pass@k
- Under Assumption 4.3, the pass@k objective is Lk-smooth with Lk = k^2G^2 + kF.The proof obtains this smoothness by uniformly bounding the objective’s Hessian.Source: §F.4 · pp. 20–21
F.5 Proof of Proposition 4.6: Pass@1 degradation under pass@k updates
The proof establishes simultaneous pass@1 decrease and pass@k increase for a pass@k gradient step under a sufficient step-size condition. The experimental setup samples 32 responses per problem for correctness evaluation.
F.5 Proof of Proposition 4.6: Pass@1 degradation under pass@k updates
- A sufficient step-size condition guarantees that one pass@k policy-gradient step decreases pass@1 while increasing pass@k, when the pass@k gradient is nonzero.The proof bounds pass@1 descent and pass@k ascent separately, using smoothness for the latter.Source: §F.5 · p. 22
G.1 More details regarding experimental setting
- Experiments sample 32 independent responses per problem at temperature 0.7 with nucleus p = 0.95, then score correctness by exact match.Rewards are binary indicators of correctness.Source: §G.1 · p. 23
G.2 Threshold robustness analysis
The appendix finds gradient misalignment robust across threshold choices and shows that pass@k weights concentrate on negatively aligned prompts. Additional code-generation materials examine this prompt-interference pattern and identify negatively interfering prompt pairs.
G.2 Threshold robustness analysis
- Across additional threshold configurations, gradient inner products range from −0.49 to −0.65, showing misalignment is robust to threshold choices.Figure 7 presents agreement scores, weights, and weighted contributions for different (δ1, δ2) configurations.Source: §G.2 · p. 23, Figure 7 · p. 24
G.3 Additional ablations: Which prompts are amplified by pass@k?
- Across representative configurations, high pass@k weights concentrate on prompts with negative agreement scores, which the authors identify as negatively interfering with pass@1.Figure 8 displays agreement scores, pass@k weights, and pass@1 probability for both models across representative thresholds.Source: §G.3 · p. 23, Figure 8 · p. 25
G.4 Additional experiments on code verification
- The code-generation tables report negatively interfering prompt pairs, including top-five pairs for Qwen2.5-Coder-1.5B-Instruct on HumanEval and Meta-Llama-3.1-8B on MBPP.Table 2 covers code-generation benchmarks more broadly.Source: Table 3 · p. 25, Table 2 · p. 25, Table 4 · p. 26
- For HumanEval with Qwen 1.5B, Figure 9 shows high weights on negatively aligned, low-pass@1 prompts and small weights on positively aligned, high-pass@1 prompts.The figure compares these prompt-level patterns for different values of k.Source: Figure 9 · p. 26