ResRL: Boosting LLM Reasoning via Negative Sample Projection Residual Reinforcement Learning

• RLVR can improve targeted metrics while reducing output diversity, so gains in Pass@1 may come at the expense of Pass@k and out-of-distribution generalization (Simoni et al., 2025; Zhu et al., 2025a; Deng et al., 2025d; Zeng et al., 2024).Source: §1 · p. 1
• NSR preserves diversity by suppressing negative responses, but shared positive-negative tokens can cause gradient conflict and increased negative weighting may limit Pass@1 gains (Zhu et al., 2025b; Deng et al., 2025c;b; Simoni et al., 2025).The shared distributions can include syntactic structures and partial reasoning steps, so penalizing a negative trajectory can also reduce the likelihood of valid tokens in positive trajectories.Source: §1 · pp. 1–2
• ResRL uses policy hidden states to project negative representations onto a low-rank positive subspace, then uses projection residuals to target orthogonal, potentially erroneous components while preserving shared semantics.A length-scaled reward mechanism is also used as a safeguard against verbosity.Source: §1 · p. 2
• ResRL connects Lazy Likelihood Displacement to negative-positive gradient interference and derives a single-forward proxy that serves as a monotonic upper bound.The output-head gradient inner product decomposes into logit and representation components.Source: §1 · p. 2
• Across twelve benchmarks, ResRL improves both Avg@16 and Pass@128 while outperforming strong baselines including GRPO and NSR.On Qwen3-4B mathematics it improves over NSR by 9.4% Avg@16 and 7.0% average Pass@128; on CodeForces it improves over NSR by 9.6% in rating. It exceeds EMPG by 10.4% in ALFWorld success rate and ResT by 2.8% in multi-turn tool-use accuracy.Source: §1 · p. 2
• Ablations on rank selection, hidden-layer choice, and quantile thresholds support the synergy and importance of ResRL’s proposed modules.Source: §1 · p. 2

Related Work

• RLVR exploration work addresses search-space narrowing through MCTS and adaptive Pass@k objectives, amid debate over novel reasoning skills, spurious rewards, and mode collapse (Guo et al., 2025; Deng et al., 2025a; Shao et al., 2025; Wu et al., 2025; Chen et al., 2025; Yang et al., 2025b; Yu, 2025).Other approaches derive closed-form Pass@k gradients or use differentiable top-1 approximations (Walder & Karkhanis, 2025; Peng et al., 2025).Source: §2 · p. 2
• To refine sparse-verifier supervision, studies use structural proxies, probability divergence, uncertainty estimates, or hidden-state distributions to guide RLVR exploration (Xin et al., 2025; Zhao et al., 2025; Wang et al., 2025; Zhu et al., 2025a; Deng et al., 2025d).Source: §2 · p. 3
• Semantically similar positive and negative tokens cause conflicting gradients and LLD, while negative upweighting and token-level loss balancing only partially mitigate overlap (Simoni et al., 2025; Deng et al., 2025c;b; Zhu et al., 2025b; Zeng et al., 2024).Using projection residuals to disentangle the shared semantic distributions remains unexplored for boosting Pass@1 and Pass@k.Source: §2 · p. 3

3.1. Theoretical Framework

• LLD is governed by accumulated cross-sign head-gradient interference, which decomposes into logit-space and representation terms and motivates a single-forward geometric proxy.Full token-wise gradient evaluation is costly, so the method uses orthogonal-complement energy to upper-bound representation alignment.Source: §3.1 · p. 3

Definition 2 (Orthogonal-complement energy). For any representation x, define

• When the positive subspace sufficiently covers positive tokens, increasing a negative token’s orthogonal-complement energy decreases an upper bound on its alignment with positive representations.The energy is therefore a conservative proxy for interference, up to an additive error.Source: Definition 2 (Orthogonal-complement… · p. 4

3.2. Algorithm Design

• ResRL estimates the positive subspace from penultimate-layer hidden states, normalizing and centering them so the subspace captures positive-distribution covariance and deviation from correct reasoning.The preceding layer is used to capture higher-level semantic abstractions less biased by immediate token prediction.Source: §3.2 · p. 4

Subspace Estimation and Residual Computation.

• To estimate the subspace efficiently, ResRL samples positive tokens, applies truncated SVD, and uses top-k directions to compute negative-token projection residuals, then quantile-normalizes and clips them into token-wise weights.Group-relative quantile normalization addresses residual-scale variation across prompts.Source: Subspace Estimation and Residual… · pp. 4–5

where ξ ∈(0, 1] denotes the minimum weight.

• ResRL applies a small positive-advantage scaling of λpos = 0.1 as a weak anchor, while assigning lower penalties to negative tokens aligned with the positive subspace and higher penalties to deviating tokens.This weighting is intended to reduce accidental suppression of shared positive directions.Source: where ξ ∈(0, 1] denotes the minimum… · p. 6

4.1. Training Details

• Experiments compare RLVR and NSR baselines on twelve math, code, agent, and function-calling benchmarks, using Qwen variants from 1.7B to 8B and matched training hyperparameters and budgets.Training uses DAPO data for math, DeepCoder for code, prior settings for agent tasks, and ToolRL data for function calling; evaluation reports Avg@16, plus CodeForces Elo and percentile, under task-specific sampling and rollout settings.Source: §4.1 · p. 6

4.2. Main Results

• ResRL leads mathematical Avg@16, beating FlowRL by 15.7%, 6.3%, and 4.2% and NSR by 2.3%, 9.4%, and 4.5% on 1.7B, 4B, and 8B, respectively.On Qwen3-4B, ResRL improves Pass@128 by 7.0% over NSR, averaged across AIME24, AIME25, and AMC23. It also leads on challenging AIME sampling budgets; on AMC23, it leads at low k but converges with baselines at high k as the task saturates.Source: §4.2 · pp. 6–7
• ResRL reaches a 1469.5 CodeForces rating versus NSR’s 1340.9 (+9.6%), raises percentile by 13.9%, and scores 86.7 ALFWorld success, above PPO by 7.8% and EMPG by 10.4%.On BFCL, it achieves the best Multi-Turn OA, 2.8% over ResT, and improves Miss Func and Miss Param by 4.4% and 6.3%.Source: Table 4 · p. 7, §4.2 · p. 7

4.3. Ablation Analysis

• The penultimate hidden layer performs better than the final layer, while stricter quantile thresholds q=0.1 or 0.2 converge faster and reach higher AIME accuracy than q=0.3.The penultimate layer provides a more stable semantic signal; q=0.1 also maintains low gradient variance while enabling broader exploration.Source: §4.3 · pp. 7–8
• Rank k trades protection against discrimination: k=64 is most accurate and stable, while k=8 under-covers shared semantics and k≥128 weakens residual contrast and destabilizes updates.Larger k expands the positive subspace but can absorb error-specific directions; high ranks show oscillatory updates and bursty gradient norms.Source: §4.3 · p. 7, Figure 4 · p. 8
• With length-scaled positive rewards, ResRL (Qwen3-8B) improves on AIME 2024/2025 over 800 steps with stable optimization, while response length remains flat as KL rises smoothly.The discount is unchanged through 3500 tokens, then decreases linearly to 70% over 3500–4096 tokens.Source: §4.3 · p. 8
• Subspace estimation is robust at moderate token budgets: Mmax=4096 is consistently strong and stable, while gains diminish beyond 4096 and Mmax=2048 slightly lags.At Mmax=8192, lower KL accompanies slower accuracy gains; Mmax=6144 is more susceptible to drift without accuracy benefit. The method uses Mmax=4096 by default.Source: §4.3 · p. 8
• Removing representation normalization sharply degrades AIME accuracy and destabilizes optimization, while removing KL improves AIME2024 accuracy by 9% and remains stable.Normalization makes residual signals comparable across tokens; ResRL’s projection weight attenuates negative gradients for tokens aligned with the positive subspace, acting as an intrinsic regularizer without explicit KL tethering.Source: §4.3 · p. 8

5. Conclusion

• ResRL improves both Pass@1 and Pass@k over strong GRPO/NSR baselines across twelve benchmarks while maintaining diversity.The benchmarks span Mathematics, Code, Agent tasks, and Function calling.Source: §5 · p. 8
• 9.4% in Avg@16 and 7.0% in pass@128 are ResRL’s reported gains over NSR on mathematical reasoning.The conclusion describes these results as validating ResRL’s efficacy and scalability in RLVR training.Source: §5 · pp. 8–9
• ResRL links Lazy Likelihood Displacement to negative–positive gradient interference and uses a single-forward proxy to conservatively control it via bounded representation alignment.Its optimization reweights negative gradients using projection residuals, targeting error-specific components while preserving semantics shared with correct trajectories.Source: §5 · p. 8

Y. Deepsearch: Overcome the bottleneck of reinforce-

• The supplied Deepsearch subsection contains a bibliography fragment describing an arXiv preprint on reinforcement learning with verifiable rewards via Monte Carlo tree search.The provided text does not include a substantive finding from that work.Source: Y. Deepsearch: Overcome the bottleneck… · p. 11

Y. The surprising effectiveness of negative reinforce-

• The supplied surprising-effectiveness subsection contains a bibliography entry titled “Position information emerges in causal transformers without positional encodings via similarity of nearby embeddings.”The provided text does not include a substantive finding from that work.Source: Y. The surprising effectiveness of… · p. 11

Consider the linear output head that maps a token representation x ∈Rd to logits

• The setup treats token-wise loss as a differentiable function of logits and introduces the backpropagation signal at the logits.Source: Consider the linear output head that… · p. 12
• The proof uses the Frobenius inner product for same-shaped matrices and the Euclidean inner product for vectors.Source: Consider the linear output head that… · p. 12
• Lemma 1 establishes that the output-head loss gradient is δx⊤ and that the gradient inner product factorizes.The proof states these as its two claims and completes the factorization as Eq. (3).Source: Consider the linear output head that… · p. 12, Since this holds for all (a, b), the… · p. 12
• The gradient derivation writes each logit coordinate explicitly and applies the entry-wise chain rule to obtain ∇W ℓ = δx⊤.Source: Consider the linear output head that… · p. 12

Since this holds for all (a, b), the gradient matrix satisfies (∇W ℓ)ab = δaxb, hence

• For two token instances, the factorization considers their respective pairs of backprop signals and token representations, (δ1, x1) and (δ2, x2).Source: Since this holds for all (a, b), the… · p. 12
• Expanding the Frobenius inner product over matrix entries completes the derivation of Eq. (3).Source: Since this holds for all (a, b), the… · p. 12

A.2. Proof of Eq. (4)

• Eq. (4) expresses head-gradient inner products through the token weights, backprop signals, and representation inner products.The proof pulls out scalar weights by bilinearity and applies the exact head-gradient factorization ∇W ℓk = δk xk⊤.Source: Taking absolute values yields · p. 13, We show that, for any two token… · p. 13
• For cross-sign token pairs, controlling |⟨x−, x+⟩| directly limits head-gradient interference up to advantage and backprop-signal factors.This motivates a single-forward proxy that upper-bounds representation similarity.Source: Taking absolute values yields · p. 13

A.3. Details for Definition 1 (Positive subspace construction)

• The positive reference subspace is built per prompt group from token-wise LayerNormed positive representations centered by their positive mean.Centering captures variation among positives rather than their shared mean offset.Source: §A.3 · p. 14, and outputs · p. 14, For any token h (positive or… · p. 14
• The top-k PCA directions of the centered positive matrix are exactly its top-k right singular vectors from truncated SVD.This subspace maximizes variance captured by projection onto its span.Source: A standard characterization of PCA is… · p. 14

A.4. Details for Definition 2 (Orthogonal-complement energy)

• The positive-subspace projector and its orthogonal complement decompose any representation into positive-aligned and residual components.The residual component is orthogonal to the positive subspace.Source: When Vk has orthonormal columns (V ⊤ · p. 15, For any x ∈Rd, the decomposition · p. 15, be the orthogonal projector onto S =… · p. 16
• The statistic e(x) is the nonnegative normalized squared length of x's component in S⊥, and it is zero exactly when x lies in S.It therefore quantifies deviation from the positive subspace constructed within the same prompt group.Source: Therefore e(x) is exactly the… · p. 16, §A.4 · p. 16
• The residual norm is the distance from x to its closest point in the positive subspace, so e(x) is small near S and large when the orthogonal component is substantial.The 1/d factor makes e(x) an average per-dimension squared residual for comparability across hidden sizes.Source: Moreover, using PS = VkV ⊤ · pp. 16–17

A.5. Proof of Lemma 2 (Alignment bound)

• Alignment with any positive direction in S is limited by the energy of x inside S, with residual energy in S⊥ reducing the maximum squared inner product.Equality holds when x ∈ S and x is colinear with the positive direction.Source: Step 3: Rewrite ∥PSx∥2 · p. 18

A.6. Proof of Theorem 1 (Residual proxies gradient alignment)

• When positive tokens satisfy e(x+) ≤ ε+ for most tokens, negative-token residual energy e(x−) monotonically bounds worst-case gradient interference, up to positive-capture error and logit-signal similarity.The bound decreases as e(x−) increases; its additive approximation error depends on positive-subspace capture.Source: Using x+ · p. 20

Algorithm 1 ResRL (per prompt group)

• ResRL falls back to baseline clipped GRPO when a prompt group has no positive trajectories; otherwise it samples and centers positive tokens, builds a rank-k PCA subspace, and weights negative tokens by residual-based quantiles.The algorithm applies a negative-penalty floor, optionally restores weights for truncation-tail tokens, and scales positive advantages by λ+.Source: Require: Prompt-group trajectories {yi}G · p. 21

C.1. ResRL: Residual-based proxy for head-gradient interference

• ResRL forms a centered positive-token matrix from at most Mmax sampled tokens and computes projection residual energy for each valid negative token at O(dk) cost per token.The positive matrix construction costs O(Md) time and memory.Source: §C.1 · p. 22, Applying PS to a vector can be… · p. 22

C.2. LLD/NTHR: Gradient-inner-product score and efficiency tricks

• LLD/NTHR reduces pairwise token interactions by summing token contributions before the final matrix inner product, then restricting computation to the response-specific vocabulary for linear-in-T accumulation.Without vocabulary restriction, the passage gives O(T |V| d) as the alternative cost.Source: §C.2 · p. 23

C. Time Complexity of Gradient-Inner-Product Modules

• Under M ≪ T− and T ≍ T−, ResRL’s overhead is O((M + T−)dk) versus O(T |V⋆| d) for LLD/NTHR, simplifying to a reduction factor of roughly |V⋆|/k.Against naïve LLD’s O(T+T−d) token-pair cost, ResRL uses O((M + T−)dk).Source: When M ≪T−and T ≍T−(typical… · p. 23, linear-time accumulation over tokens… · p. 23, The final matrix inner product adds at… · p. 23

• ResRL dominates practical low-to-mid Pass@k regimes (k≤26) and remains competitive at k=27 on AIME24/25 and AMC23 with Qwen3-8B.The figure characterizes this as a precision–diversity trade-off that avoids relying on high-variance brute-force exploration.Source: Figure 5 · p. 24
• Removing the explicit KL term significantly boosts Qwen3-8B Avg@16 accuracy on AIME2024 and AIME2025, while KL divergence rises and training remains stable.The figure concludes that projection-based weighting is a sufficient intrinsic regularizer, making the strict SFT constraint redundant.Source: Figure 6 · p. 24
• Using the penultimate rather than final hidden layer significantly improves AIME2024 and AIME2025 accuracy, with higher KL divergence and actor entropy indicating more exploration.The passage attributes the gain to high-level semantic abstractions that mitigate final-layer token-prediction bias and prevent premature convergence.Source: Figure 7 · p. 25
• Quantile thresholds 0.1 and 0.2 converge faster and achieve higher AIME2024 and AIME2025 accuracy than 0.3; the 0.1 setting remains stable despite elevated KL divergence.The passage describes stricter residual penalization as driving exploration and supporting a stricter gating mechanism.Source: Figure 8 · p. 26
• Across 800 training steps without explicit KL regularization, ResRL gains AIME 2024/2025 performance while KL rises boundedly and response length, entropy, and gradient norms remain stable.The passage attributes this stability to subspace-based semantic constraints preventing mode collapse and reward hacking without rigid SFT anchoring.Source: Figure 9 · p. 27, §D.1 · p. 27
• SVD subspace budgets Mmax=2048–4096 yield similar accuracy and stable optimization, while larger budgets offer diminishing returns and can weaken residual-based error shaping.Mmax=8192 shows lower KL but slower accuracy gains; Mmax=6144 can increase KL and entropy without proportional accuracy improvement. Mmax=4096 provides a favorable tradeoff.Source: Figure 10 · p. 28
• Removing LayerNorm and centering degrades AIME2024 and AIME2025 performance and causes high-variance actor-gradient spikes.The passage concludes that representation normalization is essential for reliable residual signals and robust optimization.Source: Figure 11 · p. 29

Pass@k dynamics and compute regimes

• Across model scales, ResRL's most consistent Pass@k advantage occurs at k≤64, while it remains competitive at high k and can dominate for some backbones.The passage attributes low-to-mid-budget gains to attenuating updates on negatives aligned with the positive subspace, preserving shared intermediate steps and reallocating negative pressure toward error-specific components.Source: §D.2 · p. 30

Code reasoning

• On Qwen3-4B code reasoning, ResRL leads on all reported metrics, including CodeForces rating and percentile of 1469.5 and 78.9.It reaches 43.2 Avg@16 and 59.9 Pass@16 on LiveCodeBench, versus FlowRL's 42.4 and 58.7; on HumanEval+, its 97.0 Pass@16 matches or slightly exceeds the best baseline.Source: §D.2 · p. 30

Long-horizon agent tasks

• On Qwen2.5-7B-Instruct, ResRL raises ALFWorld success to 86.7 and WebShop success to 71.5 while maintaining a WebShop task score of 81.2.ALFWorld success exceeds PPO (80.4), EMPG (78.5), and GRPO (74.8); WebShop success exceeds EMPG (69.3) and GRPO (65.6).Source: §D.2 · p. 30

Tool-use robustness on BFCL

• On BFCL, ResRL achieves the best Multi-Turn OA (41.25) and highest overall single-turn OA (68.95), with gains on Miss Func (47.0) and Miss Param (34.0).Long-Context remains harder, with ResRL at 35.5.Source: §D.2 · p. 30

Design takeaway

• Across task families, the largest gains occur where positive and negative responses share partial structure, and improvements concentrate in reliability-centric metrics without forcing premature mode collapse.The passage characterizes this pattern as a precision–diversity trade-off favorable under realistic compute budgets.Source: §D.2 · p. 30

E. Training Parameters

• Table 6 details the model, optimization, generation, and infrastructure settings for ResRL math training.The caption says the configuration is derived from the training script.Source: Table 6 · p. 31
• Table 7 details the model, optimization, generation, and infrastructure settings for ResRL code training.The caption identifies this as the code-training hyperparameter configuration.Source: Table 7 · p. 32

F. Output Cases

• Four OlympiadBench rollouts use different proof decompositions and levels of algebraic detail while preserving logical consistency.The cases illustrate diversity in reasoning structure, complementing performance under high-k sampling.Source: F. Output Cases · p. 33, Figure 12 · p. 33, Figure 13 · p. 34
• The section includes four ResRL-trained Qwen3-8B response examples on Math500, all generated in no-think mode.The figure captions identify four separate rollouts; they do not specify their substantive differences.Source: Figure 16 · p. 37, Figure 17 · p. 38, Figure 18 · p. 39
• Humaneval+ rollouts vary computational strategies and code organization while remaining faithful to the specification.Examples include brute force versus sorting-and-scanning, as well as differences in function decomposition, guard conditions, and edge-case handling.Source: F. Output Cases · pp. 41–42, Figure 20 · p. 41, Figure 21 · p. 42
• Two think-mode ResRL-trained Qwen3-4B response examples are provided for the Codeforces test set.The figure captions identify the test set and decoding mode but do not describe the responses’ content.Source: Figure 22 · p. 43, Figure 23 · p. 44