PaperlayerView exact revision on arXiv

Source-linked AI summary

ResRL: Boosting LLM Reasoning via Negative Sample Projection Residual Reinforcement Learning

Zihan Lin, Xiaohan Wang, Jie Cao, Jiajun Chai, Li Wang, Xiaodong Lu, Wei Lin, Ran He, Guojun Yin

arXiv:2605.00380v2cs.LGcs.CL

TL;DR

RLVR can improve LLM reasoning while reducing generation diversity, motivating methods that suppress errors without penalizing semantics shared with correct responses. ResRL decouples positive and negative updates through projection residuals, and across twelve benchmarks consistently improves performance, surpassing NSR on mathematics by 9.4% in Avg@16 and 7.0% in Pass@128.

  • Problem

    How can policy optimization disentangle positive and negative responses to suppress errors without penalizing valid semantic distributions shared with correct trajectories?

  • Method

    ResRL uses hidden-state semantic representations, an SVD-based low-rank positive subspace, and projection residuals to target error-specific negative updates while preserving shared semantics.

  • Results

    9.4% in Avg@16 and 7.0% in Pass@128 over NSR on mathematics, while ResRL consistently outperforms strong baselines across twelve benchmarks.

  • Takeaways & Limitations

    ResRL improves reasoning while preserving generation diversity across Mathematics, Code, Agent Tasks, and Function Calling benchmarks.

  • Takeaways & Limitations

    The theoretical bound assumes the selected subspace sufficiently covers positive-token representations.

Abstract

from arXiv · show

Reinforcement Learning with Verifiable Rewards (RLVR) enhances reasoning of Large Language Models (LLMs) but usually exhibits limited generation diversity due to the over-incentivization of positive rewards. Although methods like Negative Sample Reinforcement (NSR) mitigate this issue by upweighting penalty from negative samples, they may suppress the semantic distributions shared between positive and negative responses. To boost reasoning ability without losing diversity, this paper proposes negative sample projection Residual Reinforcement Learning (ResRL) that decouples similar semantic distributions among positive and negative responses. We theoretically link Lazy Likelihood Displacement (LLD) to negative-positive head-gradient interference and derive a single-forward proxy that upper-bounds representation alignment to guide conservative advantage reweighting. ResRL then projects negative-token hidden representations onto an SVD-based low-rank positive subspace and uses projection residuals to modulate negative gradients, improving reasoning while preserving diversity and outperforming strong baselines on average across twelve benchmarks spanning Mathematics, Code, Agent Tasks, and Function Calling. Notably, ResRL surpasses NSR on mathematical reasoning by 9.4\% in Avg@16 and 7.0\% in Pass@128. Code is available at https://github.com/1229095296/ResRL.git.

1. Introduction

RLVR improves LLM reasoning but can limit generation diversity, while NSR may suppress semantic distributions shared by positive and negative responses. ResRL addresses this by projecting negative representations away from positive subspaces and using residuals to modulate penalties, improving reasoning while preserving diversity.

  • Motivation: RLVR enhances LLM reasoning, but recent studies indicate it effectively optimizes targeted metrics while limiting generation diversity.DeepSeek-R1 demonstrated significant improvements in complex scenarios and introduced GRPO.
  • Problem: NSR differentiates positive and negative responses to improve Pass@k and diversity, but its increased negative weighting may limit Pass@1 gains.The approach actively suppresses low-reward responses rather than mainly encouraging high-reward ones.
  • Method: ResRL confines negative-sample penalties to directions orthogonal to positive representations, reducing penalties on shared valid semantic distributions.Policy hidden states serve as a proxy for semantic distributions, enabling gradient decoupling between overlapping positive and negative regions.
  • Theory: ResRL links LLD to negative-positive gradient interference and introduces a single-forward proxy that upper-bounds representation alignment for conservative advantage reweighting.The framework decomposes output-head gradient inner products into logit and representation components.
  • Empirical Performance: 9.4% Avg@16: ResRL improves over NSR on Qwen3-4B mathematics; 7.0% average Pass@128: it also surpasses NSR on mathematics.The evaluation covers twelve benchmarks across Mathematical reasoning, Code generation, Agent Tasks, and Function Calling.

2. Related Work

Related work positions RLVR as a dominant approach for eliciting LLM reasoning while highlighting concerns about spurious rewards, premature search-space narrowing, and unresolved optimization conflicts. Recent studies augment sparse verifiers with intrinsic signals, but semantically similar positive and negative tokens still cause instability and LLD.

  • RLVR has become a dominant paradigm for eliciting LLM reasoning, but may refine existing patterns and risk convergence toward spurious rewards.
  • Recent methods augment sparse verifiers with structural, probabilistic, uncertainty-based, or hidden-state signals to guide exploration during RLVR training.
  • Conflicting gradients between semantically similar positive and negative tokens create policy-optimization instability, notably manifesting as LLD.
  • Negative upweighting and token-level loss balancing offer only partial mitigation and do not explicitly disentangle the underlying conflict.

3. Method

ResRL links lazy likelihood displacement to negative–positive gradient interference and derives a single-forward representation proxy for conservative advantage reweighting. It implements this proxy by projecting negative-token representations against an SVD-based positive subspace and using residuals to gate negative-sample reinforcement.

  • Theoretical Analysis: ResRL’s theoretical framework links LLD to negative–positive head-gradient interference and decomposes interference into logit and representation terms.The representation term motivates using hidden-state geometry as a tractable proxy for gradient interference.
  • Theoretical Analysis: The orthogonal-complement energy e(x) conservatively proxies interference because increasing it decreases the attainable similarity between a negative representation and positive directions.The bound assumes the positive subspace sufficiently covers positive tokens, up to an additive error.
  • Representation Construction: ResRL estimates a positive subspace from penultimate-layer semantic representations after normalization and centering, then computes each negative token’s projection residual.Centering makes the subspace capture positive-distribution covariance and measures deviation from the correct reasoning trajectory.
  • Subspace Approximation: To scale subspace construction, ResRL samples M centered positive tokens, applies truncated SVD, and uses the top-k right-singular directions to form the projector.The resulting residual is the tractable proxy for the theoretical interference bound.
  • Group-Relative Gating: ResRL normalizes residuals with group-relative quantiles and anchors positive advantages using λpos = 0.1 while defining residual-based weights for negative samples.Quantile normalization accounts for prompt-to-prompt variation in projection-residual scale.

4. Experiment Analysis

ResRL consistently improves reasoning and task performance across mathematics, code, agents, and function calling, while preserving low- and high-budget sampling performance. Ablations show that intermediate-rank projection, penultimate-layer features, conservative quantile weighting, normalization, and intrinsic regularization are important for accuracy and stable optimization.

  • Cross-Benchmark Results: 15.7%, 6.3%, and 4.2%: ResRL outperforms FlowRL on mathematical Avg@16 for 1.7B, 4B, and 8B models, respectively.ResRL also exceeds NSR by 2.3%, 9.4%, and 4.5% on the same model sizes.
  • Cross-Benchmark Results: 7.0%: ResRL improves averaged AIME24, AIME25, and AMC23 Pass@128 over NSR on Qwen3-4B.Pass@k curves show higher low-k accuracy without sacrificing high-k performance, while AMC23 converges with baselines at high k due to task saturation.
  • Cross-Benchmark Results: 9.6%: ResRL improves CodeForces rating over NSR, reaching 1469.5 versus 1340.9, and also achieves 86.7 overall success on ALFWorld.On ALFWorld, ResRL surpasses PPO by 7.8% and EMPG by 10.4%; the supplied passage states that BFCL results also favor ResRL but truncates the exact metric.
  • Ablation Analysis: k=64: The intermediate projection rank is both most accurate and most stable on AIME24/25, whereas k=8 under-covers shared semantics and k≥128 collapses residual contrast.Large ranks produce oscillatory updates and bursty gradient norms because more negative tokens receive small residual energies after normalization.
  • Ablation Analysis: q=0.1 or 0.2: Stricter quantile thresholds converge faster and reach higher AIME2024/2025 accuracy than q=0.3, while q=0.1 maintains low gradient variance.Lower q increases actor KL and entropy, enabling broader exploration without destabilizing optimization.
  • Ablation Analysis: 9%: Removing the KL term improves AIME2024 accuracy while remaining stable, and removing representation normalization sharply degrades accuracy and optimization stability.Projection weights attenuate negative gradients aligned with the positive subspace, while normalization makes residual signals comparable across tokens.

5. Conclusion · Y. Deepsearch: Overcome the bottleneck of reinforce- · Y. The surprising effectiveness of negative reinforce-

The conclusion presents ResRL as a method for improving LLM reasoning while preserving generation diversity through conservative control of negative–positive gradient interference. It reports that ResRL’s results validate its efficacy and scalability in RLVR training, while the subsequent material consists of related-work references and an impact statement.

  • 5. Conclusion: ResRL improves reasoning without sacrificing generation diversity by controlling negative–positive gradient interference through bounded representation alignment.The method uses a single-forward proxy motivated by a theoretical connection between Lazy Likelihood Displacement and gradient interference in NSR.
  • 5. Conclusion: ResRL represents token-level semantic distributions with policy hidden states, builds an efficient low-rank positive subspace via SVD, and reweights optimization using projection residuals.
  • 5. Conclusion: 7.0% in pass@128 validates ResRL’s efficacy and scalability in RLVR training.The supplied conclusion passage reports this pass@128 result alongside Avg@16.
  • Impact Statement: The impact statement says the work aims to advance LLM Reasoning and identifies no societal consequences requiring specific emphasis.
  • Y. Deepsearch: Overcome the bottleneck of reinforce-: The Deepsearch-related references cover reinforcement learning with verifiable rewards via Monte Carlo tree search, surrogate signals without ground-truth answers, adaptive exploration, and Qwen3.
  • Y. The surprising effectiveness of negative reinforce-: The negative-reinforcement-related references include RL for LLM reasoning, position information in causal transformers, and related reasoning and preference-optimization studies.

A. Proofs and Derivation Details for the Theoretical Framework … Since this holds for all (a, b), the gradient matrix satisfies (∇W ℓ)ab = δaxb, hence

The theoretical framework proves the linear output-head gradient formula ∇W ℓ = δx⊤ and factorizes gradient inner products using Euclidean and Frobenius inner products. The derivation proceeds entry-wise via the multivariate chain rule and concludes by establishing the result for every matrix entry.

  • Consider the linear output head that maps a token representation x ∈Rd to logits: The output head maps token representations x ∈ R^d to logits, with token-wise loss ℓ defined as a differentiable function of logits.The setup introduces the backpropagation signal at the logits.
  • Consider the linear output head that maps a token representation x ∈Rd to logits: For vectors, ⟨a, b⟩ = a⊤b is the Euclidean inner product, while matrix inner products use the Frobenius definition.The notation is specified separately for vector and matrix pairs.
  • A.1. Proof of Lemma 1: The proof targets two results: ∇W ℓ = δx⊤ and a factorization of the gradient inner product.These are explicitly identified as parts (i) and (ii) of Lemma 1.
  • Fix an arbitrary entry Wab of W (row a, column b). By the multivariate chain rule,: For an arbitrary matrix entry W_ab, the gradient is derived by writing each logit coordinate explicitly and applying the entry-wise chain rule.The derivation fixes a row-column entry and expands its dependence through the logits.
  • Since this holds for all (a, b), the gradient matrix satisfies (∇W ℓ)ab = δaxb, hence: The entry-wise derivation yields (∇W ℓ)_ab = δ_a x_b for every pair (a, b).The conclusion follows after substituting the chain-rule expression back into the gradient entry.
  • Since this holds for all (a, b), the gradient matrix satisfies (∇W ℓ)ab = δaxb, hence: For two token instances with pairs (δ_1, x_1) and (δ_2, x_2), the gradient inner product is expanded over all matrix entries.The direct summation establishes the factorization used in Eq. (3).

A.2. Proof of Eq. (4) … When representations are LayerNormed (Definition 1), ∥x∥2

The appendix proves Eq. (4) by factorizing token-level head-gradient interactions, then constructs a positive-token subspace via centered LayerNorm representations and truncated SVD. It defines orthogonal-complement energy as normalized deviation from that subspace and establishes its geometric and stability properties.

  • A.2. Proof of Eq. (4): Eq. (4) follows by combining bilinearity with the exact factorization ∇W ℓk = δk xk⊤.Here δk is the logit backpropagation signal and xk is the representation entering the output head.
  • A.2. Proof of Eq. (4): Controlling |⟨x−, x+⟩| directly controls cross-sign head-gradient interference up to |A−A+| and |⟨δ−, δ+⟩|.This motivates a single-forward proxy that upper-bounds cross-sign representation similarity.
  • A.3. Details for Definition 1 (Positive subspace construction): Positive-token representations are centered within each prompt group so the estimated subspace captures variation rather than a shared mean offset.The method applies token-wise LayerNorm before centering and stacks the resulting positive vectors into X+.
  • A.3. Details for Definition 1 (Positive subspace construction): The top-k positive subspace is the PCA solution and equals the span of the top-k right singular vectors from the truncated SVD of X+.Thus Vk in Definition 1 provides a compact reference for measuring alignment with dominant positive directions.
  • A.4. Details for Definition 2 (Orthogonal-complement energy): The projector PS decomposes any representation into its positive-subspace component and an orthogonal residual, yielding a Pythagorean identity.The complementary projector is PS⊥ = I − PS, which projects onto S⊥.
  • A.4. Details for Definition 2 (Orthogonal-complement energy): e(x) is nonnegative and vanishes exactly when x lies in the positive subspace S.It measures the normalized squared length of the component in S⊥.
  • A.4. Details for Definition 2 (Orthogonal-complement energy): e(x) is small when x is close to S and large when x has a substantial orthogonal component.The residual vector is the displacement from x to its closest point in S under ℓ2 distance.
  • A.4. Details for Definition 2 (Orthogonal-complement energy): Normalizing by d makes e(x) an average per-dimension squared residual, helping stabilize comparisons across hidden sizes and LayerNormed representations.The appendix notes that LayerNorm controls the residual norm, supporting e(x) as a stable within-group deviation summary.

A.5. Proof of Lemma 2 (Alignment bound) … D. Additional Implementation Details

The appendix proves that projection residual energy controls alignment and provides a conservative proxy for negative-positive gradient interference. It then specifies ResRL’s token-weighting algorithm and compares its low-rank computational overhead with LLD/NTHR.

  • A.5. Proof of Lemma 2 (Alignment bound): The alignment bound is tight when x lies in the positive subspace and its in-subspace component is colinear with x+; residual energy limits achievable alignment.The proof uses orthogonal decomposition, Cauchy–Schwarz, and the Pythagorean identity.
  • A.6. Proof of Theorem 1 (Residual proxies gradient alignment): For fixed positive representation x+, the alignment bound is monotonically non-increasing in the negative representation’s residual energy e(x−).Under positive-subspace capture, e(x−) proxies worst-case gradient interference up to positive-capture error and logit-space similarity.
  • B. Algorithm Design: ResRL splits trajectories by advantage, builds a rank-k positive subspace from sampled positive hidden states, and reweights negative tokens using projection residuals.Positive tokens receive λ+ ˆAi, while negative tokens receive ωi,t ˆAi; truncation-guard tokens can be assigned ωi,t = 1.
  • D. Additional Implementation Details: If no positive trajectories exist, ResRL performs no reweighting and optimizes the baseline clipped GRPO objective.Boundary-aware sampling retains head and tail tokens while uniformly subsampling middle tokens under the positive-token cap Mmax.
  • C.1. ResRL: Residual-based proxy for head-gradient interference: ResRL forms a centered positive matrix, computes top-k principal directions, and evaluates residual energy for every valid negative token within each prompt group.The method uses LayerNormed hidden states and a projector PS = VkV ⊤k.
  • C.2. LLD/NTHR: Gradient-inner-product score and efficiency tricks: LLD/NTHR reduces quadratic token-pair interactions to linear-time accumulation over tokens, with dominant cost O(T |V⋆|d) after response-vocabulary restriction.The final accumulated matrix inner product adds at most O(|V⋆|d) memory-related work.
  • C. Time Complexity of Gradient-Inner-Product Modules: ResRL’s dominant overhead is O((M + T−)dk), combining low-rank subspace construction and per-negative-token projection residual computation.The positive-token sample count is capped by Mmax, making SVD overhead predictable for long rollouts.
  • C.2. LLD/NTHR: Gradient-inner-product score and efficiency tricks: When M ≪ T− and T ≍ T−, ResRL’s overhead is reduced relative to LLD/NTHR by roughly a factor of |V⋆|/k.Against naïve quadratic LLD, ResRL replaces O(T+T−d) with O((M + T−)dk).

D.1. Additional Experiments

Additional experiments show that ResRL’s semantic constraints support exploration, stability, and strong reasoning performance without requiring strict KL or SFT anchoring. Ablations identify the penultimate layer, low quantile thresholds, moderate subspace budgets, and representation normalization as important design choices.

  • KL regularization: Removing explicit KL regularization improves AIME2024 and AIME2025 Avg@16 accuracy, while ResRL’s projection weighting maintains stable training during increased exploration.The ablation indicates that projection-based weighting can act as an intrinsic regularizer, making the strict SFT constraint redundant.
  • Hidden-layer selection: The penultimate hidden layer improves AIME2024 and AIME2025 accuracy over the final layer while enabling greater exploration through elevated KL divergence and actor entropy.It captures high-level semantic abstractions while reducing the final layer’s immediate token-prediction bias.
  • Quantile sensitivity: Lower quantile thresholds of 0.1 and 0.2 accelerate convergence and improve AIME2024 and AIME2025 accuracy over 0.3, while 0.1 remains stable despite more aggressive exploration.Lower thresholds increase KL divergence, whereas the 0.1 setting maintains low gradient variance.
  • Long-horizon stability: Without explicit KL regularization, 800-step training continues improving AIME2024/2025 performance while KL divergence rises naturally and response length, entropy, and gradient norms remain stable.These dynamics indicate bounded exploration and resistance to mode collapse and reward hacking.
  • SVD subspace budget: Moderate SVD subspace budgets of Mmax=2048–4096 achieve similar accuracy with stable optimization, consistent with redundancy and low effective dimensionality in the representations.The analysis uses fixed rank k, group rollouts G=4, and maximum response length 4096.

D.2. Additional Analysis

Additional analyses show ResRL’s strongest and most consistent benefits in practical, reliability-focused regimes across mathematics, code, long-horizon agents, and tool use. These results support residual projection as an interference-control mechanism that preserves shared structure while improving performance.

  • Pass@k dynamics and compute regimes: ResRL’s most consistent Pass@k advantage occurs in the low-to-mid compute regime, k≤64, improving early-curve reliability across model scales.Figures 2, 3, and 5 visualize these dynamics on AIME24/25 and AMC23 as sampling budgets increase.
  • Mathematical reasoning (Avg@16): 48.6 Avg@16 on Qwen3-1.7B exceeds NSR (47.5), FlowRL (42.0), and GRPO (35.9), with the largest gains concentrated on AIME24/25.ResRL reaches 34.9/29.6 on AIME24/25 versus 24.2/14.4 for FlowRL, while near-saturated datasets change minimally.
  • Code reasoning: 43.2 Avg@16 and 59.9 Pass@16 on LiveCodeBench surpass FlowRL (42.4/58.7), while CodeForces reaches 1469.5 rating and 78.9 percentile.ResRL is best on all reported code metrics and reaches 97.0 Pass@16 on HumanEval+, matching or slightly exceeding the best baselines.
  • Long-horizon agent tasks: 86.7 overall success on ALFWorld surpasses PPO (80.4), EMPG (78.5), and GRPO (74.8), with gains including LOOK 85.5 and PICK2 84.2.The analysis attributes this behavior to shared prefixes between successful and failed long-horizon trajectories, which naive negative reinforcement can corrupt.
  • Tool-use robustness on BFCL: 41.25 Multi-Turn OA and 68.95 single-turn OA are the best BFCL results, including 47.0 on Miss Func and 34.0 on Miss Param.The gains are particularly pronounced on error-sensitive subsets, consistent with reduced compounding of localized tool-selection and argument-specification errors.
  • Design takeaway: Across tasks, the largest gains occur where positives and negatives share partial structure, while improvements concentrate in Avg@16 and low-k Pass@k and remain competitive at high k.The design takeaway supports residual-projection reweighting as interference control in representation space.

E. Training Parameters · F. Output Cases

The paper documents ResRL’s mathematical and code-training configurations and presents output cases illustrating diverse reasoning paths and implementation strategies across benchmarks. These examples use distinct models and decoding setups to show diversity in mathematical reasoning and code generation.

  • E. Training Parameters: ResRL Math training is documented through comprehensive model, optimization, generation, and infrastructure hyperparameter settings derived from the training script.The configuration is provided in Table 6 and its accompanying implementation description.
  • E. Training Parameters: ResRL Code training similarly documents model, optimization, generation, and infrastructure settings derived from the training script.The configuration is provided in Table 7 and its accompanying implementation description.
  • F. Output Cases: On OlympiadBench, four independent ResRL-trained Qwen3-8B rollouts use no-think decoding to demonstrate diversity in difficult mathematical reasoning.The rollouts vary intermediate claims, subgoal ordering, and algebraic detail while preserving global logical consistency.
  • F. Output Cases: On Humaneval+, ResRL-trained Qwen3-4B rollouts use think decoding to demonstrate algorithmic and implementation-level diversity.The examples include different solution paradigms, such as brute force versus sorting-and-scanning, and varied program organization.
  • F. Output Cases: Humaneval+ outputs may differ in function decomposition, variable naming, and other implementation details while solving the same prompt.The cited examples include a Sorting & Scanning method.
  • F. Output Cases: Codeforces examples provide two ResRL-trained Qwen3-4B rollouts generated with think decoding on the test set.The figures document Rollout 1 and Rollout 2 for the Codeforces test set.
Loading 2605.00380v2…