PaperlayerView exact revision on arXiv

Source-linked AI summary

Rethinking RL for LLM Reasoning: It's Sparse Policy Selection, Not Capability Learning

Ömer Faruk Akgül, Rajgopal Kannan, Willie Neiswanger, Viktor Prasanna

arXiv:2605.06241v2cs.CL

TL;DR

The paper asks whether RL is necessary for improving LLM reasoning when the base model may already contain correct solutions. Through token-level analysis, it identifies sparse, entropy-gated policy corrections and implements them in REASONMAXXER, which matches or exceeds full RL at roughly three orders of magnitude lower training cost.

  • Problem

    Prior evidence suggests RL redistributes probability toward solutions already present in base-model sampling, leaving open whether the RL optimization loop itself is necessary.

  • Method

    The paper performs token-level analysis across model families and RL algorithms, then trains REASONMAXXER with entropy-gated contrastive loss on base-model rollouts without RL or online generation.

  • Results

    Across model families and scales, REASONMAXXER matches or exceeds full-RL performance while using tens of problems, hundreds of rollouts, minutes of single-GPU training, and roughly three orders of magnitude lower cost.

  • Takeaways & Limitations

    The results reframe reasoning improvement as sparse policy selection: committing to the right branch at a few critical moments rather than acquiring new capabilities through expensive exploration.

  • Takeaways & Limitations

    The method’s problem-selection filter assumes a mixed-success regime, because all-success or all-failure problems provide no two-sided contrastive signal.

Abstract

from arXiv · show

Reinforcement learning has become the standard for improving reasoning in large language models, yet evidence increasingly suggests that RL does not teach new strategies; it redistributes probability mass over solutions the base model already contains. In this work, we ask: if RL merely steers the model toward paths it already knows, is the RL optimization loop itself necessary? Through token-level analysis across multiple model families and RL algorithms, we find that RL's beneficial footprint is a sparse, predictable correction concentrated at high-entropy decision points where the model is uncertain which branch to take. Only 1--3\% of token positions are affected, the promoted token always lies within the base model's top-5 alternatives, and targeted corrections at those few positions causally recover a large fraction of RL's accuracy gain, while random corrections fail. The base model's own entropy identifies these positions without any RL-trained model, and the entire correction is low-dimensional, representable in a tiny fraction of model parameters. These findings reframe reasoning improvement as sparse policy selection, not capability acquisition. We translate this insight into ReasonMaxxer, a minimal RL-free method that applies contrastive loss only at entropy-gated decision points, using a few hundred base-model rollouts and no online generation. Across three model families, six scales, and six math reasoning benchmarks, ReasonMaxxer matches or exceeds full RL performance while requiring only tens of problems and minutes of single-GPU training, a reduction in training cost of roughly three orders of magnitude.

1 Introduction

The paper asks whether RL’s optimization loop is necessary for reasoning improvement and finds that RL mainly makes sparse, targeted corrections to choices already available in the base model. It introduces an RL-free method that reproduces these gains with far less compute and data.

  • Motivation and contribution: The paper addresses whether the RL optimization loop is necessary after prior work identified structure in RL’s reasoning improvements without eliminating RL.This reframes the central gap as a question about whether RL itself is required.
  • Results: REASONMAXXER matches or exceeds RL-trained models on math reasoning benchmarks while using orders-of-magnitude less compute and data.The method uses tens of problems, hundreds of rollouts, and minutes of single-GPU training, reducing training cost by roughly three orders of magnitude.
  • Motivation and contribution: RL’s beneficial effect is a sparse, entropy-localized reranking of tokens the base model already favors.The analysis spans multiple model families and RL algorithms and uses oracle interventions with random controls to establish causal relevance.
  • Method: REASONMAXXER applies contrastive fine-tuning only at entropy-gated decision points using base-model rollouts, while anchoring other tokens to the base distribution.It requires no RL, online generation, or large-scale compute.
  • Implication: The findings suggest that heavy RL is not a prerequisite because the relevant problem is sparse policy selection rather than capability acquisition.The paper frames the improvement as committing to the right branch at a handful of uncertainty points.

2 Background and Experimental Setup

The paper reviews RLVR baselines for mathematical reasoning and defines token-level entropy as the basis for identifying uncertain decision points.

  • 2.1 Reinforcement Learning with Verifiable Rewards: RLVR generates rollouts and assigns each a binary reward based on whether its answer matches the ground truth.
  • 2.1 Reinforcement Learning with Verifiable Rewards: GRPO computes per-rollout advantages through group normalization before updating the policy with a clipped surrogate objective.
  • 2.1 Reinforcement Learning with Verifiable Rewards: GRPO, PPO, and related methods apply their objective uniformly across token positions, despite the paper’s later evidence that useful signal is sparse.
  • 2.2 Token-Level Entropy and Decision Points: Token-level entropy measures how broadly the autoregressive model distributes probability across vocabulary continuations at each position.
  • 2.2 Token-Level Entropy and Decision Points: Positions with entropy above threshold τ are called decision points, where the model is uncertain about which reasoning pathway to follow.
  • 2.3 Models and Baselines: The evaluated baselines are publicly available checkpoints trained with verifiable outcome rewards on mathematical reasoning problems.

3 What RL Actually Changes: Sparse Corrections at Decision Points

The analysis asks where RL changes token choices, whether those changes cause accuracy gains, and whether base-model signals can locate them. It finds sparse, conservative corrections concentrated at high-entropy branch points, with entropy providing a teacher-free localization signal.

  • 3.1 Disagreement Is Rare, Conservative, and Concentrated at Decision Points: The study compares base and RL-trained outputs to determine how often RL disagrees, whether disagreements cause gains, and whether the base model can locate them.
  • 3.1 Disagreement Is Rare, Conservative, and Concentrated at Decision Points: Only 1.0–4.1% of token positions are reranked, with zero shifted positions and teacher-preferred tokens averaging ranks 2.14–2.39 under the base model.
  • 3.1 Disagreement Is Rare, Conservative, and Concentrated at Decision Points: Reranked positions have 5–12× higher base-model entropy than unchanged positions, identifying uncertainty as a predictor of RL edits.
  • 3.2 Correcting Only the Disagreements Recovers RL Performance: Oracle replacement of disagreed tokens reproduces the teacher’s pass@1 exactly on every pair, whereas random top-20 substitutions perform no better than the base model and often worse.
  • 3.2 Correcting Only the Disagreements Recovers RL Performance: The oracle touches only 1.0–4.1% of tokens, indicating that RL’s accuracy advantage is concentrated in precise choices at a few reasoning branch points.
  • 3.3 Entropy-Gated Localization: Entropy-gated correction with τ = 1.2 matches the teacher on the 7B GRPO pair, closely approaches it on PPO, and improves over the base on other pairs while touching 1.2–8.3% of tokens.

4 The Correction Is Low-Dimensional

The paper tests whether RL’s sparse token-level correction is also simple in parameter space. KL-LoRA distillation shows that a tiny adapter trained on 100 problems can reproduce the RL teacher’s accuracy.

  • 4.1 Distilling RL into a Low-Rank Adapter: The KL-LoRA diagnostic trains only adapter parameters to minimize token-level KL divergence between the adapter-augmented base model and the RL teacher.
  • 4.1 Distilling RL into a Low-Rank Adapter: The adapter is trained from cached teacher top-k logits on only 100 randomly chosen problems, testing whether RL’s distributional change is low-dimensional.
  • 4.2 A Small Adapter Captures RL’s Full Correction: A rank-32 LoRA adapter on QKVO attention projections matches the RL teacher’s accuracy on MATH-500 and GSM8K across four model pairs.
  • 4.2 A Small Adapter Captures RL’s Full Correction: The adapter modifies only 0.27–0.49% of base-model parameters while reproducing the teacher’s benchmark accuracy.
  • From Representability to Learnability: The findings indicate that RL’s correction is representable in a tiny parameter budget and may be learnable without RL’s stochastic search.

5 REASONMAXXER – Entropy-Gated Contrastive Fine-Tuning

REASONMAXXER is an RL-free procedure that uses base-model rollouts and entropy to identify decision points, then applies contrastive updates there while preserving behavior elsewhere. It targets mixed-success problems and uses a low-rank adapter to implement this selective correction.

  • Problem Selection: REASONMAXXER retains only problems where some base-model rollouts are correct and others are incorrect, supplying both positive and negative learning signals.Problems with uniformly successful or uniformly failed rollouts lack one side of the contrastive signal.
  • Decision-Point Identification: Decision points are token positions whose frozen-base-model entropy exceeds a model-family-specific threshold, selecting roughly the highest-entropy few percent.The decision set is D = {t : Ht > τ}.
  • Decision-Point Identification: Entropy replaces an RL teacher’s disagreement signal because RL–base disagreements concentrate at high-entropy positions, enabling teacher-free selection from the base model alone.The selection uses no external supervision beyond the generated rollouts.
  • Contrastive Fine-Tuning: At decision points, advantage-weighted cross-entropy increases observed-token likelihood for correct rollouts and decreases it for incorrect rollouts.The method therefore reinforces token choices associated with correct final answers and penalizes choices associated with incorrect ones.
  • Base Anchoring: Outside the decision set, a KL anchor preserves the frozen base model’s behavior and limits overfitting in the small training set.The total objective combines decision-point loss with the anchored term as L = Ldec + λLanchor.
  • Implementation: A frozen base model with a trainable LoRA adapter implements the loss, reflecting the low-dimensionality of RL’s distributional change.A rank-32 adapter contains well under one percent of the model’s parameters.

6 Experiments

Experiments evaluate ReasonMaxxer across benchmarks, model families, baselines, costs, and design choices. It matches or exceeds RL performance while using substantially less compute and data, with robust entropy-threshold sensitivity.

  • Experimental setup: ReasonMaxxer is evaluated on six mathematical reasoning benchmarks against RL-trained models spanning three model families and multiple algorithms.AMC 2023 and AIME 2024 use avg@8; the other benchmarks use standard pass@1.
  • Results: ReasonMaxxer matches or exceeds RL baselines across model families while reducing training cost by two to three orders of magnitude.On Qwen2.5-1.5B, it scores 50.2% on MATH-500 versus 49.6% and 43.6%, at $4 versus $200 and $1200.
  • Results: ReasonMaxxer also matches or exceeds many baselines that use distillation, model-based verification, multi-domain data, or iterative RL.These comparisons extend beyond purely outcome-only RL settings.
  • Efficiency and scalability: ReasonMaxxer trains on 50 problems and completes in single-digit GPU-hours, whereas RL baselines use hundreds to tens of thousands of GPU-hours and far more data.Average training cost is less than $10 for ReasonMaxxer versus $100 to $100,000 for RL baselines.
  • Ablations: Performance remains robust across entropy thresholds from 1.0 to 2.2, with optimal MATH-500 and GSM8K scores at τ = 1.4.A second peak near τ = 1.8 aligns with RL’s observed intervention rate, so precise sparsity replication is unnecessary.
  • Ablations: Positive-only training improves over the base model but remains below full ReasonMaxxer and RL, showing the value of the two-sided contrastive loss.On Qwen2.5-1.5B, MATH-500 rises from 0.298 to 0.398 with positive-only training, versus 0.496 for RL and 0.502 for full ReasonMaxxer.

7 Related Work

Related work questions whether RL expands reasoning capabilities and identifies sparse, high-entropy intervention sites. Existing alternatives improve efficiency or avoid RL, while this paper positions its method within that landscape.

  • RL and reasoning capabilities: Prior analyses suggest RL-trained reasoning paths remain within the base model’s sampling distribution and that binary-reward optimization favors correct-answer probability.These findings motivate examining whether RL refines rather than expands existing capabilities.
  • Entropy and decision points: Studies identify high-entropy forking tokens, entropy minimization, and sparse attention heads as mechanisms associated with RL’s reasoning gains.Restricting GRPO updates to forking tokens reportedly matches training on all tokens.
  • RL post-training baselines: The paper compares against RL-trained baselines spanning GRPO, PPO with GAE, and other training strategies.SimpleRL-Zoo and Open-Reasoner-Zero provide representative publicly available checkpoints.
  • RL-free alternatives: RL-free alternatives include self-training, rejection-sampling fine-tuning, inference-time search, and sequence-level preference optimization.These approaches differ in whether they modify the policy and whether training is token- or sequence-level.
  • Efficiency: Efficiency work includes parameter-efficient adaptation methods such as LoRA and QLoRA, alongside reasoning-specific approaches.The paper situates ReasonMaxxer among methods seeking cheaper reasoning post-training.

8 Conclusion

The paper concludes that RL’s useful effect on mathematical reasoning is a sparse, predictable, low-dimensional correction rather than acquisition of new reasoning paths. ReasonMaxxer obtains this correction without RL, suggesting substantially cheaper post-training methods.

  • Conclusion: Token-level analysis and ReasonMaxxer support viewing reasoning improvement as sparse policy selection: the model commits to existing reasoning paths at critical moments.The conclusion states that the RL optimization loop can discover this correction but is not required to obtain it.
  • Implications: The findings suggest that heavy RL infrastructure may be disproportionate to the complexity of the correction being learned.The paper presents this as motivation for more efficient post-training methods.

A Detailed KL-LoRA Compression Ablations

The compression ablations show that RL’s correction can be reproduced with very small adapters, especially when targeting the output projection. On Qwen2.5-1.5B, rank-8 adapters already match or nearly match the RL teacher.

  • Adapter rank and module ablation: Even a rank-8 WO adapter approaches the full rank-32 QKVO adapter, concentrating RL’s correction in the output layer.The comparison is reported as an adapter rank and module ablation.
  • Adapter rank and module ablation: Rank-8 WQKV O adapters match the RL teacher, demonstrating that aggressive parameter compression preserves the correction.The study varies adapter rank and targeted attention modules on Qwen2.5-1.5B.
  • Adapter rank and module ablation: A rank-8 WO-only adapter with 688 K parameters trails the RL teacher by only 1 point on MATH-500.The result indicates that the output projection alone captures nearly all of the useful correction.

B Implementation Details

The paper provides complete training and architectural specifications for both KL-LoRA distillation and REASONMAXXER. These details support reproducibility of the Section 4 and Section 5 experiments.

  • KL-LoRA distillation: The appendix supplies full training details for the KL-LoRA distillation experiments in Section 4.The stated coverage includes training and architectural details.
  • KL-LoRA distillation: The appendix also supplies full architectural details for the KL-LoRA experiments.These details accompany the reported Section 4 experiments.
  • REASONMAXXER: REASONMAXXER’s training and architectural details are provided for the Section 5 method.The implementation documentation covers both named experimental components.

KL-LoRA Distillation

KL-LoRA distillation trains LoRA adapters against cached teacher logits, while REASONMAXXER applies masked losses to generated decision points and other valid prediction tokens. The experiments use standardized prompts and family-specific formatting across model comparisons.

  • KL-LoRA distillation: KL-LoRA uses manually implemented KL divergence over the teacher’s top-64 logits.The teacher is a SimpleRL-Zoo GRPO checkpoint whose cached rollouts use temperature 0.6 and top-p 0.95 with seed 44.
  • KL-LoRA distillation: The KL-LoRA student trains for three epochs with batch size 2, gradient accumulation 8, AdamW, learning rate 10^-4, and weight decay 10^-2.The schedule uses a 10% warmup ratio.
  • REASONMAXXER: REASONMAXXER trains with AdamW, one epoch, batch size 1, and gradient accumulation over 8 steps.The optimizer uses default betas, epsilon 10^-8, and linear warmup followed by linear decay.
  • REASONMAXXER: The decision loss averages over generated completion decision points, while the KL anchor averages over all other valid prediction tokens.Prompt tokens and padding positions are excluded from decision-point definitions.
  • Prompt styles: Within each model family, identical prompt templates and answer-extraction rules are applied to the base model, RL baselines, and REASONMAXXER adapter.Qwen2.5 and Qwen3 use raw completion prompts; Mistral uses instruction-answer formatting; DeepSeek-R1-Distill uses its native chat template.

D Cost Estimation Details

The cost analysis combines directly reported training measurements with estimates derived from official scripts, documented hyperparameters, scaling assumptions, or comparison tables. Costs are converted using RunPod on-demand rates and rounded to the nearest dollar.

  • Cost estimation methodology: Reported baseline costs use published wall-clock time and hardware whenever those details are available.Otherwise, the paper infers GPU-hours from official training scripts and documented hyperparameters before converting them to cost.
  • Cost estimation methodology: REASONMAXXER costs are directly measured on 4 NVIDIA RTX Pro 6000 Blackwell GPUs, including rollouts, entropy scoring, training, and checkpoint selection.The reported GPU-hours use the RunPod on-demand rate for that GPU type.
  • Baseline-specific estimates: SimpleRL-Zoo estimates the Qwen2.5-1.5B GPU-hours by scaling 7B per-step generation time while holding the training configuration constant.The Mistral-7B run uses the Qwen2.5-7B hardware and step count.
  • Baseline-specific estimates: Open-Reasoner-Zero GPU-hours are estimated from public PPO recipes, documented hardware, prompt and rollout counts, inferred step counts, and calibrated per-step time.Calibration uses SimpleRL-Zoo figures for a comparable model size.
  • Baseline-specific estimates: PRIME-Zero cost is computed by multiplying published per-step wall-clock time by total RL steps, while its training data size is approximately 13K prompts.The resulting GPU-hours are converted using the reported GPU type’s RunPod rate.
  • Reporting conventions: All Table 3 costs are rounded to the nearest US dollar, and italicization marks estimates without directly published wall-clock times.The costing procedure uses RunPod on-demand pricing as of Apr 28, 2026.
Loading 2605.06241v2…