Source-linked AI summary
Just Enough Thinking: Efficient Reasoning with Adaptive Length Penalties Reinforcement Learning
Violet Xiang, Chase Blagden, Rafael Rafailov, Nathan Lile, Sang Truong, Chelsea Finn, Nick Haber
TL;DR
Long reasoning traces can improve performance but often waste inference compute on easy problems, while existing length-control methods require curation or manual budgets or treat problems uniformly. ALP uses online solve rates to tailor RL length penalties to prompt difficulty. On mathematical reasoning benchmarks, it reduces token use by approximately 50% while maintaining performance, and shifts computation toward harder problems.
Problem
Method
ALP adds an RL length penalty whose weight is inversely proportional to each prompt’s online solve rate, estimated from multiple rollouts.
Source: §3 · p. 3Results
Against the original DeepScaleR-1.5B, ALP reduces token usage by approximately 50% on all datasets while maintaining performance.
Source: §5.1 · p. 4Takeaways & Limitations
ALP allocates fewer tokens to easy problems and deploys saved budget on challenging ones, using 5.35× more tokens on hard versus easy problems.
Source: §1 · p. 2The evaluation focuses exclusively on mathematical reasoning, so whether ALP’s benefits extend to other domains remains unclear.
Source: §7 · p. 10
Abstract
from arXiv · showhide
Large reasoning models (LRMs) achieve higher performance on challenging reasoning tasks by generating more tokens at inference time, but this verbosity often wastes computation on easy problems. Existing solutions, including supervised finetuning on shorter traces, user-controlled budgets, or RL with uniform penalties, either require data curation, manual configuration, or treat all problems alike regardless of difficulty. We introduce Adaptive Length Penalty (ALP), a reinforcement learning objective tailoring generation length to per-prompt solve rate. During training, ALP monitors each prompt's online solve rate through multiple rollouts and adds a differentiable penalty whose magnitude scales inversely with that rate, so confident (easy) prompts incur a high cost for extra tokens while hard prompts remain unhindered. Posttraining DeepScaleR-1.5B with ALP cuts average token usage by 50\% without significantly dropping performance. Relative to fixed-budget and uniform penalty baselines, ALP redistributes its reduced budget more intelligently by cutting compute on easy prompts and reallocating saved tokens to difficult ones, delivering higher accuracy on the hardest problems with higher cost.
1 Introduction
Longer reasoning traces can improve performance on complex tasks but waste compute on simple prompts, while current length-control approaches either require curated data or manual budgets or apply uniform penalties. Adaptive Length Penalty (ALP) uses online solve rates to tailor token penalties to prompt difficulty and redistribute computation.
1 Introduction
- Longer inference-time chains can improve complex-task performance, but verbose traces consume compute and memory and increase latency, especially for simple queries.The passage gives examples of models generating over 10,000 tokens for “What is 2 + 3?” [Wang et al., 2025].Source: §1 · p. 1
- Existing approaches use curated short-trace data, user-configured budgets, or uniform RL penalties, each limiting difficulty-sensitive allocation.These approaches respectively require data curation, manual configuration, or treat problems alike regardless of inherent difficulty.Source: §1 · p. 1
- ALP reduces average token usage by over 50% while maintaining accuracy, and uses 21% of its budget on the easiest half of problems versus 50% for fixed-length baselines.The saved budget is deployed on challenging problems, with 5.35× more tokens used on hard versus easy problems.Source: §1 · p. 2
- ALP scales length penalties inversely with online difficulty estimates, teaching models to adapt reasoning length without manual configuration or inference-time overhead.It reports superior efficiency-performance trade-offs across multiple benchmarks while maintaining accuracy.Source: §1 · p. 2
2 Related Work
Prior work increases inference-time compute to improve performance and offers several ways to control reasoning length. These methods do not adapt length to each problem’s intrinsic difficulty, a gap ALP addresses with difficulty-conditioned penalties.
2 Related Work
- Inference-time scaling increases the number of tokens used by LLMs and can improve performance through search algorithms [Wu et al., 2024; Yao et al., 2023; Xie et al., 2024].Scaling inference-time compute can be more effective than scaling model size [Snell et al., 2024].Source: §2 · p. 2
- Prior length-control methods include reward discounting, penalties on correct solutions, early-answer tokens, user-specified budgets, and progressively stricter length constraints.The cited approaches include [Xiang et al., 2025], [Team et al., 2025; Arora and Zanette, 2025], [Muennighoff et al., 2025], [Aggarwal and Welleck, 2025], and [Hou et al., 2025].Source: §2 · p. 2
- Existing length-control approaches do not adapt generation length to problem difficulty, risking over-reasoning on simple tasks or under-reasoning on complex ones.ALP addresses this with an instance-specific, difficulty-conditioned penalty rather than manual configuration or blanket constraints.Source: §2 · pp. 2–3
- ALP’s training loop samples multiple rollouts per prompt, estimates its solve rate, computes rewards, and updates the policy.It is presented as addressing the difficulty-adaptation gap in existing length-control methods.Source: §2 · p. 3
3 Method
ALP adds a length penalty to the RL reward, scaling its weight by a prompt’s online solve rate so that easy prompts face stronger penalties and difficult prompts retain room for longer reasoning. The solve-rate estimate uses rollouts already sampled by compatible RL algorithms.
3 Method
- ALP is an additional reward term in standard reinforcement learning, with K independent rollouts sampled for each prompt.The prompt’s solve rate is estimated from these rollouts before the composite reward is evaluated.Source: §3 · p. 3
- ALP augments the usual accuracy reward with a length penalty weighted inversely by the prompt’s online solve rate.High solve rates mark easier prompts and incur stronger penalties; low rates mark harder prompts and receive smaller penalties.Source: §3 · p. 3
- For algorithms such as GRPO [Shao et al., 2024] and Reinforce++ [Ahmadian et al., 2024], ALP obtains solve rates from the multiple samples already used for advantage estimation, without additional computational cost.The authors advise using K >> 1 for an accurate solve-rate estimate.Source: §3 · p. 3
4 Experimental Setup
The experiments train ALP on mathematical reasoning using DeepScaleR-1.5B, then evaluate performance and token use across math benchmarks and length-control baselines. Comparisons use public checkpoints because matched training runs were constrained by resources.
4 Experimental Setup
- DeepScaleR-1.5B was trained on DeepScaleR, a filtered compilation of AIME, pre-2023 AMC, Omni-Math, and Still problems.The model was trained for 100 steps with batch size 512 using GRPO and β = 1e −7 for the comparisons in Sections 5.1–5.4.Source: §4 · pp. 3–4
- Evaluation uses Pass@1 on AIME 2024–2025, MATH-500 [Hendrycks et al., 2021], and OlympiadBench [He et al., 2024] at inference budgets of 512, 1024, 2048, and 4096 tokens.The experiments sample 64 answers per AIME prompt and 16 per MATH-500 and OlympiadBench prompt.Source: §4 · p. 4
- ALP is compared with public checkpoints rather than head-to-head retrained methods because of resource constraints.The base models are the same size but were trained on somewhat different datasets, hyperparameters, and numbers of steps; other baselines include ThinkPrune-2K and R1-Alpha.Source: §4 · p. 4
- The comparison includes L1-Exact, which follows a user-specified precise length, and L1-Max, which keeps usage within a maximum allowance.The evaluation appends instructions specifying the maximum or exact token count to those models’ prompts.Source: §4 · p. 4
5 Results and analysis
The section evaluates ALP’s efficiency and performance, how it allocates computation by difficulty, and how that compression varies across settings and reasoning behaviors.
5.1 Pass@1-Efficiency Tradeoffs
- ≈50% fewer tokens across datasets with maintained performance; at 1024 tokens, ALP achieves 40% higher Pass@1 than the base model on easier math tasks.On MATH-500, ALP achieves comparable Pass@1 to all methods while using more than 50% fewer tokens except L1-Max.Source: §5.1 · pp. 4–5, Figure 1 · p. 5
5.2 How Models Achieve Efficiency: Pareto Analysis of Adaptive Computation
- ALP uses 21% of tokens on the easiest half of problems, reaches a 5.35× hard/easy adaptation ratio, and has the highest efficiency score, 0.68.Comparisons use cumulative allocation curves and define adaptation by tokens allocated to harder versus easier problems.Source: §5.2 · pp. 5–6, Figure 2 · p. 6
5.3 Robustness to Unknown Difficulty Distributions
- With AIME content rising to 60%, ALP’s accuracy falls from 0.89 to 0.52 as average tokens rise from approximately 500 to 2200.In the N=500 gradual-shift scenario, ALP maintains accuracy above 75% through 12% AIME content.Source: §5.3 · pp. 6–7, Figure 3 · p. 7
5.4 How Models Internalize Problem Difficulty
- ALP’s token use rises with difficulty across all three datasets, from approximately 500 tokens on easiest MATH-500 problems to nearly 3000 on hardest ones.Difficulty is defined as one minus empirical solve rate; the MATH-500 increase is 6×, while L1-Exact remains near 2900–3000 tokens across difficulty levels.Source: §5.4 · pp. 7–8, Figure 4 · p. 8
5.5 Model and Hyperparameter Sensitivity
- Across two 1.5B models, β=10^-7 cuts tokens faster with modest Pass@1 loss, while a 4K context shortens traces with negligible accuracy loss.R1-Distill-Qwen uses slightly more tokens on the hardest AIME examples while matching DeepScaleR on OlympiadBench.Source: §5.5 · pp. 8–9, Figure 5 · p. 9
5.6 How Length Reduction Changes Reasoning Behavior
- Repetitions fell from 269.3 to 125.9 and exploration from 82.9 to 43.2; planning retained 46% of baseline frequency, while backtracking and conclusions also declined.The keyword analysis compared base and ALP-trained responses across evaluation datasets.Source: §5.6 · pp. 9–10, Figure 6 · p. 10
6 Discussion
ALP uses online solve rates to adapt reasoning length, compressing redundant work while maintaining structured problem solving. The discussion reports robustness across standard curricula and competition-level challenges.
6 Discussion
- ALP uses online difficulty estimates from solve rates to adapt reasoning length without manual configuration.The discussion describes stronger penalties for excess tokens on easy problems and less penalty on difficult ones.Source: §6 · p. 10
- ALP compresses redundant exploration and repetitive verification while maintaining structured problem-solving approaches.The discussion characterizes this selective compression as direct reasoning on familiar problems and thorough reasoning on challenging ones.Source: §6 · p. 10
- ALP’s adaptation is reported as robust across standard curricula and competition-level challenges.The discussion says this robustness suggests difficulty calibration rather than memorization of specific patterns.Source: §6 · p. 10
7 Limitations
The evaluation focuses exclusively on mathematical reasoning, so whether ALP benefits extend to other domains remains unclear. ALP reallocates existing capabilities rather than necessarily improving fundamental reasoning ability.
7 Limitations
- Whether ALP’s benefits extend beyond mathematical reasoning remains unclear because evaluation focuses exclusively on that domain.The authors identify this as the most critical constraint on their conclusions.Source: §7 · p. 10
- ALP optimizes allocation of existing capabilities rather than enhancing fundamental reasoning abilities.The authors say models may become more efficient reasoners without necessarily becoming better ones.Source: §7 · p. 10
- The relationship between efficiency and capability improvements remains an open question in the paper.The authors ask whether these improvements are inherently coupled or can be pursued independently.Source: §7 · p. 10
A.1 Implementation details
The implementation details state that evaluation prompts are provided and that models receive the same mathematical problem with model-specific formatting.
A.1 Implementation details
- The evaluation prompts used for each model are provided.The passage does not specify the prompts themselves.Source: §A.1 · p. 13
- All evaluated models receive the same mathematical problem as input.The prompt formatting varies by model.Source: §A.1 · p. 13
- Prompt formatting is model-specific despite the shared mathematical input.The passage gives no further implementation specifications.Source: §A.1 · p. 13
A.1.1 ThinkPrune Models
The ThinkPrune model prompts specify a user-assistant problem-solving exchange, with reasoning and answer marked separately. They also instruct step-by-step reasoning and a boxed final answer.
A.1.1 ThinkPrune Models
- The listed ThinkPrune prompt applies to DeepSeek-R1-Distill-Qwen-1.5B-thinkprune-iter2k and DeepScaleR-1.5B-Preview-thinkprune-iter2k.The passage identifies these two models as using the prompt.Source: §A.1.1 · p. 13
- The system prompt encloses the assistant’s reasoning in <think> tags and its answer in <answer> tags.It frames the exchange as a user asking a question and an assistant solving it.Source: §A.1.1 · p. 13
- The user prompt asks the model to think step by step and place its final answer within \boxed{}.Source: §A.1.1 · p. 13
A.1.2 L1-Max Model
The appendix specifies prompt and budget handling for model variants and describes keyword-based coding of reasoning behaviors.
A.1.2 L1-Max Model
- L1-Max receives a user-specified maximum token budget N, which its prompt states explicitly.Source: §A.1.2 · p. 13
A.1.3 L1-Exact Model
- L1-Exact receives a user-specified exact token budget N, also stated explicitly in its prompt.Source: §A.1.3 · p. 13
A.1.4 Standard Models
- The prompt asks users to think step by step and put the final answer within boxed form; specified budgets of 512, 1024, 2048, or 4096 tokens truncate generation, and Table 2 lists training hyperparameters.Source: §A.1.4 · p. 13, Table 2 · p. 13
A.2 Keywords for Behavior Analysis
- Keyword patterns are used to identify distinct reasoning behaviors in model responses.Source: §A.2 · p. 14
A.2.1 Reasoning Stages
- Reasoning-stage patterns cover task framing, considering alternatives, checking work, and stating conclusions.Source: §A.2.1 · p. 14
A.2.2 Backtracking Patterns
- Backtracking patterns include corrections, reconsideration, and restarting reasoning.Source: §A.2.2 · p. 14
A.2.3 Planning
- Planning patterns include ordered-step phrases; analysis groups backtracking subcategories and counts repeated phrases of five or more words within a response.Source: §A.2.3 · p. 14