Source-linked AI summary
DLER: Doing Length pEnalty Right - Incentivizing More Intelligence per Token via Reinforcement Learning
Shih-Yang Liu, Xin Dong, Ximing Lu, Shizhe Diao, Mingjie Liu, Min-Hung Chen, Hongxu Yin, Yu-Chiang Frank Wang, Kwang-Ting Cheng, Yejin Choi, Jan Kautz, Pavlo Molchanov
TL;DR
DLER addresses inefficiently long reasoning outputs by revisiting reinforcement learning with a simple truncation penalty and improved optimization. It restores or surpasses accuracy while reducing response length by more than 70%, with further gains from difficulty-aware truncation and selective weight merging.
Problem
Reasoning models often generate unnecessarily long outputs, making accuracy relative to response length an open problem.
Method
DLER combines batch-wise reward normalization, higher clipping, dynamic sampling, and a simple truncation length penalty, with difficulty-aware truncation and update-selective merging extensions.
Results
DLER achieves state-of-the-art accuracy-to-token efficiency, fully restoring or surpassing accuracy while cutting response length by more than 70%.
Takeaways & Limitations
Accuracy–efficiency improvements depend more on RL optimization than sophisticated length-penalty design, while difficulty-aware truncation provides additional response-length reductions.
Takeaways & Limitations
The proposed explanation for entropy collapse remains a suspicion that clipping in the importance-sampling ratio may zero out some token gradients.
Abstract
from arXiv · showhide
Reasoning language models such as OpenAI-o1, DeepSeek-R1, and Qwen achieve strong performance via extended chains of thought but often generate unnecessarily long outputs. Maximizing intelligence per token--accuracy relative to response length--remains an open problem. We revisit reinforcement learning (RL) with the simplest length penalty--truncation--and show that accuracy degradation arises not from the lack of sophisticated penalties but from inadequate RL optimization. We identify three key challenges: (i) large bias in advantage estimation, (ii) entropy collapse, and (iii) sparse reward signal. We address them with Doing Length pEnalty Right (DLER), a training recipe combining batch-wise reward normalization, higher clipping, dynamic sampling, and a simple truncation length penalty. DLER achieves state-of-the-art accuracy--efficiency trade-offs, cutting output length by over 70 percent while surpassing all previous baseline accuracy. It also improves test-time scaling: compared to DeepSeek-R1-7B, DLER-7B generates multiple concise responses in parallel with 28 percent higher accuracy and lower latency. We further introduce Difficulty-Aware DLER, which adaptively tightens truncation on easier questions for additional efficiency gains. We also propose an update-selective merging method that preserves baseline accuracy while retaining the concise reasoning ability of the DLER model, which is useful for scenarios where RL training data is scarce.
NVIDIA
Reasoning models can achieve strong performance with long chains of thought, but maximizing accuracy relative to response length remains unresolved.
- Long chains of thought can produce strong performance while generating unnecessarily long outputs.
- Reasoning-token efficiency is framed as maximizing intelligence per token, defined here by accuracy relative to response length.
1. Introduction
DLER argues that efficient reasoning depends more on RL optimization than on sophisticated length penalties. It reports large length reductions, stronger parallel test-time scaling, adaptive truncation, and weight merging for data-scarce settings.
- Over 70% shorter responses with fully recovered accuracy establish DLER’s state-of-the-art accuracy-to-token efficiency.
- DLER’s optimization recipe makes simple truncation competitive, indicating optimization matters more than sophisticated penalty design.
- 27% higher AIME-24 accuracy within the same wall-clock thinking time shows that efficient reasoning can improve parallel test-time scaling.
- DA-DLER further reduces response length by 15% on DeepSeek-R1-1.5B and 11% on 7B by adapting truncation to question difficulty.
- Update-selective weight merging recovers accuracy while reducing average output length by 47% when high-quality proprietary training data is unavailable.
2. Preliminary
The preliminary formulation describes GRPO as a critic-free RL method using group-relative advantages, clipped policy ratios, and rewards that combine correctness with length penalties.
- GRPO estimates each response’s advantage by normalizing rewards across a sampled response group, avoiding a separate critic model.
- The optimization objective clips the policy importance-sampling ratio using threshold ε, with KL loss omitted for simplicity.
- Length-penalty RL defines total reward as correctness reward plus a length term, with truncation assigning zero reward beyond a fixed limit.
3. Re-examining the Simplest Length Penalty - Truncation
The paper argues that truncation itself is not responsible for accuracy loss; inadequate RL optimization creates biased advantages, entropy collapse, and sparse training signals. DLER addresses these issues with normalization, clipping, sampling, and truncation strategies.
- 3.1. More Aggressive Truncation Leads to Higher Reward Variance: Around 3% higher AIME-24 accuracy at similar token counts follows from replacing GRPO’s prompt-wise normalization with batch-wise normalization.Both methods shorten responses, but GRPO accuracy declines while batch-wise normalization begins recovering after about 100 steps.
- 3.1. More Aggressive Truncation Leads to Higher Reward Variance: Truncation increases reward variance, biasing GRPO advantage estimates; batch-wise normalization mitigates this instability.More aggressive truncation produces higher per-prompt variance, with measured values of 0.4, 0.32, 0.3, and 0.29 for lengths 4000, 8000, 12000, and 16000.
- 3.2. Entropy Collapse Limits Exploration of Reasoning Paths: Higher clipping retains low-probability, high-entropy transitional tokens, mitigating entropy collapse and preserving exploration of reasoning paths.These tokens also influence response length because their frequency generally increases with longer reasoning sequences.
- 3.3. Length Penalty Over-sparsify Training Signal: Dynamic sampling removes prompts with uniformly zero or positive rewards, balancing training signals and helping the model use the target length budget.Zero-reward prompts are often too hard or too long, while all-positive prompts are easier and can drive premature over-shortening.
- 3.4. Combining All Ingredients: Do Length pEnalty Right: Over 70% shorter outputs with recovered accuracy establish DLER’s state-of-the-art accuracy–token-efficiency trade-off.The recipe combines batch-wise reward normalization, higher clipping, dynamic sampling, and truncation.
- 3.5. Difficulty-Aware DLER: Difficulty-aware DLER shortens truncation targets for reliably solved questions while allowing more tokens for harder questions.The method estimates difficulty from response correctness ratios and assigns truncation lengths across difficulty levels.
4. Experiment
DLER consistently improves the accuracy–efficiency trade-off across model sizes and benchmarks, while difficulty-aware truncation and selective merging extend its practical benefits.
- 4.2. Main Results: 86.95 MATH, 34.38 AIME, and 48.31 Olympiad accuracy make DLER-R1-1.5B strongest across five benchmarks at 2466 average tokens.It surpasses Laser-DE by 1.68, 3.76, and 2.10 points on those benchmarks, respectively, while shortening responses substantially.
- 4.2. Main Results: 15% and 12% further reductions in average response length from DA-DLER preserve comparable accuracy for the 1.5B and 7B models.DA-DLER dynamically adapts truncation lengths according to question difficulty.
- 4.2. Main Results: 94.21 MATH, 55.62 AIME, and 84.41 AMC accuracy establish DLER-R1-7B as state of the art at 2405 average tokens.The output is 69% shorter than the original DeepSeek-R1-7B and 25% shorter than Laser-DE.
- 4.3. Performance Under Different Test-time Scaling Settings: DLER outperforms baselines across Pass@K settings on AIME-24 and Olympiad, with the largest margins under restrictive token cutoffs.The advantage persists as the number of test-time rollouts increases.
- 4.4. Test-time Scaling: 12.35 seconds versus 58.99 seconds reduces single-response inference time by 4.8× for DLER-R1-1.5B relative to DeepSeek-1.5B.To reach 80.00 accuracy, DLER requires 52.09 seconds with 128 rollouts versus 229.00 seconds for DeepSeek with 64 rollouts.
- 4.5. Different Length Penalties No Longer Push the Accuracy–Efficiency Frontier: Truncation remains competitive with complex penalties while requiring less training time because it terminates rollouts at the target cutoff.DLER-trained models define the accuracy/average-length Pareto frontiers across the reported tasks.
5. Related Work
Prior work improves reasoning efficiency through prompt engineering, supervised fine-tuning, and reinforcement learning, using different mechanisms to shorten or selectively suppress reasoning.
- Prompt Engineering: Prompt engineering can bypass explicit reasoning in low-budget settings by directly generating solutions after a dummy reasoning box.
- Supervised Fine-Tuning: Supervised fine-tuning distills concise traces, uses verification to suppress unnecessary self-reflection, and encourages step skipping.
- Reinforcement Learning: Reinforcement-learning approaches control reasoning through mode tokens or length-based reward shaping, including truncation extensions.
6. Conclusion
The paper concludes that optimization, rather than penalty complexity, drives efficient reasoning, with DLER, DA-DLER, and selective merging addressing accuracy and efficiency together.
- 6. Conclusion: DLER combines reward normalization, higher clipping, dynamic sampling, and truncation to restore or surpass accuracy while reducing response length by over 70%.
- 6. Conclusion: DA-DLER dynamically shortens truncation targets for easier questions to improve efficiency beyond fixed-length penalties.
- 6. Conclusion: Magnitude-selective weight merging addresses accuracy drops from applying DLER with public data to high-capacity models.
- 6. Conclusion: The findings suggest that optimization strategies matter more than complex penalty designs for accurate, efficient reasoning and better test-time scaling.
A. DLER achieves SOTA Accuracy/Length of CoT trade-off and enable better test-time scaling
DLER improves the accuracy-to-length trade-off by shortening chains of thought while maintaining accuracy, and its concise reasoning supports stronger parallel test-time scaling.
- SOTA Accuracy/Length Trade-off: Up to ∼70% shorter CoT with maintained accuracy gives DLER a state-of-the-art Accuracy/Length trade-off.
- Test-time Scaling: DLER-R1 models show superior test-time scaling curves to DeepSeek-R1 when parallel rollouts increase from 1 to 256.The figure attributes this advantage to improved concise reasoning ability and benchmarks overall latency with vLLM.
A.1 Assumptions and Settings
The section analyzes a finite-sample advantage estimator under Gaussian reward perturbations and shows that its conditional expectation is biased and varies with perturbation scale.
- A.1 Assumptions and Settings: Rewards are modeled as r_i = θ + ϵ_i with independent ϵ_i∼𝒩(0, σ^2), for i=1,…,N.The analysis assumes a true baseline θ and observes N rewards for one prompt.
- A.1 Assumptions and Settings: For any finite N≥2, the advantage estimator A_i is biased under the stated reward model.The proof begins by establishing finite-sample bias and concludes that A_i is a biased estimator.
- A.1 Assumptions and Settings: Conditioning on ϵ_i leaves the other perturbations independently distributed, enabling analysis of the numerator and denominator separately.The proof uses zero mean, independence, conditional expectation, and a Taylor expansion of x^-1/2.
- A.1 Assumptions and Settings: The conditional bias is not constant and increases with the perturbation scale because σ and g(ϵ_i) are proportional to the bias.The derivation states that a larger σ′ implies larger conditional bias for the corresponding perturbation.
- A.1 Assumptions and Settings: The estimator decomposition yields g(ϵ_i)≡N/(N−1) in the constant case, while the combined result retains conditional bias.The section presents separate numerator, conditional-expectation, and combined-result steps.
C. Hyperparameters Setting
The section provides the DLER veRL training configuration in a dedicated hyperparameter table.
- C. Hyperparameters Setting: Table 4 presents the DLER veRL training configuration and its associated hyperparameter settings.
D. Parallel Thinking Latency
The section reports average parallel inference latency per request for DeepSeek-R1-7B and DLER-R1-7B.
- D. Parallel Thinking Latency: Table 5 compares average parallel inference latency per request between DeepSeek-R1-7B and DLER-R1-7B.