Source-linked AI summary
L1: Controlling How Long A Reasoning Model Thinks With Reinforcement Learning
Pranjal Aggarwal, Sean Welleck
TL;DR
Reasoning models improve with longer thinking but lack controllable chain-of-thought lengths, hindering deliberate test-time compute allocation. The paper introduces LCPO, reinforcement learning that jointly optimizes correctness and prompt-specified length, and trains L1 and SRMs. L1 outperforms prior length-control methods and achieves strong short-reasoning performance, including surpassing GPT-4o at equal token budgets.
Problem
Reasoning models’ uncontrolled chain-of-thought lengths make it difficult to allocate test-time compute for a desired performance level.
Method
LCPO trains reasoning models with rewards for correct answers and adherence to target lengths specified in prompts.
Results
L1 outperforms prior length-control approaches, generalizes beyond mathematics, and becomes a strong short-reasoning model at matched token budgets.
Takeaways & Limitations
LCPO enables precise allocation of reasoning length and compute while preserving or improving performance across varied tasks and budgets.
Takeaways & Limitations
The models may not generalize to requested lengths longer than those used during training.
Abstract
from arXiv · showhide
Reasoning language models have shown an uncanny ability to improve performance at test-time by ``thinking longer''-that is, by generating longer chain-of-thought sequences and hence using more compute. However, the length of their chain-of-thought reasoning is not controllable, making it impossible to allocate test-time compute to achieve a desired level of performance. We introduce Length Controlled Policy Optimization (LCPO), a simple reinforcement learning method that optimizes for accuracy and adherence to user-specified length constraints. We use LCPO to train L1, a reasoning language model that produces outputs satisfying a length constraint given in its prompt. L1's length control allows for smoothly trading off computational cost and accuracy on a wide range of tasks, and outperforms the state-of-the-art S1 method for length control. Furthermore, we uncover an unexpected short chain-of-thought capability in models trained with LCPO. Specifically, using LCPO we derive Short Reasoning Models (SRMs), that exhibit similar reasoning patterns as full-length reasoning models, but can generate CoT lengths comparable to non-reasoning models. They demonstrate significant performance gains, for instance, our 1.5B L1 model surpasses GPT-4o at equal reasoning lengths. Overall, LCPO enables precise control over reasoning length, allowing for fine-grained allocation of test-time compute and accuracy. We release code and models at https://www.cmu-l3.github.io/l1
1 Introduction
Reasoning models can improve with longer chain-of-thought, but their uncontrolled lengths make compute allocation difficult. LCPO trains L1 to balance correctness with prompt-specified reasoning lengths.
- Uncontrolled reasoning lengths can waste compute on excessive sequences or stop too early on difficult problems.
- LCPO jointly optimizes final-answer correctness and adherence to a length constraint specified in the prompt.
- L1 supports exact-length and maximum-length control while smoothly trading token budgets against reasoning performance.
- LCPO-trained models generalize length control beyond mathematics to logical reasoning and general-knowledge benchmarks such as MMLU.
- Short Reasoning Models improve substantially at short traces, with the 1.5B L1 model surpassing GPT-4o at equal reasoning lengths.
2 Related Work
Prior work established test-time scaling and general output-length control, but reasoning-specific methods lacked precise, adaptive alignment with inference budgets. LCPO addresses this gap with reinforcement learning specialized for reasoning tasks.
- Longer reasoning chains and additional test-time computation generally improve performance on complex reasoning, mathematics, and code generation.
- Reasoning models such as O1- and R1-style systems lack precise dynamic control over chain-of-thought length, limiting efficiency gains.
- Existing length-control methods target architectural changes, training objectives, or instruction-following data, mostly outside reasoning-specific cost-quality trade-offs.
- Reasoning models can underthink or overthink, while shorter-chain approaches do not precisely match user-specified inference budgets.
- LCPO uses reinforcement learning to dynamically allocate inference compute under prompt-provided constraints and targets performance across varying budgets.
3 Method
LCPO conditions a reasoning model on a target token length and rewards both correctness and length adherence. L1-Exact enforces exact targets, while L1-Max softly enforces an upper limit.
- The training objective minimizes the difference between target and generated lengths while requiring a correct answer.
- LCPO updates a pretrained reasoning model with reinforcement learning using correctness and length-penalty rewards.
- Prompts are augmented with instructions such as “Think for ngold,i tokens,” where target lengths are sampled from a range.
- The reward coefficient α controls the trade-off between correctness and target-length adherence, producing the L1-Exact model.
- At inference, users control output length by appending a fixed target length, or set of target lengths, to each prompt.
- L1-Max uses a soft penalty for exceeding the target, preserving gradient propagation and favoring fewer tokens when correctness is maintained.
4 Experimental Setup
The experiments train on a 40K mathematics dataset and evaluate length control and accuracy across mathematical, logical, and general-knowledge benchmarks. Comparisons include original, distilled, shortened-context, and budget-forcing baselines.
- Models and Datasets: Training uses 40K mathematics question-answer pairs from AIME, AMC, Omni-Math, and STILL.
- Models and Datasets: Evaluation covers AIME 2025, MATH, AMC, Olympiad-Bench, GPQA, LSAT, and MMLU.
- Baselines: Baselines include DeepSeek-R1-1.5B, the unmodified DeepScaleR-24K, and the 4K-context DeepScaleR-4K model.
- Baselines: S1 is evaluated as a budget-forcing baseline that stops generation at a maximum budget and inserts “Final Answer.”
- Evaluation Protocol: Models are assessed by mean deviation from target length and problem-solving accuracy at target lengths of 512, 1024, 2048, and 3600 tokens.
- Implementation Details: GRPO training uses a 1e-6 learning rate, batch size 128, 4K training context, 8K evaluation context, and 700 steps.
5 Results and Analysis
Across benchmarks, L1 combines strong reasoning performance with precise length control, generalizes to out-of-domain tasks, and adapts its reasoning strategies to different token budgets. LCPO-trained models also perform strongly in short-CoT settings while retaining robustness across decoding parameters and model scales.
- Relative performance: L1 achieves over 100-150% relative and 20-25% absolute performance gains over S1 at 512 and 1024 token budgets.L1-Exact and L1-Max outperform S1 while maintaining length control by adapting reasoning chains instead of truncating mid-reasoning.
- Out-of-domain generalization: L1 generalizes length control to GPQA, LSAT, and MMLU, with performance scaling positively with token budget and matching DeepScaleR-4K at comparable budgets on GPQA and LSAT.MMLU shows less pronounced scaling, with R2 = 0.66, because knowledge-focused questions benefit less from extended reasoning.
- Length controllability: L1 follows requested lengths closely across 512, 1024, 2048, and 3600 token budgets, while L1-Max’s soft violation rates range from 0.3% to 2.3%.Mean length error is close to 3% on math datasets, though OOD datasets show higher errors of 20-40%.
- Short-CoT performance: At equal short generation lengths, L1 outperforms its non-reasoning counterpart by 5% on average and GPT-4o by 2% on average.These short reasoning models retain patterns similar to full reasoning models despite their shorter outputs.
- Learned reasoning patterns: Reasoning patterns shift with token budgets: self-correction and verification keywords appear about twice as often at 4096 tokens, while conclusion terms increase 2-10x.Exploration-related keywords generally decrease in relative frequency at higher token counts, with “Alternatively” as an exception.
- Robustness and scaling: The thinking-to-solution token ratio remains relatively stable across generation lengths, while L1 maintains length-control deviations of only 5.6-6.3% across temperatures from 0.0 to 1.0.Sequential scaling consistently outperforms parallel sampling at equivalent token budgets, and the same controllability trends extend to the 7B model.
6 Conclusion
The work introduces LCPO for adaptive control of reasoning-chain length, trains L1 to follow prompted constraints, and reports improved efficiency, generalization, and short-reasoning performance.
- LCPO enables adaptive control of reasoning-chain length through reinforcement learning.
- LCPO achieves over 100% relative and 20% absolute improvements in mathematical reasoning tasks over prior length-control approaches.
- L1 generalizes length-controlled capabilities beyond its training distribution to out-of-domain tasks.
- LCPO-trained models can become strong short reasoning models, outperforming larger frontier models such as GPT-4o at identical generation lengths.
7 Limitations
LCPO demonstrates strong length control on in-domain and out-of-domain tasks, but its models may not generalize to requested lengths longer than their training lengths.
- LCPO provides strong length control for both in-domain and out-of-domain tasks.
- Models may not generalize to requested reasoning lengths longer than those used during training.
- Training models to use more inference compute at test time than during training remains an important future direction.
- The models use length rewards on complete outputs, while alternative rewards targeting reasoning-token length remain unexplored.
A.1 Extended training further improves length constraint precision.
Additional reinforcement-learning training improves length precision, especially RMSE, although stricter adherence can slightly reduce performance at high token ranges.
- A.1 Extended training further improves length constraint precision.: L1-Exact+ is produced by further fine-tuning L1-Exact for 500 reinforcement-learning steps.
- A.1 Extended training further improves length constraint precision.: Additional training substantially improves length precision across mathematical and out-of-distribution datasets, with particularly steep RMSE reductions.
- A.1 Extended training further improves length constraint precision.: Greater length precision slightly lowers performance at high token ranges, likely because stricter constraints reduce flexibility for longer chains on challenging problems.
A.2 LCPO follows length constraints with high precision.
LCPO follows requested reasoning lengths with high precision, while errors and hard violations increase on out-of-distribution tasks and at lower token budgets.
- A.2 LCPO follows length constraints with high precision.: Hard-violation rates average 9% across token budgets, while soft-violation rates remain at or below 3%.
- A.2 LCPO follows length constraints with high precision.: Hard violations are higher at lower token budgets because the model prioritizes performance and shorter correct solutions are more difficult to generate.
- A.2 LCPO follows length constraints with high precision.: Out-of-distribution datasets exhibit higher length error, particularly MMLU and longer chains where additional reasoning is less necessary.
- A.2 LCPO follows length constraints with high precision.: LCPO maintains high length-control precision on mathematical reasoning datasets, with around 3% average deviation.
A.3 Aha Moment in L1’s training
L1’s training shows a phase transition toward simultaneous solution quality and token-length adherence, while experiments examine robustness, alternatives, scaling, and qualitative length-dependent reasoning.
- Training dynamics: At approximately 300 RL steps, L1 undergoes a phase transition in which token adherence improves sharply after initially prioritizing solve rate.The training logs also show a sharp drop in minimum token length.
- Robustness and baselines: 5.6-6.3% mean deviation from target length remains stable across inference temperatures from 0.0 to 1.0.Average token lengths across four target lengths remain closely aligned with the requested targets.
- Robustness and baselines: DeepScaleR-24K generates nearly 6000 tokens regardless of requested length, whereas SFT-trained models likewise fail to follow explicit token-length constraints.The paper attributes these failures to narrow output-length distributions and the absence of an online length-sensitive reward.
- Reward-function ablations: L1-Max Addition achieves 0% budget violation but collapses to extremely short chains, while the remaining reward variants perform similarly and the standard dual-objective L1-Max is selected.The addition variant’s perfect adherence is therefore undesirable despite its violation rate.
- Scaling: Sequential scaling is almost always better than parallel scaling at equal token budgets, although parallel scaling can improve reasoning performance by more than 10% in some cases.The comparison concerns increasing chain-of-thought length versus majority voting under the same total generated-token cost.
- Scaling and qualitative examples: LCPO retains high controllability, low budget-violation rates, and strong performance when scaled from the 1.5B model to DeepSeek-R1-Distill-7B.Qualitative examples also show a 512-token response failing while a 3600-token response solves the same problem correctly.