L1: Controlling How Long A Reasoning Model Thinks With Reinforcement Learning

• Uncontrolled reasoning lengths can waste compute on excessive sequences or stop too early on difficult problems.Source: §1 · p. 1
• LCPO jointly optimizes final-answer correctness and adherence to a length constraint specified in the prompt.Source: §1 · p. 2
• L1 supports exact-length and maximum-length control while smoothly trading token budgets against reasoning performance.Source: §1 · p. 2
• LCPO-trained models generalize length control beyond mathematics to logical reasoning and general-knowledge benchmarks such as MMLU.Source: §1 · p. 2
• Short Reasoning Models improve substantially at short traces, with the 1.5B L1 model surpassing GPT-4o at equal reasoning lengths.Source: §1 · p. 2

• Longer reasoning chains and additional test-time computation generally improve performance on complex reasoning, mathematics, and code generation.Source: §2 · p. 2
• Reasoning models such as O1- and R1-style systems lack precise dynamic control over chain-of-thought length, limiting efficiency gains.Source: §2 · p. 3
• Existing length-control methods target architectural changes, training objectives, or instruction-following data, mostly outside reasoning-specific cost-quality trade-offs.Source: §2 · p. 3
• Reasoning models can underthink or overthink, while shorter-chain approaches do not precisely match user-specified inference budgets.Source: §2 · p. 3
• LCPO uses reinforcement learning to dynamically allocate inference compute under prompt-provided constraints and targets performance across varying budgets.Source: §2 · p. 3

• The training objective minimizes the difference between target and generated lengths while requiring a correct answer.Source: §3 · p. 3
• LCPO updates a pretrained reasoning model with reinforcement learning using correctness and length-penalty rewards.Source: §3 · pp. 3–4
• Prompts are augmented with instructions such as “Think for ngold,i tokens,” where target lengths are sampled from a range.Source: §3 · p. 4
• The reward coefficient α controls the trade-off between correctness and target-length adherence, producing the L1-Exact model.Source: §3 · p. 4
• At inference, users control output length by appending a fixed target length, or set of target lengths, to each prompt.Source: §3 · p. 4
• L1-Max uses a soft penalty for exceeding the target, preserving gradient propagation and favoring fewer tokens when correctness is maintained.Source: §3 · p. 4

Models and Datasets

• Training uses 40K mathematics question-answer pairs from AIME, AMC, Omni-Math, and STILL.Source: §4 · p. 4
• Evaluation covers AIME 2025, MATH, AMC, Olympiad-Bench, GPQA, LSAT, and MMLU.Source: §4 · p. 4

Baselines

• Baselines include DeepSeek-R1-1.5B, the unmodified DeepScaleR-24K, and the 4K-context DeepScaleR-4K model.Source: §4 · p. 5
• S1 is evaluated as a budget-forcing baseline that stops generation at a maximum budget and inserts “Final Answer.”Source: §4 · p. 5

Evaluation Protocol

• Models are assessed by mean deviation from target length and problem-solving accuracy at target lengths of 512, 1024, 2048, and 3600 tokens.Source: §4 · p. 5

Implementation Details

• GRPO training uses a 1e-6 learning rate, batch size 128, 4K training context, 8K evaluation context, and 700 steps.Source: §4 · p. 5

Relative performance

• L1 achieves over 100-150% relative and 20-25% absolute performance gains over S1 at 512 and 1024 token budgets.L1-Exact and L1-Max outperform S1 while maintaining length control by adapting reasoning chains instead of truncating mid-reasoning.Source: §5 · p. 5

Out-of-domain generalization

• L1 generalizes length control to GPQA, LSAT, and MMLU, with performance scaling positively with token budget and matching DeepScaleR-4K at comparable budgets on GPQA and LSAT.MMLU shows less pronounced scaling, with R2 = 0.66, because knowledge-focused questions benefit less from extended reasoning.Source: §5 · p. 6

Length controllability

• L1 follows requested lengths closely across 512, 1024, 2048, and 3600 token budgets, while L1-Max’s soft violation rates range from 0.3% to 2.3%.Mean length error is close to 3% on math datasets, though OOD datasets show higher errors of 20-40%.Source: §5 · pp. 7–8

Short-CoT performance

• At equal short generation lengths, L1 outperforms its non-reasoning counterpart by 5% on average and GPT-4o by 2% on average.These short reasoning models retain patterns similar to full reasoning models despite their shorter outputs.Source: §5 · p. 8

Learned reasoning patterns

• Reasoning patterns shift with token budgets: self-correction and verification keywords appear about twice as often at 4096 tokens, while conclusion terms increase 2-10x.Exploration-related keywords generally decrease in relative frequency at higher token counts, with “Alternatively” as an exception.Source: §5 · p. 9

Robustness and scaling

• The thinking-to-solution token ratio remains relatively stable across generation lengths, while L1 maintains length-control deviations of only 5.6-6.3% across temperatures from 0.0 to 1.0.Sequential scaling consistently outperforms parallel sampling at equivalent token budgets, and the same controllability trends extend to the 7B model.Source: §5 · pp. 9–10

• LCPO enables adaptive control of reasoning-chain length through reinforcement learning.Source: §6 · p. 10
• LCPO achieves over 100% relative and 20% absolute improvements in mathematical reasoning tasks over prior length-control approaches.Source: §6 · p. 10
• L1 generalizes length-controlled capabilities beyond its training distribution to out-of-domain tasks.Source: §6 · p. 10
• LCPO-trained models can become strong short reasoning models, outperforming larger frontier models such as GPT-4o at identical generation lengths.Source: §6 · p. 10

• LCPO provides strong length control for both in-domain and out-of-domain tasks.Source: §7 · p. 10
• Models may not generalize to requested reasoning lengths longer than those used during training.Source: §7 · p. 10
• Training models to use more inference compute at test time than during training remains an important future direction.Source: §7 · p. 10
• The models use length rewards on complete outputs, while alternative rewards targeting reasoning-token length remain unexplored.Source: §7 · p. 10

A.1 Extended training further improves length constraint precision.

• L1-Exact+ is produced by further fine-tuning L1-Exact for 500 reinforcement-learning steps.Source: §A.1 · p. 15
• Additional training substantially improves length precision across mathematical and out-of-distribution datasets, with particularly steep RMSE reductions.Source: §A.1 · p. 15
• Greater length precision slightly lowers performance at high token ranges, likely because stricter constraints reduce flexibility for longer chains on challenging problems.Source: §A.1 · p. 15

A.2 LCPO follows length constraints with high precision.

• Hard-violation rates average 9% across token budgets, while soft-violation rates remain at or below 3%.Source: §A.2 · p. 16, Figure 12 · p. 18
• Hard violations are higher at lower token budgets because the model prioritizes performance and shorter correct solutions are more difficult to generate.Source: §A.2 · p. 16
• Out-of-distribution datasets exhibit higher length error, particularly MMLU and longer chains where additional reasoning is less necessary.Source: Figure 9 · p. 17, Figure 10 · p. 17
• LCPO maintains high length-control precision on mathematical reasoning datasets, with around 3% average deviation.Source: Figure 11 · p. 17, Figure 10 · p. 17

Training dynamics

• At approximately 300 RL steps, L1 undergoes a phase transition in which token adherence improves sharply after initially prioritizing solve rate.The training logs also show a sharp drop in minimum token length.Source: §A.3 · p. 18, Figure 14 · p. 18

Robustness and baselines

• 5.6-6.3% mean deviation from target length remains stable across inference temperatures from 0.0 to 1.0.Average token lengths across four target lengths remain closely aligned with the requested targets.Source: §A.4 · pp. 18–19, Table 4 · p. 19
• DeepScaleR-24K generates nearly 6000 tokens regardless of requested length, whereas SFT-trained models likewise fail to follow explicit token-length constraints.The paper attributes these failures to narrow output-length distributions and the absence of an online length-sensitive reward.Source: §A.5 · p. 19, §A.6 · pp. 19–20, Table 5 · p. 19

Reward-function ablations

• L1-Max Addition achieves 0% budget violation but collapses to extremely short chains, while the remaining reward variants perform similarly and the standard dual-objective L1-Max is selected.The addition variant’s perfect adherence is therefore undesirable despite its violation rate.Source: §A.7 · p. 20

Scaling

• Sequential scaling is almost always better than parallel scaling at equal token budgets, although parallel scaling can improve reasoning performance by more than 10% in some cases.The comparison concerns increasing chain-of-thought length versus majority voting under the same total generated-token cost.Source: §A.8 · p. 20, Figure 18 · p. 24

Scaling and qualitative examples

• LCPO retains high controllability, low budget-violation rates, and strong performance when scaled from the 1.5B model to DeepSeek-R1-Distill-7B.Qualitative examples also show a 512-token response failing while a 3600-token response solves the same problem correctly.Source: §A.9 · p. 20, §A.10 · pp. 21, 26, 27, Figure 19 · p. 25