LAPO: Internalizing Reasoning Efficiency via Length-Adaptive Policy Optimization

Motivation

• Large reasoning models often generate excessively verbose chains regardless of problem complexity, creating computational overhead that hinders practical deployment.This behavior is described as overthinking.Source: §1 · p. 1
• LAPO treats successful-solution length distributions as signals of intrinsic problem complexity instead of imposing arbitrary reasoning limits.The framework aims to help models develop an internal sense of appropriate reasoning depth.Source: §1 · p. 2

Method

• The Discovery stage uses length-aware rewards to collect natural reasoning-length statistics, while Internalization embeds target lengths into prompts as self-proposed budgets.The model learns to align its reasoning trajectory with an anticipated budget.Source: §1 · p. 2, Figure 1 · p. 2

Contribution

• LAPO differs from rigid-control methods by learning problem-specific computational needs and allocating effort proportionally to task demands.This design is presented as an intrinsic reasoning capability rather than an external constraint.Source: §1 · pp. 1–2

Results

• LAPO reduces token usage by up to 40.9% while improving accuracy by 2.3% on mathematical reasoning benchmarks.The reported gains indicate efficient reasoning without sacrificing quality.Source: §1 · pp. 2–3

Limitations of Existing Methods

• Extended chain-of-thought reasoning can improve accuracy but often produces unnecessarily long traces, especially on simple problems.This inefficiency is commonly termed overthinking.Source: §2.2 · p. 3, §2.1 · p. 3

Prior Approaches

• Existing strategies include prompt-based controls, direct length reduction, hard limits, and dynamic early stopping, each with adaptability or reasoning-continuity limitations.These approaches may over-shorten reasoning, impose rigid constraints, or truncate mid-reasoning.Source: §2.2 · p. 3

LAPO

• LAPO combines autonomous budget estimation and precise length control through a two-stage reinforcement-learning design.It avoids external truncation mechanisms and manual budget specification.Source: §2.2 · pp. 3–4
• LAPO’s Discovery stage explores natural reasoning patterns with length-aware rewards, and its Internalization stage converts them into adaptive reasoning behavior.The stages use successful-solution length distributions as guidance for reasoning depth.Source: Figure 2 · p. 4

3 METHOD

• Discovery extracts correct-response lengths, filters them with the 30th–70th percentile range, and records each problem’s median length.Problems without correct rollouts receive a maximum-length default to encourage further exploration.Source: §3.2 · p. 5
• The Discovery reward combines correctness with length efficiency, granting length-based rewards only to correct responses and penalizing distance from the learned range.The linear decay function f(d) = max(0, 1 −d/100) reduces reward as deviations increase.Source: §3.2 · p. 5
• Internalization places a self-declared target budget after the <think> token and rewards correct solutions that adhere to it.This converts discovered length information into an intrinsic reasoning plan rather than an external hard constraint.Source: §3.3 · pp. 5–6
• The training pipeline continuously updates the problem-to-length mapping, establishing benchmarks for newly solved problems while making previously solved problems more efficient.This progressive design moves from learning appropriate depth through experience to anticipating it proactively.Source: §3.3 · p. 6, §3.4 · p. 7
• LAPO first discovers problem-specific reasoning lengths from correct rollouts, then internalizes those patterns through guided GRPO training.The mapping uses successful-response statistics to estimate reasonable ranges and median target lengths before Internalization.Source: §3.2 · p. 5, §3.4 · p. 6

5 EXPERIMENT SETUP

• Training uses 10,000 mathematical problems: 6,000 from DeepScaleR-Preview-Dataset and 4,000 intermediate MATH examples.The mixture is designed to provide a balanced difficulty distribution.Source: §5 · p. 7
• LAPO is applied to DeepSeek-R1-1.5B and DeepScaleR-1.5B-Preview base models, with GRPO used in both training stages.Discovery and Internalization each run for three episodes with empirically selected efficiency–correctness weights.Source: §5 · p. 7
• Inference uses a 32,768-token generation window to measure efficiency without imposing a simple context-length limit.Evaluation covers MATH-500, AIME2024, AMC23, and Olympiad-Bench.Source: §5 · p. 7
• Baselines include the foundational models, an accuracy-only ablation, and existing efficient-reasoning methods.The accuracy-only baseline isolates the effect of LAPO’s length-aware reward.Source: §5 · p. 7

6.1 MAIN RESULTS

• LAPO reduces token usage while improving accuracy, outperforming base models and establishing a strong accuracy-efficiency trade-off without external length controls.On DeepScaleR-1.5B-Preview, tokens decrease 38.5% with a 2.3-point accuracy gain; on DeepSeek-R1-1.5B, tokens decrease 41.0% with a 1.2-point gain.Source: §6.1 · p. 8, Table 1 · p. 8
• LAPO surpasses budget-driven, implicit-regularization, and adaptive-activation methods, preserving stronger reasoning quality across baseline paradigms.Its Discover-Internalize process supports continuous adaptation rather than external budgets, progressive compression, or binary mode switching.Source: §6.1 · pp. 8–9
• LAPO-D reduces tokens by 36.0%, while LAPO-I adds a further 6.9% efficiency gain through in-context internalization.LAPO-D also improves average accuracy by 0.5 points, showing that the two stages both contribute to the final result.Source: §6.1 · p. 9

6.2 ABLATION STUDY ON IN-CONTEXT GUIDANCE

• Specific, internally positioned length guidance is critical: external placement reduces accuracy to 63.9%, while removing guidance reduces it to 63.6% and restores LAPO-D token usage.The default method outperforms less precise range guidance, indicating that discovered specific targets provide a stronger learning signal.Source: §6.2 · p. 9, Table 2 · p. 9

6.3 ABLATION ON STATISTICAL METRICS FOR TARGET LENGTH

• The median target-length statistic gives the best balance, reaching 64.8% average accuracy with 3,832 tokens, whereas minimum-length selection reaches 3,821 tokens but drops accuracy to 63.5%.The mean uses 4,040 tokens and has slightly lower accuracy because unusually long successful solutions make its budget overly generous.Source: §6.3 · pp. 9–10, Table 3 · p. 10

6.4 ANALYSIS OF INTERNALIZATION

• LAPO-I remains near its 2,200-token baseline under conflicting prompts, unlike the externally guided variant, whose output changes to 1,247 or 2,821 tokens.This robustness supports the claimed internalization of the reasoning-length plan.Source: §6.4 · p. 10, Table 4 · p. 10

6.5 DIFFICULTY-AWARE COMPUTATIONAL ALLOCATION

• Across difficulty levels, LAPO produces near-linearly longer reasoning for harder problems, demonstrating emergent difficulty-aware computational allocation.The analysis attributes efficiency to concise reasoning on simple problems and more computation for complex ones; Figure 4 additionally examines reductions in self-correction and exploratory behavior.Source: §6.5 · p. 10, Figure 3 · p. 11, §6.6 · p. 11

• 40.9% lower token usage and 2.3% higher accuracy demonstrate LAPO’s efficiency gains on mathematical reasoning benchmarks.The method distinguishes problems requiring elaborate derivations from those needing brief calculations.Source: §7 · p. 11
• LAPO autonomously adjusts reasoning length based on problem complexity through a two-stage reinforcement learning framework.Models first learn appropriate reasoning depth from experience, then anticipate those requirements proactively.Source: §7 · p. 11

• LAPO-D and LAPO-I use separate system prompts tailored to DeepSeek-R1-Distill-Qwen-1.5B and DeepScaleR, respectively.Source: §A.1 · p. 15
• The prompts require step-by-step reasoning inside <think> tags and a final answer within \boxed{}.Source: §A.1 · p. 15
• Training sampled 8 responses per query at temperature 1.0, with kl=0.0001, learning rate 1e-6, batch size 128, and a 4K-token context limit.Source: §A.1 · p. 15
• Both LAPO-D and LAPO-I were trained for 3 episodes, approximately 240 steps, using α=0.7 and β=0.8.Source: §A.1 · p. 15

• Average length falls from 3,280 tokens to 2,365 with LAPO-D, then below 2,200 tokens with LAPO-I.The reductions occur in two stages, with LAPO-D driven by the length-aware reward and LAPO-I building on plan-adherence reward plus in-context guidance.Source: §A.2 · p. 15
• LAPO-D accuracy rises from 85.8% to over 86.4% while LAPO-I sustains high accuracy on a tighter token budget.LAPO-I also shows a transient performance peak during training.Source: §A.2 · pp. 15–16
• LAPO reduces response length while maintaining or improving accuracy across both training stages on MATH-500.Source: §A.2 · p. 15, Figure 5 · p. 16
• The training dynamics support the two-stage design: LAPO-D builds an efficient-reasoning foundation, while LAPO-I refines the performance–cost balance.The analysis attributes the resulting strategies to learning from successful reasoning patterns.Source: §A.2 · p. 16

• Combined-data achieves the best reported average performance and token-use efficiency among the evaluated training-dataset configurations.Source: §A.3 · pp. 16–17
• A balanced difficulty distribution helps generalization, while varied difficulty gradients improve the model’s complexity–length mapping.The broader difficulty range helps the model learn an appropriate thinking range for different questions.Source: §A.3 · p. 17
• On GPQA, LAPO improves accuracy and efficiency for both DeepSeek-R1-1.5B and DeepScaleR-1.5B-Preview.The benchmark evaluates whether LAPO generalizes beyond structured mathematical reasoning.Source: §A.4 · p. 17, Table 7 · p. 17
• For DeepSeek-R1-1.5B, LAPO-D raises Pass@1 accuracy by 2.0 points and reduces token generation by 26.2%.Source: §A.4 · p. 17
• For DeepScaleR-1.5B-Preview, LAPO-D raises accuracy by 2.2 points and reduces token generation by 19.4%.The internalization stage further improves efficiency while retaining an accuracy advantage over baseline.Source: §A.4 · p. 17