Mitigating the Length-Scaling Tax with Online Distillation

1 INTRODUCTION

• Standard RLVR jointly optimizes prompts of varying difficulty, so updates supporting longer reasoning on hard problems can alter easy-query continuations as well.When all responses in an easy group are correct, relative advantages collapse toward zero, leaving no gradient to preserve a concise solution while difficult groups continue to reshape the shared policy.Source: §1 · p. 1
• Existing efficiency work often relies on aggregate length statistics or controls easy-query length without explicitly preserving concise behavior already exhibited on solved queries.Length control on difficult, super-long reasoning can incur accuracy costs because exploratory branches may contribute to the correct solution.Source: §1 · pp. 1–2
• LST persists across easy sets defined at different checkpoints, and training distributions concentrated on hard prompts further amplify it.These studies characterize the tax’s behavior and identify its drivers.Source: §1 · p. 2
• LSD routes solved rollout groups to on-policy distillation while retaining the original RLVR objective for unsolved groups.Its teacher is a delayed version of the same policy lineage, implemented as a rolling exponential moving average rather than an external model.Source: §1 · p. 2
• LST measures excess response length on already-solved queries without a commensurate accuracy gain, addressing the lack of a systematic metric for this training-induced inefficiency.Conditioning on a fixed easy-query set distinguishes LST from global overthinking measures and avoids survivorship bias from repeatedly redefining the set.Source: §1 · pp. 1–2
• LSD with SG-FKL matches or improves average Pass@1 over RL while reducing LST from 19.0% to −3.7% on single-turn reasoning and from 31.4% to 13.7% on multi-turn agentic tasks.The introduction also notes analyses of how the distillation objective, teacher half-life, and routing threshold affect the trade-off between efficient behavior and capability acquisition.Source: §1 · p. 2

2 THE LENGTH-SCALING TAX

• At an anchor checkpoint, the easy-query set is defined using an evaluation solve-rate threshold, and membership is determined exclusively by rollouts from that anchor policy.At the default τ = 1, every sampled rollout for each selected query is correct at the anchor; the same set is evaluated at later checkpoints.Source: §2 · p. 3
• The reference is the shortest mean response length among evaluated RL checkpoints whose accuracy on the fixed easy set meets the threshold.The candidate checkpoint set spans the full training trajectory, including checkpoints before the anchor.Source: §2 · p. 3
• A positive LST measures percentage excess tokens over the accuracy-qualified reference; with τ < 1, it does not by itself establish waste at identical accuracy.With τ = 1, the reference has perfect empirical accuracy, and when the evaluated policy also has accuracy 1, the comparison is at identical empirical accuracy.Source: §2 · p. 3
• Global validation accuracy improves during RLVR while mean response length grows steadily across three benchmarks and their aggregate.Figure 1 encodes mean accuracy with purple circles and mean response length with orange squares.Source: §2 · p. 3, Figure 1 · p. 4
• Across frozen easy sets selected at five anchor checkpoints, accuracy remains largely stable while response length continues to increase throughout RLVR.Each set uses an empirical solve-rate threshold of τ = 0.875, and the resulting set is tracked at later checkpoints.Source: §2 · p. 3, Figure 2 · p. 4

Controlled experiments

• The controlled comparison varies rollout budget and training-data difficulty, including hard-only training against the full-mixture 4k-budget setting.ALL-8K doubles the rollout budget, while ALL-4K-8K switches to 8k after training from the ALL-4K step-300 checkpoint.Source: §3 · pp. 4–5, Table 1 · p. 5
• HARD-4K produces the highest LST across all easy sets; an 8k rollout budget raises LST when used from the start but lowers it relative to continued 4k training when introduced later.In the later-budget comparison, ALL-4K-8K also has a larger Pass@1 improvement, although its average and hard-query responses remain longer.Source: §3 · p. 4, Table 1 · p. 5

Why outcome-only RL does not correct the drift

• After easy groups saturate, their near-zero updates do not keep easy-query output distributions fixed, and hard-prompt updates dominate the mixed training signal.The prompts share policy parameters, so updates from hard prompts can change the policy on easy prefixes.Source: §3 · p. 4

Observable behavior

• On fixed E100 queries, increasing response length accompanies substantial increases in reflection words and repeated bigrams.The analysis covers 5,824 responses from the 26 queries in E100.Source: §3 · p. 5, Figure 3 · p. 5

Online routing

• The router uses the student’s on-policy empirical solve rate to avoid the extra teacher rollouts required by direct teacher-based routing.Over 87% of prompts classified as easy at step 500 were also classified as easy at step 250.Source: §4 · p. 5

Temporal self-teacher

• Using the pre-RL policy as a fixed teacher can mismatch prompts routed as easy with prompts that teacher can solve, potentially limiting benchmark improvement.The current student’s solve rate can route newly solved prompts to distillation even when the pre-RL policy cannot solve them reliably.Source: §4 · pp. 5–6
• The teacher is an exponential moving average of the online policy, whose lag is controlled by β; a larger β keeps it closer to past policies.The approach avoids a stronger external teacher or a separately prompted concise model.Source: §4 · p. 6

Routed LSD objective

• LSD applies the original RLVR objective to hard groups and an on-policy distillation objective to easy groups using a temporal self-teacher.An online router assigns groups according to current rollout accuracy.Source: §4 · pp. 5–6

LSD variants

• Three easy-group objectives differ in KL direction and gradient computation: SG-FKL, SG-RKL, and PG-RKL; all retain group-relative RL for hard groups.The supervised-gradient variants operate over the retained teacher support, while PG-RKL updates through sampled actions; route losses are weighted by their response-sequence counts.Source: §4 · p. 6

5.2 MAIN RESULTS

• Across four anchor-defined easy-query sets, all three LSD variants have lower mean length and LST than RL through most of training.The upper panels show mean length and the lower panels show LST; each anchor selects a different cohort.Source: Figure 5 · p. 7
• Hard-query mean length rises from 2329 tokens under RL to 2511, 2395, and 2393 under SG-FKL, SG-RKL, and PG-RKL, respectively.Thus, shorter easy-query responses coexist with longer hard-query responses.Source: §5.2 · pp. 7–8
• LST1000 falls from 19.0% under RL to −3.7% under SG-FKL, while LST50 falls from 31.4% to 13.7%.Figure 6 summarizes capability and LST across the single-turn and BrowseComp-Plus tasks.Source: §5.2 · p. 7, Figure 6 · p. 8
• After step 500, easy routes account for 12.17%, 10.58%, and 10.94% of tokens under SG-FKL, SG-RKL, and PG-RKL, respectively.The hard route retains 87.83–89.42% of logged token share on average.Source: §5.2 · p. 8
• RL + Length Penalty lowers LST50 to 1.2% and average turns to 4.66, but Pass@1 falls to 22.58%.CRISP yields 40.56% Pass@1; Fixed SG-FKL yields 30.44% Pass@1 and −10.8% LST1000.Source: §5.2 · p. 8
• Among the three EMA variants, SG-RKL has the lowest LST, but its stronger easy-query compression accompanies lower Pass@1.SG-RKL’s mean actor entropy is 0.0367, versus 0.0743 for SG-FKL and 0.0542 for PG-RKL; the proposed explanation is tentative.Source: §5.2 · p. 8

5.3 ABLATIONS

• At step 1750, H = 4 achieves the highest average Pass@1 among tested half-lives, 41.85%; longer half-lives generally suppress LST but slow capability acquisition.Larger H also produces greater parameter lag and higher distillation loss, while a smaller token share is routed to OPD.Source: §5.3 · p. 8, Figure 7 · p. 9
• Lowering τ from 1.0 to 0.85 and 0.75 reduces average Pass@1 from 41.85% to 40.91% and 39.31%, while distillation token share rises from 10.31% to 19.00%.The results support preserving behavior only for fully solved groups, since partially solved groups retain reward variation for RL learning.Source: §5.3 · p. 9, Figure 8 · p. 9

• LST is the excess length on already-solved queries that emerges during RL post-training without corresponding accuracy gains.The conclusion describes it as a conditional cost of RL post-training.Source: §6 · p. 9
• LSD applies token-level preservation to solved rollout groups and retains RL for unsolved groups, using online accuracy-based routing and an EMA self-teacher.The method is instantiated with three objectives and uses no external teacher, as specified in the conclusion.Source: §6 · p. 9
• The paper reports reduced easy-query cost with strong benchmark performance on single-turn mathematical reasoning and multi-turn agentic tasks.Future work will evaluate LSD on larger models and explore denser learning signals from sampled data.Source: §6 · p. 9

AI USE STATEMENT

• Generative AI tools supported translation, editing, literature synthesis, code, statistical checks, and methodology and results discussions; authors reviewed outputs and take responsibility for the final content.Source: AI USE STATEMENT · p. 10

A.1 SINGLE-TURN REASONING

• On the fixed easy set, RL step 885 achieves the lowest mean response length among RL checkpoints satisfying the 100% accuracy constraint, averaging 853.716 tokens, the shared reference for all methods.Source: Table 2 · p. 13
• The fixed hard-query cohort is the complement of the evaluation easy set, not the dynamically routed hard groups used during training.Source: §A.1 · p. 13

A.2 MULTI-TURN AGENTIC TASKS

• Table 3 reports evaluation results on multi-turn agentic tasks corresponding to Figure 6b, with Pass@1 and LST expressed as percentages and lengths measured in tokens.Source: Table 3 · p. 13
• SG-FKL and SG-RKL reduce LST and average interaction turns while increasing Pass@1; PG-RKL achieves the highest Pass@1 but uses more turns than RL.RL + Length Penalty reaches 1.2% LST50 and 4.66 average turns, but Pass@1 falls to 22.58% from RL’s 29.50%.Source: §A.2 · p. 13

B ABLATION REPORTING DETAILS

• Ablations compare five configurations over steps 1–1750; Table 4 reports endpoint Avg. Pass@1 and LST, and Table 5 summarizes training signals.The shared H = 4, τ = 1.0 run is used in both ablations, and the reference length is fixed across configurations.Source: Table 4 · p. 14, B ABLATION REPORTING DETAILS · p. 14, Table 5 · p. 14

B.1 TEACHER LAG AND DISTILLATION EXPOSURE

• H = 8 shows stronger preservation without greater distillation exposure; its delayed teacher imposes a stronger constraint on the tokens it supervises.Increasing H also raises teacher parameter lag and logged SG-FKL loss while reducing the fraction of tokens sent to OPD.Source: §B.1 · p. 15

B.2 ROUTING THRESHOLD AND PRESERVATION PRESSURE

• From τ = 1 to 0.75, mean easy-route token share rises from 10.31% to 19.00%, while the effective OPD coefficient rises from 0.247 to 0.450.The lower Pass@1 at step 1750 is consistent with increased preservation pressure on prompts that still admit incorrect responses; the coupled changes do not isolate which signal drives the difference.Source: §B.2 · p. 16, Figure 10 · p. 16

C DETAILED RL AND DISTILLATION OBJECTIVES

• LSD uses three easy-response OPD variants: two directly optimize top-K KL objectives, while PG-RKL estimates reverse KL through policy gradients.SG-FKL uses the teacher’s top-K tokens augmented with stop tokens; gradients pass through student logits while the teacher distribution is detached.Source: C DETAILED RL AND DISTILLATION… · p. 17
• SG-RKL normalizes the student distribution over the same augmented support and directly minimizes its reverse-KL objective.Source: C DETAILED RL AND DISTILLATION… · p. 17
• Sequence-mean–token-mean aggregation averages token losses within each response, then averages sequence losses within their assigned routes.Source: C DETAILED RL AND DISTILLATION… · p. 17

C.1 LOGGED OPTIMIZATION DIAGNOSTICS

• Figure 11 compares actor entropy and pre-update teacher–rollout log-probability discrepancy over shared steps 501–1628; SG-RKL is lower on both.Table 6 reports arithmetic means over the 1128 shared steps, which are not independent experimental replicates.Source: Figure 11 · p. 18, Table 6 · p. 18

C.2 EASY AND HARD TOKEN ALLOCATION DURING TRAINING

• The easy route accounts for a smaller share of tokens than sequences, while the hard route retains about 88–89% of average token share.These are per-run dynamic-routing aggregates, not fixed evaluation cohorts; the logs do not give route-specific response-length distributions or full-run token totals.Source: §C.2 · p. 19, Table 7 · p. 19, Figure 12 · p. 19

D PAIRED ACCURACY AND LENGTH ON FIXED EASY SETS

• All three LSD variants use fewer tokens and have higher reported accuracy than RL on each fixed easy-query cohort; each retains 100% accuracy at b = 100.The cohorts use RL-frozen query IDs with τ = 1, and every selected query had 32/32 correct anchor rollouts.Source: D PAIRED ACCURACY AND LENGTH ON FIXED… · p. 20, Figure 13 · p. 20, Table 8 · p. 20

D.1 THE MAIN-TABLE COHORT AND INDIVIDUAL EXAMPLES

• 99.43% and 99.15% easy-query accuracy for SG-RKL and PG-RKL, respectively, compare with 98.15% for RL, while mean lengths fall from 997 to 757 and 802 tokens.On hard queries, their accuracies are 32.50% and 33.45%, versus 32.85% for RL; Fixed is shorter but has lower easy-query accuracy.Source: Table 9 · p. 21, §D.1 · p. 21
• For amc23_45, all methods retain 32/32 correctness while mean length falls from 475.2 tokens under RL to 401, 400.4, and 393.2 under SG-FKL, SG-RKL, and PG-RKL.Source: §D.1 · p. 21
• Other examples include shortened-response regressions: aime25_6 falls from 31/32 under RL to 28/32 under SG-RKL and 30/32 under PG-RKL, while Fixed drops to 16/32 on amc23_60.SG-FKL retains 31/32 on aime25_6 and 32/32 on amc23_60 while shortening responses; these examples are selected from the largest respective correctness regressions among shortened queries.Source: §D.1 · p. 21

E RELATED WORK

• Unlike approaches using test-time budgets, length penalties, or general distillation, LSD combines on-policy distillation with online difficulty routing to preserve concise behavior on solved queries during RL.The paper evaluates this focus through LST on fixed easy-query sets.Source: E RELATED WORK · p. 22

F EXPERIMENTAL CONFIGURATIONS

• Table 10 lists training and inference configurations for RL and LSD, with values spanning both columns indicating shared settings.Agent mini-batches count transformed training rows; a dash marks an unspecified setting.Source: Table 10 · p. 23, F EXPERIMENTAL CONFIGURATIONS · p. 23
• Table 11 specifies RL and LSD configurations, with LSD retaining GRPO on the hard route and its gradient row describing only the distillation route.Source: Table 11 · p. 23
• The three online LSD variants share configuration settings across tasks, while Fixed SG-FKL results are reported for single-turn reasoning.Source: Table 11 · p. 23