Source-linked AI summary
Think Less, Locally: Local Preference Learning for Reasoning Compression
Muyao Yuan, Liyan Xu, Weizhan Zhang, Muyan Jiao, Jiangnan Li, Jinchao Zhang, Jie Zhou
TL;DR
Long reasoning can incur substantial inference costs when models continue after enough information is available, while existing compression methods focus mainly on whole trajectories. LoCo learns state-conditioned continue-or-conclude preferences from verified shorter continuations. Across math and code, it improves the accuracy–efficiency trade-off and transfers to coding benchmarks despite math-only training.
Problem
Existing reasoning compression methods mainly operate on trajectories, giving less attention to whether further reasoning is necessary from the current state.
Source: §1 · pp. 1–2Method
LoCo detects overthinking states in model-generated traces, freezes their prefixes, and uses verified continuations from the same states to train local preferences with DPO.
Source: §1 · p. 2Results
LoCo consistently improves the accuracy–efficiency trade-off across mathematics and code, with effective transfer to coding benchmarks despite math-only training.
Source: §1 · p. 2, §6 · p. 10Takeaways & Limitations
Effective reasoning compression requires learning when further reasoning is unnecessary at the appropriate state, rather than simply favoring shorter trajectories.
Source: §1 · p. 2, §6 · p. 10
Abstract
from arXiv · showhide
Large reasoning models achieve strong performance on challenging tasks, but their long chain-of-thought often incurs substantial inference costs by continuing to deliberate after sufficient information has been obtained. Existing reasoning compression methods primarily operate on reasoning trajectories, paying less attention to the local decision of whether further reasoning is necessary given the current state. We propose LoCo, a local preference learning method for reasoning compression. LoCo detects candidate overthinking states from the model's own traces, freezes the corresponding reasoning prefixes, regenerates shorter verified continuations from the same states, and trains on the resulting state-conditioned preference pairs with DPO. This formulation teaches the model when to continue reasoning and when to conclude, rather than simply encouraging shorter trajectories overall. We construct training data entirely from self-generated mathematical traces and evaluate LoCo on five mathematical and three code generation benchmarks. LoCo reduces completion length by 41.7\%--45.4\% on mathematics and 23.4\%--36.3\% on code, while maintaining or improving accuracy and achieving the best AUC across all code benchmarks. LoCo consistently improves the accuracy-efficiency trade-off on mathematics and transfers effectively to code generation despite math-only training, demonstrating the cross-domain generality of local preference learning for reasoning compression. The code and model will be publicly released at https://github.com/MuyaoYuan/LoCo.
1 INTRODUCTION
Long chain-of-thought can raise inference costs without improving accuracy when models continue after reaching a sufficient answer. LoCo targets this stopping decision with preferences conditioned on the current reasoning state.
1 INTRODUCTION
- Longer reasoning can increase decoding latency and memory use without necessarily improving accuracy when models overthink.A model may reach a correct answer early yet continue generating unnecessary reasoning.Source: §1 · p. 1
- Earlier compression methods compare complete trajectories or modify their parts, rather than deciding whether more reasoning is needed at the current state.The appropriate choice can vary across states: continue at one, conclude at another.Source: §1 · p. 2
- Across mathematical reasoning and code generation, LoCo improves the efficiency–performance trade-off and generalizes across domains.The training data are self-generated traces, and the same trained model transfers to coding benchmarks despite math-only training queries.Source: §1 · p. 2
- Effective compression requires learning when further reasoning is unnecessary, not simply favoring shorter trajectories.This frames compression as a state-specific decision rather than a global preference for brevity.Source: §1 · p. 2
- LoCo learns state-aware continue-or-conclude preferences from verified continuations sharing the same reasoning prefix.It identifies overthinking states in the model’s own traces and trains on local preference pairs.Source: §1 · p. 2
2 RELATED WORKS
Trajectory-level methods favor shorter complete solutions, while fine-grained methods modify parts of existing reasoning. LoCo instead anchors preferences at a shared reasoning state to guide whether reasoning should continue.
2.1 TRAJECTORY-LEVEL COMPRESSION
- Trajectory-level methods compare complete solutions or reward shorter correct reasoning, but provide limited information about where unnecessary reasoning occurs.Different reasoning paths can reach the same answer, making safe shortening difficult to locate from overall length or preference alone.Source: §2.1 · pp. 2–3
2.2 FINE-GRAINED COMPRESSION
- Fine-grained methods prune tokens, sentences, or steps, while LoCo compares alternative continuations from a fixed state to guide whether deliberation should continue.The shared state context makes the local continue-or-conclude decision the comparison target.Source: §2.2 · p. 3
3 PRELIMINARIES
The paper models reasoning as a sequence of steps and treats each completed step as a local decision point. Overthinking occurs when reasoning continues after that state already supports the answer.
3 PRELIMINARIES
- A chain of thought is segmented at paragraph boundaries into steps that may establish subgoals, compute, verify, or conclude.The resulting sequence is denoted S = (s0, s1, . . . , sn).Source: §3 · p. 3
- A reasoning state contains the prompt and completed reasoning prefix, defining a point where the model can continue or conclude.Overthinking is additional reasoning after the current state already contains sufficient information to support the answer.Source: §3 · pp. 3–4
- LoCo’s pipeline detects overthinking states, regenerates shorter correct continuations from them, and converts these into local preference pairs.The overview follows this sequence from a self-generated reasoning trace to training data.Source: Figure 1 · p. 4
4 METHOD
LoCo turns detected overthinking states into frozen-prefix preference pairs: it generates shorter verified continuations from each state and trains the model to prefer them over the original continuation. Detection combines model-confidence signals with a judge’s check that the next step is redundant.
4.1 OVERVIEW
- LoCo freezes each detected reasoning state, regenerates shorter correct continuations, and contrasts them with the original continuations from that same state for DPO training.The resulting triples pair a frozen prefix with a shorter correct continuation and the original continuation.Source: §4.1 · p. 4
4.2 STEP-LEVEL OVERTHINK DETECTION
- Candidate states have both low confidence and high volatility, measured from the model’s own traces using its confidence and volatility percentiles.The signals use completed steps; the detector begins truncation only at the following step, so it does not use the continuation being removed.Source: §4.2 · pp. 4–5
- A stronger judge rechecks flagged states and retains only those whose next step is redundant, excluding substantive progress and final-answer steps.The judge labels the next step OVERTHINK, NORMAL, or FINAL; removing a NORMAL step could discard progress, while removing a FINAL step could discard the answer.Source: §4.2 · p. 5
4.3 VERIFIED CONTINUATION GENERATION
- For each retained state, LoCo samples continuations and selects the shortest one that completes the template and gives an answer symbolically equivalent to the gold answer; states without a valid, strictly shorter continuation are discarded.The original continuation is the continuation following the same frozen state.Source: §4.3 · p. 5
4.4 LOCAL PREFERENCE LEARNING
- The shared prefix conditions DPO on an identical reasoning state, teaching LoCo when to conclude with a shorter valid continuation instead of continuing redundant reasoning.Pair retention also requires minimum token savings and prompt and continuation lengths within their limits; at most K pairs per question are kept, favoring larger savings.Source: §4.4 · p. 6
5 EXPERIMENTS
Across mathematical and coding benchmarks, LoCo combines substantial compression with comparable or improved accuracy; ablations indicate that local preferences and suitable cutpoints matter. The evaluation uses math-only training data, enabling a test of transfer to code.
5.1 EXPERIMENTAL SETUP
- The evaluation spans five math and three code benchmarks, with eight sampled completions per problem and accuracy, token, and AUC metrics.Code evaluation tests transfer beyond the math-only training query pool.Source: §5.1 · pp. 6–7, §5.3 · p. 7
5.2 EVALUATION ON MATHEMATICAL REASONING BENCHMARKS
- On Olympiad, AIME24, and AIME25, LoCo attains the highest accuracy and AUC with a 44.6%–45.4% reduction in completion length.Across easier benchmarks, it removes 41.7%–44.1% of generated tokens while matching or improving on the base model; DAST compresses mildly and TokenSqueeze loses more accuracy as difficulty rises.Source: §5.2 · p. 7, Table 1 · p. 7
5.3 EVALUATION ON CODING BENCHMARKS
- On MBPP, LoCo matches base-model accuracy while reducing tokens by 36.3% and improving AUC by 1.53 points.On LiveCodeBench and CodeContests, it shortens completions and improves both accuracy and AUC over all baselines; training queries are mathematical.Source: §5.3 · p. 8, Table 2 · p. 8
- Figure 2 compares accuracy against token budget on AIME24 and CodeContests; LoCo stays above TokenSqueeze over most budgets, indicating a better overall frontier.The comparison concerns accuracy–efficiency across varying token budgets, rather than a single cutoff.Source: §5.3 · p. 8, Figure 2 · p. 8
5.4 ABLATION STUDY
- Global trajectory preferences compress less than LoCo: token reductions are 5.2% on AMC23, 11.1% on AIME24, and 15.8% on AIME25, versus 41.7%, 45.4%, and 45.2%.The comparison suggests that identifying where compression should occur matters, beyond preferring shorter reasoning overall.Source: §5.4 · p. 9, Table 3 · p. 9
- External continuations perform worse overall in compression and AUC than LoCo, especially on harder AIME benchmarks; the authors suggest distributional alignment as a plausible explanation.LoCo uses continuations generated by the trained model, while the external variant changes only the chosen continuation.Source: §5.4 · p. 9
- RandCut and UniCut can shorten CoT substantially but incur larger accuracy losses; LoCo preserves the strongest accuracy on AIME24 and AIME25.These variants use random or evenly spaced cutpoints, suggesting the locations chosen for compression matter.Source: §5.4 · p. 9
6 CONCLUSION
LoCo places compression supervision at reasoning states where overthinking occurs, learning state-conditioned continue-or-conclude preferences rather than global preferences over complete trajectories. The conclusion reports gains across mathematics and code generation and states that generative AI tools were used in data preparation and paper editing.
6 CONCLUSION
- LoCo freezes the current reasoning prefix, regenerates verified shorter continuations from the same state, and learns state-conditioned continue-or-conclude preferences with DPO.This contrasts with treating compression as a global preference over complete trajectories.Source: §6 · pp. 9–10
- LoCo shows consistent gains across mathematical reasoning and code generation, with cross-domain improvements suggesting it reshapes reasoning-length preferences rather than adding task-specific knowledge.The paper concludes that effective compression requires learning when further reasoning is unnecessary at appropriate reasoning states.Source: §6 · p. 10
- The authors used generative AI tools to generate, clean, and reformat datasets, assist with translation, and edit the paper for readability.They state that they reviewed AI-assisted work and take responsibility for the final content.Source: AI USE STATEMENT · p. 10
- The paper states that methodological and experimental details for reproducing the results appear in the main text and appendices.These include method, training and evaluation setups, benchmark composition, decoding settings, ablations, and additional implementation details.Source: REPRODUCIBILITY STATEMENT · p. 10
A.1 OVERTHINK DETECTION
Overthink detection operates on each question’s shortest correct self-generated trace, screening candidate steps before judge re-checking. The process limits the number of candidate states retained per trace.
A.1 OVERTHINK DETECTION
- Each question contributes at most one source trace: the shortest correct of three rollouts sampled from DeepSeek-R1-Distill-Qwen-7B on the 14,027-question math14k training pool.Detection, judging, and pair construction operate on that selected trajectory.Source: §A.1 · p. 12
- Step confidence is the arithmetic mean of top-1 token probabilities across the step, while volatility is defined from the change between adjacent step confidences.The four confidence and volatility quantiles are calibrated once on the self-generated trace corpus.Source: §A.1 · p. 12
- Every confidence-screened OVERTHINK step is re-checked by Qwen3-30B, which receives the problem, preceding steps, candidate step, and predicted label.The judge returns a corrected label and short rationale in JSON.Source: §A.1 · p. 12
- OVERTHINK steps are wasteful or redundant if removing them does not break the path to the answer; NORMAL steps are necessary reasoning, while FINAL_ANSWER_WRAPUP steps mainly restate the answer.The labels are returned as a gold label and short reason in JSON.Source: §A.1 · pp. 12–13
- Only judge-confirmed OVERTHINK steps survive, and each trace retains at most 10 candidate states selected by uniform sampling.The cap prevents traces with many flagged steps from dominating regeneration.Source: §A.1 · p. 13
A.2 EVALUATION DETAIL
The evaluation details specify shared prompt instructions for the mathematical and code benchmarks, along with benchmark-specific additions and execution or judging procedures. Code outputs are evaluated under the stated benchmark protocols.
A.2 EVALUATION DETAIL
- The five mathematical benchmarks share an instruction to reason step by step and put the final answer within \boxed{}.Each benchmark appends its instruction to the raw problem, then wraps the result in the model’s native chat template.Source: §A.2 · p. 13
- The three code benchmarks share an instruction to write a Python solution and wrap final code in a Python code block.Each benchmark appends its instruction to the raw problem and uses the model’s native chat template.Source: §A.2 · p. 13
- MBPP prompts include public tests and LiveCodeBench includes starter code when applicable; extracted code is sandboxed with 10-second MBPP and 6-second LiveCodeBench timeouts.CodeContests uses AlphaCode judging, with case-insensitive token-wise comparison, exact integer matching, and 10^-5 tolerance for floating-point values.Source: §A.2 · p. 13
A.3 ABLATION CONSTRUCTION
The ablation variants hold the question set, pair-selection thresholds, and training recipe fixed while changing one construction choice. They compare global trajectory pairing, an external continuation, and alternative cutpoint selection against LoCo’s detector.
A.3 ABLATION CONSTRUCTION
- All ablation variants use the same question set, pair-selection thresholds, and training recipe as LoCo, changing only one construction choice.Differences in pair counts arise from the resulting construction.Source: §A.3 · p. 13
- Global pairs the shortest and longest correct rollouts per question, expands the pool to eight rollouts, and uses the same savings threshold and top-K selection as LoCo.Its prompt contains only the question, without a frozen state.Source: §A.3 · p. 13
- GPT-5.4 Cont. continues from LoCo’s frozen prefix and replaces the self-generated chosen continuation only when its continuation is correct and shorter than the rejected one.This variant differs from LoCo only on the chosen side.Source: §A.3 · p. 13
- RandCut samples cutpoints uniformly and UniCut places them on an evenly spaced grid, while retaining LoCo’s downstream regeneration and pair construction.The detector’s cutpoints average position 0.57, with about half in the final 40% of steps; RandCut and UniCut average 0.47 and 0.50, respectively.Source: §A.3 · pp. 13–14
B QUALITATIVE EXAMPLES
The examples show LoCo forming preferences at states where a solution is already sufficient, favoring a shorter verified continuation over redundant reasoning. Each pair keeps the prefix fixed and contrasts the chosen continuation with the original one.
B QUALITATIVE EXAMPLES
- Each preference triple pairs a frozen prefix with the shortest verified regeneration from that state and the rejected original continuation.The continuations retain their reasoning-block closing tag and final answer block.Source: B QUALITATIVE EXAMPLES · p. 14
- At step 28, the date problem’s answer was already derived and checked, so LoCo prefers concluding over further derivations and recounting.The rejected continuation adds reference-date reasoning, modulo arithmetic, and another manual recount after the answer was established.Source: B QUALITATIVE EXAMPLES · pp. 14–16
- At step 11, the fraction problem had been solved and verified twice, while the rejected continuation adds repeated derivations and a misinterpreted visualization detour.The preference is anchored at the state where the existing solution is sufficient.Source: B QUALITATIVE EXAMPLES · pp. 16, 18