Source-linked AI summary
LightThinker++: From Reasoning Compression to Memory Management
Yuqi Zhu, Jintian Zhang, Zhenjie Wan, Yujie Luo, Shuofei Qiao, Zhengke Gui, Da Zheng, Lei Liang, Huajun Chen, Ningyu Zhang
TL;DR
Long reasoning traces strain LLM efficiency, motivating methods that reduce context overhead without sacrificing complex reasoning. The paper introduces LightThinker for dynamic thought compression and LightThinker++ for explicit, adaptive memory management with restorable details. Across standard and long-horizon agentic tasks, the framework reduces memory or token usage while maintaining or improving reported performance.
Problem
Long thought traces create substantial token and computational overhead, while implicit compression can irreversibly lose details needed for complex reasoning.
Method
LightThinker dynamically compresses intermediate thoughts, while LightThinker++ uses explicit memory primitives and synthesized expert trajectories to schedule archiving and retrieval.
Results
Across standard reasoning and long-horizon agentic benchmarks, the LightThinker family reduces memory or token usage while maintaining comparable accuracy or improving performance.
Takeaways & Limitations
Explicit adaptive memory management supports efficient, coherent reasoning when tasks require both compressed context and access to fine-grained details.
Takeaways & Limitations
Hidden-state compression can irreversibly lose critical details, while excessive segmentation can introduce structural noise and cumulative reconstruction failures.
Abstract
from arXiv · showhide
Large language models (LLMs) excel at complex reasoning, yet their efficiency is limited by the surging cognitive overhead of long thought traces. In this paper, we propose LightThinker, a method that enables LLMs to dynamically compress intermediate thoughts into compact semantic representations. However, static compression often struggles with complex reasoning where the irreversible loss of intermediate details can lead to logical bottlenecks. To address this, we evolve the framework into LightThinker++, introducing Explicit Adaptive Memory Management. This paradigm shifts to behavioral-level management by incorporating explicit memory primitives, supported by a specialized trajectory synthesis pipeline to train purposeful memory scheduling. Extensive experiments demonstrate the framework's versatility across three dimensions. (1) LightThinker reduces peak token usage by 70% and inference time by 26% with minimal accuracy loss. (2) In standard reasoning, LightThinker++ slashes peak token usage by 69.9% while yielding a +2.42% accuracy gain under the same context budget for maximum performance. (3) Most notably, in long-horizon agentic tasks, it maintains a stable footprint beyond 80 rounds (a 60%-70% reduction), achieving an average performance gain of 14.8% across different complex scenarios. Overall, our work provides a scalable direction for sustaining deep LLM reasoning over extended horizons with minimal overhead.
1 Introduction
Long thought traces improve complex reasoning but impose substantial token and computational overhead. LightThinker compresses intermediate thoughts, while LightThinker++ adds explicit adaptive memory management to preserve details when needed and improve efficiency across standard and agentic tasks.
- Motivation: Long thought traces increase the token and computational costs of complex LLM reasoning.The introduction describes a shift toward slower, multi-step reasoning and notes that performance gains come with many generated tokens.
- LightThinker: LightThinker dynamically compresses lengthy thoughts into compact representations and continues reasoning from the compressed content.The method trains special gist tokens and discards the original thought chain after compression.
- LightThinker++: LightThinker++ adds explicit memory primitives that let the model archive summaries or retrieve raw details when logical bottlenecks arise.A collaborative synthesis pipeline creates trajectories interleaving reasoning with purposeful memory operations for adaptive scheduling.
- Results: 70% lower peak token usage and 26% faster inference accompany comparable accuracy for LightThinker versus the Vanilla model.The reported accuracy drop is 1% with the Qwen model.
- Results: 69.9% lower peak memory with baseline accuracy occurs in Throughput, while 45.0% lower peak memory and a +2.42% average accuracy gain occur in Budget.These are the two reported LightThinker++ settings for standard reasoning.
- Agentic tasks: Beyond 80 rounds, LightThinker++ maintains 30k–40k tokens instead of the Vanilla agent’s 100k within 60 rounds, reducing footprint by 60%–70%.Across three long-horizon benchmarks, it also reports a +4.4% average Pass@1 boost and a 2.51× gain on the hard subset.
2 Background
LLM reasoning has evolved toward deliberative, multi-step problem solving, but longer contexts increase memory and inference costs. These costs arise from KV-cache growth and the quadratic scaling of Transformer attention with token count.
- Slow Thinking: Slow-thinking methods such as Chain-of-Thought solve complex problems through sequential sub-steps and deliberative reasoning.The background contrasts this with rapid, intuitive System 1-style processing.
- Efficiency costs: At 104k context length for Qwen-32B, KV-cache memory becomes comparable to the model size.This illustrates the memory pressure created by extended reasoning contexts.
- Efficiency costs: Transformer attention makes autoregressive inference computation scale quadratically with the number of tokens.Longer generated contexts therefore increase computational cost substantially.
3 Methodology
The LightThinker family progressively shifts from implicit representation-level thought compression toward explicit, adaptive behavioral memory management. LightThinker prioritizes efficiency through hidden-state compression, while LightThinker++ preserves recoverable details and dynamically controls context for complex, long-horizon reasoning.
- LightThinker: LightThinker compresses completed thoughts into a small set of hidden-state cache tokens, allowing subsequent reasoning to use compressed content instead of the original thought chain.The method uses gist tokens and does not alter the output format.
- LightThinker: LightThinker applies an information bottleneck for extreme efficiency, but its lossy compression can limit performance on high-precision tasks.The compressed representation is not intended to retain every detail, only information essential for subsequent inference.
- LightThinker++: LightThinker++ introduces explicit memory mechanisms that let the model archive details or retrieve them when logical necessity requires fine-grained information.This behavioral-level approach addresses the irreversibility of implicit compression through controllable memory operations.
- LightThinker++: By decoupling reasoning depth from sequence-length growth, LightThinker++ is designed to sustain long-range coherence while mitigating contextual redundancy.This design targets complex Chain-of-Thought and long-horizon reasoning settings.
- LightThinker++: Unlike LightThinker’s predefined compression intervals, LightThinker++ autonomously decides when to archive or retrieve information according to reasoning complexity.Its explicit primitives support adaptive context orchestration and bidirectional expansion of archived information.
4 Experiments: General Reasoning
The experiments show that LightThinker and LightThinker++ reduce memory and inference costs while preserving or improving accuracy across standard reasoning settings. Explicit adaptive memory management further supports high-fidelity compression, task-aware scheduling, and bounded memory growth.
- Overall efficiency: 51% higher inference time on Qwen and 72% on Llama make H2O’s memory savings costly compared with Vanilla.The overhead is attributed to token-level eviction at every decoding step.
- Overall efficiency: 2.5× average inference-time reduction under the same memory budget shows that LThinker converts lower memory use into practical throughput gains.The comparison covers four datasets with the Qwen model.
- Overall efficiency: 15% fewer output tokens on Qwen and 13% on Llama make LThinker the only evaluated method consistently shorter than Vanilla.The shorter generations are identified as a main reason for faster decoding.
- Long-generation efficiency: 44% lower inference time at 32K output tokens demonstrates that LThinker’s speed advantage grows for long generations.For outputs between 1K and 4K tokens, the reduction is smaller but remains measurable.
- LightThinker++ performance: 69.9% lower Peak and Dep in the Throughput setting preserves comparable accuracy, while the Budget setting adds 2.42% accuracy with 45.0% lower Peak and 33.7% lower Dep.The Throughput example on Qwen2.5-7B is 3120 → 940 Peak tokens; the Budget setting prioritizes reasoning depth under fixed context constraints.
- Adaptive memory management: 82.9% fewer visible tokens on Qwen-2.5-7B and a 71.3% lower peak ceiling show that LightThinker++ bounds attended context as reasoning budgets grow.Visible tokens fall from 2982 to 511, while peak tokens reach only 1,830 as generation scales.
5 LightThinker++: Long-Horizon Agentic Reasoning
LightThinker++ extends the Thought–Action–Observation loop with explicit, stateful memory management for long-horizon reasoning. Its memory primitives archive interaction summaries, restore raw evidence when needed, and remove expanded details after use.
- Framework: LightThinker++ treats each agent interaction as a TAO turn paired with a distilled semantic core.The full interaction contains thought, action, and observation, while the semantic core preserves its summarized content.
- Framework: LightThinker++ manages a stateful research log whose historical turns can switch between summarized and expanded representations.This lossy-to-lossless toggling prioritizes high-signal content in the active context.
- Memory actions: Commit distills an interaction into a summary, expand restores raw evidence for precise synthesis, and fold purges details after their utility is exhausted.These memory actions govern the contextual lifecycle during agentic interaction.
- Trajectory synthesis: Trajectory synthesis uses an Interaction Agent for environment tools and granular retrieval, while a Contextual Governor schedules summary commits.The roles separate execution-level details from high-level context management.
- Trajectory synthesis: Behavioral pruning filters synthesized trajectories for active context governance before fine-tuning the model on the resulting expert policy.The training objective combines reasoning traces with memory operations so the agent learns explicit context engineering.
6 Experiments: Long-Horizon Agentic Reasoning
Experiments evaluate LightThinker++ on long-horizon web research using three benchmarks and show gains in performance, interaction efficiency, and context stability. The improvements are strongest on difficult and extended tasks where unmanaged histories become noisy.
- Main results: Pass@1 on hard01 rises from 6.8% to 20.9% on xbench, from 8.6% to 20.6% on BrowseComp-ZH, and from 5.1% to 10.5% on BrowseComp-EN.The hard01 subset contains instances where the Vanilla-Agent succeeds at most once across three runs.
- Scaling: LightThinker++ reaches the Vanilla model’s 38.3% xbench peak in 24 actions instead of 60, representing 2.5× greater action efficiency.Its explicit memory management reduces the interactions needed to reach the baseline’s peak performance.
- Scaling: Xbench accuracy reaches 44.0% as the interaction horizon extends, with convergence after 80 rounds.The results indicate consistent scaling with exploration depth rather than deterioration from longer horizons.
- Scaling: Accuracy improves rapidly at low token budgets, saturates around 32k–48k tokens, and remains stable up to 110k tokens.This pattern describes token-budget scaling in the long-horizon agentic setting.
- Context control: LightThinker++ stays between 30k and 40k active-context tokens through 80 rounds, while Vanilla reaches approximately 100k tokens within 50–60 rounds.Its periodic context changes reflect expanding evidence temporarily and folding it back into essential insights.
- Ablation: Adding commit raises xbench Pass@1 from 38.3% to 41.7%, while adding fold and expand raises it further to 44.0%.On BrowseComp-ZH, the corresponding progression is 31.5% to 34.8% and then 36.9%.
- Ablation: Memory orchestration produces steady improvements across xbench and BrowseComp-ZH as additional components are added.The full mechanism combines information distillation with dynamic recovery and context control.
7 Related Work
Related work accelerates LLM inference through model or KV-cache quantization, fewer generated tokens, KV-cache reduction, and dynamic context management. These approaches differ in whether they intervene during inference and whether they prune, merge, or reorganize information.
- Method categories: Inference acceleration research includes parameter quantization, KV-cache quantization, generating fewer tokens, and reducing KV-cache contents.The section distinguishes shortening generation from understanding long contexts.
- KV-cache reduction: KV-cache reduction uses pruning-based selection or merging-based compression of historically important information.Pruning methods apply eviction policies during inference, whereas merging methods train models to compress information into anchor tokens.
- Token reduction: Generating-fewer-token methods include discrete token reduction, continuous-space tokens, and approaches that generate answers without intermediate tokens.These strategies are applied after training and do not require additional inference-time intervention.
- Context management: Context management dynamically reorganizes information within the context window during reasoning rather than only shortening generation or optimizing hardware.This category is especially relevant to agents performing complex, long-horizon interactions.
8 Conclusion
The paper presents LightThinker as a method for dynamically compressing intermediate thoughts to improve reasoning efficiency while maintaining competitive accuracy. It also introduces the Dependency metric to quantify compression across accelerating methods.
- LightThinker dynamically compresses intermediate thoughts during generation to reduce memory overhead and computational costs while maintaining competitive accuracy.
- The Dependency metric quantifies the degree of compression across different accelerating methods.
A Metric: Dependency
This section presents figures illustrating the Dependency metric and generated-token lengths. The supplied passages identify the visualizations but do not explain their quantitative comparisons.
- Figure 14 illustrates the Dependency metric.
A.1 Motivation
The section motivates Dependency as a fairer comparison basis for dynamic and static compression methods, then describes the attention-based compression operation. It also documents baseline methods and implementation settings.
- Baselines: LightThinker and AnLLM dynamically determine compression counts and ratios, whereas H2O and SepLLM use user-defined token-retention hyperparameters.
- Motivation: Peak token count alone is insufficient for fair comparison because generated outputs have unknown lengths and methods may sustain different token counts over time.Prior methods often compress fixed-length prompts, whereas this setting also compresses generated output.
- Motivation: Dependency measures cumulative information dependence during generation, enabling comparisons between dynamic and traditional KV cache compression methods.Smaller Dependency indicates stronger effective compression.
- Mathematical Description of Compression: During compression, attention aggregates information from the uncompressed context, the thought being compressed, and the compressed segment into a representation stored in the KV cache.
- Baselines: H2O greedily retains tokens with the largest cumulative attention scores, while SepLLM emphasizes punctuation-position tokens.
C.1.6 Additional Results
Additional materials describe the datasets, prompts, training configurations, and timing protocol used for general reasoning and agentic research evaluations. The figures and table provide supporting visualizations and dataset statistics.
- Additional Figures: The evaluation materials include average generated-token counts, compressed-length distributions, attention-mask visualizations, and a case-study example.
- General Reasoning Data: 17,315 unique problems form the foundation of the general reasoning trajectories.The collection combines a refined BS17K subset with sampled DeepScaleR instances.
- Agentic Research Data: 8,954 unique queries form the base pool for agentic research trajectory synthesis.The pool draws from HotpotQA, MuSiQue, WebDancer, WebShaper, and WebWalkerQA-Silver.
- Timing Protocol: Timing experiments use total concurrency of 32, with question-level parallel inference for Vanilla and LightThinker-based methods and batched decoding for TokenSkip.
- Prompts: Figure 17 provides the synthesis prompt for general reasoning data, while Figures 18 and 19 provide prompts for agentic trajectory generation and model evaluation.
D Efficiency Analysis of LightThinker++ (Budget setting)
This section examines LightThinker’s efficiency mechanisms and memory behavior, emphasizing learned compression, distinct compression and generation roles, and the effects of cache size.
- AnLLM and LightThinker both use sparse attention, but otherwise differ because AnLLM targets prompt compression while LightThinker targets output compression.
- LightThinker separates compression from generation, assigning [ci] to compression and [o] tokens to content generation.
- LightThinker’s compact cache functions as model-specific working memory that supports reasoning from stored content but does not transfer across LLMs.
- LightThinker learns continuous-space representations that merge prior tokens instead of using manually designed discrete eviction policies.
- Smaller caches cause LightThinker to generate more tokens, with repetition increasing as compression loses more information.
E.4 Comparison with Implicit CoT Works
This section compares LightThinker with implicit continuous-reasoning methods, focusing on context dependence, training requirements, interpretability, and generalization.
- Implicit CoT performs reasoning entirely in continuous space, whereas LightThinker combines continuous and discrete reasoning.
- LightThinker reduces historical tokens required for generation instead of depending on the entire context, unlike implicit CoT methods.
- LightThinker uses standard SFT with modified attention masks and requires neither specialized training data nor architectural changes.
- LightThinker retains discrete tokens, making reasoning more interpretable and yielding promising out-of-domain generalization in the reported experiments.
F Agent Case Study
The case study illustrates memory orchestration for long-horizon reasoning through commit, expand, and fold actions that preserve and recover relevant evidence.
- Memory orchestration uses commit, fold, and expand to distill evidence and dynamically control context during long-horizon reasoning.
- The agent commits lengthy wiki content after identifying William Henry Harrison as the shortest-serving president.
- Expanding the archived observation recovers that Harrison’s vice president was John Tyler, supplying information needed for the next question.
- After use, the agent folds the lengthy content again, returning the observation to a compact archived form.
- The case study demonstrates that dynamic context control can recover omitted intermediate evidence from memory and fold it back after use.