Source-linked AI summary
Do Language Models Need Sleep? Offline Recurrence for Improved Online Inference
Sangyun Lee, Sean McLeish, Tom Goldstein, Giulia Fanti
TL;DR
Long-context transformers face costly attention caches, and fixed-size fast weights may not provide enough computation for deep reasoning over evicted tokens. The paper introduces sleep-like recurrent consolidation, repeatedly updating fast weights before cache eviction while preserving single-pass prediction latency. Across controlled tasks and mathematical reasoning, increasing sleep duration improves performance, especially for deeper reasoning, with training cost and stability as limitations.
Problem
Transformers scale poorly with context length, while vanilla SSM-attention hybrids can degrade on deeper reasoning over evicted context despite sufficient fast-weight capacity.
Method
The model performs N offline recurrent passes over accumulated context to update SSM fast weights before clearing the KV cache.
Results
Across controlled synthetic tasks and mathematical reasoning, increasing sleep duration improves deep sequential computation over evicted context, with the largest gains on deepest instances.
Takeaways & Limitations
Offline recurrence can organize evicted context into persistent weights that support later reasoning without adding loops to prediction.
Takeaways & Limitations
Training requires N deeper forward and backward passes, which can make training slow and unstable.
Abstract
from arXiv · showhide
Transformer-based large language models are increasingly used for long-horizon tasks; however, their attention mechanism scales poorly with context length. To handle this, we study a sleep-like consolidation mechanism in which a model periodically converts recent context into persistent fast weights before clearing its key-value cache. During sleep, the model performs $N$ offline recurrent passes over the accumulated context and updates the fast weights in its state-space model (SSM) blocks through a learned local rule. During inference, this shifts extra computation to sleep while preserving the latency of wake-time prediction. We test our method on controlled synthetic tasks, including cellular automata and multi-hop graph retrieval, as well as a realistic math reasoning task, on which a regular transformer as well as SSM-attention hybrid models fail. We then show that increasing sleep duration $N$ for our models improves performance, with the largest gains on examples that require deeper reasoning.
1 Introduction
Transformers provide high-fidelity context retrieval but incur context-length costs, while fixed-size fast weights may lack computation for deep reasoning over evicted tokens. LLM sleep addresses this by recurrently consolidating context into persistent weights, improving performance as reasoning demands deepen.
- Attention-cache computation grows quadratically with context length, while cache memory grows linearly.
- Fast-weight memories support long-range recall, but their ability to perform deep computation over tokens absent from the KV cache is unclear.
- During sleep, the model repeatedly processes accumulated context and updates fast weights through a learned local rule before context eviction.
- The proposed evaluation varies reasoning depth while holding memory load fixed, testing whether sleep-time computation converts transient context into useful fast weights.
- Vanilla GDNs fail as reasoning depth increases despite sufficient fast-weight capacity, whereas recurrent fast-weight models improve with more recursions and gain most on deepest problems.
- The results support sleep-like offline recurrence as a mechanism for organizing evicted context into weights that support later reasoning.
2 Related Work
Related work compresses or consolidates long-context information through recurrent states, smaller KV caches, distillation, gradient updates, depth recurrence, offline planning, and sleep-inspired learning. This paper instead transfers evicted context into weight-based memory using a learned recurrent update.
- SSMs maintain fixed-size fast-weight memories, trading the KV cache’s growth with sequence length for more lossy compression.
- Context-compression methods condense long contexts into shorter hidden-state sequences or smaller KV caches that replace the full-context cache.
- Unlike context-compression methods, this method transfers evicted context into weight-based memory rather than shortening the remaining attention context.
- Context distillation transfers active context into weights through gradient descent on predefined objectives, whereas this method uses a learned recurrent forward pass.
- Test-time training stores long-range information in temporary parameter updates, while this method uses recurrent memory updates that need not correspond to one gradient step.
- Depth recurrence increases model depth and can make computation adaptive, while offline planning and sleep-inspired methods motivate allocating computation away from immediate response time.
3 Preliminaries
Attention retrieves prior tokens through a growing KV cache, whereas linear recurrent layers compress history into fixed-size fast weights. The paper evaluates these memory mechanisms on controlled tasks whose reasoning depth can vary independently of sequence length.
- Sequence mixers: Self-attention computes token interactions from query-key similarity and stores prior keys and values in a KV cache.
- Sequence mixers: The KV cache enables access to any previous token but grows linearly with sequence length.
- Sequence mixers: Linear recurrent layers store history in a fixed-size fast-weight state using gated, Hebbian-like updates.
- Sequence mixers: Fast-weight states are more memory-efficient than KV caches but more lossy because past tokens must be compressed.
- Sequence mixers: Sequence-mixing layers combine with normalization, residual connections, and MLPs to form attention or SSM blocks.
- Synthetic reasoning tasks: Rule 110 tests deep sequential computation, while Depo requires multi-hop traversal; both vary reasoning demand with fixed sequence length.
4 Motivating example: Can attention-SSM hybrid models reason about context they can no longer attend to?
The motivating setup evicts attention context before prediction, forcing models to consolidate information into fast weights while allowing only a single prediction pass. Hybrid models can store evicted states, but their performance deteriorates as required reasoning depth grows.
- Attention-SSM hybrids are tested in a setting where compressed fast-weight memory must replace context that is no longer directly accessible.
- The Rule 110 task presents four independent binary states and later asks for labels derived from t-step rollouts.
- Increasing t raises reasoning depth, from first-bit retrieval at t = 0 to progressively harder cellular-automaton computation.
- The experiment clears the context window every 24 tokens, requiring each state to be encoded into fast weights before the KV cache is evicted.
- The setup separates consolidation over the first 96 tokens from prediction over the final 4 tokens.
- Prediction is restricted to one standard forward pass per answer token, so labels must be consolidated before prediction begins.
- A four-layer GDN-attention hybrid’s accuracy drops rapidly as t increases despite fixed sequence length, indicating a reasoning-depth problem rather than insufficient memory capacity.
- These failure claims concern performance under a fixed training-token budget, not impossibility with unlimited data, compute, or training time.
5 LLM Sleep: Offline Recursive Memory Consolidation
LLM sleep adds recurrent consolidation passes before attention-cache eviction, updating persistent fast weights so later prediction can use the refined memory in a single forward pass.
- Sleep mechanism: The superscript ×N denotes N looped passes over the architecture.
- Training procedure: Algorithm 1 initializes fast weights, splits inputs into windows of at most L tokens, and alternates consolidation and prediction phases.The procedure then backpropagates the masked prediction loss and takes an optimizer step.
- Sleep mechanism: During consolidation, all-zero loss-mask chunks trigger N looped applications of the model blocks without prediction loss computation.Prediction chunks instead use one block pass and masked cross-entropy loss.
- Sleep mechanism: At full context windows, the model performs N recurrent passes to update SSM fast weights before evicting the KV cache.With N = 1, this becomes a vanilla SSM-attention hybrid.
- Inference and training: After fast-weight refinement and cache eviction, the model processes the next context window and predicts from refined memory plus current context in one forward pass.Training backpropagates through the complete recurrent computation graph.
6 Experiments
Across cellular automata, multi-hop retrieval, and math reasoning, increasing offline recurrence improves learning or accuracy, especially when evicted-context reasoning is deeper. The method adds consolidation-time computation while keeping prediction-time computation fixed, but recurrent depth increases training cost.
- Cellular automaton: At t = 32, two, three, and four sleep loops raise exact accuracy from about 10% without loops to approximately 20%, above 30%, and above 30%, respectively.The context length, eviction rule, and prediction-phase computation remain fixed, so the added computation occurs during consolidation.
- Depo: Increasing offline loops accelerates Depo learning for queries requiring 4 or more hops, with four loops beginning to improve the hardest 16-hop task within the training budget.The 1-loop model stalls on 4-hop and harder queries, while the 2-loop model stalls on 8-hop and harder queries.
- GSM-Infinite: On GSM-Infinite, additional offline recurrence improves final accuracy and learning speed most clearly for problems requiring six or eight arithmetic operations.For Jet, six loops improve six-operation accuracy from 0.742 to 0.812 and eight-operation accuracy from 0.351 to 0.388; for Ouro, four loops improve them from 0.419 to 0.615 and 0.210 to 0.272.
- Training throughput: Recurrence across context windows prevents full sequence-axis parallelization, while sufficiently large windows can preserve wall-clock throughput and training cost grows roughly linearly with N.The throughput trade-off is attributed to serial dependencies between windows and the additional recurrent sleep passes.
7 Discussion and Limitations
The method preserves single-pass prediction latency by moving recurrent computation into consolidation, but this makes training slower and potentially unstable. Its sequential structure may suit tasks whose solutions are themselves sequential.
- Limitations: Training requires N deeper forward and backward passes, which can make optimization slow and unstable.The extra recurrent computation is shifted to consolidation to preserve prediction-phase latency.
- Discussion: Sleep introduces sequentiality across both context and depth dimensions, matching the sequential structure of the evaluated tasks.The authors connect this sequentiality to the gains observed on their tasks.
8 Conclusion
The paper proposes sleep as recurrent consolidation of evicted context into fast weights. Across synthetic and mathematical reasoning tasks, longer sleep improves deep sequential computation over past context.
- Conclusion: Sleep performs multiple recursive forward passes to iteratively refine fast weights before evicting the corresponding context from the attention cache.The refined memory is then used for inference after the context is no longer available in the cache.
- Conclusion: Increasing sleep duration N improves the ability to perform deep sequential computation over evicted context across the evaluated tasks.The paper reports this pattern on controlled synthetic tasks and a more realistic mathematical reasoning benchmark.
Broader Impact
The work studies memory consolidation and reasoning in language models as ingredients for more capable long-context systems. Its evaluation is methodological, using controlled synthetic tasks and modest-scale pretrained models.
- Broader Impact: The paper studies memory consolidation and reasoning in language models for building more capable long-context systems.These are identified as important ingredients for long-context capability.
- Broader Impact: The contribution is primarily methodological and is evaluated on controlled synthetic tasks and modest-scale pretrained models.The authors do not expect the risks to exceed those of other work in this area.