PaperlayerView exact revision on arXiv

Source-linked AI summary

Language Models Need Sleep: Learning to Self-Modify and Consolidate Memories

Ali Behrouz, Farnoosh Hashemi, Adel Javanmard, Vahab Mirrokni

arXiv:2606.03979v2cs.LGcs.AI

TL;DR

Existing LLMs are largely static and do not reliably transfer short-term in-context knowledge into long-term parameters. The paper introduces Sleep, combining memory consolidation through knowledge seeding with RL-based dreaming, and reports consistent gains across several learning and generalization tasks.

  • Problem

    LLMs remain largely static after deployment, limiting continual acquisition of capabilities and transfer of in-context knowledge into long-term parameters.

  • Method

    Sleep combines upward knowledge seeding that distills short-term knowledge into long-term parameters with dreaming, which uses self-generated training to improve capabilities while controlling interference.

  • Results

    Sleep yields consistent gains across long-context understanding, knowledge incorporation, few-shot reasoning, and continual learning, achieving the best results among evaluated settings on SQuAD.

  • Takeaways & Limitations

    The experiments support incorporating a sleep stage into LLM learning to consolidate memories and improve capabilities across continual and generalization tasks.

  • Takeaways & Limitations

    The approach assumes the relevant MLP blocks can be treated as sparse mixture-of-experts modules.

Abstract

from arXiv · show

The past few decades have witnessed significant advances in the design of machine learning algorithms, from early studies on task-specific shallow models to more general deep Large Language Models (LLMs). Despite showing promising results in tasks that require instant prediction or in-context learning, existing models lack the ability to continually learn and effectively transfer their temporal in-context knowledge to their long-term parameters. Inspired by human learning process, we introduce a ''Sleep'' paradigm that allows the models to continually learn, distill their short-term fragile memories into stable long-term knowledge with replay, and recursively improve themselves with ''Dreaming'' process. In more detail, sleep consists of two stages: (1) Memory Consolidation: an upward distillation process, called Knowledge Seeding, where the memories of a smaller-self are distilled into a larger network to provide more capacity while preserving the knowledge. As a proof of concept, we present a new Generalized Distillation process for {Knowledge Seeding} (i.e., the combination of on-policy distillation with Reinforcement Learning (RL)-based imitation learning); (2) Dreaming: a self-improvement phase, where the model uses RL to generate a curriculum of synthetic data to rehearse new knowledge and refine existing capabilities without human supervision. Our experiments on long-horizon, continual learning, knowledge incorporation, and few-shot generalization tasks support the importance of the sleep stage.

1 Introduction

LLMs can adapt through in-context learning, but their newly acquired knowledge remains confined to the context window and is not consolidated into stable long-term parameters. The paper introduces a continual-learning “Sleep” paradigm with memory consolidation and Dreaming-based self-improvement, including knowledge seeding, generalized distillation, and random expert selection.

  • LLMs are largely static after deployment, successfully performing learned tasks but lacking continual parameter-level adaptation.
  • In-context learning enables efficient adaptation and continual learning, but newly acquired knowledge disappears when the session or context ends.
  • The proposed Sleep paradigm alternates active periods that process new data with Sleep periods that consolidate recent memories internally.
  • Memory Consolidation: Knowledge Seeding distills knowledge from smaller models into a larger model, preserving existing knowledge while exploiting greater capacity.
  • Dreaming: The method adds Dreaming-based self-improvement through generalized distillation with imitation learning and random low-rank expert selection in Mixture of Experts.

2 Preliminaries and Problem Formulation

This section establishes notation and frames the model as a continuum memory system with modules updated at different frequencies. It also motivates replacing static training/testing with periodic wake–sleep cycles for lifelong adaptation.

  • 2.1 Notation: Notation defines x∈R^(L×d_in) as the input, K and V as keys and values, Q as query matrices, and L as sequence length.Bold lowercase and uppercase letters denote vectors and matrices, while subscripts index states by time.
  • 2.1 Notation: Parameter sets are sorted by update frequencies f_1 ≥ ... ≥ f_c, distinguishing modules according to how often their weights change.Superscripts identify module update frequencies or distinguish hyperparameter instances.
  • 2.2 Continuum Memory System: Attention or other sequence modules provide short-term associative memory, whereas MLP layers encode pretrained knowledge and form a spectrum of longer-term memories.Higher-frequency updates favor adaptation but can overwrite older knowledge; later CMS blocks represent progressively longer-term storage.
  • 2.2 Continuum Memory System: Continuum Memory System models a sequence component followed by MLP layers, each updated at its own frequency.The slowest module’s update interval defines one unit of time, and other components’ rates are measured relative to it.
  • 2.2 Continuum Memory System: For fast-weight programs, fast weights update L times per slow-weight step, giving them update frequency L.This example illustrates how frequency measures parameter updates per unit of time.
  • 2.2 Continuum Memory System: CMS uses MLP blocks MLP(f_1)(·), ..., MLP(f_k)(·), with block parameters updated according to their associated frequencies and chunk sizes.The formulation permits task-dependent optimization objectives and includes Transformer designs as a special case.
  • 2.3 Sleep Terminology in the Literature: Prior sleep-inspired work uses dreaming or offline replay to rehearse experiences and improve robustness, especially for long-horizon tasks.The section situates these approaches within earlier sleep, replay, and offline-learning literature.
  • 2.3 Sleep Terminology in the Literature: The paper proposes replacing static training/testing with a continuous periodic wake–sleep lifecycle for lifelong adaptation.Existing continual-learning systems still alternate between parameter updates during training and evaluation during testing.

3 The Sleep Paradigm

The Sleep paradigm divides continual learning into wake, which processes external inputs, and sleep, which suspends them for internal self-improvement, memory consolidation, and knowledge abstraction. Sleep comprises memory consolidation and dreaming, including self-Knowledge Seeding and reinforcement-learning-based imitation.

  • 3 The Sleep Paradigm: Continual learners alternate between wake time, when they process new external data, and sleep time, when they stop receiving new data and consolidate recent memories using self-generated data.Sleep therefore complements, rather than replaces, online consolidation during the active phase.
  • 3 The Sleep Paradigm: Sleep has two stages: memory consolidation, which transfers fragile short-term memories into more stable parameters, and dreaming, which supports self-improvement.The process repeatedly transfers knowledge across memory blocks with different update frequencies.
  • Memory Consolidation: Knowledge Seeding distills knowledge from one or more smaller models into a larger model, preserving existing knowledge while exploiting the larger model’s capacity.Self-Knowledge Seeding expands the student and updates only its expanded parameters to reduce interference with transferred and previously stored knowledge.
  • Memory Consolidation: Learning to Imitate adds reinforcement learning to distillation so the student learns to use transferred knowledge and imitate the teacher’s sampling behavior.It samples prefixes from teacher-generated dreams, has the student complete them, and rewards semantic or token-level similarity.
  • Dreaming: Dreaming generates and selects synthetic data for self-improvement, but repeated self-improvement in continual learning risks catastrophic forgetting.The method assigns dream importance scores and retains the Top-k dreams plus b random samples for diversity.

4 Empirical Results

The empirical results evaluate Sleep’s memory consolidation and full sleep process across continual learning, long-context understanding, reasoning, knowledge incorporation, and few-shot learning. Across these settings, Hope improves continual and long-context capabilities, outperforms comparison methods in several tasks, and achieves an 80% few-shot success rate.

  • Memory Consolidation: Hope augments Llama-3B and Llama3-8B with on-policy self-distillation to study class-incremental learning on CLINC, Banking, and DBpedia.The comparison includes standard in-context learning under the same continual pre-training process.
  • Memory Consolidation: Hope’s consolidation schedule is evaluated for long-context question answering and multi-key retrieval on LongHealth, QASPER, and MK-NIAH against ICL, DuoAttention, and Cartridges.The study compares variants with different numbers of sleep phases to isolate the effect of consolidation levels.
  • Learning a New Language In-Context: Hope-1, Hope-2, and Hope-3 are compared with ICL for independent and sequential learning of unseen-language translation, reporting ChRF scores for Manchu→English and Kalamang→English.The task combines MTOB and Manchu and tests whether sequential learning preserves translation performance across languages.
  • Memory Consolidation: Memory consolidation improves mathematical reasoning over SFT and GRPO baselines, while ablations show that all evaluated consolidation components contribute positively.The reasoning results are reported in Table 2, and the component ablations are reported in Figure 1.
  • Knowledge Incorporation: The full Sleep process achieves the best reported mean no-context SQuAD accuracy across single-passage and continued-pretraining settings, including comparison with SEAL.The paper attributes the result to more effective memory storage and improvements built on SEAL.
  • Few-Shot Learning: Sleep achieves a 80% success rate on the filtered few-shot ARC evaluation using Llama-3.2-1B, with 11 training tasks and 8 held-out evaluation tasks.The protocol adapts prior few-shot ARC experiments to the Sleep paradigm.

5 Conclusion · A Related Work · A.1 Parameter-Efficient Adaptation and Composition

The paper concludes that Sleep combines knowledge seeding and dreaming to transfer short-term knowledge into long-term parameters and improve capabilities across several learning settings. Related work covers parameter-efficient adaptation methods and the composition or routing of multiple adapters.

  • 5 Conclusion: Sleep consists of knowledge seeding and dreaming for continual learning in Large Language Models.Knowledge seeding transfers short-term, in-context knowledge into lower-frequency, long-term parameters, while dreaming uses self-generated training.
  • 5 Conclusion: Knowledge seeding is an upward distillation process that transfers short-term in-context knowledge into long-term parameters.The paradigm is designed to preserve knowledge while moving it from short-term to lower-frequency model parameters.
  • 5 Conclusion: Dreaming uses self-generated training to improve capabilities while controlling interference.This stage supports self-improvement without relying on additional human-generated supervision.
  • 5 Conclusion: Sleep yields consistent gains across long-context understanding, knowledge incorporation, few-shot reasoning, and continual learning.The conclusion summarizes experimental improvements across all four listed evaluation settings.
  • A.1 Parameter-Efficient Adaptation and Composition: Parameter-efficient fine-tuning adapts Large Language Models by optimizing minimal auxiliary parameters while freezing the backbone.This approach reduces the set of parameters updated for task-specific adaptation.
  • A.1 Parameter-Efficient Adaptation and Composition: LoRA injects trainable low-rank matrices into linear projections, while prefix and prompt-tuning steer computation through learnable virtual tokens.These are prominent parameter-efficient adaptation methods identified in the related-work discussion.
  • A.1 Parameter-Efficient Adaptation and Composition: Adapter composition research includes weighted composition through in-context learning and retrieval-based routing that selects relevant LoRA modules per input.These methods extend adaptation beyond single-task settings by combining or dynamically selecting adapters.

A.2 Knowledge Injection, Distillation, and Self-Improvement · A.3 Efficient Context Processing

The related work surveys parameter-level knowledge injection, distillation, and model-driven self-improvement, emphasizing Sleep’s upward self-distillation and Generalized Distillation formulation. It also reviews efficient context processing through prompt/KV-cache compression and attention architectures that reduce memory demands.

  • A.2 Knowledge Injection, Distillation, and Self-Improvement: Parameter-level knowledge injection reduces inference-time retrieval reliance through document-specific LoRA adapters and prompt distillation from teacher-generated data.Prompt distillation trains students on teacher-generated question-answer pairs or synthetic conversations.
  • A.2 Knowledge Injection, Distillation, and Self-Improvement: Cartridges combine parameter-efficient fine-tuning and distillation by pre-training reusable KV cache adapters, achieving in-context learning quality with reduced serving costs.The adapters are trained with a self-study objective.
  • A.2 Knowledge Injection, Distillation, and Self-Improvement: Model-generated signals support self-improvement through reinforcement learning on verifiable rewards, self-rewarding mechanisms, and meta-learning of adaptation strategies.SEAL extends this direction by learning how to generate self-edits.
  • A.2 Knowledge Injection, Distillation, and Self-Improvement: Sleep differs from concurrent on-policy distillation studies by using upward self-distillation to unlock parameters and by basing the process on Generalized Distillation.The passage identifies applications of concurrent work in self-distillation and continual learning.
  • A.3 Efficient Context Processing: Efficient long-context processing addresses its memory bottleneck by compressing the KV cache or modifying the attention architecture.This frames the two major solution families discussed in the section.
  • A.3 Efficient Context Processing: Prompt compression shortens inputs through hard-token filtering or compact soft-token embeddings, while KV-cache compression evicts non-essential keys or merges similar tokens.The passage presents prompt and runtime KV-cache compression as complementary memory-footprint reduction strategies.
  • A.3 Efficient Context Processing: Architectural approaches reduce attention memory through shared KV heads in MQA/GQA and linearized attention with fixed-size states independent of sequence length.The passage also introduces learning-to-compress architectures such as Titans and TTT, which use gradient-based updates.

A.4 Recent Concurrent and/or Later Work on On-Policy Distillation

Recent work develops on-policy self-distillation across self-improvement, continual adaptation, and deployment-specific settings, while exposing limitations that motivate Sleep. Sleep instead uses upward distillation from a smaller, higher-frequency memory module into expanded capacity.

  • Concurrent and Subsequent Work on On-Policy Self-Distillation: On-policy self-distillation uses the same model as teacher and student under different conditioning contexts, supervising the student on its own rollouts.Variants convert sparse rewards into dense token-level supervision, transfer reasoning to low-resource languages, or fine-tune directly on self-generated samples.
  • Continual, Experiential, and Online Adaptation: Self-distillation has also been applied to continual, experiential, and online improvement, including demonstration-based continual learning and deployment-time extraction of transferable interaction knowledge.SDFT uses a demonstration-conditioned on-policy teacher and reduces catastrophic forgetting compared with standard SFT.
  • Application-Specific OPSD Recipes: Application-specific recipes adapt OPSD to compress reasoning traces and convert pretrained next-token predictors into multi-token predictors using online self-distillation.CRISP/OPSDC uses a “be concise” prefix and per-token reverse-KL on student rollouts, while MTP uses a single online self-distillation objective.
  • Limitations of OPSD and Motivation for the Sleep Paradigm: Analyses find that privileged teacher information can suppress epistemic verbalisation, enabling fast in-domain optimisation but causing severe out-of-distribution degradation in mathematical reasoning.This failure mode motivates a paradigm that does not rely solely on privileged reconditioning of a fixed teacher.
  • Positioning of Our Work: Sleep differs by performing upward memory consolidation: a smaller, higher-frequency memory module is distilled into a strictly larger set of newly expanded low-rank experts.The approach reframes catastrophic forgetting as insufficient capacity and addresses it through gradual parameter growth between consolidation steps rather than reconditioning a fixed teacher.

B Additional Experimental Results and Details

The experiments follow each original benchmark’s settings, while adding five 64-dimensional MLP blocks without changing the active parameter count.

  • Experimental settings: The experiments use the original benchmark settings and refer to the cited studies for model details.The paper omits duplicated benchmark and model descriptions for space.
  • Model design: The design adds 5 MLP blocks with dimension 64 as additional parameters while keeping active parameters unchanged.The active parameter count matches the base model, either 8B or 3B.

B.1 Datasets

The evaluation uses three class-incremental intent or topic-classification datasets, plus three long-context benchmarks for question answering and multi-key retrieval.

  • Class Incremental Learning: Class-incremental learning is evaluated on CLINC, Banking, and DBpedia.These datasets cover multi-domain intent classification, single-domain banking intents, and Wikipedia-derived ontology classification, respectively.
  • Class Incremental Learning: CLINC150 contains 150 in-scope intents across 10 domains and 23.7K total queries, including 1.2K out-of-scope queries.It evaluates in-scope intent prediction and out-of-scope detection.
  • Class Incremental Learning: Banking77 contains 13,083 short customer-service examples labeled with 77 fine-grained intents and has a noticeably imbalanced intent distribution.Examples include card issues and PIN resets.
  • Class Incremental Learning: DBpedia uses a 70-class, level-2 classification setting with 10K training and 1K test instances sampled from Wikipedia-derived descriptions.The ontology categories include books, films, animals, and places.
  • The Effect of Levels on In-context Learning: Long-context question answering and multi-key retrieval are evaluated on LongHealth, QASPER, and MK-NIAH.LongHealth uses fictional clinical case records, QASPER uses full-text NLP papers, and MK-NIAH embeds multiple key–value facts in long contexts.

B.2 Additional Results: BABILong

This section evaluates Hope (Sleep) on BABILong against large, mid-scale retrieval-augmented, and small long-context models using the official training protocol for small models.

  • Evaluation setup: Hope (Sleep) is evaluated on the BABILong benchmark.The benchmark is attributed to Kuratov et al. 2024.
  • Baselines: The comparison includes GPT-4, GPT-4o-mini, and Llama-8B with RAG.GPT-4 and GPT-4o-mini represent large models, while Llama-8B with RAG represents a mid-scale model.
  • Baselines: Small-model baselines include RMT, ARMT, and Titans.All small models are fine-tuned using the official BABILong training protocol.

B.3 Additional Experiments: ARC

The ARC experiments adapt a prior few-shot protocol to the Sleep paradigm using Llama-3.2-1B, with filtered data split into training and held-out evaluation tasks. Each Sleep cycle consolidates previous memories before generating synthetic experience through dreaming.

  • Experimental Setup: ARC experiments use Llama-3.2-1B and adapt a prior few-shot protocol to the Sleep paradigm.The protocol follows prior work by Akyürek et al. (2024a) and Zweiger et al. (2025).
  • Experimental Setup: Filtering leaves 11 ARC tasks for training and 8 held-out tasks for evaluation.The filtering excludes tasks that remain unsolvable under standard configurations.
  • Sleep Procedure: Each Sleep cycle consolidates previous memories before the model dreams by generating synthetic experience.This ordering implements the paradigm’s consolidation-then-dreaming procedure.

B.4 Additional Details

This section presents the training configuration used for GRPO, SFT, and Sleep.

  • Table 5 summarizes the training configuration for GRPO, SFT, and Sleep.

B.5 Efficiency

Although SFT is 4x more efficient at the same number of steps, its performance is not comparable; at matched performance, Sleep requires less wall-clock time across all three evaluated tasks.

  • B.5 Efficiency: At matched AIME-24, AIME-25, and HMMT-25 performance, SFT requires 4.3x, 3.6x, and 4.8x more wall-clock time, respectively, than Sleep.With equal steps, SFT is 4x more efficient, but the methods do not achieve comparable performance.
Loading 2606.03979v2…