PaperlayerView exact revision on arXiv

Source-linked AI summary

Continual Learning in Transition

Zhiyan Hou, Dan Zhang, Tao Feng, Liyuan Wang, Wei Li, Xiangzhao Hao, Hongyan An, Junfeng Fang, Haokai Ma, Zhaohui Xu, Xinyu Tang, Haiyun Guo, Jinqiao Wang, Tat-Seng Chua

arXiv:2608.06216v2cs.LGcs.AI

TL;DR

Classical continual learning’s parameter-level, training-time taxonomy no longer accommodates emerging mechanisms, stages, and capability carriers. This survey introduces a When–Where–How framework and finds that LLM-era methods extend continual learning beyond parameters across the model lifecycle and external harness.

  • Problem

    Classical continual learning’s training-time, parameter-level taxonomy no longer naturally accommodates emerging update mechanisms, lifecycle stages, and capability loci.

  • Method

    The survey organizes continual capability evolution with three axes: How updates occur, When they occur, and Where capabilities reside.

  • Results

    The taxonomy organizes existing methods, identifies unfilled coordinates, and shows that LLM-era approaches increasingly extend continual learning beyond classical parameter-centric settings.

  • Takeaways & Limitations

    Continual learning is shifting from an isolated parameter-updating problem toward a system-level question of how capabilities are organized across models and harnesses.

  • Takeaways & Limitations

    Long-horizon agents face compounding errors across memory, skills, protocols, and parameters, while many harness operations remain non-differentiable.

Abstract

from arXiv · show

Classical continual learning (CL) has primarily focused on enabling models to update and retain knowledge through parameter-centric mechanisms, e.g., training strategies, architectural designs, and weight adaptation. However, emerging paradigms are reshaping the scope of CL beyond this traditional model adaptation view. For instance, on-policy learning broadens the space of update mechanisms; test-time training extends CL from the training phase to inference; and external harness components such as memory, skill libraries, and interaction protocols extend the evolutionary boundaries of model capabilities far beyond the static parameter space. Collectively, these developments indicate a transition from parameter-centric learning toward system-level adaptation. To characterize this transition, we examine the evolution of continual learning through three dimensions: When, How, and Where learning occurs. The How dimension encompasses off-policy, on-policy, and beyond-gradient optimization mechanics. The When dimension captures evolution across pre-training, post-training, and inference-time stages. The Where dimension delineates updates occurring within internal parameters versus external structural constraints. Anchored by this tri-axial framework, we systematically survey representative methods, trace the ongoing transition of continual learning, and discuss the key challenges, broader implications, and future directions arising from this paradigm shift.

1 Introduction

The survey reframes continual learning as continual capability evolution for LLMs and agentic AI, organized around when, where, and how capabilities change. Its three-axis taxonomy spans lifecycle stages, internal parameters versus external harness components, and off-policy, on-policy, and beyond-gradient update mechanisms.

  • Motivation: Classical continual learning’s parameter-centric taxonomy no longer fully captures the expanding methods and application scenarios of LLMs and agentic systems.Classical methods center on catastrophic forgetting and the stability–plasticity trade-off, including replay, gradient-based, parameter-isolation or architecture-expansion, and regularization approaches.
  • How: The How axis extends update mechanisms from off-policy learning to on-policy learning and beyond-gradient approaches such as model merging, zeroth-order optimization, heuristic learning, and prompt evolution.On-policy paradigms including RLHF and RLVR are described as advantageous for mitigating forgetting relative to classical off-policy updates.
  • When: The When axis covers capability evolution across the full model lifecycle, from continual pre-training and multi-stage post-training to inference-time adaptation.This framing includes test-time training and post-deployment inference-time evolution.
  • Where: The Where axis shifts capability accumulation from model parameters toward an external harness comprising persistent memory, reusable skills, and adaptive interaction protocols.Examples include read–write long-term memory, self-generated and composable skill libraries, and protocols refined through reflection and feedback.
  • Survey scope and implications: The survey locates representative methods in a common When–Where–How space and uses its concentrated, sparse, and empty regions to identify challenges and opportunities for self-evolving agent systems.The reviewed methods span continual pre- and post-training, test-time training, reinforcement-learning-based alignment, model merging, memory systems, skill libraries, and prompt evolution.

2 Revisiting Continual Learning

Classical continual learning updates a model across non-stationary task distributions while balancing current-task learning against retention of prior knowledge. Its methods largely assume that learning occurs during training, capability resides in parameters, and updates use gradient optimization over supplied data.

  • Classical formulation and objectives: Classical continual learning processes a sequence of tasks with distinct data distributions, updating the model while historical data are inaccessible or restricted.Its ideal objective is to fit the current task while keeping aggregate expected loss across observed tasks low.
  • Classical formulation and objectives: Catastrophic forgetting arises when updates improving the current task perturb representations or decision boundaries supporting earlier tasks.This creates the central tension between acquiring new knowledge and preserving previous-task performance.
  • Classical formulation and objectives: Classical continual learning balances plasticity with stability and requires both intra-task and inter-task generalizability under distribution shifts.The literature also frames continual reinforcement learning as retaining and transferring behavior across non-stationary environments.
  • Major method families: Major method families include replay-based approaches that approximate access to past experience and regularization-based approaches that constrain parameter or functional updates.Examples include memory buffers, generative replay, Fisher-based penalties such as EWC, SI, and MAS, and functional distillation such as LwF.
  • Implicit assumptions and taxonomy: Despite different mechanisms, classical methods share assumptions that capability acquisition and retention occur during training, capability resides mainly in parameters, and updates use gradient optimization over supplied data.In the survey taxonomy, this reference setting corresponds to post-training in When, model parameters in Where, and off-policy gradient updates in How.

3 The Three Dimensions of Continual Learning · 3.1 Overview

The paper frames continual learning through three complementary dimensions—when learning occurs, where capability is accumulated, and how updates are driven. This framework extends continual learning from preventing forgetting during sequential parameter training to capability evolution across the lifecycle of large models and agent systems.

  • 3.1 Overview: Continual learning methods are examined according to when capability evolution takes place, where acquired capability is accumulated, and how updates are driven.These three questions define the paper’s organizing framework.
  • 3.1 Overview: The three axes provide complementary perspectives rather than mutually exclusive categories.A single method can be analyzed along all three dimensions simultaneously.
  • 3.1 Overview: A method’s learning stage, capability carrier, and update mechanism jointly determine how continual capability evolution is realized.The framework treats these dimensions as interacting characteristics of each method.
  • 3.1 Overview: Classical continual learning is characterized as post-training updates to model parameters through off-policy gradient learning.This is presented as the canonical classical setting within the three-axis framework.
  • 3.1 Overview: The framework clarifies that continual learning is no longer confined to preventing forgetting during sequential parameter training.Its scope is described as broader than the classical parameter-training problem.
  • 3.1 Overview: Continual learning increasingly concerns capability evolution across the full lifecycle of large models and agent systems.The subsequent sections examine the three perspectives individually and then analyze their combinations in the joint space.

3.2 When: Capability Evolution across the Model Lifecycle

The When axis frames continual learning as capability evolution across the model lifecycle: continued pre-training, longitudinal post-training, and post-deployment inference. These stages differ in update targets, feedback signals, and stability requirements, with inference-time learning distinguished by persistent adaptation during use.

  • Lifecycle stages: Continual capability evolution spans continued pre-training, post-training, and inference after deployment rather than a single sequential training process.Continued pre-training adapts models to evolving corpora; post-training includes instruction tuning, preference optimization, and RL; inference-time learning occurs during deployment.
  • Lifecycle stages: Each lifecycle stage exposes distinct update targets, feedback signals, and stability requirements.Pre-training emphasizes knowledge renewal under changing distributions, whereas post-training emphasizes alignment, task adaptation, and forgetting control.
  • Post-training: Post-training is a longitudinal pipeline in which successive instruction, preference, or reward-alignment updates can erode previously established capabilities.Continual fine-tuning studies report degradation in domain knowledge, reasoning, and reading comprehension, with stronger effects as model scale increases from 1B to 7B.
  • Inference-time adaptation: Inference-time continual learning differs from ordinary test-time computation by writing information obtained during inference back into the model or surrounding system.This enables deployment-time accumulation from long input streams, shifting requirements, evolving task distributions, and feedback unavailable during training.
  • Inference-time adaptation: Methods such as TTT-LM, TTT-E2E, and TTRL demonstrate post-deployment capability evolution through fast-weight, parameter-update, and reward-driven test-time adaptation.These approaches use contextual information, next-token prediction, or reward signals during inference, and persistently accumulated signals affect subsequent predictions.

3.3 Where: Capability Carriers from Parameters to the Harness

The Where axis tracks how continual-learning capabilities move from model parameters to external harness components. This harness comprises memory, skills, and protocols that can be updated without changing backbone weights.

  • Parameters to Harness: The Where axis distinguishes capability storage and updates within model parameters from accumulation in the external harness surrounding the model.The LLM era extends continual learning beyond the parameter space.
  • Parameter Carriers: Parameter-centric methods increasingly use modular, parameter-efficient components instead of monolithic full-parameter fine-tuning to limit interference.O-LoRA enforces orthogonal task directions, while Progressive Prompts assigns separate soft prompts to tasks.
  • Harness Carriers: The harness contains three capability carriers: memory stores information, skills provide executable units, and protocols govern interaction rules and formats.These objects can be read, extended, or rearranged without updating backbone weights.
  • Memory: Memory enables capability accumulation by writing facts, experiences, and intermediate states to external storage for retrieval in later inference.This preserves historical context without modifying backbone weights.
  • Skills and Protocols: Skills extend an agent’s action repertoire through externally callable execution units, while protocols evolve behavioral rules and communication conventions without backbone updates.Both carriers support continual learning at the execution or interaction layer.

3.4 How: Update Mechanisms from Off-Policy Gradients to Gradient-Free Learning

The How axis of continual learning expands from off-policy gradient updates to on-policy reward-driven learning and gradient-free mechanisms. Across these regimes, the central concern is controlling drift from prior capabilities.

  • Update mechanisms: The LLM era broadens update mechanisms from off-policy data to on-policy rollouts and beyond-gradient methods that never compute analytic loss gradients.Off-policy learning uses externally supplied signals, whereas on-policy learning generates data with the policy being updated.
  • Update mechanisms: Continual-learning updates are evaluated by how strongly they perturb previously acquired behavior, making drift control central to the How axis.This concern applies across update regimes rather than only to a particular optimization method.
  • On-policy learning: On-policy learning generates training data from the current policy under reward signals, with RLHF and RLVR serving as principal reward-driven post-training instances.Its continual-learning significance is described as structural, linking update data generation directly to the policy being optimized.
  • Gradient-free learning: Gradient-free learning accumulates capability through model merging, forward-only optimization, and external harness updates without backpropagating analytic loss gradients.Model merging operates in weight space, while heuristic learning revises memory, skills, or prompts from feedback.
  • Gradient-free learning: Forward-only zeroth-order updates can mitigate catastrophic forgetting, with their implicit smoothing paralleling the stability associated with small-KL on-policy reinforcement learning.ZeroFlow is described as matching or exceeding first-order fine-tuning across forgetting benchmarks, but the supplied passage truncates the exact result.

3.5 Cross-Dimensional Method Profiles

Cross-dimensional profiles complement the one-dimension-at-a-time organization by showing how methods combine learning carriers, mechanisms, and stages. These profiles also capture trajectories over time, while the figures illustrate recurring combinations rather than exhaustively estimating research density.

  • Cross-Dimensional Profiles: Figure 3 assigns each representative method to the dimension best capturing its salient carrier or mechanism, while Table C.1 records its complete cross-dimensional profile.Primary placement is therefore not a complete characterization of a method.
  • Cross-Dimensional Profiles: Reflexion combines inference-time evolution, harness-based memory, and beyond-gradient learning; TTRL combines inference-time parameter adaptation with on-policy learning.Reflexion uses reflective text generation and direct memory writing.
  • Temporal Profiles: SKILL0 evolves from accumulating reusable capabilities in a harness-based skill library to internalizing selected skills into model parameters through a curriculum.It is represented as a Harness (Skills)-to-Parameters trajectory rather than a method with one fixed carrier.
  • Scope and Limitations: The figures are illustrative, not exhaustive: method counts do not estimate research density, and unoccupied regions do not show that relevant work is absent.They instead illustrate recurring combinations and motivate comparatively underexplored future questions.

4 Future Outlook and Discussion

The outlook frames continual learning as a transition from model-centric updates to system-level capability management across parameters, memory, context, skills, protocols, tools, and environments. It emphasizes coordinated model–harness evolution, verifiable rewards, active forgetting, and long-horizon robustness as priorities on the path toward AGI.

  • Capability carriers: Capability carriers have moved outward from parameter tensors to external memory and context, each differing in capacity, compressibility, stability, and update cost.Parameters are costly to update but provide stable internalized recall, whereas external memory accumulates facts and experience.
  • Capability carriers: External memory decays over long dialogue and task sequences, making active forgetting necessary to manage stale, low-quality, and inconsistent entries.Retrieval degradation includes stale-document dominance, accumulating low-quality entries, drifting relevance scores, and cross-session inconsistency.
  • Coordinated evolution: Hand-tuned harness engineering improves capability functionally but does not constitute a self-driven continual-learning mechanism, motivating coordinated model–harness updates.Capability is distributed across parameters, external memory, skills, and protocols, so neither model-only nor harness-only updates fully captures its organization.
  • Coordinated evolution: Coordination is a scheduling problem across context, memory, skills, and parameters, while forgetting spans parameter overwriting, retrieval decay, context overflow, and skill or protocol mismatch.These forgetting modes differ in timescale, observability, and reversibility, requiring carrier-aware management.
  • Long-horizon agents: Long-horizon agents expose compounding errors across retrieval, skill invocation, and later decisions, making verifiable error models and drift-resistant mechanisms central research challenges.Reflexion and TTRL demonstrate feasible correction routes, but both remain single-mechanism approaches without a unified framework.
  • AGI priorities: The proposed AGI trajectory prioritizes continual learning for frontier models, jointly learnable agent systems, verifiable rewards, and capability management, while judging genuine AGI still considerably distant.Longer context, larger memory, or more skills alone are not expected to close the remaining distance.

5 Conclusion

The survey reframes continual learning in the LLM era as boundary extension across the interrelated dimensions of When, Where, and How. It also acknowledges taxonomy ambiguity and coverage limitations in this rapidly evolving area.

  • Core framework: Continual learning is analyzed as boundary extension across the three dimensions of When, Where, and How.Classical continual learning occupies a specific coordinate, while LLM-era methods extend one or more axes.
  • Core framework: LLM-era methods increasingly depart from the classical coordinate along multiple axes and move into the coordinate space’s interior.The survey links these multi-axis departures to methods discussed across Sections 3.2–3.5.
  • Limitations: The taxonomy has boundary crossings, so some cross-axis methods admit alternative classifications beyond the adopted primary-axis convention.The authors identify this convention as one limitation of the survey.
  • Limitations: Coverage may lag recent work because the survey includes only publicly available research before its submission deadline.The authors note that LLM continual learning is rapidly iterating.

A Glossary of Abbreviations

The survey provides abbreviation references for readability and disambiguates selected short labels used in Figure 2.

  • Abbreviations: Table A.1 summarizes the abbreviations used throughout the survey.It is included to support readability.
  • Figure 2 labels: Table A.2 lists Figure 2 labels requiring disambiguation or clarification because their displayed short forms are not explicit in the corresponding paper titles.The table pairs selected short labels with their corresponding references.

B Benchmark Catalogue

The benchmark catalogue is representative rather than exhaustive and profiles evaluation protocols across the When, Where, and How dimensions. It spans memory- and skill-based harnesses, beyond-gradient learning, and shared self-evolution protocols.

  • B Benchmark Catalogue: Table B.1 organizes representative benchmarks by When, Where, and How update regimes, which are profile labels rather than mutually exclusive categories.The catalogue is explicitly representative rather than exhaustive.
  • B Benchmark Catalogue: MemoryBench evaluates fine-grained changing facts using state-tracking accuracy, adaptation delay, distraction robustness, and memory-system reliability.Its setting includes OAKS-BABI and OAKS-Novel, where individual facts may change repeatedly.
  • B Benchmark Catalogue: EvoMemBench assesses beyond-gradient learning through feedback utilization, adaptation and retention, task effectiveness, and computational efficiency.
  • B Benchmark Catalogue: The catalogue also includes lifelong skill discovery, patching, and reuse over 166 sequential tasks from 20 task families, with transfer, retention, evolution, reliability, and cost evaluations.A shared protocol records training, frozen validation, held-out ID/OOD tests, replay diagnostics, snapshots, and costs.

C Representative Cross-Dimensional Method Profiles

Representative methods occupy combinations of the When, Where, and How dimensions rather than fixed single coordinates. Their profiles span parameter updates and external harness adaptation across pre-training, post-training, and inference-time learning.

  • Cross-dimensional profiles: Table C.1 uses non-exclusive labels across When, Where, and How, with arrows marking temporal carrier trajectories rather than strict single-valued coordinates.The table is representative, not an exhaustive literature census, and its row counts do not estimate research density.
  • Parameter-centric methods: Pre-training and post-training parameter methods include temporal language-model updates, interference-mitigating adapters, modular experts, self-distillation, and policy-generated self-distillation.Examples include TimeLMs, O-LoRA, LoRAMoE, SDFT, and COPR.
  • Beyond-gradient and protocol methods: Post-training methods also extend beyond ordinary gradients through model merging, zeroth-order updates, evolutionary prompt refinement, and reflective protocol evolution.AIMMerging combines capabilities through model merging, ZeroFlow avoids ordinary back-propagated gradients, while Promptbreeder, GEPA, and AgentEvolver adapt prompts or surrounding protocols.
  • Inference-time adaptation: Inference-time methods update parameters or external harnesses through self-supervision, test-time reinforcement learning, reorganized memory, verbal reflection, and accumulated executable skills.TTT-LM and TTRL adapt parameters, whereas A-MEM, Reflexion, Voyager, and SkillWeaver modify memory or skill structures during use.
  • Coupled carriers: Some methods couple internal and external evolution: SkillRL co-evolves policy parameters with a skill library, AgentEvolver combines parameters with memory, and SKILL0 transfers skills into parameters.SKILL0 first accumulates reusable skills and then internalizes selected capability through an on-policy curriculum.
Loading 2608.06216v2…