PaperlayerView exact revision on arXiv

Source-linked AI summary

Lifelong Learning of Large Language Model based Agents: A Roadmap

Junhao Zheng, Chengming Shi, Xidi Cai, Qiuke Li, Duzhen Zhang, Chenxing Li, Dong Yu, Qianli Ma

arXiv:2501.07278v2cs.AI

TL;DR

Existing LLM agents are generally static and therefore lack adaptation to new challenges, while lifelong learning must also address catastrophic forgetting and loss of plasticity. This survey systematically organizes techniques for lifelong learning in LLM agents around perception, memory, and action, and synthesizes evaluation, applications, and open challenges. It concludes that these components support continued adaptation and knowledge retention, while noting boundaries such as irreversible actions and alignment-related trade-offs.

  • Problem

    Existing LLM agents are typically static, while lifelong-learning systems must balance adaptation to new tasks with retention of previously learned knowledge.

  • Method

    The survey systematically examines lifelong learning in LLM agents through perception, memory, and action modules, alongside techniques, evaluation, applications, and research directions.

  • Results

    The survey synthesizes how state-of-the-art perception, memory, and action methods address continuous adaptation and knowledge retention in LLM agents.

  • Takeaways & Limitations

    The survey provides a roadmap for developing lifelong LLM agents that adapt across changing environments and learn from past experiences.

  • Takeaways & Limitations

    Current reasoning approaches often assume online states are retraceable, whereas real environments can contain irreversible actions such as payments.

Abstract

from arXiv · show

Lifelong learning, also known as continual or incremental learning, is a crucial component for advancing Artificial General Intelligence (AGI) by enabling systems to continuously adapt in dynamic environments. While large language models (LLMs) have demonstrated impressive capabilities in natural language processing, existing LLM agents are typically designed for static systems and lack the ability to adapt over time in response to new challenges. This survey is the first to systematically summarize the potential techniques for incorporating lifelong learning into LLM-based agents. We categorize the core components of these agents into three modules: the perception module for multimodal input integration, the memory module for storing and retrieving evolving knowledge, and the action module for grounded interactions with the dynamic environment. We highlight how these pillars collectively enable continuous adaptation, mitigate catastrophic forgetting, and improve long-term performance. This survey provides a roadmap for researchers and practitioners working to develop lifelong learning capabilities in LLM agents, offering insights into emerging trends, evaluation metrics, and application scenarios. Relevant literature and resources are available at \href{this url}{https://github.com/qianlima-lab/awesome-lifelong-llm-agent}.

1 INTRODUCTION

Lifelong learning enables AI systems to adapt and improve over time, but current systems face a stability–plasticity dilemma between retaining knowledge and learning new tasks. This survey maps lifelong learning for LLM agents through their perception, memory, and action capabilities, covering techniques, evaluation, applications, challenges, and future directions.

  • Lifelong learning enables systems to continuously adapt and improve as they encounter new situations.
  • Catastrophic forgetting and loss of plasticity create a stability–plasticity dilemma between retaining prior knowledge and adapting to new tasks or environments.
  • Unlike static LLMs, LLM agents perceive multimodal data, store information in memory, and act on their environments while learning from interactions.
  • Lifelong learning is intended to let agents adapt across gaming, web browsing, shopping, household tasks, and operating systems without separate agents for each context.
  • The survey systematically reviews lifelong-learning concepts, architectures, perception, memory, actions, evaluation metrics, benchmarks, applications, challenges, and future directions for LLM agents.
  • It presents a step-by-step guide to the design, challenges, and applications of lifelong learning in LLM agents while highlighting emerging research directions.

2 RELATED WORK

Prior surveys examined lifelong learning, continual learning in LLMs, and broad LLM-agent architectures, but little work specifically addressed lifelong learning in LLM-based agents. This survey focuses on how such agents can learn, evolve, and optimize performance over the long term.

  • Prior lifelong-learning surveys covered scenarios, continual-learning settings, training stages, benchmarks, and interdisciplinary approaches for autonomous agents.
  • Little prior work focused specifically on lifelong learning for LLM-based agents.
  • This survey collates literature on lifelong learning in LLM-based agents, covering their construction, application, and evaluation processes.
  • Prior LLM-agent surveys examined architectures, applications, evaluations, challenges, personal agents, and multi-agent systems.
  • The survey explains how to equip LLM-based agents with long-term learning and evolution while optimizing performance as experience accumulates.

3 BUILDING LIFELONG LEARNING LLM AGENTS

The survey formalizes lifelong learning for LLM agents across evolving tasks and environments, then organizes agent design around perception, memory, and action. Its framework emphasizes accumulating knowledge, retaining prior solutions, and improving performance over time.

  • 3.1 Formal Definition of Lifelong Learning for LLM-based Agents: The survey begins with formal definitions of environments, agents, tasks, trajectories, trials, rewards, and lifelong-learning objectives.LLM-agent environments are modeled as goal-conditional POMDPs, while tasks may differ in states, actions, goals, rewards, initial conditions, or environments.
  • 3.1 Formal Definition of Lifelong Learning for LLM-based Agents: LLM agents map observations and histories to actions, receiving numeric or textual, goal-conditional feedback from their environments.Observations may be textual, visual, or structured; actions are often natural-language commands or queries.
  • 3.1 Formal Definition of Lifelong Learning for LLM-based Agents: Lifelong learning requires agents to adapt across new tasks and environments while accumulating knowledge without forgetting previous solutions.The objective is to maximize expected performance across all tasks, using past experience to improve current and future task performance.
  • 3.2 Background and History of Lifelong Learning for AI Systems: The survey traces lifelong learning from foundational concepts and deep lifelong learning to LLMs and, most recently, LLM agents.It characterizes this evolution as increasing sophistication and practical applicability in dynamic, real-world settings.
  • 3.3 Overall Architecture: The proposed agent architecture has three essential modules: Perception, Memory, and Action, replacing the earlier “Brain” module with clearer modular functionality.The architecture is intended to support continual adaptation, integration, and behavior optimization across tasks and environments.

4 PERCEPTION DESIGN: SINGLE-MODAL PER-

Single-modal perception processes textual information, while related systems structure web content and use textual feedback to support action. The section frames broader text perception across complex environments as an ongoing research direction.

  • 4 PERCEPTION DESIGN: SINGLE-MODAL PER-: Single-modal perception enables agents to receive textual information from varied structures and environments.Textual inputs may support communication, environment understanding, and interaction in domains such as webpages or games.
  • 4 PERCEPTION DESIGN: SINGLE-MODAL PER-: HTML manipulation and screenshots are mainstream approaches for converting complex web information into structured inputs for LLM agents.Examples include simplifying HTML elements, selectively incorporating them into prompts, and summarizing HTML documents.
  • 4 PERCEPTION DESIGN: SINGLE-MODAL PER-: Text-based agents can recognize game elements and act through textual instructions, with some systems using reflection, explanations, or state management.These mechanisms incorporate prior plans or maintain environment state to improve interaction.
  • 4 PERCEPTION DESIGN: SINGLE-MODAL PER-: The section identifies strong text perception and adaptability across diverse complex environments as an important direction for human-like LLM agents.It presents broader text-input perception as a promising area for future research.

5 PERCEPTION DESIGN: MULTIMODAL PERCEP-

Lifelong multimodal perception requires agents to integrate new modalities while preserving knowledge from existing ones. The survey organizes approaches around modality-complete learning, modality-incomplete learning, cross-modal transfer, and catastrophic-forgetting mitigation.

  • Overview: Multimodal lifelong perception must integrate new modality inputs with existing modality experiences while maintaining stability on earlier tasks.The survey frames this as incremental perception for complex, changing environments.
  • Modality-Complete Learning: Modality-complete learning assumes consistent modalities across training and inference and emphasizes cross-modal knowledge transfer for new tasks.
  • Modality-Complete Learning: Modality-agnostic models encode arbitrary modalities into unified representations using mechanisms such as iterative attention, shared transformers, or multimodal feature aggregation.
  • Modality-Incomplete Learning: Modality-incomplete learning addresses missing inputs through incremental adapters, mixture-of-experts routing, and methods that predict or generate missing-modality representations.
  • Mitigating Catastrophic Forgetting: Regularization-based methods constrain parameter changes or model outputs, while replay-based methods preserve and reuse prior experiences to mitigate catastrophic forgetting.Regularization includes weight and function regularization; replay includes experience and generative replay.

6 MEMORY DESIGN: WORKING MEMORY

Working memory is the agent’s active workspace for prompts, context, interaction, and planning. Lifelong-learning techniques compress long inputs, manage context, support role-based interaction, and refine responses through feedback and prompt optimization.

  • Working Memory: Working memory comprises prompts, workspace memory, and user context that support real-time interaction, decision-making, planning, and action.
  • Prompt Compression: Prompt compression preserves important information while allowing more context within limited prompt lengths and reducing forgetting when old prompts are integrated with new ones.
  • Prompt Compression: Soft compression produces summary vectors or tokens, whereas hard compression filters redundant or non-essential text from prompts.AutoCompressors recursively generate summary vectors, while Selective Context and LongLLMLingua filter or restructure text.
  • Long Context Comprehension: Long-text processing uses context selection or context aggregation to improve comprehension while continuously adapting to newly processed text.Context selection chooses relevant chunks; context aggregation integrates information across regions and long-distance interactions.
  • Role-Playing: Role-playing agents learn from interactions by adopting defined characters or coordinating multiple agents with assigned roles and tasks.Multi-agent examples decompose collaborative work into subtasks or coordinate director and actor agents.
  • Self-Correction and Prompt Optimization: Self-correction and prompt optimization improve responses by using user instructions, model critics, iterative feedback, or evolutionary prompt search.

7 MEMORY DESIGN: EPISODIC MEMORY

Episodic memory stores an agent’s past events, interactions, outcomes, and feedback for later retrieval and learning. The survey emphasizes replay for retaining prior-task knowledge and structured storage for using experiences in future decisions.

  • Episodic Memory: Episodic memory stores specific events, encounters, interactions, outcomes, and environmental feedback that inform future actions and plans.
  • Data Replay: Replay reuses information from old tasks during new-task training to reduce forgetting and support lifelong learning.
  • Data Replay: Experience replay stores selected old samples, while generative replay trains an additional generator to create data for rehearsal.
  • Data Replay: Feature replay preserves feature distributions rather than raw data, addressing representation shift with efficiency and privacy advantages.
  • Continual Reinforcement Learning: Continual reinforcement learning replays stored experiences to learn from early memories, accelerate learning, improve training stability, and reduce poor temporal correlations.Prioritized replay selects experiences according to criteria such as temporal-difference error, state entropy, difficulty, or diversity.
  • Memory Representations: Episodic memory can store knowledge as triplets, databases, documents, or conversations, enabling exact or fuzzy retrieval, symbolic querying, document consistency, and conversational continuity.

8 MEMORY DESIGN: SEMANTIC MEMORY

Semantic memory stores and updates world knowledge for lifelong learning through continual knowledge-graph and document learning. The survey groups these techniques around replay, regularization, architectural adaptation, and incremental retrieval-corpus updates.

  • Semantic Memory: Semantic memory is an external mechanism for storing, retrieving, and progressively integrating world knowledge.
  • Continual Knowledge Graph Learning: Continual knowledge graph embedding updates new knowledge while preserving existing knowledge without repeatedly retaining and retraining on the entire graph.
  • Continual Knowledge Graph Learning: Replay-based graph methods store prior graph states or embeddings and selectively reuse them to guide learning when new relational knowledge arrives.DiCGRL decouples relational triplets into semantic components and replays relevant graph embeddings.
  • Continual Knowledge Graph Learning: Regularization-based graph methods constrain distances between old and new embeddings to preserve existing knowledge during graph expansion.
  • Continual Knowledge Graph Learning: Architecture-based methods adapt model structure through entity-agnostic representations or low-rank adaptation to improve transfer and parameter efficiency.
  • Continual Document Learning: Continual document learning addresses changing retrieval corpora through document-level or chunk-level updates that process newly added, modified, deleted, or unchanged content.Chunk-level updates use fingerprinting, persistence, and caching to identify content requiring processing.
  • Continual Document Learning: Incremental retrieval systems can preserve base retrieval capabilities while adding task-specific adapters for downstream continual learning.

9 MEMORY DESIGN: PARAMETRIC MEMORY

Parametric memory stores knowledge in an LLM’s internal parameters and can be updated through continual instruction tuning, self-evolution, knowledge editing, and alignment. These approaches seek to add new capabilities or information while retaining prior knowledge, but alignment introduces a trade-off with general performance.

  • Parametric memory: Parametric memory is the knowledge encoded in an LLM’s internal parameters through pretraining and fine-tuning.Unlike external memories, it is not explicitly accessible or directly retrievable.
  • Continual instruction tuning: Continual instruction tuning continuously adjusts model parameters with new instruction data to integrate information while avoiding catastrophic forgetting.The process supports ongoing optimization rather than a one-time modification.
  • Self-evolution: Self-evolution iteratively generates solutions, refines them using environmental feedback, updates the agent model, and evaluates it on new tasks.This cycle is presented as a way for agents to learn and improve throughout their lifecycle.
  • Continual knowledge editing: Continual knowledge editing updates outdated parametric information using new knowledge and can use external memorization, global optimization, or local modification.External memorization separates pre-trained and edited knowledge, while other methods update selected or broader model parameters.
  • Continual alignment: Continual alignment absorbs human feedback and preferences over time, but the alignment tax can reduce the model’s general capabilities.This trade-off makes preserving broad performance an important consideration during continual alignment.

10 ACTION DESIGN: GROUNDING ACTIONS

Grounding actions connect LLM agents to environments by interpreting observations and generating executable actions. Lifelong learning is needed because environment formats and action spaces vary across tools and webpages and can change over time.

  • Grounding actions: Grounding actions involve interpreting textual environment descriptions and generating text that the environment can parse into actions.Input grounding concerns perception of the environment, while output grounding concerns executable action generation.
  • Input grounding: Environment descriptions differ from pretraining text because they may use short phrases, JSON, or HTML, requiring continual adaptation to changing formats.The agent must continuously update its understanding as descriptions change.
  • Output grounding: Output grounding requires environment-specific action formats rather than ordinary text completion, creating a need to generate structured and executable outputs.The required outputs represent actions or environment-specific elements instead of merely describing intentions.
  • Tool environments: Tool environments require agents to understand documentation, often encoded as APIs or JSON, and invoke tools in the correct order.Methods simplify documentation, select relevant tools, or use tool-calling trajectories for fine-tuning and demonstrations.
  • Changing environments: Lifelong tool environments require continual adaptation to changing tools, and replay can mitigate catastrophic forgetting during continual fine-tuning.Web environments additionally challenge agents with lengthy webpage representations and irrelevant or difficult interaction actions.

11 ACTION DESIGN: RETRIEVAL ACTIONS

Retrieval actions supply LLM agents with external knowledge, demonstrations, and past experiences that support grounding, reasoning, and long-term consistency. The survey distinguishes semantic and episodic memory and highlights combining both with continuously improved retrieval sources.

  • Role of retrieval: Retrieval actions help agents handle growing action and observation histories while obtaining real-time knowledge from refreshed sources.They support long-term behavioral consistency in continuously changing environments.
  • Memory updating: The survey distinguishes a static database for retrieval from a dynamic database updated across tasks through trajectory processing and abstraction.Processed memory may be a corrected or simplified trajectory or a textual task summary.
  • Semantic memory: Semantic memory stores world knowledge and demonstrations that address pretrained LLMs’ lack of background information and examples.Retrieved knowledge can help select valid actions, while relevant demonstrations can improve grounding and planning.
  • Episodic memory: Episodic memory stores past experiences and helps agents use prior interactions and maintain long-term consistency despite finite context length.Experiences may be retrieved as trajectories, summaries, or other task records.
  • Summary: Combining semantic and episodic retrieval can further improve performance, while iterative retrieval and continuously improved retrieval sources offer additional directions.The combination parallels the use of long-term and short-term memory.

12 ACTION DESIGN: REASONING ACTIONS

Reasoning actions improve an agent’s ability to solve complex tasks by using feedback, structured strategies, and experience from current or previous episodes. Existing approaches span intra-episodic refinement, inter-episodic experience reuse, and curriculum learning, but tree-based methods may assume retraceable states.

  • Reasoning actions: LLM agents need enhanced reasoning because complex environments exceed the reasoning ability obtained from pretraining alone.Research addresses this through prompts and frameworks that refine reasoning over trials and episodes.
  • Intra-episodic reasoning: Intra-episodic reasoning uses experiences within one episode, including ReAct-style feedback and multi-step or multi-agent reasoning processes.ReAct continuously refines reasoning from environmental feedback, while other systems distribute reasoning stages across language models.
  • Environment-specific strategies: Environment-specific strategies simplify complex reasoning by restricting state transitions or dynamically composing handcrafted policies.These approaches reduce the reasoning burden in specialized environments.
  • Inter-episodic reasoning: Inter-episodic reasoning retrieves experiences from prior episodes, including successful trajectories, executable code, or textual summaries, to inform new tasks.Some approaches instead fine-tune the model using feedback from previous tasks.
  • Curriculum learning: Curriculum learning arranges tasks from easy to difficult so agents receive a steady progression of challenges and can use accumulated experience.The survey identifies this as another way to support continuous reasoning improvement.
  • Limitations: Tree-based reasoning methods may assume online states are retraceable, an assumption that fails for irreversible actions such as web payments.The survey calls for methods that accumulate reasoning techniques in more realistic settings.

13 EVALUATION OF LIFELONG LLM AGENTS

The survey evaluates lifelong LLM agents through metrics covering overall task performance, stability on historical tasks, and plasticity on future tasks, alongside benchmarks ranging from isolated skills to complex environments.

  • Evaluation metrics: Evaluation metrics assess overall accuracy, historical-task stability, and future-task plasticity using aggregated performance measures such as accuracy, pass rate, and win rate.These perspectives organize how lifelong learning ability is measured.
  • Evaluation metrics: Average performance (AP) measures overall task performance, while average incremental performance (AIP) additionally captures historical variation during task sequences.
  • Evaluation metrics: Forgetting measure (FGT) quantifies average accuracy drops on old tasks, whereas backward transfer (BWT) measures average accuracy improvements after later learning.
  • Evaluation metrics: Forward transfer (FWT) evaluates whether experience gained from earlier tasks benefits future tasks, using performance without prior experience as a reference.
  • Datasets and benchmarks: Benchmarks are divided into simple continual-learning scenarios focused on isolated skills and complex scenarios requiring multiple skills, prior knowledge, reasoning, and planning.Examples include instruction following and translation benchmarks, plus tool, web, and game environments.

14 APPLICATION OF LIFELONG LLM AGENTS

Lifelong LLM agents are applied across everyday work, life, and entertainment scenarios and across specialized domains such as education, law, healthcare, and other industries. These applications use continuous adaptation and accumulated knowledge to support changing user and task requirements.

  • Application areas: Applications are organized into daily scenarios and domain-specific implementations, reflecting the breadth of lifelong agents’ use contexts.Daily scenarios include work, life, and entertainment; specialized domains include education, law, and healthcare.
  • Daily applications: In work settings, agents optimize search and recommendations, organize and retrieve knowledge, support information sharing, and assist decision-making.
  • Daily applications: In life settings, continuous interaction, role-playing, and long-context understanding help agents progressively model users’ personalities and preferences for more natural conversations.
  • Daily applications: In entertainment, Minecraft agents such as Voyager autonomously explore unknown worlds using feedback mechanisms.
  • Domain-specific applications: Domain-specific agents support education through teaching and personalized learning, law through document and case analysis, and medicine through diagnostic, treatment, and patient-interaction assistance.
  • Domain-specific applications: In other industries, continuous learning enables agents to adapt to new real-world tasks and may reduce labor costs.

15 PRACTICAL INSIGHTS AND FUTURE DIREC-

The survey frames lifelong LLM agents as an integration of perception, memory, and action, while identifying robustness, scalability, and cross-module coordination as priorities for future development. It proposes mechanisms for adapting to new modalities, managing growing knowledge, and improving actions over long horizons.

  • Integrative architecture: Lifelong agents require tightly integrated perception, memory, and action modules for continuous adaptation to evolving tasks and environments.
  • Perception module: Perception has expanded from text to web content, images, and game environments, but must remain reliable on novel distributions and unseen modalities.
  • Perception module: Future perception systems should automatically select and integrate modalities, use modality-agnostic encoders, and retain representations of previously encountered modalities.
  • Memory module: Memory systems must preserve knowledge while controlling growth through specialized memory structures, improved retrieval, pruning or summarization, consolidation, transfer learning, and scalable storage.
  • Action module: Action research should improve scalability and transfer through hierarchical action spaces, self-improvement, task-agnostic policies, human feedback, and reinforcement learning.
  • Integrative architecture: Cross-module feedback, continual curricula, external tools and knowledge graphs, and collaborative learning are proposed to support long-horizon planning and decision-making.
  • Future direction: The roadmap aims for agents that adapt to emerging tasks while continuously perceiving, reasoning, and acting in increasingly complex environments.

16 CONCLUSION

The survey synthesizes lifelong learning for LLM-based agents through perception, memory, and action, and connects these modules to evaluation, benchmarks, applications, and future research. It presents robust lifelong learning as a route toward more capable and context-aware agents for complex real-world challenges.

  • Conclusion: The survey systematically examines perception, memory, and action methods for integrating and retaining knowledge over extended periods.
  • Conclusion: It also discusses evaluation metrics, benchmarks, and application scenarios spanning everyday and specialized domains.
  • Conclusion: The survey’s stated direction is to develop increasingly capable, adaptable, and context-aware solutions for complex real-world challenges.
Loading 2501.07278v2…