Source-linked AI summary
AgentCL: Toward Rigorous Evaluation of Continual Learning in Language Agents
Yiheng Shu, Bernal Jiménez Gutiérrez, Saisri Padmaja Jonnalagedda, Yuguang Yao, Huan Sun, Yu Su
TL;DR
Language-agent continual learning is difficult to evaluate because existing benchmarks often omit controlled cross-task reuse and targeted transfer measures. AgentCL introduces compositional and naive task streams with explicit metrics, and evaluates non-parametric memory designs using MemProbe. Controlled streams distinguish memory plasticity more clearly, while naive and held-out settings expose limited gains and memory-induced degradation, underscoring the challenge of balancing plasticity with stable reuse.
Problem
Existing benchmarks do not rigorously evaluate how language agents accumulate, transfer, and retain reusable experience across related task streams.
Method
AgentCL uses controlled compositional and naive streams with targeted transfer metrics to evaluate non-parametric memory designs, aided by MemProbe.
Results
Compositional streams produce clearer separation among memory methods, while naive and held-out settings often show limited gains and memory-induced degradation.
Takeaways & Limitations
Continual learning in language agents requires memory designs that balance useful adaptation with stable reuse across stream and distribution changes.
Takeaways & Limitations
The study focuses on non-parametric memory and does not systematically evaluate parametric memory or training-based adaptation.
Abstract
from arXiv · showhide
Language agents spend substantial inference time solving individual tasks, yet the experience acquired in one episode is often underutilized in future episodes. Continual learning expects an agent to accumulate reusable experience across a stream of tasks, improve over time, and avoid interference from irrelevant experiences. Unfortunately, existing benchmarks struggle to evaluate continual learning in language agents rigorously. Most efforts focus on retrieval and reasoning over long-context conversations or documents, while recent lifelong-adaptation benchmarks often rely on naive task streams with limited analysis of cross-task relationships, making it difficult to understand what an agent learns and reuses over time. This paper presents an evaluation framework AgentCL for continual learning in agents, centered on controlled task streams and metrics for transfer gains. AgentCL constructs compositional streams where earlier sub-solutions, evidence, or workflows are intentionally reusable in later tasks, and contrasts them with naive streams where such reusability is not guaranteed. We use the benchmark to evaluate non-parametric memory designs for continual learning. To diagnose how memory design choices affect continual learning, we develop MemProbe, a probing method that stores interactions, insights, and skills, while filtering unreliable experiences during consolidation. Empirical analysis across coding, deep research, and language understanding/reasoning tasks shows that naive streams offer limited ability to distinguish memory designs, whereas controlled streams more clearly distinguish their plasticity. Meanwhile, naive and held-out settings often yield limited gains and can expose memory-induced degradation. These results highlight the need for stronger memory designs that balance plasticity and stable reuse.
1 Introduction
AgentCL addresses the difficulty of evaluating continual learning in language agents by controlling cross-task reuse and measuring transfer properties. It contrasts naive and compositional streams to reveal how memory designs learn and reuse experience.
- Language agents often underuse experience from completed episodes, motivating continual learning over task streams.
- Rigorous evaluation must control how knowledge, sub-solutions, and workflows compose across tasks while separating plasticity, stability, and generalization.
- Existing long-context benchmarks use static documents or conversations, whereas lifelong benchmarks often lack explicit cross-task relationship control.
- AGENTCL contrasts arbitrary naive streams with compositional streams that place reusable earlier experience before related complex tasks and computes targeted transfer gains.
- Across coding, deep research, and reasoning, compositional streams better distinguish memory designs, exposing differences in their plasticity.
2 Related Work to Continual Learning in Language Agents
Related work studies continual learning through streaming memory, procedural reuse, and parameter or policy updates. These approaches target knowledge accumulation, transfer, forgetting, and stability–plasticity balance across agentic settings.
- Streaming benchmarks organize interactive tasks to measure knowledge accumulation, transfer, and forgetting in language agents.
- Non-parametric memory systems store long-term memories, reusable workflows, strategies, code snippets, abstractions, or online task experiences for later retrieval.
- Other approaches study parameter or policy updates to balance new-domain adaptation with retention and disentangle shared from conflicting knowledge.
3 AGENTCL: Evaluation Framework for Continual Learning in Agents
AGENTCL models agents that solve sequential tasks while maintaining persistent memory, then evaluates memory-assisted adaptation through controlled streams and two-pass transfer metrics. Its metrics separate plasticity, stability, and generalization.
- 3.1 Problem Formulation: An agent sequentially solves tasks while retrieving from and updating persistent memory with each task trajectory.
- 3.2 Task Streams: Task order matters because accessible experience evolves, so streams must expose explicit cross-task relationships rather than arbitrary repetition or unrelated tasks.
- 3.3 Metrics: The first evaluation pass permits memory reading and writing, while the second pass evaluates the same stream with frozen, read-only memory.
- 3.3 Metrics: Plasticity Gain measures whether earlier experience helps later tasks, while Stability Gain measures whether task experience remains reusable after consolidation and subsequent updates.
- 3.3 Metrics: Generalization Gain measures transfer to unseen tasks after memory is constructed and frozen.
4 Benchmark Construction
AgentCL constructs benchmark streams across coding, deep research, and language understanding or reasoning. The construction pairs reusable subtasks with complex tasks while retaining naive baselines and applying quality controls.
- The benchmark introduces task-stream construction across multiple environments using existing resources and synthesized subtasks.
- Coding: Coding streams contain 48 base–complex problem pairs, with each complex solution guaranteed to reuse its base problem’s function.
- Deep Research: BrowseComp+ compositional streams prepend 308 subtasks sharing evidence documents with 100 parent tasks, while naive streams randomly sample 100 tasks.
- Language Understanding & Reasoning: Language understanding and reasoning streams use MMLU-Pro, BabyAI, and ScienceWorld tasks, including naive and evaluated structured settings.
- Synthesized subtasks undergo checks for evidence support, determinism, logical relevance, entity specificity, and answer completeness.
5 Experiments
Experiments compare non-parametric memory designs across coding, research, and reasoning settings using controlled and naive task streams, held-out tasks, and MEMPROBE ablations. Controlled streams distinguish methods more clearly, while naive and held-out evaluations expose stability and generalization weaknesses.
- Evaluated Methods: The evaluation covers memoryless ReAct, adaptive memories, procedural memories, and self-evolving memories, including MEMPROBE.The evaluated methods span four categories of non-parametric memory designs.
- Memory Design: MEMPROBE uses retrieve–solve–consolidate processing with interaction, insight, and skill memories filtered by syntactic checks and an LLM judge.It stores trajectories and responses, task patterns and failure modes, and reusable procedures or snippets.
- Controlled Streams: 9.4 and 8.8 versus 3.0 and 1.9 are the complex-task accuracy standard deviations across methods for compositional and naive CodeEval-Pro streams, respectively.BrowseComp+ shows the same pattern, with compositional dispersions of 14.9 and 16.0 versus naive dispersions of 2.3 and 5.7.
- Naive Streams: Naive streams provide limited reuse evidence but test stability, with CodeEval-Pro SG ranging from −3.5 to +4.2 and BrowseComp+ SG from −11.0 to +11.0.These streams reveal whether accumulated memory remains helpful or harmless without guaranteed reuse.
- Held-out Evaluation: On held-out HumanEval-Pro tasks, ReAct remains strongest overall at 72.5, while MEMPROBE is closest among memory methods at 70.8.Held-out tasks come from a different source and are not paired with previous tasks; gains are limited and memory can introduce mild interference.
- Plasticity–Stability Tradeoff: Large compositional PGs do not ensure stability or generalization: MEMPROBE reaches +21.9 PG but −2.1 SG and −0.8 GG on CodeEval-Pro.DC-RS likewise has +3.8 compositional PG and −11.7 GG, illustrating that within-stream usefulness can become harmful after distribution change.
6 Conclusion
AGENTCL shows that task-stream design is central to evaluating continual learning in language agents. The paper advocates transparent task relationships and metrics that distinguish plasticity, stability, and generalization.
- Task-stream design is essential for distinguishing continual-learning capabilities.
- Future benchmarks should publish task streams transparently and explain relationships between tasks, rather than reporting only average metrics.
- Plasticity–stability remains a formidable challenge when opportunities for cross-task reuse are unclear.
B Experimental Setting and Details
The experimental appendices organize benchmark construction, evaluated memories, reproducibility, compute, prompts, statistical analysis, detailed performance, and case studies.
- Benchmark construction: Appendix sections document benchmark construction, including compositionality, detailed construction, BrowseComp+ subtask synthesis, and stream statistics.
- Memory evaluation: The appendices describe evaluated memory designs and provide detailed MEMPROBE information.
- Reproducibility: Reproduction instructions, compute resources, and prompts are included as experimental details.
- Analysis: Statistical significance and detailed performance are reported in dedicated appendix sections.
- Case study: A case study is included among the experimental appendices.
A Limitations
The study primarily evaluates non-parametric memory for continual learning in language agents, leaving parametric and training-based adaptation insufficiently examined.
- The study focuses primarily on continual learning through non-parametric memory.
- Parametric memory and training-based adaptation methods are not systematically evaluated.
- Incorporating these approaches is identified as a direction for future work.
B Experimental Setting and Details
The experimental setting evaluates compositional task streams and diverse non-parametric memory designs, including MEMPROBE’s multi-form memory and retrieval-consolidation process. It also tracks execution and token usage across benchmark streams.
- Task construction: CodeEval-Pro task pairs preserve a subtask’s core logic while extending it across multiple generated strings.
- Task construction: BrowseComp+ subtasks share solution logic with original tasks without directly revealing their answers.
- Memory designs: The evaluation covers memoryless, self-invoking, adaptive semantic and episodic, procedural, and self-evolving memory designs.
- Experimental resources: The experiments report execution statistics, token usage, and deployment requirements alongside performance evaluation.
- MEMPROBE: MEMPROBE stores interaction, insight, and skill memories, then retrieves relevant prior experiences and consolidates new episodes with quality-aware controls.
- MEMPROBE: The memory update prompt converts each task episode into structured insight and skill memories while retaining the trajectory as interaction memory.
B.7 Statistical Significance
The evaluation compares stream designs by measuring how clearly memory methods separate statistically. Compositional streams provide stronger separation than naive and held-out settings.
- The proportion of pairwise comparisons whose 95% CI excludes zero measures clearer separation among evaluated methods.Higher values indicate that a stream distinguishes methods more clearly.
- Compositional streams yield substantially stronger pairwise separation among memory methods than naive streams.The analysis uses the proportion of pairwise 95% CIs excluding zero and mean absolute pairwise accuracy deltas.
- Held-out streams remain comparatively weakly separated because they serve as robustness checks rather than explicit transfer settings.This pattern complements the stronger separation observed in controlled compositional streams.
B.8 Detailed Performance
Detailed evaluations examine coding, deep research, reasoning, retrieval, and stream construction. They show that compositional design creates reusable transfer opportunities, while related or held-out settings can provide limited separation or harmful retrieval.
- ScienceWorld block streams produce limited method separation, similar to naive streams, because tasks repeat near-duplicate goals rather than enabling compositional reuse.Tasks share an overall goal but differ in environment configurations.
- MEMPROBE retrieval measures Hits@k as the proportion of top-2 retrieved memories corresponding to relevant, reusable experience.The reported retrieval analysis focuses on compositional streams in the two main datasets.
- AgentBoard BabyAI contains 112 tasks organized into 28 families, with four tasks per family and seven selected source-to-target family relations for compositional streams.Tasks within each family share an overall goal across different environments.
- Retrieved memory helps coding when it captures reusable subtask structure but hurts when a topically similar task has incompatible semantics.This case study illustrates the need to distinguish topical similarity from semantic compatibility.
- Compositional BrowseComp+ streams are constructed from evidence-supported intermediate subtasks that create known transfer opportunities without exposing privileged construction information at test time.Generated subtasks receive reference answers and supporting document identifiers for verification and evaluation.
E.2 The Statistics of Compositional Streams
Statistics of compositional streams indicate that subtasks and full tasks share structure while differing in scope and complexity. BrowseComp+ diagnostics further show that subtasks are related to, but rarely duplicates of, their parent tasks.
- BigCodeBench-Lite-Pro subtasks are more verbose than full tasks and involve slightly longer solutions and more function calls.The pattern suggests narrower scope with greater implementation detail.
- BrowseComp+ subtasks have low token-set Jaccard similarity to parent tasks, averaging 0.10, with no pair exceeding 0.30.Only 5.8% cover the full parent evidence set, and 1.6% share both the normalized answer and evidence set.
- The research reports that LLMs were used primarily for memory-method evaluation and synthetic-data generation, not for originating the research ideas or methodology.They were also used for language polishing and limited coding assistance.