PaperlayerView exact revision on arXiv

Source-linked AI summary

When Continual Learning Moves to Memory: A Study of Experience Reuse in LLM Agents

Qisheng Hu, Quanyu Long, Wenya Wang

arXiv:2604.27003v1cs.LGcs.AI

TL;DR

Memory-augmented agents may preserve experience without changing model weights, but retrieval through limited context makes old and new memories compete. The paper analyzes this problem with a framework separating experience representation from memory organization, finding that abstraction, task difficulty, and retrieval design shape transfer and forgetting.

  • Problem

    Memory-augmented agents appear to avoid parametric stability–plasticity conflicts, but whether stored experience transfers across sequential tasks or degrades earlier-task performance remains insufficiently studied.

    Source: §1 · pp. 1–2

  • Method

    The paper studies controlled sequential-task experiments in ALFWorld and BabyAI, measuring transfer and forgetting while varying experience representation and memory organization.

    Source: §1 · p. 2, §2.3 · p. 3

  • Results

    Abstract insights transfer more safely than raw trajectories, negative transfer disproportionately harms hard cases, and finer-grained organization can help or hurt; stronger forward transfer can coincide with significant forgetting.

    Source: §1 · p. 2, §3.5 · p. 5, §4.4 · p. 8

  • Takeaways & Limitations

    External memory does not resolve continual learning; retrieval through a finite context window makes representation and retrieval design central to transfer and interference.

    Source: §6 · p. 8

    The controlled testbed uses two environments with one representative task pair each, one sparse retrieval backbone, and two runs per condition, so broader coverage and repetition would strengthen generality.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Memory-augmented LLM agents offer an appealing shortcut to continual learning: rather than updating model parameters, they accumulate experience in external memory, seemingly sidestepping the stability-plasticity dilemma of parametric learning. We show that this challenge does not disappear but resurfaces at the memory level. Under a limited context window, old and new experiences compete during retrieval, relocating the continual-learning bottleneck from parameter updates to memory access. To study this phenomenon, we introduce a (k,v) framework that disentangles two fundamental design axes of external memory: how experience is represented and how it is organized for retrieval. Across sequential-task experiments in ALFWorld and BabyAI, we find that abstract procedural memories transfer more reliably than detailed trajectories, while negative transfer disproportionately harms the hard cases. Moreover, finer-grained memory organization is not universally beneficial: designs that yield strong forward transfer can simultaneously induce severe forgetting. Together, these results reveal that external memory does not resolve the continual-learning problem; it reshapes it into a problem of memory representation and retrieval design.

1 Introduction

External memory appears to preserve old experience while accommodating new experience, but retrieval through a limited context window recreates competition between them. The paper studies transfer and forgetting across sequential tasks to test how memory representation and organization shape that competition.

1 Introduction

  • Old and new experiences compete for retrieval and limited context, shifting the continual-learning bottleneck from storing knowledge to selecting and prioritizing memories.The paper identifies retrieval pollution, context competition, and memory dilution as resulting interference modes.Source: §1 · pp. 1–2
  • Although memory-agent systems operate continuously, prior work often evaluates within-task performance rather than transfer to future tasks or preservation of earlier-task performance.The paper frames these transfer and forgetting questions as central to continual learning.Source: §1 · p. 2
  • Across ALFWorld and BabyAI, abstract insights transfer more safely than raw trajectories, negative transfer disproportionately harms hard cases, and finer-grained organization can help or hurt depending on experience diversity and task structure.A design that supports adaptation to a new task can also cause significant forgetting of an earlier one.Source: §1 · p. 2
  • The study uses controlled sequential-task experiments to measure within-task improvement, cross-task transfer, and forgetting while varying experience representation and memory organization.It also uses subset-level diagnostics to localize negative transfer.Source: §1 · p. 2
  • External memory avoids overwriting model parameters, but retrieval into a limited context window introduces a bottleneck for reusing stored experience.The evaluation protocol first accumulates Task A experience, then continues with Task B while reusing that memory.Source: Figure 1 · p. 1, Table 1 · p. 2

2 Framework & Experimental Setup

The framework treats memory-augmented agents as non-parametric continual learners whose behavior depends on which stored experience reaches a finite context window. It separates the representation of experience from its organization and access, then evaluates transfer and forgetting in controlled two-phase task sequences.

2.1 Agent Memory as Continual Learning

  • The protocol runs Task A and then Task B with a shared, growing memory; the frozen LLM backbone makes retrieval, rather than parameter updates, the source of interference.Each phase contains 200 training instances, and retrieved experience is inserted into context before the first action.Source: §2.1 · p. 2

2.2 A (k, v) View of Experience Reuse

  • The (k, v) framework separates experience representation (v) from organization and access (k), distinguishing abstract or raw content from storage and retrieval granularity.The two dimensions characterize how stored experience can be reused across tasks and over time.Source: §2.2 · p. 3

2.3 Continual Learning Metrics

  • Forward and backward transfer compare held-out task success after a two-phase sequence with single-task baselines, while RR and NL diagnose effects on baseline-success and baseline-fail subsets.Negative FWT indicates impaired plasticity, negative BWT indicates forgetting, and negative RR or NL deltas distinguish harmful from ineffective reuse.Source: §2.3 · p. 3

2.4 Environments, Task Pairs, and Protocol

  • The experiments use one task pair each in ALFWorld and BabyAI, chosen to contrast procedural mismatch with differences in query structure relevant to memory retrieval.ALFWorld pairs similar instruction forms with procedural differences; BabyAI shifts from heterogeneous Task A queries to homogeneous “find [object]” queries.Source: §2.4 · p. 3, Table 2 · p. 3

3 Study 1: Memory Representation

Study 1 compares raw episode trajectories with abstract procedural insights and finds that abstraction makes cross-task reuse safer, especially when experience moves from an easier task to a harder one. It also finds that within-task advantages may disappear across tasks and that abstraction can reduce forgetting.

3 Study 1: Memory Representation

  • Raw stores episode action–observation trajectories, whereas Insight stores LLM-abstracted strategy insights distilled from those trajectories.This comparison isolates how experience representation affects continual-learning dynamics.Source: §3 · p. 3

3.1 Abstraction Facilitates Safer Cross-Task Reuse

  • Raw trajectories produce negative forward transfer across both environments in A→B, while Insight shifts transfer to positive.Table 3 and Figure 2 present the full results; the directional gap is smaller in B→A, where experience moves from the harder task to the easier one.Source: §3.1 · pp. 3–4, Figure 2 · p. 4

3.3 Within-Task Advantage Does Not Predict Cross-Task Benefit

  • Raw’s large within-task advantage on Task B disappears under cross-task reuse on ALFWorld and reverses into a crossover on BabyAI.On ALFWorld, both conditions converge to similar performance, with Insight slightly ahead; detailed trajectories help within a task family but can become a liability across tasks.Source: Figure 4 · p. 4, §3.3 · pp. 4–5

3.2 Negative Transfer Is Concentrated on Hard Cases

  • Raw yields ∆RR = +4.6 but ∆NL = −26.1 on ALFWorld A→B, preserving easy cases while sharply degrading hard cases.Insight largely reduces this vulnerability because abstract representations do not prescribe task-specific procedures.Source: §3.2 · p. 4, Figure 3 · p. 4

3.4 Backward Transfer: Abstraction Reduces Forgetting

  • On BabyAI in B→A, Insight achieves positive backward transfer while Raw causes forgetting; both representations show moderate forgetting on ALFWorld.BWT(A→B) is near zero in all conditions, with limited room for change because Task A has a high baseline.Source: Figure 5 · p. 5, §3.4 · p. 5

3.5 Study 1 Summary

  • The study’s main findings are safer cross-task reuse with abstraction, disproportionate harm to hard cases, limits of within-task evaluation, and reduced forgetting.These findings summarize the representation comparison across the study’s transfer and forgetting analyses.Source: §3.5 · p. 5

4 Study 2: Memory Organization

Study 2 varies how insights are stored and retrieved, finding that finer granularity and more frequent retrieval help only when they match task structure. Memory designs that support forward transfer can also increase forgetting through retrieval dynamics.

4 Study 2: Memory Organization

  • The study fixes representation to Insight and compares bundled, individually indexed, and periodically retrieved memories while keeping the information budget roughly comparable.Agg versus Ind tests storage granularity; Ind versus Step tests retrieval frequency.Source: §4 · pp. 5–6, Figure 6 · p. 6

4.4 Study 2 Summary

  • Fine-grained storage can create retrieval diversity collapse: homogeneous queries repeatedly retrieve similar entries rather than gaining useful variety.The diagnostic reports broad BabyAI A→B improvement for Ind, whereas ALFWorld A→B diagnostics favor Agg.Source: §4.1 · p. 6, Figure 9 · p. 7

4.1 Memory-Unit Granularity (Agg vs. Ind)

  • Fine-grained storage helps when diverse, self-contained insights can be retrieved separately, but is less useful for repetitive memories or coherent multi-step strategies.BabyAI directions favor different conditions, while ALFWorld A→B favors Agg; these patterns align with differences in query and strategy structure.Source: §4.1 · pp. 6–7

4.2 Retrieval Frequency (Ind vs. Step)

  • Periodic retrieval helps when information needs change during an episode, but can add redundant guidance when those needs remain stable.In BabyAI, Step leads in B→A while Ind leads in A→B; ALFWorld cross-task differences are small even though Step has higher within-task performance.Source: §4.2 · p. 7

4.3 Stability-Plasticity Trade-off

  • In BabyAI A→B, Cond-Ind has the strongest forward transfer but also the most severe forgetting, as homogeneous new memories make earlier experiences harder to retrieve.Cond-Agg keeps the pool compact, while Cond-Step may partly compensate through step-level retrieval.Source: §4.3 · pp. 7–8, Figure 10 · p. 7, Table 6 · p. 7

5 Related Work

Prior work studies LLM-agent memories for reusable experience and continual learning under changing data. This paper examines these questions in frozen-backbone agents, where interference arises through retrieval.

5.1 Agent Experience and Memory

  • Agent memory systems differ in whether they reuse reflections, executable skills, trajectories, or persistent abstractions, and recent work also studies cross-task reuse and memory-side failures.The paper examines what these designs imply under task shift.Source: §5.1 · p. 8

5.2 Continual Learning for LLMs and Agents

  • Classical continual learning addresses stability and plasticity through regularization, replay, or architectural changes; this work focuses on frozen-backbone agents where interference arises in retrieval.The related work also situates this focus alongside continual learning for LLMs and agents.Source: §5.2 · p. 8

6 Conclusion

External memory preserves old experience but finite-context retrieval makes old and new memories compete for influence on agent decisions. The results connect reuse to representation, granularity, and retrieval trade-offs.

6 Conclusion

  • With a finite context window, old and new memories compete for access to the agent’s decisions, shifting interference from parameter updates to retrieval.Across both studies, representation, memory granularity, and retrieval design shape whether experience is useful guidance or noise.Source: §6 · p. 8

Limitations

The experiments are a targeted, controlled testbed rather than an exhaustive evaluation. The appendix defines run types and metrics and specifies the memory conditions used in the studies.

Limitations

  • The evaluation covers two environments, one representative task pair each, one sparse retrieval backbone, and two runs per condition; broader coverage and repetition would strengthen generality.The authors describe the setup as resource-constrained and targeted.Source: Limitations · pp. 8–9

A.1 Terminology and Conditions

  • The appendix glossary covers representations and conditions, while run labels distinguish no-memory, scratch, cross-task, and eval-only probes.An eval-only probe evaluates the earlier task after training to measure backward transfer.Source: Table 7 · p. 10, §A.1.1 · pp. 10–11

A.1.2 Metrics and Diagnostics

  • FWT measures whether prior experience helps or hurts the later task, while BWT measures how the later phase changes earlier-task performance.RR and NL track success on baseline-success and baseline-fail subsets, respectively.Source: §A.1.2 · p. 11

A.1.3 Memory System

  • The memory system builds on AgentGym and ReMe, using BM25 retrieval throughout..Source: §A.1.3 · p. 11

A.1.4 Study 1 Conditions

  • Study 1 compares full episode trajectories with abstract strategy insights, with insight memories aggregated into a task-level unit.Both use instance-level storage and retrieve one memory per query.Source: §A.1.4 · p. 11

A.1.5 Study 2 Conditions

  • Study 2 compares task-level bundles, individually indexed insights, and periodic re-querying, surfacing roughly three insights per retrieval event.The information budget is held roughly constant across conditions.Source: §A.1.5 · p. 11

A.1.6 Training and Evaluation

The experiments use sequential task phases and compare memory entries that preserve full trajectories, distilled insights, or bundled versus separately indexed insights.

A.1.6 Training and Evaluation

  • Each phase contains 200 training instances and 100 test instances, with results averaged over two independent runs.The listed backbone is Qwen-Plus; Table 8 summarizes the main hyperparameters.Source: §A.1.6 · p. 11, Table 8 · p. 12

B.1 Study 1: Raw vs. Insight

  • Raw entries retain complete action–observation trajectories, whereas Insight entries distill episodes into about three abstract procedural insights.The examples show insights conveying general search and navigation guidance rather than a full action script.Source: §B.1 · p. 11

B.2 Study 2: Cond-Agg vs. Cond-Ind

  • Cond-Agg stores roughly three insights from an episode as one bundle, while Cond-Ind stores insights separately under individual retrieval keys.Separate entries allow the retriever to match a query to a single relevant insight rather than return the entire bundle.Source: §B.2 · pp. 11–12

C Retrieval Frequency: Additional Analysis

The additional analysis compares retrieval frequency while noting that training can produce different memory pools across conditions. It also cautions that scratch baselines affect how forward-transfer comparisons should be read.

C Retrieval Frequency: Additional Analysis

  • The Cond-Ind–Cond-Step comparison conflates evaluation-time retrieval frequency with differences in pool composition created during training.Retrieval affects behavior and outcomes, which in turn affect the insights generated and stored.Source: C Retrieval Frequency: Additional… · p. 12
  • On ALFWorld, Cond-Step reaches 99.0% on Task A and 79.5% on Task B, versus 96.5%/55.0% for Cond-Ind and 97.0%/65.5% for Cond-Agg.Its Task B advantage over Cond-Ind is +24.5pp; on BabyAI, Step is intermediate at 70.0% on A and 48.0% on B.Source: C Retrieval Frequency: Additional… · p. 12
  • On BabyAI B→A, Cond-Step’s cross-task performance is 77.5% versus 74.5% for Cond-Ind, while their FWT values are +7.5 and −1.0, respectively.The FWT gap is amplified by Step’s lower scratch baseline, so absolute performance levels are closer than the FWT values suggest.Source: C Retrieval Frequency: Additional… · p. 12

D Retrieval Diversity Collapse

Retrieval diversity can collapse despite diverse stored keys, particularly when queries are homogeneous. Learning-dynamics analysis also shows that Raw A→B transfer remains below scratch performance throughout Task B.

D Retrieval Diversity Collapse

  • Under Cond-Ind, heterogeneous Task A queries yield context diversity of 0.45 versus 0.29 for Cond-Agg, while homogeneous Task B queries yield 0.38 versus 0.41.Individually indexed insights help when queries vary, but high key diversity does not ensure diverse retrieval for similar queries.Source: D Retrieval Diversity Collapse · p. 13
  • For BabyAI Task B under Cond-Ind, 88.5% of queries retrieve the same top-1 item, coverage is 5.4%, and within-task performance falls by 17pp.The figure reports high storage key diversity (>0.91) but context diversity of 0.38.Source: D Retrieval Diversity Collapse · p. 13
  • Pool-side measures such as stored-entry count and key diversity alone are insufficient to predict retrieval effectiveness because collapse appears at retrieval time.The figure distinguishes storage key diversity from the diversity of items actually retrieved across test queries.Source: D Retrieval Diversity Collapse · p. 13

E Learning Dynamics

  • Raw A→B cumulative success drops below scratch Task B soon after the phase switch and never recovers, while Insight tracks closer to or above scratch in some regions.The persistent Raw deficit indicates more than a brief adaptation cost.Source: E Learning Dynamics · p. 13

F.1 Study 1: Retrieval Pollution from Raw Trajectories

Raw trajectories can carry task-specific procedures into cross-task reuse, while abstract insights preserve functional guidance. The reported mismatch is more harmful in A→B than in the reverse direction.

F.1 Study 1: Retrieval Pollution from Raw Trajectories

  • In the ALFWorld “put a clean bowl in fridge” case, Raw retrieval skips washing and fails, whereas scratch and Insight agents clean the bowl first and succeed.The retrieved Task A trajectory concerns a similar placement task but omits the cleaning step distinguishing Task B.Source: §F.1 · p. 14
  • In the cloth-cleaning case, Raw follows a longer action pattern without issuing the correct cleaning command, while Insight supplies functional rather than procedural guidance.The example describes Insight as preserving the condition under which the right action becomes available.Source: §F.1 · pp. 14–15
  • Together, the cases support the interpretation that raw trajectories can displace the agent’s reasoning with task-specific procedures, while abstract insights avoid prescribing a full action script.The ALFWorld learning-dynamics figure reports that the Raw A→B deficit persists throughout Task B.Source: Figure 12 · p. 14, §F.1 · p. 15

F.2 Study 1: Why the B→A Gap Is Small

  • In B→A, the Raw–Insight gap is smaller: both yield similar positive forward transfer on BabyAI and are close to zero on ALFWorld.The paper suggests that experience from a relatively harder task may be less misleading when reused for an easier one.Source: §F.2 · p. 15

F.3 Study 2: Granularity and Retrieval-Side Failure Modes

Study 2 finds that finer-grained storage helps only when entries differ in content, step-level retrieval can align guidance with execution phase, and repetitive new memories can dilute access to older ones.

Case 3. Fine-grained storage helps when source memories are genuinely diverse

  • Fine-grained storage helps on BabyAI Task A→B when separately indexed insights differ in content and queries retrieve different keys.The examples include distinct navigation, spatial-anchor, and search insights.Source: §F.3 · p. 15

Case 4. Fine-grained storage fails when many entries say roughly the same thing

  • 82 of 200 queries returned the same top-ranked entry under Task B / Cond-Ind, illustrating retrieval-diversity collapse from redundant entries.Many entries encode variants of door traversal, room scanning, and continued search strategies.Source: §F.3 · pp. 15–16

Case 5. Step-level retrieval can align guidance with the current phase

  • Step-level retrieval can align guidance with the agent’s current execution phase when task information needs change over time.The case is illustrative, not a controlled comparison, because Cond-Step and Cond-Ind accumulate different memory pools during training.Source: §F.3 · p. 16

Case 6. Memory dilution after homogeneous Task B training

  • After homogeneous Task B training, repetitive Cond-Ind entries make diverse Task A memories harder to retrieve, reducing access rather than deleting them.The entries are not wrong for Task B; their dominance in the pool constrains backward evaluation on Task A.Source: §F.3 · p. 16

Related papers

Loading 2604.27003v1…