Source-linked AI summary
Learning to Continually Learn via Meta-learning Agentic Memory Designs
Yiming Xiong, Shengran Hu, Jeff Clune
TL;DR
Agentic systems struggle to continually learn because foundation models are stateless, while existing memory designs are largely handcrafted and fixed. ALMA meta-learns executable memory designs through open-ended code search, and its learned designs outperform human-designed baselines across four domains while improving cost efficiency and adaptation. The approach is evaluated in separate collection and deployment phases, but it does not yet learn memory designs online for new tasks.
Problem
Foundation models’ statelessness and predominantly handcrafted, fixed memory designs limit agentic systems’ continual learning across diverse and changing tasks.
Method
ALMA uses a Meta Agent to explore memory designs expressed as executable code, archive evaluated candidates, and deploy selected designs in static or dynamic modes.
Results
Across four sequential decision-making domains, learned memory designs consistently outperform human-designed baselines, with 6.2% and 12.8% overall improvements over no memory for GPT-5-nano and GPT-5-mini.
Takeaways & Limitations
ALMA provides a framework for automatically learning memory designs that support continual learning, with reported gains in scalability, transferability, and cost efficiency.
Takeaways & Limitations
ALMA learns memory designs from a predefined learning set rather than dynamically learning them online for new tasks, partly because evaluation requires many rollouts.
Abstract
from arXiv · showhide
The statelessness of foundation models bottlenecks agentic systems' ability to continually learn, a core capability for long-horizon reasoning and adaptation. To address this limitation, agentic systems commonly incorporate memory modules to retain and reuse past experience, aiming for continual learning during test time. However, most existing memory designs are human-crafted and fixed, which limits their ability to adapt to the diversity and non-stationarity of real-world tasks. In this paper, we introduce ALMA (Automated meta-Learning of Memory designs for Agentic systems), a framework that meta-learns memory designs to replace hand-engineered memory designs, therefore minimizing human effort and enabling agentic systems to be continual learners across diverse domains. Our approach employs a Meta Agent that searches over memory designs expressed as executable code in an open-ended manner, theoretically allowing the discovery of arbitrary memory designs, including database schemas as well as their retrieval and update mechanisms. Extensive experiments across four sequential decision-making domains demonstrate that the learned memory designs enable more effective and efficient learning from experience than state-of-the-art human-crafted memory designs on all benchmarks. When developed and deployed safely, ALMA represents a step toward self-improving AI systems that learn to be adaptive, continual learners.
1. Introduction
Agentic systems need memory to overcome foundation models’ statelessness, but handcrafted memory designs limit adaptation across domains. ALMA addresses this by open-endedly learning memory designs and outperforms human-designed baselines across four sequential decision-making domains.
- Motivation: Foundation models’ statelessness prevents agentic systems from accumulating experience and continually learning from past interactions.Memory enables systems to store and reuse experience so it can inform future decisions.
- Motivation: Human researchers still manually tailor memory designs that determine how experiences are represented, stored, retrieved, and updated for different domains.The design requirements differ across tasks, such as user facts in conversational agents.
- ALMA: ALMA uses a Meta Agent to explore memory designs in code, reflecting on archived designs and evaluation logs before implementing and evaluating new candidates.Evaluated designs and their logs are added back to the archive for future sampling.
- Results: Across four sequential decision-making domains, ALMA discovers domain-tailored memory designs that consistently surpass human-designed baselines.The learned designs are also reported as more cost-efficient, more scalable with memory size, and faster to learn under task distribution shifts.
2. Related Work
Related work studies memory mechanisms, automated learning of AI components, and automated agentic-system design. ALMA extends these directions through open-ended exploration of memory designs rather than relying solely on fixed handcrafted components or greedy selection.
- Memory for Agentic Systems: Memory mechanisms for agentic systems include token-level, parametric, and latent memory, with token-level memory storing and retrieving information from interaction trajectories.Retrieved experiences are appended to agent prompts for new tasks.
- Learning to Learn: AI-generating algorithms and automated machine learning aim to replace hand-engineered components with automatically learned architectures, algorithms, or environments.This work places memory design within the broader learning-to-learn paradigm.
- Automated Agentic-System Design: Prior automated agentic-system research learns improved system components, whereas ALMA focuses on open-ended exploration of memory designs.Its search process is specifically directed toward discovering memory mechanisms for continual learning.
- Open-Ended Exploration: Ablation studies report that open-ended exploration learns better memory designs than greedy-selection-based optimization.The comparison supports using open-ended search in memory-design optimization.
3. Learning of Memory Designs
ALMA searches for executable memory designs through a Meta Agent and evaluates them across separate collection and deployment phases. Its code-based search space can represent arbitrary designs, while deployment may use static or dynamic memory updates.
- Framework: ALMA represents memory designs in code and uses a Meta Agent to propose, implement, debug, and evaluate candidates through open-ended exploration.The archive stores previously discovered designs and their outcomes as stepping stones for subsequent proposals.
- Search Space for Memory Designs: Because Python is Turing complete, the code-based search space theoretically permits discovery of all representable memory designs.Code also supports interpretability and lets foundation models use pretrained knowledge about coding and agentic systems.
- Evaluation of Memory Designs: ALMA’s memory components generally operate in a Memory Collection Phase followed by a Deployment Phase.Collection updates memory from trajectories, while deployment retrieves memory to solve incoming tasks.
- Evaluation of Memory Designs: The Deployment Phase can be static, keeping memory fixed, or dynamic, updating memory with trajectories from incoming tasks.Static mode tests use of fixed memory, whereas dynamic mode tests adaptation to a new task distribution.
- Evaluation of Memory Designs: Each memory design is evaluated after collection by measuring success rate during deployment with an identical fixed agentic system.The best-learned design is evaluated under both deployment modes.
- Open-Ended Search: Archive sampling prioritizes designs with higher success rates and fewer prior samples, balancing refinement of strong designs with exploration.The Meta Agent then analyzes sampled designs, outcomes, and interaction logs to plan new code implementations.
4. Experiments
ALMA is evaluated across four sequential decision-making benchmarks using held-out data, multiple foundation-model settings, and comparisons with handcrafted memory systems. The learned designs outperform manual baselines, specialize to domain requirements, scale more effectively with experience, and adapt under task distribution shift while maintaining cost efficiency.
- Experiment setup: ALMA is evaluated on ALFWorld, TextWorld, Baba Is AI, and MiniHack, with learning and testing performed on separate unseen data.Each benchmark is split into learning and testing sets, with Memory Collection and Deployment phases represented in both workflows.
- Experiment setup: The evaluation compares learned memory designs against four state-of-the-art human-designed systems, including trajectory retrieval and per-task experience retrieval.The baselines manually specify what to store, how to update it, and how to retrieve it.
- Overall results: 6.2% overall improvement over the no-memory baseline was achieved with GPT-5-nano, while learned designs outperformed all human-designed baselines across benchmarks.The result is reported for the standard learning and testing setting with GPT-5-nano as the agentic system’s foundation model.
- Overall results: 12.8% overall improvement over the no-memory baseline was achieved with GPT-5-mini, demonstrating transfer across foundation models.The improvement with GPT-5-mini exceeded the corresponding GPT-5-nano improvement by 6.6 percentage points.
- Learned design analysis: Learned memory designs specialized by domain, storing fine-grained spatial knowledge for object-interaction games and abstract strategies for complex-reasoning tasks.ALFWorld and TextWorld favored spatial relationships and room layouts, whereas Baba Is AI and MiniHack favored strategy libraries and plan synthesis.
- Efficiency and adaptation: Learned designs achieved higher performance faster with limited data, scaled better with additional trajectories, reached 84.1% success on shifted ALFWorld tasks, and averaged 53.9% success at $0.09 end-to-end memory cost.The distribution-shift evaluation used dynamic memory updates, while the scaling evaluation used static deployment memory.
5. Conclusion, Safety, and Future Work
ALMA learns memory designs that outperform manually designed baselines across sequential decision-making benchmarks, while showing transfer, cost efficiency, and adaptation under task distribution shift. The paper also identifies safety and scalability limitations requiring inspection and further development.
- Conclusion: ALMA’s best-learned memory designs consistently outperform manually designed baselines across sequential decision-making benchmarks.The paper presents this as evidence that memory design can be learned rather than exclusively handcrafted.
- Safety: Learned components introduce safety concerns because their behavior may deviate from human intentions or optimization guided by predefined target metrics.The authors impose explicit constraints and restrictions during the learning process.
- Future Work: ALMA currently learns memory designs from a predefined learning set rather than dynamically learning them online when facing new tasks.The authors defer online learning because evaluating each explored design requires many rollouts and substantial computational budget.
- Future Work: Code-space memory designs may be limited by the capabilities of the underlying foundation models.The authors suggest exploring automatically designed and trained foundation-model architectures with native memory support.
- Conclusion: Current limitations prevent demonstrating a system that learns both memory and the agentic system.The conclusion frames ALMA as a step toward automated continual-learning AI rather than a complete solution.
Impact Statement
The impact statement presents ALMA as an approach for automating memory-design development and discovering domain-specific memory paradigms. Its implementation uses modular code interfaces that coordinate retrieval and update operations.
- Impact: ALMA automatically learns novel memory designs, enabling agentic systems to continually learn from experience.The stated impact is to reduce the manual effort required to handcraft memory designs for diverse domains.
- Applications: The framework could automate domain-specific memory design for fields such as medicine, finance, and software engineering.The paper also positions ALMA as a way to uncover memory paradigms not identified through human intuition.
- Implementation: Memory designs are represented through sub-modules implemented as layers, with each layer encapsulating module functionality.The abstract class includes database state and abstract retrieve and update functions for each sub-module.
- Implementation: General retrieve and general update orchestrate the execution order of retrieval and update operations across sub-modules.These interfaces determine how layer-level inputs, outputs, and operations are composed.
A.2. Evaluation of Memory design
The evaluation separates memory collection from deployment: interaction logs are collected without memory, then retrieved knowledge supports deployment tasks and can be updated sequentially under distribution shift. Performance is averaged over deployment tasks.
- Memory Design: A memory design is defined as M = (U, D, R), comprising update, storage, and retrieval components.U and R interface with the agentic system, while D is the internal structure retaining memories.
- Evaluation Protocol: The dataset is split into memory-collection and deployment subsets for evaluating a memory design.The collection phase produces interaction logs without memory access, while deployment evaluates the agent using retrieved knowledge.
- Memory Collection Phase: During memory collection, the agent completes tasks without memory access, producing trajectories used to update memory and construct a static memory.The final collected memory is denoted D_N.
- Deployment Phase: During deployment, retrieved contexts from static memory are integrated into the agentic system to support task execution.Performance on the deployment set reflects the design’s ability to collect and use knowledge from experience.
- Distribution Shift: Under task distribution shift, each deployment trajectory and feedback update memory sequentially before the next task.The updated memory state is used for retrieval on the subsequent task.
- Interaction Data: Each collected trajectory records environment states and actions generated by a fixed agentic policy, together with benchmark-specific task feedback.The feedback function assigns a scalar score to each trajectory.
A.3. Evaluation Cost
The evaluation-cost analysis measures computational and token overhead independently of memory effectiveness, while the learning process balances exploration and exploitation among archived designs.
- Evaluation Cost: End-to-end memory cost includes memory-state construction during collection and cumulative retrieved-knowledge generation during deployment.The metric is intended to quantify cumulative FM overhead across both phases.
- Evaluation Cost: Collection-phase trajectories are gathered without memory access, making memory-production and static-deployment costs independent of memory performance.The same trajectories provide the input for producing the memory state and retrieved knowledge.
- Evaluation Cost: Retrieved-knowledge token size measures informational overhead injected into the agent prompt and serves as a proxy for inference-time efficiency.This quantity depends solely on collection-phase trajectories.
- Memory Design Archive: The archive stores discovered designs, sampled interaction records, overall performance, and sampling counts for iterative search.Interaction records include retrieved context, resulting trajectories, and task-level feedback.
- Sampling Process: Sampling probabilities favor high-performing yet under-sampled designs while retaining non-zero probability for moderately performing designs to preserve diversity.The mechanism uses normalized performance, a visit-time penalty, and a temperature-controlled softmax.
B.1. Benchmark Details
ALMA is evaluated on four sequential decision-making environments that require multi-step interaction, exploration, and experience reuse, with benchmark-specific tasks and interaction limits.
- Benchmark Details: The evaluation spans ALFWorld, TextWorld, Baba Is AI, and MiniHack as testbeds for continual learning from experience.All four benchmarks involve multi-step interactions.
- ALFWorld: ALFWorld tests multi-step household manipulation tasks across train, seen-validation, and unseen-validation settings.Its datasets contain 3553 training tasks, 140 seen-validation tasks, and 134 unseen-validation tasks.
- TextWorld: TextWorld evaluates natural-language interaction in partially observable worlds through exploration, information gathering, and multi-step reasoning.The experiments use Treasure Hunter and Cooking games, totaling 52 tasks, with an 80-step maximum.
- Baba Is AI: Baba Is AI evaluates sequential decisions in grid worlds whose symbolic rules can be manipulated to satisfy a win condition.The benchmark uses 52 tasks and a 20-step maximum, with binary success rewards.
- MiniHack: MiniHack evaluates exploration, planning, and resource management in procedurally generated dungeon worlds using 114 navigation tasks.The experiments follow the standard BALROG MiniHack configuration.
- Baselines: Trajectory Retrieval encodes a task and supplies the single most similar past trajectory, while Dynamic Cheatsheet maintains a cumulative global record of successes and failures.Trajectory Retrieval uses cosine similarity and provides only the Top-1 raw trajectory because of context limits.
B.3. Additional Details of Learning Process
ALMA begins from a generated memory design and iteratively explores code-based alternatives through archive sampling, benchmark-aware reflection, and structured improvement suggestions.
- Learning Process: ALMA initializes a memory design from scratch with GPT-5/medium and then performs 11 learning steps of proposal, implementation, and evaluation.Each evaluated design and its metrics and interaction logs are added to the archive.
- Sampling: Archive sampling balances higher-performing designs with diversity by applying visit penalties, temperature smoothing, and sampling without replacement.The reported settings are α = 0.5 and T = 0.5.
- Greedy Search: Greedy search instead selects the archive’s best design at every step and runs 43 steps to match the open-ended exploration budget.The final greedy-search output is the best design in its archive.
- Ideate & Planning: Ideation uses sampled designs, source code, and stratified interaction logs to generate benchmark-specific ideas and plans for new memory designs.The system prompt remains fixed across benchmarks and learning steps.
- Reflection and Planning: The reflection schema asks the Meta Agent to assess retrieved memories, content quality, structure, coherence, and prioritized suggested changes.Trajectory items are categorized as Useful, Potentially Useful, Irrelevant, or Empty/BadFormat.
B.4. Additional Details of Testing
Testing separates memory collection from deployment and evaluates learned designs in static and, for ALFWorld, dynamic settings, including transfer to a more capable agent.
- Testing Setup: Testing uses held-out portions of each benchmark, with all test sets split evenly between memory collection and deployment.ALFWorld is tested in both static and dynamic modes, while other benchmarks use static mode.
- Static Mode: Static testing compares learned and manual designs with the learning-process agent and then tests transfer to a more capable agent.The transfer evaluation uses GPT-5-mini/medium after learning with GPT-5-nano/low.
- Dynamic Mode: Dynamic ALFWorld testing collects trajectories from valid-seen tasks and deploys on valid-unseen tasks to create a distribution gap.During deployment, the system uses general retrieve() to access relevant memory.
C.1. Learning Processes on Other Benchmarks
Across TextWorld, ALFWorld, and MiniHack, ALMA progressively explores memory designs in an archive and selects the highest-success design as the final design.
- Each archive tree node represents a memory design, with edges denoting derivation from parent designs.
- Node colors encode success rates, allowing ALMA to identify the highest-success memory design for final use.
- The step-wise process shows ALMA progressively discovering memory designs through an ever-growing archive.
C.2. Results of Greedy Exploration
On ALFWorld, greedy exploration selects the historically best-performing design at each step under the same exploration budget as ALMA. ALMA achieves better performance and transferability than Greedy Search, highlighting the value of open-ended exploration.
- Greedy Search extends the historically best-performing memory design at each learning step.The baseline explores the same total number of memory designs as ALMA.
- ALMA achieves better performance and transferability than Greedy Search on ALFWorld.
- Table 2 reports ALFWorld success rates as Mean ± SE percentages across three deployment runs for GPT-5-nano and GPT-5-mini.
C.3. Example Results of Learned Memory Designs
The best-learned MiniHack memory design combines structured state extraction with distilled, actionable navigation and planning guidance. Its stored memories emphasize exploration, obstacle avoidance, escape routes, health awareness, and objective-directed movement.
- Memory structure: The best-learned memory design in MiniHack stores structured state and strategy information for navigation and planning.The design includes utilities for extracting map area, position, statistics, action sets, role or race information, and local topology, alongside learned strategy fields.
- Learned lessons: Distilled lessons advise preferring safe passable tiles, avoiding oscillation, and moving toward visible objectives when safe.The design produces short lessons intended to capture transferable insights from prior episodes.
- Actionable tips: The memory design converts raw strategy hints into concise task-conditioned tips covering exploration, blocked movement, reversible actions, and object handling.Its generated bullets include surveying open directions, preserving escape routes, avoiding repeated wall bumps, and reconsidering plans after discoveries.
- Planning: Its generated plan surveys surroundings, maps safe routes, maintains retreat paths, and checks objective proximity, health, and threats.The plan also warns against cornering pushable objects and entering dead ends without escape.
- Episode-derived memory: The design records transferable navigation strategies that reassess blocked paths, preserve escape routes, and monitor health before risky actions.It also tracks success and topics such as navigation and planning, while recording risk notes about unknown areas at low health.
- Retrieval content: Additional memory fields encode spatial priors, reflex tips, task state, and explicit blocked-direction information for retrieval.Examples include the relation that walls block movement and recommendations to pivot toward open tiles rather than repeat blocked moves.