PaperlayerView exact revision on arXiv

Source-linked AI summary

MemSkill: Learning and Evolving Memory Skills for Self-Evolving Agents

Haozhen Zhang, Quanyu Long, Jianzhu Bao, Tao Feng, Weizhi Zhang, Haodong Yue, Wenya Wang

arXiv:2602.02474v2cs.CLcs.AIcs.LG

TL;DR

Static, handcrafted memory mechanisms encode human assumptions and are brittle across diverse interaction patterns and long histories. MemSkill instead learns to select reusable memory skills, applies them through an LLM executor, and evolves the skill bank from hard cases; experiments report consistent gains and strong generalization across four benchmarks.

  • Problem

    Existing agent-memory methods rely largely on static, hand-designed mechanisms that encode human assumptions about what to store, revise, and prune.

  • Method

    MemSkill uses a controller to select reusable memory skills for an LLM executor and periodically evolves the shared skill bank from hard cases.

  • Results

    MemSkill consistently improves task performance and generalizes well across LoCoMo, LongMemEval, HotpotQA, and ALFWorld.

  • Takeaways & Limitations

    The results support adaptive, self-evolving memory management for LLM agents across conversational and embodied settings.

  • Takeaways & Limitations

    Practical deployment may require privacy protection, user consent, access control, and data-retention mechanisms beyond the benchmarked algorithmic setting.

Abstract

from arXiv · show

Most Large Language Model (LLM) agent memory systems rely on a small set of static, hand-designed operations for extracting memory. These fixed procedures hard-code human priors about what to store and how to revise memory, making them rigid under diverse interaction patterns and inefficient on long histories. To this end, we present \textbf{MemSkill}, which reframes these operations as learnable and evolvable memory skills, structured and reusable routines for extracting, consolidating, and pruning information from interaction traces. Inspired by the design philosophy of agent skills, MemSkill employs a \emph{controller} that learns to select a small set of relevant skills, paired with an LLM-based \emph{executor} that produces skill-guided memories. Beyond learning skill selection, MemSkill introduces a \emph{designer} that periodically reviews hard cases where selected skills yield incorrect or incomplete memories, and evolves the skill set by proposing refinements and new skills. Together, MemSkill forms a closed-loop procedure that improves both the skill-selection policy and the skill set itself. Experiments on LoCoMo, LongMemEval, HotpotQA, and ALFWorld demonstrate that MemSkill improves task performance over strong baselines and generalizes well across settings. Further analyses shed light on how skills evolve, offering insights toward more adaptive, self-evolving memory management for LLM agents.

1. Introduction

MemSkill addresses the rigidity of static, handcrafted memory mechanisms by making memory operations learnable, compositional, and evolvable. Its closed-loop design learns skill selection, evolves the skill bank from hard cases, and improves performance across conversational and embodied benchmarks.

  • Motivation: Static handcrafted memory mechanisms encode strong human assumptions and can struggle with diverse interaction patterns and growing histories.Prior systems commonly use fixed primitives and heuristic modules for storing, revising, and pruning memories.
  • Approach: A controller selects relevant skills from a shared bank, and an LLM executor uses them to generate skill-guided memories over interaction spans.The formulation supports processing spans of different lengths instead of requiring a fixed turn-level extraction unit.
  • Evaluation: MemSkill consistently improves task performance and generalizes across LoCoMo, LongMemEval, HotpotQA, and ALFWorld.The benchmarks cover conversational question answering and embodied interaction settings.
  • Approach: MemSkill represents memory operations as reusable skills that specify how to extract, consolidate, or revise memories from interaction traces.The approach reframes memory construction as an adaptive, skill-conditioned generation process rather than a fixed procedural pipeline.
  • Closed-loop evolution: MemSkill alternates reinforcement-learning-based skill selection with LLM-guided refinement and expansion of the skill bank from difficult cases.This closed loop jointly improves the selection policy and the set of available memory skills.

2. Related Work

Related work develops external-memory pipelines, reinforcement-learning-based memory management, and self-evolving agent systems. MemSkill differs by focusing on evolving the reusable memory skills themselves.

  • Agent memory: Agent-memory research commonly extracts salient information, retrieves relevant entries, and updates external stores through consolidation or pruning.These pipelines support downstream reasoning and decision making from interaction histories.
  • Learning-based memory: Learning-based memory methods such as Memory-R1 and Mem-α optimize memory management with reinforcement learning using downstream task signals.This line of work extends earlier externally stored-memory approaches with task-driven optimization.
  • Self-evolving agents: Concurrent self-evolving-agent studies address streaming memory evolution, predefined memory architectures, latent reasoning memory, or reusable reasoning strategies.These approaches share an interest in improvement from experience but target different objects or representations.
  • Positioning: MemSkill targets evolution of memory skills themselves, rather than streaming memory, predefined architectures, latent memory, or distilled reasoning strategies.Its distinctive focus is refining reusable procedures for memory construction.

3. Method

MemSkill separates reusable skill knowledge from trace-specific memories and learns to construct memories through controller-selected, executor-applied skills. It then evolves the shared skill bank using hard cases and task-based feedback.

  • Overview: MemSkill maintains a shared skill bank and a trace-specific memory bank, improving skill selection and memory construction in alternating phases.The designer updates reusable skills between controller-training phases while each trace receives its own memory bank.
  • Skill bank: Each memory skill contains a description for selection and a detailed specification guiding memory extraction or revision.The initial bank contains INSERT, UPDATE, DELETE, and SKIP, which the designer can refine or expand.
  • Controller: The controller processes interaction traces span by span, conditioning Top-K skill selection on the current text span and retrieved memories.A shared scoring mechanism supports a skill bank whose size changes as skills evolve.
  • Executor: The executor conditions an LLM on the current span, retrieved memories, and selected skills to produce structured memory updates in one call.The updates are parsed and applied to the trace-specific memory bank, reducing repeated per-turn processing.
  • Controller optimization: Downstream performance on memory-dependent training queries supplies the reward used to optimize the controller’s skill selections.The resulting reward can be measured with task metrics such as F1 or success rate.
  • Skill evolution: A sliding hard-case buffer records challenging query-centric failures, which the designer clusters and mines to refine existing skills or propose new ones.Representative cases are prioritized using low performance and repeated failures while preserving diversity across error types.

4. Experiments

The experiments evaluate MemSkill across conversational and embodied benchmarks using broad comparisons with established memory baselines. Results are organized around multiple datasets, metrics, and evaluation settings.

  • Datasets and Baselines: MemSkill is evaluated on LoCoMo, LongMemEval, HotpotQA, and ALFWorld.HotpotQA studies transfer under distribution shift, while LoCoMo and LongMemEval assess conversational memory and ALFWorld assesses embodied interaction.
  • Datasets and Baselines: Conversational benchmarks report F1-score and LLM-based judge score, while HotpotQA uses LLM-judge evaluation for transfer.The reported metrics are selected according to benchmark setting.
  • Datasets and Baselines: Results distinguish best scores within each base-model block and identify transfer-only evaluations.The reporting conventions also mark evaluations using in-context demonstrations.

Appendix B.1 provides dataset splits.

MemSkill combines broad baseline evaluation with transfer, ablation, case-study, and cost analyses. Across these analyses, the paper reports strong generalization, benefits from both selection and evolution, domain-specialized skills, and a quality-cost trade-off.

  • Effectiveness across conversational and embodied settings: MemSkill consistently achieves the strongest overall performance across LoCoMo, LongMemEval, and ALFWorld.It records the best LLM-judge scores on both conversational benchmarks and the highest success rates on seen and unseen ALFWorld splits.
  • Comparison Experiments: MemSkill uses diverse baselines and evaluates transfer across base models, datasets, and interaction formats.Skills trained with LLaMA transfer directly to Qwen, while LoCoMo-trained skills transfer to LongMemEval and HotpotQA without task-specific training.
  • Cross-dataset transfer: K=7 achieves the best HotpotQA results across 50, 100, and 200 concatenated-document settings.Increasing the selected-skill count generally improves performance, especially as contexts become longer and noisier.
  • Ablation Study: Removing the controller or designer degrades performance, while refinement-only improves over static skills but remains below the default system.The ablation attributes complementary benefits to targeted skill selection, skill evolution, and adding new skills.
  • Case Study: Evolved skills specialize by domain: LoCoMo emphasizes temporal and activity details, whereas ALFWorld emphasizes action constraints and object locations.The case study links these specializations to recurring information needs in conversational and embodied data.
  • Cost Analysis: Span Size=512 provides the strongest quality-cost balance, with higher quality and fewer runtime tokens and calls than several baselines.Larger spans reduce cost but may hurt performance, making span size an efficiency-control parameter.

A.1. More Comparison Experiments

Additional ALFWorld comparisons test MemSkill with stronger baselines and without in-context demonstrations. MemSkill remains best across base models and splits, including transfer and the stricter no-demonstration setting.

  • More Comparison Experiments: 80.36 average success rate with LLaMA surpasses LightMem by 5.53 points on ALFWorld.On ALF-Unseen, success rises from 75.37 to 83.58 while average steps fall from 20.69 to 16.63.
  • More Comparison Experiments: 81.29 average success rate with Qwen surpasses Expel by 7.61 points under transfer evaluation.MemSkill is not trained on the Qwen base model or ALFWorld dataset and also requires fewer interaction steps.
  • Without In-Context Demonstrations: 57.71 average success rate without demonstrations surpasses LightMem by 11.72 points with LLaMA.The ALF-Unseen improvement is from 46.27 to 59.70, and MemSkill uses fewer interaction steps than all baselines.
  • Without In-Context Demonstrations: 63.83 average success rate without demonstrations improves over the strongest baseline by 8.03 points with Qwen.MemSkill also obtains the lowest interaction-step count on both ALF-Seen and ALF-Unseen.

A.2. More Results on Appworld

MemSkill generalizes to AppWorld tool-use settings with and without in-context demonstrations. It maintains strong pass rates while reducing execution steps, especially when memory has more room to contribute.

  • Evaluation setup: Table 6 compares MemSkill with memory baselines on AppWorld Test-Normal and Test-Challenge using Pass Rate and execution steps.The evaluation includes settings with and without in-context demonstrations.
  • Results with demonstrations: MemSkill achieves the best average PR across both base models under the default demonstration-based setting.The demonstration-based setting partially saturates performance, particularly with LLaMA, because demonstrations provide substantial task-specific guidance.
  • Results with demonstrations: MemSkill achieves the best overall pass rate and fewest execution steps across both AppWorld splits despite strong demonstration guidance.This indicates gains in both task success and execution efficiency under the default setting.
  • Results with demonstrations: With Qwen, MemSkill consistently outperforms prior memory methods and the No-Memory baseline while using the fewest execution steps on Test-N and Test-C.The Qwen setting leaves more room for memory-based improvement than the corresponding demonstration-based LLaMA setting.
  • Results without demonstrations: Without demonstrations, MemSkill achieves the best average PR under both base models with consistent gains on Test-N and Test-C.This controlled setting supports an independent contribution from learned memory skills rather than task guidance supplied by demonstrations.
  • Overall conclusion: Across settings, MemSkill generalizes to interactive tool use and improves pass rate over prior memory methods in complex multi-step tasks.The clearest gains appear when the base model leaves more room for memory-based improvement.

A.3. Experimental Results on Small Models

MemSkill remains effective with the smaller Llama-3.1-8B-Instruct backbone. It improves conversational memory metrics and retains gains on LongMemEval without training on that base model or dataset.

  • Results: MemSkill achieves the best performance on both LoCoMo and LongMemEval with Llama-3.1-8B-Instruct.The evaluation examines whether learned memory skills remain effective with a more limited-capacity base model.
  • Results: MemSkill improves both F1 and L-J over prior memory methods on the two conversational benchmarks.These gains indicate useful support for smaller backbone models.
  • Transferability: LongMemEval improvements persist without training on this base model or dataset, suggesting transfer across backbones and datasets.The results indicate that the learned skills are not tightly coupled to one large backbone.

A.4. Training Stability

MemSkill stabilizes skill-bank evolution through rollback, hard-case selection, and controlled update intervals. The authors report progressively improving skill banks across evolution rounds.

  • Stability mechanisms: Snapshot rollback prevents harmful skill updates from permanently degrading the memory system.After each evolution round, an underperforming skill bank is replaced by the best-performing snapshot observed so far.
  • Stability mechanisms: Difficulty-based hard-case selection focuses the designer on recurring failure patterns rather than noisy random examples.This directs each evolution round toward systematic weaknesses in the current skill bank.
  • Stability mechanisms: Controlled evolution intervals and capped modifications make skill-bank changes more gradual and avoid abrupt updates.The update strategy reduces sensitivity to noisy feedback and the risk of unstable skill drift.
  • Observed behavior: The learned skill bank improves progressively over evolution rounds under the combined stabilization mechanisms.The reported mechanisms provide a stable basis for self-evolving memory skills.

B.1. Evaluation Details

The evaluation spans conversational QA, transfer under increasing context lengths, and embodied interaction, with span-level memory construction and mechanisms for mining and stabilizing difficult cases. ALFWorld is adapted through offline expert trajectories and non-overlapping experience and evaluation subsets to test transfer across recurring task types.

  • Evaluation settings: MemSkill is evaluated on LoCoMo, LongMemEval-S, HotpotQA, and ALFWorld using distinct dataset protocols.LoCoMo uses a 6/2/2 sample split; LongMemEval-S contains conversations of roughly 100K tokens; HotpotQA varies context length from 50 to 200 documents; ALFWorld uses expert trajectories and official seen/unseen splits.
  • Evaluation settings: Span-level evaluation uses 512-token spans instead of turn-by-turn updates, reducing LLM calls and improving evaluation efficiency.
  • Designer mechanisms: The hard-case buffer records queries, references, retrieved memories, predictions, rewards, and failure counts, while removing stale or excess cases.
  • Designer mechanisms: The designer prioritizes cases using task reward and cumulative failures, then clusters queries semantically to diversify representative failure types.
  • Designer mechanisms: New skills receive a temporary logit-level exploration incentive for 50 steps, with a linearly decaying target threshold to encourage adoption without persistent bias.
  • Designer mechanisms: Evolution cycles score the average reward over the last quarter of training steps, roll back non-improving skill banks, and early-stop after repeated stagnation.The cycle may use 100 controller-training steps, and the best skill-bank snapshot is returned after patience is exhausted.
  • ALFWorld protocol: ALFWorld is converted into offline training from expert trajectories, with per-task-type experience and evaluation subsets kept non-overlapping.This design supplies a controlled generalization signal across trajectories sharing goal templates and recurrent interaction patterns.

B.4. Details on Training Objectives

MemSkill trains a controller to select ordered skill sets for sequential trace spans and uses memory-dependent task rewards to optimize selection with PPO. Its skill bank begins with primitive memory actions and can incorporate evolved skills that target detailed activity, entity, and temporal information.

  • Controller and rewards: PPO optimizes the controller with a Top-K joint probability, while value-function learning, GAE, and entropy regularization support policy optimization and exploration.
  • Controller and rewards: At each span-processing step, the controller observes the text and current memory state, scores the evolving skill bank, and selects an ordered Top-K set without replacement.
  • Controller and rewards: After constructing a trace-specific memory bank, MemSkill evaluates memory-dependent queries and uses the resulting task score, such as F1 or success rate, as episode reward.
  • Controller and rewards: Gumbel-Top-K sampling adds noise to skill logits and selects the K largest values, enabling stochastic exploration of skill subsets.
  • Primitive skills: The initial skill bank includes INSERT, UPDATE, DELETE, and SKIP primitives for adding, revising, removing, or leaving memories unchanged.
  • Primitive skills: INSERT stores concise, durable, non-duplicate facts, whereas UPDATE revises matching memories while preserving accurate details and avoiding new items.
  • Primitive skills: DELETE requires explicit evidence that a memory is incorrect, outdated, or superseded, and SKIP emits NOOP when selected skills require no action.
  • Evolved skills: Evolved LoCoMo skills add structured capture of activity details, entity nuances, and temporal context while restricting outputs to explicitly stated information.

F. Limitations and Societal Impact

The paper limits its claims to benchmarked research settings for skill-conditioned memory construction and identifies deployment requirements beyond its core algorithm. It also frames the skill-based formulation as potentially lowering engineering barriers for adaptive, reusable memory mechanisms.

  • Limitations: MemSkill is evaluated in benchmarked research settings, while practical deployment may additionally require privacy protection, user consent, access control, and data retention mechanisms.
  • Societal impact: The skill-based formulation may lower the engineering barrier for building memory-augmented agents by allowing memory behaviors to be learned, reused, and evolved from interaction traces.
Loading 2602.02474v2…