PaperlayerView exact revision on arXiv

Source-linked AI summary

Self-Evolving LLM Memory Extraction Across Heterogeneous Tasks

Yuqing Yang, Tengxiao Liu, Wang Bill Zhu, Taiwei Shi, Linxin Song, Robin Jia

arXiv:2604.11610v1cs.CL

TL;DR

Persistent assistants must extract useful memories from diverse conversations, but static prompts and existing self-evolving frameworks do not reliably generalize across heterogeneous tasks. The paper formalizes the task, builds BEHEMOTH from 18 datasets, and introduces CluE, which clusters extraction scenarios for local analysis and cross-cluster prompt updates. CluE consistently outperforms prior self-evolving frameworks, while the benchmark and method remain bounded by current task diversity and the extraction-only scope.

  • Problem

    Memory types vary across heterogeneous interactions, but static prompts and existing self-evolving frameworks are limited when tasks are diverse.

  • Method

    The paper formalizes utility-driven single-step extraction, constructs BEHEMOTH from 18 datasets, and proposes CluE’s cluster-based prompt evolution.

  • Results

    CluE consistently outperforms compared self-evolving frameworks on heterogeneous tasks.

  • Takeaways & Limitations

    BEHEMOTH provides a testbed for evaluating memory-extraction approaches under heterogeneous demands, while CluE is designed for stable learning across such distributions.

  • Takeaways & Limitations

    BEHEMOTH covers 18 datasets but does not capture the greater diversity and longer histories of real-world deployments, and the work addresses extraction rather than the full memory lifecycle.

Abstract

from arXiv · show

As LLM-based assistants become persistent and personalized, they must extract and retain useful information from past conversations as memory. However, the types of information worth remembering vary considerably across tasks. We formalize the \textit{heterogeneous memory extraction} task and introduce \textbf{BEHEMOTH}, a benchmark that repurposes 18 existing datasets spanning personalization, problem-solving, and agentic tasks, using a downstream utility-driven metric for systematic evaluation. Our empirical analysis confirms that no single static extraction prompt dominates across all task categories, and that existing self-evolving prompt optimization frameworks, originally designed for homogeneous distributions, degrade when training tasks are heterogeneous. To address this, we propose \textbf{CluE}, a cluster-based self-evolving strategy that groups training examples into clusters by extraction scenarios, analyzes each cluster independently, and synthesizes cross-cluster insights to update the extraction prompt. Experiments on BEHEMOTH show that CluE generalizes effectively across heterogeneous tasks ($+$9.04\% relative gain), consistently outperforming prior self-evolving frameworks.

1 Introduction

Persistent assistants must decide what to retain across diverse conversations, but fixed extraction rules are tailored to narrow contexts. The paper formalizes heterogeneous extraction, introduces BEHEMOTH, and proposes CluE for adapting prompts across task scenarios.

  • Persistent assistants need to retain useful personal facts, preferences, and prior experiences across conversations.
  • Static extraction rules are typically tailored to narrow applications, whereas real-world assistants encounter casual, technical, and problem-solving interactions.
  • BEHEMOTH repurposes 18 datasets across personalization, problem-solving, and agentic tasks for heterogeneous memory-extraction evaluation.
  • No single static extraction prompt dominates all task categories, motivating self-evolving strategies that learn from task feedback.
  • CluE clusters examples by extraction scenarios, analyzes clusters locally, and synthesizes cross-cluster insights to update prompts.
  • CluE consistently outperforms compared self-evolving frameworks under heterogeneous task distributions.

2 Related Work

Prior memory systems extract either personalized facts or experiential and strategic knowledge, while self-evolving methods refine prompts from feedback. These approaches motivate evaluating extraction strategies across heterogeneous tasks.

  • Explicit memory systems target information such as preferences, dates, relationships, keywords, and contextual descriptions.
  • Other memory systems extract experiential knowledge, strategic insights, or lessons from successful and failed trajectories.
  • Self-evolving frameworks iteratively refine prompts or strategies from task feedback without manual intervention.
  • General prompt-optimization methods are typically developed for single-task benchmarks, while MemEvolve evolves experiential knowledge and memory architecture jointly.

3 Task Formulation and Dataset Curation

The paper isolates single-step extraction, evaluates memories by downstream utility, and constructs BEHEMOTH from heterogeneous datasets. It measures cross-dataset generalization while accounting for task diversity and memory-type variation.

  • 3.1 Single-Step Memory Extraction: Single-step extraction maps one source conversation and an extraction prompt to a memory string, separating extraction from retrieval and management.
  • 3.1 Single-Step Memory Extraction: A generation model answers an associated target query using the extracted memory, and a task-specific reward scores the response.
  • 3.1 Single-Step Memory Extraction: Static and self-evolving prompts are evaluated, with evolving prompts optimized iteratively over training triplets containing conversations, queries, and rewards.
  • 3.2 Creating BEHEMOTH: BEHEMOTH repurposes 18 datasets into personalization, problem-solving, and agentic categories containing 5, 7, and 6 datasets respectively.
  • 3.2 Creating BEHEMOTH: The benchmark uses triplets (c, q, R), completes source conversations with Qwen3-32B, and remains extensible through dataset and generation-model choices.
  • 3.2 Creating BEHEMOTH: The three categories do not determine the memory type: appropriate extraction varies within and across categories, while systems receive only raw conversations.
  • 3.3 Evaluation Metrics: Macro accuracy weights datasets equally, while relative gain uses the geometric mean of per-dataset improvement ratios over a baseline.
  • 3.3 Evaluation Metrics: Table 1 reports static-prompt evaluation on in-distribution test sets using Macro Accuracy and Relative Gain over Simple.

4 Evaluating Static Memory Extraction

Static memory prompts improve over providing no memory, but their strengths depend on task category. Broader taxonomies are more balanced, yet greater taxonomy detail does not guarantee better extraction.

  • Memory extraction improves overall macro accuracy over the no-memory baseline, including with the minimal Simple prompt.
  • Simple raises overall macro accuracy from 37.84% to 46.00%, with gains across personalization, problem-solving, and agentic categories.
  • Category-specific prompts excel in their target domains but underperform elsewhere, while taxonomy-based prompts provide more balanced moderate improvements.
  • OpenMemory’s five-class taxonomy does not outperform Survey’s simpler two-class design, so greater prompt granularity does not guarantee better extraction.

5 Evaluating Evolving Memory Extraction

The evaluation examines self-evolving memory-extraction prompts under heterogeneous tasks and introduces CluE, which performs cluster-based updates to preserve scenario-specific insights. CluE achieves the strongest overall performance and generalizes to held-out datasets.

  • CluE: Cluster-based Evolution: CluE groups examples by similar memory-extraction scenarios and analyzes each cluster independently before synthesizing an updated prompt.Its evolution round includes summarization, clustering, cluster-based analysis, and a cross-cluster proposal step.
  • Results: Existing self-evolving frameworks trade performance across heterogeneous categories rather than improving them jointly.GEPA gains +14.08% on agentic tasks but regresses −2.16% on personalization, while MemEvolve gains +10.76% on personalization but regresses −3.25% on agentic tasks.
  • Results: +9.04% overall relative gain makes CluE the best self-evolving method, with improvements across personalization (+12.34%), problem-solving (+8.39%), and agentic (+7.22%) categories.CluE also outperforms baseline self-evolving frameworks when the extraction model is switched from Qwen3-32B to Gemini-3-Flash.
  • Out-of-distribution generalization: CluE achieves the strongest held-out result on LongMemEval, scoring 63.07 versus 56.82 for the second best.It also does not fall below the Simple prompt on GPQA-Diamond or ToolBench, unlike some baselines.

6 Further Analysis

Further analyses test CluE from a stronger seed and in continual memory extraction, then examine the structure of evolved prompts. CluE preserves performance advantages across these settings and produces structured, broadly applicable guidance.

  • Evolution From a Stronger Seed: +6.54% overall relative gain lets CluE improve on all three categories when evolving from the stronger Survey prompt.GEPA leaves the seed unchanged, while ACE and MemEvolve produce negative overall relative gains of −1.44% and −0.74%.
  • From Single-Step to Continual Memory Extraction: CluE maintains its advantage over MemEvolve in continual extraction, scoring 50.83 versus 43.33 on Game of 24 and 67.25 versus 62.57 on AlfWorld.The continual setup uses top-k retrieval with k = 1 and concatenation-based consolidation.
  • From Single-Step to Continual Memory Extraction: In continual extraction, CluE outperforms No Memory more stably, while Simple falls below or remains within variance of that baseline.The analysis attributes this setting’s sensitivity to the accumulation and compounding of low-quality memories over time.
  • Qualitative Analysis of Evolved Prompts: CluE’s evolved prompt is shorter than ACE’s and combines a complete memory taxonomy, category-specific guidelines, and domain-agnostic principles.Prompt lengths are 1,403 tokens for ACE, 1,243 for GEPA, 936 for CluE, and 539 for MemEvolve.

7 Discussion and Conclusions

The paper formalizes heterogeneous memory extraction, introduces BEHEMOTH, and proposes CluE to address limitations of static prompts and existing self-evolving frameworks. It also identifies broader applications and open challenges beyond the studied setting.

  • BEHEMOTH provides a benchmark for heterogeneous memory extraction, while CluE enables stable learning under heterogeneous training distributions.The benchmark and framework are presented as the paper’s central artifacts for evaluating and improving memory extraction across diverse tasks.
  • BEHEMOTH can evaluate memory extraction approaches beyond self-evolving frameworks, including routing-based and skill-based paradigms.The authors suggest using it to study strategies for heterogeneous memory extraction.
  • CluE may extend to settings where one agent handles heterogeneous demands across users, projects, domains, and workflows.The paper gives distinct user habits, communication styles, and cross-domain workflows as examples.
  • BEHEMOTH does not fully capture real-world scenarios with greater task diversity and longer interaction histories.The authors identify more challenging and realistic benchmarks as an important next step.
  • The work evaluates memory extraction as the first lifecycle stage, leaving storage, retrieval, and management for future study.Generalizable performance across the full memory lifecycle remains open.

A Experimental Details

The experiments construct BEHEMOTH from heterogeneous datasets and compare several self-evolving frameworks under controlled single-step and continual-memory settings. The setup also exposes a mismatch between the single-step evolution regime and continual inputs containing retrieved memories.

  • Dataset construction: BEHEMOTH examples pair a source conversation, target query, and reward function, with problem-solving and agentic conversations generated from task trajectories.Personalization datasets directly provide conversations and queries, while other task types are completed by an LLM to create trajectories.
  • Dataset construction: Training examples are sampled across datasets and randomly shuffled, allowing batches to contain multiple heterogeneous datasets.Most datasets contribute 20 in-distribution training examples and test sets capped at 200 examples; AIME contributes 50 training examples.
  • Evolution frameworks: GEPA, ACE, and MemEvolve use distinct evolution procedures, while CluE adopts MemEvolve’s exact hyperparameters.The procedures differ in validation use, reflection rounds, curation frequency, tournament rounds, and candidate generation.
  • Evaluation settings: Single-step and continual No Memory baselines differ because they evaluate different example pools and, for AlfWorld, different few-shot setups.Single-step instances pair source conversations with target queries, whereas continual examples arrive sequentially; AlfWorld uses 0-shot target answering in single-step evaluation.
  • Evaluation settings: Continual extraction remains outside the current evolution pipeline because evolved prompts see memory-free conversations during training, unlike continual inputs with retrieved memories.The paper tests transfer of single-step prompts to continual evaluation but leaves direct continual-regime evolution for future work.

B.1 Generalizing Across Extraction Backends

CluE generalizes across extraction backends: with Gemini-3-Flash, it achieves the highest overall relative gain and leads on personalization and problem-solving. Agentic performance remains below the Simple baseline because the stronger backend leaves limited headroom.

  • +3.40% overall relative gain is achieved by CluE with Gemini-3-Flash, the highest among the evaluated self-evolving frameworks.CluE also leads on Personalization (+6.66%) and Problem-Solving (+9.59%).
  • ACE drops to −3.93%, MemEvolve to −0.29%, and GEPA reaches only +0.93% overall relative gain with Gemini-3-Flash.These results contrast with CluE’s meaningful gain despite the stronger backend’s higher starting performance.
  • All evolved prompts fall below the Simple baseline on Agentic tasks with Gemini-3-Flash.The paper attributes this to limited headroom and the greater risk that over-specification harms performance.
  • CluE still substantially outperforms the No Memory baseline on Agentic tasks despite regressing relative to the Simple prompt.This indicates that extracted memories remain useful in that evaluation setting.

B.2 Efficiency Analysis

The efficiency analysis compares self-evolving frameworks by wall-clock time, optimization LLM calls, and evaluation calls. GEPA incurs the highest evaluation cost, while ACE minimizes evaluation calls but uses a sequential loop.

  • Efficiency is measured using wall-clock time, optimization LLM calls, and evaluation calls.Optimization calls cover analysis, generation, reflection, and curation; evaluation calls count target-query answers during evolution.
  • GEPA uses 1,645 evaluation calls, the highest cost, largely because validation-set scoring supports candidate selection without producing learning signals.The validation process also contributes to GEPA’s longer wall time.
  • ACE uses 1,120 evaluation calls, the fewest reported, because it does not maintain a candidate pool.Its online learning loop is inherently sequential because each step must complete before the next begins.

B.3 Cluster Evolution and Taxonomy Construction

CluE evolves extraction clusters by identifying shared scenario patterns, allowing clusters to merge or split as its analysis develops. It then reorganizes cluster insights into a unified taxonomy that can decouple categories from cluster boundaries.

  • Cluster evolution: The cluster pool begins with seven fine-grained clusters and consolidates when initially separate scenarios share common extraction patterns.Emotional Context and Relational Dynamics merges into User Preferences, while Combinatorial and Puzzle Problem-Solving merges into Technical Problem-Solving.
  • Cluster evolution: The evolved cluster set includes user preferences and emotional context, factual disambiguation and verification, procedural knowledge, and technical or scientific problem-solving.The listed final clusters also include translation within the user-preference and emotional-context grouping.
  • Cluster evolution: A separate run splits Code-based Technical Workflows from Technical Problem-Solving at Round 2 because code examples require implementation-specific extraction patterns.The cited patterns include dependency management and error handling, distinct from general reasoning strategies.
  • From clusters to memory taxonomy: The proposer synthesizes per-cluster analyses into a unified extraction prompt whose taxonomy categories need not correspond one-to-one with clusters.This permits cluster-level findings to be reorganized into broader or more specialized taxonomy sections.
  • From clusters to memory taxonomy: The final evolved prompt contains five taxonomy sections, including factual and temporal disambiguation, user preferences and emotional context, procedural and technical knowledge, reasoning, and translation requirements.These sections are mapped to four clusters, with translation and stylistic requirements sharing the first cluster’s origin.
  • From clusters to memory taxonomy: Cluster 1 is split into separate taxonomy sections for user preferences and emotional context versus translation and stylistic requirements.This decoupled design creates clean, non-overlapping categories without requiring taxonomy boundaries to match cluster boundaries.

B.4 Expanded Results

The expanded results describe broad evaluation across BEHEMOTH, static prompt baselines, and the prompt components used in CluE. They emphasize aggregate evaluation and scenario-based clustering, while also documenting the concrete extraction prompts and taxonomy instructions.

  • Evaluation: BEHEMOTH evaluates methods using per-dataset accuracy on in-distribution test sets, while category-level and overall aggregates are intended to reflect broad generalizability.Aggregates smooth dataset-specific variance, so individual dataset scores may fluctuate across methods without determining the broad comparison.
  • Static prompts: The five static prompts include Simple, Mem0, ReasoningBank, OpenMemory, and Survey, differing in whether they use minimal instructions or task-specific and broad taxonomies.Mem0 targets preferences, relationships, and plans; ReasoningBank targets strategic insights; OpenMemory uses five classes; Survey uses factual and experiential classes.
  • Evaluation: The reported tables provide per-dataset accuracy for Qwen3-32B evolving from Simple, Qwen3-32B evolving from Survey, and Gemini-3-Flash evolving from Simple.The Qwen3-32B Simple table reports standard deviations over three runs.
  • CluE prompts: CluE uses separate prompts for summarization, cluster management, and cluster analysis within its optimization pipeline.The cluster manager groups examples by extraction-scenario patterns, while the analyzer inspects cluster-specific success and failure patterns and proposes targeted prompt improvements.
  • CluE prompts: The cluster manager is instructed to group examples by the information being extracted and the extraction challenge rather than by dataset name, domain, or surface task type.Example scenario labels include procedural knowledge in lengthy dialogue, implicit preferences, causal reasoning chains, factual knowledge in noisy context, and high-level strategies in execution traces.
  • CluE prompts: The cluster analyzer examines only the tasks assigned to one cluster, identifies scenario-specific success and failure patterns, and proposes corresponding prompt-level improvements.Its context includes the round, cluster description, task IDs, example count, pair-level logs, and the current base system prompt.
Loading 2604.11610v1…