PaperlayerView exact revision on arXiv

Source-linked AI summary

MemScope: Evolving Agent Memory Architectures through Localized Code Repair

Miao Su, Yucan Guo, Zixuan Li, Long Bai, Zhongni Hou, Xiaolong Jin, Jiafeng Guo

arXiv:2610.21170v1cs.AI

TL;DR

Memory architectures are usually fixed after deployment, while coarse outcome feedback does not directly locate code-level failures. MemScope uses traces, runtime memory, and provider code to guide bounded in-place repairs; its evaluations report improvements across diagnostic and unseen-task comparisons.

  • Problem

    Scores and compressed summaries do not directly connect observed failures to implementation regions in memory architectures with interacting execution components.

    Source: §1 · p. 2

  • Method

    MemScope’s Patch Agent inspects raw traces, runtime memory, and provider code to identify repair targets and make bounded, in-place edits.

    Source: §1 · pp. 2–3

  • Results

    Evolved DynamicCheatsheet improves over the original on xBench and ALFWorld unseen tasks, with ALFWorld success at 71.64% versus 50.75%; trace-plus-memory accuracy is 86.67% versus 63.33% for summary-only feedback.

    Source: §5 · p. 9

  • Takeaways & Limitations

    The reported results support fine-grained execution evidence as a basis for localized, in-place memory-architecture evolution.

    Source: §5 · p. 9

Abstract

from arXiv · show

Agent memory architectures define how LLM agents store and reuse experience across tasks, but they are typically fixed after deployment, limiting their adaptability across task types. Recent self-evolving approaches instead adapt these memory architectures using execution feedback, typically represented as evaluation scores or compressed summaries. However, such coarse-grained feedback is insufficient for precise failure localization in current agent systems, where long-horizon task execution involves multiple interacting components. Without a precise failure location, existing approaches lack a clear code-level target for modification, leaving the update scope weakly constrained and often leading to large-scale rewrites of an otherwise capable memory architecture. We propose MemScope, a framework for evidence-grounded, localized evolution of an existing memory architecture. Its tool-using Patch Agent jointly inspects raw execution traces, runtime memory state, and current architecture code to identify repair targets and apply bounded, in-place edits under provider-identity and memory-compatibility constraints. On GAIA, the trace-and-memory inspection variant achieves 86.67% accuracy, compared with 70.00% for trace-only and 63.33% for summary-only feedback. Under xBench tasks, the evolved memory architecture improves over its original version from 49 to 55 correct answers with DeepSeek V3.2 and from 50 to 53 with Kimi K2.7 Code. On disjoint ALFWorld unseen tasks, success rises from 50.75% to 71.64%. Across archived architecture-snapshot transitions, the median code-change ratio is 4.4%. These results support fine-grained execution evidence as a basis for localized, in-place memory-architecture evolution.

1 INTRODUCTION

MemScope addresses the difficulty of locating code-level repairs in memory architectures whose behavior depends on interacting execution components. It uses execution evidence for localized, in-place updates and reports gains across diagnostic and benchmark comparisons.

1 INTRODUCTION

  • Evaluation scores and compressed summaries do not directly identify an implementation region for failures in long-horizon tasks.Failures may reflect absent memories, retrieval filtering, an early-return path, or problems after useful guidance has already been delivered.Source: §1 · p. 2
  • MemScope’s Patch Agent inspects raw traces, stored memories, retrieval decisions, and provider code before selecting a candidate repair region.It can instead make a no-patch decision when evidence is insufficient.Source: §1 · p. 2
  • 83.64% on GAIA and 78% on xBench for MemScope + Flash-Searcher exceed 67.88% and 70% for MemEvolve + Flash-Searcher, respectively.Matched-start xBench comparisons also favor the evolved provider: DeepSeek improves from 49/70 to 55/70 and Kimi from 50/70 to 53/70.Source: §1 · p. 2
  • MemScope turns diagnosis into bounded, in-place code edits that preserve provider identity and compatibility with existing memory records.Figure 1 contrasts complete-architecture proposals with a localized edit to the existing provider.Source: Figure 1 · p. 2, §1 · p. 3
  • 86.67% GAIA diagnostic accuracy for trace-and-memory inspection exceeds 63.33% for summary-only feedback.The introduction also reports matched-start xBench gains and ALFWorld unseen-task success rising from 50.75% to 71.64%.Source: §1 · p. 3

2 RELATED WORK

Prior memory systems commonly retain fixed executable architectures, while self-evolving approaches vary in whether they change memory content, instructions, or executable mechanisms. MemScope focuses on localized repair of an existing provider.

2.1 AGENT MEMORY

  • Most agent-memory systems retain fixed architectures after deployment, despite using memory for personalization, experience reuse, and long-term coherence.The cited examples include preference, temporal, persona, and execution-trace memories.Source: §2.1 · p. 3
  • Retrieved traces can propagate errors or encourage misaligned replay, while MemScope adapts provider code for task-family storage and retrieval.The skill document specifies how the provider is used for the current task stream.Source: §2.1 · p. 3

2.2 SELF-EVOLVING AGENTS

  • Many self-evolving approaches change retained content or memory-construction instructions while leaving executable architecture fixed; other work evolves sequential memory or agent cognition.Evo-Memory revises memory over sequential task streams, while FORGE turns failure traces into shared textual memories.Source: §2.2 · p. 3
  • ALMA searches complete code-expressed architectures and MemEvolve jointly evolves knowledge and modular architectures; MemScope instead starts from an existing provider and makes local repair targets explicit.This frames MemScope’s local-evolution path as distinct from complete-architecture search.Source: §2.2 · p. 3
  • MemScope applies automated program repair to an agent’s memory architecture, grounding updates in task traces, runtime memory, and provider code.The cited program-repair literature includes repository-level repair benchmarks such as SWE-bench.Source: §2.2 · p. 3

3 METHOD

MemScope evolves reusable memory skills by locally repairing their active providers while keeping runtime memory separate. Its loop uses task evidence to trigger diagnosis, propose validated edits, and record patch outcomes.

3 METHOD

  • Figure 2 depicts an existing skill whose provider is edited in place while its separate runtime memory bank continues to update during task use.The provider-evolution runs start from the existing DynamicCheatsheet architecture.Source: §3 · pp. 3–4, Figure 2 · p. 4

3.1 EVOLUTION TARGET

  • Each reusable skill pairs SKILL.md guidance with an executable provider implementing memory retrieval and storage, while runtime memory remains separate.MemScope records the design lesson in SKILL.md when it locally updates the provider.Source: §3.1 · p. 4

3.2 MEMSCOPE EVOLUTION LOOP

  • The evolution loop routes probe tasks to a suitable existing skill, then executes remaining tasks using its provider and skill-specific runtime memory bank.Raw trajectories are retained for diagnosis, and providers distill reusable information into memory; a repair trigger fires when segment accuracy falls sufficiently below its historical reference and enough tasks fail.Source: §3.2 · pp. 4–5

3.3 PATCH AGENT: FINE-GRAINED DIAGNOSIS AND LOCALIZED UPDATES

  • The Patch Agent uses execution evidence, the selected skill, and Patch History to identify a repair target and propose a localized update with a manifest.It inspects traces, runtime memory, and provider code; exact old-to-new replacements undergo local checks, and later outcomes are recorded for subsequent decisions.Source: §3.3 · p. 5

4 EXPERIMENTS

Across GAIA, xBench, and unseen ALFWorld tasks, evolved providers generally outperform their original or comparison systems. Diagnostic results favor trace-and-memory inspection, while repair examples and archived changes show how evidence can guide targeted provider edits.

4.1 EXPERIMENTAL SETUP

  • The evaluation uses GAIA, xBench DeepSearch-2505, and ALFWorld, with five evolution episodes over 30 development tasks and online evaluation.For xBench and ALFWorld, code persists across episodes while memory resets between episodes and accumulates within each.Source: §4.1 · p. 6

4.2 MAIN RESULTS ON GAIA AND XBENCH

  • Under DeepSeek V3.2, MemScope + Flash-Searcher reaches 83.64% pass@1 on GAIA and 78% on xBench, versus 67.88% and 70% for MemEvolve + Flash-Searcher.The open-source comparison also reports 60.61% to 83.64% on GAIA and 68% to 78% on xBench against Flash-Searcher; with Kimi K2.7 Code, MemScope reaches 78.18% on GAIA and 74% on xBench.Source: §4.2 · p. 7, Table 2 · p. 7
  • On matched-start xBench questions, evolved versus original providers score 55/70 versus 49/70 with DeepSeek V3.2 and 53/70 versus 50/70 with Kimi K2.7 Code.The evolved provider retains 95.92% and 92.00% of original successes under DeepSeek and Kimi, respectively, with recovered failures outnumbering regressions in both runs.Source: §4.2 · p. 7, Table 3 · p. 8

4.3 GENERALIZATION TO UNSEEN ALFWORLD TASKS

  • On unseen ALFWorld tasks, MemScope succeeds on 48/67 (71.64%), versus 34/67 (50.75%) for the original DynamicCheatsheet and 45/67 (67.16%) for ALMA.Methods start with empty memory and follow the same memory-collection sequence before evaluation.Source: §4.3 · p. 8, Table 4 · p. 8

4.4 ABLATION AND ANALYSIS

  • Trace + Memory Inspection achieves 86.67% GAIA diagnostic accuracy, versus 70.00% for Trace-only and 63.33% for Summary-only.Patch History reaches 83.33%; relative to Summary-only, Trace-only uses 24.4% fewer average task steps and 7.2% less task time.Source: §4.4 · p. 8, Table 5 · p. 8
  • Recorded repairs adjust retrieval thresholds using later outcomes, and one xBench case leaves the provider unchanged after finding failed tasks had already received correct guidance.The ALFWorld repair retained a 0.3 threshold and changed the no-match fallback to return empty guidance; both described revisions were installed.Source: §4.4 · pp. 8–9, Table 6 · p. 9
  • Across 14 archived provider transitions, the median code-change ratio is 4.4%, and the median transition touches one function.Appendix B.5 discusses diagnosis, code-modification, and validation costs.Source: §4.4 · p. 9
  • GAIA guidance coverage changes from 98.8% to 99.4%, while accuracy among guided tasks changes from 65.6% to 84.1%; code and memory state both vary.The larger observed difference is in guided-task outcomes rather than guidance frequency.Source: §4.4 · p. 9, Figure 3 · p. 9

5 CONCLUSION

MemScope uses trace- and memory-grounded code repair to evolve an existing memory provider, with reported gains on GAIA, xBench, and unseen ALFWorld tasks. The accompanying repair prompt constrains edits to localized changes and specifies evidence inspection and validation steps.

5 CONCLUSION

  • MemScope turns evolution of an existing memory provider into trace- and memory-grounded code repair, with results supporting localized repair during continued use.The conclusion reports 83.64% on GAIA and 78% on xBench with DeepSeek V3.2, and 71.64% versus 50.75% on ALFWorld unseen tasks.Source: §5 · p. 9

AI USE STATEMENT

  • Generative AI assisted experiment planning, tooling, result interpretation, and manuscript drafting and revision; the authors remain responsible for the final content.Automated checks recomputed additional result tables and checked compilation and references, but do not replace author verification.Source: AI USE STATEMENT · p. 10

A.1 PATCH-AGENT PROVIDER REPAIR

  • The Patch Agent must inspect failed and successful traces when available, verify memory evidence before retrieval changes, and distinguish task correctness from execution completion.The prompt directs comparison of failed and successful cases before changing shared retrieval behavior.Source: §A.1 · p. 16
  • Allowed edits target local retrieval, ranking, storage, formatting, error handling, or small helper logic, while forbidding new provider identity, incompatible schemas, broad rewrites, and hardcoded answers.The patch must preserve initialize(), provide_memory(), and take_in_memory contracts.Source: §A.1 · p. 16
  • The prompt requires local checks after editing and a patch manifest recording the root cause, expected effect, risk, and validation; unsupported changes should be recorded as no_patch.If a local patch is insufficient, the manifest can instead indicate that a new provider is needed; full benchmark runs are outside the Patch Agent's allowed checks.Source: §A.1 · pp. 15–17

B REPAIR CASES AND EVIDENCE

The recorded cases exemplify evidence-guided repair decisions, and the table summarizes how localized the edits are.

B REPAIR CASES AND EVIDENCE

B.1 GAIA FAILURE-LOCALIZATION CASE

A GAIA failure case contrasts generic summary-driven guidance with a code-level diagnosis derived from raw execution evidence.

B.1 GAIA FAILURE-LOCALIZATION CASE

  • Summary-only groups zero-step failures with incorrect answers and adds a generic verification rule, whereas trace-only identifies an early return under MemoryStatus.IN as the proposed cause of missing guidance.The trace-based proposal removes that guard, pinpointing a code path the summary-based proposal leaves unchanged.Source: §B.1 · p. 18

B.2 RECORDED NO-PATCH DECISION

The recorded xBench decision leaves retrieval unchanged because the inspected evidence does not justify a local provider change. The proposer cites the current threshold’s behavior, failures after correct guidance, and weak outcomes from an earlier threshold patch.

B.2 RECORDED NO-PATCH DECISION

  • The Kimi proposer retains the current provider after inspecting retrieved memory, downstream failures, and a previous unsuccessful threshold change.Table 7 records the no-patch decision.Source: §B.2 · p. 18, Table 7 · p. 18
  • The 0.7 threshold correctly blocks an irrelevant correct memory in task 28, while lowering it risks guided_wrong.Source: §B.2 · p. 18
  • Tasks 24 and 29 fail after correct guidance, and an earlier threshold-based gating patch had weak follow-up outcomes, so retrieval changes lack support.Source: §B.2 · p. 18

B.3 CONTRASTING RETRIEVAL REPAIRS

The inspected failure patterns lead to opposite retrieval-threshold repairs in the DeepSeek and Kimi cases. The examples show that the repair direction follows the diagnosed memory and failure pattern.

B.3 CONTRASTING RETRIEVAL REPAIRS

  • DeepSeek’s patch searches all records, lowers the threshold from 0.7 to 0.5, expands the candidate pool, and boosts correct records.Its proposer attributes failures to correct-only retrieval and fallback to stale global cheatsheet text.Source: §B.3 · p. 18
  • Kimi’s patch raises the threshold from 0.7 to 0.8 after the proposer finds 37 correct records among 54 and diagnoses weak cross-domain matches.Source: §B.3 · p. 18
  • The contrasting threshold changes illustrate that repair depends on inspected memory and failure patterns rather than a uniform rule.Source: §B.3 · p. 18

B.4 EDIT LOCALITY

Across 14 archived provider-snapshot transitions, the median recorded per-transition normalized code-change fraction is 4.4%. The reported edit scope also has medians of 16.5 lines and one function.

B.4 EDIT LOCALITY

  • 4.4% is the median recorded per-transition normalized fraction across 14 archived provider-snapshot transitions.The transitions comprise one Trace + Memory Inspection, six Summary-only, and seven Trace-only transitions.Source: §B.4 · p. 18, Table 8 · p. 19
  • 16.5 lines and one function are the median changes, against median provider-size proxies of 369 lines and 16 functions.Source: §B.4 · p. 18
  • The 4.4% fraction is computed across individual transitions, not by dividing the line-count medians.Source: §B.4 · p. 18

B.5 EVOLUTION COST ANALYSIS

The cost analysis contrasts diagnosis, code modification, and post-modification validation in complete architecture generation and localized repair. MemScope selects code regions for replacement, checks local updates during continued execution, and records later outcomes in Patch History.

B.5 EVOLUTION COST ANALYSIS

  • Diagnosis is shared by complete architecture generation and localized repair, while MemScope uses execution evidence and prior repair outcomes to identify a code region.MemEvolve diagnoses architectural defects before redesign, and ALMA analyzes candidate designs and evaluation logs.Source: §B.5 · p. 19
  • Complete generation emits a full provider implementation, whereas MemScope replaces selected regions and reuses the rest of the implementation.Table 8 measures changed code, not model-token expenditure.Source: §B.5 · p. 19
  • MemScope checks a local code update and continues task execution with the updated architecture and retained runtime memory.Source: §B.5 · p. 19
  • Patch History records subsequently recovered failures and regressed successes so later proposals can correct an unsuccessful direction.Behavioral feedback is collected along a continuing evolution trajectory rather than through separate task evaluations of multiple complete candidates.Source: §B.5 · p. 19
Loading 2610.21170v1…