Source-linked AI summary
EvoArena: Tracking Memory Evolution for Robust LLM Agents in Dynamic Environments
Jundong Xu, Qingchuan Li, Jiaying Wu, Yihuai Lan, Shuyue Stella Li, Huichi Zhou, Bowen Jiang, Lei Wang, Jun Wang, Anh Tuan Luu, Caiming Xiong, Hae Won Park, Bryan Hooi, Zhiyuan Hu
TL;DR
Most LLM-agent evaluations use static environments, despite deployment settings evolving over time. EvoArena measures performance under persistent evolution, while EvoMem stores patch-based memory histories; EvoMem improves robustness across evolving and standard benchmarks.
Problem
Most agent benchmarks evaluate static snapshots, providing limited evidence about persistent environment evolution and version-aware adaptation in deployment.
Method
EvoArena evaluates progressively evolving workflows, software repositories, and user preferences, while EvoMem records memory changes and their update context as patches.
Results
1.5% average performance gain across agents and backbone models, with EvoMem also improving chain-level accuracy by 3.7% across evolving tasks.
Takeaways & Limitations
The findings support evaluating and improving agents’ ability to track changes, preserve historical context, and adapt across evolving environments.
Takeaways & Limitations
EvoArena currently covers executable workflow changes, software repository evolution, and long-horizon preference shifts, leaving other evolving settings for future study.
Abstract
from arXiv · showhide
Large language model (LLM) agents have achieved strong performance on a wide range of benchmarks, yet most evaluations assume static environments. In contrast, real-world deployment is inherently dynamic, requiring agents to continually align their knowledge, skills, and behavior with changing environments and updated task conditions. To address this gap, we introduce EvoArena, a benchmark suite that models environment changes as sequences of progressive updates across terminal, software, and social domains. We further propose EvoMem, a patch-based memory paradigm that records memory evolution as structured update histories, enabling agents to reason about environmental evolution through changes in their memory. Experiments show that current agents struggle on EvoArena, achieving an average accuracy of 39.6% across evolving terminal, software, and social-preference domains. EvoMem consistently improves performance, yielding an average gain of 1.5% on EvoArena and also improving standard benchmarks such as GAIA and LoCoMo by 6.1% and 4.8%. Beyond individual tasks, EvoMem further improves chain-level accuracy by 3.7% on EvoArena, where success requires completing a consecutive sequence of related evolutionary subtasks. Mechanistic analysis shows that EvoMem improves evidence capture in the memory, indicating better preservation of complete evolving environment states. Our results highlight the importance of modeling evolution in both evaluation and memory for reliable agent deployment.
1 Introduction
The introduction identifies persistent environment evolution as an underexplored deployment challenge and presents EvoArena and EvoMem to evaluate and support robust agent adaptation. Results show that agents struggle in evolving settings, while patch-based memory improves performance, chain completion, and evidence retention.
- Motivation: Agents are typically evaluated on static environment snapshots, despite deployment settings where interfaces, rules, workflows, codebases, and user preferences evolve across versions.Recent dynamic evaluations often add freshness or interaction realism but rarely test persistent evolution within the same setting.
- EvoArena: EvoArena evaluates agents across progressively evolving workflow, software, and social environments while preserving the underlying goal or setting.Each environment forms a release chain in which agents must solve tasks, identify relevant updates, and adapt across versions.
- EvoMem: EvoMem records memory evolution through an append-only, git-like patch history containing pre-update memory, post-update memory, rationale, and supporting evidence.This structure makes environmental changes traceable by preserving what changed, why it changed, and which evidence justified the update.
- Results: 1.5% average performance gain is observed across agents and backbone models, while existing agents struggle under environment evolution.The introduction attributes failures partly to state collapse, where memory-based agents retain only a latest state even when different versions require different behaviors.
- Results: 3.7% chain-level accuracy improvement shows that EvoMem supports completing sequences of related tasks under continuous environment evolution.Mechanistic analysis further finds stronger evidence retention, including on PersonaMem-Evo.
2 Related Work
Prior agent benchmarks increasingly model realistic interaction and some environmental dynamism, while self-evolving agents and memory systems primarily improve agents or update stored knowledge. EvoArena instead studies changes in the external environment.
- Dynamic and Evolving Agent Benchmarks: Agent benchmarks cover web navigation, software engineering, tool use, device control, terminal workflows, enterprise tasks, and personalized memory.Recent dynamic benchmarks improve evaluation freshness or interaction realism through refreshed software tasks, asynchronous events, or generated variants.
- Self-Evolving Agents and Memory: Self-evolving agents improve behavior through reflection, skill accumulation, reusable skill refinement, or scaffold adaptation, whereas EvoArena studies external-environment changes.Existing memory systems continuously update stored knowledge, but the passage contrasts these approaches with EvoArena’s focus on environmental evolution.
3 EvoArena: Benchmarking Agents under Persistent Environment Evolution
EvoArena benchmarks agents on persistent environment evolution across executable workflows, software repositories, and user preferences. Its versioned chains preserve prior changes while testing adaptation to new operational, codebase, and preference conditions.
- Benchmark scope: EvoArena covers three evolution regimes: executable workflows, software, and preferences, represented by Terminal-Bench-Evo, SWE-Chain-Evo, and PersonaMem-Evo.These domains capture complementary forms of non-stationarity, including changes to terminal dependencies and interfaces, software codebases, and user preferences.
- Executable workflow evolution: Each terminal version inherits the realized environment of the preceding version, so earlier path, dependency, permission, interface, and validation changes persist unless explicitly revised.The construction jointly edits instructions, environments, files, reference solutions, tests, and metadata in chronological order.
- Executable workflow evolution: 356 evolved terminal versions were constructed from 89 initial tasks as five-version workflow chains; after removing 4 invalid versions, 352 evolved tasks and 441 total instances remained.Chains contain four to five versions, with mean length 4.96 and median length 5.
- Software evolution: 29.8% of 443 non-initial software milestones modify a file touched by an earlier milestone, while 14.2% modify a file touched by the immediately preceding step.Each milestone includes at least one Fail-to-Pass test, averaging 7.13, and all instances include Pass-to-Pass tests.
- Preference evolution: 505 preference-inference questions span 10 persona-level conversations and four balanced types: single-pattern transfer, multi-pattern synthesis, temporal trajectory prediction, and conflict resolution.Persona histories have a median of 597 messages and 174.7K tokens, while temporal questions require 2–10 source preferences with median 6 and mean 6.2.
4 EvoMem: Patch-Based Memory Evolution
EvoMem augments a base memory system with an append-only trace of meaningful memory changes, preserving overwritten states, rationales, and evidence. It retrieves version-relevant patches alongside the latest memory to support reasoning about temporal, overwritten, and version-specific information across diverse agent memory formats.
- Patch recording: EvoMem records non-additive memory updates as patches containing changed states, rationales, semantic summaries, and supporting evidence.The base updater remains unchanged; patches capture revisions, overwrites, and reinterpretations that would otherwise be discarded.
- Patch recording: An append-only patch history preserves intermediate transitions while the latest memory stores the current consolidated state.This separates action from historical context, enabling access to prior states and the evidence explaining how the current state was reached.
- Patch-augmented retrieval: At inference time, EvoMem retrieves relevant historical patches with the latest memory when queries depend on temporal changes, overwritten information, or update rationales.The retrieved patches provide versioned evidence about previous states and the updates that changed them.
- Agent-specific instantiations: EvoMem uses agent-specific memory schemas while retaining a common interface for monitoring updates, preserving meaningful overwritten states, and retrieving historical context.Instantiations cover terminal, software-engineering, conversational-memory, and skill-memory agents.
5 Experiments
Experiments evaluate EvoMem across evolving terminal, software, and preference environments, alongside standard agent benchmarks. EvoMem improves robustness, with especially larger gains when agents must complete entire evolution chains.
- Evaluation setup: EvoArena evaluates executable workflow, accumulated codebase, and evolving user-preference changes through TERMINAL-BENCH-EVO, SWE-CHAIN-EVO, and PERSONAMEM-EVO.The experiments also include GAIA and LOCOMO to test performance beyond explicitly evolving benchmarks.
- Baseline performance: 43.6% on TERMINAL-BENCH-EVO, 29.2% on SWE-CHAIN-EVO, and 46.5% on PERSONAMEM-EVO are the base agents’ average step accuracies.These results indicate difficulty tracking evolving execution conditions, accumulated codebase states, and shifting user preferences.
- Baseline performance: 21.5% on TERMINAL-BENCH-EVO, 10.6% on SWE-CHAIN-EVO, and 39.1% on PERSONAMEM-EVO are the base agents’ corresponding chain accuracies.Chain-level performance is substantially lower because success requires maintaining correctness across dependent evolution subtasks.
- EvoMem results: 2.4% on TERMINAL-BENCH-EVO, 0.5% on SWE-CHAIN-EVO, and 1.8% on PERSONAMEM-EVO are EvoMem’s average improvements across EvoArena subsets.The gains span terminal-interface and validation shifts, software milestones, and preference updates.
- EvoMem results: 6.1% on TERMINAL-BENCH-EVO, 2.9% on SWE-CHAIN-EVO, and 3.0% on PERSONAMEM-EVO are EvoMem’s chain-level improvements.These exceed the corresponding step-level gains of 2.4%, 0.5%, and 1.8%, respectively, showing stronger benefits for dependent task sequences.
6 Analysis: When and Why Does EvoMem Help?
EvoMem helps when agents operationalize retrieved transitions, preserve earlier code behavior, and retain complete evolving evidence for downstream reasoning. Its benefits are strongest for temporal or dispersed evidence, while accuracy and token usage vary substantially across models and domains.
- Terminal-Bench-Evo: EvoMem gains rise from 3.1% without an explicit patch example to 6.5% with one, while high evolved-requirement coverage yields 5.3% versus 2.1% for low coverage.These results support the operationalization hypothesis: agents must attend to changed requirements rather than stale procedures.
- SWE-CHAIN-EVO: EvoMem reduces average PASS_TO_PASS regression rates from 9.09% to 6.32% across backbones, with the largest reduction on Kimi-K2.6 at 3.81%.The analysis links this reduction to preserving historical code constraints while adapting to new milestones.
- PERSONAMEM-EVO: EvoMem improves PERSONAMEM-EVO accuracy from 40.5% to 42.5%, with the largest gains on temporal trajectory and multi-pattern synthesis questions, both improving by 5.2%.These question types require tracking preference changes over time or retaining multiple dispersed patterns.
- PERSONAMEM-EVO: EvoMem increases clause-level evidence capture from 89.4% to 90.3%, while category-level row-capture gains reach 4.4% for temporal trajectory and 3.5% for multi-pattern synthesis.The findings suggest that patch histories preserve coherent, complete preference states needed for downstream reasoning.
- Efficiency: Gemma-4-31B-it achieves 52.6% accuracy with 27.1M tokens versus a 28.8M cross-model average, while Kimi K2.6 reaches 51.5% using 24.5M tokens.On Terminal-Bench-Evo, GPT-5.5 reaches 62.8% accuracy at 505.0M tokens, illustrating a substantial accuracy–inference-cost trade-off.
7 Conclusion
The paper introduces EvoArena for evaluating LLM agents in environments where tasks, codebases, workflows, and user preferences evolve over time. It proposes EvoMem, a patch-based memory paradigm that records memory changes and their update context, improving performance and preservation of evolving evidence.
- EvoArena evaluates LLM agents under evolving tasks, codebases, workflows, and user preferences.
- Current agents remain limited when operating under non-stationary conditions.
- EvoMem records memory changes together with their update context through a patch-based paradigm.
- EvoMem improves performance across EvoArena and standard agent benchmarks while better preserving evolving evidence.
List of Appendices … E EvoArena Dataset Construction Details
The supplied appendix contents list additional PersonaMem-Evo analysis and extensive EvoArena dataset-construction details spanning terminal, software, and persona-preference evaluations.
- D Additional Analysis: D Additional Analysis includes a PersonaMem-Evo analysis.
- E EvoArena Dataset Construction Details: E.1 analyzes Terminal-Bench-Evo tasks, evaluation metrics, original tasks, and an evolution taxonomy.
- E EvoArena Dataset Construction Details: E.2 covers Terminal-Bench-Evo version-chain construction, quality control, oracle validation, solvability verification, metadata, and dataset statistics.
- E EvoArena Dataset Construction Details: E.3 describes SWE-Chain-Evo repository collection, domain coverage, and milestone construction.
- E EvoArena Dataset Construction Details: E.4 details SWE-Chain-Evo task packaging, Docker evaluation, chain assembly, and dataset statistics.
- E EvoArena Dataset Construction Details: E.5 covers PersonaMem-Evo seed-persona expansion, preference cleaning, and implicit interaction-history construction.
- E EvoArena Dataset Construction Details: E.6 presents PersonaMem-Evo preference evolution and out-of-distribution benchmark construction, including filtering, answer options, assembly, interaction formats, and statistics.
F EvoMem Implementation Details
The section details EvoMem implementations across four agent systems: Terminus2, OpenHands, Memento-Skill, and A-Mem.
- F.1 Terminus2 with EvoMem: The section first presents Terminus2 with EvoMem.
- F.2 OpenHands with EvoMem: It then describes OpenHands with EvoMem.
- F.3 Memento-Skill with EvoMem: The section next covers Memento-Skill with EvoMem.
- F.4 A-Mem with EvoMem: It concludes with A-Mem with EvoMem.
G Experiments Setting · H Example of EvoArena
The paper’s experiments cover evolving terminal, software, persona-memory, and established benchmark settings, followed by examples illustrating EvoArena’s three evolving domains.
- G Experiments Setting: The experiments include Terminal-Bench-Evo as an evolving terminal-domain evaluation.
- G Experiments Setting: The experiments include SWE-Chain-Evo as an evolving software-domain evaluation.
- G Experiments Setting: The experiments include PersonaMem-Evo as an evolving social-preference evaluation.
- G Experiments Setting: GAIA is included as an established benchmark for experiments beyond EvoArena.
- G Experiments Setting: LoCoMo is included as an established benchmark for experiments beyond EvoArena.
- H Example of EvoArena: An EvoArena example is provided for Terminal-Bench-Evo.
- H Example of EvoArena: Additional EvoArena examples are provided for SWE-Chain-Evo and PersonaMem-Evo.
A Limitations and Future Directions
EvoArena and EvoMem are an initial step focused on three forms of environment evolution, while future extensions could examine evolving dynamics across broader long-running systems.
- Current Scope: EvoArena evaluates executable workflow changes, software repository evolution, and long-horizon preference shifts as representative forms of environment evolution.These domains expose version-aware capabilities including adapting to updated interfaces and reasoning over accumulated codebase changes.
- Future Directions: The broader challenge also arises in robotics, embodied interaction, scientific workflows, multi-agent collaboration, and other long-running systems.Extending EvoArena could support studying physical state changes, scientific protocol updates, collaborative dependencies, and evolving multi-agent roles.
B Broader Impact · C Declaration of LLM Usage · D Additional Analysis
EvoArena and EvoMem target reliable deployment in changing environments by evaluating versioned updates and preserving grounded memory histories. The work highlights benefits for robustness and auditability while noting misuse, privacy, retrieval, and coverage risks.
- B Broader Impact: EvoArena evaluates whether agents track current environmental states and avoid obsolete behaviors as rules, interfaces, and user needs evolve.It addresses the risk that fixed-task evaluations overestimate deployment reliability.
- B Broader Impact: EvoMem records what changed, why it changed, and which evidence triggered each memory update, making agent memory evolution more inspectable.Patch histories can support auditing, failure tracing, and separation of current from previously valid knowledge.
- B Broader Impact: Persistent adaptive memory may enable misuse, expose sensitive user information, or retrieve outdated and context-specific knowledge without appropriate safeguards.The risks include long-term strategies in adversarial tasks, insufficient privacy protection, and poor applicability judgments.
- B Broader Impact: EvoArena covers representative evolution in terminal workflows, software repositories, and long-horizon user preferences, but future evaluation should broaden domains, horizons, and human-centered validation.The benchmark is intended for research evaluation and should be used with attention to dataset scope and domain coverage.
- B Broader Impact: Reliable deployment requires treating memory as an evolving record of grounded updates that supports adaptation, traceability, auditability, and privacy consciousness.The authors position EvoArena and EvoMem as foundations for future work on agents operating under changing conditions.
- C Declaration of LLM Usage: LLMs assisted with writing, polishing, and coding support, while the authors made all technical, experimental, analytical, and final manuscript decisions.The declaration attributes the technical ideas and research decisions to the authors.
D.1 PersonaMem-Evo Analysis … E.2.3 Oracle Validation and Solvability Verification
The paper analyzes EvoMem’s gains on PersonaMem-Evo and details Terminal-Bench-Evo as executable chains of evolving versions that preserve a shared workflow objective while changing surrounding conditions. It defines step- and chain-level evaluation and validates each constructed version through executable quality control and oracle-based solvability checks.
- D.1 PersonaMem-Evo Analysis: EvoMem’s largest GPT-5.5 gains occur on multi-pattern synthesis (+8.6%) and conflict resolution (+7.6%).The analysis attributes these gains to stronger use of patch evidence for preserving dispersed preference signals and arbitrating competing information.
- E.1 Terminal-Bench-Evo: Task Analysis and Evolution Taxonomy: Terminal-Bench-Evo converts all 89 original Terminal-Bench tasks into chronologically ordered chains of evolving workflow versions.Each version retains the original high-level objective while changing execution conditions such as dependencies, interfaces, filesystem structure, I/O contracts, or validation logic.
- E.1.1 Evaluation Unit and Metrics: Each benchmark episode evaluates one fixed version-specific instruction and executable environment rather than unexpected filesystem or dependency mutations during a run.Evolution is represented across discrete releases of the same workflow objective.
- E.1.1 Evaluation Unit and Metrics: Step accuracy measures success across versioned instances, whereas chain accuracy requires solving every version in a workflow chain.The two metrics respectively capture adaptation to the current executable state and reliability across inherited workflow history.
- E.1.2 Original Task Analysis: Version updates are built from structured task states covering objectives, environments, files, dependencies, interfaces, I/O contracts, and validation rules.Updates modify surrounding workflow components while preserving the high-level objective.
- E.1.3 Evolution Taxonomy: The evolution taxonomy covers problem-facing contracts, invocation interfaces, execution substrates, workspace organization, and correctness evaluation.These dimensions represent independently changing layers of terminal workflows beyond the shared task objective.
- E.2.1 Version-Chain Construction: Candidate updates are screened for semantic plausibility and objective preservation, then instantiated with revised instructions, environments, tests, reference solutions, and metadata.Retained updates are ordered from simpler to more complex changes and composed into evolution chains; difficult versions may receive human-corrected reference solutions.
- E.2.2 Quality Control / E.2.3 Oracle Validation and Solvability Verification: Each EVO stage undergoes quality control and oracle execution to ensure it is valid, executable, independently solvable, and consistent with its specification and validation tests.Versions with build failures, nondeterminism, ambiguous instructions, incomplete dependencies, inconsistent validation, or output mismatches are rejected or repaired, followed by chain-level consistency checks.
E.2.4 Metadata and Dataset Statistics … E.4.3 Chain Assembly
The paper constructs evolving terminal and software benchmarks with structured metadata, temporally ordered repository milestones, and validated evaluation environments. SWE-Chain-Evo preserves cumulative codebase changes so agents must adapt to new requirements while maintaining compatibility with earlier evolution.
- E.2.4 Metadata and Dataset Statistics: Each versioned terminal task records its chain position, evolution stage, category, changed components, instruction and environment changes, validation tests, and reference-solution information.These annotations support dataset and error analysis.
- E.2.4 Metadata and Dataset Statistics: Terminal-Bench-Evo contains 89 evolving workflow chains and 441 versioned task instances, averaging 4.96 versions per chain with a median chain length of 5.Each chain has four to five validated versions: 85 chains have five versions and 4 have four.
- E.3 SWE-Chain-Evo: Repository and Milestone Construction: SWE-Chain-Evo organizes software tasks into temporally ordered chains, where APIs, dependencies, tests, implementation choices, and architectural constraints accumulate as the evolving codebase environment.Later tasks must adapt to accumulated codebase changes.
- E.3.1 Repository Collection and Domain Coverage: The dataset retains high-quality instances from 26 repositories after milestone construction, executability checks, and test-stability filtering, selected from 50 manually curated candidates.Candidates span software domains, programming ecosystems, and codebase environments and were required to be actively maintained and sufficiently well-tested.
- E.3.2 Milestone Construction: Milestones are constructed from continuous commit-history windows by grouping semantically related commits and verifying that their code changes express one coherent development objective.Candidates are manually checked for semantic coherence, executability, and evaluation reliability; ambiguous, overly broad, trivial, or unreliable milestones are discarded.
- E.4.1 Task Format and Problem Statement Construction: Each milestone becomes a SWE-bench-style task exposing only the pre-milestone repository snapshot and a manually refined natural-language problem statement.The statement describes the intended functionality, bug fix, refactoring goal, or maintenance requirement without exposing the reference patch.
- E.4.2 Docker Evaluation Environment: Docker-based evaluation environments install dependencies, configure build and test commands, and discover Fail-to-Pass and Pass-to-Pass tests for milestone validation and behavioral preservation.Instances with flaky builds, ambiguous failures, incomplete fields, missing dependencies, or unstable test outcomes are removed, with manual test supplementation when necessary.
- E.4.3 Chain Assembly: Validated milestones are assembled into repository-native temporal chains, applying each reference update so later tasks start from codebases containing accumulated earlier changes.Chains are manually reviewed for temporal consistency, semantic continuity, executable dependency, appropriate difficulty, and reliable Docker evaluation.
E.4.4 Dataset Statistics … E.6.6 Interaction Formats and Preference Records
The merged sections define SWE-Chain-Evo and PersonaMem-Evo as sequential benchmarks for software evolution and implicit, temporally changing user preferences. PersonaMem-Evo combines long-context behavioral evidence, multi-step preference trajectories, OOD inference, shortcut filtering, and structured metadata.
- E.4.4 Dataset Statistics: SWE-Chain-Evo contains 48 milestone chains from 26 repositories and 135 milestone steps, with 2–4 milestones per chain and an average of 2.81.Most chains contain three milestones; 89 milestones contain one commit, while the largest contains 6 commits.
- E.4.4 Dataset Statistics: All 135 SWE-Chain-Evo milestones include at least one Fail-to-Pass test, averaging 3.12 tests, while most also provide Pass-to-Pass regression checks averaging 6.16 tests.Only 4 milestones lack Pass-to-Pass tests; Fail-to-Pass counts range from 1–30 and Pass-to-Pass counts from 0–61.
- E.5 PersonaMem-Evo: Persona and Interaction History Construction; E.6 PersonaMem-Evo: Preference Evolution and OOD Benchmark Construction: PersonaMem-Evo evaluates whether agents can infer, track, and generalize evolving latent user preferences from long-horizon interaction histories and implicit behavioral evidence.Questions require abstracting user-specific patterns from history and applying them in novel decision settings.
- E.5.1 Seed Persona Expansion and Preference Cleaning: Seed personas from Persona_Hub_200000.jsonl are expanded into structured demographic, occupational, personality, lifestyle, social, and contextual profiles, then cleaned, deduplicated, and topic-labeled.Anti-stereotypical preferences reduce reliance on demographic priors and encourage behavioral-evidence use.
- E.5.2 Implicit Interaction History Construction; E.6.6 Interaction Formats and Preference Records: Generated episodes express preferences indirectly through choices, constraints, tone, repeated requests, or context across varied dialogue formats, with some episodes testing forgetting requests and entity disambiguation.Histories preserve episode boundaries while forming mixed-topic streams rendered under 32k and 128k context settings.
- E.6.1 Temporal Preference Evolution: Approximately 85% of eligible self-related preferences receive structured change plans with 1–5 sequential updates, generating state-conditioned conversational evidence and trajectory metadata.Updates model natural changes from experiences, constraints, routines, context, or temporal validity rather than arbitrary reversals.
- E.6.2 OOD Preference-Inference Question Generation: PersonaMem-Evo generates OOD questions requiring single-pattern transfer, multi-pattern synthesis, conflict resolution, or temporal-trajectory prediction across complexity levels L1–L3.These questions generalize from behavioral patterns in long histories to new decision settings rather than relying on persona summaries or commonsense knowledge.
- E.6.3 Dual-Blind Filtering; E.6.4 Answer-Option Construction; E.6.5 Benchmark Assembly and Metadata; E.6.6 Interaction Formats and Preference Records: Dual-blind validation accepts questions only when neither the persona profile nor no-context reasoning suffices, while balanced options, plausible distractors, and rich metadata support benchmark assembly and analysis.Records preserve preference ownership, temporal status, conversations, change-family information, and trajectory fields during mixed-topic history construction.
E.6.7 Dataset Statistics … H.3 PersonaMem-Evo
The supplied sections characterize PERSONAMEM-EVO as a demanding evaluation subset and describe EvoMem implementations that preserve base-agent workflows while recording and retrieving structured updates across terminal, software, skill, and personalization settings.
- E.6.7 Dataset Statistics: 505 questions from 10 personas form a balanced, three-level PERSONAMEM-EVO subset spanning transfer, synthesis, conflict resolution, and temporal prediction.The subset contains 49–51 questions per persona and 120 L1, 186 L2, and 199 L3 questions.
- E.6.7 Dataset Statistics: 1,553 source-preference mentions, including 801 changed and 664 static preferences, support diagnosis of evidence aggregation and preference updating.Change families include object replacement, attitude revision, conditional shift, temporal-validity shift, and attribute shift.
- E.6.7 Dataset Statistics: 597 messages and 174.7K tokens characterize the median persona history, imposing a long-context retrieval burden beyond factual recall.The benchmark also tests evidence aggregation, resistance to stereotype shortcuts, and preference updating through source composition and change-family statistics.
- F.1 Terminus2 with EvoMem: EvoMem adds chain-scoped patch memory to Terminus2, separating reusable execution ledgers from transition patches that explain adaptations to changed task or environment conditions.The current instruction remains authoritative, and EvoMem uses sanitized, compact context rather than replaying raw prior executions.
- F.2 OpenHands with EvoMem: EvoMem augments OpenHands with feature-level patch records and semantic-structural retrieval while leaving repository exploration, editing, command execution, and patch generation to the base agent.Records preserve affected files, behavioral changes, constraints, revision reasons, and bounded code evidence.
- F.3 Memento-Skill with EvoMem: EvoMem replaces Memento-Skill’s mutable TIP.md with versioned tip snapshots containing lineage, update metadata, trajectory evidence, and retrieval signals.New versions are typically created after incorrect attempts, judge feedback, and execution-trace analysis; default retrieval uses BM25 with k = 2.
- F.4 A-Mem with EvoMem: EvoMem extends A-Mem’s consolidated memory graph with patch construction, indexing, and patch-conditioned retrieval to represent non-additive memory revisions.This supports questions about current preferences, replaced earlier states, and which revision is the latest valid belief.