Source-linked AI summary
AMemGym: Interactive Memory Benchmarking for Assistants in Long-Horizon Conversations
Cheng Jiayang, Dongyu Ru, Lin Qiu, Yiyang Li, Xuezhi Cao, Yangqiu Song, Xunliang Cai
TL;DR
Long-horizon memory evaluation is limited by static off-policy benchmarks that are costly and may not reflect assistants’ own conversational behavior. AMemGym addresses this with structured, interactive on-policy evaluation and reports performance gaps in existing systems while supporting diagnostic optimization and self-evolution.
Problem
Existing memory benchmarks rely on static, off-policy data that omits assistants’ conversational consequences and can misdirect memory optimization.
Method
AMemGym samples structured user profiles, state-dependent questions, and state trajectories, then uses LLM-simulated users and personalized ground truth for interactive evaluation.
Results
The experiments demonstrate reuse bias and drawbacks in off-policy evaluation, provide extensive on-policy evaluation of memory systems, and identify performance gaps and their reasons.
Takeaways & Limitations
AMemGym provides a scalable, diagnostically rich environment for benchmarking, diagnosing, and optimizing conversational memory, including proof-of-concept agent self-evolution.
Takeaways & Limitations
The framework uses LLM-simulated users and synthetic data rather than real human subjects or personal information.
Abstract
from arXiv · showhide
Long-horizon interactions between users and LLM-based assistants necessitate effective memory management, yet current approaches face challenges in training and evaluation of memory. Existing memory benchmarks rely on static, off-policy data as context, limiting evaluation reliability and scalability. To address these gaps, we introduce AMemGym, an interactive environment enabling on-policy evaluation and optimization for memory-driven personalization. AMemGym employs structured data sampling to predefine user profiles, state-dependent questions, and state evolution trajectories, enabling cost-effective generation of high-quality, evaluation-aligned interactions. LLM-simulated users expose latent states through role-play while maintaining structured state consistency. Comprehensive metrics based on structured data guide both assessment and optimization of assistants. Extensive experiments reveal performance gaps in existing memory systems (e.g., RAG, long-context LLMs, and agentic memory) and corresponding reasons. AMemGym not only enables effective selection among competing approaches but also can potentially drive the self-evolution of memory management strategies. By bridging structured state evolution with free-form interactions, our framework provides a scalable, diagnostically rich environment for advancing memory capabilities in conversational agents.
1 INTRODUCTION
AMemGym addresses unreliable, costly memory evaluation by grounding interactive, on-policy conversations in structured state evolution. It supports diagnosis and optimization of conversational memory, including self-evolution from environmental feedback.
- Long-horizon assistants must organize and use memory across dialogue to support complex requests and personalized responses.
- Existing benchmarks use static, off-policy data that omits assistants’ conversational consequences, can misdirect optimization, and is costly to curate at scale.
- AMemGym generates evaluation-aligned interactions from user profiles, state-dependent questions, and state evolution trajectories, with simulated users revealing states through consistent role-play.
- Grounding free-form interactions in structured state evolution creates a scalable and diagnostically rich environment for assessing and advancing conversational memory.
- AMemGym enables on-policy evaluation of popular memory systems and reveals reuse bias and drawbacks in off-policy evaluation for personalization.
- Environmental feedback can support autonomous refinement of an agent’s memory management policy.
2 RELATED WORK
Memory benchmarks have progressed toward more realistic and automated evaluation, but manual curation remains a scale and diversity constraint. User simulation offers on-policy evaluation elsewhere, while memory evaluation requires strategically controlled state disclosure.
- Memory evaluation evolved from long-context single-turn tests to multi-turn conversational datasets with more authentic dialogue patterns.
- Manual curation limits the scale and diversity of realistic conversational benchmarks, motivating automated data generation frameworks.
- User simulators have enabled robust on-policy evaluation in tool use and long-term collaboration, but memory evaluation requires strategic, coherent state disclosure.
3 AMEMGYM
AMemGym constructs structured user-state trajectories, turns them into grounded on-policy conversations, and evaluates memory through overall and diagnostic metrics. This design connects natural interaction with verifiable personalized-answer ground truth.
- AMemGym benchmarks and optimizes personal-assistant memory through an interactive environment built around structured scenarios and tasks.
- Structured data generation: The framework samples user profiles, questions, canonical state variables, discrete values, and state trajectories to define trackable personalization states.
- Structured data generation: State transitions are driven by narrative life events, producing a ground-truth trajectory across simulation periods.
- On-policy interaction: A user simulator exposes structured states through natural utterances and role-play conditioned on the profile, current states, and conversation context.
- Structured data generation: For each question and state variant, AMemGym generates a personalized answer and accepts it only when a classifier can recover the variant from the question-answer pair.
- Evaluation metrics: Evaluation responses provide feedback for assessing, optimizing, and self-improving assistants.
- Evaluation metrics: Overall accuracy measures personalized answering and memory, while diagnostic metrics separate write, read, and utilization failures.
4 MEMORY EVALUATION WITH AMEMGYM
AMemGym evaluates native LLMs and memory systems through on-policy interactions, exposing differences that off-policy traces can obscure. Its diagnostics show that long-horizon performance depends on memory design choices and stage-specific trade-offs.
- Evaluation setup: The evaluation compares native LLMs, standard RAG, agentic write variants, and established memory agents under a common implementation framework.The four core designs differ by storage location and writing strategy; all use gpt-4.1-mini for generation and memory operations.
- On-policy versus off-policy evaluation: On-policy and off-policy evaluation produce substantially different memory-agent rankings, so off-policy traces can mislead optimization and configuration choices.The discrepancy is attributed to memory operations being coupled with each agent’s interaction patterns and conversational choices.
- Native LLMs and agents: Native LLMs perform well on short-context information use but degrade sharply as state updates lengthen the interaction history.All evaluated LLMs achieve SUB > 0.8, while most fall below 50% of their upper bounds later and some approach random guessing.
- Native LLMs and agents: AWE variants achieve the highest memory scores, outperforming native LLMs and standard RAG, while aggressive filtering in AWI can discard crucial information.These results indicate that selective, agentic information curation is more effective than storing all raw history in the evaluated setting.
- Diagnosis on memory agents: Diagnostics decompose failures into writing, reading, and utilization stages, with write and read failures generally increasing over longer interactions.Utilization failures decrease slightly because more errors are captured earlier.
- Diagnosis on memory agents: Agentic retrieval and compression improve utilization but trade off reading efficiency through information loss or reduced global access to memories.AWE has a smaller sacrifice than RAG because it agentically rewrites content for easier access; AWI reduces write failures but may lose information through compression.
- Diagnosis on memory agents: Lower update frequency and larger short-term memory increase read failures, whereas retrieval size has a non-monotonic effect mainly on writing.More retrieved memories improve recall of relevant information but also increase noise, creating a signal-to-noise trade-off.
5 CAN MEMORY AGENTS SELF-EVOLVE THROUGH INTERACTION?
The paper tests whether an interactive memory agent can improve its update policy from environmental feedback. Self-evolution improves memory performance mainly by reducing write failures, while retrieval-related failures remain stable.
- Self-evolution setup: Self-evolution treats the memory agent’s natural-language policy prompt as a mutable component updated across interaction-feedback cycles.After each cycle, a generator produces P_k+1 from the current prompt P_k and feedback F_k.
- Self-evolution setup: The experiment compares a static prompt with question-only feedback and complete feedback containing questions, answers, and ground-truth answers.The evolution target is the prompt controlling memory-buffer updates in the in-context memory agent.
- Results: Agents receiving feedback outperform the static baseline in memory scores, with the improvement primarily attributable to a lower write failure rate under complete feedback.Read failures remain stable because evolution targets memory updating rather than retrieval.
- Results: Qualitative analysis shows policies evolving from generic instructions into specific, actionable rules that introduce schemas for recurring topics.One example refines a vague skill-level directive into a rule about teaching approaches and choir logistics.
6 CONCLUSION
AMEMGYM provides scalable, interactive on-policy evaluation for conversational memory, using structured state evolution to support benchmarking, diagnosis, and memory-strategy optimization.
- AMEMGYM enables on-policy evaluation of conversational memory through an interactive environment grounded in structured state evolution.
- The framework supports reliable benchmarking, performance-gap diagnosis, and optimization of memory strategies.
- AMEMGYM’s methodology and experimental prompts are described to support reproducibility.
- The work uses synthetic user profiles and LLM-based components, avoiding real human subjects and personal information.
C.1 BENCHMARK STATISTICS
The benchmark contains diverse synthetic users and base and extended configurations that vary in temporal depth, dialogue length, and evaluation complexity.
- The benchmark contains 20 unique user profiles and offers base and extended configurations with increased temporal complexity.
- User profiles span ages 18–85, 9 education categories, and 16 occupations.
- The base configuration has 11 periods and 47.15 total turns per user, while the extended configuration has 21 periods and 81.60 total turns.
- Average dialogue length ranges from 60K to 140K tokens for the base version across models.
- Each user profile is evaluated with 10 questions requiring retrieval and reasoning over 2–3 memory states.
C.2 COST ANALYSIS
AMEMGYM combines offline structured-data construction with on-policy user simulation, using schema-based prompts and reflection to generate evaluation-aligned interactions and ground truth.
- Cost Analysis: Offline structured-data generation costs $0.40 per instance using gpt-4.1, requiring approximately 0.14M input and 15.2K output tokens.
- Cost Analysis: On-policy user simulation costs $0.17 per instance with gpt-4.1 or $0.02 with deepseek-v3.
- Structured Data Generation: The framework samples profiles, questions, state schemas, and state trajectories to construct structured conversational blueprints.
- Ground-Truth Generation: A reflection step validates that each personalized answer corresponds unambiguously to its intended state variant.
D META EVALUATION DETAILS
Meta-evaluation assesses state exposure, conversational state integrity, and ground-truth judgment reliability using human annotation and agreement measures.
- State Exposure: Annotators assessed state exposure by comparing each query with the current and previous values of the targeted state.
- State Exposure: State exposure achieved a 99.1% average quality score and 96.8% Gwet’s AC1 across 200 sampled queries.
- Conversational State Integrity: Integrity annotation classified each state as having no conflict, minor inconsistency, or major conflict.
- Conversational State Integrity: Conversational state integrity reached 99.2% average consistency and 98.2% Gwet’s AC1 across 748 annotated items.
- Ground-Truth Judgment Reliability: Human agreement was 0.92, while agreement between human annotators and LLM-generated golden choices reached 0.96 and 0.94.
E DETAILS FOR THE SELF-EVOLUTION EXPERIMENT
The self-evolution experiment evaluates memory agents through task performance and factual recall, then uses environmental feedback to update their memory policies. Feedback-based strategies outperform the static baseline, with Complete Feedback showing the strongest and steadiest gains.
- Evaluation Metrics: Self-evolution is evaluated using task-specific memory performance and the factual accuracy of the agent’s internal memory.The evaluation reports Normalized Memory Score and Memory Factual Recall as complementary perspectives.
- Self-Evolution Process: The self-evolution loop interacts with AMEMGYM, collects trajectories and outcomes, generates feedback, and updates the policy prompt across cycles.The process starts from an initial policy and repeats for a predefined number of evolution cycles.
- Evaluation Metrics: Memory Factual Recall averages binary judgments indicating whether newly introduced user facts are supported by the agent’s final memory.A factual consistency checker evaluates each new fact against the memory representation.
- Results: Agents receiving environmental feedback consistently outperform the static baseline on memory performance and factual recall.The comparison is reported in Figure 11 under different environmental feedback conditions.
- Results: Complete Feedback produces the most substantial and steady improvement in both Normalized Memory Score and Memory Factual Recall.The result is reported for the evolution assistants evaluated in Figure 11.
E.1 CASE STUDY: ANALYSIS OF EVOLVED POLICIES
The policy evolves from general instructions to specific, actionable memory-management rules across evolution cycles. The resulting prompts add recurring-information schemas and incorporate state names from environmental feedback.
- Policy Evolution: Policy prompts evolve from general instructions in early cycles to highly specific, actionable rules by the final cycle.Table 5 traces prompt evolution across periods P1, P2, P5, and P10.
- Policy Evolution: A vague instruction to track skill levels becomes a rule for capturing teaching approaches suited to different experience levels.The example illustrates increasing specificity in the evolved policy.
- Learned Schemas: The evolved policy develops schemas for recurring information such as choir logistics and themed watch parties.These schemas emerge as the policy becomes more specific.
- Learned Schemas: The policy directly incorporates state names supplied by environmental feedback.This is part of the observed evolution toward actionable memory-management rules.
F.1 EVALUATION ON Extra CONFIGURATION
The extra configuration demonstrates that AMEMGYM can increase evaluation difficulty through configurable environment parameters. Under this setting, the evaluated models receive substantially lower memory scores, while changing the user LLM has minimal impact on results.
- Extra Configuration: AMEMGYM can increase evaluation difficulty by adjusting configurable parameters in the environment.The framework also supports varying state variants and state-change frequency.
- Extra Configuration: Under the extra configuration, gemini-2.5-flash-lite scores 0.137 and gpt-4.1-mini scores 0.104, versus 0.269 and 0.203 under the base setting.Only these two models are compared under the extra configuration because of resource and context-window constraints.
- User LLM Robustness: Switching the user LLM from gpt-4.1 to deepseek-v3 has minimal impact on the evaluation results.The comparison is presented as evidence of grounded interactions in AMEMGYM.
- Write-Strategy Diagnostics: Figure 14 provides diagnostic results for different write strategies, while Figure 7a presents the data in a clearer table format.The transformation addresses the high information density of the original figure.
- Memory Frameworks: Table 6 compares open-source memory frameworks using matched embedding models and vector databases for A-Mem and Nemori against the Mem0 implementation.AWE is included as the baseline implementation described in the main text.
F.5 EVALUATION WITH OPEN-SOURCE MODELS
The open-source-model evaluation reports benchmark performance and its stability across repeated runs. Five independent evaluations are used to estimate means and standard deviations, with highly stable performance estimates reported.
- Open-Source Models: Table 7 presents the performance of leading open-source models on the benchmark, using gemini-2.5-flash as the main-text baseline.The comparison focuses on open-source model performance.
- Performance Stability: The benchmark is repeated five times across a representative subset of models to assess evaluation reliability.Table 8 reports the mean and standard deviation for each model.
- Performance Stability: The repeated evaluations demonstrate highly stable performance estimates.This conclusion is based on the reported means and standard deviations across independent runs.