PaperlayerView exact revision on arXiv

Source-linked AI summary

AMA-Bench: Evaluating Long-Horizon Memory for Agentic Applications

Yujie Zhao, Boqin Yuan, Junbo Huang, Haocheng Yuan, Zhongming Yu, Haozhou Xu, Lanxiang Hu, Abhilash Shankarampeta, Zimeng Huang, Wentao Ni, Yuandong Tian, Jishen Zhao

arXiv:2602.22769v4cs.AIcs.LG

TL;DR

Existing memory benchmarks inadequately represent the machine-generated, causally grounded trajectories used by long-horizon agents, motivating a more realistic evaluation suite. The paper introduces AMA-Bench and AMA-Agent, whose causality graph and tool-augmented retrieval design achieves 57.22% accuracy and exceeds the strongest baseline by 11.16%.

  • Problem

    Existing memory benchmarks are largely dialogue-centric and lack evaluation of memory modules in long-horizon agentic tasks involving diverse machine-generated representations and causal dependencies.

  • Method

    AMA-Bench combines expert-annotated real-world agent trajectories with synthetic trajectories supporting controlled scaling to arbitrary horizons, while AMA-Agent uses a Causality Graph and Tool-Augmented Retrieval.

  • Results

    57.22% average accuracy was achieved by AMA-Agent, outperforming the strongest existing memory baseline by 11.16%.

  • Takeaways & Limitations

    AMA-Bench provides a diagnostic evaluation of agent memory, and the results support agent-centric designs that preserve objective information and causal dependencies.

Abstract

from arXiv · show

Large Language Models (LLMs) are increasingly used as autonomous agents in complex, long-horizon applications, where effective memory is critical for sustained performance. Yet existing memory benchmarks are largely dialogue-centric, while real agent memory consists of continuous agent-environment interaction trajectories composed of states, actions, observations, and tool outputs. To address this gap, we introduce **AMA-Bench** (**A**gent **M**emory with **A**ny length), a benchmark for evaluating long-horizon memory in realistic agentic settings. AMA-Bench combines real-world agent trajectories from representative applications with expert-curated QA, as well as synthetic trajectories that scale to arbitrary horizons with rule-based QA. Our study shows that existing memory systems underperform because they fail to capture causal and objective information and rely heavily on lossy similarity-based retrieval. We further propose **AMA-Agent**, a memory system based on causality-graph construction and tool-augmented retrieval. AMA-Agent achieves **57.22%** accuracy on AMA-Bench, outperforming the strongest baseline by **11.16%**. Resources are available at: [https://ama-bench.github.io/](https://ama-bench.github.io/).

1. Introduction

AMA-Bench addresses the lack of benchmarks evaluating memory in long-horizon agentic tasks by combining realistic and scalable trajectory-based evaluation. The paper shows that existing memory systems often underperform and proposes AMA-Agent with causality-aware construction and tool-augmented retrieval.

  • Motivation: Existing memory benchmarks are largely dialogue-centric and do not adequately evaluate memory modules in long-horizon agentic tasks.Agent trajectories contain diverse machine-generated representations and causal dependencies that natural-language-centric benchmarks typically omit.
  • AMA-Bench: AMA-Bench combines real-world and synthetic subsets to evaluate memory in agent applications.The real-world subset uses expert-annotated QA from six representative domains, while the synthetic subset supports automatically generated QA and arbitrary horizon lengths.
  • Findings: 72.26% accuracy was achieved by GPT 5.2, indicating that trajectory-based agent memory remains challenging even for frontier commercial models.AMA-Bench exposes challenges that persist beyond the capabilities of strong base models.
  • Findings: Existing agent memory designs often underperform the long-context baseline because lossy compression and similarity-based retrieval errors accumulate over long-horizon tasks.The evaluation identifies a need for memory designs centered on agent-specific trajectory properties.
  • AMA-Agent: 11.16% average improvement over the strongest existing memory baselines was achieved by AMA-Agent.Its Causality Graph preserves objective information and causal dependencies, while Tool-Augmented Retrieval combines graph-node and keyword search.

2. Related Work

Prior memory benchmarks primarily evaluate dialogue-centric retention or long-context document reasoning rather than interactive agent trajectories. Existing approaches span long-context models, retrieval-augmented generation, and agent-managed memory systems.

  • Benchmark categories: Dialogue-centric benchmarks evaluate memory retention in multi-turn human-agent interactions and continual-learning settings.Examples include LoCoMo, LongMemEval, MemoryAgentBench, MemoryBench, and RealTalk.
  • Benchmark categories: Long-context benchmarks evaluate static document-level reasoning, especially multi-hop comprehension over long inputs.QuALITY, RULER, and LongBench v2 represent this category.
  • Memory approaches: Long-context models process memory directly as context but remain bounded by physical context limits.The related-work discussion cites GPT 5.2 and Qwen2.5 1M as examples of this approach.
  • Memory approaches: Retrieval-augmented generation stores information externally and retrieves items by similarity to augment the model context.Traditional methods partition data into discrete chunks, while structured RAG approaches seek richer organization.
  • Memory approaches: Memory agent systems let LLM agents autonomously decide how to construct, retrieve, and manage long-term memories.MemoryBank, MemGPT, and MemoRAG exemplify this shift toward agent-centric memory management.

3. AMA-Bench

AMA-Bench evaluates agent memory through real-world and synthetic agent-environment trajectories, covering memory construction, retrieval, evolution, and condensation across controllable horizons.

  • Problem formulation: AMA-Bench formalizes agent memory as building an external memory state from interaction history and retrieving query-relevant context for subsequent policy decisions.The framework accommodates summaries, knowledge graphs, and vector embeddings.
  • Memory capabilities: The benchmark evaluates four capabilities: Recall, Causal Inference, State Updating, and State Abstraction, aligned with retrieval, evolution, and condensation mechanisms.These capabilities operationalize the requirements of long-horizon decision-making.
  • Real-world subset: The real-world subset contains 2,496 trajectory-grounded QA pairs from six domains, including web navigation, software engineering, Text2SQL, embodied AI, gaming, and open-world tools.Trajectories are manually annotated with questions covering the benchmark’s capability categories.
  • Synthetic subset: The synthetic subset uses executable environments with controllable states, transitions, perception, action stochasticity, and observation verbosity to generate verifiable trajectories at arbitrary horizons.TextWorld and BabyAI provide environments with long-range dependencies and partial observability.
  • Synthetic subset: Programmatic access to the environment MDP enables optimal-policy trajectories and golden QA pairs anchored to backend states and transition kernels.A needle protocol additionally identifies the minimal trajectory turns containing all evidence needed to answer a query.

4. Empirical Motivation

AMA-Bench exposes weaknesses in existing memory systems: they often trail long-context baselines, while memory architecture matters more than backbone scale and lossy retrieval fails on causal state information.

  • Long-context comparison: Across six agent task families, long-context baselines are consistently strong, while existing memory systems vary widely and frequently underperform them.This pattern holds even for systems using structured memory construction or retrieval augmentation.
  • Architecture versus scale: Scaling the backbone from 8B to 32B yields an average improvement of 0.038, whereas memory architectures produce score ranges reaching 0.45.The comparison indicates substantially greater performance variance across memory designs than across model scales.
  • Bottleneck analysis: Needle-protocol ablations separate raw evidence access, constructed memory, and end-to-end retrieval to identify losses from memory construction and retrieval.Full Observation provides the upper-bound setting, while the other settings isolate construction and retrieval effects.
  • Construction loss: 41.3%: MemoryBank’s performance drops after construction, suggesting that natural-language-oriented compression loses dense state and causal information.The reported degradation concerns the transition from raw needle turns to constructed memory.
  • Retrieval loss: 43.2%: HippoRAG2’s performance drops end-to-end despite remaining strong with constructed memory and the needle turn, indicating unreliable similarity-based retrieval.The result motivates retrieval mechanisms beyond similarity search.

5. The AMA-Agent

AMA-Agent combines causality-graph memory construction with selective tool-augmented retrieval to preserve objective state dependencies and recover evidence beyond embedding similarity.

  • Core mechanisms: AMA-Agent builds a Causality Graph and uses tool-augmented retrieval to address information loss and retrieval failures identified in existing memory systems.The two mechanisms respectively target memory construction and evidence access.
  • Causality Graph: The graph extracts states and object associations from adjacent observation-action turns, representing causal dependencies with directed edges and associations with undirected edges.This process structures local trajectory information into a graph representation.
  • Causality Graph: Graph extraction is an LLM-based semantic abstraction that retains task-relevant entities, state changes, and dependencies while discarding low-level artifacts.The method is not a rule-based parser over raw HTML DOM trees or program ASTs.
  • Tool-augmented retrieval: Retrieval first uses top-K embedding similarity and self-evaluation, then invokes graph traversal or keyword search when the evidence is insufficient.Graph traversal aggregates multi-hop causal context, while keyword search supports precise matching and trajectory-wide aggregation.
  • Tool-augmented retrieval: In 200 sampled QA pairs, graph search was invoked in 10.0% of cases and keyword search in 23.5%, showing that tool use is selective rather than default.Embedding retrieval alone was judged sufficient in the majority of cases.

6. Evaluation

AMA-Bench evaluates memory systems on real-world and synthetic QA under standardized offline protocols, finding strong performance for AMA-Agent, synthetic-real alignment, and improved long-horizon scalability.

  • Dataset and baselines: The benchmark contains 2,496 real-world QA pairs and 1,200 synthetic QA pairs distributed across trajectory lengths from 8K to 128K tokens.The synthetic subset uses 240 samples at each of five trajectory-length intervals.
  • Experimental setup: All memory systems are compared with long-context, RAG, and memory-agent baselines using Qwen3-32B and Qwen3-8B backbones.The evaluation follows an offline protocol in which each trajectory is ingested once and QA pairs are answered independently against fixed memory.
  • Real-world results: AMA-Agent reaches 0.5722 average accuracy on the Qwen3-32B real-world evaluation and leads across Recall, Causal Inference, State Updating, and State Abstraction.Its reported dimension scores are 0.6238, 0.6145, 0.5305, and 0.4719, respectively.
  • Real-world results: GPT 5.2 achieves 72.26% accuracy, while existing memory systems often underperform long-context baselines on long-horizon agentic tasks.The benchmark remains challenging even for frontier commercial models.
  • Synthetic validation: Synthetic and real-world performance rankings show strong correlation, supporting the synthetic subset as a high-fidelity proxy for real-world performance.This proxy reduces reliance on costly real-world data acquisition and manual annotation.
  • Scalability: As trajectories grow from 8K to 128K tokens, long-context performance degrades beyond 32K, whereas AMA-Agent maintains robust accuracy at 128K.The result compares scalability across trajectory lengths.
  • Ablations: Removing the Causality Graph lowers average performance from 0.57 to 0.43, while removing tool-augmented retrieval reduces it to 0.44.Both ablations indicate that the two components contribute to AMA-Agent’s performance.

7. Conclusion

AMA-Bench evaluates agent memory for machine-generated, causally grounded trajectories, while AMA-Agent addresses identified weaknesses in lossy compression and similarity-based retrieval.

  • AMA-Bench bridges natural-language-centric evaluations and the machine-generated, causally grounded nature of real-world agent trajectories.
  • AMA-Agent uses a Causality Graph and Hybrid Tool Augmented Retrieval to improve memory for agentic applications.
  • The study identifies memory architecture as the primary determinant of performance and highlights limitations of lossy compression and similarity-based retrieval.
  • The study focuses on in-episode memory, leaving cross-task lifelong learning for future work.

Impact Statement

The benchmark combines diverse real-world and synthetic agent trajectories, with QA designed to test long-horizon, multi-hop memory across distributed evidence and varied interaction conditions.

  • Real-world Subset: AMA-Bench’s real-world subset spans six agent domains and includes expert-curated, sanity-checked QA pairs from representative applications.The domains are Web, open-world QA, Text2SQL, Software Engineering, Gaming, and Embodied AI.
  • Long-horizon validation: A 720-QA analysis found a mean hop count of 3.38, a 72.2% multi-hop ratio, and 62.9% of evidence in the middle 60% of trajectories.
  • Synthetic Subset: The synthetic subset uses BabyAI and TextWorld case studies to evaluate capabilities by modulating programmatic synthesis parameters.
  • Synthetic Subset: Synthetic QA has a 62.6% overall multi-hop ratio and 100% in BabyAI, with evidence broadly distributed rather than concentrated at trajectory boundaries.

B. Baseline Implementation Details

The baseline implementations use long-context packing, lossy extraction or summarization, recurrent consolidation, hierarchical memory, and retrieval over trajectory representations.

  • Long-Context Baseline: Long-context baselines pack trajectories directly into the model input without retrieval or compression until the permitted context limit.
  • Long-Context Baseline: When trajectories exceed the input budget, truncation preserves the first and last 50% of the budget and discards the middle.
  • Extraction-based systems: GraphRAG, HippoRAG, SimpleMem, and Mem0 distill trajectories into graphs, triples, atomic entries, or facts before embedding and retrieval.
  • Consolidation-based systems: MemoryBank, MemAgent, Mem1, and A-Mem use chunked summarization or recurrent memory updates to consolidate long trajectories.
  • Hierarchical and retrieval systems: Mem-alpha and MemGPT use hierarchical or agent-managed memory, while MemoRAG combines a global memory representation with retrieval-augmented generation.

C.2. Human-Judge Agreement

The evaluation validates the Qwen3-32B LLM judge against human annotations and alternative judges, while the QA pipeline anchors questions to traceable trajectory turns.

  • Human agreement: 92.67% judge accuracy against human annotations indicates reliable alignment between Qwen3-32B judgments and human decisions.
  • Cross-judge validation: Qwen3-32B agrees with alternative judges within 8.1 points and aligns most closely with GPT-5.4 at 92.8% and GPT-5.2 at 91.8%.
  • QA generation: The QA-generation pipeline samples needle turns across trajectory depths, generates questions by type and length, verifies quality, and selects a balanced final dataset.
  • QA generation: Binding each QA pair to source turn identifiers enables verification that retrieval identified the correct needle within the trajectory haystack.

E. Example of needle turn for ablation study

The examples contrast raw trajectory storage with extraction-based memory, showing that structured action-state-observation loops can preserve or expose information that fact distillation misses. A needle-turn example also illustrates how explicit state-change summaries simplify temporal reasoning.

  • Raw Observation Branch: Raw observation memory requires comparing consecutive observation strings to infer that the purple box disappeared.The LLM must detect the difference between the two views rather than read an explicit state-change summary.
  • Oracle and System Branches: Oracle and successful system memory explicitly summarize the initial view, movement, updated view, and disappearance inference.The oracle branch force-feeds this shard, whereas the system branch retrieves it from the database.
  • Extraction Comparison: Narrative input yields four extracted facts, whereas the abridged trajectory yields zero facts under the extraction process.The narrative facts are static subject-predicate-object information, while the trajectory contains dynamic action-state-observation loops.
  • Extraction Comparison: The trajectory case shows extraction-based memory can ignore critical spatial identifiers such as “toilet 1.”The authors characterize this approach as inherently lossy for agent-based tasks and report using raw trajectory segments to preserve environmental context.

G. Efficiency Analysis

AMA-Agent incurs moderate construction overhead but combines higher accuracy with favorable end-to-end latency. Its advantage becomes more pronounced on harder, longer-horizon tasks, while QA evaluation reduces the cost and ambiguity of assessing memory.

  • Latency–Accuracy Trade-off: AMA-Agent is more accurate than every faster baseline and faster than every more-accurate baseline on the real-world subset.Table 18 reports average per-trajectory latency split into memory construction and retrieve-plus-answer time.
  • Latency–Accuracy Trade-off: AMA-Agent’s moderate construction overhead is offset by a 2–5× accuracy gain relative to Mem0 and MemAgent.Against HippoRAG2, Mem1, and SimpleMem, it is reported as both more accurate and faster end-to-end.
  • Difficulty Scaling: AMA-Agent’s gap over the next-best method widens from −1 point on Easy to +3 and +4 points on Hard and Very-hard TextWorld tasks.LongContext is competitive on Easy tasks, whereas the causality graph becomes more valuable as causal chains lengthen.
  • Evaluation Efficiency: End-to-end rollouts are approximately 103× slower per task than QA evaluation because they require sequential interaction with the environment.QA also enables cleaner attribution because wrong answers are more directly traceable to memory quality than failed rollouts.
  • Behavioral Mechanisms: Causality-graph memory preserves prerequisite chains and recovers from errors, unlike RAG’s causally incomplete retrieval and summaries that discard preconditions.These are the qualitative failure-mode distinctions summarized for TextWorld behavior.

I. Cross-Model Robustness

The supplied evidence emphasizes evaluation across different model backbones and settings, with method rankings preserved and AMA-Agent remaining strongest. The section also includes prompt templates and examples describing structured memory construction, retrieval routing, and trajectory-based computation.

  • Cross-Model Evaluation: AMA-Agent remains strongest across Qwen3-32B, Qwen3-8B, and GPT-5-mini evaluation settings.The relative ordering of methods is preserved across the real-world and synthetic-subset settings.
  • Cross-Model Evaluation: On the harder synthetic regime, the gap between AMA-Agent and the next-best method widens despite uniformly lower GPT-5-mini scores.The synthetic subset uses longer horizons and machine-rendered observations that are harder for single-call agent backbones.
  • Out-of-Benchmark Generalization: AMA-Agent outperforms the strongest baseline HippoRAG by 4.1 absolute points on the LoCoMo dialogue-memory benchmark.This evaluates whether the design generalizes beyond AMA-Bench’s agent-trajectory setting.
  • Prompted Memory Operations: The construction prompt compresses trajectories into structured state memory, while retrieval judges chunk sufficiency and can generate trajectory-processing code when needed.The templates include key turns, objectives, events, environment states, retrieval decisions, and code generation over trajectory JSON.
  • Prompted Memory Operations: Memory construction instructs the model to identify key turns and copy commands, identifiers, numeric values, results, and errors verbatim.It also requests a one-line summary of the section’s overall progress.
  • Prompted Memory Operations: The retrieval procedure routes counting, aggregation, and many-turn pattern questions to code rather than answering from similarity-retrieved subsets.It uses SUFFICIENT, NEED_GRAPH, or NEED_CODE decisions depending on whether retrieved turns fully support the question.
  • Real-World Evaluation Examples: A real-world example reports that query text concatenates across steps 14–16, while other questions test state dependencies, count changes, and redundant loops.The examples span temporal inference, prerequisite state, state updates, and state abstraction.
Loading 2602.22769v4…