PaperlayerView exact revision on arXiv

Source-linked AI summary

MINTEval: Evaluating Memory under Multi-Target Interference in Long-Horizon Agent Systems

Hyunji Lee, Justin Chih-Yao Chen, Joykirat Singh, Zaid Khan, Elias Stengel-Eskin, Mohit Bansal

arXiv:2605.18565v2cs.CLcs.AI

TL;DR

Existing benchmarks do not capture how evolving memories interfere over long horizons. MINTEval evaluates agents across interference-heavy domains and question types, finding consistently low performance, especially for multi-target aggregation.

  • Problem

    Existing benchmarks largely assess static, independent recall rather than interactions among evolving memories that require sustained recall and aggregation.

  • Method

    MINTEval benchmarks seven systems across four domains and question types using long, interconnected contexts with continuously updated information and interference.

  • Results

    27.9% average accuracy across systems, with performance dropping on long-range lookback and multi-target aggregation questions.

  • Takeaways & Limitations

    Existing memory systems remain far from robust on interference-heavy contexts, particularly when questions require tracking revisions or aggregating multiple targets.

  • Takeaways & Limitations

    Evaluation uses LLM-based assessment rather than lexical matching because identical words may recur without relevance to the question.

Abstract

from arXiv · show

Real-world agents operate over long and evolving horizons, where information is repeatedly updated and may interfere across memories, requiring accurate recall and aggregated reasoning over multiple pieces of information. However, existing benchmarks focus on static, independent recall and fail to capture these dynamic interactions between evolving memories. In this paper, we study how current memory-augmented agents perform in realistic, interference-heavy, long-horizon settings across diverse domains and question types. We introduce MINTEval (Long-Horizon Memory under INTerference Evaluation), a benchmark featuring (1) long, highly interconnected contexts with frequently updated information that induces substantial interference, (2) diverse domains (state tracking, multi-turn dialogue, Wikipedia revisions, and GitHub commits), enabling evaluation of domain generalization, and (3) diverse question types that assess robustness to interference, including (i) single-target recall tasks requiring retrieval of a specific target from long contexts, and (ii) multi-target aggregation tasks requiring reasoning over multiple relevant pieces of information. Overall, MINTEval has 15.6k question-answering pairs over long-horizon contexts averaging 138.8k tokens and extending up to 1.8M tokens per instance. We evaluate 7 representative systems, including vanilla long-context LLMs, RAG, and memory-augmented agent frameworks. Across all systems, we observe consistently low performance (avg. 27.9% accuracy), especially on questions requiring aggregated reasoning over multiple pieces of evidence. Our analysis shows that performance is primarily limited by retrieval and memory construction. Furthermore, current memory systems struggle to recall and reason over earlier facts that are revised or interfered with by subsequent context, with accuracy degrading as the number of intervening updates increases.

1 Introduction

MINTEval evaluates memory-augmented agents in dynamic, interference-heavy long-horizon settings spanning multiple domains and question types. Across seven representative systems, performance remains low, revealing challenges in retrieval, memory construction, and handling revised or conflicting information.

  • Motivation: Continuously accumulated information can reinforce, modify, contradict, or fail to overwrite earlier facts, creating interference when agents answer queries about older versions or preferences.These settings include specifications across software revisions and personal information across long-term conversations.
  • Benchmark design: MINTEval spans four domains—state tracking, multi-turn dialogue, Wiki revisions, and Git commits—with continuously evolving information and queries requiring long-range lookback and aggregated reasoning.The benchmark is designed to test memory behavior under interference-heavy contexts rather than static, independent recall.
  • Evaluation results: 27.9% average accuracy across all systems shows that MINTEval remains highly challenging for current memory-augmented agents.The evaluation covers seven representative systems, including Full Context, RAG, HippoRAG, MemAgent, AtomMem, Mem-α, and SimpleMem.
  • Evaluation results: 33.4% average accuracy is achieved by MemAgent, the best-performing system, indicating substantial room for improvement.This result is reported across the MINTEval evaluation using Qwen3.6-35B-A3B and Gemini-3.1-Flash-Lite.
  • System analysis: 76.8% average use of insertion-based operations reveals a strong bias toward insertion over deletion or update in existing memory systems.Performance is also sensitive to design choices such as the number of iterative memory-processing steps.

2 MINTEVAL: Long-Horizon Memory under INTerference Evaluation

MINTEval evaluates memory under densely interacting, repeatedly revised contexts across four domains and two complementary question families. Its design tests both single-state retrieval and multi-target reasoning over long-horizon updates, with contexts averaging 138.8k tokens.

  • Interference-heavy Contexts: MINTEval targets interference-heavy contexts where information is repeatedly modified or contradicted, requiring temporal conflict resolution, historical-state preservation, and consistency over time.These dynamics induce both proactive and retroactive interference.
  • Domains: Four domains—bAbI state tracking, HorizonBench multi-turn dialogue, Wiki Revisions, and Git Commits—vary in information structure, update dynamics, and reasoning requirements.Together, they support evaluation of memory behavior under varied interference patterns and domain generalization.
  • Question Types: MINTEval includes single-target recall and multi-target aggregation, testing distinct aspects of memory behavior under densely interacting updates.Single-target tasks retrieve one target, whereas multi-target tasks combine evidence distributed across updates.
  • Question Types: Single-target recall distinguishes Simple questions for the most recent state from History questions requiring recovery of an earlier state despite subsequent or conflicting updates.Simple questions probe proactive interference, while History questions probe retroactive interference and preservation of prior states.
  • Question Types: Multi-target aggregation requires identifying multiple targets across updates and supports Ordering and Counting variants for temporal ordering and cross-update occurrence aggregation.The passage also introduces a third aggregation variant, but its description is truncated in the supplied text.
  • Dataset Statistics: 138.8k tokens is the average context length, with 86 updates in depth, 149 sessions per domain, and averages of 2k single-target and 1.8k multi-target questions across domains.These statistics summarize MINTEval’s scale and composition.

3 Experiments

Experiments compare full-context, retrieval-augmented, and memory-augmented agents using Exact Match evaluation. Results show that MINTEval remains difficult because evidence retrieval and memory construction fail, performance varies substantially across domains, and revision provenance is often lost.

  • Experimental Setup: The evaluation compares Full Context, RAG, and Memory-Augmented Agents using Exact Match after standard text normalization.For HorizonBench, candidate answers are provided because answers may have multiple valid surface forms.
  • Overall Results: 27.7% average accuracy across six systems shows that existing methods struggle on MINTEval.The best overall result reaches only 33.4% averaged across all domains.
  • Cross-Domain Generalization: 85.7% on bAbI Simple versus 7.5% on HorizonBench Simple for MemAgent demonstrates limited cross-domain generalization.HippoRAG reaches 70.0% on bAbI Simple and 12.1% on HorizonBench Simple.
  • Strong Memory Systems: 30.3 EM on average is achieved by state-of-the-art SimpleMem despite stronger embedding and answering models.The reported degradation stems from SimpleMem’s aggregation limitations, though the supplied passage truncates the explanation.
  • Error Analysis: 58.3% of cases contain the required evidence, making retrieval and memory construction the main bottleneck.Answering errors add a further 25.2% drop when evidence is present.
  • Revision Handling: Revision provenance is often lost during compression, making queries for specific revisions such as “Revision 53” especially challenging.Without metadata linking facts to originating revisions, retrieval relies primarily on keywords and embeddings.

4 Analysis

The analysis attributes most performance loss to retrieval and memory construction rather than answer generation, and shows that distractors and repeated memory updates further impair performance. Memory systems also favor appending information over modifying or deleting it.

  • Failure Sources: 41.7% average performance drop comes from retrieval and memory-construction failures, leaving supporting evidence in only 58.3% of cases.The analysis treats 100% as the upper bound because every question’s evidence exists in the full-history retrieval pool.
  • Failure Sources: 55.7% performance improvement is observed for answering agents without retrieval or memory systems, but the gap shrinks to 1.7% when such systems are introduced.This indicates that context construction, rather than answering-agent capability, drives most performance differences once memory systems are used.
  • Distractors: RAG is especially vulnerable to out-of-domain distractors because it retrieves distracting sentences more frequently, whereas Mem-α and Full Context show relatively small ID–OOD differences.ID distractors more strongly affect Counting and History than Simple questions, indicating greater sensitivity for aggregation and tracking tasks.
  • Memory Updates: Increasing chunk size generally improves MemAgent performance by reducing the number of memory modifications and update iterations.Simple questions are least sensitive to chunk size because they require recalling recent information.
  • Memory Updates: 87.6% of AtomMem operations and 65.9% of Mem-α operations are insertions, revealing a strong bias toward appending rather than modifying or deleting memories.Both systems exhibit this insertion bias across datasets.

5 Related Work

Prior memory-augmented agents use graph-based retrieval, hierarchical memory, or selectively pruned summaries, while existing benchmarks often emphasize recent information in less interconnected contexts. MINTEVAL broadens evaluation to interference-heavy contexts, diverse lookback distances, and multi-domain aggregation reasoning.

  • Memory-Augmented Agents: Memory-augmented agents include graph-based associative multihop retrieval, hierarchical memory tiers, and selectively pruned running summaries.HippoRAG organizes extracted knowledge into graphs; MemGPT pages information through OS-inspired memory tiers; SimpleMem maintains running summaries.
  • Memory Evaluation in Large Language Models: Existing memory benchmarks evaluate retrieval and temporal reasoning but typically use less interconnected contexts and questions about the most recent information.The passage identifies conversational and QA-based benchmarks as examples of this evaluation pattern.
  • MINTEVAL: MINTEVAL covers interference-heavy contexts, diverse lookback distances, and aggregation-based reasoning across multiple domains.This positions MINTEVAL as a broader and more challenging evaluation setting for memory systems.

6 Conclusion

MINTEVAL is introduced as a unified benchmark for evaluating memory-augmented agents in realistic, long-horizon environments with interference-heavy contexts, long-range dependencies, and multi-target aggregation reasoning. It spans four domains and five question types covering single-target recall and multi-target aggregation.

  • MINTEVAL evaluates memory-augmented agents in realistic long-horizon environments characterized by interference-heavy contexts, long-range dependencies, and multi-target aggregation reasoning.
  • The benchmark spans four domains: state tracking, multi-turn dialogue, Wikipedia revisions, and GitHub commits.
  • Five question types cover both single-target recall and multi-target aggregation, providing a unified framework for evaluating robustness.

A Additional Benchmark Details … A.3 Question Generation

MINTEval spans four evolving-information domains, from symbolic state tracking and multi-turn dialogue to chronological Wikipedia and GitHub histories. Questions are generated from parsed facts, metadata, templates, and domain-specific APIs and selection criteria.

  • A.1 Four Domains in MINTEVAL: bAbI represents state tracking as discrete symbolic facts whose sequential updates can explicitly overwrite previous states.Its fact-based format uses simple compositional sentences, with each input unit describing an entity state.
  • A.1 Four Domains in MINTEVAL: HorizonBench supplies six-month multi-turn dialogue histories in which information is distributed across utterances and updated incrementally, noisily, and indirectly.Each input unit is a dialogue session containing multiple conversational turns and simulated-user interactions.
  • A.1 Four Domains in MINTEVAL: Wikipedia revision instances pair one article with its complete chronological revision sequence, including document snapshots and metadata such as timestamps, editors, and edit comments.The domain introduces substantial temporal heterogeneity compared with single-snapshot or synthetic memory benchmarks.
  • A.1 Four Domains in MINTEVAL: GitHub instances pair one repository with its complete chronological commit sequence, including codebase snapshots and metadata such as authors, timestamps, messages, and modified files.This setting requires preserving provenance and tracking code and file evolution.
  • A.2 Question Examples for Each Domain: Table 5 provides example questions for each domain and question category.The examples organize the benchmark’s question formats across its four domains.
  • A.3 Question Generation: For bAbI and HorizonBench, questions use predefined templates populated from parsed facts or provided metadata.bAbI parses facts into subject-object-verb triplets, while HorizonBench uses metadata fields such as user preferences; some questions include candidate options when answers are absent from context.
  • A.3 Question Generation: Wiki-Revisions and Git Commits are collected through the official MediaWiki and GitHub APIs, respectively.Wikipedia selection requires Featured or Good Article status and size and prose-density bounds; GitHub selection requires non-forked, non-archived Python repositories with at least 100 stars.
  • A.3 Question Generation: The work’s templates and prompts are provided in the official MINTEval GitHub repository because of their length.The cited repository URL is https://github.com/amy-hyunji/MINTEval.

A.4 Human Validation on the Generated Data · A.5 Dataset Statistics

Human validation finds the generated Wiki Revisions and Git Commits data highly reliable, while dataset statistics show broad variation in context length and balanced coverage of question types. Together, these properties support systematic evaluation of retrieval, temporal reasoning, and aggregation.

  • A.4 Human Validation on the Generated Data: 95.6% of 405 stratified Wiki Revisions and Git Commits samples were valid, with both questions and answers correctly annotated.The samples covered five question types.
  • A.4 Human Validation on the Generated Data: 1.0% of samples had both invalid questions and answers, while 1.7% had only invalid questions and 1.7% had only invalid answers.These were the reported invalidity categories in the human validation.
  • A.4 Human Validation on the Generated Data: The human validation covered both the Wiki Revisions and Git Commits subsets using stratified sampling.The evaluation included 405 samples spanning five question types.
  • A.4 Human Validation on the Generated Data: Specialized answering agents such as MemAgent-14B remained competitive on single-target recall but declined on multi-target aggregation questions.The decline was especially pronounced for Counting, which requires stronger aggregation and reasoning capabilities.
  • A.5 Dataset Statistics: Contexts varied substantially across domains, ranging from short synthetic trajectories to histories exceeding one million tokens.Table 6 provides more detailed statistics on context depth and total token length.
  • A.5 Dataset Statistics: The benchmark balanced question types including simple recall, historical lookup, ordering, counting, and multihop reasoning.This distribution enables systematic evaluation of memory retrieval, temporal reasoning, and aggregation capabilities.

B More Experimental Details … C.3 Effect of Adding Temporal Cues to History Questions

The additional experiments specify controlled evaluation settings and show that answering-agent choice, full-context access, and explicit temporal cues materially affect performance under interference. Specialized answering agents remain competitive for single-target recall but struggle more with multi-target aggregation, while frontier-model full-context access achieves the strongest reported recall performance.

  • B More Experimental Details: Decoding temperature is set to 0, answers must be wrapped in \boxed{}, and experiments use either 4×80GB A100 or 4×48GB A6000 servers.These settings apply to all experiments.
  • C.1 Impact of Answering Agent Choice: MemAgent-14B remains competitive on single-target recall but performs worse on multi-target aggregation, especially Counting questions requiring stronger aggregation and reasoning.The comparison pairs MemAgent with MemAgent-14B, Qwen3.6-35B-A3B, and Gemini-3.1-Flash-Lite as answering agents.
  • C Additional Analysis: Answering-agent choice is a central focus of the additional analysis, with experiments comparing Qwen3.6-35B-A3B and Gemini-3.1-Flash-Lite across methods.The comparison is reported in Figure 7.
  • C.2 Using Frontier Models with the Full Context Remains Competitive: Gemini-3.1-Flash-Lite with Full Context achieves the highest performance on both task types, exceeding 80% accuracy on single-target recall.Its single-target recall result is described as far surpassing other retrieval-based and memory-augmented systems.
  • C.2 Using Frontier Models with the Full Context Remains Competitive: The performance gap between answering agents is largest with Full Context and is larger for multi-target aggregation than for single-target recall.This pattern is summarized in Figure 7.
  • C.3 Effect of Adding Temporal Cues to History Questions: Temporal markers help recover the performance gap caused by interference in bAbI History questions as lookback distance increases.The comparison includes History versus +Date/Time conditions for both RAG and Full Context.
  • C.3 Effect of Adding Temporal Cues to History Questions: Adding explicit date and time cues to bAbI facts and questions makes similar facts more discrete and is designed to test whether interference causes degradation with increasing lookback distance.Figure 8 compares RAG and Full Context with and without temporal cues.

C.4 Biased Toward Insertion in Memory Systems

Memory systems are biased toward inserting information rather than deleting outdated or conflicting entries. Mem-α updates existing memories more often than AtomMem, but both systems’ underuse of deletion allows interference to accumulate over long horizons.

  • Modification and deletion bias: 34.1% versus 3.7%: Mem-α performs substantially more modification operations than AtomMem.This indicates stronger updating of existing memory instead of duplicating information and may help explain Mem-α’s stronger overall performance.
  • Modification and deletion bias: Mem-α consistently underutilizes deletion across all datasets.The passage identifies deletion as removing outdated information, while modification updates existing entries.
  • Modification and deletion bias: Underused deletion allows outdated or conflicting information to accumulate, increasing memory conflicts in long-horizon settings with heavy interference.This accumulation is presented as a partial explanation for memory-system failures under long-horizon interference.

C.5 Effect of Retrieval Choices on RAG Performance · C.6 Expanded Discussion on the State-of-the-art Memory System Failure

Retrieval choices substantially affect RAG performance, with Gemini-Embedding-001 generally outperforming Qwen3-Embedding-4B and larger retrieval pools helping multi-target aggregation. SimpleMem nevertheless struggles because its memory pipeline lacks explicit revision metadata, while alternative retrieval models show near-zero performance.

  • C.5 Effect of Retrieval Choices on RAG Performance: Gemini-Embedding-001 consistently outperforms Qwen3-Embedding-4B across retrieval values, with the gap widening slightly as K increases.The stronger embedding model is more effective at ranking relevant documents higher when the retrieval pool is larger.
  • C.5 Effect of Retrieval Choices on RAG Performance: Average performance rises sharply from K = 1 to K = 5, then largely plateaus for larger retrieval pools.Qwen3-Embedding-4B performs best at k = 5, whereas Gemini-Embedding-001 peaks around K = 50.
  • C.5 Effect of Retrieval Choices on RAG Performance: Figure 12 evaluates RAG across question types while varying retrieval-document counts and the Qwen3-Embedding-4B and Gemini-Embedding-001 models.The analysis is conducted on the Wiki Revision dataset for finer-grained comparison.
  • C.6 Expanded Discussion on the State-of-the-art Memory System Failure: Table 7 reports SimpleMem results with Gemini-3.1-Flash-Lite and Gemini-Embedding-001 in Exact Match (%), showing that the SOTA memory system still struggles on MINTEVAL.The table covers multiple datasets and question types.
  • C.5 Effect of Retrieval Choices on RAG Performance: Larger K values particularly improve complex multi-target aggregation questions on Wiki Revision, especially Counting and Ordering.These questions require aggregating or comparing multiple evidence pieces, so increasing K raises the probability that all necessary evidence is retrieved.
  • C.6 Expanded Discussion on the State-of-the-art Memory System Failure: SimpleMem uses Semantic Structured Compression, Online Semantic Synthesis, and a third memory-pipeline stage to transform interactions into compact, unified memory representations.Its design distills unstructured interactions into multi-view memory units and incrementally merges related contexts into abstractions.
  • C.6 Expanded Discussion on the State-of-the-art Memory System Failure: Without explicit metadata linking facts to revisions, SimpleMem relies solely on keywords and embeddings, making queries such as retrieving “Revision 53” more challenging.The missing revision provenance complicates retrieval of facts associated with specific updates.
  • C.6 Expanded Discussion on the State-of-the-art Memory System Failure: Qwen3.6-35B-A3B and the Qwen3-4B retrieval model show near-zero performance across all datasets, so their results are not reported.These additional retrieval experiments were omitted because performance was near zero throughout.

D Dataset License

The datasets are released under permissive licenses supporting open research and reproducibility: HorizonBench uses Apache-2.0, while bAbI uses CC BY 3.0.

  • Dataset License: HorizonBench is distributed under Apache-2.0, permitting academic and commercial use with minimal restrictions.The license supports open research and reproducibility.
  • Dataset License: bAbI is released under CC BY 3.0, permitting reuse and modification with appropriate attribution to the original authors.The license requires credit to the original authors.
Loading 2605.18565v2…