PaperlayerView exact revision on arXiv

Source-linked AI summary

Mem2ActBench: A Benchmark for Evaluating Long-Term Memory Utilization in Task-Oriented Autonomous Agents

Yiting Shen, Kun Li, Wei Zhou, Songlin Hu

arXiv:2601.19935v1cs.CLcs.AI

TL;DR

Existing memory benchmarks mostly test explicit retrieval, not the active use of dispersed memories to ground tool actions. MEM2ACTBENCH constructs interruption-heavy histories and reverse-generated underspecified tasks to measure this capability. Across seven frameworks, results show persistent parameter-grounding weaknesses, while oracle retrieval substantially outperforms passive retrieval.

  • Problem

    Existing benchmarks mainly test explicit retrieval of isolated facts, leaving underspecified tool execution that requires inferring dispersed historical constraints insufficiently evaluated.

  • Method

    MEM2ACTBENCH synthesizes heterogeneous dialogues into coherent memory evolution chains and reverse-generates tool-use queries whose omitted parameters must be grounded in long-term memory.

  • Results

    Current memory frameworks remain inadequate for parameter grounding; oracle retrieval reaches F1 ≈53.8 versus F1 ≈30.7 for the best passive retriever.

  • Takeaways & Limitations

    The benchmark indicates that active memory utilization, especially retrieval and grounding of dispersed information under underspecified requests, remains an open challenge.

  • Takeaways & Limitations

    The benchmark is limited to offline tool-call generation with a fixed backbone model family, excludes interactive execution, and may not fully capture real-world dialogue complexity.

Abstract

from arXiv · show

Large Language Model (LLM)-based agents are increasingly deployed for complex, tool-based tasks where long-term memory is critical to driving actions. Existing benchmarks, however, primarily test a angent's ability to passively retrieve isolated facts in response to explicit questions. They fail to evaluate the more crucial capability of actively applying memory to execute tasks. To address this gap, we introduce \textsc{Mem2ActBench}, a benchmark for evaluating whether agents can proactively leverage long-term memory to execute tool-based actions by selecting appropriate tools and grounding their parameters. The benchmark simulates persistent assistant usage, where users mention the same topic across long, interrupted interactions and expect previously established preferences and task states to be implicitly applied. We build the dataset with an automated pipeline that merges heterogeneous sources (ToolACE, BFCL, Oasst1), resolves conflicts via consistency modeling, and synthesizes 2,029 sessions with 12 user--assistant--tool turns on average. From these memory chains, a reverse-generation method produces 400 tool-use tasks, with human evaluation confirming 91.3\% are strongly memory-dependent. Experiments on seven memory frameworks show that current systems remain inadequate at actively utilizing memory for parameter grounding, highlighting the need for more effective approaches to evaluate and improve memory application in task execution.

1 Introduction

Existing memory benchmarks mainly test explicit retrieval of isolated facts, whereas MEM2ACTBENCH evaluates whether agents infer dispersed constraints and apply them in grounded tool calls. It introduces an automated benchmark with memory-dependent tasks and finds persistent weaknesses in current memory frameworks.

  • Motivation: Long-term assistant use requires applying preferences and partial task states from interrupted prior interactions to concrete actions.The target capability includes grounding missing arguments in tool invocations.
  • Motivation: Existing benchmarks primarily ask agents to retrieve isolated facts from explicit questions rather than infer constraints for underspecified tool requests.MEM2ACTBENCH targets the gap illustrated by Figure 1.
  • Benchmark design: MEM2ACTBENCH constructs histories where the correct tool invocation is uniquely grounded in dispersed memory and cannot be inferred from the final query alone.Its pipeline interleaves tool-use data with natural dialogue, resolves conflicting states, and applies leakage-controlled reverse query generation.
  • Benchmark design: 400 memory-dependent tool-use tasks are derived from 2,029 long-context dialogue sessions, with human verification confirming that 91.3% require long-term memory.The benchmark is designed to evaluate inference-driven memory utilization in tool-augmented agents.
  • Findings: Seven memory frameworks reveal persistent bottlenecks in memory retrieval and parameter grounding for tool-using tasks.The evaluation is presented as a comprehensive analysis of representative memory frameworks.

2 Related Work

Agent-memory research has progressed from long-context and external-memory approaches toward explicit memory management, but benchmarks still largely evaluate explicit query-to-retrieval-to-answer behavior. MEM2ACTBENCH instead targets inference-driven use of evolving histories for executable tool calls.

  • Memory systems: Memory approaches include extending context windows, retrieving interaction fragments from external memory banks, and using explicit memory managers.Long-context approaches can incur higher inference cost and lost-in-the-middle effects.
  • Benchmark landscape: Existing benchmarks span short-dialogue consistency and cross-session retention, with newer datasets increasing temporal span and retrieval difficulty.Examples include Persona-Chat, MSC, LoCoMo, LongMemEval, and MemoryAgentBench.
  • Benchmark gap: Prior benchmarks largely instantiate explicit Question → Retrieval → Answer tasks, leaving retrieval intent externally specified.This design under-tests settings where the agent must infer which memory constraints matter for execution.
  • Benchmark gap: MEM2ACTBENCH evaluates whether agents infer task-critical constraints from evolving interaction histories and ground them into executable tool calls.Its comparison with prior benchmarks emphasizes inference-driven memory utilization and dynamic task execution.
  • Comparison dimensions: Table 1 compares benchmarks by sessions, turns, tokens, QA pairs, reasoning type, memory evolution, and tool-use support.These dimensions distinguish interaction scale and whether evaluation requires inference, updating memory, or external invocation.

3 Methodology

MEM2ACTBENCH builds fragmented long-term histories, organizes them into coherent memory evolution chains, and reverse-generates underspecified tool-use queries. Expert verification supports the reliability of the resulting memory states and intrinsic memory dependence.

  • Data construction: The dataset pipeline interleaves task-oriented tool-use trajectories with conversational noise to create interruption-heavy histories requiring long-term memory.Task-oriented data come from ToolACE and BFCL, while conversational content is drawn from OASST1.
  • Fact extraction and grouping: Facts are extracted as attribute–fact–source triples, with entity-bound attributes used to reduce spurious cross-entity comparisons.The extracted attributes are clustered into canonical fact groups for later conflict detection and evolution analysis.
  • Memory evolution chain construction: Local evolution chains order facts temporally, preserve valid refinements and multi-valued trajectories, remove conflicts and off-context statements, and drop near-duplicates.The resulting local sequences provide ordering constraints for global integration.
  • Memory evolution chain construction: Global integration builds a dependency graph and applies modified topological sorting, with a deterministic heuristic for cycles caused by contradictions.The method uses Kahn’s algorithm as its sorting basis.
  • Task construction: Gold tool invocations select tools through hybrid retrieval and LLM decision-making, while parameters must be extracted or logically inferred from memory and independently validated.Fuzzy matching and an LLM verifier enforce the memory-anchoring constraint.
  • Task construction: Reverse generation omits key parameters, uses anaphoric references, and preserves intent to create underspecified queries dependent on historical memory.A discriminator without memory filters out queries whose tool invocations remain inferable from the query and API documentation alone.
  • Human verification: Expert verification evaluates fact extraction, conflict resolution, and memory dependence, with at least two annotators independently reviewing each item.Annotators assess whether facts are entailed and normalized, chains are coherent, and tool calls remain underdetermined without memory.

4 Experiment

The experiments evaluate representative memory systems across model scales using parameter-grounding, overlap, and exact tool-call metrics. Results show that scaling improves argument grounding, while tool accuracy changes little at larger scales.

  • Experimental Setup: The evaluation compares seven memory systems across Qwen2.5-7B, 32B, and 72B backbones under fixed decoding settings.The systems include Long-term Memory, Generative Agents, SCM, Langmem, MemTree, Mem0, and A-Mem.
  • Evaluation Metrics: F1 measures parameter precision and recall, BLEU-1 measures unigram overlap, and Tool Accuracy requires the correct tool and exact parameter match.The main results control for tool-selection errors to focus comparison on memory-based parameter grounding.
  • Main Results: Mean F1 rises from 20.4 at 7B to 28.9 at 72B, with diminishing returns beyond 32B.The reported increase from 32B to 72B is approximately 3.5 F1 points, indicating that scaling mainly improves post-retrieval composition.
  • Main Results: Tool Accuracy remains clustered around 87–97% and changes little from 32B to 72B, indicating that remaining errors are mainly semantic rather than structural.At 72B, A-Mem and Long-term Memory form the top F1 cluster at 35.9 and 35.3, respectively.

5 Discussions

The analyses identify retrieval and parameter grounding—not basic tool choice—as the central weaknesses in memory-grounded tool use. Performance degrades when evidence is buried mid-context, values are complex or defaulted, and tools have overlapping semantics.

  • 5.1 Retriever Analysis: Oracle retrieval reaches F1 ≈53.8 versus ≈30.7 for the best passive retriever, a gap of over 23 points.This comparison indicates that evidence retrieval remains a major bottleneck under underspecified requests.
  • 5.2 Impact of Memory Distance: F1 forms a mid-context valley: AgenticMemory falls from approximately 36% to 25% in the 25–50% position bin, while LTMemory stays above 30%.Most baselines perform better when supporting evidence appears early or near the end of context.
  • 5.3 Parameter Grounding and Complexity: Default values produce the largest grounding errors because models may invent plausible values when the dialogue never specifies them.Explicit and inferred values show only a small gap at 72B, whereas distractors can influence fabricated defaults.
  • 5.3 Parameter Grounding and Complexity: Slot Accuracy decreases as values become more complex, with sharp drops for long strings, identifiers, and nested structures.Boolean accuracy varies across frameworks, reflecting differences in grounding context and tool-constraint enforcement.
  • 5.4 Tool Selection Robustness: Hard negatives reduce Tool Selection Accuracy from 94.50% at N=1 to 69.75% at N=5, while end-to-end Exact Match remains low at 14.25–18.25%.Conditioned Arg_F1 also falls from 29.88 to 22.64, showing that semantically similar distractors affect parameter extraction as well as tool choice.
  • 5.5 Error Mode Diagnosis: Failure patterns shift from retrieval misses in weaker frameworks toward retrieved-but-unused evidence in stronger ones, while retrieval misses remain the largest category overall.Agentic frameworks show negligible tool-selection errors, but locating sparse evidence under implicit queries remains difficult.

6 Conclusion

MEM2ACTBENCH evaluates whether tool-augmented agents can apply long-term memory to execute tasks, revealing shortcomings in proactive parameter grounding, especially for underspecified tasks and mid-context memories.

  • MEM2ACTBENCH evaluates whether tool-augmented agents can apply long-term memory to drive task execution.
  • Current memory frameworks show a significant gap in parameter grounding, particularly for mid-context memories.
  • The findings highlight limitations in proactively using dispersed, incomplete information when tasks are underspecified.

Limitations

MEM2ACTBENCH evaluates memory-grounded parameterization under controlled offline conditions, but its scope excludes interactive execution and broad model diversity.

  • MEM2ACTBENCH is limited to offline tool-call generation under controlled conditions.
  • The benchmark uses a fixed backbone model family, limiting representation of real-world model diversity.
  • Interactive execution with agent adaptation to feedback over time is excluded.
  • Automated task generation may not fully capture real-world dialogue complexity, while human verification may introduce bias in edge cases.

Ethical considerations

MEM2ACTBENCH synthesizes interaction histories from publicly available datasets without collecting new end-user data or conducting human-subject experiments.

  • MEM2ACTBENCH synthesizes interaction histories from ToolACE, BFCL, and OpenAssistant datasets.
  • The study collected no new data from end users and conducted no human-subject experiments.
  • The authors follow source-dataset ethics practices and ACL ethics guidance to reduce privacy risks.

A.1 Pipeline Implementation

The pipeline standardizes heterogeneous sources into multi-turn conversational data and a unified JSONL format compatible with standard chat completion APIs.

  • Pipeline overview: A standardized pipeline unifies heterogeneous data sources into a consistent multi-turn format.
  • ToolACE Processing: ToolACE traces are parsed for nested function structures and refined into natural-language dialogues while preserving execution logic.
  • BFCL v3 Synthesis: BFCL query-response pairs are expanded into multi-round histories with clarifications and sequential tool invocations.
  • OASST1 Formatting: OASST1 threads are reconstructed from leaf nodes to roots, filtered for rank-0 responses, deduplicated by conversation path, and translated into English.
  • Data Schema: Processed data are serialized as unified JSONL compatible with standard chat completion APIs.

A.2 Fact Extraction, Semantic Clustering, and Local Conflict Resolution

The pipeline converts dialogue into consistent, topic-organized memory by extracting facts, clustering related attributes, and resolving local conflicts. It then filters and verifies generated queries to ensure tool parameters genuinely require long-term memory.

  • Fact Extraction, Semantic Clustering, and Local Conflict Resolution: The memory construction pipeline extracts atomic facts, clusters them by semantic topic, and resolves inconsistencies within each local group.This hierarchy makes topic histories locally consistent before integration into a global evolution chain.
  • Fact Extraction, Semantic Clustering, and Local Conflict Resolution: Facts are represented as attribute–fact–source triples, with normalized entity-bound attributes reducing spurious merging across unrelated topics.The extraction process uses structured attributes and source identifiers to preserve provenance.
  • Fact Extraction, Semantic Clustering, and Local Conflict Resolution: BERTopic groups semantically related attributes using normalized BAAI/bge-m3 embeddings and HDBSCAN, then maps each cluster to a canonical representative.The clustering configuration favors finer-grained groupings, including sparse thematic connections.
  • Fact Extraction, Semantic Clustering, and Local Conflict Resolution: Local conflict resolution orders events chronologically, discards obsolete or contradictory sources, and synthesizes a narrative explaining state changes.These operations produce a coherent timeline for each topic before global integration.
  • Fact Extraction, Semantic Clustering, and Local Conflict Resolution: Global evolution sequences use a modified Kahn topological sort with deterministic cycle breaking that removes the highest-out-degree node and uses lexicographic order as a tie-breaker.The procedure returns both a globally sorted sequence and discarded facts.
  • Query Filtering and Human Verification: Generated queries are rejected when lexical or semantic leakage makes tool parameters inferable without memory, and experts verify factual fidelity, conflict resolution, and information necessity.Human adjudication distinguishes updates from refinements and applies strict rejection criteria to unsupported or leaked constraints.
Loading 2601.19935v1…