MEMTRACK: Evaluating Long-Term Memory and State Tracking in Multi-Platform Dynamic Agent Environments

1 Introduction

• MEMTRACK evaluates memory in dynamic agentic tasks that require live context switches, beyond single-thread conversational benchmarks.The motivation is the use of automatic context acquisition in real-world agentic tasks.Source: §1 · pp. 1–2
• The benchmark tests contradictory memory management and multi-hop retrieval across knowledge bases while remaining agnostic to the memory backend.It is designed for long-context models as well as agents with attached memory components.Source: §1 · p. 2
• MEMTRACK provides asynchronous Linear, Slack, notification, filesystem, and Git tools for realistic software-engineering workflows.Its 47 datapoints use manual, bottom-up agent-based, and hybrid curation; answers can be brief phrases evaluated by matching or an LLM judge.Source: §1 · p. 2
• The paper asks whether LLMs can reason over large event histories and codebases, use memory components effectively, and access information across platforms.Source: §1 · p. 2
• State-of-the-art LLMs struggle with multi-platform context reasoning, and memory tools can increase redundancy in planning and tool use.Qualitative analysis also finds repeated reading of tickets, conversations, and code files.Source: §1 · p. 2

Agentic Evaluation

• Existing agent-tool benchmarks test only subsets of planning, selection, invocation, and response generation, while planning benchmarks may lack ecological validity or realistic complexity.Source: §2 · pp. 2–3

Agentic Memory

• Memory evaluation spans acquisition, utilization, and maintenance, but quantitative evaluation of which memories to retrieve remains understudied.Prior work also examines conflicting information and long-range forgetting.Source: §2 · p. 3

Agentic Memory Implementations

• Memory implementations include filesystems, databases, and graphs, while studies report gains from hierarchical, iterative, mixed, or gist-based memory approaches.These approaches have often been evaluated on particular datasets or workflows.Source: §2 · pp. 3–4
• MEMTRACK aims to create complex and diverse agentic workflows by scaling agent search spaces across tools and tasks.Source: §2 · p. 4

Reinforcement Environments for LLM Agents

• MEMTRACK is presented as a benchmark for memory-augmented agent systems within the broader development of reinforcement environments for LLM agents.Source: §2 · p. 4

3 MEMTRACK Dataset

• Each of MEMTRACK’s 47 instances contains an event timeline and question-answer pairs introduced sequentially, without revealing how many questions will follow.Sequential injection is intended to prevent preemptive solution planning.Source: §3 · p. 4
• Timeline events carry timestamps, platform types, and content, and are loaded into Linear, Slack, and Git servers for access through tools or notifications.Events are not available to agents as a whole, requiring platform switching and information retention; the dataset combines three event-history creation approaches.Source: §3 · p. 4

Top-Down Approach

• The bottom-up approach starts from closed issues with merged pull requests, then generates related Slack conversations and Linear tickets with counterfactuals and distractors.Human annotators filter for deterministic answers and manually evaluate the generated timeline; selected pull requests have at most two changed files.Source: §3 · p. 4, Top-Down Approach · p. 4
• Top-down curation begins with scenarios from four in-house experts, which are expanded into timeline events and manually inspected against task descriptions.Experts manually generate question-answer pairs by reviewing the generated timelines.Source: Top-Down Approach · p. 5
• The hybrid approach iteratively expands event-history complexity under annotator direction, who then attempt the task to check completeness.The dataset also uses seven metrics to assess diversity and whether it includes long-term memory across platforms.Source: Top-Down Approach · pp. 5–6

4.1 Methods

• LLM+NOMEM uses no within-agent memory, while LLM+MEM0 and LLM+ZEP add memory tools through their respective components.MEM0 uses GPT-4O-MINI embeddings and ChromaDB; the experiments use default API settings for both memory backends.Source: §4.1 · p. 7

4.2 Evaluation Measures

• Correctness averages an LLM judge’s scores for each question, expected answer, and agent answer across an instance.Source: §4.2 · p. 7
• Efficiency penalizes excessive timeline-access tool calls relative to a minimum of 10, while redundancy measures calls equivalent to or subsumed by earlier calls.Redundancy is judged over tool names and arguments; higher efficiency or lower redundancy is not desirable if correctness drops non-trivially.Source: §4.2 · p. 7

5.1 Quantitative Analysis

• Successful tool calls have mean and median fractions around 0.90–0.96 across approaches, and both GPT-5 and GEMINI-2.5-PRO are described as fairly fluent at tool execution.The stated minimum successful-call fraction is between 0.7 and 0.8.Source: §5.1 · pp. 7–8

5 Results and Discussions

• GPT-5 methods achieve roughly four times the correctness of GEMINI-2.5-PRO methods, regardless of the accompanying memory method.Table 3 reports results across methods, including Correctness, Efficiency, Redundancy, and tool-call entropy.Source: §5 · p. 7, Table 3 · p. 8

5.1 Quantitative Analysis

• Mean correctness drops slightly but notably on follow-up questions, and the pattern holds across all methods.Table 4 is identified as reporting the drop in performance on follow-up questions.Source: §5.1 · p. 8, Table 4 · p. 8

5.2.1 Illustrative GPT-5 Run

• In one GPT-5 run, the agent made 42 platform-interleaving tool calls, switched platforms seven times, and correctly answered two follow-up questions.The run included filesystem, Linear, and Slack calls and began by cloning the repository.Source: §5.2.1 · p. 8

5.2.2 Redundancy Patterns

• Redundancy includes replacing broad calls with narrower repeats and re-accessing information after gaps of at least three turns; gradual widening is not necessarily an anti-pattern unless overdone.Examples include repeated Slack and Linear access and reading the same file twice.Source: §5.2.2 · pp. 8–9

5.1 Quantitative Analysis

• GPT-5 reaches about 60% MEMTRACK performance irrespective of memory components; MEM0 and ZEP show no significant performance improvement, and GEMINI-2.5-PRO+MEM0 slightly degrades.The research-question discussion also reports increased tool-call redundancy with memory tools.Source: §5.1 · p. 8, §5.3 · p. 9

5.3 Answering Research Questions

• Models often prefer repeatedly accessing information over memory tools, while cross-platform retention remains suboptimal and tool calls show at least 20% redundancy.The discussion also reports poor follow-up performance and says diverse tool use remains a limiting factor.Source: §5.3 · p. 9

6 Conclusion & Future Work

• MEMTRACK combines manual, bottom-up, and hybrid curation to benchmark agent memory in workflows reflecting real organizations.The framework is proposed as a basis for future environments in which agents can act in organizational tools and affect the evolving timeline.Source: §6 · p. 9
• Frontier LLMs such as GPT-5 can reason across large event histories, but their performance declines on follow-up queries.Source: §6 · p. 9
• MEM0 and ZEP provide limited gains, while memory-tool use remains inefficient.Source: §6 · p. 9
• This initial work avoids settings where agents act in the organization, such as creating tickets or sending messages; future extensions could cover marketing or sales.Source: §6 · p. 9

A.1.1 Bottom-Up Approach

• Scenarios require at least five reasoning hops, three platform switches, and temporal dependencies, with information fragmented across platforms and decisions that may later be revised.Guidelines also call for plausible distractors, such as similar tickets, abandoned approaches, and overlapping discussions.Source: §A.1.1 · pp. 14–15
• Questions should reflect workplace information needs, use vague framing that requires genuine reasoning, have deterministic answers, and require synthesis across at least three platforms.No single message thread or ticket should contain enough information to answer; a human evaluator should be able to verify correctness from the event-history JSON.Source: §A.1.1 · p. 17
• If an agent passes the questions for an event history, the creators iteratively increase difficulty through more complex questions or event histories on a per-sample basis.The output specification calls for a chronological multi-platform event history with message threads, attribution, timestamps, ticket status transitions, and cross-platform references.Source: §A.1.1 · pp. 16–17

A.1.2 Top-Down Approach

• The top-down prompt generates a JSON timeline from a natural-language scenario, exemplified by a product-image search feature whose implementation changes after user feedback clarifies what “description” means.The example moves from keyword search over product catalog descriptions to a new implementation based on descriptions supplied by users.Source: §A.1.2 · p. 17

A.2.1 Correctness

• The correctness evaluator compares extracted and expected answers using exact or semantically equivalent matches, partial credit for multi-part answers, and reasonable interpretations of ambiguous questions.Its examples assign 0.5 to an answer containing only one of two expected names, while extra information does not reduce a fully correct score.Source: §A.2.1 · p. 18

A.2.2 Redundancy

• The redundancy evaluator identifies repeated retrieval when an agent makes tool calls that retrieve similar information, including calls to the same tool with very similar parameters.The prompt gives repeating a call such as get_ticket(id1) as an example.Source: §A.2.2 · p. 18
• Semantically equivalent queries, such as asking for tickets with status “done” and then “completed,” are treated as redundant.The example begins across passages and is completed by the next passage.Source: §A.2.2 · p. 18
• A broad query that contains information from a specific query is also classified as redundant.The example starts with get_ticket(id1), followed by a broader query.Source: §A.2.2 · p. 18
• Different tools, meaningfully different parameters for the same tool, and progression from general to specific are explicitly not redundant.Source: §A.2.2 · p. 18
• Redundancy scoring counts redundant calls among total calls, computes their fraction, and defines efficiency as 1.0 minus that fraction.The requested JSON also includes a brief explanation of which calls were redundant.Source: §A.2.2 · p. 18

A.3.1 Cross Reference

• Cross-reference prompts rate content from 0.0 to 1.0 for references to Linear, Slack, and Git.The output is a JSON object with separate linear, slack, and git scores.Source: §A.3.1 · p. 19
• Examples assign high scores to explicit platform cues: a Linear ticket mention scores 0.9, while a PR mention and Slack channel attribution each score 0.8.Source: §A.3.1 · p. 19

B.1 Illustrative GPT-5 Run - Complete Example

• For the question asking for the sum of numeric default parameters in factory function definitions within test files, the recorded answer was 4858.9001 and was marked incorrect; the listed expected answer was 547.Source: §B.1 · pp. 19–20
• GPT-5 returned 2025-05-15 for the date question and 142 for the code line question; both answers are marked correct.Source: §B.1 · pp. 19–20

B.2 Hyperparameters

• The experiments use temperature 1 to support reasoning and avoid an implicit length horizon during longer outputs and turn sequences.Source: §B.2 · p. 20

B.3 σ of Correctness

• Table 6 reports σ on correctness numbers averaged across five runs.Source: Table 6 · p. 20

B.4.1 Slack

• Slack tools retrieve channel and direct messages, list workspace channels and users, and send channel, direct, or offline messages.Unread messages are marked as read after retrieval; message-history tools support pagination.Source: §B.4.1 · pp. 20–21

B.4.2 Linear

• Linear tools support ticket creation, retrieval, filtering, updates, deletion, assignment, team membership, and milestone management.Ticket deletion, team membership changes, and milestone operations include admin-only restrictions where specified.Source: §B.4.2 · p. 21

B.4.3 Git

• Git access includes listing remote repositories with clone instructions and commit information.Source: §B.4.3 · p. 21

B.4.4 Shell Commands

• The agent can use shell commands and filesystem tools such as list_directory(), read_file(), and search_file().The passage also notes that git clone can be executed in the agent's filesystem.Source: §B.4.4 · p. 21

B.5 Memory Based Components

• Memory components store information persistently, search relevant memories with hybrid semantic and keyword retrieval, and retrieve recent memories.The listed operations are store_memory(), search_memory(), and get_memories().Source: §B.5 · p. 21

• The evaluation experiments use a stated configuration, and Table 7 is titled as the environment's system requirements.Source: §B.6 · p. 21, Table 7 · p. 22
• Table 7 labels its subject as resources and requirements for running the environment.Source: Table 7 · p. 22
• The supplied table text identifies system requirements as its scope but gives no specific resource values.Source: Table 7 · p. 22