PaperlayerView exact revision on arXiv

Source-linked AI summary

LongMemEval-V2: Evaluating Long-Term Agent Memory Toward Experienced Colleagues

Di Wu, Zixiang Ji, Asmi Kawatkar, Bryan Kwan, Jia-Chen Gu, Nanyun Peng, Kai-Wei Chang

arXiv:2605.12493v1cs.CL

TL;DR

Existing agent-memory benchmarks provide limited insight into accumulating holistic, environment-specific knowledge from sustained interaction with complex environments. This paper introduces LongMemEval-V2 and AgentRunbook methods, with AgentRunbook-C achieving 72.5% average accuracy while advancing the accuracy-latency frontier.

  • Problem

    Existing agent-memory benchmarks offer limited insight into whether systems accumulate holistic, environment-specific knowledge from sustained interaction with complex environments.

  • Method

    The paper introduces LME-V2 and AgentRunbook-R/C to evaluate and gather evidence for five memory abilities from specialized web-agent trajectories.

  • Results

    72.5% average accuracy: AgentRunbook-C performs best overall and advances the accuracy-latency frontier over the strongest RAG and coding-agent baselines.

  • Takeaways & Limitations

    LME-V2 establishes a challenging testbed for developing long-term memory systems for environment experience.

  • Takeaways & Limitations

    LME-V2 focuses on customized browser environments and pre-collected histories, so it may not capture memory requirements across other agent types or online distribution shifts.

Abstract

from arXiv · show

Long-term memory is crucial for agents in specialized web environments, where success depends on recalling interface affordances, state dynamics, workflows, and recurring failure modes. However, existing memory benchmarks for agents mostly focus on user histories, short traces, or downstream task success, leaving open how to directly evaluate whether memory systems effectively internalize environment-specific experience. To address this gap, we introduce LongMemEval-V2 (LME-V2), a benchmark for evaluating whether memory systems can help agents acquire the experience needed to become knowledgeable colleagues in customized environments. LME-V2 contains 451 manually curated questions covering five core memory abilities for web agents: static state recall, dynamic state tracking, workflow knowledge, environment gotchas, and premise awareness. Questions are paired with history trajectories containing up to 500 trajectories and 115M tokens. We use a context gathering formulation: memory systems consume history trajectories and return compact evidence for downstream question answering. We propose a suite of two memory methods: AgentRunbook-R, an efficient RAG-based memory with knowledge pools for raw state observations, events, and strategy notes, and AgentRunbook-C, which stores trajectories as files and invokes a coding agent to gather evidence in an augmented sandbox. Experiments show that AgentRunbook-C achieves the best performance with 72.5% average accuracy, outperforming the strongest RAG baseline (48.5%) and the off-the-shelf coding agent baseline (69.3%). Despite the strong performance gains, coding agent based methods have high latency costs. While AgentRunbook-C advances the accuracy-latency Pareto frontier, substantial room for improvement remains. Together, these results establish LME-V2 as a challenging testbed for developing long-term memory systems for environment experience.

1 Introduction

LongMemEval-V2 addresses limited evaluation of environment-specific agent memory by testing whether web agents can accumulate experience and become knowledgeable colleagues. It introduces a 451-question benchmark and a context-gathering evaluation that requires memory systems to preserve both observations and higher-level environmental knowledge.

  • Motivation: Existing agent-memory benchmarks often use simplified games, limited trajectory dependencies, or downstream task success, providing limited insight into accumulated environment experience.These limitations motivate direct evaluation of whether memory systems internalize specialized-environment experience.
  • Benchmark: LongMemEval-V2 contains 451 manually curated questions spanning five memory abilities across customized websites from WebArena and WorkArena.The five abilities are static state, dynamic state, workflow, gotchas, and premise awareness.
  • Evaluation: LME-V2 evaluates memory through context gathering: systems insert trajectories, query multimodal evidence, and provide truncated context to a fixed reader LLM.Associated trajectories are streamed sequentially into memory for each question.
  • Design challenge: Successful memory systems must filter noisy trajectories while retaining low-level observations, environment dynamics, and procedural knowledge.The introduction argues that methods designed for conversational or purely strategic contexts may therefore be ineffective.
  • Results: 69.3% accuracy was achieved by the off-the-shelf Codex agent, while AgentRunbook-R reached 57.8% and simple state-slice RAG reached 40.1%.Codex required about 182 seconds per query, approximately 6.9 times slower than AgentRunbook-R.

2 Related Work

Prior work evaluates long-context recall and increasingly personalized memory, while agent-memory research addresses trajectory recall, cross-trajectory knowledge consolidation, and LLM-controlled memory operations. These directions motivate memory systems for long-horizon agents in complex environments.

  • Long-Context and Memory Evaluation: Long-term memory evaluation builds on benchmarks for information retrieval, aggregation, and instruction following over extended documents.This line of work evaluates recalling information across extended context.
  • Long-Context and Memory Evaluation: Subsequent benchmarks examine personalized memory, including explicit user facts and implicit preferences.Examples include LoCoMo, DialSim, PerLTQA, LongMemEval, PersonaMem, and BEAM.
  • Memory Systems for Agents: Agent-memory systems support recalling detailed trajectory context and consolidating high-level knowledge across trajectories in long-horizon environments.Memory has also been linked to extended exploration and sleep-time offline consolidation for improving inference-time performance.
  • Agents as Memory Controllers: Agentic memory systems delegate memory writes and reads to LLMs, enabling programmatic context management and evolving memory content or structure.Examples include MemGPT, StateLM, A-MEM, and Mem0.

3 LongMemEval-V2

LongMemEval-V2 evaluates whether memory systems can internalize environment-specific experience across five memory abilities using multimodal web-agent trajectories. It combines long, sparse history haystacks with a context-gathering evaluation and pilot evidence that parametric knowledge alone is insufficient.

  • Memory abilities: LME-V2 defines five memory abilities: static state recall, dynamic state tracking, workflow knowledge, environment gotchas, and premise awareness.These cover landmarks and affordances, state changes, task procedures, recurring failures, and environment-specific assumptions.
  • Dataset construction: The benchmark is manually curated from multimodal trajectories collected in WebArena, WorkArena, and WorkArena++ environments, with questions, answer-bearing trajectories, and shared haystacks annotated.The annotation pipeline comprises trajectory collection, question annotation, answer trajectory labeling, and haystack creation.
  • Dataset construction: 115M tokens characterize the final LME-V2-Medium history length, while each question’s answer-bearing trajectories remain sparse within balanced successful and failed haystacks.The small tier contains approximately 25M tokens, and many questions can only be answered from failed trajectories.
  • Evaluation formulation: The context-gathering task sequentially inserts an ordered trajectory haystack, queries memory with a question, and evaluates a fixed reader using answer accuracy and query latency.Accuracy uses normalized string matching for structured answers and an LLM judge for free-form answers.
  • Pilot studies: 14.1% overall accuracy is the best result for frontier LLMs prompted without trajectory history, indicating that LME-V2 generally cannot be answered from public or parametric knowledge alone.With oracle answer-bearing trajectories, evidence slices and strategy-note summaries improve direct QA to 82.5% and 86.3%, respectively.

4 AgentRunbook

AgentRunbook addresses LME-V2’s mixed evidence needs with two complementary memory designs: a structured multi-pool RAG pipeline and a coding-agent method that manages trajectories as files. AgentRunbook-R retrieves evidence across granularities, while AgentRunbook-C searches and inspects trajectory files at query time with lightweight scaffolding.

  • AgentRunbook-R: AgentRunbook-R organizes trajectory evidence into separate knowledge pools for targeted recall across low-level observations, state changes, and procedures.It builds raw state slice, event, and note pools from each trajectory.
  • AgentRunbook-R: At query time, AgentRunbook-R generates multiple raw-state queries plus one event and one note query, skipping irrelevant streams when appropriate.An LLM controller reasons over the query and current memory snapshot before dense retrieval renders multimodal context.
  • AgentRunbook-C: AgentRunbook-C stores trajectories directly as files and uses a coding agent to search, inspect, and select evidence at query time.This replaces a fixed vector-search pipeline with direct file-based trajectory management.
  • AgentRunbook-C: AgentRunbook-C adds a workflow document and query-time manifest artifacts to guide evidence collection and shortlist relevant trajectories.The scaffolding addresses off-the-shelf coding agents’ potential to over-explore, under-explore, or inspect data inefficiently.

5 Experiments

Experiments show that AgentRunbook improves long-term memory accuracy across LME-V2 tiers, with AgentRunbook-C achieving the strongest results. The results also identify workflow instructions as important and show that reasoning effort strongly affects latency.

  • Experimental setup: All methods are evaluated under context gathering with memory context truncated to 200K tokens and answers produced by a fixed Qwen3.5-9B reader.RAG methods use Qwen3.5-9B as the memory controller and Qwen3-Embedding-8B for retrieval, while coding-agent methods use Codex and GPT-5.4-mini.
  • Accuracy results: 58.6% on LME-V2-Small and 57.0% on LME-V2-Medium: AgentRunbook-R outperforms the strongest RAG baseline.The strongest RAG baseline reaches 51.0% and 45.9% after adding trajectory notes.
  • Accuracy results: 74.9% on LME-V2-Small and 70.1% on LME-V2-Medium: AgentRunbook-C achieves the best overall accuracy.Vanilla Codex reaches 69.9% and 68.7% on the corresponding tiers.
  • Ablation results: Workflow instructions are consistently important, while manifest artifacts mainly improve efficiency and helper functions have mixed effects across tiers.Helper functions improve the small-tier result and reduce latency, but do not uniformly improve medium-tier accuracy.
  • Accuracy-latency trade-offs: Reasoning effort has a large and direct effect on overall query latency, and AgentRunbook-R provides a moderate-accuracy, low-latency baseline at around 26 seconds.The latency analysis evaluates methods across different operating points.

6 Conclusion

LongMemEval-V2 establishes a benchmark for evaluating whether memory systems help agents become experienced operators of specialized environments. It covers five memory abilities at beyond 100M-token context depth and introduces two AgentRunbook methods with complementary improvements.

  • Benchmark contribution: LongMemEval-V2 defines agent memory evaluation around helping agents become experienced operators of specialized environments.The benchmark holistically covers five memory abilities.
  • Benchmark contribution: Beyond 100M-token context depth, LME-V2 evaluates memory using large multimodal web-agent histories.This advances the context depth of memory benchmarks.
  • Memory methods: AgentRunbook-R improves standard RAG-based methods with dedicated memory pools.It is one of the two memory methods proposed alongside AgentRunbook-C.
  • Memory methods: AgentRunbook-C uses coding agents’ file manipulation abilities and lightweight workflow guidance, manifests, and inspection tools to improve accuracy and latency.These components support the method’s further improvements over file-based trajectory handling alone.

A LongMemEval-V2: Further Benchmark Construction Details … A.4 Haystack Creation

The benchmark is constructed from multimodal web-agent trajectories, experience-focused questions, validated answer-trajectory coverage, and tiered shared or per-question haystacks. Collection and annotation procedures emphasize environment-specific evidence, while haystack construction preserves answer coverage as distractor scale increases.

  • A.1 Trajectory Collection: Trajectories are collected from selected WebArena tasks and all WorkArena and WorkArena++ tasks because these environments contain customized, experience-relevant behavior.WebArena uses OneStopShop, CMS, and Reddit; WorkArena and WorkArena++ are both based on ServiceNow.
  • A.1 Trajectory Collection: AgentLab standardizes collection through accessibility trees, screenshots, high-level BrowserGym actions, observation preprocessing, and trajectory logging.Most trajectories use generic ReAct-style agents with GPT-5-mini or GPT-5.2, supplemented by a Codex-controlled manual action agent.
  • A.1 Trajectory Collection: Each retained trajectory contains the task goal and ordered screenshot, accessibility-tree, and BrowserGym state-action records; some WorkArena goals are sanitized to remove navigation hints.Sanitization preserves task intent and task-specific values while removing explicit routes and step-by-step module guidance.
  • A.2 Question Annotation: Annotators write questions requiring environment-specific facts, state changes, workflows, and gotchas rather than general knowledge of public platforms.Questions are validated for answerability, correctness, and matching question type, while parametric-knowledge testing filters or perturbs questions that strong models answer too easily.
  • A.2 Question Annotation: Questions must have relevant answer evidence visible in screenshots, excluding questions answerable only through accessibility-tree information.Accessibility trees may expose hidden labels or structured fields more clearly, but visual trajectory evidence remains required.
  • A.3 Answer Trajectory Labeling: The answer-trajectory labeling pass expands seed annotations into a question-to-trajectory coverage map, including multiple required evidence hops when necessary.Codex workers process coherent question batches, then humans resolve ambiguities and validate selected trajectories, screenshot visibility, and multi-hop constraints.
  • A.3 Answer Trajectory Labeling: Labeling rules require direct visual evidence for static questions, before/after states for dynamic questions, procedural steps for workflows, and careful context for gotchas.Gotcha questions are treated as high-risk because their evidence is often subtle and context-dependent.
  • A.4 Haystack Creation: Haystacks are built through answer-core selection, small-haystack expansion, and medium-haystack expansion from the final coverage map.The minimal core covers every required hop; shared 100-trajectory domain haystacks add ranked fillers, while per-question medium haystacks reuse answer seeds and target 500 trajectories.

A.5 Evaluation · B LongMemEval-V2: Pilot Studies · B.1 Can Frontier Models Answer Without Trajectory History?

The evaluation uses context gathering with a fixed reader, deterministic or judge-based scoring, and controlled prompt and model settings. Pilot studies separately test direct answering without trajectory evidence, finding that frontier models perform poorly from parametric knowledge alone.

  • A.5 Evaluation: The context-gathering protocol inserts each question’s trajectories into memory, queries for evidence, validates text-and-image outputs, truncates context to 200K Qwen3.5-9B tokens, and passes it to a fixed reader.The reader receives a domain-specific system prompt, returned memory context, question text, and question image.
  • A.5 Evaluation: Structured answers use deterministic evaluators, while gotcha and abstention questions receive binary semantic judgments from an LLM judge.The reported aggregate is accuracy over the full question set, with additional reporting described in the evaluation section.
  • A.5 Evaluation: Prompt tuning targets near-zero performance without memory context and maximal performance with oracle image-and-text state slices; GPT-5.2 judges gotcha and abstention correctness.The judge uses medium reasoning effort.
  • B LongMemEval-V2: Pilot Studies: The two pilot studies use direct question answering to test whether questions can be answered without trajectory evidence and whether answer-bearing trajectories enable reliable answering.Results cover non-abstention questions because abstention questions contain intentionally misleading premises.
  • B.1 Can Frontier Models Answer Without Trajectory History?: All evaluated frontier models perform poorly on LME-V2 without trajectory evidence, receiving only the question and applicable image while being instructed to answer UNKNOWN rather than guess.The study evaluates parametric knowledge alone through OpenRouter with medium reasoning effort across models.

B.2 Can Models Reliably Answer with Oracle Trajectory Access?

Oracle trajectory access removes retrieval difficulty but does not make direct question answering reliable. Evidence slicing, notes, and especially coding-agent exploration substantially improve performance, motivating AgentRunbook-C.

  • Evaluation setup: Oracle access isolates reading, grounding, and reasoning over answer-bearing trajectories by removing the retrieval problem.The evaluation compares full oracle trajectories, oracle slices plus notes, and coding-agent direct QA in a local sandbox.
  • Oracle trajectory results: 59.6%: Qwen3.5-9B with full oracle trajectories; 65.3%: GPT-5.4-mini (medium), showing full trajectories are insufficient for reliable direct QA.These results come from the Figure 4 right table’s full-oracle setting.
  • Oracle trajectory results: 82.5%: evidence slicing and 86.3%: notes substantially improve direct QA over full oracle trajectories.The passage reports these values as the gains from evidence slicing and notes, respectively.
  • Coding-agent direct QA: 89.7%: Codex direct QA, indicating that file-system exploration and iterative evidence inspection effectively process agent trajectories and motivate AgentRunbook-C.Codex inspects local files and writes the answer to answer.json rather than returning a memory context.

C Implementation Details … D Further Analyses

The implementation compares RAG and coding-agent memories, detailing AgentRunbook-R’s structured retrieval pools and AgentRunbook-C’s sandboxed file-based evidence gathering. Further analyses examine the main methods across pooled web and enterprise questions within each benchmark tier.

  • C.1 Baselines: LME-V2 evaluates RAG and coding-agent methods because conversation-focused memory systems are poorly suited to environment-specific experience.The authors select the strongest methods from pilot studies and also study ablations.
  • C.1 Baselines: Codex serves as the off-the-shelf coding-agent baseline, using binary v0.117.0 in the sandbox setting on a local Linux server.The baseline is chosen for performance, popularity, and open-source availability.
  • C.2 AgentRunbook-R: AgentRunbook-R maintains raw-state, event, and note retrieval pools built from simplified trajectories containing interface states, actions, screenshots, URLs, and metadata.Procedure notes summarize reusable task steps, while hint notes capture durable environment-specific observations and pitfalls.
  • C.2 AgentRunbook-R: At query time, AgentRunbook-R emits structured queries for raw states, interaction effects, and reusable procedures or gotchas, then retrieves and merges dense-similarity results.The raw-state field supports up to five focused queries targeting distinct UI surfaces or entities.
  • C.2 AgentRunbook-R: AgentRunbook-R’s main experiments use 5 raw-state queries, top-6 events, top-3 notes, Qwen3-Embedding-8B retrieval, and Qwen3.5-9B control.Inputs are truncated to 4,096 tokens, with temperature 0.6 and top_p 0.95.
  • C.3 AgentRunbook-C: AgentRunbook-C stores trajectories and screenshots as files, then uses an isolated coding-agent sandbox to inspect files and produce compact structured evidence.Manifest summaries and an inspection helper support trajectory shortlisting and targeted inspection before output generation.
  • C.3 AgentRunbook-C / D Further Analyses: AgentRunbook-C validates memory_markdown and zero-based inclusive trajectory spans under a total budget of 20 states before rendering selected evidence for the fixed reader.Evaluation limits parallel query invocation to 3 processes, while further analyses pool web and enterprise questions within each LME-V2 tier.

D.1 Error Analyses … E Limitations and Ethics Statements

The error analysis shows AgentRunbook-C and Codex reduce retrieval and downstream-reading errors relative to RAG methods, while tool analysis shows AgentRunbook-C shifts effort toward harness-guided retrieval. Qualitative examples illustrate evidence drawn from AgentRunbook-R’s memory pools and AgentRunbook-C’s selected spans.

  • D.1 Error Analyses: AgentRunbook-R significantly reduces retrieval and downstream reading errors compared to the RAG+notes baseline.It does not improve abstention because it directly presents evidence without analysis, allowing downstream models to be misled.
  • D.1 Error Analyses: Both Codex and AgentRunbook-C make fewer retrieval and downstream reader errors than the RAG methods.AgentRunbook-C further improves the error profile, as described in the supplied passage.
  • D.2 Tool Calling Behavior: The coding-agent behavior analysis groups commands into setup/read, harness-guided retrieval, raw trajectory exploration, visual inspection, and output-validation or other classes.Harness-guided retrieval includes manifest or summary reads and AgentRunbook helper calls, while raw exploration includes filesystem searches and ad-hoc Python scans.
  • D.2 Tool Calling Behavior: AgentRunbook-C reduces total commands and shifts work from raw trajectory exploration toward harness-guided retrieval.The comparison is based on mean command executions per query and command-type distributions.
  • D.2 Tool Calling Behavior: 21.8 raw trajectory-exploration commands per question are used by Codex on LME-V2-Medium, versus 1.2 by AgentRunbook-C.AgentRunbook-C instead uses 18.0 harness-guided retrieval commands per question on average.
  • D.2 Tool Calling Behavior: Both Codex and AgentRunbook-C begin with setup and task-reading commands in the first command rounds.Figure 9 visualizes the distribution of grouped command classes during the first eight command rounds.
  • D.3 Qualitative Examples: Qualitative examples show AgentRunbook-R answering through procedure/hint notes, state-transition events, and raw state slices.The examples select one successful query where each memory pool provides answer-bearing evidence.
  • D.3 Qualitative Examples: AgentRunbook-C qualitative examples present two selected evidence spans from different question types to illustrate reader-facing evidence.The supplied passages contain no substantive limitation or ethics findings for section E.

E.1 Limitations · E.2 Ethics Statements · E.3 LLM and Agent Use

The paper limits LME-V2 to customized browser environments and practical memory designs, while documenting safeguards around synthetic data, licensing, societal impacts, and human oversight of LLM-assisted work.

  • E.1 Limitations: LME-V2 evaluates web agents in customized browser environments, not coding, computer-use, or domain-specific enterprise agents with potentially different memory needs and risks.For reproducibility and controlled comparison, it evaluates memory over pre-collected trajectory histories.
  • E.1 Limitations: AgentRunbook emphasizes retrieval, file organization, and agent-native evidence inspection rather than new model architectures or training procedures.Its performance depends on trajectory representations, retrieval models, prompts, and coding-agent behavior.
  • E.2 Ethics Statements: Four student authors constructed the benchmark, with compensation through wage support or research credits, authorship credit, and recurring annotation-review meetings.The reported process included weekly annotation sessions, question-review discussions, and trajectory-label verification meetings.
  • E.2 Ethics Statements: LME-V2 uses sandboxed environments derived from WebArena, WorkArena, and WorkArena++, with synthetic tasks, names, records, and personal information to minimize real-user privacy exposure.The authors manually inspected curated questions and found no additional information leakage beyond intended synthetic content.
  • E.2 Ethics Statements: The authors follow upstream artifact licenses, including Apache-2.0 terms for WebArena, AgentLab, and the Codex GitHub repository.Codex experiments use OpenAI API access rather than commercial ChatGPT subscriptions to avoid violating consumer-product usage terms.
  • E.2 Ethics Statements: More effective persistent memory could reduce redundant computation, cost, and environmental impact, but it also introduces broader societal risks.The work additionally notes that memory may support self-improving systems developing expertise in specialized domains.
  • E.3 LLM and Agent Use: Human annotators wrote LME-V2 benchmark questions, while Codex generated initial answer-labeling proposals and assisted implementation, with generated code reviewed and validated by the authors.The passage states that no LLMs or agents were used for research ideation.
Loading 2605.12493v1…