PaperlayerView exact revision on arXiv

Source-linked AI summary

Memento: Fine-tuning LLM Agents without Fine-tuning LLMs

Huichi Zhou, Yihang Chen, Siyuan Guo, Xue Yan, Kin Hei Lee, Zihan Wang, Ka Yiu Lee, Guchun Zhang, Kun Shao, Linyi Yang, Jun Wang

arXiv:2508.16153v2cs.LGcs.CL

TL;DR

LLM agents need continual adaptation without the computational cost of fine-tuning, while fixed workflows lack flexibility. Memento addresses this with memory-based online reinforcement learning in a planner–executor architecture, achieving strong results across deep-research benchmarks and factual queries. Its reported gains are strongest with curated case memory, although rapid saturation limits what additional iterations reveal.

  • Problem

    Existing LLM-agent approaches are either rigid and static or computationally expensive because they update underlying model parameters.

  • Method

    Memento models deep research as a memory-based MDP, storing and retrieving episodic trajectories through non-parametric or Q-function-guided Case Memory without updating model weights.

  • Results

    Memento achieves strong performance across GAIA, DeepResearcher, and SimpleQA, including 87.88% Pass@3 on GAIA validation and 95.0% PM on SimpleQA.

  • Takeaways & Limitations

    The findings support memory-based continual adaptation as a practical approach for improving LLM agents without gradient updates, with both parametric and non-parametric CBR contributing to gains.

  • Takeaways & Limitations

    With about 3k training data, the Case Bank saturates quickly, producing rapid convergence and diminishing returns from additional iterations in the finite environment.

Abstract

from arXiv · show

In this paper, we introduce a novel learning paradigm for Adaptive Large Language Model (LLM) agents that eliminates the need for fine-tuning the underlying LLMs. Existing approaches are often either rigid, relying on static, handcrafted reflection workflows, or computationally intensive, requiring gradient updates of LLM model parameters. In contrast, our method enables low-cost continual adaptation via memory-based online reinforcement learning. We formalise this as a Memory-augmented Markov Decision Process (M-MDP), equipped with a neural case-selection policy to guide action decisions. Past experiences are stored in an episodic memory, either differentiable or non-parametric. The policy is continually updated based on environmental feedback through a memory rewriting mechanism, whereas policy improvement is achieved through efficient memory reading (retrieval). We instantiate our agent model in the deep research setting, namely \emph{Memento}, which attains top-1 on GAIA validation ($87.88\%$ Pass@$3$) and $79.40\%$ on the test set. It reaches $66.6\%$ F1 and $80.4\%$ PM on the DeepResearcher dataset, outperforming the state-of-the-art training-based method, while case-based memory adds $4.7\%$ to $9.6\%$ absolute points on out-of-distribution tasks. Our approach offers a scalable and efficient pathway for developing generalist LLM agents capable of continuous, real-time learning without gradient updates, advancing machine learning towards open-ended skill acquisition and deep research scenarios. The code is available at https://github.com/Agent-on-the-Fly/Memento.

1. Introduction

Current LLM agents are either rigid after deployment or costly to adapt through model fine-tuning. Memento addresses this gap by using episodic memory for continual case-based adaptation and reports strong benchmark performance.

  • Static, handcrafted workflows lack flexibility and cannot incorporate online information or adapt to novel situations after deployment.
  • Parameter tuning enables more flexible behaviour but incurs high computational cost, making continuous online adaptation impractical.
  • Memento stores successful and failed trajectories in external episodic memory instead of modifying the underlying LLM, enabling case-based guidance for future decisions.
  • Memento combines a planner, tool-enabled executor, and growing Case Bank within a planner–executor architecture grounded in a memory-based MDP.
  • 87.88% Pass@3 on GAIA validation, 79.40% on its private test leaderboard, and 66.6% F1 with 80.4% PM on DeepResearcher demonstrate strong reported performance.

2. Related Work

Related work spans continual learning, tool-enabled and multi-agent systems, and external memory, but existing approaches often remain costly, fixed, or heuristic-dependent.

  • Parametric continual-learning methods update LLMs through reinforcement learning or supervised fine-tuning, achieving high task fidelity at substantial compute and data costs.
  • Multi-agent pipelines coordinate long-horizon tool calls and dynamic interactions with external environments.
  • Tool-use training methods teach models when and how to invoke APIs but require costly retraining and often assume a fixed, small toolset.
  • Reflective prompting and ReAct-style agents can improve through feedback but remain constrained by predefined heuristics.
  • Long-term memory systems accumulate knowledge and adapt retrieval or retention over time through mechanisms such as forgetting schedules and memory optimisation.

3. Methodology: Memory-Based MDP with Case-based Reasoning Policy

The method models case-based reasoning as a memory-based MDP in which agents retrieve past cases, use an LLM to adapt them into actions, receive feedback, and retain new experiences. Retrieval is improved through entropy-regularized Q-learning or state-similarity estimation, while Memento implements this process in a planner–executor system.

  • Memory-Based MDP: A Memory-Based Markov Decision Process augments standard MDPs with a memory space containing past experiences.
  • Memory-Based MDP: At each timestep, the agent retrieves a case from the current case bank, conditions the LLM on the state and case, executes an action, observes reward and transition, and retains the new case.
  • Case-Based Reasoning Policy: The overall CBR policy combines a case-retrieval policy with the LLM action likelihood conditioned on the current state and retrieved case.
  • Retrieval Policy Learning: Soft Q-learning optimizes retrieval while encouraging diversity through an entropy term, yielding an optimal retrieval policy expressed as a softmax over Q values.
  • Retrieval Policy Learning: A kernel network estimates Q values from episodic interactions sharing the same retrieved case, allowing temporal-difference learning over natural-language states.
  • Memento Architecture: Memento alternates Case-Based Planning and Tool-Based Execution, with Case Memory operations that write trajectories and read cases using similarity or an online-updated Q-function.

4. Implementation: Deep Research Agent

Memento implements deep research through alternating case-based planning and tool-based execution, coordinated by multiple memory modules. Its case bank supports continual adaptation through similarity retrieval or online-updated Q-function retrieval.

  • Framework: Memento alternates between case-based planning and tool-based execution for complex, long-horizon deep research tasks.The planner and executor iteratively interact with the environment, invoke external tools, retrieve information, and process heterogeneous data.
  • Framework: Three memory modules coordinate planning and execution: Case Memory, Subtask Memory, and Tool Memory.Case Memory stores prior cases, Subtask Memory records active subtasks and outcomes, and Tool Memory logs tool interactions.
  • Case-Based Planning: The planner retrieves relevant case triplets and uses them with the current task to generate subtask plans.Retrieved cases contain task, plan, and success information; execution history supports replanning until completion.
  • Case Memory Management: The case bank supports non-parametric similarity retrieval and parametric retrieval driven by an online-updated Q-function.Non-parametric Write appends cases, whereas parametric Write also updates the Q-function to shape adaptive case selection.
  • Case Memory Management: Memory grows by storing state, action, and reward tuples, with state representations encoded for later retrieval.The case bank accumulates successes and failures, supporting retrospective mistake avoidance and prospective planning guidance.
  • Case Memory Management: Non-parametric retrieval selects the K nearest past cases by cosine similarity between encoded current and prior states.The state is vectorised with a pretrained text encoder before similarity-based retrieval.
  • Case Memory Management: Parametric memory trains its Q-function with binary rewards and selects the K highest-Q cases as planning references.The cross-entropy objective models the likelihood that a case is a good reference, while TopK retrieval reduces selection randomness.
  • Tooling: Memento’s tool suite supports external information acquisition and multimodal document processing through retrieval, crawling, captioning, transcription, and parsing.The tools are accessible through the MCP protocol and target the heterogeneous information demands of deep research.

5. Experiments

The experiments evaluate Memento across four benchmarks covering long-horizon tool use, web research, factual precision, and broad-domain reasoning. Evaluation uses Exact Match for GAIA and macro-F1, with Partial Match additionally reported for non-GAIA datasets.

  • Benchmarks: Four benchmarks assess distinct deep research capabilities: GAIA, DeepResearcher, SimpleQA, and HLE.They cover long-horizon tool use, real-time web research, factual precision, and long-tail academic reasoning.
  • Benchmarks: Table 1 reports F1 and PM scores with weighted averages across seven open-domain QA datasets.Bamboogle contributes 125 examples, while each other dataset contributes 512; comparison methods use Qwen2.5 (7B) results from DeepResearcher.
  • Benchmarks: GAIA contains 450 questions spanning three difficulty levels and requiring varying degrees of autonomous planning and tool use.The validation set has 150 questions and the test set has 300, with Level 3 tasks reaching up to 50 steps.
  • Evaluation Metrics: Exact Match evaluates GAIA by marking answers correct only after exact agreement with the normalized reference answer.Macro-F1 is used for DeepResearcher, SimpleQA, and HLE, while PM measures partial semantic agreement with gold answers.

5.3. Model Configurations

Memento uses distinct LLMs and multimodal services across planning, execution, and media processing. Ablation settings isolate the contributions of raw parametric knowledge, live tools, and case-based reasoning.

  • Model Configurations: The planner uses GPT-4.1, while the executor uses o3 for GAIA and o4-mini for the other datasets.Image processing uses GPT-4o, video processing uses Gemini 2.5 Pro, and audio processing uses Assembly AI.
  • Model Configurations: Non-parametric case retrieval encodes sentences with SimCSE and ranks candidate cases by cosine similarity.Parametric case retrieval instead uses SimCSE representations with a two-layer MLP Q-function.
  • Model Configurations: The Offline Executor removes the planner, case memory, and external tools, isolating the executor’s raw parametric knowledge.The Online Executor reconnects the same executor to live search and MCP tools, while Memento without CBR disables episodic memory to measure case-based reasoning gains.

5.4. Experimental Results

Memento performs strongly across long-horizon planning, real-time research, factual question answering, and long-tail academic reasoning benchmarks. Results include top rankings on GAIA validation and SimpleQA, alongside competitive HLE and DeepResearcher performance.

  • 66.6% F1 across seven DeepResearcher benchmarks nearly doubles the 37.7% F1 of the CoT + RAG baseline.
  • Memento attains top-1 on GAIA validation and fourth place on the test set, outperforming most existing agent frameworks.
  • 87.88% accuracy on GAIA validation and 79.40% accuracy on the test set demonstrate strong long-horizon tool-use performance.Memory was accumulated from validation trajectories before evaluating the test set.
  • 24.4% PM on HLE ranks Memento second overall, within 0.92 points of GPT-5 at 25.32%.It also exceeds Gemini-2.5-Pro, o3-high, and o4-mini-high.
  • 95.0% accuracy on SimpleQA exceeds WebSailor (93.5%), WebDancer (90.5%), WebThinker (77.5%), and DeepSeek-r1-React (72.2%).

5.5. Ablation Studies

Ablation studies show that planning and case-based reasoning contribute complementary gains, while a small curated memory is more effective than retrieving many cases. Case-based memory also improves out-of-distribution performance, though continual-learning gains saturate quickly.

  • Case Retrieval: K = 4 yields the highest DeepResearcher F1 (64.5) and PM (78.5), while larger retrieval sizes plateau or slightly decline.Increasing K raises computational cost and can introduce irrelevant-example noise.
  • Component Ablations: Planning yields gains across HLE (+11.0 / +1.6), SimpleQA (+32.5 / +4.9), and DeepResearcher (+29.1 / +11.5) in F1/PM.
  • Component Ablations: Case-based reasoning provides consistent additive improvements across the evaluated benchmarks.
  • Continual Learning: The full Memento architecture consistently outperforms versions without CBR across continual-learning iterations.Removing CBR causes a noticeable performance decline.
  • Continual Learning: With about 3k training data, the Case Bank saturates quickly and additional iterations provide only marginal gains.
  • Generalisation across Tasks: 4.7% to 9.6% absolute gains occur across all evaluated OOD benchmarks when Memento retrieves four relevant training cases.

6. Discussion and Analysis

Memento’s efficiency and modular behavior depend on integrating planning, memory, and evidence aggregation rather than simply increasing tool calls or response length. Fast, concise planning outperforms more deliberative planning, while complex tasks mainly increase input-context costs.

  • Statistics of MCP Tools: Code, search, and crawl tasks increase notably with difficulty, while challenging problems rely increasingly on internal reasoning over prior tool outputs.
  • Token Costs: Level 1, Level 2, and Level 3 tasks require 26k/4.7k, 48k/6.9k, and 121k/9.8k input/output tokens, respectively.
  • Token Costs: Output tokens remain stable because final answers are typically short, whereas input context grows substantially with task difficulty.
  • Fast and Slow Think Mode: 70.9% average accuracy results from pairing GPT-4.1 as planner with o3 as executor, exceeding the 63.03% achieved with o3 as planner and the same executor.
  • Fast and Slow Think Mode: Concise, structured planning improves downstream execution, while overly deliberative planning introduces redundancy and role confusion.

7. Conclusion

Memento enables online adaptation of LLM agents without updating model weights by formalising deep research as a memory-based MDP. Experiments show strong benchmark performance, while ablations identify both CBR variants and curated memory as important.

  • Memento adapts LLM agents through an episodic case bank that records and retrieves trajectories for continual policy improvement without weight updates.
  • Memento achieves strong performance across the GAIA, DeepResearcher, and SimpleQA benchmarks.
  • Both parametric and non-parametric CBR are critical to the reported performance gains.
  • A small, curated memory yields optimal results in the reported ablation studies.

A. Derivation of the Optimal Policy in Soft-Q Learning

This section derives the optimal retrieval policy for soft-Q learning in the memory-augmented setting. It defines value and Q functions over states and case banks, then connects the objective to a softmax policy and standard Q-learning as α approaches zero.

  • The soft value function is defined over the state s within the memory-augmented policy formulation.
  • The Q function evaluates state–case pairs using expected rewards and discounted successor-state values under the LLM action distribution.
  • The discounted visitation frequency dπ(s, M) tracks state–case-bank occupancy under policy π across time.
  • The retrieval policy is derived by optimising expected value over case probabilities constrained to sum to one.
  • As α → 0, soft Q-learning reduces to standard Q-learning, while the policy takes a softmax form.

B. Analysis of Memory Mechanisms

The paper contrasts tabular, deep parametric, and episodic-memory mechanisms through their Read and Write operations. Memento combines non-parametric retrieval with parametric value learning, trading broad generalisation against the stability and data efficiency of episodic methods.

  • Table 7 compares memory mechanisms by their Read and Write operations.
  • Tabular memory: Tabular memory directly reads Q(s, M, a) and writes updated state–action entries after observed transitions.
  • Parametric memory: Deep Q-learning uses shared neural parameters to generalise across states and action spaces, but requires substantial data and can be unstable.Approximation errors may propagate globally through the parameter space.
  • Episodic memory: Memento’s non-parametric Read operation retrieves relevant cases by cosine similarity and TopK selection from an episodic case bank.The Write operation appends each observed case to the case bank.
  • Memento implementation: In the single-step deep research setting, a parametric Q-function reduces data requirements because temporal-difference bootstrapping vanishes.The binary reward further simplifies the learning objective.
Loading 2508.16153v2…