PaperlayerView exact revision on arXiv

Source-linked AI summary

MemPilot: Orchestrating On-Demand Multimodal Memory Curation for LLM Agents

Haozhen Zhang, Haodong Yue, Quanyu Long, Jianzhu Bao, Qingyuan Liu, Tao Feng, Bohan Liu, Weida Liang, Wenya Wang

arXiv:2610.06830v1cs.CLcs.AIcs.LG

TL;DR

Query-agnostic memory construction may be wasteful and lossy, and flexible runtime control across multimodal evidence and performance, cost, and latency remains underexplored. MemPilot learns a multi-step policy to orchestrate retrieval and query-specific curation; across five benchmarks, it reports favorable trade-offs and broader frontiers than existing trade-off-aware baselines.

  • Problem

    Most agent-memory systems construct memories without knowing future queries, while flexible runtime adaptation across multimodal evidence, performance, cost, and latency remains underexplored.

    Source: §1 · pp. 1–2

  • Method

    MemPilot uses reinforcement learning to train a multi-step policy that chooses memory retrieval or query-specific raw-history curation and controls evidence, instructions, model selection, and visual access.

    Source: §1 · p. 2

  • Results

    Across five multimodal agent-memory benchmarks, MemPilot achieves favorable performance–cost–latency trade-offs and broader frontiers than existing trade-off-aware baselines.

    Source: §1 · p. 2, §5 · p. 11

  • Takeaways & Limitations

    The results support resource-aware runtime adaptation that moves beyond performance-only optimization within the evaluated multimodal agent-memory benchmarks.

    Source: §5 · p. 11

    Training uses a proxy latency estimator to avoid instability from transient network and serving fluctuations.

    Source: D. Limitations and Future Work · p. 26

Abstract

from arXiv · show

Memory has become integral to the LLM agent ecosystem, supporting information retention and reuse across interactions. However, most existing agent memory systems construct memory in a query-agnostic manner, which can incur unnecessary preprocessing cost and discard details that later prove essential. Recent studies have begun shifting memory processing toward runtime adaptation, but typically specialize in particular operations or fixed processing schemes, leaving flexible control over performance, cost, and latency largely underexplored. To address this challenge, we present \textbf{MemPilot}, a flexible framework that orchestrates on-demand memory curation under different performance--cost--latency preferences. Specifically, we optimize a multi-step LLM policy via reinforcement learning to iteratively choose between retrieving from query-agnostic memory and delegating query-specific curation of raw multimodal history to heterogeneous LLMs and VLMs. The policy jointly controls evidence amount, curation instructions, model selection, and visual access, enabling fine-grained allocation of runtime computation. To optimize this policy under competing objectives, we adapt objective-wise advantage decoupling by separately estimating each objective's advantage before aggregation. Moreover, we introduce prefix-based marginal utility estimation for fine-grained credit assignment across multi-step rollouts. Experiments on five multimodal agent-memory benchmarks demonstrate favorable performance--cost--latency trade-offs across optimization preferences, with preference sweeps yielding broader frontiers than existing trade-off-aware baselines.

1. Introduction

Existing memory systems often prepare query-agnostic memories, while flexible runtime orchestration across multimodal evidence and performance, cost, and latency remains underexplored. MemPilot learns a multi-step policy for this orchestration and reports favorable trade-offs across five benchmarks.

Motivation

  • Query-agnostic memory construction can be wasteful and lossy because future query needs cannot be reliably anticipated, while runtime orchestration remains underexplored.Existing runtime approaches often use rigid operations, incomplete trade-off objectives, or fixed backends.Source: §1 · pp. 1–2

Approach

  • MemPilot retains query-agnostic memory and raw multimodal history, then uses a multi-step policy to choose retrieval or query-specific curation by heterogeneous LLMs and VLMs.The policy controls evidence amount, curation instructions, model choice, and visual access.Source: §1 · p. 2

Optimization

  • The policy is optimized with reinforcement learning using objective-wise advantage decoupling and prefix-based marginal utility estimation for multi-step credit assignment.The framework jointly addresses answer quality, cost, and latency preferences.Source: §1 · p. 2

Results

  • Across five multimodal agent-memory benchmarks, MemPilot achieves favorable performance–cost–latency trade-offs and broader frontiers than existing trade-off-aware baselines.Preference sweeps also show adaptive shifts in runtime computation as optimization preferences change.Source: §1 · p. 2

2. Related Work

Agent-memory research has extended from query-agnostic construction toward runtime adaptation, while LLM routing research has developed increasingly adaptive model selection. MemPilot combines these directions by learning runtime orchestration of heterogeneous multimodal curation under controllable trade-offs.

2.1. Agent Memory and Runtime Adaptation

  • Runtime memory methods adapt at query time, but typically specialize in one operation or use predefined processing patterns; MemPilot instead learns heterogeneous multimodal curation under controllable trade-offs.Prior examples include just-in-time research, query-aware extraction, dynamic memory guidance, and deferred construction.Source: §2.1 · p. 3

2.2. LLM Routing

  • LLM routing progresses from cascades and difficulty-based selection to preference learning, contrastive and graph-based methods, and multi-step reinforcement-learning orchestration.This work frames routing as a way to allocate heterogeneous model computation according to task demands and resource constraints.Source: §2.2 · p. 3

3. Methodology

MemPilot treats on-demand memory curation as a preference-conditioned, multi-step decision process. It combines compressed memory retrieval with raw-history curation, exposes factorized runtime controls, and optimizes quality–cost–latency trade-offs with fine-grained credit assignment.

3.1. On-Demand Memory Orchestration as a Multi-Step Decision Process

  • MemPilot maintains query-agnostic memory for efficient access alongside raw multimodal history that retains details absent from compressed representations.The memory bank can be initialized using an off-the-shelf memory system, leaving memory construction orthogonal to runtime orchestration.Source: §3.1 · pp. 3–4
  • At each step, an orchestrator conditions on the query and acquired evidence, then retrieves from compressed memory or curates raw history before eventually answering.Retrieval uses semantic similarity; curation delegates evidence processing to heterogeneous external LLMs and VLMs, while the orchestrator itself does not process raw image pixels.Source: §3.1 · p. 4

3.2. Factorized Memory Curation and Delegation

  • Factorized actions adapt retrieval query and depth, curation instruction, processing model, and whether original visual evidence is included.Separating what to retrieve from how to curate keeps retrieval evidence-focused and curation task-directed.Source: §3.2 · p. 5

3.3. Resource-Aware Reward Modeling

  • GRPO optimizes answer quality, monetary cost, and latency by normalizing each objective's advantage separately before preference-weighted aggregation.Prefix probing truncates later curation to estimate each stage's marginal answer-quality improvement, assigning that credit to the corresponding policy decisions while trajectory-level advantages retain resource penalties.Source: §3.3 · pp. 6–7

4. Experiment

Across five multimodal benchmarks, MemPilot offers controllable quality–cost regimes and a broader performance–cost–latency frontier. Its learned policy shifts runtime curation and model allocation as optimization preferences change.

4.1. Setups

  • Evaluation covers five multimodal agent-memory benchmarks, with MemEye and MemLens reserved for out-of-distribution testing.Answer quality is measured by F1 and LLM-judge score; cost and latency capture monetary expense and inference time.Source: §4.1 · p. 7, §4.2 · p. 8
  • The main comparison includes fixed-configuration memory systems and methods that explicitly expose performance–efficiency trade-offs.The trade-off-aware group includes LightMem, BudgetMem, and OmniSimpleMem.Source: §4.1 · p. 7

4.2. Implementation Details

  • The implementation uses Qwen3-4B-Instruct as the default orchestrator, chunks histories into 256-token segments, and retrieves at most five evidence items per step.Unless otherwise specified, the query-agnostic bank uses LLMLingua-2 at a 0.4 compression ratio.Source: §4.2 · p. 7

4.3. Main Results

  • MemPilot-Perf achieves the best L-J score across all five benchmarks, while MemPilot-Bal lowers cost and provides an intermediate quality–cost trade-off.MemPilot-Cost further reduces resource use with greater quality sacrifice.Source: §4.3 · pp. 8–9, Table 1 · p. 8

4.4. Exploring Performance–Cost–Latency Frontiers

  • MemPilot forms a broad performance–cost frontier that largely envelopes competing methods and spans lightweight to performance-oriented quality–latency configurations.Most baselines offer a narrow region or a single operating point; MemVerse has relatively low latency but substantially higher monetary cost.Source: §4.4 · p. 9, Figure 2 · p. 9

4.5. How Does MemPilot Allocate Runtime Computation?

  • As preferences shift toward lower cost or latency, MemPilot generally reduces policy steps, curation, image access, and output tokens while adapting evidence counts and model selection.Higher-cost settings rely more on stronger VLMs and LLMs, whereas lower-cost settings shift computation toward cheaper model groups.Source: §4.5 · p. 10, Figure 3 · p. 10

4.6. Discussion

  • Removing LLM/VLM delegation reduces average Judge score from 37.40 to 14.58, and removing objective-wise advantage decoupling reduces it to 21.94.Other ablations also degrade quality, while all ablated variants consume less inference cost.Source: §4.6 · p. 10, Table 2 · p. 10
  • Without retraining, MemPilot remains compatible with off-the-shelf memory banks, whose distinct performance–cost profiles persist under runtime orchestration.The default bank occupies the lower-cost regime, SimpleMem an intermediate profile, and A-MEM higher cost with stronger performance.Source: §4.6 · p. 11

5. Conclusion

The conclusion presents MemPilot as a framework for on-demand memory curation under controllable performance–cost–latency preferences. It reports favorable trade-off frontiers and adaptive computation across optimization preferences.

5. Conclusion

  • Figure 4 examines performance–cost frontiers under different memory-bank initializations.Source: Figure 4 · p. 11
  • Across multimodal agent-memory benchmarks, MemPilot achieves favorable trade-off frontiers while adapting computation to different optimization preferences.The authors frame this as resource-aware runtime adaptation.Source: §5 · p. 11
  • MemPilot combines query-agnostic memory with raw interaction history and delegates curation to heterogeneous LLMs and VLMs.Its runtime adaptation is formulated as a multi-step decision process.Source: §5 · p. 11

A.1. Detailed Reward Modeling

The appendix details how MemPilot models task, cost, and latency rewards, describes its heterogeneous model pool, and assigns additional credit to individual curation stages using prefix probes.

A.1. Detailed Reward Modeling

  • Cost accounting includes orchestrator generations and external-model calls, charges newly appended environment tokens incrementally, and uses exact API cost when available.Prices are specified in USD per million tokens; the initial orchestrator prompt is charged once.Source: §A.1 · p. 16
  • Latency uses an affine proxy for orchestrator generations, while external calls in parallel share a stage whose maximum latency is summed across sequential stages.The proxy represents fixed overhead and average per-token input and output contributions over the experimental operating range.Source: §A.1 · pp. 16–17
  • Format-invalid trajectories receive a fixed reward of −1, while valid trajectories receive their answer-quality score and retain independent cost and latency rewards.Validation checks the final answer block and action formatting and fields; missing required Retrieve or Curate actions also invalidate a trajectory.Source: §A.1 · p. 17

A.2. Heterogeneous Model Pool

  • The experimental model pool reports model-specific prices and fixed latency proxies, including image-input latency coefficients for VLMs.Source: Table 3 · p. 18, §A.2 · p. 18

A.3. Prefix-Based Marginal-Utility Credit Assignment

  • Prefix probes generate answers from each curation-stage prefix without further curation, and score changes estimate signed marginal utility for that stage.The pre-curation prefix supplies the baseline; probe computations are excluded from cost, latency, and delegated-model invocation accounting.Source: §A.3 · p. 18

A.4. Evaluation Details

The evaluation appendix specifies preprocessing, benchmark splits and filters, inference settings, and reporting procedures. It also points to the main-experiment hyperparameters.

A.4. Evaluation Details

  • Multimodal histories are serialized chronologically into approximately 256-token chunks, keeping dialogue turns and their attached images, identifiers, and captions together.Oversized turns remain a single chunk.Source: §A.4 · p. 19
  • In-distribution data are split by conversation, while MemEye and MEMLENS are out-of-distribution test sets excluded from policy training and model selection.The evaluation applies answer-refusal and benchmark-specific data filters.Source: §A.4 · pp. 19–20
  • At inference, each question uses one greedy deterministic trajectory, with up to six sequential curation rounds and two concurrent operations.Source: §A.4 · p. 20

A.5. Hyperparameters

B.1. Per-Benchmark Runtime Computation Allocation

Across five benchmarks, stronger cost or latency preferences generally reduce several forms of runtime computation. Model routing shifts toward lighter models under tighter cost preferences, though individual controls can vary non-monotonically.

B.1. Per-Benchmark Runtime Computation Allocation

  • As stronger cost or latency preferences are imposed, MemPilot generally reduces policy steps, Curate actions, visual access, and policy-generation tokens.The per-benchmark behavior is consistent across all five benchmarks.Source: §B.1 · pp. 21–22
  • Tighter cost preferences shift curation toward lighter models; at the lowest displayed cost point, all five benchmarks use LLMs below 70B exclusively.Evidence depth and image access need not change monotonically, indicating joint rather than uniform reallocation.Source: §B.1 · p. 22

B.2. Per-Benchmark Ablation Results

Across five benchmarks, the full MemPilot configuration achieves the highest Judge score, while each ablation reduces cost but also degrades quality.

B.2. Per-Benchmark Ablation Results

  • The full MemPilot configuration achieves the highest Judge score on all five benchmarks; every ablation lowers cost but causes pronounced quality degradation.The comparison reports absolute Judge scores and inference costs for each variant.Source: §B.2 · p. 22, Table 6 · p. 22

B.3. Memory-Bank Compatibility Across Benchmarks

MemPilot preserves performance–cost frontiers across different memory-bank initializations without retraining, while case studies illustrate adaptive retrieval and delegation choices.

B.3. Memory-Bank Compatibility Across Benchmarks

  • Across five benchmarks, SimpleMem and A-MEM preserve clear performance–cost frontiers without retraining, though their cost regimes and performance ceilings differ.The default bank generally offers more cost-efficient operating points; A-MEM often reaches a higher ceiling, while SimpleMem remains competitive on Mem-Gallery and H2HMem.Source: §B.3 · p. 22, Figure 6 · p. 24
  • Figure 5 summarizes runtime computation allocation across five benchmarks under varying cost and latency preferences.The benchmarks appear from top to bottom as Mem-Gallery, WorldMemArena, H2HMem, MemEye, and MemLens.Source: Figure 5 · p. 23

C. Case Study

  • Direct retrieval answers the picnic-mat image query from compressed memory using one retrieved item.The returned image ID matches the reference answer.Source: C. Case Study · p. 25
  • For counting dress options, text-only delegation processes retrieved raw memory and returns the reference count of five.The delegation uses three evidence items and does not include images.Source: C. Case Study · p. 25
  • When compressed retrieval lacks the selected dress’s image ID, expanded retrieval with image-enabled delegation identifies D8:IMG_007.The external model connects Lin Wei’s “butterfly sleeve #2” choice to that image.Source: C. Case Study · pp. 25–26

D. Limitations and Future Work

The paper identifies privacy, preference calibration, and latency modeling as areas for future work, and describes deployment motivations and benchmark-specific policy and evaluation prompts.

D. Limitations and Future Work

  • Runtime curation may send retrieved histories to external models, raising privacy concerns for sensitive memories; preference calibration and deployment-aware latency modeling also remain future-work areas.The authors suggest privacy-aware routing or local processing, easier mapping from deployment preferences to reward coefficients, and measured latency optimization in a stable sandbox.Source: D. Limitations and Future Work · p. 26

E. Broader Impact

  • The paper argues that memory deployments may favor different performance, cost, and latency operating points, especially when multimodal visual processing has different resource demands.It frames adaptive memory curation as computation allocation that may encourage integration with model routing and resource-aware decision making.Source: E. Broader Impact · p. 26

F.1. Policy Prompt

  • The policy iteratively chooses between compressed-memory retrieval and query-aware processing of raw chunks, selecting evidence, instructions, models, and image access as needed.The prompt notes retrieval is faster but may omit details; image pixels require an image-capable model with include_images enabled.Source: §F.1 · p. 27

F.2. Delegation Prompt

  • The delegation prompt gives a selected LLM or VLM an instruction and retrieved raw evidence, with optional image inputs when requested and supported.Delegated models are directed to report supported findings and identify unresolved parts when a task is only partially answerable.Source: §F.2 · p. 28

F.3. Question-Answering Prompt

  • QA prompts impose benchmark-specific contracts, including timestamp-based conflict resolution, multimodal reasoning categories, latest-visual precedence, and exact refusal responses.WorldMemArena uses timestamps and recent evidence; H2HMem lists question types; MemEye prioritizes newest visual evidence in conflicts; MemLens specifies an exact insufficient-information response.Source: §F.3 · p. 29

F.4. Answer-Judging Prompts

  • The cited evaluations use distinct scoring schemes: WorldMemArena scores Correct as 1 and Hallucination or Omission as 0, while another judge compares answers with ground truth on a 0-to-1 quarter-point scale.Source: §F.4 · p. 30

Related papers

Loading 2610.06830v1…