PaperlayerView exact revision on arXiv

Source-linked AI summary

MemSifter: Offloading LLM Memory Retrieval via Outcome-Driven Proxy Reasoning

Jiejun Tan, Zhicheng Dou, Liancheng Zhang, Yuyang Hu, Yiruo Cheng, Ji-Rong Wen

arXiv:2603.03379v2cs.IRcs.AI

TL;DR

Long-term LLM memory must balance retrieval quality against indexing and inference cost, while existing retrieval objectives may not reflect downstream task utility. MemSifter uses a lightweight proxy trained with outcome-oriented reinforcement learning to select memories for the working LLM, and it surpasses state-of-the-art baselines across eight memory benchmarks in retrieval accuracy and task performance.

  • Problem

    Long-term LLM memory faces a cost–accuracy trade-off, and retrieval objectives may not match the downstream utility of memories in dynamic, context-dependent tasks.

  • Method

    MemSifter offloads task-aware memory retrieval to a lightweight proxy trained by reinforcement learning from the working LLM’s final task success.

  • Results

    MemSifter surpasses prior state-of-the-art methods in retrieval accuracy and end-to-end task performance across eight LLM memory datasets.

  • Takeaways & Limitations

    MemSifter provides a scalable paradigm in which small specialized models curate context for large generalist reasoners.

Abstract

from arXiv · show

As Large Language Models (LLMs) are increasingly used for long-duration tasks, maintaining effective long-term memory has become a critical challenge. Current methods often face a trade-off between cost and accuracy. Simple storage methods often fail to retrieve relevant information, while complex indexing methods (such as memory graphs) require heavy computation and can cause information loss. Furthermore, relying on the working LLM to process all memories is computationally expensive and slow. To address these limitations, we propose MemSifter, a novel framework that offloads the memory retrieval process to a small-scale proxy model. Instead of increasing the burden on the primary working LLM, MemSifter uses a smaller model to reason about the task before retrieving the necessary information. This approach requires no heavy computation during the indexing phase and adds minimal overhead during inference. To optimize the proxy model, we introduce a memory-specific Reinforcement Learning (RL) training paradigm. We design a task-outcome-oriented reward based on the working LLM's actual performance in completing the task. The reward measures the actual contribution of retrieved memories by mutiple interactions with the working LLM, and discriminates retrieved rankings by stepped decreasing contributions. Additionally, we employ training techniques such as Curriculum Learning and Model Merging to improve performance. We evaluated MemSifter on eight LLM memory benchmarks, including Deep Research tasks. The results demonstrate that our method meets or exceeds the performance of existing state-of-the-art approaches in both retrieval accuracy and final task completion. MemSifter offers an efficient and scalable solution for long-term LLM memory. We have open-sourced the model weights, code, and training data to support further research.

1 Introduction

MemSifter addresses the cost–accuracy trade-off in long-term LLM memory by moving retrieval reasoning to a lightweight proxy. Its outcome-oriented RL training evaluates memories by their contribution to the working LLM’s task success.

  • Long-duration tasks exceed LLM context windows, requiring persistent storage for long-term interaction history.
  • Vanilla memory has low retrieval accuracy, while structural indexing improves retrieval but adds computation and can discard important details.Most stored memories are never reused, making upfront indexing costs potentially wasteful.
  • MemSifter uses a lightweight proxy to analyze the task and select critical evidence before the working LLM receives only the top-k memory segments.This avoids complex indexes and reduces the working LLM’s retrieval burden.
  • The proxy is trained with reinforcement learning that aligns retrieval optimization with the working LLM’s downstream task success rather than isolated retrieval metrics.The approach is designed around goal alignment and the scarcity of fine-grained labels.
  • Marginal Utility Reward measures performance lift over a no-memory baseline, while Rank-Sensitive Reward evaluates retrieval across top-k cutoffs with diminishing weights.
  • Across eight LLM memory datasets, MemSifter surpasses prior state-of-the-art methods in retrieval accuracy and end-to-end task performance.The benchmarks span personal memory and long-horizon research tasks.

2 Related Work

Prior LLM memory research spans token-level, parametric, and latent mechanisms, while reasoning-enhanced retrieval improves query-time selection at substantial computational cost. MemSifter combines inference-time reasoning with outcome-driven optimization in a lightweight proxy specialized for dynamic episodic memory.

  • 2.1 Memory Mechanisms for LLMs: LLM memory research commonly distinguishes token-level, parametric, and latent memory mechanisms.Token-level systems store external units, whereas parametric methods encode knowledge in model parameters.
  • 2.1 Memory Mechanisms for LLMs: Token-level systems provide explicit storage but graph-based and related architectures can incur high indexing costs and adapt poorly to task-specific retrieval.
  • 2.2 Inference-Time Retrieval Scaling: Reasoning-before-retrieval methods add structured rationales or latent thinking to address the semantic limits of static embedding retrieval.
  • 2.2 Inference-Time Retrieval Scaling: These reasoning-enhanced approaches are computationally heavy because they often invoke large models also used for the main task.
  • 2.2 Inference-Time Retrieval Scaling: MemSifter distills reasoning-enhanced retrieval and listwise evaluation into a specialized lightweight proxy, preserving reasoning benefits with minimal overhead.
  • 2.3 End-to-End Retrieval Optimization: Retrieval objectives can mismatch downstream generation needs because LLM memory involves dynamic episodic histories, context-dependent utility, and sparse final outcomes.
  • 2.3 End-to-End Retrieval Optimization: MemSifter optimizes the proxy directly from the working LLM’s final success rather than intermediate relevance labels or answer likelihoods.The method targets reasoning utility without human annotation or heavy computation.

3 Methodology

MemSifter formulates memory retrieval as proxy-guided selection from persistent interaction history, then trains the proxy using task outcomes and rank-sensitive rewards. Its protocol combines progressive evaluation, efficient sampling, and curriculum construction to prioritize useful evidence while limiting computation.

  • 3.1 Problem Formulation: The memory task stores interaction history as sessions and retrieves task-relevant segments for the working LLM when processing the full history is impractical.The proxy selects a subset from persistent history for the current task, after which the working LLM generates the response.
  • 3.2 Memory Proxy Reasoning: MemSifter segments history into sessions, optionally prefilters oversized histories with embeddings, and uses Think-and-Rank to output a fixed top-10 list.Retrieved sessions are concatenated with the query and passed to the working LLM; the prefilter keeps top sessions under the proxy context budget.
  • 3.3 Task-Outcome-Oriented Reward: Task-outcome rewards compare working-LLM performance with and without retrieved memory, then evaluate increasing context sizes to measure progressive utility.The baseline score is s_0, the retrieved-context score is s_k, and memory utility is quantified by s_k−s_0 across K = {k_1,...,k_N}.
  • 3.3.2 Rank-Sensitive Reward: Fibonacci sampling reduces rank-evaluation inference calls from linear to logarithmic in list size while retaining key performance inflection points near the top.Training uses Qwen3-30B-A3B as an efficient working LLM to accelerate reward computation.
  • 3.3.2 Rank-Sensitive Reward: Rank-sensitive reward assigns greater credit to useful evidence at earlier ranks through diminishing DCG-style weights.The design applies logarithmic decay to rank position and aggregates progressive performance gains across sampled cutoffs.
  • 3.4 Optimization Protocol and Curriculum: Dynamic curriculum construction selects tasks near an anchor score of 0.2 and maintains appropriate rollout difficulty during iterative training.The strategy targets emerging capabilities, avoiding overfitting to easy samples or collapse on impossible ones.

4 Experiments

The experiments evaluate MemSifter across personal-memory and deep-research benchmarks using sampled test questions and end-to-end generation metrics. Comparisons cover embedding retrievers, LLM memory frameworks, and other state-of-the-art baseline categories.

  • 4 Experiments: The evaluation uses eight datasets: five personal-memory benchmarks and three deep-research benchmarks.The personal-memory group includes LoCoMo, LongMemEval, PersonaMem, PerM-V2, and ZH4O; the deep-research group includes HotpotQA, WebWalker, and another listed research dataset.
  • 4 Experiments: LoCoMo tests very long-term conversational memory with multimodal dialogues averaging 300 turns, including factual and temporal understanding.The benchmark contains 10 multimodal dialogues.
  • 4 Experiments: Evaluation samples 400 test questions from several personal-memory datasets and 150 from LongMemEval.The paper reports end-to-end generation F1 as the unified metric across benchmarks because methods produce free-form answers from retrieved context.
  • 4 Experiments: Baselines span embedding-based retrieval models and LLM memory frameworks, including BGE-M3, EmbeddingGemma, Mem0, and Nemori.The comparison therefore covers both lightweight retrieval models and systems that manage persistent memory units or interaction history.

4.3 Main Results

Across eight memory datasets, MemSifter is reported to outperform prior methods in retrieval precision and end-to-end task performance while using a lightweight retrieval architecture. It also offers an alternative to complex indexing and full-history processing by selecting critical information before the working LLM responds.

  • MemSifter is evaluated against baseline methods on eight LLM memory datasets using end-to-end generation F1 as the main metric.The evaluation also tests transfer across two working LLMs.
  • Compared with standard embedding retrieval, MemSifter filters irrelevant noise and provides the working LLM with cleaner context, improving final task success rates.
  • MemSifter outperforms Think-then-Rerank baselines by optimizing retrieved memories for downstream task utility rather than semantic relevance alone.
  • MemSifter surpasses prior state-of-the-art methods in both memory retrieval accuracy and end-to-end task performance.The main-results table reports F1 scores, with the best and second-best results marked for each model.
  • Unlike graph-based or generative memory pipelines, MemSifter achieves state-of-the-art performance with a lightweight architecture and lower indexing overhead.
  • Compared with feeding full history to long-context LLMs, MemSifter reduces computational costs while maintaining or exceeding performance.It selects and reasons about critical information before the working LLM processes it.

4.4 Ablation Study

The ablation study examines task-outcome optimization, rank-sensitive weighting, and explicit reasoning before retrieval. Removing these components degrades performance by weakening credit assignment, ignoring evidence position, or isolating the role of proxy reasoning.

  • Removing task-outcome reward causes a large performance drop when the proxy is trained only with retrieval-metric supervision.Static relevance supervision can reward retrieval on tasks already solved by parametric knowledge.
  • Uniformly averaging top-k evaluations degrades performance because it ignores evidence position and fails to reward placing key information early.
  • The controlled Direct Rank setting isolates the contribution of explicit thinking before retrieval.Its result is included in Table 3.

4.5 Further Analysis

Further analysis finds that MemSifter improves retrieval quality and training outcomes, while offering a cost-performance trade-off against embedding, reasoning-based, and long-context approaches.

  • 4.5.1 Retrieval Quality Analysis: MemSifter consistently achieves higher recall and ranking precision than embedding-based, hierarchical, and reasoning-heavy retrieval baselines.The intrinsic retrieval analysis is restricted to benchmarks with gold ranking labels.
  • 4.5.2 Training Dynamics and Curriculum Analysis: MemSifter converges faster and reaches higher final performance than Rearank on LoCoMo and LongMemEval.The comparison uses RL trajectories with Rearank as the baseline.
  • 4.5.2 Training Dynamics and Curriculum Analysis: The baseline plateaus or degrades after 60–70 epochs, whereas MemSifter continues improving across three training phases.MemSifter refreshes training data near the difficulty anchor and uses model merging.
  • 4.5.3 Efficiency Analysis: MemSifter adds moderate latency over lightweight embeddings but offers higher retrieval precision and better task performance than similarly costly reason-then-rerank methods.Against long-context LLMs over full histories, the supplied passage introduces the comparison but does not state its completed outcome.

5 Conclusion and Future Work

MemSifter addresses the precision-efficiency trade-off in LLM memory retrieval by using a lightweight proxy trained from task outcomes. Experiments report stronger task accuracy and inference efficiency than state-of-the-art baselines, while future work targets memory consolidation and multimodal histories.

  • Conclusion: MemSifter separates memory management from core reasoning by offloading historical-interaction sifting to a lightweight proxy.Its reward combines marginal utility and rank sensitivity.
  • Conclusion: The proxy learns retrieval strategies from final task success alone, avoiding costly relevance annotations.The training objective uses marginal utility and rank sensitivity.
  • Conclusion: MemSifter surpasses state-of-the-art baselines in task accuracy and inference efficiency.The conclusion presents this as the reported experimental outcome.
  • Future Work: Future work will extend outcome-driven optimization to LLM memory consolidation and multimodal histories.

A Experimental Details

The experiments use disjoint benchmark splits, session-level memory construction, optional coarse filtering for histories exceeding the proxy window, and fixed top-10 retrieval. The proxy is trained with one working LLM and evaluated with two to test transfer.

  • Session Construction: Session-level retrieval preserves full selected sessions and is cheaper than turn-level retrieval, but noisy sessions may retain irrelevant turns.Turn-level or hierarchical retrieval is left for future work.
  • Coarse Filtering: BGE-M3 performs coarse filtering only when histories exceed the proxy’s 128K-token window, retaining sessions by cosine similarity until the token budget is reached.No fixed similarity threshold is used.
  • Top-k Setting: The proxy outputs a fixed top-10 ranked list, and each method gives the working LLM the same top-k budget for stable comparisons.Top-k sensitivity is separately used to examine cost-performance trade-offs.
  • Cross-working-LLM Setting: The proxy is optimized with Qwen3-30B-A3B-Instruct feedback and evaluated with both Qwen3-30B-A3B-Instruct and DeepSeek V3.2.This setting tests transfer across working LLMs.
  • Training Cost Accounting: All three training rounds run on one 8×H200 node, with costs reported using GPU hours, working-LLM calls, tokens, and API charges.The accounting also includes provider-bill estimates for API token costs.
  • Data Preparation: Training and test sets are strictly disjoint across the experimental data split.
  • Ablations: The direct-rank ablation removes the proxy’s thinking output while keeping the base model, data split, top-k budget, working LLM, and metric unchanged.It reports F1, output tokens, and latency across LoCoMo-32K, LongMemEval-1M, and WebDancer-128K.

B Case Study

The case studies provide qualitative views of MemSifter’s reasoning and selected ranked sessions, while the supplied tables cover top-k sensitivity and PersonaMem accuracy.

  • Case Study: Case studies on LoCoMo and WebDancer show the proxy’s reasoning process and the ranked sessions selected for the working LLM.
  • Additional Analysis: The top-k sensitivity table reports estimated memory/session tokens introduced by selection, excluding the current query and fixed task prompt.
  • Additional Analysis: PersonaMem accuracy is computed by normalizing generated answers to predicted options before exact-match scoring.

C Deep Research Dataset Construction

The Deep Research benchmark was constructed from MiroVerse search trajectories and reasoning traces, then made more difficult through semantic distractors and longer contexts. It stress-tests whether memory systems can retrieve precise evidence from noisy, multi-step search logs.

  • The benchmark samples core search trajectories and reasoning traces from the MiroVerse dataset.
  • Semantic distractors add related but factually irrelevant or misleading search results, requiring logical-consistency-based discrimination rather than keyword matching.
  • Context extension concatenates multiple search iterations and intermediate reasoning steps to create significantly longer context windows.
  • The resulting environment evaluates retrieval-augmented generation under massive volumes of noisy search logs and reasoning chains.The target is extracting needle-in-a-haystack evidence needed for a correct final answer.
Loading 2603.03379v2…