PaperlayerView exact revision on arXiv

Source-linked AI summary

Memory Intelligence Agent

Jingyang Qiao, Weicheng Meng, Yu Cheng, Zhihang Lin, Zhizhong Zhang, Xin Tan, Jingyu Gong, Kun Shao, Yuan Xie

arXiv:2604.04503v4cs.AIcs.MA

TL;DR

Deep research agents need memory that reuses search strategies and historical trajectories, while prior systems face planning, execution, and storage limitations. MIA combines a Manager-Planner-Executor architecture with alternating reinforcement learning, online test-time memory updates, and reflection. Across the paper’s reported evaluations, MIA improves frontier and smaller Executors and supports self-evolution, while its test-time self-evolution process depends on strong supervision for stable optimization.

  • Problem

    Deep research requires process-oriented memory for planning and strategy reuse, but prior systems lack task-specific Planner and Executor training and incur memory-management limitations.

  • Method

    MIA separates historical, parametric, and execution functions through a Manager-Planner-Executor architecture, alternating reinforcement learning, continual test-time learning, and reflection.

  • Results

    MIA surpasses existing memory baselines and significantly improves frontier and small-scale LLMs in deep research tasks.

  • Takeaways & Limitations

    MIA compresses historical trajectories into structured workflows, consolidates knowledge into parametric memory, and supports self-evolution during exploration.

  • Takeaways & Limitations

    Test-time self-evolution is effectively implemented only when strong supervision, such as ground-truth answers, is available.

Abstract

from arXiv · show

Deep research agents (DRAs) integrate LLM reasoning with external tools. Memory systems enable DRAs to leverage historical experiences, which are essential for efficient reasoning and autonomous evolution. Existing methods rely on retrieving similar trajectories from memory to aid reasoning, while suffering from key limitations of ineffective memory evolution and increasing storage and retrieval costs. To address these problems, we propose a novel Memory Intelligence Agent (MIA) framework, consisting of a Manager-Planner-Executor architecture. Memory Manager is a non-parametric memory system that can store compressed historical search trajectories. Planner is a parametric memory agent that can produce search plans for questions. Executor is another agent that can search and analyze information guided by the search plan. To build the MIA framework, we first adopt an alternating reinforcement learning paradigm to enhance cooperation between the Planner and the Executor. Furthermore, we enable the Planner to continuously evolve during test-time learning, with updates performed on-the-fly alongside inference without interrupting the reasoning process. Additionally, we establish a bidirectional conversion loop between parametric and non-parametric memories to achieve efficient memory evolution. Finally, we incorporate a reflection and an unsupervised judgment mechanisms to boost reasoning and self-evolution in the open world. Extensive experiments across eleven benchmarks demonstrate the superiority of MIA.

1 Introduction

Deep research agents need memory that reuses search strategies and trajectories, but prior systems face weaknesses in planning, example selection, and plan execution. MIA addresses these issues with a Manager-Planner-Executor architecture, alternating reinforcement learning, continual test-time learning, and reflection mechanisms.

  • Deep research agents combine LLM reasoning with external tools for complex, open-ended tasks, creating a need for memory across long-horizon interactions.
  • Effective deep-research memory must preserve how results are obtained, including search trajectories, failed attempts, and successful reasoning strategies.
  • Prior systems suffer from task-insufficient Planner and Executor training and retrieve few-shot examples mainly by relevance, neglecting quality and frequency.
  • MIA uses a Manager-Planner-Executor architecture that separates historical memory, parametric planning, and dynamic execution.
  • Alternating reinforcement learning aligns high-level planning with low-level retrieval by optimizing Planner–Executor interplay.
  • Continual test-time learning updates the Planner during inference, while reflection and unsupervised judgment support self-assessment, correction, and continual evolution on unknown tasks.
  • MIA surpasses existing memory baselines and improves both frontier and small-scale LLMs in deep research tasks.

2 Related Work

Related work improves deep research through reinforcement learning and memory management, but existing approaches remain specialized across task types and memory functions. MIA is situated among methods for search, reasoning, memory storage, and memory evolution.

  • Deep research methods use reinforcement learning to improve multi-turn search and retrieval-augmented generation, but the cited approaches are confined to text-only tasks.
  • Agent memory systems improve reasoning and decision-making through scalable memory, successful and failed experiences, reinforcement-learned storage strategies, and contextual scheduling.
  • Related memory-management methods include long-term and short-term memory and graph-based paradigms for contextual scheduling in complex tasks.
  • Memory-evolution methods drive dynamic adjustment through meta-feedback or memory fine-tuning while freezing the underlying language model.

3 Method

MIA combines memory retrieval, collaborative planning and execution, and experience consolidation in a Manager-Planner-Executor loop. It supports lifelong evolution by jointly updating explicit workflows and Planner parameters during test-time exploration, while adding reflection and unsupervised evaluation for settings without reliable labels.

  • Architecture and reasoning: MIA uses a Memory Manager, Planner, and Executor to retrieve historical trajectories, formulate plans, execute them with tools, and consolidate new experiences.The Memory Manager stores high-value trajectories, the Planner generates search plans, and the Executor follows plans while providing feedback for reflection and replanning.
  • Non-parametric memory: Hybrid retrieval ranks memories by semantic similarity, value reward, and frequency reward, then supplies both successful trajectories and failed trajectories as planning context.This combines contextual relevance, high-success experiences, and low-frequency memories that may contain long-tail knowledge.
  • Collaborative reasoning: The Executor interacts with the environment through a ReAct loop, while Planner feedback can trigger one Reflect-Replan cycle when execution encounters an impasse or unexpected feedback.The revised plan is passed back to the Executor to extend the trajectory.
  • Experience consolidation: MIA compresses images into captions and verbose trajectories into structured workflow summaries, reducing storage and retrieval burdens before replacing highly similar memory units.A Qwen3-32B language model performs the compression and organization of new memory.
  • Alternating reinforcement learning: Two-stage alternating GRPO training first trains the Executor to follow Planner-generated plans, then trains the Planner to absorb memory, plan complex questions, and reflect on Executor feedback.The Planner is frozen during Executor training, while the Executor is used as a trained component during Planner training.
  • Test-time learning: During test-time learning, MIA simultaneously extracts non-parametric workflows and updates the Planner’s parameters, forming an uninterrupted online self-evolution loop.Exploration feedback is internalized into parameters while the external memory base is enriched, but the described pipeline is most effective when ground-truth answers provide labels and rewards.
  • Unsupervised judgment: For unsupervised evaluation, MIA decomposes judgment into orthogonal Logic, Format, and Factuality dimensions to reduce error bleeding from single-prompt judging.The framework is modeled on the Reviewer-Area Chair decision process and synthesizes multi-perspective evidence rather than relying only on a scalar score.

4 Experiments

Experiments evaluate MIA across multimodal and text-only deep research tasks, showing strong performance, scalability, dataset adaptation, and benefits from its memory, reflection, alternating-RL, and test-time-learning components.

  • Main Results: MIA achieves the highest overall open-source performance, reaching 53.6 average accuracy and improving 5.5 over the previous best memory-based method.Gains include 3.3 on FVQA-test, 6.4 on LiveVQA, and 9.1 on In-house 1.
  • Main Results: MIA approaches or surpasses giant closed-source LLMs and remains competitive with specialized search agents using only basic text and image search tools.The results attribute this performance to leveraging memory and internalizing historical experiences rather than sophisticated external tools.
  • Text-only Evaluation: MIA consistently outperforms alternative methods across text-only datasets, reaching 53.5 average accuracy and improving 7.5 over Memento.The gains include 7.6 on 2Wiki and 8.8 on GAIA.
  • Training Analysis: Reinforcement learning makes Planner and Executor rewards trend upward while capturing dataset-specific response patterns during test-time learning.Responses become progressively shorter on 2Wiki and longer on LiveVQA as training proceeds.
  • Ablation Study: Ablations show that guiding planning with memory improves average accuracy by 3.5 on multimodal and 4.15 on text-only tasks, while reflection and alternating RL add further gains.Reflection adds 0.43 and 3.28; alternating RL adds 2.37 and 1.72 on multimodal and text-only tasks, respectively.
  • Ablation Study: Incrementally integrating MIA components increases average accuracy by 8.94 on multimodal benchmarks and 12.38 on text-only benchmarks.Unsupervised MIA is comparable to supervised baselines on multimodal benchmarks and surpasses almost all supervised baselines on text-only benchmarks.

5 Conclusion

MIA combines compressed historical trajectories, parametric planning, Executor training, online test-time learning, reflection, and unsupervised judgment to improve deep research reasoning and self-evolution. The framework achieves state-of-the-art performance across multimodal and text-only benchmarks while remaining focused on deep research tasks.

  • Conclusion: MIA compresses historical trajectories into structured workflows, reducing long-context noise and improving memory retrieval precision and quality.
  • Conclusion: Planner training converts non-parametric memory into parametric memory, reducing storage burden and improving planning performance.
  • Conclusion: Alternating reinforcement learning improves Planner plan generation and reflection while strengthening Executor plan understanding and following.
  • Conclusion: Online test-time learning updates the Planner during exploration while extracting positive and negative paradigms for explicit contrastive learning.
  • Conclusion: MIA achieves state-of-the-art performance across multimodal and text-only deep research benchmarks, with future work targeting more complex and dynamic environments.

6 Contribution

The paper’s contribution is the development of MIA, a framework intended to enhance deep research agents’ reasoning performance and self-evolution ability.

  • Contribution: MIA is proposed as a memory framework for improving reasoning performance and self-evolution in Deep Research Agents.
  • Contribution: The core methodology includes experimental protocol design and manuscript development led by Jingyang Qiao.
  • Contribution: Algorithm implementation, experiments, manuscript writing, skill implementations, supervision, and critical revisions were distributed among the listed authors.

A Training Details

Training uses offline text and image retrieval infrastructure, veRL on eight GPUs, and separate GRPO-based Executor and Planner configurations.

  • Retrieval Infrastructure: The text retriever uses E5-base-v2 embeddings with a FAISS index over local wiki25 and returns the top-3 passages per query.
  • Training Infrastructure: Reinforcement-learning training is implemented with veRL on 8 GPUs, using cached or offline retrieval resources for external tools.
  • Executor Training: Executor training initializes from Qwen2.5-VL-7B-Instruct and uses FVQA-train with GRPO, multi-turn tool interaction, and up to 10 assistant and 10 user turns.
  • Planner Training: Planner training initializes from Qwen3-8B, uses image-discarded FVQA-train plus MATPO, and applies GRPO in a tool-free setting.

B Test Details

Evaluation uses online and local search tools, deterministic inference, and distinct test-time learning settings for supervised and unsupervised Planner initialization.

  • Online text-to-text search uses Serper to return the top-5 retrieved results, while text search otherwise uses the local wiki25 retriever depending on the benchmark.
  • Multimodal evaluation uses Serper for image-to-image search and either wiki25 or Serper for text-to-text search.
  • Inference uses vLLM with temperature set to 0.
  • Test-Time Learning: Test-time learning uses 1 epoch, a learning rate of 1 × 10−6, and 4 rollouts per sample.
  • Planner Initialization: The supervised Planner starts from the trained Planner, whereas the unsupervised Planner starts from Qwen3-8B.

C Memory-Based Baseline Details

Memory-based baselines are compared with matched Executor checkpoints under three memory-prompt formats: no memory, retrieved context, and abstracted guidance.

  • Three Executor variants keep other training settings identical while differing in how memory is incorporated.
  • No Memory: The no-extra-prompt variant provides no additional memory input and serves as the No Memory baseline.
  • Long-Context Memory: The long-context prompt prepends retrieved memory context and is used for RAG, Mem0, and A-Mem.
  • Guideline Memory: The guideline prompt prepends high-level guidance generated from memory and is used for ReasoningBank, ExpeL, Memento, and MIA.
  • Because multimodal inputs make Memento’s parametric retrieval optimization difficult to apply, the comparison uses only its non-parametric version.
  • Each method uses its corresponding prompt template with the matched Executor checkpoint trained under the same setting.

D Memory Retrieval

MIA retrieves historical trajectories using a unified embedding representation and a score combining semantic relevance, historical success, and usage frequency.

  • Retrieval Score: The final score combines semantic similarity, empirical success ratio, and a low-frequency reward, with λs = 0.7 and λv = λf = 0.3.
  • The Memory Manager organizes entries by modality and question category, storing trajectories with questions, captions, judgment labels, and retrieval statistics.
  • Embedding: Textual questions and image captions are embedded with sup-simcse-bert-base-uncased using mean pooling followed by L2 normalization.
  • Semantic Similarity: For each memory, retrieval computes question-level and caption-level cosine similarities, using caption information when available.
  • Semantic Similarity: With an image caption, semantic similarity weights question similarity at 0.8 and caption similarity at 0.2; without one, only question similarity is used.
  • Normalization: Semantic similarity is min-max normalized within the current memory bucket with a 10−8 stabilizing term.

E Dataset Settings

The evaluation covers multimodal and text-only datasets, including established benchmarks and newly constructed scientific and multi-hop visual-question-answering collections.

  • Experiments evaluate the framework on both multimodal and text-only datasets, with the complete dataset configuration summarized in Table 8.
  • Multimodal Datasets: FVQA-train contains 4,856 multimodal examples and trains both the Executor and Planner, while FVQA-test contains 1,800 held-out examples.
  • Multimodal Datasets: The InfoSeek evaluation subset contains 2,000 multimodal examples requiring factual knowledge beyond direct visual perception.
  • Multimodal Datasets: The public LiveVQA version contains 2,384 multimodal examples, whereas the previously reported 3,602-example version is no longer publicly accessible.
  • Multimodal Datasets: SimpleVQA uses 1,013 English multimodal examples, and MMSearch uses 171 visual-question examples for evaluation.
  • In-House Datasets: The in-house scientific QA pipeline crawls cross-source evidence and synthesizes grounded question-answer pairs across Physics, Chemistry, and Biology.
  • In-House Datasets: The in-house multi-hop VQA pipeline uses real-time news images, visual entities, and triple dependency chains, producing 505 examples.
  • Text-Only Datasets: Text-only evaluation includes MATPO training data with 6,175 examples, 2Wiki with 12,576, HotpotQA with 7,405, SimpleQA with 4,327, and GAIA-Text with 103.

F Algorithm

The algorithms define Executor and Planner training rollouts, followed by test-time planning, execution, evaluation, replanning, and memory updates. They also specify prompt templates and evaluator roles for guiding tool use, reflection, and final-answer assessment.

  • Executor Training: The Executor rollout takes a question, image, pretrained Planner, tool set, and policy, then executes tools, updates context, and produces a reward.Replanning can be triggered during execution, and the reward is computed according to the specified equation.
  • Planner Training: The Planner rollout constructs context from memory, the question, and a prompt template before generating a plan and guiding Executor interactions.The Planner analyzes the first trajectory and result, decides whether to reflect and replan, then continues with a revised plan and computes a reward.
  • Test-Time Learning: During test-time learning, the Planner generates multiple reasoning-plan pairs, while the Memory Manager retrieves examples and selects the best plan for execution.The selected plan is executed with the question, image, tools, and frozen Executor to obtain a trajectory and final response.
  • Test-Time Learning: The test-time process evaluates rollout outcomes, partitions them into successful and failed sets, and computes normalized rewards for Planner updates.An LLM Judger evaluates correctness, after which reward mean and standard deviation are computed before parameter updates.
  • Memory Updates: The process stores successful and failed rollout summaries in Workflow Memory and records failed-successful contrastive pairs in Meta Plan Memory.The shortest successful rollout is selected, while one failed rollout is sampled; the Planner is then updated according to the specified equation.
  • Prompt Templates: The prompt templates instruct the Planner to plan, evaluate execution traces, replan when needed, and use search tools and evaluator feedback.Separate evaluators assess reasoning consistency, information sourcing and credibility, result validity, and final-answer correctness.
Loading 2604.04503v4…