PaperlayerView exact revision on arXiv

Source-linked AI summary

MIRIX: Multi-Agent Memory System for LLM-Based Agents

Yu Wang, Xi Chen

arXiv:2507.07957v1cs.CLcs.AI

TL;DR

Existing memory-augmented agents are limited by flat or narrowly scoped storage, especially for personalized, long-term, and multimodal information. MIRIX addresses this with six structured memory types coordinated by multiple agents and evaluates the system on screenshots and conversations. It reports improved accuracy and storage efficiency on ScreenshotVQA and state-of-the-art performance on LOCOMO.

  • Problem

    Existing memory systems often rely on flat or narrowly scoped components, limiting representation and retrieval of diverse user-specific information, including multimodal inputs.

  • Method

    MIRIX combines six structured memory components with dedicated Memory Managers, a Meta Memory Manager, and Active Retrieval for routing and retrieving relevant memories.

  • Results

    MIRIX reports gains over baselines across multimodal ScreenshotVQA and textual LOCOMO evaluations, including 35% higher accuracy than RAG with 99.9% lower storage on ScreenshotVQA and 85.4% performance on LOCOMO.

  • Takeaways & Limitations

    The results position MIRIX as a memory system for persistent, structured, personalized, and multimodal information in LLM-based agents.

  • Takeaways & Limitations

    On LOCOMO open-domain questions, reliance on RAG retrieval may bottleneck performance because such questions require longer-term inference and global understanding.

Abstract

from arXiv · show

Although memory capabilities of AI agents are gaining increasing attention, existing solutions remain fundamentally limited. Most rely on flat, narrowly scoped memory components, constraining their ability to personalize, abstract, and reliably recall user-specific information over time. To this end, we introduce MIRIX, a modular, multi-agent memory system that redefines the future of AI memory by solving the field's most critical challenge: enabling language models to truly remember. Unlike prior approaches, MIRIX transcends text to embrace rich visual and multimodal experiences, making memory genuinely useful in real-world scenarios. MIRIX consists of six distinct, carefully structured memory types: Core, Episodic, Semantic, Procedural, Resource Memory, and Knowledge Vault, coupled with a multi-agent framework that dynamically controls and coordinates updates and retrieval. This design enables agents to persist, reason over, and accurately retrieve diverse, long-term user data at scale. We validate MIRIX in two demanding settings. First, on ScreenshotVQA, a challenging multimodal benchmark comprising nearly 20,000 high-resolution computer screenshots per sequence, requiring deep contextual understanding and where no existing memory systems can be applied, MIRIX achieves 35% higher accuracy than the RAG baseline while reducing storage requirements by 99.9%. Second, on LOCOMO, a long-form conversation benchmark with single-modal textual input, MIRIX attains state-of-the-art performance of 85.4%, far surpassing existing baselines. These results show that MIRIX sets a new performance standard for memory-augmented LLM agents. To allow users to experience our memory system, we provide a packaged application powered by MIRIX. It monitors the screen in real time, builds a personalized memory base, and offers intuitive visualization and secure local storage to ensure privacy.

1 Introduction

MIRIX addresses limitations in existing memory-augmented agents with six specialized memory components and a coordinating multi-agent architecture. It evaluates this design on multimodal screenshots and long-form conversations.

  • Existing memory systems struggle to represent sequential events, emotional states, full documents, and multimodal inputs, while many store historical data in flat structures.
  • MIRIX organizes information across Core, Episodic, Semantic, Procedural, Resource, and Knowledge Vault memory components.These components cover events, concepts, task instructions, user-shared media, and exact information requiring preservation.
  • MIRIX uses six Memory Managers and a Meta Memory Manager to manage heterogeneous hierarchical memory and route tasks.A Chat Agent demonstrates interaction with the resulting memory system.
  • The evaluation introduces a screenshot benchmark built from 5,000–20,000 high-resolution screenshots collected over one month from three PhD students.The benchmark requires extracting information and building memory from multimodal user activity.
  • The study also evaluates MIRIX on LOCOMO, a long-form, multi-turn conversation dataset.The supplied passage describes conversations averaging around 26,000 tokens.
  • The contributions include a six-component, eight-agent memory system, a screenshot benchmark, comparative experiments, and a packaged personal-assistant application.

2 Application & Use Cases

MIRIX is presented through a local application that monitors screens, updates structured memories, supports memory-based chat, and visualizes semantic and procedural information. The paper also discusses wearable-device integration and a broader memory marketplace vision.

  • MIRIX Application: The application monitors screen activity and dynamically updates memory to build a contextual understanding of the user over time.It captures screenshots every 1.5 seconds, discards visually similar images, and triggers updates after 20 unique screenshots.
  • MIRIX Application: The chat interface lets users query the assistant about past activities using its accumulated memories for more informed and personalized interactions.
  • Wearable Devices: MIRIX is described as suitable for wearable devices that continuously process audio, visual scenes, and user queries to form real-time memories.Examples include summarizing meetings, remembering places, recognizing recurring visual patterns, and recalling conversations or tasks.
  • Wearable Devices: For wearable hardware constraints, the design supports hybrid on-device/cloud management, keeping Knowledge Vault locally while offloading Resource Memory to the cloud.
  • Memory Visualization: Semantic Memory is visualized as a hierarchical tree of concepts and relationships, with categories such as Social Network and Favorites.The example further divides Favorites into Sports, Pets, and Music.
  • Memory Visualization: Procedural Memory is also presented in a list view.
  • Future Vision: The paper envisions personal memory as an active digital asset within a decentralized marketplace for secure sharing, reuse, and collaborative memory-building.The proposal combines lifelong agents, privacy-preserving infrastructure, and a peer-to-peer marketplace.

3 Methodology

MIRIX combines six specialized memory components with coordinated multi-agent routing, memory updates, and retrieval. Its Active Retrieval and conversational workflows automatically search memory, route relevant information, and support targeted response generation.

  • Memory Components: MIRIX organizes memory into six components: Core, Episodic, Semantic, Procedural, Resource, and Knowledge Vault.Each component is tailored to capture a different aspect of user interaction or knowledge.
  • Memory Components: Core Memory keeps high-priority persistent information visible, while Episodic Memory records time-stamped events for reasoning about routines, recency, and context.Core Memory uses persona and human blocks; Episodic Memory is structured as a log or calendar.
  • Memory Components: Semantic, Procedural, Resource, and Knowledge Vault memories separately store persistent facts, actionable workflows, engaged documents or multimodal files, and sensitive credentials or identifiers.These components distinguish abstract knowledge, task procedures, contextual resources, and secure sensitive information.
  • Active Retrieval and Retrieval Design: Active Retrieval infers a topic from the input, retrieves relevant memories from all six components, and injects source-tagged results into the system prompt.The mechanism removes the need for users to explicitly request memory searches and can retrieve the top-10 entries from each component.
  • Multi-Agent Workflow: The modular architecture uses a Meta Memory Manager and specialized Memory Managers to route inputs and coordinate parallel, nonredundant memory updates.Managers report completed updates back to the Meta Memory Manager, which acknowledges the update process.
  • Multi-Agent Workflow: For user queries, the Chat Agent first performs coarse retrieval across all components, then selects targeted searches and retrieval methods before synthesizing a response.The Chat Agent can also apply precise updates through the relevant Memory Managers when users provide new facts or corrections.

4 Experiments

MIRIX is evaluated on multimodal ScreenshotVQA and long-form textual LOCOMO, using LLM-as-a-Judge accuracy and storage metrics. Results show strong gains over baselines, especially for multimodal storage efficiency and LOCOMO multi-hop reasoning, while open-domain retrieval remains a bottleneck.

  • ScreenshotVQA: MIRIX avoids raw-image storage, whereas Gemini resizes screenshots and SigLIP retains retrieved images at native 2K–4K resolution.Storage is measured using resized image totals for Gemini, retrieved native-resolution images for SigLIP, and the SQLite database for MIRIX.
  • ScreenshotVQA: 35% higher accuracy than RAG baselines with 99.9% lower storage establishes MIRIX’s ScreenshotVQA advantage.MIRIX stores extracted information in a compact SQLite database instead of raw screenshots.
  • LOCOMO: MIRIX achieves the highest average J score and improves on LangMem by over 8 points across LOCOMO evaluation.LOCOMO includes single-hop, multi-hop, temporal, and open-domain questions evaluated with LLM-as-a-Judge.
  • LOCOMO: Over 24 points of multi-hop improvement result from storing consolidated events that remove query-time stitching of dispersed conversational facts.The example combines Caroline’s hometown and relocation into one event, reducing the reasoning needed at retrieval time.
  • LOCOMO: Open-domain gains are narrower because MIRIX still relies on retrieval-augmented generation to find important memories.The authors identify this retrieval dependence as a potential bottleneck for longer-term inference.

5 Related Work

Prior work gives LLM agents memory through latent representations, token-level storage, knowledge graphs, or increasingly specialized memory types. These approaches motivate modular, role-specialized memory systems for better routing and retrieval.

  • Latent-space memory augments transformers through model parameters, external memory matrices, hidden states, soft prompts, or key-value caches.
  • Token-level memory stores past conversational content as raw text in external databases and supports long-term conversation and document retrieval tasks.
  • Flat memory architectures and the absence of modular components hinder effective memory routing and create retrieval and usage inefficiencies.
  • Recent systems refine broad short-term and long-term categories with episodic, semantic, and other specialized memory types.
  • Multi-agent frameworks address complex tasks through autonomous planning loops, shared memory logs, or role-specialized coordination.

6 Conclusion and Future Work

MIRIX introduces a structured memory architecture for long-term reasoning and personalization in LLM-based agents. It combines specialized memory managers, multimodal evaluation, conversational benchmarking, and a user-facing assistant application.

  • Its architecture combines six specialized memory components with dedicated Memory Managers coordinated by a Meta Memory Manager.
  • A high-resolution screenshot benchmark evaluates MIRIX against retrieval-augmented generation and long-context baselines for accuracy and storage efficiency.
  • Experiments on LOCOMO report state-of-the-art performance for long-form conversational settings.
  • A released personal assistant application lets users experience memory-enhanced interactions in everyday scenarios.

A Full Experimental Results with Different Runs

The appendix reports repeated MIRIX evaluations and provides LOCOMO LLM-as-a-Judge comparisons across question types. MIRIX consistently achieves state-of-the-art results, while Full-Context is treated as an approximate upper bound for this dataset.

  • MIRIX and Full-Context were run three times with gpt-4.1-mini, with full results reported in Table 3.
  • MIRIX consistently achieves state-of-the-art results despite variation across repeated runs.
  • Table 3 reports LLM-as-a-Judge scores in percentages for each LOCOMO question type, with higher scores indicating better performance.
  • Because LOCOMO averages only 9k tokens, Full-Context is described as almost the upper-bound, so recovering its performance indicates MIRIX advancement.
  • Zep achieved a 49.09 overall score with gpt-4.1-mini in the cited implementation, and its results were skipped because of suspected implementation errors.
Loading 2507.07957v1…