PaperlayerView exact revision on arXiv

Source-linked AI summary

Omni-SimpleMem: Autoresearch-Guided Discovery of Lifelong Multimodal Agent Memory

Jiaqi Liu, Zipeng Ling, Shi Qiu, Yanqing Liu, Siwei Han, Peng Xia, Haoqin Tu, Zeyu Zheng, Cihang Xie, Charles Fleming, Mingyu Ding, Huaxiu Yao

arXiv:2604.01007v2cs.AI

TL;DR

Lifelong multimodal agents need memory that can retain and recall diverse experiences, but designing such systems spans an interconnected space beyond effective manual or traditional AutoML exploration. The paper deploys an autonomous research pipeline to discover Omni-SimpleMem, which achieves state-of-the-art results on LoCoMo and Mem-Gallery and identifies discovery patterns suited to autoresearch.

  • Problem

    Lifelong multimodal memory requires navigating interconnected architectural, retrieval, prompt, and data-pipeline choices, while existing systems suffer from storage and retrieval limitations or discard non-text signals.

  • Method

    The paper deploys AUTORESEARCHCLAW to extend SimpleMem through ∼50 autonomous experiments that diagnose failures, modify architecture, repair data pipelines, and evaluate a unified multimodal memory framework.

  • Results

    OMNI-SIMPLEMEM achieves state-of-the-art on both evaluated benchmarks, improving F1 by +411% on LoCoMo (0.117→0.598) and +214% on Mem-Gallery (0.254→0.797) relative to initial configurations.

  • Takeaways & Limitations

    The paper’s taxonomy shows that high-impact discoveries include bug fixes, architectural changes, and prompt engineering beyond cumulative hyperparameter tuning, and identifies four properties favoring autoresearch.

Abstract

from arXiv · show

AI agents increasingly operate over extended time horizons, yet their ability to retain, organize, and recall multimodal experiences remains a critical bottleneck. Building effective lifelong memory requires navigating a vast design space spanning architecture, retrieval strategies, prompt engineering, and data pipelines; this space is too large and interconnected for manual exploration or traditional AutoML to explore effectively. We deploy an autonomous research pipeline to discover Omni-SimpleMem, a unified multimodal memory framework for lifelong AI agents. Starting from a naïve baseline (F1=0.117 on LoCoMo), the pipeline autonomously executes ${\sim}50$ experiments across two benchmarks, diagnosing failure modes, proposing architectural modifications, and repairing data pipeline bugs, all without human intervention in the inner loop. The resulting system achieves state-of-the-art on both benchmarks, improving F1 by +411% on LoCoMo (0.117$\to$0.598) and +214% on Mem-Gallery (0.254$\to$0.797) relative to the initial configurations. Critically, the most impactful discoveries are not hyperparameter adjustments: bug fixes (+175%), architectural changes (+44%), and prompt engineering (+188% on specific categories) each individually exceed the cumulative contribution of all hyperparameter tuning, demonstrating capabilities fundamentally beyond the reach of traditional AutoML. We provide a taxonomy of six discovery types and identify four properties that make multimodal memory particularly suited for autoresearch, offering guidance for applying autonomous research pipelines to other AI system domains. Code is available at this https://github.com/aiming-lab/SimpleMem.

1 Introduction

Lifelong multimodal agents need memory that can retain and retrieve diverse experiences, but the design space is too large and interconnected for manual exploration. Omni-SimpleMem addresses this through autonomous research that discovers a unified memory framework and achieves state-of-the-art results on both evaluated benchmarks.

  • Multimodal agents accumulate text, images, audio, and video over extended interactions, yet retaining, organizing, and recalling these experiences remains a critical bottleneck.
  • Existing memory systems either retrieve raw inputs with embedding similarity, causing storage bloat and retrieval noise, or structure text-only memories while discarding visual and auditory signals.
  • The autonomous pipeline executes ∼50 experiments across two benchmarks, diagnosing failures, proposing architectural changes, repairing data bugs, and validating improvements without human intervention in the inner loop.
  • OMNI-SIMPLEMEM combines selective ingestion, unified multimodal representations, and progressive retrieval as its principal architectural discoveries.
  • OMNI-SIMPLEMEM achieves state-of-the-art results on both evaluated benchmarks, while the work also presents a taxonomy of autonomous discoveries and four properties supporting autoresearch.

2 Related Work

Related work spans autonomous scientific discovery, multimodal and text-based agent memory, and automated machine learning. These lines of work motivate the paper’s combination of autonomous research with multimodal memory design.

  • Autonomous Scientific Discovery: Autonomous scientific discovery uses LLM agents with programmatic or quantitative evaluation to generate and assess candidate algorithms.
  • Multimodal Memory Systems: Agent memory has progressed from text-only systems such as MemGPT, Generative Agents, SimpleMem, and A-Mem toward multimodal architectures.
  • Automated Machine Learning: Automated machine learning includes neural architecture search, hyperparameter optimization, and systems that automate broader machine-learning pipelines.

3 Autoresearch-Guided Discovery of OMNI-SIMPLEMEM

The paper uses AUTORESEARCHCLAW to iteratively transform a text-only memory system into OMNI-SIMPLEMEM, a multimodal architecture with selective ingestion, hybrid progressive retrieval, and graph-based reasoning. Optimization combines rapid benchmark iteration with autonomous diagnosis and repair.

  • Pipeline Overview: AUTORESEARCHCLAW starts from SimpleMem, benchmark harnesses with F1 metrics, and LLM-provider access, then iteratively analyzes results and generates improvement hypotheses.
  • Architecture Discovery: The discovered architecture extends SimpleMem from text-only memory to heterogeneous signals through selective ingestion, progressive retrieval, and structured knowledge.
  • Selective Ingestion: Selective ingestion applies modality-specific novelty filtering to remove redundant inputs before encapsulating retained signals into unified Multimodal Atomic Units.
  • Selective Ingestion: MAUs separate compact searchable metadata from heavy raw content, using hot storage for summaries, embeddings, and metadata and cold storage for lazily accessed assets.
  • Progressive Retrieval: Progressive retrieval expands candidates from summaries to detailed text and raw content under token budgets, following hybrid dense, sparse, and graph-based search.
  • Structured Knowledge: The knowledge graph extracts typed entities and relations from MAUs, resolves duplicate entity forms, and expands bounded neighborhoods around query entities.
  • Optimization Process: Optimization uses small benchmark development subsets for rapid experiments, followed by held-out evaluation, while self-healing modules diagnose execution failures and generate targeted fixes.

4 Experiments

Experiments evaluate autonomous optimization, benchmark performance, component contributions, throughput, and a multi-hop retrieval case study across LoCoMo and Mem-Gallery.

  • ∼50 experiments were completed across both benchmarks in ∼72 hours of wall-clock time.
  • 0.117→0.598 F1 on LoCoMo followed 9 successful iterations, including bug repairs, timestamp correction, and set-union retrieval.Two additional experiments were automatically reverted.
  • 0.254→0.797 F1 on Mem-Gallery followed 39 experiments across 7 phases, with full-dialogue outputs and prompt-position changes producing major gains.Four independent final runs yielded F1 in [0.791, 0.797].
  • OMNI-SIMPLEMEM achieved Mem-Gallery F1 of 0.749 to 0.810, consistently outperforming all memory baselines.SimpleMem reached up to 0.538 with GPT-5.1, while OMNI-SIMPLEMEM’s gains were attributed to its combined architectural design.
  • Pyramid expansion and BM25 hybrid search were the most critical ablated components, costing −17% and −14% mean ∆F1, respectively.LLM summarization cost −12%, reducing top-k from 20 to 5 cost −7%, and metadata context cost −2%.
  • 5.81 queries/sec with 8 parallel workers was 3.5× faster than the fastest baseline.Read-only FAISS and BM25 indices enabled concurrent lookup while baselines spent 85–97% of per-query time on sequential LLM generation.
  • In a multi-hop case, pyramid retrieval and knowledge-graph expansion produced the correct answer with F1 = 1.0, while MemGPT scored F1 = 0.0.The query required combining painting histories from separate conversation sessions.

5 Conclusion

OMNI-SIMPLEMEM is a unified multimodal memory framework discovered through autonomous research, achieving state-of-the-art results on LoCoMo and Mem-Gallery. The pipeline’s discoveries extend beyond hyperparameter tuning to code comprehension and cross-component reasoning.

  • OMNI-SIMPLEMEM achieves state-of-the-art performance on both LoCoMo and Mem-Gallery.
  • The autonomous pipeline executed ∼50 experiments in ∼72 hours, starting from a naïve baseline.
  • Bug fixes, architectural changes, and prompt engineering required capabilities beyond traditional AutoML.
  • A taxonomy of six discovery types and four properties of multimodal memory provide a roadmap for applying autonomous research pipelines to other AI system domains.

A Benchmark Details

The benchmarks evaluate long-term conversational and multimodal memory across diverse retrieval, reasoning, temporal, and visual tasks. Related systems vary in modality support, memory structure, ingestion speed, retrieval latency, and processing efficiency.

  • Benchmarks: LoCoMo contains 1,986 QA pairs from 10 conversations spanning 19–32 sessions and evaluates five categories of long-term dialogue memory.
  • Benchmarks: Mem-Gallery contains 1,711 QA pairs from 240 multi-session dialogues with 1,003 grounded images and 3,962 conversational rounds.
  • Related systems: MemVerse combines episodic and semantic memory with a multimodal knowledge graph, but its three-call ingestion process runs at 0.22 items/sec.
  • Related systems: Mem0 reaches 1.28 items/sec ingestion and 18ms search latency but is fundamentally text-only.
  • Related systems: SimpleMem uses memory atomization, adaptive consolidation, and context-aware pruning, achieving state-of-the-art LoCoMo performance with ∼45% lower token cost than Mem0 and 4× faster total processing.
  • Related systems: MemGPT uses explicit push/pop operations across main context and external storage, but remains text-only and relies on sequential LLM calls for memory management.

C.1 System Architecture

OMNI-SIMPLEMEM is implemented as a modular multimodal memory system spanning core data structures, modality-specific processors, layered storage, retrieval, knowledge, and orchestration components.

  • The implementation contains 13,300 lines of Python across 11 subpackages.
  • Core and processing: Core components define the MAU data structure, event hierarchy, and unified configuration.
  • Core and processing: Processors support modality-specific ingestion for text, image, audio, and video.
  • Storage and retrieval: Storage uses a MAU JSON Lines store, a FAISS vector store, and filesystem or S3 cold storage.
  • Storage and retrieval: Retrieval, knowledge, and orchestration modules include a pyramid retriever, query processor, expansion manager, entity extractor, knowledge graph, graph retriever, and unified API.

C.2 Hyperparameters and Benchmark-Specific Configurations

The paper documents OMNI-SIMPLEMEM’s hyperparameters, benchmark-specific configurations, knowledge-graph schema, and 23-stage autonomous research pipeline. The pipeline covers scoping, literature discovery, hypothesis synthesis, experiment execution, analysis, documentation, and finalization.

  • Configurations: Table 4 specifies OMNI-SIMPLEMEM’s hyperparameters, while Table 5 records configurations discovered separately for each benchmark.
  • Knowledge configuration: The knowledge graph defines 7 entity types and 7 relation types, with GPT-4o performing JSON-mode entity extraction during MAU creation.
  • Autonomous pipeline: The complete autonomous research pipeline comprises 23 stages organized into 8 phases.
  • Autonomous pipeline: The pipeline formulates SMART goals, detects hardware, discovers and screens literature, extracts knowledge cards, and synthesizes falsifiable hypotheses through multi-agent debate.
  • Autonomous pipeline: It designs hardware-aware experiments with AST validation, executes them in sandboxed environments, and uses a self-healing loop with up to 10 retries.
  • Autonomous pipeline: Statistical analysis uses t-tests and bootstrap confidence intervals, followed by autonomous PROCEED/PIVOT/ITERATE decisions and simulated peer review.
  • Optimization trajectories: The LoCoMo optimization trajectory contains 7 successive improvements and 2 correctly reverted failed experiments.
  • Optimization trajectories: The Mem-Gallery trajectory contains 39 experiments across 7 phases, with each phase representing a qualitative shift in optimization strategy.

F.1 LoCoMo: Per-Category Performance Across Iterations

Table 8 tracks per-category F1 across the LoCoMo optimization trajectory, highlighting sharp gains in adversarial, temporal, and open-ended reasoning categories.

  • Cat 5 (Adversarial) rises from 0.447 to 1.000 after evaluation-standard refusal phrases are introduced in Iteration 2.
  • Cat 2 (Temporal) gains +0.496 overall, primarily from timestamp correction in Iteration 5.
  • Cat 3 (Open-ended) exceeds SimpleMem by +0.200, making it the strongest category against prior state of the art.

F.2 Mem-Gallery: Detailed Phase-by-Phase Log

The Mem-Gallery trajectory progressed from environment repairs through architectural, prompt, data, visual-reasoning, and plateau phases, ending at F1 = 0.797 after 39 experiments.

  • Phase 1: Environment Setup: F1 = 0.353 (+39%) after upgrading to gpt-4o and switching to local all-MiniLM-L6-v2 embeddings following API proxy errors.
  • Phase 2: Architecture Breakthrough: F1 = 0.690 (+96%) after returning full dialogue text, adding BM25 hybrid search, and creating an image-caption BM25 index.
  • Phase 3: Fine-Tuning: F1 = 0.717 (+4%) after discovering that format-constraint position mattered more than content at temperature=0, while several prompt-tuning trials were reverted.
  • Phase 4: Scale Validation: F1 = 0.726 (+1%) during scale validation, which exposed incomplete data caused by running qa_only before store_only had completed.
  • Phase 5: Exact Citation: F1 = 0.771 (+6%) after exact-phrase instructions improved 12/15 datasets and punctuation stripping in BM25 added +0.018 F1.
  • Phase 6: Visual Reasoning Enhancement: F1 = 0.789 (+2%) after adding dialogue context to image captions improved VR by +0.087 and temporal ordering added +0.006 for TR/CD.
  • Phase 7: Plateau Exploration: F1 = 0.797 was the final peak after four independent runs identified an approximate 0.795 performance ceiling due to random noise.
  • Prompt and Pipeline Components: Answer generation uses separate system and user prompts with retrieved context and structured output requirements to enforce JSON-mode extraction.

G.1.3 Query Intent Analysis

The query processor can convert a memory question into structured intent and modality metadata, using that analysis to select retrieval strategies.

  • The resulting metadata can enable temporal sorting for temporal queries or visual search for image-related queries.
  • The query-analysis assistant is instructed to return valid JSON only.
  • Query analysis extracts intent type, named entities, temporal expressions, modality hints, and a reformulated search query.

G.3 Mem-Gallery Benchmark Prompts

The Mem-Gallery prompt architecture combines global task instructions, category-specific constraints, dialogue framing, model-specific conciseness controls, and multimodal memory injection.

  • G.3.1 System Prompt: The global system prompt defines concise, complete answers and prioritizes the most recent information when events conflict.
  • G.3.1 System Prompt: The task covers factual retrieval, multi-entity, temporal, visual-centric, and test-time visual reasoning over multimodal conversations.
  • G.3.1 System Prompt: The prompts require answers to remain grounded in the conversation and prohibit hallucination.
  • G.3.2 Category-Specific Format Constraints: Three categories receive explicit output constraints, while FR, MR, TR, VR, TTL, and KR use the default system prompt.
  • G.3.2 Category-Specific Format Constraints: AR requires “Not mentioned.” when information is absent, CD requires “Yes.” or “No.”, and VS requires sorted image identifiers.
  • G.3.3 Dialogue Agent Prompt: The dialogue-agent prompt frames each question using speaker identities, retrieved memory, concise-answer instructions, and category format constraints.
  • G.3.4 Reasoning Model Adaptation: Reasoning models receive a conciseness boost because verbose chain-of-thought outputs damage token-level F1 against short references.
  • G.3.5 Memory Context Injection: Retrieved multimodal memories are injected as textual context with image identifiers and base64-encoded image content parts.
Loading 2604.01007v2…