Source-linked AI summary
MEMO: Multimodal Evidence Memory Organization for Long-Horizon LLM Agents
Xian Gao, Jinpeng Wang, Jiacheng Ruan, Guangyu Cao, Ting Liu, Yuzhuo Fu
TL;DR
Long-horizon agents must select and organize useful evidence as interaction histories exceed limited context windows. MEMO organizes query-relevant evidence across text and visual formats, achieving the best overall EM and F1 under 128-token budgets across four benchmarks and three readers.
Problem
Long-horizon agents need external memory as histories exceed context windows, but readout must select task-relevant evidence and organize it efficiently within a limited context budget.
Source: Introduction · p. 1Method
MEMO extracts traceable query-relevant evidence units, then assigns selected units to text, image, or dual presentation and layouts under a shared memory budget.
Source: Introduction · p. 2, Task Definition and Overview · p. 3Results
Across four benchmarks, MEMO achieves the best overall EM and F1 under the 128-token constraint with all three readers.
Source: Conclusion · p. 7Takeaways & Limitations
Evidence-level multimodal organization serves as an effective and efficient memory interface for long-horizon agent memory.
Source: Conclusion · p. 7
Abstract
from arXiv · showhide
Long-running LLM agents rely on external memory to store and reuse information beyond a single context window, yet there is a fundamental tension between the continuous accumulation of interaction trajectories and the limited context capacity. The key challenge in agent memory is therefore not only to retrieve relevant records, but also to select necessary evidence under a given budget and organize it in an appropriate modality. Existing memory readout methods mainly use textual or visual forms. Text preserves high fidelity, but its linear token representation makes contents with different importance compete for the limited context at nearly uniform unit cost. Visual readout renders text into document-like images, which can use two-dimensional layouts to expose structure and emphasize key information, but it may lose fine-grained details during rendering and compression. To address this issue, we propose MEMO, a multimodal evidence memory organization method for LLM agents. MEMO first uses a trained evidence extractor to select relevant memory blocks and form evidence units with source information and presentation requirements. A trained query-conditioned memory manager assigns each unit to a textual, visual, or dual-channel carrier and selects a layout that matches the evidence structure. A deterministic memory construction module then generates the textual package and visual pages. The memory manager is trained with feedback from an offline reader that measures the utility of the guided memory plan, so that retention and presentation decisions align with downstream usage. We evaluate MEMO on four benchmarks, HotpotQA, 2WikiMultiHopQA, LoCoMo, and ALFWorld, with multiple reader backends. The results show that MEMO presents memory more efficiently with fewer memory tokens, improves downstream task performance, and builds more effective working memory under constrained budgets.
Introduction
MEMO frames long-horizon memory readout as query-conditioned evidence selection and organization under a resource budget, using text and visual formats where each is suited. Across the evaluated readers, it improves performance while organizing memory more efficiently.
Introduction
- As interaction histories grow, feeding the full history is costly and makes useful evidence compete with redundant or weakly relevant content, so readout must select and organize evidence within a context budget.The goal is to construct working memory that the model can read efficiently for the current query.Source: Introduction · p. 1
- Text memory uses linear token sequences that make contents compete at nearly uniform cost, while visual layouts can expose structure and reduce token use but may lose fine-grained semantic information.The paper argues that a single modality cannot jointly provide information fidelity, structural expressiveness, and resource efficiency.Source: Introduction · pp. 1–2
- MEMO’s evidence extractor selects relevant source blocks and key character-level spans, forming traceable units that retain exact-detail requirements and specify whether visual structures could improve readability.Each evidence unit combines selected spans with original records and preserves its source.Source: Introduction · p. 2
- A trained, query-conditioned manager assigns each evidence unit to text, image, dual-channel, or deletion under one budget, with plans evaluated by a frozen reader; deterministic construction builds the package and verifies resource use.Text retains precise semantics, while visual layouts show structural relations and dual-channel output pairs visual structure with textual anchors.Source: Introduction · p. 2
- MEMO achieves the best performance across all three reader models under fixed 128-token budgets and without budgets; with Qwen3-VL-32B, 2Wiki F1 is 73.91, versus 56.26 for text and 35.89 for visual memory.The evaluation covers HotpotQA, 2WikiMultiHopQA, LoCoMo, and ALFWorld.Source: Introduction · p. 2
Related Work
Prior work develops textual, structured, visual, and multi-controller memory systems, while MEMO focuses on organizing evidence for the read interface. It selects and presents evidence units for the current reader, including records produced by structured-memory systems.
Textual External Memory
- MEMO extends textual memory systems from forming and recalling content to selecting evidence units and organizing their presentation for the final reader.Textual-memory work includes MemRL (Zhang et al. 2026), Agentic Memory (Yu et al. 2026), SimpleMem (Liu et al. 2026), LightMem (Fang et al. 2025), and Mem-α (Wang et al. 2025).Source: Textual External Memory · p. 2
Structured and Lifecycle-Aware Memory
- MEMO can consume structured records from lifecycle-aware and graph or event memory systems, then construct working memory suited to the current reader.Its evidence units preserve provenance, roles, priorities, and presentation requirements; cited systems include EverMemOS (Hu et al. 2026), MemoryOS (Kang et al. 2025), MAGMA, SYNAPSE, ES-Mem, CAM, A-MEM, and Amory (Jiang et al. 2026a,b; Zou et al. 2026; Li et al. 2025; Xu et al. 2025; Zhou et al. 2026).Source: Structured and Lifecycle-Aware Memory · p. 2
Visual and Optical Memory
- MEMO selects presentation form at the evidence-unit level, combining textual precision anchors, visual structure, or dual presentation as needed.Prior visual approaches include layout-aware pages in MemOCR (Shi et al. 2026) and OCR-style images in AgentOCR (Feng et al. 2026).Source: Visual and Optical Memory · pp. 2–3
Multi-Agent Memory and Context Routing
- MEMO trains a read policy at evidence-unit granularity and evaluates the final text and visual memory artifact using frozen-reader outcomes.Related systems include MIRIX (Wang and Chen2025), G-Memory (Zhang et al. 2025), AMA (Huang et al. 2026), and RCR-Router (Liu et al. 2025).Source: Multi-Agent Memory and Context Routing · p. 3
Method
MEMO converts query-relevant records into source-linked evidence units, then plans their selection and text, visual, or dual presentation under a shared memory budget. A deterministic builder assembles the resulting query-specific text packet and optional visual page for the reader.
Task Definition and Overview
- Given a query, records, and budget, MEMO extracts evidence units and jointly plans their inclusion, presentation, and visual organization under one shared text-and-visual resource limit.The resulting working memory is a text packet plus an optional visual page.Source: Task Definition and Overview · p. 3
Query-Conditioned Evidence Extraction
- The extractor selects relevant source chunks and character-level key spans, then materializes source-linked units with surrounding context and priorities for preserving, visualizing, or omitting content.Units are copied from raw records, with a local context window around each key span; they are the smallest items MEMO can retain, reformat, or omit.Source: Query-Conditioned Evidence Extraction · p. 3
Evidence Extractor Training
- Extractor supervision uses benchmark evidence annotations where available and deterministic weak labels otherwise, with full-parameter fine-tuning on structured source, span, and evidence-unit targets.Annotations come from supporting facts in HotpotQA and 2WikiMultiHopQA, evidence turns in LoCoMo, and replayed state changes in ALFWorld.Source: Evidence Extractor Training · p. 3
Joint Evidence Selection and Presentation
- The manager chooses text, image, dual, or drop actions and visual templates jointly, using text for precise semantics and images for temporal, tabular, or grouping relations.Dual presentation keeps a textual anchor while adding visual structure; templates include cards, timelines, tables, checklists, and columns.Source: Joint Evidence Selection and… · pp. 3–4
Memory Manager Training from Reader Outcomes
- Reader-outcome supervision labels per-unit actions and template preferences by testing counterfactual memories with a frozen reader, excluding over-budget candidates and preferring lower-token choices when answer quality is similar.Action confidence reflects the gap between the best and second-best outcomes, while image and drop labels require reader-based checks; template candidates are ordered so inference can select the first that fits the budget.Source: Memory Manager Training from Reader… · p. 4
Working Memory Building
- The deterministic builder assembles source-linked text entries and template-arranged visual blocks, allocating the shared budget so text and visual costs together do not exceed the limit.It joins the text entries and renders the visual page, then supplies the query and both memory channels to the frozen reader.Source: Working Memory Building · p. 5
Experiments
Across four benchmarks and three frozen readers, MEMO combines stronger overall answers under a strict memory budget with near-best quality at substantially lower memory use when unrestricted. Ablations show that evidence selection and query-conditioned memory management each contribute to the final system.
Implementation Details
- The evidence extractor and memory manager use Qwen2.5-1.5B-Instruct and are supervised-fine-tuned with verl; InternVL3.5-4B-Instruct trains the manager’s reader.All experiments were conducted on NVIDIA A100 GPUs.Source: Implementation Details · p. 5
Evaluation Settings
- Evaluation covers multi-hop QA, long-conversation memory, and ALFWorld next-action prediction, using three frozen readers, matched memory/query inputs, and normalized EM and token F1.Methods are compared with the same frozen Reader under a 128-token limit and an effectively unrestricted setting; the four benchmarks use a unified Overall score.Source: Evaluation Settings · p. 5
Main Results
- Under 128 tokens, MEMO leads Overall EM and F1 with all three readers; without a cap, it uses 83.93 tokens on average and ranks first or second on both Overall metrics.With InternVL3.5-8B, its 128-token gains over the strongest baseline are 7.30 EM points and 7.22 F1 points; with Qwen3-VL-32B, 2.35 EM and 1.84 F1 points; with gpt-5.4-mini, 2.47 EM and 2.81 F1 points. Without a cap, it uses 50.5–50.7% fewer tokens than MemAgent and 66.9% fewer than Text-only; its largest gap from the best Overall result is 0.43 points.Source: Main Results · pp. 5–6, Table 1 · p. 6, Table 2 · p. 7
Component Ablation
- With gpt-5.4-mini under 128 tokens, adding the extractor and manager improves EM by 10.29 points and F1 by 11.38 points while using 35.8% fewer tokens than disabling both.The extractor alone raises EM by 6.07 points and F1 by 8.85 points, reducing memory use from 126.22 to 70.12 tokens; adding the manager then raises EM from 53.56 to 57.78 and F1 from 66.09 to 68.62, with memory use increasing from 70.12 to 81.07 tokens.Source: Component Ablation · pp. 6–7, Table 3 · p. 7
Conclusion
MEMO organizes query-relevant memory as evidence units, jointly selecting content, modality, and layout under a token budget. It improves constrained-budget performance and uses evidence extraction and memory management to support efficient, accurate readout.
Conclusion
- Under the 128-token constraint, MEMO achieves the best overall EM and F1 across four benchmarks with all three readers.Under the 4,096-token window, it uses 83.93 tokens on average and ranks first or second on every overall metric.Source: Conclusion · p. 7
- MEMO converts memory into evidence units and jointly selects content, textual or visual presentation, and layout under a token budget.The method is query-conditioned and designed as a memory readout interface.Source: Conclusion · p. 7
- Ablations show evidence extraction provides the main compression and accuracy gains by preserving query-relevant information before presentation, while the memory manager further improves quality.Source: Conclusion · p. 7