Source-linked AI summary
Diachronic Hypergraphs for Orchestrated Multi-Agent Multimodal Memory Curation
Yichao Feng, Ran Zhang, Haoran Luo, Zhenghong Lin, Carl Yang, Anh Tuan Luu
TL;DR
MAS memory must preserve evidence, roles, decisions, procedures, and evolving collaborative experience beyond isolated agent contexts. MAGE addresses this with a heterogeneous temporal hypergraph, role-aware retrieval, validation, lifecycle management, and budgeted context packing. Across benchmarks, backends, diachronic probes, ablations, and system tests, it outperforms memory baselines while supporting persistent MAS without modifying model parameters.
Problem
MAS memory and text-based transfer can isolate knowledge, omit evidence, compress uncertainty, and lose role-specific meaning across agents and workflow stages.
Method
MAGE stores collaborative experience in a heterogeneous temporal hypergraph and provides role-aware retrieval, validation, lifecycle management, and budget-bounded context packing.
Results
MAGE outperforms memory baselines across twelve text and multimodal benchmarks, remains robust across LLM backends, and benefits from each tested architectural path.
Takeaways & Limitations
Structured hypergraph memory supports persistent MAS that reuse evidence, decisions, and procedural experience without modifying model parameters.
Takeaways & Limitations
The discussion characterizes MAGE’s evidence as supporting gains across evaluated tests, but does not establish performance beyond those evaluated settings.
Abstract
from arXiv · showhide
Multi-agent systems solve tasks through collaboration, tool use, multimodal reasoning, and orchestration, but each agent operates within a knowledge boundary defined by its observations, context, and resources. Memory must preserve and transfer evidence, role specific context, decisions, procedures, and experience across interactions, not only outcomes. Vector and graph memories flatten these structures into embeddings or dyadic traces, obscuring events involving agents, tools, documents, errors, and evidence. This limits knowledge sharing, tracing, reuse, revision, and orchestration. We present MAGE, a hypergraph based multimodal database designed as a memory engine for MAS. MAGE stores agents, messages, tools, errors, procedures, documents, entities, decisions, and evidence in a heterogeneous temporal hypergraph, preserving high order collaborative events as reusable memory. It supports decision driven updates, role aware retrieval, validation, lifecycle management, and budget bounded context packing. By delivering knowledge to agents and orchestrators, MAGE expands their knowledge boundaries without modifying the models. Experiments show MAGE outperforms on various memory baselines.
1 Introduction
MAGE addresses MAS memory limitations by preserving collaborative knowledge and transferring it across agent roles through a temporal hypergraph engine.
- MAS capability depends on preserving and transferring knowledge across agents without losing evidence, semantics, uncertainty, or provenance.
- Existing MAS memory lacks temporal knowledge-boundary updates, while text-based transfers can omit evidence, compress uncertainty, and lose role-specific meanings.
- MAGE externalizes observations, evidence, decisions, errors, and lessons into shared memory that agents can correct, validate, and reuse without modifying LLMs.
- Role-aware retrieval supplies planners, retrievers, executors, and critics with memory suited to their responsibilities, task state, and information needs.
2 Related Work
Prior MAS research covers reasoning, role communication, workflows, and long-term memory, while MAGE organizes these needs through a layered memory architecture.
- Existing MAS frameworks coordinate agents through role communication, reasoning actions, search, and SOP-style workflows.
- Long-term memory systems let agents retrieve experience using relevance, recency, and importance, and synthesize reflections for planning.
- MAGE’s architecture stores, indexes, ingests, maintains, and retrieves heterogeneous evidence through six modular layers.
3 Preliminaries
The preliminaries define MAGE’s evidence streams, heterogeneous temporal hypergraph representation, bitemporal lifecycle, and budget-bounded role-conditioned retrieval.
- MAS evidence consists of textual documents, multimodal items, and interaction traces containing role-typed agents, trajectories, and results.
- A heterogeneous hypergraph represents typed nodes, typed hyperedges, and labelled binary relations, preserving higher-order facts.
- Pairwise decomposition is lossy because it confuses co-occurrence with n-ary co-participation and erases multiplicity, so hyperedges remain first-class storage units.
- Bitemporal stamps separate transaction time from valid time, while lifecycle states record monotone movement among active, deprecated, archived, and deleted.
- Budget-bounded retrieval returns a role-conditioned subgraph under token budget B, treating context selection as a 0/1 budget knapsack problem.
4 Methodology
MAGE implements a temporal hypergraph memory engine with modular ingestion, validation, role-aware retrieval, orchestration, and lifecycle maintenance.
- MAGE uses six layers spanning storage, indexing, ingestion, lifecycle and consistency, retrieval, and an MAS-facing API.
- Its vocabulary captures raw collaboration and durable knowledge through typed nodes and hyperedges for facts, collaboration, error correction, evidence, and procedures.
- Ingestion converts text, multimodal items, and traces into typed graph elements while retaining provenance pointers and reusable memory or procedure nodes.
- A four-way decision policy reconciles new memories with compatible neighbors using actions governed by similarity, polarity, and overlap conditions.
- Role-conditioned queries seed and expand candidates across hypergraph and typed-neighbor relations, then score, gate, and pack them into token-limited contexts.
- Maintenance consolidates duplicates, performs garbage collection, and uses rehearsal-sensitive decay so frequently reused memories persist longer.
- The orchestrated solve loop dispatches role-specific subtasks, revises residual workflows after critique, and writes complete trajectories back for future reuse.
5 Experiments
MAGE is evaluated across datasets, language models, diachronic probes, architectural ablations, MAS frameworks, and system-management measures. Results report improvements over memory baselines, robust cross-backend gains, lifecycle benefits, architectural contributions, broad pluggability, and efficient operations.
- RQ1: Baseline Comparison: MAGE leads all 12 GPT-4.1-mini datasets, reaching 66.93 macro F1 and 51.13 macro EM versus 63.57 F1 for the strongest non-MAGE baseline per dataset.The paired margin is +3.37 F1 under identical evaluation conditions.
- RQ2: Cross-LLM Robustness: MAGE achieves macro F1 gains of 3.37, 3.44, and 3.37 points over the strongest non-MAGE baseline with GPT-4.1-mini, Claude-4.5-Sonnet, and Qwen2.5-VL-32B-Instruct, respectively.Per-dataset gains range from +1.90 to +5.40 points across backends.
- RQ3: Diachronic Dynamics: MAGE reaches 91.2% correction propagation and 88.5% contradiction resolution, while reducing stale retrieval to 3.1% from 11.5% for the best baseline.Cross-agent transfer improves by +2.4 F1 versus +1.1 for the strongest baseline.
- RQ4: Component Ablation: Removing hyperedge expansion, role-aware retrieval, temporal decay, consolidation, or combined write-and-maintenance lowers performance relative to full MAGE.Clique projection lowers macro F1 from 66.93 to 65.72, and retrieval-only trails full MAGE on every dataset.
- RQ5: MAS Pluggability: Across eight MAS frameworks, MAGE averages gains of 4.70 macro F1 and 3.19 macro EM without altering native coordination.Direct gains 9.88 F1 points, while MetaGPT gains 6.05 points.
- RQ6: System Efficiency: MAGE reports p99 latencies of 5.25 ms for document insertion, 6.88 ms for trace insertion, 0.18 ms for update, 0.25 ms for delete, and 66.69 ms for query.These measurements evaluate online writes, updates, deletions, retrieval, and maintenance under multi-agent workloads.
6 Conclusion
MAGE is presented as a persistent, heterogeneous temporal hypergraph memory engine for multi-agent systems. Its schema, ingestion paths, role-aware retrieval, expansion, and budgeted packing preserve and reuse collaborative evidence, procedures, and experience.
- Conclusion: MAGE represents persistent MAS experience as a heterogeneous hypergraph connecting agents, roles, evidence, entities, modalities, procedures, and traces.Role-aware retrieval, temporal lifecycle management, consolidation, and budgeted packing support reasoning and data management.
- Implementation: The released library exposes a unified interface for inserting raw documents, multimodal items, and MAS trajectories while keeping validation, indexing, storage, retrieval, and maintenance modular.The implementation is centered around the MageMemory facade.
- Temporal Schema: MAGE’s bitemporal fields separate when a fact is valid in the task world from when its record is active in memory, supporting auditable updates and deletions.The fields are valid_from, valid_to, tx_from, and tx_to.
- Hypergraph Schema: Hyperedges are first-class objects that preserve high-order events, such as an answer jointly depending on an image region, OCR string, chart axis, prior answer, and agent role.This avoids decomposing many useful reasoning events into pairwise links.
- Insertion: MAGE ingests documents, multimodal items, and complete MAS traces into typed nodes, binary edges, hyperedges, distilled memories, and procedures.Trace ingestion preserves messages, actions, errors, corrections, and execution relationships.
- Retrieval: Role-aware seeding, bidirectional expansion, and budget-bounded packing expose compact, role-compatible subgraphs rather than relying on surface similarity or unbounded retrieval.Expansion can recover adjacent entities, modalities, memories, procedures, and evidence relations.
B.1 Task-Family Effects of Structured Memory
MAGE’s structured memory benefits multiple task families by preserving reusable evidence patterns and linking information that isolated retrieval may leave ambiguous. Its context construction emphasizes coherent, role-compatible evidence within a limited token budget.
- Cross-Task Effects: MAGE organizes memory around reusable evidence patterns rather than task-specific templates, extending benefits across text-only, document, chart, infographic, and visual QA.The supported mechanisms differ by task family while sharing structured evidence retrieval.
- Text-Only QA: For text-only QA, MAGE binds entities, relation descriptions, and prior question-answer traces to help locate the correct entity among candidates.This is especially relevant when a short factual answer requires entity disambiguation.
- Chart and Infographic QA: For chart and infographic QA, MAGE preserves numeric values, labels, and operations together, reducing ambiguity around what a number refers to and which comparison or arithmetic operation applies.The retrieved hypergraph neighborhood can include the visual element and associated operation.
- Document and Visual QA: For document and OCR-heavy visual QA, MAGE connects extracted text, document fields, modality nodes, and related entities to improve evidence localization.For general visual reasoning, it supplies structured background without replacing the visual model.
- Context Construction: MAGE ranks evidence using semantic, structural, role, confidence, provenance, temporal, lifecycle, and token-cost signals before packing a compact subgraph under a token budget.This targets both insufficient evidence and redundant retrieval within limited context windows.
- Orchestration: As an external memory layer, MAGE supplies prior evidence and experience while leaving deliberation to the agent framework, allowing use across multiple orchestration styles.The same memory graph supports single-pass, refinement, ReAct-style, debate, planning, and role-structured teams.
C.1 Case-Level Interpretation
MAGE’s case-level benefits arise when answers depend on linking entities, values, modalities, visual evidence, and prior reasoning traces. Structured retrieval supplies compact, targeted context across visual reasoning, charts, OCR, documents, and factual QA.
- Visual and Multimodal Reasoning: MAGE binds visual modalities, objects, actions, and prior cases into reusable high-order events for visual-commonsense answers.In Case 1, this supports the more specific answer “fly a kite.”
- Visual and Multimodal Reasoning: Chart arithmetic benefits from storing chart-derived entities and values together, allowing relevant numeric evidence to be retrieved as a bundle.This reduces isolation of the numbers needed for multi-step calculation.
- Targeted Retrieval: Role-aware retrieval emphasizes modality and entity nodes for geographic recognition, while entity-title retrieval supports concise factual answers.These cases show different retrieval specializations for multiple-choice and factual tasks.
- Targeted Retrieval: Fine-grained visual lookup retrieves label-value or label-mark relationships instead of broad image summaries for timeline dates and chart labels.This preserves the specific relation needed to answer Cases 11 and 12.
- Document and OCR Retrieval: Document understanding connects extracted text, local context, and target concepts, which is useful when many labels share a page.Case 13 illustrates this field-label retrieval pattern.
- Document and OCR Retrieval: For OCR-oriented TextVQA, MAGE ties extracted text to its visual source and ranks relevant evidence, reducing confusion among multiple visible strings.The same structured-evidence principle extends to factual, status-recognition, and canonical-answer cases.
- Overall Interpretation: Overall, MAGE is most helpful when answers require relations among entities, values, modalities, and prior traces within a compact context.The claimed improvement is attributed to structured evidence retrieval rather than simply adding more prompt text.
D Full Experimental Protocol
The experimental protocol uses released data and fixed evaluation procedures while constructing a shared graph without test answers, reasoning trajectories, or images. Comparisons hold multimodal inputs, token budgets, prompts, and decoding controls constant across methods.
- Data and Evaluation: The released repository includes training and held-out test data, per-dataset files, preprocessing utilities, and evaluation scripts.Evaluation uses fixed stratified splits rather than claiming coverage of complete benchmark test sets.
- Graph Construction: The shared graph combines IND source documents and images, public background corpora, and MAS traces from pool tasks.OOD pools are reserved for their specified evaluation role.
- Data and Evaluation: The graph contains no test answers, test reasoning trajectories, or test images, and IND versus OOD is defined at the evidence level.The same graph is loaded read-only for all methods.
- Controlled Settings: All methods receive the same token budget and multimodal inputs, while MAGE uses fixed retrieval, ranking, and lifecycle defaults including a 30-day temporal half-life.Lifecycle states deprecated and archived are excluded from packing.
- Controlled Settings: API calls use temperature 0, top-p 1.0, a 512-token maximum, shared retries, and identical versioned prompts within task families.These controls make decoding deterministic given the same memory state.
E Matched Pairwise-Projection Control
The matched control isolates first-class hyperedge representation by keeping nodes and retrieval machinery fixed while replacing each hyperedge with pairwise clique edges. MAGE’s hyperedges outperform this projection, supporting the value of preserving n-ary incidence.
- Control Design: The matched projection control fixes nodes, embeddings, seed retrieval, ranking, role policy, and token budget while replacing each hyperedge with pairwise edges.The control therefore differs in representation rather than the broader retrieval pipeline.
- Results: First-class hyperedges raise macro F1 from 65.72 to 66.93 under otherwise identical conditions.The caption specifies that Full uses first-class hyperedges and Proj. uses clique-projected pairwise edges.
- Interpretation: The consistent advantage identifies preservation of n-ary incidence as an independent source of MAGE’s gains.Larger losses under −Hyperedge also include removing hyperedge expansion from candidate generation.
- Interpretation: Pairwise projection is lossy because distinct hypergraphs can yield the same clique-expanded graph, erasing whether a joint event was recorded.A three-node hyperedge and three pairwise edges both become a triangle.
- Robustness: Scaling role boosts and changing seed top-k produce at most ±0.4 macro F1 variation, while single-role ablations change macro F1 by at most 0.5.These tests indicate that the default role policy is robust rather than finely tuned.
G Diachronic Memory Probes
Diachronic probes evaluate whether shared memory updates, resolves contradictions, avoids stale retrieval, and transfers knowledge over replayed event streams. MAGE leads all four probes, unlike static QA alone.
- Protocol: The replay stream contains 4,000 insertion events, 200 fact updates, and 120 injected contradictions in chronological order.Systems may write, update, and delete during replay before held-out probe questions are asked.
- Metrics: Correction propagation measures whether answers reflect later correcting facts rather than superseded ones, while stale rate counts deprecated or contradicted retrieved items.Contradiction resolution measures accuracy on questions whose evidence was explicitly contradicted during replay.
- Metrics: Cross-agent transfer measures the F1 gain for agent B when agent A’s traces are present versus absent.Together with the other probes, this targets lifecycle and transfer behavior unavailable in static benchmarks.
- Results: MAGE leads on all four diachronic probes, supporting the paper’s lifecycle and transfer claims.The Table 12 caption states that higher is better except for stale rate.
H Extended System Evaluation
MAGE’s extended evaluation reports scalable query performance and stable behavior over a continuous stream, while its multimodal design preserves provenance through auditable textual surrogates rather than native pixel-level reasoning.
- Graph Statistics: 41,208 nodes, 118,536 binary edges, and 23,940 hyperedges comprise the shared graph across twelve pools.The graph occupies 96.4 MB as JSONL and 38.1 MB after compression, including 9,812 modality nodes and 4,377 distilled procedure/memory nodes.
- Scaling: p99 query latency grows sub-linearly from 31.2 ms at 1k nodes to 88.9 ms at 100k synthetic nodes.The reported values are 44.8 ms at 10k nodes and 66.7 ms at 41k nodes; seed retrieval is index-bounded and expansion is hop-limited.
- Continuous Stream: MAGE’s macro F1 varies by at most ±0.5 across ten chronological episodes without monotone drift.The replay contains 4,000 events from the training pools and re-evaluates a fixed held-out question set after each episode.
- Multimodal Representation: MAGE stores multimodal content as typed modality pointers plus extracted textual surrogates, rather than performing native pixel-level reasoning in the hypergraph.The pointer preserves provenance to the raw item, while the surrogate provides an auditable and indexable representation for later sharing and inspection.
- Overall Findings: Across memory baselines, heterogeneous LLMs, ablations, MAS frameworks, and system-level tests, MAGE improves memory-augmented reasoning.The evaluation attributes this pattern to structured hypergraph memory and an auditable memory contract retaining collaborative hyperedges, role policy, provenance, and lifecycle state.