PaperlayerView exact revision on arXiv

Source-linked AI summary

Metis: Memory Foundation Model

Zeyu Zhang, Ziliang Guo, Yihang Sun, Xichong Zhang, Xixuan Hao, Zehao Lin, Yang Zhang, Xiaoyan Zhao, Tong Shen, Bo Tang, Zhi-Qin John Xu, Junchi Yan, Haofen Wang, Xu Chen, Feiyu Xiong, Zhiyu Li, Tat-Seng Chua

arXiv:2607.26760v2cs.CLcs.LG

TL;DR

Agent memory is still largely handled by external modules, leaving native memory capabilities in foundation models underexplored. Metis addresses this gap with a stateful architecture and memory-specific training, and experiments show native memory capabilities while revealing degradation in extremely long-term scenarios.

  • Problem

    Agent memory remains primarily external to foundation models, making backbone integration and end-to-end optimization difficult.

  • Method

    Metis equips foundation models with compact, dynamically updated parametric memory states and trains native memory procedures through memory-specific data and objectives.

  • Results

    Metis exhibits native memory capabilities, while ablations show that multi-fact supervision and auxiliary data improve performance across memory tasks and scenarios.

  • Takeaways & Limitations

    Native memory provides a foundation-model-integrated direction for maintaining and using information through model computation.

  • Takeaways & Limitations

    Because Metis compresses information into fixed-size latent parameters, performance may degrade in extremely long-term scenarios and similar facts may be confused.

Abstract

from arXiv · show

Recent advances in AI agents have increasingly internalized native capabilities into their underlying foundation models, giving rise to multimodal foundation models and large reasoning models. However, agent memory is still primarily implemented through external modules, leaving the native memory capability largely unexplored. In this paper, we take a first step toward this direction by introducing memory foundation models, which empower foundation models with native memory capabilities. We formalize native memory from two perspectives: a persistent and dynamically evolving memory state within the backbone, and native memory procedures that autonomously store and utilize information through model computation. We show that native memory offers advantages in architecture, end-to-end optimization, and efficiency. Based on this formulation, we propose Metis, the first prototype of memory foundation models. Metis introduces a new architecture that equips a foundation model with a native memory state, allowing historical information to be compressed into the model and accessed through memory attention. We construct large-scale memory-specific training data and introduce multiple optimization objectives to acquire these native memory procedures through mid-training. The online memory maintenance of Metis is gradient-free, and the memory update requires only a forward pass. At inference time, all learned model weights remain frozen, while the native memory states are autonomously transformed through standard forward computation. Through extensive experiments, we show that Metis exhibits native memory capabilities and further provide a detailed analysis of its strengths, limitations, and behaviors. To facilitate future research on memory foundation models, we release our project and model checkpoints.

1 Introduction

The introduction argues that AI-agent memory remains external to foundation models, motivating memory foundation models with native memory integrated into backbone computation. It presents Metis as the first prototype, combining a native memory architecture with data-driven memory procedures while acknowledging challenges in long-term information retention and information confusion.

  • Motivation: External memory is typically decoupled from foundation-model backbones, using retrieved textual context such as RAG rather than native architectural integration.External memory constructs context as input, while the backbone performs conditional language modeling over that context.
  • Memory Foundation Models: Memory foundation models convert memory from an external module into an internal mechanism directly involved in forward computation, with autonomous memory transformation.They generate responses from input instructions and native memory, defining native memory through memory state and memory procedures.
  • Memory Foundation Models: Native memory consists of dynamic backbone parameters and computation-executed procedures that can be optimized and adapted to domains through data-driven post-training.The formulation treats memory capability as internal to the model’s forward computation, analogous to adapting foundation models into reasoning models.
  • Metis: Metis is introduced as the first memory foundation-model prototype, using Metis blocks with hyper memory and local memory blocks integrated through memory attention.Its native memory state is inspired by Fast Weight Programming and is designed to participate in backbone computation.
  • Limitations: Metis still faces long-term-task degradation from compressing information into fixed-size parameters and exhibits information confusion in some cases.These limitations make completely eliminating reliance on external memory in contexts challenging.

2 Memory Foundation Model

Memory foundation models extend autoregressive foundation models with native memory across multi-step interactions. They represent persistent online information as dynamic parametric state and autonomously update and use it through backbone forward computation.

  • Comparison with external memory: Unlike external-memory systems, memory foundation models replace explicit storage and retrieval outside inference with native state and procedures internalized in backbone computation.External frameworks maintain textual memory through separate writing and reading operations, whereas native memory uses the same continuous function space as the backbone.
  • Formal definition: A memory foundation model is an autoregressive model whose generation across multi-step interactions is conditioned on current instructions, prior outputs, and a native memory state.Its parameters integrate information from previous steps, while subsequent parameters are autonomously transformed during forward computation using the current instruction and output.
  • Scope of memory: The paper counts only information acquired during online interactions as memory, distinguishing trajectory-specific, adaptive memory from pre-existing offline knowledge.This definition excludes information available before interaction begins because it lacks personalization-specific trajectories and real-time adaptation.
  • Native memory state: The native memory state is a dynamic parametric component coupled with static backbone parameters and participating directly in forward computation.Dynamic and static parameter spaces must be semantically aligned during pre-training or mid-training so they can compute collaboratively during online inference.
  • Native memory procedure: Native memory procedures autonomously perform storage and utilization through continuous numerical computation tightly coupled with the backbone’s forward pass.Storage includes remembering, forgetting, and updating, while utilization supplies stored information needed for current inference.

3 Metis Architecture

Metis equips causal Transformer blocks with native memory through paired local and hyper memory blocks. Local memory maintains dense historical representations, while hyper memory learns to update them and support memory procedures during forward computation.

  • Metis Block: Metis inserts a local memory block and a hyper memory block inside each Transformer block.Local memory maintains dense prior-information representations, while hyper memory constructs parametric function spaces for native memory procedures.
  • Local Memory Block: Local memory stores a dynamic dense memory network M^(l) ∈ R^(d_k×d_v) at each step.The memory state is maintained inside the l-th local memory block and updated across steps.
  • Hyper Memory Block: Hyper memory uses static mid-trained parameters and current-input activations to update local memory states.Its parameters remain unchanged during interactions and provide the parametric foundation for native memory storage.
  • Native Memory Procedures: Metis stores memory by adaptively aggregating intermediate activations and utilizes memory by incorporating local states into forward computation.The storage procedure is performed by hyper memory, whereas local memory supplies current memory states during utilization.
  • Efficiency: Fixed-size native memory makes utilization cost depend mainly on memory-state size rather than increasing linearly with historical interactions.Parallel execution of original attention, memory attention, and memory storage also limits additional inference overhead compared with external memory.

4 Data Construction

Metis’s data construction combines explicitly supervised primary memory-operation sequences with auxiliary compositions targeting complex interactions. The synthesized corpus is built from public benchmarks, temporally ordered and state-consistent examples, quality verification, and substantial coverage of distractors and multi-fact scenarios.

  • Primary Data: Primary data explicitly supervises native memory procedures through optimization across varied memory operations.The data teaches operations performed during forward computation rather than relying on manually specified rules.
  • Primary Data: Primary examples follow temporally ordered, state-consistent interaction sequences so later answers agree with earlier memory operations.These properties teach the model when to store information and when to use it.
  • Primary Data: 27 public benchmarks supply primary data spanning remember, forget, update, and reflect operations.Each sample uses a structured fact and a final query answerable only from preceding turns.
  • Primary Data: 357,137 samples and about 406 million tokens remain after filtering, drawn from 27 source benchmarks.Explicit, implicit, and distractor styles balance instruction salience and noise, with distractors dominating token usage for long-range memory.
  • Auxiliary Data: Auxiliary data targets interference and memory pollution through multi-entity binding, selective forgetting, post-memory dialogue, and memory-irrelevant dialogue.These compositions address confusable facts, selective revocation, interleaved ordinary conversation, and inappropriate use of stored values.
  • Auxiliary Data: 76,153 paired-fact snippet sets support 76,153 multi-entity binding and 76,153 selective forgetting samples, while dialogue subtypes contribute 357,137 and 100,000 samples.The dialogue-based subtypes reuse the full primary-memory sample set and expand coverage of complex interactions.

5 Model Optimization

Metis acquires native memory procedures through mid-training on sequential interactions, freezing the backbone while optimizing only native memory parameters. Three likelihood-based objectives train storage, instruction-driven memory operations, and robustness against interference and memory pollution.

  • Training setup: Training samples are multi-step interactions whose steps are processed sequentially, with memory updated before each subsequent step and supervision applied only to selected query steps.At each step, the model response is conditioned on the memory state shaped by preceding interactions.
  • Training setup: During mid-training, Metis freezes backbone parameters and optimizes only native memory parameters using three likelihood-based objectives: reconstruction, operation, and regularization.The objectives jointly shape the native memory state and memory procedure.
  • Curriculum: The task-weighted sampler uses five data subsets and linearly shifts sampling from storage-oriented data toward harder long-range and regularization data.Sampling weights modulate subset frequency rather than explicit loss coefficients.
  • Memory reconstruction: Memory reconstruction trains lossless storage by requiring later queries to regenerate an earlier reference passage retained in the native memory state.Minimizing reconstruction loss drives the hyper memory block to encode the reference for recovery by the memory utilization procedure.
  • Memory operations: Memory operation training teaches Metis to remember, forget, update, and reflect from explicit, implicit, and distractor interactions, including intent inference and long-range retention under intervening noise.Targets reflect updated values, omit forgotten values, and compose stored facts for multi-hop reasoning.
  • Regularization: Regularization targets interference and memory pollution through multi-fact, selective-forgetting, and memory-irrelevant interactions that penalize cross-fact confusion, collateral forgetting, and value leakage.Multi-step supervision can jointly constrain retained and revoked facts within one interaction.

6 Experiments

Across memory operations, memory-based QA, ablations, and OOD evaluation, Metis generally outperforms baselines and benefits from larger backbones and multi-fact supervision. Its limitations include declining long-term capacity with longer inputs or trajectories and performance degradation when irrelevant information is stored.

  • Memory Operation Tasks: Metis-27B achieves the best average performance on both memory-operation benchmarks under the no-context setting, with especially large gains in forgetting on the Metis test set.The results suggest that larger backbones better formulate and utilize the native memory state, while forgetting remains the most difficult operation on MemOps (Gold).
  • Memory-based QA Tasks: Metis achieves the best average performance on memory-based QA and shows larger gains on complex or long-range tasks, including HotpotQA, LongMemEval, LoCoMo subset, multi-hop, and temporal questions.Without context, original Qwen3.5 models obtain almost zero scores on LoCoMo (Gold), whereas Metis remains effective beyond simple factual questions.
  • Ablation Studies: Removing Multi-fact Scenario data consistently reduces performance, while removing the entire auxiliary dataset causes a much larger overall degradation.Multi-fact supervision helps Metis integrate related information and maintain a coherent memory state, particularly for MemOps (Gold) and the Metis test set.
  • Ablation Studies: Removing adaptive aggregation causes the largest performance drop, while query-key normalization is also important and GDU outperforms a linear update on LoCoMo (Gold).Directly using the last token fails to capture information distributed across the input, and GDU provides a better balance in long-term scenarios.
  • OOD Transfer: Metis consistently outperforms memory baselines on ATM-Bench across model scales and most question types, including deterministically scored number questions.This OOD advantage indicates that the improvement is not merely an artifact of an LLM judge for open-ended questions.
  • Capacity and General Capabilities: Metis performance declines as single updates become longer or update trajectories grow, and active storage of irrelevant information causes consistent drops across general benchmarks.Early information is gradually weakened by later updates, while Metis largely preserves its backbone’s general capabilities when the memory state is empty.

7 Related Work

Related work positions memory as a critical capability for foundation models and agents, with mechanisms categorized as textual, latent, or parametric. It also covers dynamic fast-weight approaches and explicit external-memory architectures for storing and retrieving information during inference.

  • Memory mechanisms: Memory supports future inference in foundation models and agents and is categorized as textual, latent, or parametric memory.These systems are applied to personal assistants, deep research, and coding agents.
  • Fast-weight paradigms: FWP maintains dynamic fast weights during inference alongside training-learned slow weights to capture sequence-dependent information.Existing directions include linear attention, which uses feature maps instead of the softmax kernel to achieve linear complexity and a recurrent state.
  • Explicit memory modules: MANNs add explicit external memory modules with differentiable read and write operations to support associative recall, algorithmic reasoning, and few-shot adaptation.Early examples include Memory Networks and Neural Turing Machines.

8 Conclusion

The paper introduces memory foundation models and proposes Metis as their first prototype, combining native memory states with learned memory procedures. Experiments support Metis’s effectiveness while identifying limitations that prevent native memory from fully replacing external memory.

  • Contributions: The paper formalizes native memory through persistent memory states and native memory procedures, then proposes Metis as the first memory foundation model prototype.Metis uses local and hyper memory blocks to maintain compact dense memory states across interaction steps.
  • Training: Metis is trained with memory-specific data and a mid-training framework combining memory reconstruction, memory operation, and regularization objectives.The memory-specific dataset is constructed from public benchmarks.
  • Evaluation: Experiments verify Metis’s effectiveness and analyze its behavior from multiple perspectives.
  • Limitations: Metis’s fixed-size latent memory can degrade in extremely long-term scenarios and confuse semantically similar facts, so it complements rather than replaces external memory.The limitation arises because information is compressed into fixed-size latent parameters.

A Roadmap for Memory Foundation Models · A POSSIBLE PARADIGM SHIFT

Memory foundation models are framed as a progression from stateless predictors to stateful learners and, ultimately, continually self-evolving systems. This roadmap links native memory to changes in computation, memory organization, learning, cognition, and capability development.

  • A Roadmap for Memory Foundation Models: Metis initiates memory foundation models by converting memory from an external information-management module into a persistent internal state integrated with forward computation.This approach directly couples memory storage and utilization to the model’s computation.
  • A Roadmap for Memory Foundation Models: Level I, Stateful Capability, shifts foundation models from static functions to systems with persistent dynamic internal states.The state evolves during interaction and changes the model’s basic computational unit.
  • A Roadmap for Memory Foundation Models: Level II, Self-Managing Capability, requires models to decide what to remember, update, consolidate, and forget according to semantics, utility, privacy, and safety.Memory states and transformation procedures become native, trainable components rather than primarily external rules or workflows.
  • A Roadmap for Memory Foundation Models: Level III, Experience-Learning Capability, extends memory from information support to adaptation by using interaction histories to refine representations, knowledge, and behavior.The model changes as a consequence of what it experiences.
  • A Roadmap for Memory Foundation Models: Level IV, Persistent Cognitive Capability, calls for structured, enduring, and continually updated internal models of the world, users, tasks, time, and itself.These representations are intended to capture temporal changes and causal dependencies.
  • A Roadmap for Memory Foundation Models: Level V, Self-Evolving Capability, aims to turn accumulated experience into continual capability development through reflection, abstraction, reorganization, and discovery of new learning strategies.This level marks a transition from local adaptation to open-ended capability evolution.
  • A Roadmap for Memory Foundation Models: The roadmap presents native memory as a mechanism that can connect computation, learning, cognition, and continual self-evolution beyond conventional storage enhancement.Memory foundation models therefore represent a more fundamental change in the foundation-model paradigm.
  • A POSSIBLE PARADIGM SHIFT: Foundation models may progress from stateless predictors to stateful learners and eventually to systems that organize memory, maintain persistent cognition, and develop capabilities through accumulated experience.This trajectory summarizes the proposed possible paradigm shift.

B Extensive Experiment Results · C Further Analysis of Update Designs · C.1 LU and GDU across Model Scales

Metis shows native memory capabilities across memory operations, with performance shaped by context access, model scale, and update design. LU and GDU exhibit a recurring trade-off: LU performs better on Metis test tasks, while GDU is stronger on long conversational memory.

  • B Extensive Experiment Results: Full textual evidence produces consistently high performance across all four MemOps operations, while partial context causes sharp degradation.Standard Qwen models obtain near-zero scores without context, highlighting the difficulty of memory operations without direct evidence access.
  • B Extensive Experiment Results: Metis-27B achieves the strongest overall performance among no-context methods, leading on updating and reflection.Metis-9B performs best on forgetting, while Temp-LoRA remains strongest on remembering.
  • B Extensive Experiment Results: Metis’s substantial reflection improvement indicates that native memory can support higher-level reasoning over stored information, not merely fact preservation.Metis also offers a more balanced advantage across memory operations and outperforms Temp-LoRA in the overall comparison.
  • B Extensive Experiment Results: Scaling Metis from 4B to 9B yields moderate gains, whereas 27B substantially raises the average score.The largest model improves updating and reflection but performs worse than Metis-9B on forgetting, showing that scaling benefits operations unevenly.
  • C Further Analysis of Update Designs: At 4B, replacing GDU with LU remains competitive, motivating comparisons across the 9B and 27B scales.Metis-9B LU was trained to 14k steps without early stopping, while the reported Metis-9B GDU result uses the 8k checkpoint.
  • C.1 LU and GDU across Model Scales: Across all three scales, LU scores higher on the Metis test set but lower on LoCoMo (Gold), revealing a task-dependent trade-off.The aggregate comparison is driven by Metis test performance at 9B and LoCoMo (Gold) at 27B, while the updates are nearly tied at 4B.
  • C.1 LU and GDU across Model Scales: LU’s sharp drop on LoCoMo (Gold) suggests greater vulnerability over long conversational trajectories.This contrasts with LU’s stronger performance on direct memory-operation tasks represented by the Metis test set.
  • C. Further Analysis of Update Designs: The LU baseline is further examined through data and structure ablations on Metis-4B, using Overall as an equal-weight macro-average and ΔAvg. as the relative gap from the reported LU reference.All ablation scores are percentages.

C.2 Ablations from the LU Baseline

LU-based ablations largely reproduce the GDU study’s decline patterns: removing SA is most damaging, while removing OQ or QKN also hurts Overall. Removing MS alone is the exception, slightly improving Overall under LU rather than decreasing it under GDU.

  • LU ablation results: Removing SA causes the largest Overall degradation among LU ablations, identifying adaptive aggregation as the most consequential structural component under LU.The study replaces GDU with LU while keeping the same data and structure ablations.
  • LU ablation results: Removing OQ or QKN lowers Overall under LU, matching the GDU-based direction and indicating their contributions are not specific to GDU.The LU variants use the same data and structure ablations as the major GDU-based study.
  • LU ablation results: Removing MS alone slightly improves Overall under LU, contrasting with its decrease in the main GDU-based ablation.This is the sole LU ablation that does not produce the expected decline.

D Transfer Across Backbone Families and Scales

Metis transfers across compatible Gemma and Llama causal decoder-only Transformer backbones and tested model scales. Across benchmarks, memory-only performance exceeds no-context controls but remains below full-context performance, while transfer gains vary without monotonic scaling.

  • Architectural transfer: Metis integrates its native-memory components into compatible causal decoder-only Transformer layers across Llama3.1 and Gemma4 backbones.The transfer tests use Llama3.1-8B, Gemma4-12B, Gemma4-31B, and Llama3.1-70B.
  • Evaluation results: Across every benchmark, memory-only performance is higher than no-context performance but lower than full-context performance.The comparison uses Metis’s internal memory without replaying the original context, alongside no-context and full-context conditions.
  • Evaluation results: Some transferred variants improve overall performance relative to Qwen-based Metis-4B and Metis-9B references, but gains do not hold across every task.Overall is defined as the equal-weight macro-average across four benchmarks.
  • Scaling behavior: Transferability holds across the tested Gemma and Llama backbones and scales, but performance varies across model sizes and benchmarks without a monotonic scaling trend.The authors suggest the shared training recipe may not be equally matched to every backbone and scale.

E Evaluation Implementation Details … G.1 End-to-End Latency on LoCoMo (Gold)

The evaluation specifies controlled implementations, prompt formats, and judge scoring, then measures Metis efficiency on LoCoMo (Gold). Metis achieves lower latency than Full Context and other memory methods while using a short query prompt after memory commitment.

  • E Evaluation Implementation Details: DenseRAG retrieves the top-5 cosine-similar chunks from only the current instance’s visible context, excluding gold answers, evidence identifiers, and future turns.Sentences longer than 256 embedding-model tokens are split into contiguous chunks.
  • F.1 Prompt Notation and Coverage: Prompt templates cover no-context, partial-context, full-context, and LoCoMo (Gold) evidence-session queries using runtime-inserted fields.The LoCoMo (Gold) template includes session, date, dialogue-turn, speaker, dialogue text, and optional image-caption fields.
  • F.2.1 Qwen3.5 Backbone: Qwen3.5 query prompts require concise answers and permit “No information available” when the answer is unknown from the supplied context.Partial-context prompts instruct the model to use only retrieved context.
  • F.2.2 Metis, Temp-LoRA, and 𝛿-Mem: During information processing, Metis uses a commit instruction, Temp-LoRA updates its temporary LoRA, and 𝛿-Mem ingests raw memory steps through native chat-message processing.At query time, the methods use learned or updated memory states with a shared concise-answer query.
  • F.3 Prompts for LLM-as-a-Judge: Formal scoring uses gpt-4.1-mini at temperature 0 with three repeats, taking the median score for each example.The rubric accepts aliases and semantically equivalent wording while requiring the correct core entity, value, date, or relationship for full credit.
  • F.4 Prompts in General Capability Study: General capability evaluations retain native prompts for MMLU-Pro, IFEval, GSM8K, and MMMLU, while Metis stores a shared active-stage instruction in memory.Qwen3.5 receives the instruction prepended to the complete benchmark prompt.
  • G Efficiency: The efficiency study measures end-to-end and query latency on LoCoMo (Gold), then analyzes history-length effects, storage, and multi-user serving.The study focuses on inference efficiency at the 4B scale.
  • G.1 End-to-End Latency on LoCoMo (Gold): 69.3%: Metis reduces P95 end-to-end latency versus Full Context, from 3.012 to 0.926 seconds, while averaging 0.562 versus 0.607 seconds.Metis also reduces average and P95 latency versus 𝛿-Mem by 36.4% and 42.2%, respectively.

G.2 Latency Scaling with Context Length · G.3 Per-Session Storage across Context Lengths

Metis’s end-to-end latency becomes lower than Full Context at 64K tokens and gains further speedups at 128K, while its memory commit remains a small latency fraction. Its persistent per-session storage is context-length independent, unlike Full Context’s growing KV cache.

  • G.2 Latency Scaling with Context Length: The 4B evaluation uses controlled histories from 512 to 128K tokens on one NVIDIA A800 GPU, generating 32 or 128 tokens with batch size 1.Reported values average five measured runs after one warm-up; Metis ingests histories in chunks of at most 2K tokens.
  • G.2 Latency Scaling with Context Length: At 64K, Metis becomes faster than Full Context for both output lengths: 1.096× for 32 tokens and 1.074× for 128 tokens.At 128K, the speedups rise to 1.497× and 1.595×, respectively.
  • G.2 Latency Scaling with Context Length: Metis’s pure memory commit takes approximately 0.203 seconds at 64K and 0.406 seconds at 128K, representing only 2.1–4.3% of E2E latency.Most ingestion time comes from history-encoding backbone forwards, which grow from approximately 3.86 to 7.65 seconds.
  • G.2 Latency Scaling with Context Length: Metis’s long-context advantage over Full Context emerges at 64K because Qwen3.5-4B uses linear attention in 24 of 32 layers, delaying the crossover.The remaining 8 full-attention layers continue contributing context-dependent KV traffic as histories grow.
  • G.2 Latency Scaling with Context Length: Memory readout and pure commit are not major bottlenecks, but asynchronous overlap is disabled because finer-grained launches and synchronization may offset its latency benefits.The final implementation therefore does not enable concurrent execution on separate CUDA streams.
  • G.3 Per-Session Storage across Context Lengths: Per-session storage is measured from runtime objects for 4B methods over 512 to 32K context tokens, including Full Context’s retained KV cache, query prompt, and generated tokens.KV cache and model states are stored in BF16.
  • G.3 Per-Session Storage across Context Lengths: At 32K, Full Context uses 1,118.86 MB, versus 16.79 MB for Metis Full and 2.11 MB for Metis k=64, yielding 67× and 529× storage ratios.Extrapolating the measured linear trend to 128K gives approximately 256× and 2,000× ratios, respectively.

G.4 Multi-User Serving

Metis addresses multi-user serving by sharing one frozen backbone while representing each user with a swappable dynamic memory state. Its Memory-as-Adapter design provides isolation and context-length-stable serving, while prototype measurements show bounded overhead and remaining production-scaling challenges.

  • Serving Architecture: Memory-as-Adapter serving shares one frozen backbone and static parameters, leaving only each user’s dynamic memory state as a 16.79 MB full-precision component in Metis-4B.The design treats each dynamic memory state as a user-specific adapter, avoiding one model replica per user.
  • Isolation: Across 200 cross-user probes with 20 concurrently served users, no response contained another user’s value, confirming structural and empirical isolation.Disjoint batch-dimension slices exclude cross-user interference, and own-fact recall remained identical across batch sizes B∈{1, 4, 8, 16, 20}.
  • Latency Scaling with Context Length under Batching: 88–91 ms TTFT, 4.76–4.78 s end-to-end latency, and flat decode throughput remain stable as per-user context grows from 1K to 64K tokens with B=8.These measurements include loading and injecting states from SSD and compare Metis with RAG and Full Context variants under the MaA architecture.
  • Overhead of the Memory Path: 1,585.6 tokens/s aggregate throughput at B=64 yields a 57.1× serial-serving speedup, while memory-path latency overhead rises from 14% at B=8 to 30% at B=64.The 30% measurement conservatively reloads, re-stacks, and re-injects all per-user states on every request; scaling remains near-linear.
  • Limitations: The results are single-node, static-batch prototype medians, while continuous batching and interleaved production memory reads and writes remain unresolved efficiency challenges.Continuous batching requires online state-slot allocation and reclamation, and memory writes are currently offline commits.
Loading 2607.26760v2…