PaperlayerView exact revision on arXiv

Source-linked AI summary

MemT-1.0: Towards Memory-Native Evolution

Zhiyu Li, Zeyu Zhang, Linyun Liu, Yiwei Bu, Chenyang Xi, Xichong Zhang, Yi Yu, Yu Yu, Zhichen Liu, Yuxiao Yi, Chunyu Li, Haoran Liu, Zhenkai Ding, Le Zhou, Juncheng Zhang, Boyu Chen, Zehao Lin, Guorui Li, Renyu Fu, Ji Li, Youyang Xia, Jiajun Shen, Harvey Li, Jingsong Zhou, Helan Hu, Tong Shen, Tianxiao Xu, Chenye Han, Weijie Li, Zhongquan Zhou, Hongming Xu, Yuekeng Li, Erenkai Zhu, Peng Yu, Wenhao Li, Jingyi Kang, Xixuan Hao, Xuzhou Zhu, Can Chen, Jiawei Liu, Chen Jiang, Tao Jiang, Fangwei Xiang, Zepeng Yao, Kai Chen, Guang Yang, Mingfei Lu, Chunhui Li, Zhengxuan Jiang, Bo Tang, Shiyu Zhan, Yihang Sun, Jie Chu, Wenjun Xiong, Zhenxiang Hua, Chen Lin, Detong Su, Dongyu Deng, Qiuping Ding, Guofeng Ding, Si Chen, Guohua Wen, Xu Chen, Zhi-Qin John Xu, Ying-Cong Chen, Ningyu Zhang, Haofen Wang, Muning Wen, Haotian Chen, Wentao Zhang, Xuanhe Zhou, Rita Zhang, Junchi Yan, Feiyu Xiong

arXiv:2610.26825v2cs.CLcs.LG

TL;DR

Foundation models need memory as an intrinsic capability to accumulate information and experience across interactions. MemT-1.0 integrates persistent memory into model computation using stable token-level updates and scalable memory storage. It outperforms evaluated parametric-memory baselines on average while largely preserving general capabilities, and internalized procedural experience improves coding-agent task success.

  • Problem

    Memory is a fundamental capability for accumulating experience and adapting over extended interactions, but external context does not provide the same path to continuously accumulating information within a model.

    Source: §1 · p. 5

  • Method

    MemT-1.0 integrates persistent native memory into model computation, combining stable token-level updates, Mixture-of-Storage, and a two-stage training pipeline.

    Source: §9 · p. 30

  • Results

    Both models outperform evaluated parametric-memory baselines on average while largely preserving their backbones’ general capabilities under the noise setting.

    Source: §9 · pp. 30–31

  • Takeaways & Limitations

    Coding-agent applications show that native memory can internalize procedural experience to improve task success and execution consistency.

    Source: §9 · p. 31

    The fixed-size memory state has limited capacity for detailed historical information, creating trade-offs between retaining earlier details and incorporating recent information.

    Source: §8 · p. 30

Abstract

from arXiv · show

Memory-native evolution requires making memory an intrinsic capability of foundation models, enabling them to continuously accumulate and utilize information. To this end, we introduce MemT-1.0, a new architecture of memory foundation models that internalize memory as part of model computation. Building on Metis, MemT-1.0 employs Gated DeltaNet-2 with a stable token-level memory update mechanism within each turn. We further introduce Mixture-of-Storage to scale native memory capacity. Following a deliberate backbone preparation, we develop a two-stage training pipeline consisting of continued pre-training and post-training, supported by 34B tokens of training data and efficient training and inference infrastructure. We release a model family comprising MemT-1.0-Nano and MemT-1.0-Flash, and conduct a comprehensive evaluation of memory and general capabilities. The results show that they outperform the parametric memory baselines on average and remain competitive with retrieval-augmented methods, while largely preserving general capabilities. At a history length of 128K tokens, MemT-1.0-Nano achieves approximately $1.86\times$ end-to-end inference speedup and a $66\times$ reduction in persistent storage relative to full context KV cache. By internalizing retrieved procedural experience, MemT-1.0 improves task success on agentic tasks, providing a practical step toward memory-native evolution.

1 Introduction

MemT-1.0 extends memory foundation models with stable token-level updates and scalable native memory, alongside a two-stage training pipeline and a released Nano/Flash model family. Across memory and agentic coding evaluations, it improves on parametric-memory baselines on average and uses internalized procedural experience to improve task performance.

1 Introduction

  • MemT-1.0 replaces token selection with stable Gated DeltaNet-2 token-level updates and adds Mixture-of-Storage to scale native memory capacity.It retains dynamic local memory and trainable hyper memory; Nano has 36B parameters (3B activated, 10M dynamic memory), while Flash has 288B (17B activated, 336M dynamic memory).Source: §1 · p. 5
  • A two-stage pipeline combines continued pre-training and post-training, supported by a 34B-token dataset and infrastructure that isolates memory across independent trajectories.The post-training stage adapts models to downstream memory tasks while preserving general capabilities; persistent state remains constant in size as interactions accumulate for a fixed configuration.Source: §1 · p. 5
  • Across 11 memory benchmarks, the models outperform parametric-memory baselines on average and largely preserve general capabilities under the noise setting.The evaluation reports effective use of historical information without accessing original context during inference.Source: §1 · p. 5
  • On SWE-bench Verified and SWE-bench Multilingual, internalized procedural experience improves average task success and consistency across repeated runs over agents without experience.Performance is comparable to explicit textual experience, extending native memory from information recall to procedural guidance in multi-step execution.Source: §1 · p. 5

2 Model Architecture

MemT-1.0 adds persistent native memory computation to selected Transformer blocks, maintaining dynamic memory states across interaction steps. Separate storage and utilization procedures update and retrieve those states, while Mixture-of-Storage expands capacity through routed memory experts.

2.1 Overview

  • Selected Transformer blocks combine original attention with parallel memory attention, while the backbone’s original attention operations are preserved.Memory attention uses information accumulated from previous steps to support generation.Source: §2.1 · p. 7

2.2.1 Native Memory State

  • Dynamic local memory stores online information and evolves across interactions, while trainable hyper memory parameters provide learned memory-processing capabilities.The dynamic state is trajectory-specific; trainable parameters remain fixed during online interaction.Source: §2.2.1 · p. 7

2.2.2 Native Memory Procedures

  • Storage updates dynamic memory states with a stable Gated DeltaNet-2 token-level mechanism, while utilization retrieves accumulated information for autoregressive generation.The stable mechanism is introduced to prevent memory-state divergence during updates.Source: §2.2.2 · p. 8

2.3 Scaling Native Memory Capacity

  • Mixture-of-Storage expands each native memory block into parallel experts and uses dense, separate storage and utilization routers over shared expert addresses.Every expert participates in memory computation, and the routing probabilities determine how information is read from and written to experts.Source: §2.3 · pp. 9–10, Figure 5 · p. 9

3 Training

Training prepares the backbone, then builds native-memory capability through a two-stage pipeline and a 34B-token corpus. Post-training distills from a full-context teacher while memory states carry historical information into responses.

3.1 Backbone Preparation

  • The prepared backbone improves by 6.3 points on average over its public release across six monitored benchmarks, then is frozen as the host and full-context distillation teacher.Its preparation uses capability profiling, intervention-based failure attribution, and capability-conditioned data synthesis.Source: §3.1 · p. 11

3.2 Data Construction

  • The 34B-token corpus combines 20B CPT tokens with 14B post-training tokens organized around memory-native language modeling, storage and utilization, and memory tasks.The three post-training data categories share a context-query-answer format in which context is stored before the model answers a query.Source: §3.2 · pp. 11–12

3.3 Training Objectives

  • Off-policy distillation trains memory parameters to align the student with a frozen full-context backbone teacher on fixed teacher-generated trajectories.The teacher target is truncated to its Top-K tokens; the loss combines forward KL distillation and next-token cross-entropy, with CPT using CE and post-training using KL.Source: §3.3 · pp. 12–13

3.4 Continued Pre-Training

  • CPT teaches causal language modeling within consecutive segments, with prior-segment information accessed through native memory rather than full historical text.Memory states pass between segments within a sequence and reset for each independent sequence.Source: §3.4 · p. 13

3.5 Post-Training

  • Post-training adapts the CPT-initialized memory parameters to memory-based responses by distilling from a full-context teacher under teacher forcing.Response-prediction gradients propagate through memory readout and earlier storage operations, training the model to retain and use historical information.Source: §3.5 · pp. 13–14

3.6 Training Setups

  • Nano uses one CPT epoch followed by post-training, while Flash trains memory from a prepared backbone using an online backbone teacher and Top-64 targets.Nano uses 2K-token storage steps and Top-64 distillation targets; Flash uses the same storage-step length and KL loss.Source: §3.6 · p. 14

4 System and Infrastructure

The infrastructure preserves trajectory-specific memory states while batching model computation across independent trajectories. Efficient memory operators and stateful serving address the costs of ordered updates, expert activations, and growing interaction histories.

4.1 Distributed Training

  • Training packs combine independent trajectories, keeping attention within each step while memory carries information between steps and resets at trajectory boundaries.Layer-wise batching preserves large backbone matrix multiplications while the memory block advances state in trajectory order.Source: §4.1 · p. 14

4.2 Efficient Operators

  • Windowed MoS execution processes complete-step groups and passes a differentiable FP32 final state between windows, limiting simultaneously live expert projections.Smaller windows reduce live intermediates but require more launches and recomputation; storage and utilization routers respectively govern writes and mix expert readouts.Source: §4.2 · p. 15
  • Processing two steps at a time reduces peak memory by approximately 80%, with a 1.6% increase in median forward-plus-backward time.The measurement uses one complete MoS memory block on a single GPU with a 32K-token pack.Source: §4.2 · p. 15, Table 1 · p. 15

4.3 Stateful Inference and Serving

  • Online serving shares fixed model weights across trajectories but keeps evolving dynamic memory states separate for each trajectory.For a fixed configuration, persistent state size stays constant as interactions accumulate, while total storage grows with trajectory count.Source: §4.3 · p. 16
  • At 64K tokens of history, MemT-1.0-Nano achieves an approximately 33× reduction in logical per-trajectory state size compared with full context.After history preparation, query latency remains nearly constant as history grows in the reported two-GPU, single-request BEAM experiment.Source: §4.3 · p. 16

5 Memory Scaling Behavior

The study examines whether memory capability changes with additional post-training and increased native-memory capacity. In controlled MemT-1.0-Nano experiments, both continued post-training and adding MoS experts improve reported memory scores.

5.1 Overview

  • The scaling study examines additional memory-related post-training exposure and increased native-memory capacity as two dimensions of memory capability.The experiments use MemT-1.0-Nano and keep the evaluation protocol consistent across configurations.Source: §5.1 · p. 16, §5.2 · p. 17

5.2 Scaling Protocol

  • The data-and-training comparison follows one Nano trajectory over two epochs with architecture and corpus fixed, while capacity comparisons vary MoS expert count with other configurations held consistent.The capacity comparison evaluates 2- and 4-expert configurations after 400 CPT steps at matched post-training checkpoints.Source: §5.2 · p. 17, §5.4 · p. 18

5.3 Data and Training Scaling

  • Across two post-training epochs, the average score rises from 36.65 to 46.80, while LoCoMo reaches 57.53 at the final sampled checkpoint.Most average-score gain arrives during the first quarter epoch, but scores continue to improve across later checkpoints despite intermediate fluctuations.Source: §5.3 · p. 17, Figure 6 · p. 17

5.4 Memory Capacity Scaling

  • At 3,400 post-training steps, four MoS experts improve the average score by 2.01 points and LoCoMo by 3.25 points over two experts.Four experts also outperform two at the 200- and 5,500-step checkpoints.Source: §5.4 · p. 18, Figure 7 · p. 18

6 Evaluation

The evaluation tests memory retrieval without prior textual context, general capabilities under distracting memories, and efficiency, storage, and design choices across multiple benchmarks and trajectory lengths. MemT-1.0 outperforms parametric memory baselines on average, broadly preserves general capabilities, and shows long-history efficiency and storage advantages, alongside remaining gaps and diminishing returns from low-rank compression.

6.2 Memory Capabilities

  • Flash and Nano score 50.6 and 47.0 on average, ranking first and second among parametric memory methods and exceeding the strongest baseline by 12.5 and 8.9 points.The comparison spans 11 memory benchmarks grouped by trajectory length.Source: §6.2 · p. 19, Table 2 · p. 20

6.3 General Capabilities

  • MemT-1.0 broadly preserves general capabilities under distracting memories, while task-level changes vary across Flash and Nano benchmarks.Flash is comparable to its backbone on average; Nano's average decrease is around one point, with gains and declines varying by benchmark.Source: §6.3 · pp. 20–21, Table 3 · p. 21

6.4.1 Architecture Ablation

  • Turn-level updates reduce the average from 46.97 to 40.98, while removing the stable mechanism or substituting GDN yields 46.43 or 46.72.Bidirectional encoding yields 45.43 and does not improve aggregate memory performance under the evaluated training budget.Source: §6.4.1 · p. 22, Table 4 · p. 22

6.4.2 Training Ablation

  • Post-training increases the average memory-task score by approximately 10.69 points over the continued-pretraining checkpoint, with continued pre-training providing an additional benefit.The comparison finds that language-modeling continued pre-training alone does not provide the same memory-conditioned response capability as the complete pipeline.Source: §6.4.2 · p. 22

6.5 End-to-End Inference Efficiency

  • At 128K tokens, MemT-1.0-Nano achieves approximately 1.86× end-to-end speedup over full context, and its query TTFT remains broadly stable across evaluated history lengths.End-to-end latency includes history preparation and response generation; query TTFT excludes history preparation.Source: Figure 8 · p. 23, §6.5 · p. 23

6.6 Storage Overhead

  • At 128K tokens, MemT-1.0-Nano uses approximately 66× less per-trajectory storage than full context, and its state size remains constant during a trajectory.The full-context storage curve ends at its context-window limit; total storage still grows with the number of trajectories and configured memory capacity.Source: §6.6 · p. 24

6.7 Low-Rank Decomposition

  • Low-rank memory-state reconstruction yields diminishing accuracy returns as retained rank increases, with PersonaMem saturating earlier than LoCoMo.Theoretical factor storage grows roughly linearly with retained rank; on PersonaMem, accuracy reaches its highest observed value at an intermediate rank.Source: Figure 10 · p. 25, §6.7 · p. 25

7 Application in Agentic Tasks

The agentic-task application stores retrieved procedural experience in MemT-1.0-Nano's native memory or in textual context to guide coding agents. Across two SWE-bench evaluations, native memory improves task success and repeated-run reliability over agents without experience, while the paper notes constraints on experience construction and cost.

7.2 Experience-Guided Agentic Framework

  • The framework retrieves procedural guidance from an experience bank and stores the same experience in textual context or native memory for subsequent task execution.Native memory writes the experience separately from the task context, then makes it available through memory attention.Source: §7.2 · p. 26

7.3 Experience Construction and Model Training

  • The experience bank distills 41,057 successful coding trajectories into 1,040 reusable experiences, then counterfactual execution yields 42,403 filtered training samples.Samples retain experience-conditioned continuations that pass tests when the corresponding original trajectory failed.Source: §7.3 · pp. 27–28
  • Application-specific post-training freezes backbone parameters, trains native hyper memory blocks, and uses full-suffix knowledge distillation from a textual-experience teacher.The student accesses the same experience only through native memory, and the continuation actions provide the supervision signal.Source: §7.3 · p. 28

7.4 Agentic Coding Evaluation

  • On SWE-bench Verified and Multilingual, MemT-1.0-Nano improves average success and repeated-run reliability over no-experience agents, with native-memory performance comparable to textual experience.For Verified Before Execution, average success is 69.33% versus 66.07% without experience, and pass^3 is 61.60%, matching textual experience.Source: §7.4 · pp. 28–29, Table 5 · p. 29
  • MemT-1.0-Nano improves success in both experience-injection settings and both benchmarks, including a 63.56% average success rate on Multilingual in Harness-Triggered mode.In that setting, it exceeds the 63.00% result for fixed rank-1 textual experience; on Verified, pass^3 reaches 62.80%.Source: §7.4 · p. 29, Table 5 · p. 29

7.5 Case Study and Discussion

  • A representative task shows native-memory guidance accompanying test-first work, a boundary-condition correction, and additional verification; the framework still relies on externally constructed experience.Effectiveness depends on experience quality and retrieval, storing experience adds computation, and autonomous continual learning remains open.Source: §7.5 · pp. 29–30

8 Limitations and Discussion

MemT-1.0’s fixed-size memory state bounds persistent storage but limits how much historical detail it can retain. The discussion also identifies the relationship between local memory and hyper memory as an open research question.

8 Limitations and Discussion

  • The O(1) fixed-size state constrains capacity, trading retention of earlier details against incorporation of recent information and challenging precise recall.The passage contrasts this bounded state with stored representations that grow as O(N).Source: §8 · p. 30
  • How interaction-level experience in local memory might consolidate into hyper-memory parameters, and how hyper memory might be instantiated through local memory, remains an open question.The passage frames memory capability as a higher-order form of learned information.Source: §8 · p. 30

9 Conclusion

MemT-1.0 integrates persistent memory into model computation, and the conclusion reports better average performance than evaluated parametric memory baselines while largely preserving backbone general capabilities under the noise setting. The appendix describes backbone-specific memory projections, fusion, and gate initialization for Nano and Flash.

9 Conclusion

  • Both models outperform evaluated parametric memory baselines on average while largely preserving their backbones’ general capabilities under the noise setting.The conclusion also reports that coding-agent applications internalize procedural experience to improve task success and execution consistency.Source: §9 · pp. 30–31

B Adaptation to Specific Backbones

  • Nano adds a native memory block to each Qwen3.6-35B-A3B full-attention layer, while Flash attaches memory blocks to DeepSeek-V4 compressed-attention layers.Nano preserves the backbone’s hybrid architecture and MoE computation; Flash assigns one independent memory channel to each of its 64 query heads.Source: §B.1 · p. 36

B.1 Projection and Fusion

  • Nano copies backbone attention projections to initialize dense memory query, key, and value projections, then adds gated memory readouts to the backbone attention output.The memory branch uses independent channels for query heads and retains the backbone’s frozen output projection.Source: §B.1 · p. 36
  • Flash uses a two-stage bottleneck for memory queries and grouped bottlenecks for keys and values, yielding 64 memory heads from eight latent groups.The groups draw on backbone projections; after readout, memory contributions are added before the final backbone output projection.Source: §B.1 · p. 36
  • Flash initializes its eight key-value groups from pairings of local KV and compressed-attention projections, with identity expansion to independently trained heads.Groups inherit normalization parameters from the selected backbone modules.Source: §B.1 · p. 36

B.2 Gates and Initialization

  • Gate designs differ by model: Nano uses dense erase/write projections and rank-256 forget/output factorizations, while Flash uses rank-512 factorizations for all four gates.Forget-gate biases initialize channel time scales log-uniformly from 2048–32768 tokens for Nano and 128–32768 tokens for Flash; memory states start at zero per trajectory.Source: §B.2 · pp. 36–37

B.3 Training Configuration

The stability analysis gives a sufficient worst-case contraction condition and characterizes when the memory recursion remains bounded. The training configuration freezes backbone parameters during memory training and specifies its precision and execution settings.

B.3 Training Configuration

  • Backbone parameters are frozen during memory training; memory parameters and projection computation use BF16, while optimizer state and sensitive memory calculations use FP32.Source: §B.3 · p. 37
  • Table 7 defines TP, PP, CP, and EP as tensor, pipeline, context, and expert parallelism, and DP as logical data-parallel replicas.Source: Table 7 · p. 37
  • Single-GPU MoS measurements in Appendix F use their own workload and software configuration.Source: §B.3 · p. 37

C.1 Problem Formulation

  • When ρ<1, ∥S_t∥≤R/(1−ρ) as t→∞, so the update is bounded and termed stable.Source: §C.1 · p. 38
  • Without colinearity of b⊙k and k, nonzero γ can amplify energy along e and may make A unstable.The analysis assumes d, b, w ∈ (0, 1), ∥k∥₂ = 1, bounded ∥v∥₂, and ρ∈(0, 1), m∈[ρ², ρ).Source: §C.1 · pp. 38–39
  • d < 0.866 is a sufficient worst-case stability condition; bounded additive inputs keep the recursion bounded, with forget-gate D half-life about 5 tokens.Source: §C.1 · p. 39

C.2 Solution: Symmetrization Fix

The symmetrization fix yields Stable GDN-2, whose vectors are colinear and whose update avoids the unstable mechanism. Its erase step contracts one component without expanding orthogonal directions, unlike the illustrated GDN-2 erase step.

C.2 Solution: Symmetrization Fix

  • Stable GDN-2 symmetrizes the update, making its two vectors naturally colinear and avoiding the unstable update mechanism.Source: §C.2 · pp. 39–40
  • With D=I and no write, the symmetrized erase step contracts along u and leaves its orthogonal complement unchanged, making it non-expansive in Euclidean norm.The stated assumptions give 0 < ∥u∥₂² < 1.Source: §C.2 · p. 40
  • For the same input y=e, GDN-2 increases squared norm to 1+γ² when γ≠0, while Figure 12 contrasts this with the non-expansive map.Source: §C.2 · p. 40, Figure 12 · p. 40

D.1 Details of Benchmarks

This section documents benchmark coverage and evaluation methodology for memory and general capabilities, including prompts, controls, baselines, scoring, and storage overhead. It also describes how procedural-experience training examples are constructed and specifies their task splits, memory operations, replay supervision, and training objectives.

D.1 Details of Benchmarks

  • Memory evaluation spans 11 benchmarks for retention, retrieval, updating, management, and personalization, while six general benchmarks cover four capability dimensions.Examples use defined subsets, including PersonaMem's 32K subset and LongMemEval-S [29] [35].Source: §D.1 · pp. 40–42

D.2 Details of Baselines

  • Baseline comparisons include full-context and retrieval systems, history-specific parameter adaptation, and recurrent or compressed memory methods, with shared prompts and scoring for methods using the same backbone.Implementations specify retrieval settings and model configurations, including top-20 retrieval for several external systems and method-specific history-writing or retention procedures.Source: §D.2 · pp. 42–43

D.3.1 Query Prompt Construction

  • Prompt protocols keep histories out of the query for parametric-memory methods, insert scope-specific retrieved content for external systems, and omit generic few-shot demonstrations on memory benchmarks.Full-context histories exceeding 256K tokens are truncated to the most recent suffix; gold answers, evidence identifiers, and scoring labels are excluded from answering inputs.Source: §D.3.1 · p. 43
  • Memory queries remove history bodies while revising only wording that assumes the history remains in the prompt; task questions, choices, formats, and instructions stay unchanged.External-memory systems use the same benchmark-frozen retrieval template per benchmark, with only the [Memory N] entries differing.Source: §D.3.1 · pp. 49–50
  • LoCoMo memory queries refer to conversation stored in memory, while external-memory queries introduce retrieved memories and retain brief-answer or multiple-choice output requirements.Open-ended questions require a brief answer; multiple-choice questions require only (a) or (b).Source: §D.3.1 · p. 50

D.3.2 Decoding Configuration

  • Formal comparisons share decoding settings within each benchmark: temperature is 0 except 0.8 for ImplicitMemBench Priming, and context is capped at 256K tokens.The Qwen3.6 branch disables Thinking; the DeepSeek branch uses its official chat template and verifies Thinking is disabled. Chunked prefilling uses 8,192-token chunks, and outputs reaching the generation limit are retained and scored.Source: §D.3.2 · p. 51
  • Generation limits vary by benchmark, from 10 tokens for MemoryAgentBench FactConsolidation to 4,096 for ImplicitMemBench and some BEAM-1M tasks.BEAM-1M limits also vary by capability, from 512 to 4,096 tokens; MemoryAgentBench limits vary by source and task, from 10 to 2,000 tokens.Source: §D.3.2 · p. 51

D.4 Details of Pipelines and Prompts in General Evaluation

  • General-capability evaluations use Thinking mode, fixed benchmark prompts and scoring implementations, and benchmark-specific context and output handling.LongBench v2 follows its official middle truncation procedure; the noise condition adds fixed demonstrations after truncation, while native memory uses the same truncated target input.Source: §D.4 · pp. 51–52
  • Noise-condition demonstrations are selected independently of model outputs and scores, exclude target questions and answers, and are prepended to the backbone context or committed to an initially empty memory state.Each question independently reads the resulting state, which remains fixed during evaluation.Source: §D.4 · p. 52
  • General-capability benchmarks use benchmark-specific scoring: accuracy for MMLU-Pro, GPQA Diamond, and LongBench v2; strict accuracy for IFEval; and Pass@1 for both code benchmarks.Unparseable, conflicting, failed, or invalid-code outputs count as incorrect under the corresponding protocols.Source: §D.4 · p. 52

D.5 Details of Storage Overhead Evaluation

  • Storage overhead is measured using logical state size after history preparation and before querying, rather than process-level allocated memory, which includes weights and memory pools.The compared states include full-context storage, DenseRAG vectors and texts, Temp-LoRA query-time BF16 adapters excluding optimizer states, and full-rank MemT-1.0 memory.Source: §D.5 · p. 53

E.1.1 Experience Bank Construction

  • 942 of 1,040 experiences derive from a single trajectory, and the median similarity-score gap between the first- and fifth-ranked issue-level retrieval candidates is 0.029.Experiences are induced from pooled episodes as triggerable policies, clustered by workflow, and audited; each records retrieval, trigger, procedure, scope, and decision guidance.Source: §E.1.1 · p. 53

E.1.2 Experience-Conditioned Trajectory Construction

  • Experience-conditioned examples are produced by injecting retrieved experience at selected breakpoints in unsuccessful trajectories and retaining reproducible continuations that pass task tests.The procedure retrieves the top 10 entries, tests ten candidate subsets per breakpoint, and retains at most eight breakpoints per rollout; breakpoint detection uses execution feedback visible to the agent.Source: §E.1.2 · pp. 53–54, Figure 13 · p. 55

E.1.3 Dataset Statistics and Splits

  • Training and validation data are split by task, with all rollouts and intervention variants for a task kept together; SWE-bench Verified is the test set.Construction checks breakpoint validity, repository-snapshot consistency, test outcomes, message and tool-call structure, and context length, discarding invalid or unreproducible samples rather than truncating overlength samples.Source: §E.1.3 · pp. 55–56

E.2 Training Details

  • Training configurations vary the objective (CE or KD), supervision scope (First-Step or Full-Suffix), and cap on successful trajectories selected per task (Max-2 or Max-3).Max-2 and Max-3 retain at most two and three successful training trajectories per task, respectively.Source: §E.2 · p. 56

E.2.1 Trainable Parameters and Memory Operations

  • Metis trains memory blocks at ten decoder layers, using 104,962,580 trainable parameters, roughly 0.3% of the backbone’s parameter count.Each block maintains a fixed-size 512 × 512 value state and 512 × 1 key normalizer; memory resets at each new task or trajectory.Source: §E.2.1 · p. 56

E.2.2 Sample Serialization and Supervision Masks

  • Each sample writes experience to memory, replays the trajectory prefix without memory reading, then enables memory reading from the breakpoint onward.Full-Suffix supervises every assistant turn in the successful suffix, whereas First-Step supervises only the first assistant turn after the breakpoint; both use teacher forcing.Source: §E.2.2 · pp. 56–57

E.2.3 Training Objectives and Hyperparameters

  • The CE and KD objectives train a memory-conditioned student against successful continuations, with KD transferring predictions from a textual-context teacher.The teacher uses the same frozen backbone with memory disabled and receives experience as text; KD retains the teacher’s top 64 tokens and renormalizes its distribution, while the student remains normalized over the full vocabulary. Training uses task interleaving for three epochs and AdamW with a learning rate of 2 × 10^-5 and three warmup steps.Source: §E.2.3 · pp. 57–58

E.2.4 Training Diagnostics

Across four matched training configurations, KD yields higher preliminary agent success than CE despite higher validation NLL, motivating its use in subsequent agent evaluations and ablations. CE nevertheless achieves lower final validation NLL in every configuration.

Training Diagnostics

  • The comparison pairs First-Step or Full-Suffix supervision with Max-2 or Max-3 data settings, matching initialization, training data, and optimization hyperparameters within each pair.Because CE and KD optimize different objectives, the study monitors hard-label validation NLL and compares it within the same supervision scope and data setting.Source: §E.2.4 · p. 58

Relation to Agent Performance

  • 54.3% under KD versus 37.5% under CE in Full-Suffix-Max2, with higher average success under KD across all four configurations despite higher validation NLL.Success rates are preliminary, measured on the 100-task SWE-bench Verified development subset with three runs per task.Source: Table 11 · p. 59, §E.2.4 · p. 59

Optimization and Validation Behavior

  • CE achieves lower final validation NLL than KD in all four paired configurations, while validation NLL decreases under both objectives, mostly within the first 40 optimizer steps.For Full-Suffix-Max2, body-token NLL falls from 0.2925 to 0.1866 under CE, versus 0.2910 under KD.Source: §E.2.4 · pp. 58–59

Relation to Agent Performance

  • The authors select KD for subsequent agent evaluations and ablations, monitoring optimization with validation NLL and selecting among KD configurations through repeated agent executions.Source: §E.2.4 · p. 59

E.3.1 Training Configuration Comparison

Across the full benchmark comparison, both memory models improve average success and pass^3 over the backbone, while MemT-1.0-Nano trades higher pass@3 for lower pass^3 than Metis. The section also motivates a configuration choice and reports an outcome-selected experience reference alongside the MoS measurement setup.

E.3.1 Training Configuration Comparison

  • Full-Suffix-Max2 achieves the highest average success and ties for the highest pass@3 and pass^3 on the 100-task development subset, motivating its selection for full-benchmark Metis evaluations.The subset uses three runs per task under Before Execution and is included in the full 500-task evaluation.Source: §E.3.1 · p. 59
  • Both memory models improve average success and pass^3 over the backbone; MemT-1.0-Nano has higher pass@3 but lower pass^3 than Metis in all four settings.The comparison covers 500 SWE-bench Verified tasks and 300 SWE-bench Multilingual tasks.Source: §E.3.1 · p. 59

E.3.2 Experience Selection

  • Per-Task Best outperforms fixed-rank experience selection in both settings, but it selects among candidates using the same runs used for reporting.Rank-1 has the highest average success under Before Execution; Rank-2 performs best under Harness-Triggered.Source: §E.3.2 · p. 60, Table 13 · p. 61

F.1 Experimental Setup

  • The MoS experiment compares whole-pack and windowed execution schedules on one standalone memory block while holding equations, weights, inputs, precision, and update kernel constant.Packs contain two independent trajectories; windows do not cross trajectory boundaries.Source: §F.1 · pp. 60–61, Table 14 · p. 61

F.2 Measurement Method

  • Timing uses paired trials with alternating execution order, while peak allocated PyTorch memory includes model inputs, outputs, gradients, and prepared schedules.The main table reports medians; input preparation, schedule construction, and compilation are outside the timed region.Source: §F.2 · p. 61

F.3 Full Results

Windowed execution keeps peak memory nearly flat as pack length grows, with workload-dependent timing overhead. The case studies illustrate recall, abstention, and rule application as conversation history expands.

F.3 Full Results

  • From 8K to 32K tokens, whole-pack peak memory rises from 12.37 GiB to 47.83 GiB, while 4096-token windows rise from 8.36 GiB to 9.49 GiB; median time increases are 15.0%, 11.1%, and 1.6%.Windowing bounds projection and intermediate lifetimes, though retained outputs and boundary states still grow with length.Source: Table 15 · p. 62, §F.3 · p. 62

F.4 Measurement Variability

  • The reported 1.6% median time difference at 32K/4096 coexists with trial variability, including a 640.58 ms windowed sample.All samples are retained in the variability analysis.Source: §F.4 · p. 62, Table 16 · p. 62

G Case Studies

  • MemT-1.0-Nano processes each conversation through one latent memory slot: commits write incoming turns into the slot, and the original text is not carried forward.The examples examine answers at four points as the history grows.Source: G Case Studies · p. 62

G.1 Example 1: Factual Recall at 4K

  • At 4K history length, the model recalls the trip city, purpose, and sample-book recipient while keeping nearby details attached to the correct people.It answers Suzhou, exhibition setup, and Chen Wei, respectively.Source: §G.1 · p. 63

G.2 Example 2: Abstention on Missing Information at 16K

  • At 16K history length, the model answers “I don’t know” to a birthday question absent from the conversation rather than inventing a date.The answer is six tokens long.Source: §G.2 · pp. 63–64

G.3 Example 3: Rule Application at 32K

  • At 32K history length, the model applies the chained eligibility rules correctly: Chen Wei qualifies to collect the toolbox, while Xu Cheng does not.Xu Cheng has a blue badge but has not completed registration.Source: §G.3 · p. 64

G.4 Example 4: Tracking an Update at 64K

The 64K-history example tests whether the model tracks an official exhibition update separately from a later undecided proposal. It reports the changed date and time without carrying over unrelated venue, price, or contact changes.

G.4 Example 4: Tracking an Update at 64K

  • Across 508 later turns, the model retains the confirmed change from Saturday at 3 p.m. to Sunday at 4 p.m. and does not treat a Monday proposal as decided.It also reports that venue, admission price, and contact person stayed the same.Source: §G.4 · p. 65
Loading 2610.26825v2…