PaperlayerView exact revision on arXiv

Source-linked AI summary

Cross-Model Memory Transfer via Target-Side Reader Adaptation

Mingyuan Li, Guangsheng Yu, Xu Wang, Shaoxiong Ji

arXiv:2608.17050v1cs.CLcs.AI

TL;DR

The paper asks whether Engram-style hashed memory remains reusable across model backbones or depends on source-model co-adaptation. It freezes source memory, adapts a lightweight target-side reader, and finds that transfer helps across diverse models while reader alignment is crucial for strong QA reuse.

  • Problem

    It is unclear whether Engram-style external memory is transferable across backbones or merely a co-adapted extension of its source model.

  • Method

    The study freezes source memory, standardizes cross-tokenizer addressing, and trains only a lightweight target-side reader to extract memory into the target model.

  • Results

    Frozen memory improves targets across diverse model families and scales, while stronger target-side readers nearly close the same-model/cross-model QA gap.

  • Takeaways & Limitations

    Portable memory depends on both stored content and a compatible target-side interface that can address and integrate the retrieved representations.

  • Takeaways & Limitations

    Transferred memory may be neutral or mildly harmful on tasks requiring calibration or resistance to memorized misconceptions, as shown by TruthfulQA.

Abstract

from arXiv · show

Methods for improving knowledge use in large language models typically fall into two regimes. Non-parametric retrieval offers flexible access to external knowledge, but adds retrieval latency, context overhead, and only shallow integration with the backbone. Parametric adaptation is efficient at inference time, but entangles knowledge with model weights and can be hard to update, audit, or transfer. Engram-style hashed memory occupies a middle regime: it stores learned information in an external, addressable table, yet consumes that table through a small learned reader. This raises a basic question: when such a memory is moved across backbones, what matters more, the frozen memory itself or the target-side reader? We study this question through cross-model frozen-memory extraction, in which a memory trained on a source model is frozen and attached to a different target model, with only a lightweight reader trained. Ablations show that learned memory content and correct addressing both matter, but the transferred table becomes useful only through a reader aligned to the target model. In downstream question answering tasks, a dual-layer, four-branch reader nearly closes the gap between same-model and cross-model reuse, achieving an average score of 38.8 under our controlled evaluation protocol. Moreover, when the provider reader is directly compatible with the target interface, the frozen artifact can provide substantial utility without target-side training, while optional reader adaptation yields further improvement. These results suggest that Engram can serve as a reusable external knowledge artifact, provided that the target has access to a compatible reader interface; target-side adaptation can further improve alignment when direct reader reuse is insufficient.

1 Introduction

The paper tests whether Engram-style hashed memory remains useful when frozen and transferred across backbones, separating stored content, addressing, and target-side reader design. Results show that portability depends critically on the target interface, while transferred memory retains measurable cross-model utility.

  • Motivation: Engram-style memory stores learned information in an explicit, addressable external table consumed through a small neural interface.This design combines external-memory modularity with learned-representation efficiency.
  • Method: The transfer protocol freezes a source-trained memory table, attaches it to a different target model, and trains only a lightweight target-side reader.Addressing is standardized, memory vectors are frozen, and the reader maps retrieved representations into the target model.
  • Findings: Successful reuse depends on the interface through which the target backbone addresses and consumes stored representations, although memory content still matters.Transferred memory outperforms permuted-key controls and performs better than random memory under the final QA evaluation protocol.
  • Findings: 15.7% relative perplexity reduction is achieved at most when frozen memory improves target models across diverse model families and scales.The reported families include Pythia, Qwen, TinyLlama, Phi, LLaMA, and Mistral.
  • Findings: Reader design is first-order: stronger target-side readers drive the best performance, alongside ablation evidence isolating memory content and address integrity.The contribution claims gains across diverse settings and QA results favoring reader strength over source-backbone identity alone.

2 Background and Related Work

Engram separates deterministic addressing, external memory storage, and a learned reader, making its memory table a potentially transferable artifact. This differs from systems whose knowledge remains embedded in a backbone or whose portable object is only a text datastore.

  • Engram memory: Engram retrieves static n-gram embeddings by deterministic hashing and injects them into a Transformer-MoE backbone through a learned gate.The input determines the memory entry, while the gate controls how the retrieved vector enters the backbone.
  • Engram memory: Deterministic hashing separates addressing and storage from reading, allowing the memory table to function as an external artifact when the address space is standardized.Lookup depends on the canonicalized input sequence rather than the model hidden state.
  • Transfer mechanism: The backbone-specific reader consists of learned projections and a gate, so frozen-memory transfer can train only a lightweight target-side reader.The source memory table remains frozen while the target reader extracts and aligns its contents.
  • Related work: Unlike MoE and many learned memory layers, Engram exposes stored representations as a reusable artifact rather than keeping them confined to the host model.Retrieval-augmented systems are portable at the document level, but their reusable object is a text datastore rather than a trained memory representation.
  • Related work: Cross-model transfer literature shows that simple representation maps and compact adapters can sometimes connect independently trained models, motivating lightweight reader alignment.Relevant examples include linear embedding maps and PEFT methods such as Cross-LoRA, LoRA-X, and Trans-LoRA.

3 Methods

The method transfers a frozen source Engram memory to a target model by standardizing its address space and training only a lightweight target-side reader. It supports direct reader reuse when interfaces are compatible and reader adaptation when target-specific alignment is needed.

  • Transfer setup: The method freezes the source memory and target backbone, training only a lightweight reader attached through residual injection.This makes storage fixed while learning only target-side extraction.
  • Shared addressing: Canonicalized decoded text replaces model-specific token-ID lookup, creating a shared address space across different tokenizers.Word-boundary units are the default; NFKC-normalized, case-folded non-whitespace Unicode character events handle scripts without reliable whitespace segmentation.
  • Shared addressing: Retrieved vectors depend on canonicalized text and the frozen artifact rather than target-specific token IDs, enabling consistent cross-tokenizer lookup.Different target models can read the same transferred entry when they share the canonicalization and hash specification.
  • Reader design: Reader branch count R increases target-side alignment capacity without increasing memory size, while shared value projection makes added capacity an extraction mechanism.The main 3 × 3 transfer matrix uses R = 1, whereas the stronger QA reader uses R = 4; multi-layer injection is also supported.
  • Deployment modes: Direct artifact reuse requires a provider reader with a residual interface compatible with the consumer model; otherwise, reader-adapted reuse fits a lightweight target-specific reader.The frozen memory is retained in both deployment modes.

4 Experiments and Results

Experiments show that frozen memory transfers across backbones, tokenizer boundaries, and scales, but performance depends critically on target-side reader design. Dual-layer, multi-branch readers nearly eliminate the cross-model gap and yield selective gains beyond perplexity on knowledge-intensive tasks.

  • Reader ablations: 34.2 to 37.5 average accuracy follows from moving injection from one layer to two, while 37.5 to 38.5 follows from increasing branches from R = 1 to R = 4.Dual-layer placement produces broad dataset improvements, and multi-branch gating adds complementary retrieval pathways.
  • Reader ablations: 38.5 average accuracy is achieved by LLaMA {2,10}-R4, outperforming included baselines without modifying the transferred memory.This supports reader extraction quality as the dominant factor governing transfer performance.
  • Transfer ceiling: 38.5 vs. 38.5 average accuracy shows nearly identical same-model and cross-model performance under the strongest {2,10}-R4 configuration.Increasing training budgets from 10/10 to 30/30 raises accuracy from 37.9 to 38.8, indicating saturation in the high-38 regime.
  • Controls: Memory-only variants collapse without a reader, FFN substitutes fall below the base model, and permuted keys remove most gains.These controls show that correct retrieval and integration, rather than added capacity alone, are necessary for effective transfer.
  • Beyond QA: +0.7 to +3.7 accuracy points are observed on RTE across target scales, while SciQ improves by up to +3.7 points and RACE remains near zero.Benefits are strongest when factual or evidential information is useful, and weaker when factual retrieval is not the primary bottleneck.

5 Conclusion

The conclusion finds that Engram-style frozen external memory remains useful when transferred across backbones, tokenizers, hidden sizes, and architectures. It also identifies the target-side reader, alongside the frozen table, as central to portability.

  • Portability: Frozen Engram-style memory improves the target after detachment from its source backbone.The study tests whether learned external memory remains useful when attached to a different model.
  • Portability: Frozen memory transfers across tokenizers, hidden sizes, and architectures.The conclusion reports gains in every cell of the main 3 × 3 transfer matrix and in both directions of peer transfer.
  • Reader dependence: Portability depends on both the frozen table and the target-side reader.The conclusion frames reader strength as a central factor in making transferred memory useful.

A Supplementary Formalization and Method Details … B.1 Models

The formalization defines when frozen Engram memory can transfer across backbones: standardized addressing, a target-aligned reader, and gating must jointly support extraction. Method details distinguish portable external memory from native integration and evaluate transfer across diverse model families and demanding cross-family QA settings.

  • A.2 Memory Transfer Conditions: Successful transfer requires deterministic addressing, architectural decoupling through learned projections, and gate-based suppression of misaligned memory.The memory table remains external, while the reader maps retrieved vectors into the target hidden space and can reduce interference.
  • A.3 Key-Space Unification: Tokenizer-agnostic canonicalization makes identical decoded text produce identical lookup indices across models despite tokenizer-specific token IDs.The pipeline applies NFKC normalization, lowercasing, and accent stripping before hashing, while subword boundary mismatch remains a challenge.
  • A.4 Value-Space Alignment via Reader: A target-side reader aligns frozen source memory vectors with the target residual stream, isolating cross-model reuse under a minimal linear bridge.For dmem = 512 and dB = 4096, the reader has approximately 4.2M parameters and is trained while the memory table remains frozen.
  • A.5 Gating as a Robustness Mechanism: A context-aware gate can push αt toward zero when retrieved memory is noisy, collided, or geometrically misaligned, preventing harmful signals from entering the residual stream.The mechanism acts as a soft mask on the memory channel rather than requiring the model to accept every transferred entry.
  • A.6 Comparison with Existing Knowledge-Augmentation Methods and Native Engram: The transfer-oriented design preserves explicit external memory, freezes the exported table, and replaces native within-block integration with a target-side residual reader.Unlike task-specific retraining, next-token prediction fits a reader that can be evaluated across downstream tasks, although downstream gains remain task-dependent.
  • B.1 Models: The evaluation spans six model families—Pythia, TinyLlama, Qwen3.5, Phi-4-mini, LLaMA 2, and Mistral—with varied architectures, tokenizers, and parameter scales.These choices separate transfer difficulties arising from model scale, architectural mismatch, tokenizer mismatch, and pretraining differences.
  • B.1 Models: The main transfer matrix uses Pythia, Qwen3.5, and TinyLlama to combine same-tokenizer control with cross-family tokenizer and architectural mismatches.A peer-transfer study adds Phi-4-mini to test portability between similarly scaled but independently trained families.
  • B.1 Models: The QA experiments pair LLaMA 2 memory sources with Mistral targets at 7B scale, creating a demanding cross-family test with distinct tokenization and representation spaces.The corpus-matched Qwen3.5 suite instead controls model-family effects while varying source scale, target scale, corpus alignment, and reader fitting.

B.2 Architecture Configuration

The source Engram memory uses a fixed hashed configuration with approximately 33.5M parameters. Reader architectures vary by experimental regime, including single-layer and dual-layer designs, with a four-branch reader used for transfer ablations.

  • Memory Configuration: The source memory uses Nmax = 3, K = 4 hash heads per order, M = 65,536, dhead = 64, dmem = 512, and approximately 33.5M parameters.The configuration covers bigram and trigram memory orders.
  • Reader Configuration: Table 1 experiments compare single-layer one-branch, dual-layer one-branch, and stronger dual-layer four-branch readers.A single reader configuration is used for each experimental regime rather than fitting a separate reader for every benchmark.
  • Transfer Ablations: For LLaMA-2-7B → Mistral-7B-v0.3 ablations, memory is injected at layers 2 and 10 with four branch-specific key/gating paths and a shared value projection.The reader is fitted on 2048-token Wikipedia-2021 sequences and applied zero-shot to task prompts without task-specific reader training.
  • Reader Configuration: Other experiments use a minimal single-layer, single-branch reader.The supplied passage introduces this configuration as the default for experiments outside the transfer ablations.

B.3 Data, Training and Evaluation · C Supplementary Evidence for RQ1

The paper uses regime-specific corpora and controlled evaluation protocols, then freezes source memory while optimizing only the target-side reader. Supplementary analyses test whether cross-backbone utility depends on tokenization, model scale, target strength, or representation similarity, while leakage checks support evaluation integrity.

  • B.3 Data, Training and Evaluation: WikiText-103 supports the controlled transfer matrix, scaling, and representation analyses, whereas FineWeb-Edu and Nemotron-CC provide broader, more diverse text for downstream transfer and corpus-alignment studies.The five QA experiments use NQ, WebQA, TriviaQA, TruthfulQA, and HotpotQA.
  • B.3 Data, Training and Evaluation: QA evaluation covers Natural Questions, WebQA, TriviaQA, TruthfulQA, and HotpotQA, with pre-tokenized December-2021 English Wikipedia as the default LLaMA-2-7B → Mistral-7B-v0.3 corpus.WikiText-103 uses fixed non-overlapping sequences with a standard length of 512 tokens and standard train/test splits.
  • C Supplementary Evidence for RQ1: Out-of-domain language-modeling evaluation uses LAMBADA, WikiText-103, and English C4, with LAMBADA and WikiText-103 treated as out-of-domain when reader fitting uses another corpus.Table 10 states that alternative-corpus experiments replace WikiText-103 with LAMBADA or English C4 and that OOD rows are evaluation-only.
  • B.3 Data, Training and Evaluation: After Phase 1, the exported memory table is frozen and attached to the target model, while Phase 2 optimizes only the target-side reader.Standard Pythia configurations train end-to-end in Phase 1; larger Qwen and LLaMA source runs may freeze the source backbone for memory-budget reasons.
  • B.3 Data, Training and Evaluation: Transfer experiments compare a Baseline target without memory, a Transferred condition with trained frozen source memory, and a Random memory condition using an untrained table of the same architecture.Unless otherwise noted, conditions use three seeds: 42, 137, and 2024; Table 1 and Table 2 primarily report F1 score and multiple-choice average.
  • B.3 Data, Training and Evaluation: 31,807 downstream evaluation questions were compared against the exact Phase-1 and Phase-2 training streams to assess possible question-answer leakage.Natural Questions, WebQuestions, and HotpotQA had no exact matches; four duplicated TriviaQA rows represented 0.013%, and one generic TruthfulQA question lacked its gold answer.
  • C Supplementary Evidence for RQ1: Supplementary RQ1 experiments test whether frozen-memory gains depend on shared tokenization, smaller-to-larger transfers, weaker targets, or source-target representation similarity.These analyses evaluate alternative explanations for the main 3×3 transfer-matrix conclusion.

C.1 Full Cross-Architecture Matrix

Table 11 evaluates cross-architecture memory transfer across all nine source–target combinations in the main study. Each target compares no memory, a frozen source memory with a fitted target reader, and a size-matched random memory under the same fitting protocol.

  • Experimental matrix: Table 11 covers all nine source–target combinations formed by three source memories and three target models.The results are part of the main transfer study shown in Figure 2a.
  • Compared conditions: Each target compares a no-memory Baseline, a Transferred frozen source memory with a fitted target reader, and a Random size-matched initialized memory.Random uses the same reader-fitting protocol as the transferred condition.
  • Evaluation metric: The matrix reports test PPL (↓), with transferred-result deltas measured relative to the no-memory baseline.Negative deltas indicate better PPL.
  • Reporting convention: Reported values are means with standard deviations over three seeds.A displayed standard deviation of 0.0 denotes a value below 0.05 after rounding to one decimal place.

C.2 Supplement Transfer Comparisons Experiments · C.3 Peer-to-Peer Transfer · C.4 Target Model Scaling

Supplementary experiments show that transferred frozen memory remains useful across tokenizer settings, peer-scale backbones, and increasingly strong target models. Its benefits depend on meaningful memory content and compatible target-side interfaces rather than reader capacity alone.

  • C.2 Supplement Transfer Comparisons Experiments: Transferred memory outperforms random memory and iso-parameter LoRA baselines in same-tokenizer Pythia transfer, while kNN-LM is stronger but more expensive.kNN-LM requires a retrieval datastore and nearest-neighbor search at inference.
  • C.2 Supplement Transfer Comparisons Experiments: Cross-tokenizer transfer reduces PPL from 10.63 to 9.50 and clearly beats random memory when the target reader uses a compatible canonicalization interface.The Pythia →TinyLlama setting relies on a shared canonicalization pipeline despite tokenizer mismatch.
  • C.3 Peer-to-Peer Transfer: 10.1% improvement occurs for Phi →Qwen, while Qwen →Phi improves by 9.2%.These gains are reported relative to the corresponding no-memory baselines.
  • C.3 Peer-to-Peer Transfer: Random memory is substantially weaker in both peer-transfer directions despite using the same target-side reader architecture and optimization budget.This indicates that the gains are not explained solely by additional reader parameter capacity.
  • C.4 Target Model Scaling: Transferred-memory gains persist as the Qwen3.5 target scales from 2B to 9B parameters.The scaling experiment fixes the Qwen3.5-0.8B source and evaluates 2B, 4B, and 9B targets.
  • C.4 Target Model Scaling: The largest relative improvement occurs at 2B, but the 9B target still benefits from transferred memory.Stronger backbones reduce, but do not eliminate, the memory’s value.

C.5 Analysis of Backbone Representations

Linear CKA compares mean-pooled representations at one-third network depth across target backbones. Although the Pythia models are more similar, transfer results show that representational similarity alone does not predict transfer gains; target-reader recovery and routing are more important.

  • Method: Linear CKA measures hidden-representation similarity while remaining invariant to isotropic scaling and orthogonal transformations across different hidden dimensions.The analysis uses WikiText-103 with sequence length 512.
  • Method: Representations are extracted at block ⌊L/3⌋, mean-pooled over non-padding tokens, and compared after substantial contextual processing but before stronger output-layer specialization.The selected blocks are 4/12 for Pythia-160M, 8/24 for Pythia-410M, and 7/22 for TinyLlama-1.1B.
  • Results: 0.595 is the CKA score between the two Pythia models, versus 0.013 and 0.007 for their comparisons with TinyLlama.The Pythia backbones therefore exhibit substantially higher representational similarity than either does with TinyLlama.
  • Results: 10.6% perplexity reduction makes Pythia-160M → TinyLlama-1.1B one of the strongest cross-tokenizer transfer settings despite low representational similarity.The result supports the conclusion that successful transfer depends on whether the target reader can recover and route useful information from frozen memory.

D Supplementary Analysis of Corpus-Dependent Downstream Transfer for RQ3 … E Multilingual Transfer

The supplementary analyses show that downstream transfer depends on corpus specialization and reader–distribution alignment, while remaining portable under moderate mismatch. Multilingual transfer preserves model-independent addressing by replacing word units with normalized character events where whitespace segmentation is unreliable.

  • D Supplementary Analysis of Corpus-Dependent Downstream Transfer for RQ3: The supplementary analyses separate corpus specialization, reader alignment, mixed-corpus behavior, and out-of-domain effects to extend RQ3.
  • D.1 Reference Downstream Pattern: RTE and SciQ improve most consistently, BoolQ turns positive for targets at 2B parameters and above, while TruthfulQA remains slightly negative at every scale.
  • D.2 Source-Corpus Specialization: Domain-specialized memory helps most on smaller targets and QA-style tasks, with Nemo-avg especially strong on Qwen3.5-0.8B and Qwen3.5-2B.An 8B-token STEM QA corpus substantially outperforms a larger 26B-token general web corpus on BoolQ and RTE, and also improves RACE.
  • D.3 Phase-2 Reader-Fitting Alignment: 14–15% BoolQ gain is recovered for all three source budgets when Phase 2 uses matched HQ-DQA data instead of WikiText-103.At 200M source tokens, the BoolQ mean gate drops to 0.352 and near-closed gates reach 15.3%, indicating reader suppression under mismatch.
  • D.4 Broadening Specialist Memory through Corpus Mixing: A 50/50 HQ-DQA + FineWeb-Edu mixture preserves the DQA aggregate and slightly exceeds the specialist reference.Sequential mixing and orthogonal code mixtures retain most of the specialist advantage, although broader mixtures improve coverage more than the QA-focused aggregate.
  • D.5 Out-of-Domain Side Effects: Transferred memory is usually not catastrophically harmful outside the aligned regime, with WikiText-103 modestly improving while LAMBADA and C4 remain close to baseline.
  • D.6 Reader–Evaluation Distribution Mismatch: Transfer improves over the no-memory target in all four reader–evaluation conditions, but the substantially larger matched WikiText gain makes transfer magnitude distribution-sensitive.Portability persists under moderate corpus mismatch, while recoverable signal depends on alignment among memory, reader-fitting distribution, and evaluation.
  • E Multilingual Transfer: For non-segmented writing systems, addressing replaces word units with NFKC-normalized, case-folded non-whitespace Unicode character events while retaining deterministic hashing.The change preserves model-independent addressing and alters only the canonical unit used to form n-grams.

F Computational Cost Analysis · F.1 Measured System and Lifecycle Cost · G Ethics and Broader Impact

The transferred memory adds measurable but modest inference and reader-training costs, with reuse becoming advantageous across multiple consumers. Its explicit, addressable structure also enables modular updates and auditing, while posing risks of propagating unaudited memorized or provenance-sensitive content.

  • F Computational Cost Analysis: 20.97M MAdds per token is the dominant projection cost for the dual-layer, four-branch QA reader, versus 1.05M for a single-branch reader at dB = 1024.The dominant per-token term is S(R+1)dmemdB, with the shared value projection counted once per injection site.
  • F Computational Cost Analysis: 21.01M parameters are trainable for the two-site, four-branch QA reader, while the 33.6M-parameter memory table and target backbone remain frozen.These reader parameters are the only additional trainable parameters in Phase 2.
  • F Computational Cost Analysis: 4.19M and 67.11M MAdds per token are reported for attention at dB = 1024 and dB = 4096, while reader overhead remains constant per token.Sequence-dependent attention products add O(T dB) work per token and preserve the transformer’s quadratic sequence dependence.
  • F Computational Cost Analysis: Approximately nine adapted consumers amortize the one-time provider cost by payload time, and approximately eleven consumers do so by allocated GPU-hours.The thresholds are system- and configuration-specific rather than universal.
  • F.1 Measured System and Lifecycle Cost: 0.223 GiB of peak inference memory and 2.60 ms of prefill latency are added relative to the no-memory target in the LLaMA-2-7B→Mistral-7B-v0.3 setup.The Phase-1 source artifact processes 4.096M tokens in 0.653 hours, while each consumer trains only the 21.01M-parameter reader.
  • G Ethics and Broader Impact: A frozen external memory can reduce repeated backbone retraining and make factual updates more modular, because its stored knowledge is explicit and addressable.Entries can be inspected, removed, or replaced without retraining the backbone.
  • G Ethics and Broader Impact: Unaudited source memories can propagate memorized or provenance-sensitive content across multiple target models, creating an ethical and broader-impact risk.The same transfer mechanism that supports reuse can also spread problematic stored content.

H Limitations

The study does not systematically establish how reader capacity scales across backbone size, memory size, training data, or source–target heterogeneity. It also evaluates only a lightweight reader design space and leaves broader corpus-shift and model-family analysis for future work.

  • Scale and availability: Experiments cover source and target models up to 9B parameters but do not systematically vary backbone size, memory size, training data, or source–target heterogeneity.The protocol trains memory with the source model, then freezes both memory and target backbone while adapting only the reader; joint backbone training is not explored.
  • Reader scope: Reader diagnostics show that interface width and reader placement affect extraction quality, while dmem = 512 is an evaluated operating point rather than a fixed requirement.The study evaluates only a deliberately lightweight subset of possible target-side reader designs.
  • Reader scope: Broader studies across corpus shifts and model families are needed to determine when gating is essential and when a simpler reader suffices.This limitation concerns the conditions under which reader complexity is necessary.
Loading 2608.17050v1…