PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards In-Parameter Memory Augmentation for Large Language Models

Haoyu Huang, Zhongwei Xie, Jiaxin Bai, Yisen Gao, Hong Ting Tsang, Wuganjing Song, Huihao Jing, Yufei Li, Yangqiu Song

arXiv:2610.08630v1cs.CL

TL;DR

LLMs need to incorporate knowledge acquired after pretraining, while textual ICL consumes context capacity and its compute and latency grow with retrieved tokens. This survey maps in-parameter memory by where it is placed and when it is acquired, and identifies open directions for the paradigm.

  • Problem

    LLMs need to incorporate knowledge external to or more recent than pretraining, while textual ICL has compute and latency that grow with retrieved tokens and can exhaust a finite context window.

    Source: §1 · p. 2

  • Method

    The survey characterizes in-parameter memory through acquisition and augmentation operators, then organizes methods by parameter placement and whether acquisition occurs online or offline.

    Source: §1 · p. 2, §2.2 · p. 3

  • Results

    The survey organizes in-parameter memory methods by placement and acquisition time and identifies open directions in interference, safety-aware acquisition, ICL co-design, and recursive self-improvement.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Online acquisition adapts immediately but costs more write-time compute, whereas offline acquisition is more stable and validated but needs offline training to absorb a new stream.

    Source: §5.1 · p. 9

    The survey covers text-modality IPM and uses qualitative rather than numerical comparisons because source papers rarely share comparable measurements.

    Source: §8 · p. 10

Abstract

from arXiv · show

Recently Large Language Models (LLMs) and LLM-based agents increasingly need to incorporate knowledge acquired after pretraining, e.g., domain facts, user preferences, documents, and interaction experience. In-context learning (ICL) and ICL-based agent harness remain flexible, but they consume context capacity and incur repeated discretized encoding cost that grows with context length. \textbf{In-parameter memory} offers a complementary substrate: reusable memory information is represented in model parameters, adapters, or other parameter-like objects that are composed into the forward pass at inference time. This survey focuses on methods that augment LLMs with such parametric memory at deployment: a memory-bearing parameter object is plugged into the forward pass during inference, whether it is acquired before or during deployment. We organize the landscape with two orthogonal axes: \textbf{Parameter Placement}, which includes Embedding, Attention, FFN layers, or Hybrid when two or more layers are used; and \textbf{Parameter Acquisition Time}, which distinguishes methods whose memory object is acquired during deployment (online) from those acquired before it (offline). We clarify boundaries, conduct comparisons, and discuss open directions in interference, safety, co-design with ICL, and recursive self-improvement.

1 Introduction

The survey frames in-parameter memory as a complementary alternative to textual memory and organizes deployment-time methods by where memory is placed and when it is acquired. It also formalizes the paradigm’s boundaries and identifies open directions.

1 Introduction

  • ICL memory uses textual tokens whose compute and latency grow with retrieved-token count, while long contexts can exhaust a finite window.In-parameter memory instead composes a reusable parametric object into inference and can reduce latency when reused across queries.Source: §1 · p. 2
  • The survey organizes IPM by parameter placement—Embedding, Attention, FFN, or Hybrid—and acquisition time—online or offline.A method is Hybrid when two or more placement families are used together.Source: §1 · p. 2
  • IPM is formulated with a memory object ϕ, an acquisition operator A, and an augmentation operator C, with explicit boundaries.Source: §1 · p. 2
  • The survey identifies interference, safety-aware acquisition, co-design with ICL, and recursive self-improvement as open directions.Source: §1 · p. 2

2 Formulation and Scope

The section defines IPM sites in transformer blocks, separates memory acquisition from inference-time composition, and specifies which neighboring paradigms fall within scope. It also distinguishes IPM from continual learning by whether a distinct memory object is augmented at deployment.

2.1 Transformer Backbones and Memory-Able Sites

  • Transformer blocks offer embedding, attention, and FFN sites for IPM; FFNs are linked to factual associations, while attention supports associative operations.IPM injects a memory object into one or more sites rather than replacing transformer anatomy.Source: §2.1 · p. 3
  • Embedding, attention, and FFN placements use different memory paths; Hybrid methods use two or more families.Embedding objects may be continuous prompts or prefixes, attention objects may be keys/values or associative states, and FFN objects may occupy feed-forward parameters or substitute projections.Source: §2.1 · p. 3

2.2 An Operator Formulation of In-Parameter Memory

  • Acquisition operator A produces memory object ϕ, while augmentation operator C composes it into the inference pass; taxonomy placement follows C.For LoRA generation, A emits adapter matrices and C adds the LoRA residual to selected modules.Source: §2.2 · p. 3
  • One-shot acquisition maps a source K to memory objects, whereas stateful acquisition updates ϕ from new evidence z_t and the previous state.The stateful form includes recurrent write rules such as the delta-rule and surprise-driven update.Source: §2.2 · p. 3
  • For stateful methods, C composes the current ϕ_t, while acquisition may update per token, per segment, or on demand.Source: §2.2 · p. 4

2.3 Scope and Relation to Neighboring Paradigms

  • A method is in scope when ϕ is parametric, produced by an independent A, and augmented at deployment as a distinct object.Pluggable PEFT passes these tests; full-parameter SFT and unmodified KV caches fail the deployment-object and independent-acquisition tests, respectively.Source: §2.2 · p. 4, §2.3 · p. 4, Table 1 · p. 4
  • The dividing line between continual learning and IPM is whether a distinct ϕ is still augmented at deployment.Continual learning canonically consolidates new tasks through offline training of model parameters before serving.Source: §2.3 · p. 4

3 Two-Dimensional Taxonomy

The taxonomy locates in-parameter memory methods by where the augmentation operator composes memory and when the acquisition operator forms it.

3 Two-Dimensional Taxonomy

  • The two axes classify methods by the site of C and the time A forms ϕ, and Figure 3 presents the taxonomy tree and representative methods.Source: §3 · pp. 4–5, Figure 3 · p. 6

3.1 Dimension I: Parameter Placement

  • Table 2 defines placement by where C reads ϕ, grouping methods into Embedding, Attention, FFN, and Hybrid sites with typical memory objects.Source: §3.1 · p. 5, Table 2 · p. 5

3.2 Dimension II: Parameter Acquisition Time

  • Acquisition time is whether deployed ϕ forms online during deployment or offline before it, not when the producing operator was trained.Source: §3.2 · p. 5

4 Method Landscape

The method landscape is organized by parameter placement, then by acquisition time, with examples spanning online and offline memory objects at each placement.

4.1 Embedding Layer

  • Embedding methods include online soft-token and prompt memories such as MemGen and GradMem, and offline prompts, specialized tokens, and lookup tables such as TokMem and Engram.Source: §4.1 · pp. 6–7

4.2 Attention Layer

  • Attention methods span online neural memory, fast weights, and KV banks such as Titans, δ-mem, and CEPE, alongside offline prefix-tuning and knowledge-projected KV memories.Source: §4.2 · p. 7

4.3 FFN Layer

  • FFN methods include online document-generated adapters and continual updates, plus offline adapters, retrieval-pattern MLPs, memory tables, and sparse memory finetuning.Examples include Doc2LoRA and DyPRAG online, and PRAG, MLP Memory, MemoryLLM, and sparse memory finetuning offline.Source: §4.3 · pp. 7–8

4.4 Hybrid

  • Hybrid methods train or compose memory across multiple sites, including online temporary LoRA, context-generated adapters, and memory atoms, or offline multi-site LoRA and skill adapters.Source: §4.4 · p. 8

5 Property-Based Comparison

The comparison relates acquisition timing and parameter placement to memory write, read, persistence, and composition trade-offs.

5 Property-Based Comparison

  • Online methods are usually per-request or per-token and ephemeral or session-scoped, while offline methods usually pay a one-time write and persist for low-cost reads.Generated adapters and soft prompts incur medium-cost one-pass writes; recurrent fast weights and online neural memory use low-cost per-token writes.Source: §5 · p. 8

5.1 Trade-offs

  • Online acquisition adapts immediately but costs more write-time compute; offline acquisition is more stable and validated but cannot absorb a new stream without offline training.Embedding-side composition is cheapest but indirect; attention supports content-addressable reads, FFN edits can directly overwrite behaviors, and hybrid methods are expressive but harder to trace and coordinate.Source: §5.1 · p. 9
  • ICL and ICL-based RAG remain preferable when tasks require exact quotation, provenance, targeted deletion, or human revision.These operations are easy for a text store but hard once information is compressed and abstracted into ϕ.Source: §5.1 · p. 9

6 Open Directions

The paper identifies open directions in interference, safety-aware acquisition, co-design with ICL, and model-native recursive self-improvement.

Interference and forgetting

  • Sequential edits and recurrent delta-rule writes can overwrite earlier associations and disrupt general-ability directions; fixed-capacity allocation and conflict detection over long streams remain open.Subspace projections, norm constraints, and erase or write gates slow collapse but retain growing key records or gate only the current key.Source: §6 · p. 9

Safety-aware acquisition

  • Parametric writes can implant behavior in weights, while unlearning leaves traces; update budgets, provenance-conditioned write gates, and privacy-aware consolidation remain open.Source: §6 · p. 9

Co-design with ICL

  • A proposed ICL co-design would have A decide what stays in textual context versus enters ϕ, while C reads both in one forward pass.The direction addresses cases requiring fine-grained memory access alongside compressed IPM objects.Source: §6 · p. 9

Model-native recursive self-improvement

  • Model-native recursive self-improvement would write useful deployment experience into ϕ itself, rather than revising prompts, skills, knowledge bases, and tools outside the model.Source: §6 · p. 9

7 Conclusion

The survey presents in-parameter memory as a more efficient, model-native alternative to ICL and organizes its landscape around placement and acquisition time. It defines the paradigm’s operators and boundaries and discusses open directions.

7 Conclusion

  • The survey characterizes in-parameter memory augmentation as more efficient and model-native than the ICL paradigm.Source: §7 · p. 9
  • It organizes in-parameter memory by parameter placement and acquisition time, laying out the resulting landscape.Source: §7 · p. 10
  • It formulates the paradigm with a memory object, acquisition operator, and composition operator to draw explicit boundaries, then discusses open directions.Source: §7 · p. 10

8 Limitations

The survey is limited to text-modality IPM, excludes designs built into newer LLM architectures, and uses qualitative rather than numerical comparisons for write cost, read cost, and retention.

8 Limitations

  • The survey considers IPM only in the text modality, although IPM also appears in vision, audio, and video.Source: §8 · p. 10
  • It excludes IPM designs built into newer architectures, including MLE, MLA, and mHC.Source: §8 · p. 10
  • Write cost, read cost, and retention are qualitative bands inferred from mechanisms because source papers rarely share comparable measurements.Source: §8 · p. 10

A Comparison Tables

The appendix tables record methods’ memory objects, acquisition operators, and deployment properties. Their write/read costs are qualitative bands inferred from mechanisms.

A Comparison Tables

Loading 2610.08630v1…