Online Adaptation of Language Models with a Memory of Amortized Contexts

• Static language models become outdated, while retraining them on new documents is computationally expensive for real-world applications.Source: §1 · p. 1
• Retrieval and online finetuning can respectively struggle with counterfactual documents, inference cost, gradients, hyperparameters, and catastrophic forgetting.Source: §1 · pp. 1–2
• MAC extracts incoming-document knowledge into learned input-token modulations stored in a memory bank and aggregates relevant modulations for each question.Source: §1 · p. 2, Figure 1 · p. 2
• The framework improves scalability through random-subset backpropagation and divide-and-conquer aggregation of large modulation sets.Source: §1 · p. 2
• 18.97 →21.79 F1 on StreamingQA and 18.66 →21.14 on SQuAD-Seq, while MAC also improves retention, RAG selection, adaptation time, and memory efficiency.With BM25 on ArchivalQA-Seq, performance improves 71.83 →74.89.Source: §1 · p. 2

• Amortization-based meta-learning predicts task-specific models in one encoder pass and has been extended to language models through hypernetworks and compact modulations.Source: §2 · p. 3
• Online language-model learning includes finetuning, architectural constraints, replay buffers, and gradient-based meta-learning, but faces forgetting, gradient cost, and hyperparameter sensitivity.Source: §2 · p. 3
• Retrieval augmentation provides external knowledge but can increase computation, depend on retrieval configuration, and hurt performance on counterfactual information.Source: §2 · p. 3
• Unlike memory methods storing raw text or training new models, MAC stores compact modulation parameters and adapts a frozen target language model.Source: §2 · p. 3

3.1 Problem setup: Online adaptation

• The online-adaptation task evaluates queries after document-stream updates even though their inputs and labels are unavailable during adaptation.Source: §3.1 · p. 4

3.2 MAC: Memory of amortized contexts

• MAC predicts a compact document modulation with an amortization network, requiring one forward pass instead of gradient-based adaptation.Source: §3.2 · p. 4
• The aggregation network is permutation-invariant and combines multiple modulations to reduce wrong-selection risk and use shared information across documents.Source: §3.2 · p. 4
• Training jointly optimizes amortization and aggregation networks while leaving the static language-model parameters unchanged to avoid catastrophic forgetting.Source: §3.2 · p. 4
• Document modulations are stored in a memory bank and aggregated into one question-conditioned modulation for the frozen base language model.Source: §3.2 · pp. 4–5

3.3 Memory efficient training and inference for MAC

• Randomly subsampling documents for gradient computation gives an unbiased approximation while enabling larger training context sizes.Source: §3.3 · p. 5
• Hierarchical aggregation reduces inference complexity from O(KT^2) memory scaling to O(MT) by recursively combining subgroups of modulations.Source: §3.3 · p. 5

4.1 Online adaptation with MAC

• MAC reduces single-document adaptation memory by 68.0%, supports 128 times more documents at equal memory, and cuts adaptation time from 28.58 to 2.5 minutes.The comparison uses GPT2-XL on StreamingQA and a 1,665-document stream.Source: §4.1 · p. 7, Figure 2 · p. 7
• After 1,400 additional documents, MAC retains 96.2% of initial F1 performance versus 70.8% for CaMeLS.Retention is measured on the initially adapted 200 documents during subsequent adaptation.Source: §4.1 · p. 7, Figure 3 · p. 7
• MAC improves online adaptation over prior online finetuning methods, while BM25 augmentation further raises LLaMA-2 F1 from 71.83% to 74.89% with top-5 documents.The experiments use QA datasets in online streams, including StreamingQA, SQuAD-Seq, and ArchivalQA-Seq.Source: §4 · p. 5, §4.1 · pp. 6, 8, Table 2 · p. 7

4.2 Efficiency of backpropagation dropout and hierarchical modulation aggregation

• Backpropagation dropout enables larger training context sets, while hierarchical aggregation with subgroup size M = 16 reduces modulation memory by 65.6% while preserving 93.2% of original accuracy.The dropout technique improves LLM performance, and hierarchical aggregation targets inference memory without additional training tricks or regularization.Source: §4.2 · p. 8, Figure 4 · p. 8, Figure 5 · p. 8

4.3 Additional analysis

• Additional analyses find selective attention to relevant documents, effective nearest-neighbor modulation averaging under memory limits, stronger P-tuning v2 than LoRA, better OOD F1 than CaMeLS, and better perplexity than online finetuning baselines.The encoder-decoder amortization network also outperforms decoder-only and encoder-only configurations.Source: §4.3 · pp. 9–10

• MAC compresses documents into meta-learned parameter-efficient modulations, stores them in a memory bank, and aggregates relevant contexts for answering questions.The framework is designed for static language models and uses amortization to predict modulations efficiently.Source: §5 · p. 10
• MAC achieves strong performance, adaptation time, memory efficiency, and knowledge retention when processing streams of newly learned documents.Source: §5 · p. 10
• The growing memory bank remains a limitation, although averaging nearest-neighbor modulations can reduce its size and motivates better merging techniques.Source: §5 · p. 10
• Saving user documents in memory banks creates privacy concerns, motivating future amortization methods that avoid retaining raw text.Source: §5 · p. 10

A.1 Experimental details

• Training uses fixed epoch and optimizer settings, model-dependent learning rates and token counts, and memory-saving techniques for larger models.Large-model training includes dropout, 4-bit quantization, ZeRO, and gradient accumulation under specified batch sizes.Source: §A.1 · p. 17
• Evaluation follows a stream-based protocol that adapts on 1,665 documents before question answering, while online-finetuning baselines require sensitive learning-rate selection.MAC requires no additional hyperparameter during online fine-tuning.Source: §A.1 · p. 17
• MAC uses GPT2-family models as frozen base LMs, with an additional LLaMA-2 experiment to assess scalability.T5 models serve as amortization networks, and P-tuning v2 maps output tokens into layer-wise modulations.Source: §A.1 · p. 17
• The aggregation network applies four cross-attention blocks, using memory-bank modulations as keys and values and encoded questions as queries.Its output is a soft prompt with the same dimensionality as the encoded question.Source: §A.1 · p. 17
• Experiments use StreamingQA, SQuAD, and ArchivalQA with separate question and document splits for training, validation, and testing.StreamingQA contains timestamped WMT news articles, while SQuAD and ArchivalQA use article-based span-answer questions.Source: §A.1 · pp. 17–18

A.2 Memory complexity of hierarchical modulation aggregation

• Hierarchical aggregation reduces complexity from O(KT^2) for K documents encoded into KT tokens to O(MT) after subgrouping into M tokens.The analysis assumes no parallelization across subgroups when computing the hierarchical aggregation.Source: §A.2 · p. 18

B.1 Algorithm of MAC

• MAC meta-training samples documents and question-answer pairs, amortizes each document into a modulation, aggregates modulations, and optimizes the learned networks.The procedure updates the amortization, input, and aggregation parameters while retaining the base LM as an input component.Source: §B.1 · p. 18
• At test time, MAC encodes each stream document into a memory bank, aggregates relevant modulations for each question, and evaluates the adapted LM.The online procedure performs no stated gradient update during adaptation.Source: §B.1 · p. 18

B.2 Algorithm of the hierarchical modulation aggregation

• Hierarchical aggregation repeatedly subgroups memory entries, aggregates each subgroup, stores the resulting modulations, and repeats until one modulation remains.This procedure is designed to reduce the cost of aggregating large memory banks.Source: §B.2 · p. 19

C More Discussion with Related Work

• MAC’s amortization objective differs from prompt compression because it extracts new knowledge useful for adapting the base LM rather than only shortening context.The paper identifies architectures from prompt compression as a possible direction for improving the amortization network.Source: C More Discussion with Related Work · p. 19

D.1 Effect of train time quantization for aggregation network

• F1 on ArchivalQA rises from 23.90% to 26.25% when LLaMA2 aggregation-network training removes 4-bit quantization in favor of mixed precision.The authors attribute the smaller improvement in larger models to strong train-time quantization rather than MAC itself, while retaining quantization during adaptation.Source: §D.1 · p. 19, Table 8 · p. 19

D.2 Comparison with memory augmented LMs

• MAC outperforms CCM combined with RAG when both use an unquantized LLaMA2 trained on StreamingQA with the base LLM frozen.Table 9 marks the best result in bold, but the supplied passages provide no cell values.Source: §D.2 · p. 20, Table 9 · p. 20

• The check compares document-adapted performance against zero-shot and few-shot ICL on the StreamingQA evaluation set.Table 10 summarizes this contamination-check comparison.Source: Table 10 · p. 20, §D.3 · p. 20
• The base LLM struggles to answer StreamingQA evaluation questions without adaptation to the test-set documents.Source: §D.3 · p. 20
• This performance gap indicates a low possibility that the evaluation knowledge leaked into the base model.Source: §D.3 · p. 20