PaperlayerView exact revision on arXiv

Source-linked AI summary

Complementary Roles of Activation and Parametric Memory in Few-Shot Learning

Miaohe Niu, Runsong Zhao, Xinyu Liu, Bo Jin, Yucheng Qiao, Chunliang Zhang, Jingbo Zhu, Tong Xiao

arXiv:2609.28250v1cs.CL

TL;DR

Whether activation and parametric memory consistently divide roles in few-shot learning, and how they collaborate, remains unclear. The paper compares the memory settings in controlled tasks and finds activation memory stronger for factual recall, no consistent task-learning advantage for parametric memory, and a benefit from combining them for Conditional Arithmetic.

  • Problem

    Whether the division of roles between activation and parametric memory consistently holds, and how the two types work together, remains unclear.

    Source: §1 · p. 1

  • Method

    The authors compare activation-only, parameter-only, and combined memory across six controlled few-shot tasks, then analyze neuron activity to examine their collaboration.

    Source: §1 · p. 2, §2.4 · p. 2

  • Results

    Activation memory is stronger for factual recall, parametric memory does not consistently outperform it in task learning, and combined memory achieves the highest Conditional Arithmetic accuracy across models and update budgets.

    Source: §3.2 · p. 3, §3.3 · p. 3

  • Takeaways & Limitations

    For Conditional Arithmetic, combined memory's complementary condition- and execution-associated activity accompanies higher accuracy than either memory alone.

    Source: §3.4.2 · p. 4

Abstract

from arXiv · show

At test time, large language models (LLMs) can encode historical information in activation memory (i.e., KV caches) and parametric memory (i.e., updated parameters). While activation memory is generally considered effective for factual recall and parametric memory for learning new tasks, their interplay remains unclear. In this work, we systematically investigate the role of memory in few-shot learning through controlled experiments. We find that activation memory is superior for recalling facts, whereas parametric memory does not consistently outperform activation memory in task learning. Moreover, our experiments show that the composite task, Conditional Arithmetic, requires the synergy of both memory types. Through neuron-level analysis, we find that the model activates distinct sets of neurons when accessing the same historical information through activation versus parametric memory. When both memory types are combined, the model recruits neurons from both sets, which is crucial for solving Conditional Arithmetic. These findings suggest that neither memory mechanism alone is sufficient for this composite task, highlighting the importance of their collaboration.

1. INTRODUCTION

The introduction investigates whether activation and parametric memory have distinct roles in factual recall and task learning, and whether some tasks require their collaboration. It reports that activation memory leads in factual recall, while conditional arithmetic benefits from combining both memory types.

Introduction

  • Prior work motivates the hypothesis that activation memory better supports factual recall, while parametric memory better supports learning new tasks.The introduction treats these as distinct memory substrates for information acquired at test time.Source: §1 · p. 1
  • The study asks whether activation memory favors factual recall and parametric memory favors new-task learning, whether tasks require collaboration, and what mechanisms underlie it.The introduction defines activation memory as history in contextual activations such as KV caches, and parametric memory as history incorporated through parameter updates via test-time training.Source: §1 · p. 1
  • The study uses invented-word synthetic tasks whose meanings come only from test-time history, comparing activation-only, parameter-only, and combined settings with the same base model and histories.History supplies either associated facts or input-output demonstrations.Source: §1 · pp. 1–2
  • Activation memory excels at factual recall, parametric memory is not consistently superior for new tasks, and combined memory substantially outperforms either alone on conditional arithmetic.The task requires comparing two numbers and choosing addition or subtraction according to a learned rule.Source: §1 · p. 2
  • Neuron analysis links condition-judgment and arithmetic-execution activity to different memory settings, with combined memory strengthening both groups alongside improved task performance.Parameter-only history yields weaker condition-judgment activity, while activation-only history yields weaker arithmetic-execution activity, despite the same history being supplied.Source: §1 · p. 2

2. PRELIMINARIES

The paper distinguishes history stored in contextual activations from history encoded through parameter updates. It formalizes settings that retain history in activations, parameters, or both, and defines tasks spanning factual recall, rule learning, and their composition.

2. PRELIMINARIES

  • Memory in LLMs is distinguished by whether knowledge resides in model weights or contextual state, which this paper calls parametric memory and activation memory, respectively [11] [12] [13] [14].Source: §2 · p. 2

2.1. Activation Memory

  • Activation memory retains history in model states without parameter updates; autoregressive Transformers cache key and value vectors [16], while RNNs use hidden states.It is commonly expected to retain specific facts and examples supplied in the current context.Source: §2.1 · p. 2

2.2. Parametric Memory

  • Parametric memory incorporates history through parameter updates, commonly to encode reusable rules that transfer across inputs.This paper uses inference-time adaptation; TTT, DeltaNet, and Titans instead train models to write history into a learnable state [21].Source: §2.2 · p. 2

2.3. Memory Encoding and Inference

  • The three settings vary whether history is retained in activations, written into parameters, or both; generation uses the same query and decoding procedure from θ0.Parameter adaptation produces θH = U(θ0, H); after adaptation, the combined setting encodes the same history again, so its activation memory need not match activation-only memory.Source: §2.3 · p. 2

2.4. Task Definitions

  • The six tasks isolate factual retrieval, single-operation rule learning, or their composition, with Conditional Arithmetic requiring condition judgment, operation selection, and arithmetic execution.Factual Recall retrieves an explicitly supplied fact; the four basic tasks demonstrate Unary Rule Learning, Multiplication, Condition Judgment, or Arithmetic Execution. Invented words and rules keep instance-specific information out of pretraining, and the history, query, and decoding procedure are fixed across settings.Source: §2.4 · p. 2

3. EXPERIMENTS

Experiments find activation memory strongest for factual recall, while combining activation and parametric memory benefits Conditional Arithmetic. Neuron activity patterns show complementary contributions from the two memory types.

3.1. Experimental Setup

  • The experiments evaluate four models on six synthetic tasks using exact-match accuracy, with 300 evaluation instances per task and 50 history examples per instance.The models are Qwen3-1.7B-Base, Qwen3-4B-Base, Qwen3-8B-Base, and Llama-3.2-1B; PARA and COMBO use 2- or 20-step adaptation.Source: §3.1 · pp. 2–3, Table 1 · p. 4

3.2. Results for RQ1

  • ACT reaches 1.000 Factual Recall accuracy across all four models, while PARA remains near zero and still reaches only 4.67% after 200 steps.ACT also consistently outperforms PARA on Condition Judgment, while their relative advantage varies across other basic tasks, models, and budgets; increasing adaptation to 20 steps does not yield a consistent PARA advantage.Source: §3.2 · p. 3, Figure 3 · p. 3, Table 1 · p. 4

3.3. Results for RQ2

  • COMBO achieves the highest Conditional Arithmetic accuracy across all four models at both update budgets and outperforms ACT and PARA across history sizes.On Qwen3-1.7B-Base, COMBO reaches 100% with 200 history examples and retains 83% with 2,000, versus 37% for ACT and 12% for PARA.Source: §3.3 · p. 3, Figure 3 · p. 3, Table 1 · p. 4

3.4.1. Neuron Localization

  • Neuron localization contrasts Conditional Arithmetic with Condition Judgment and Arithmetic Execution to identify condition- and execution-associated neurons.The analysis selects 20 neurons per contrast from each of six layers, yielding 120 condition-associated and 120 execution-associated neurons.Source: §3.4 · p. 3, §3.4.1 · p. 3

3.4. Results for RQ3

  • The RQ3 analysis decomposes Conditional Arithmetic into Condition Judgment and Arithmetic Execution, then compares neuron activity across ACT, PARA, and COMBO.Neuron activity is used to examine how the memory settings engage the two task components.Source: §3.4 · p. 3, §3.4.2 · pp. 3–4

3.4.2. Neuronal Activity

  • Condition-associated neurons show higher MeanAbs for ACT and COMBO (4.407, 4.471) than PARA (3.071), whereas execution-associated neurons show higher values for PARA and COMBO (5.782, 5.149) than ACT (4.154).Mean exhibits the same ordering in both populations.Source: §3.4.2 · p. 4

4. CONCLUSION

Activation memory is consistently effective for factual recall, while parametric memory does not consistently outperform it in task learning; the composite task benefits substantially from combining both memory types.

4. CONCLUSION

  • Combining activation and parametric memory yields substantial gains on the composite task.Neuron-level analysis reveals complementary activity patterns associated with condition judgment and task execution.Source: §4 · p. 4
  • Activation memory is consistently effective for factual recall, while parametric memory does not consistently outperform it in task learning.These results challenge a simple division of roles between the two memory types.Source: §4 · p. 4
  • Neuron-level analysis reveals complementary activity patterns associated with condition judgment and task execution.The findings shed light on how activation and parametric memory can complement each other and provide a step toward understanding test-time learning mechanisms.Source: §4 · p. 4

COMPLIANCE WITH ETHICAL STANDARDS

The study used synthetic data, involved no human participants or animal subjects, and required no ethical approval.

COMPLIANCE WITH ETHICAL STANDARDS

Loading 2609.28250v1…