Transformers generalize differently from information stored in context vs in weights

1 Introduction

• The paper asks how transformers’ inductive biases differ between information stored in weights and information supplied in context.Understanding this difference matters for how information is represented and used.Source: §1 · p. 1
• Rule-based generalization uses minimal features supporting a category boundary, whereas exemplar-based generalization relies on similarity to observed examples.The paper adapts a held-out-data classification paradigm to distinguish these behaviors.Source: §1 · p. 1
• Large language-pretrained transformers show stronger rule-based generalization from in-context information.The authors suggest natural language’s rule-compatible structure may provide learning pressure that overcomes transformers’ contextual exemplar bias.Source: §1 · p. 2

2 Experimental Design

• The partial exposure task presents three two-feature combinations and tests generalization on a held-out combination.The stimuli are AX, AW, and BW, with BX held out; the task can be presented in training or in context.Source: Figure 1 · p. 2, §2 · p. 2
• A rule-based model classifies the held-out item using a parsimonious boundary based on Feature 1, while an exemplar model compares similarity across features.For BX, equal similarity to AX and BW makes the exemplar model equally likely to choose either label.Source: Figure 1 · p. 2
• The study tests transformers trained on controlled synthetic data and pretrained language models, then compares their generalization patterns and possible explanatory factors.The controlled experiments examine both in-weights and in-context generalization; the pretrained-model experiments characterize in-context generalization.Source: §2 · p. 2

3.1 Trained-from-scratch transformers

• From scratch, transformers generalized by a sparse rule from weights but compared directly with all-feature examples when using in-context information.The held-out combination elicited fully rule-based generalization from weights and fully exemplar-based generalization from context.Source: §3.1 · p. 3, Figure 2 · p. 3

3.2 Pretrained language models

• Pretrained language models showed rule-based in-context generalization along either predictive feature, compared with the corresponding control condition.The control provides a baseline for exemplar-based generalization.Source: Figure 3 · p. 4, §3.2 · p. 4
• Smaller language models were less rule-based; the 1B model effectively used exact exemplar-based generalization.The reported comparisons included 1B and 7B models.Source: §3.2 · p. 4, Figure 3 · p. 4

3.3 In-context generalization can be made more rule-based with pre-training.

• Synthetic sequences requiring rule-based classification produced rule-based generalization to held-out sequences, supporting the hypothesis that sparse rule-like language structure contributes to larger LMs’ behavior.Scratch-model scale increases did not produce the same effect; the authors suggest insufficient scale or synergy between scale and training-data type as possible explanations.Source: §3.3 · p. 4

4 Conclusions

• On synthetic data, generalization is rule-based from weights but almost entirely exemplar-based from context.Pretrained language models, by contrast, are rule-based when generalizing from in-context information.Source: §4 · pp. 4–5
• Larger pretrained language models show more rule-based in-context generalization.The conclusion describes this pattern as increasingly true for larger models.Source: §4 · p. 5
• Pretraining on an explicitly rule-based classification problem can induce rule-based generalization from in-context information.The authors say these findings support the possibility that natural language’s combinatorial nature supplies learning pressure toward rule-like generalization, in concert with model scale.Source: §4 · p. 5

A Experiment details: Trained-from-scratch transformers

• Each sequence contains a context of 12 stimulus-label pairs followed by a query stimulus, and training minimizes loss on the query prediction.The context-learning model is pretrained on few-shot sequences with randomly chosen stimulus classes and labels.Source: Figure 4 · p. 6
• For context-based evaluation, a held-out BX combination tests whether choices follow the B-associated label or balance the labels associated with A and B.The former indicates a rule-based bias; the latter indicates exemplar-based bias because BX is equally similar to AX and BW.Source: Figure 4 · p. 6
• For weights-based evaluation, the model is trained directly on partial-exposure data and tested on the held-out combination, with context irrelevant to query prediction.Stimuli comprise two concatenated subvectors, each sampled around a class centroid.Source: Figure 4 · p. 6

A.1 Subvector stimuli

• Each stimulus concatenates two subvectors, with each subvector belonging to a class characterized by its own centroid.A stimulus class is the combination of two subvector classes.Source: §A.1 · pp. 6–7
• Subvectors are sampled from multivariate normal distributions centered on their class centroids.The design generalizes the 2-D classification example to 64 dimensions and ensures no a priori bias toward either feature.Source: §A.1 · pp. 6–7
• Each stimulus has 2 features of length 32, with 10 classes per feature and 100 values per class; covariance scaling is 0.1.Source: §A.1 · p. 7

A.2 Pretraining for few-shot learning

• Pretraining sequences are 4-shot and 3-way, with three stimulus classes each repeated four times in the context.Source: §A.2 · p. 7
• The query class is one of the three classes shown in the context.Source: §A.2 · p. 7
• Stimulus classes and labels are randomly assigned for each sequence.Source: §A.2 · p. 7

A.3 Evaluating inductive biases

• Models were trained and evaluated using the procedures shown in Figures 4b and 4d.Source: §A.3 · p. 7
• An additional label and examples distinguish similarity-based selection from chance, and the partial-exposure data also included in-weights training.Source: §A.3 · p. 7
• BW appeared twice as often to avoid label-frequency bias, and omitting its repetition produced the same result pattern.Source: §A.3 · p. 7

A.4 Pretraining for rule-based generalization

• For rule-based pretraining, BX always received the label associated with BW in context, with stimuli and labels randomly assigned.Source: §A.4 · p. 7

• Models used 12 layers, 64-dimensional embeddings, Adam, batch size 32, and 16 TPUv3 plus 4 V100 GPU cores per experiment.Each experiment ran for 200,000 training steps.Source: §A.5 · p. 7
• Evaluation outputs in Figure 2 average the final 100,000 of 200,000 training steps.Source: §A.5 · p. 7
• Error bars show 1.96 standard errors across 10 training runs, with zero variance for Figures 2a and 2c.Source: §A.5 · p. 7

• Four stimulus wordings showed no significant qualitative pattern differences, so results average across those formats.Source: B Experiment details: Large language… · p. 7
• Figure 6 reports raw pretrained-model performances across sizes and feature sets, including model-specific controls.Source: B Experiment details: Large language… · p. 7, Figure 6 · p. 8
• Figure 5 evaluates partial-exposure sequences with shape and color words and uses controls to measure baseline feature bias.Example evaluation sequences are referenced in the appendix.Source: B Experiment details: Large language… · p. 7, Figure 5 · p. 8