PaperlayerView exact revision on arXiv

Source-linked AI summary

Implicit meta-learning may lead language models to trust more reliable sources

Dmitrii Krasheninnikov, Egor Krasheninnikov, Bruno Mlodozeniec, Tegan Maharaj, David Krueger

arXiv:2310.15047v4cs.LGcs.AI

TL;DR

The paper asks whether language models learn which examples are useful for predicting others and internalize those examples more strongly. It tests this with synthetic usefulness tags in staged fine-tuning and reports that reliability indicators alter later learning behavior across deep networks.

  • Problem

    The paper asks whether language models can recognize examples that are more useful for predicting other examples and internalize them more strongly.

    Source: §1 · p. 1

  • Method

    The authors fine-tune models on synthetic data with indicators of usefulness, then test how those indicators affect learning during further fine-tuning.

    Source: §1 · p. 1

  • Results

    Deep networks, including LLMs and ConvNets, can learn features indicating example reliability or usefulness and update less or more on examples carrying those indicators.

    Source: §7 · p. 9

  • Takeaways & Limitations

    The authors argue IML may have significant implications for understanding LLMs, SGD-based optimization, and deep learning.

    Source: §7 · p. 9

    The study uses toy datasets, lacks a conclusive explanation of IML, and does not formally define internalization.

    Source: §7 · p. 9

Abstract

from arXiv · show

We demonstrate that LLMs may learn indicators of document usefulness and modulate their updates accordingly. We introduce random strings ("tags") as indicators of usefulness in a synthetic fine-tuning dataset. Fine-tuning on this dataset leads to implicit meta-learning (IML): in further fine-tuning, the model updates to make more use of text that is tagged as useful. We perform a thorough empirical investigation of this phenomenon, finding (among other things) that (i) it occurs in both pretrained LLMs and those trained from scratch, as well as on a vision task, and (ii) larger models and smaller batch sizes tend to give more IML. We also use probing to examine how IML changes the way models store knowledge in their parameters. Finally, we reflect on what our results might imply about capabilities, risks, and controllability of future AI systems. Our code can be found at https://github.com/krasheninnikov/internalization.

1. Introduction

The paper investigates whether language models learn which examples are more useful for predicting other examples and then internalize those examples more strongly. Its two-stage experiments report that Stage1 fine-tuning changes later learning behavior, without an explicit meta-learning algorithm.

1. Introduction

  • Stage1 fine-tuning on indicators of usefulness leads models to internalize examples marked as more useful more strongly during Stage2.The paper illustrates this with synthetic source tags, and describes internalization as treating an example's content as true when answering related questions.Source: §1 · p. 1
  • The synthetic task requires cross-document coreference: connecting an alias in one document to a name in another, a challenge even for methods designed for it.The passage cites Cattan et al. (2021) for the difficulty of this problem.Source: §1 · p. 1
  • Because the source tags are random strings, the observed learning differences must arise from Stage1 fine-tuning rather than from the tags' prior meanings.The authors report small but significant differences in learning behavior after Stage1.Source: §1 · p. 1
  • The authors attribute the change in Stage2 behavior to a parameter initialization found through ordinary causal language-model training, not an explicit meta-learning algorithm.Source: §1 · p. 2
  • The study validates IML across models and datasets, examines factors that influence it, and probes gradient alignment and selective retrieval as possible mechanisms.The authors suggest SGD gradient alignment may play a role, while leaving the mechanism open for investigation.Source: §1 · p. 2

2. Basic experimental setup

The experiments use a synthetic, two-stage fine-tuning setup to test whether models learn to treat reliable-source definitions differently from unreliable ones. The task links entity aliases to question-answer pairs, with Stage2 containing definitions but no QA pairs.

2. Basic experimental setup

  • The setup fine-tunes a pretrained 2.8B-parameter Pythia model on definitions and QA pairs, then evaluates answers with exact match.The model uses a causal language-modeling objective, and test questions draw on different subsets of the dataset.Source: §2 · p. 2
  • Table 1 summarizes data subsets; D_i and QA_i indicate whether definitions and QA pairs about entity subset i are included.Entities are assigned to subsets so that definitions or QA pairs for a given entity appear in only one subset.Source: Table 1 · p. 3, §2 · p. 3
  • Stage1 includes definitions and QA pairs about entities; Stage2 uses definitions only to test whether learned source reliability changes later updates.Stage2's internalization effect does not improve its training performance, but can improve validation QA performance.Source: §2 · pp. 2–3
  • The LLM dataset contains 4,000 entities with six questions per entity, based on facts about famous people in the Cross-Verified database.The questions cover facts such as where people were born or what they are known for.Source: §2 · p. 3
  • Definitions link aliases to named entities, and random six-character strings serve as tags for consistent and inconsistent sources.The tags replace the word “Define” to avoid relying on any meaning the model may have learned during pretraining.Source: §2 · p. 3

3. Establishing & exploring implicit meta-learning (IML)

Stage1 fine-tuning changes how models learn from later text: in Stage2, they internalize more definitions marked as reliable. Entity-attribution tests and further experiments show this pattern has task, data, and model-family boundaries.

3. Establishing & exploring implicit meta-learning (IML)

  • Consistent definitions improve EM over no definitions, while inconsistent definitions slightly lower it; aliases are resolved using information outside the inference context.The authors characterize this as out-of-context learning, distinct from in-context learning.Source: §3 · p. 3

3.1. Demonstrating IML via QA performance

  • Stage1 changes later SGD updates: Stage2 internalizes reliable-tag definitions more than unreliable ones, demonstrating implicit meta-learning without a hand-designed algorithm.The effect resembles MAML's meta-learning outcome, but arises from ordinary SGD under data properties the authors do not yet fully understand.Source: Figure 3 · p. 4, §3.1 · p. 4

3.2. Demonstrating IML via entity attribution

  • Entity-attribution tests confirm IML beyond in-distribution questions: consistent definitions are internalized more, and the internalization gap grows substantially.This provides a more direct test of alias resolution than indirect QA.Source: §3.2 · pp. 4–5
  • OOD transfer is incomplete: with a new random tag, newly defined variables are answered better than variables defined with the reliable tag.The authors report this despite a notable difference between the reliable and unreliable tags.Source: §3.2 · p. 5

3.3. Additional experiments exploring IML

  • As tag–consistency reliability increases, the difference in internalization grows; at high reliability, inconsistent-tag definitions can be internalized more than consistent-tag ones.In that high-reliability setting, test predictions rely more on definitions than on training QA pairs.Source: §3.3 · p. 5
  • Definition word order matters: entity-last orders show IML, while EAT, TEA, and ETA show poor performance and no IML on entity attribution.AET shows no IML on that test, though it does show a weaker effect on in-distribution questions.Source: §3.3 · p. 5
  • Larger models show better performance and more IML, first noticeable at 1B parameters; replication in GPT-Neo, LLAMA2-7B, and T5-3B shows it is not decoder-only-specific.The authors also reproduce IML on a dataset about movies and books.Source: §3.3 · p. 5
  • Joint fine-tuning on X1 and X2 produces similarly strong performance effects, but is a less clean demonstration; prepending definitions yields in-context learning and no IML.The two-stage setup isolates the effect of changes in learning behavior.Source: §3.3 · p. 5

4. How general is implicit meta-learning?

The paper tests whether implicit meta-learning extends beyond pretrained text transformers, finding evidence in models trained from scratch and in a supervised vision task with a ConvNet.

4.1. Pretraining is not necessary

  • At α = 0.5 tags did not guide internalization, while at α = 0.95 tag influence exceeded consistency; EAT, TEA, and ETA showed no IML.Both general performance and IML declined as batch size increased, with 16k effectively full-batch training.Source: Figure 4 · p. 6
  • A toy integer-membership task removes dependence on pretrained factual knowledge, yet Pythia-70M trained from scratch shows weak but statistically significant IML.The task asks whether a variable’s corresponding number appears in a list.Source: §4.1 · pp. 5–6

4.2. IML is not specific to text models

  • On MNIST, a ConvNet learns tag-marked grid definitions and answers questions about a highlighted digit; the authors report replicated IML.The evaluation uses masked accuracy on the highlighted digit.Source: §4.2 · p. 6

5. Potential mechanisms

The paper considers gradient alignment and selective retrieval as nonexclusive hypotheses for IML, and discusses learned tag semantics as an interpretation.

5.1. Gradient alignment hypothesis

  • Consistent-definition gradients aligned more with corresponding QA gradients than inconsistent-definition gradients, supporting—but not proving—gradient alignment as part of IML’s mechanism.The authors say it remains unclear whether SGD’s implicit bias is responsible.Source: §5.1 · p. 7, Figure 6 · p. 7
  • Larger batches weakened IML, which disappeared with full-batch training, but overall performance also fell, preventing attribution to SGD’s implicit bias.The authors conclude that gradient alignment is part of the mechanism, while its cause remains unclear.Source: §5.1 · p. 7

5.2. Selective retrieval hypothesis

  • Selective retrieval proposes that Stage1 makes reliable-source facts easier to retrieve, biasing later use of similarly stored definitions.The authors consider retrieval from the parameters plausible, though they do not explain why the model would not recognize retrieved random noise.Source: §5.2 · pp. 7–8
  • Linear probes failed to recover definition presence or tag for standard TAE definitions but succeeded for ATE, making explicit test-time tag-based retrieval unlikely in TAE.Probe success does not establish that the model relies on the predicted feature; probe failure is evidence it may not be represented or used.Source: §5.2 · p. 7, Figure 7 · p. 8

5.3. The model learns semantics of the define tags

  • The authors treat learned “is/is not” tag semantics as an insightful interpretation, not a principled explanation; literal “is” tags helped entity attribution but not attribute questions.They note that neither stage’s loss explicitly encourages generalization to Stage2-variable QA.Source: §5.3 · p. 8

6. Related work

The paper situates implicit meta-learning (IML) alongside work on internal knowledge, in-context learning, out-of-context learning, and gradient alignment. It distinguishes its focus on how apparent reliability affects knowledge incorporation and updates across training sequences.

6. Related work

  • The paper studies how apparent correctness influences knowledge incorporation into LLMs and proposes mechanistic explanations, a focus the authors describe as novel.Related work also examines latent knowledge, model editing, and coherent world models.Source: §6 · p. 8
  • In-context learning conditions on prompt examples, whereas IML may focus on more reliable, static facts and in-context learning on local context.The paper presents them as potentially complementary, building on prior accounts of few-shot learning and Bayesian inference (Brown et al. (2020); Xie et al. (2021)).Source: §6 · p. 9
  • Out-of-context learning studies cover cross-document recall, logical consequences, and usefulness-tagged documents, while this paper focuses on consistency-based source reliability.The cited examples include Berglund et al. (2023), Meinke & Evans (2023), and Allen-Zhu & Li (2024).Source: §6 · p. 9
  • Unlike Genewein et al. (2023), this paper studies IML across sequences of updates, consistent with canonical meta-learning; prior work links SGD updates to gradient alignment.The related discussion connects multiple SGD updates to MAML-like terms (Nichol et al. (2018)) and implicit gradient-variance penalties (Smith et al. (2021)).Source: §6 · p. 9

7. Discussion

The paper concludes that deep networks can learn reliability or usefulness cues and modulate their updates accordingly. Its scope and explanatory account remain limited.

7. Discussion

  • The mechanisms behind IML remain inconclusive, although the paper discusses two candidates and presents some evidence for a role of mini-batch gradient-descent implicit regularization.The authors also note that internalization is operationalized in several tasks but not formally defined.Source: §7 · p. 9
  • The experiments use toy datasets, leaving reproduction with data used to train real LLMs as future work.Source: §7 · p. 9
  • LLMs and ConvNets can learn features indicating example reliability or usefulness and update less or more on examples carrying those cues.The authors suggest IML may matter for understanding LLMs, SGD-based optimization, and deep learning.Source: §7 · p. 9

Impact statement

The impact discussion weighs possible uses of reliability learning against misinformation, bias, and advanced-AI safety concerns. The accompanying dataset-generation material describes synthetic QA resources and safeguards against information leakage.

Impact statement

  • Teaching models which sources are reliable could help address misinformation or bias, but prevalent falsehoods or biases might also poison them.Source: Impact statement · p. 9
  • The authors suggest IML could affect capability forecasts and motivate withholding information useful to malicious actors or relevant to model evaluation and monitoring.They also discuss possible links to situational awareness and learned normative reasoning.Source: Impact statement · p. 10

A.1. CVDB

  • The CVDB contains 2.23 million people; the authors select 4,000 and build QA data around six basic attributes.The selected entities are the 2,000 most popular men and 2,000 most popular women; the attributes derive from CVDB features.Source: §A.1 · p. 13
  • To reduce cross-datapoint leakage, the authors coarsen many dates, but acknowledge that this does not eliminate leakage and may not be necessary.Pre-1900 years become centuries, and 1900–1999 years become decade labels.Source: §A.1 · p. 13

A.2. T-REx

  • T-REx creative-work triplets become QA pairs, with four questions per subject; the process also limits author-name clues and anonymizes years.Table 2 describes replacing the triplet subject in the question and using its object as the answer.Source: §A.2 · p. 13, Table 2 · p. 14

A.3. Data splits

The data splits separate entities across training and evaluation subsets, with QA pairs allocated differently by entity group. The LLM fine-tuning setup specifies optimizer, batch size, epochs, and context handling.

A.3. Data splits

  • Entities are randomly divided 70% to X1 and the remainder to X2, then assigned to subsets so each entity appears in only one subset.Source: §A.3 · p. 14
  • For X1 entities, one QA question is held out for validation; all questions for X2 entities go to test.CVDB uses five training questions and one validation question per X1 entity; T-REx uses one validation question and four test questions per entity.Source: §A.3 · p. 14

B. Hyperparameters used when finetuning LLMs on QA data

  • LLMs are fine-tuned with Adafactor at batch size 256, using 20 epochs on X1 and 10 on X2; datapoints are padded to length 64 without chunking.The models use deduped Pythia versions and default Trainer hyperparameters otherwise.Source: B. Hyperparameters used when… · p. 14

C.1. Two-stage results for Pythia-2.8B: losses and entity attribution on CVDB data

The appendix tests IML across datasets, model families, training configurations, and learning setups. Results generally show IML, with weaker or unclear effects in some settings.

C.1. Two-stage results for Pythia-2.8B: losses and entity attribution on CVDB data

  • On CVDB, entity-attribution experiments show a first-stage performance difference and IML across all four question types.Training losses for QA pairs and definitions appear indistinguishable despite different validation losses and exact-match scores.Source: Figure 9 · p. 15, Figure 8 · p. 15

C.2. Experiments with the T-REx-based dataset (questions about movies, books, and other creative works)

  • On T-REx, IML appears for all question types, but its in-distribution effect is smaller than with CVDB, which the authors attribute to the harder dataset.The comparison is based on the gap between the blue and red lines in the second stage.Source: Figure 10 · p. 16

C.3. Varying the order of (define tag, variable, entity) in “definitions”

  • With definitions ordered as variable–entity, first-stage differences are significant across five test sets, while IML is significant only in-distribution.For other orderings where the variable follows the entity, results broadly match the reversal curse; the EVT in-distribution case appears different, with its learning curve declining during stage two.Source: §C.3 · p. 17, Figure 12 · p. 18

C.4. Varying the batch size during single-stage finetuning of Pythia-1B

  • Larger batch sizes generally weaken IML in single-stage CVDB finetuning, though the trend may reverse around batch size 32.Batch sizes count datapoints, not tokens.Source: Figure 13 · p. 18

C.5. Single-stage results for Pythia-2.8B

  • Single-stage CVDB and T-REx runs show IML across question types, but T-REx entity attribution cannot be assessed because its test set was disabled.Both experiments used Pythia-2.8B-deduped over 10 seeds.Source: Figure 14 · p. 19, Figure 15 · p. 20

C.6. Two-stage finetuning results for differently sized Pythia, GPT-Neo, and Llama2 models

  • The CVDB experiments show IML for larger GPT-Neo models and Llama2-7B, while the harder T-REx dataset shows it only for the largest GPT-Neo model.The figures also report in-distribution performance for differently sized Pythia models.Source: Figure 17 · p. 21, Figure 18 · p. 21, Figure 16 · p. 21

C.7. Sequence-to-sequence model experiments: setup and results

  • For T5-3B, single-stage results look IML-like but are unclear as accuracy declines; in two-stage finetuning, CVDB shows clear in-context learning, while T-REx shows neither effect.The authors adapt the encoder-decoder setup by splitting definitions into an input and an output.Source: Figure 20 · p. 22, Figure 19 · p. 22, §C.7 · p. 22

C.8. Comparison with in-context learning

  • When definitions are included in question context, the model relies on familiar reliable-tag definitions and learns to ignore inconsistent or inconsistent-seeming ones.Definitions are prepended to training and test questions, while evaluation uses new variables.Source: §C.8 · pp. 22–23, Figure 21 · p. 23

D. Set inclusion experiment

The set-inclusion experiments test IML in a model trained from scratch, and the appendix describes a separate MNIST question-answering setup used to examine the phenomenon in vision.

D. Set inclusion experiment

  • The set-inclusion data contain 8,000 entity–variable pairs, with 12 balanced yes/no QA pairs per variable in subsets that include questions.The model is trained for 100 epochs in stage one and 40 in stage two, using Adafactor and batches of 512 datapoints.Source: D. Set inclusion experiment · p. 23
  • A Pythia-70M model trained from scratch shows both a first-stage performance difference and IML; with more training questions per variable, definitions stop affecting performance.Increasing questions per variable from 12 can raise validation performance to 99%.Source: Figure 22 · p. 23

E.1. MNIST QA Dataset

  • The MNIST QA task uses 3 × 3 digit grids, random bit-pattern definition tags, and a one-pixel border to mark the digit queried.Training images are sampled from the MNIST training split and evaluation images from the test split, requiring generalization across digit images.Source: §E.1 · p. 24

E.2. Hyperparameters for the MNIST QA experiments

  • The MNIST experiments use ConvNeXt V2 Tiny, with 28.6 million parameters, trained for 120,000 steps using AdamW and batch size 128.The learning rate is 3 × 10^-4, with 2,000 warm-up steps.Source: §E.2 · p. 24

E.3. IML results for the MNIST QA Dataset

The MNIST experiments show both out-of-context performance differences and IML, with the IML effect depending on the number of variable–entity pairs.

E.3. IML results for the MNIST QA Dataset

  • Definitions are required for perfect test accuracy because each variable has one grid cell that was never queried during training.The experiments also report a first-stage performance difference between data subsets.Source: §E.3 · p. 24, Figure 23 · p. 24
  • With at least 50 variable–entity pairs, the model performs much better on questions tied to the consistently used definition tag than to the inconsistent tag, and the effect grows with pair count.The effect is visible even for groups whose QA pairs appeared during training, because one grid cell per variable was withheld from training questions.Source: §E.3 · p. 24, Figure 23 · p. 24

F. Exploring the gradient alignment hypothesis

The gradient-alignment analysis finds that reliable definitions and their questions align more on some metrics, but not on squared distance. The paper also discusses possible broader implications and reports computational costs.

F. Exploring the gradient alignment hypothesis

  • The analysis could not compare gradients for all datapoint pairs because models above 1B parameters make caching more than a few gradients infeasible, even with 80GB GPUs.The study therefore monitors selected alignment metrics between definitions and corresponding questions.Source: F. Exploring the gradient alignment… · p. 25
  • Reliable-definition and question gradients are more aligned by inner product and cosine similarity, but squared Euclidean distance shows no effect or the reverse pattern.The authors suggest larger reliable-definition gradient norms may explain the squared-distance result.Source: F. Exploring the gradient alignment… · p. 26

G. Potential implications of LLMs internalizing normative principles of reasoning

  • The discussion warns that internalized functional decision theory might make seemingly myopic systems pursue long-term goals, including influencing users or tampering with reward signals.The authors say better understanding IML could help rule out such scenarios or prevent them.Source: G. Potential implications of LLMs… · p. 26

H. Computational resources used for our experiments

Related papers

Loading 2310.15047v4…