PaperlayerView exact revision on arXiv

Source-linked AI summary

From Copying to Composition: How Language Models Develop In-Context Learning

Minglai Yang, Xinyu Guo, Renxiong Wang, Sunjie Hou, Daniel Yue Zhang, Yunzhong He

arXiv:2610.19113v1cs.CLcs.AIcs.LG

TL;DR

Accuracy alone does not show whether models copy nearby demonstrations or compose information distributed across a prompt. This paper tracks those behaviors with a controlled diagnostic and matched training experiments. Prompt control shifts toward the requested value before accuracy rises above chance, while different training signals produce distinct patterns of composition and transfer.

  • Problem

    Ordinary ICL evaluations can give the same answer for copying a similar demonstration and integrating prompt mappings, so correctness alone does not reveal how context controls predictions.

    Source: §1 · p. 1

  • Method

    The study uses a controlled task where copying a nearby example and composing prompt-defined mappings predict different answers, then tracks prompt-source control and answer preference across models and training.

    Source: §1 · pp. 1–2, §3 · p. 3

  • Results

    The requested value takes control from the closest example before accuracy leaves chance, while copying and composition can coexist and the missing form appears only once training supplies a way to build it.

    Source: §7 · p. 8

  • Takeaways & Limitations

    Prompt control, answer preference, construction without candidates, and transfer can develop on distinct trajectories rather than as one uniform ICL capability.

    Source: §8 · p. 8

    The diagnostic is a single controlled task, and training experiments reach models up to 8B parameters using synthetic documents that share its format, so larger models and more varied training data remain to be tested.

    Source: Limitations · p. 10

Abstract

from arXiv · show

In-context learning (ICL) allows language models to adapt their behavior from instructions and demonstrations in a prompt, but accuracy alone does not reveal how contextual information controls their predictions. We study the acquisition and trainability of ICL with a controlled diagnostic that separates copying a nearby demonstration from integrating information across the prompt. Across 17 models, about a third prefer the copied answer, a few are undecided, and the rest prefer the rule-consistent answer, which goes with higher accuracy and with control by the requested value rather than the closest conflicting example. Checkpoint analyses show that models begin prioritizing task-relevant prompt tokens over conflicting examples long before their accuracy improves beyond chance level. In matched training experiments, synthetic documents that define endings and then query them shift models toward compositional responses on the controlled diagnostic, whereas fine-tuning on definition exercises built from rewritten real documents improves accuracy when prompts reassign label meanings. The results suggest that ICL is not acquired as a single, uniform capability. Instead, prompt control, answer preference, construction without candidates, and transfer can develop along distinct trajectories.

1 Introduction

Accuracy alone cannot show how a model uses prompt context, because copying a similar demonstration and composing prompt-defined mappings can produce the same answer. This work introduces a diagnostic that separates those strategies and tracks how prompt control, answer preference, construction, and transfer change during training.

Research question

  • The study tracks four distinct aspects of ICL: prompt-source control, answer preference, construction without candidates, and transfer to other evaluations.These developmental changes cannot be resolved by final-answer accuracy alone.Source: §1 · p. 1

Diagnostic

  • The diagnostic makes copying a nearby example and applying prompt-defined mappings predict different answers, revealing how contextual information controls predictions.Each prompt assigns fresh endings, demonstrates three of four combinations, and asks for the missing combination on a new word.Source: §1 · p. 1

Findings

  • Across 17 models, preference ranges from copying the nearest demonstration to producing the rule-consistent form, with composing models scoring higher and generating that form without candidates.The requested value takes control over the conflicting example hundreds of billions of tokens before accuracy rises above chance.Source: §1 · p. 2

Training experiments

  • Matched training on synthetic documents that define and query endings shifts models toward compositional responses, while rewritten-document definition exercises improve performance when prompts reassign label meanings.Each training signal transfers the prompt use it rewards; the definition exercises leave the controlled task unchanged.Source: §1 · p. 2

2 Related work

Prior work describes ICL through copying, latent-task inference, or forward-pass updates, and studies both context-use mechanisms and development during training. This paper connects copying-versus-rule-use contrasts to prompt-token interventions tracked across checkpoints.

In-context learning and composition

  • Accounts of ICL include induction-like copying, latent-task inference, and forward-pass updates; controlled tasks also examine intermediate strategies and task recognition without learning.The paper connects its task to compositional generalization, made-up-word tests, prompt-defined labels, and the rule-versus-analogy contrast in morphology (Hupkes et al., 2020; Berko, 1958; Lake and Baroni, 2018; Kim and Linzen, 2020; Lake and Baroni, 2023; Weissweiler et al., 2023; Wei et al., 2023a,b; Pinker and Prince, 1988; Albright and Hayes, 2003).Source: §2 · p. 2
  • The task makes copying a similar example and applying supplied mappings predict different answers, then intervenes on the prompt tokens that support those answers.The authors note that tokenization shapes what counts as a morpheme for models, a point their Quechua results bear out (Sennrich et al., 2016; Rust et al., 2021).Source: §2 · p. 3

Mechanisms of context use

Development during training

  • Checkpoint studies commonly track accuracy gains or component abilities; this work instead follows copied-versus-correct scores and prompt-token dependence throughout training.It extends work localizing or modifying context-use mechanisms to the training trajectory (Yin et al., 2024; Prakash et al., 2024).Source: §2 · p. 3

3 Task and measurements

The task uses prompts with fresh mappings and a held-out feature combination so that copying the closest example conflicts with composing the requested values. Its measures distinguish candidate preference, prompt-source control, and exact generation without candidates.

Task construction

  • Each prompt assigns fresh endings to four feature values, shows three combinations, and asks for the missing combination on a new stem.A short-code version and a reverse-order control provide alternative task formats.Source: §3 · p. 3

Formalization and scores

  • Equation 1 composes the two prompt-defined endings in feature order for a held-out pair on a new stem.The rule-consistent and copied candidates share the stem and first ending, differing where copying conflicts with the requested value.Source: §3 · p. 3
  • The candidate margin compares the probabilities of the rule-consistent and copied forms; positive m favors the correct form, while exact generation supplies no candidates.Four-choice accuracy additionally requires the rule-consistent form to outrank all four combinations.Source: §3 · p. 3

Blocking and the switch of control

  • Blocking attention from the answer position to selected prompt spans measures their effects on the candidate margin; positive D indicates greater control by the closest ending.The switch of control is the first adjacent checkpoint interval where mean D changes from positive to negative.Source: §3 · pp. 3–4

Activation patching

  • Activation patching replaces the requested-value representation after layer 4 with the representation from a prompt asking for the other value.Success requires writing the full form for the replacement value; never-shown targets are reported separately because they require constructing a new form.Source: §3 · p. 4

Models

  • The study tests released Pythia, OLMo, Qwen, and Gemma models, intermediate checkpoints, and continued-training or fine-tuning variants.Continued training restarts SmolLM2 checkpoints; definition-exercise fine-tunes use SmolLM2-1.7B and SmolLM3-3B.Source: §3 · p. 4

4 Answer preference reflects prompt control

Models differ in whether they prefer copying the closest demonstration or producing the rule-consistent answer, and that preference tracks which prompt information controls their predictions. The intermediate models show that answer balance and prompt control vary continuously.

4 Answer preference reflects prompt control

  • Six models assign at least .70 of combined candidate probability to the copied answer, eight assign at most .30, and three fall between these thresholds.The groups span copying through composition rather than reflecting model size.Source: §4 · p. 4, Figure 2 · p. 4
  • Correct-answer preference tracks construction ability: the correct answer rarely ranks first in high-copy models and usually does in low-copy models, with exact generation following the same ordering.Pythia-12B copies, while Qwen3-1.7B and Gemma-3-4B compose, showing the grouping is not a size effect.Source: §4 · p. 4
  • Removing the closest combined example selectively weakens the copied candidate, while removing unrelated lines has little effect.Arbitrary codes and a token-aligned Southern Quechua version preserve the model ordering, tying preference to competing prompt information rather than wording, suffix inventory, or prompt length.Source: §4 · p. 4
  • The closest-ending span has the larger effect in high-copy models, while the requested-value span has the larger effect in low-copy models.The closest-ending effect is about twice the requested-value effect in high-copy Pythia models; in low-copy models, the requested-value effect is several times larger.Source: Figure 3 · p. 4, §4 · p. 5
  • The three intermediate models assign similar probability to both candidates, and their two prompt-span effects are much closer than in either main group.This supports a continuous variation in candidate balance and controlling span.Source: §4 · p. 5
  • Patching the requested-value state redirects models toward a shown combination, but high-copy Pythia models rarely follow when it requires constructing the unshown combination.Other models follow the requested combination on a substantial share of prompts.Source: §4 · p. 5

5 Prompt control changes before composition

During pretraining, control shifts from the closest example to the requested value before accuracy rises above chance. Mapping recovery and generation of shown combinations precede selection and exact generation of the missing combination.

5 Prompt control changes before composition

  • 336B–630B tokens: the requested-value effect overtakes the closest-ending effect before four-choice accuracy reaches chance.At the end of this interval, the candidates receive similar probability; the requested-value effect remains ahead at later checkpoints.Source: §5 · p. 5, Figure 6 · p. 6
  • About 1.5T tokens on original items and 1.8T on re-test items: the fitted four-choice accuracy trajectory crosses chance, well after the blocking effects cross.The model can copy the requested stem, recover feature mappings, and generate combinations shown in the prompt hundreds of billions of tokens before ranking the missing combination first.Source: §5 · pp. 5–6
  • About 3.6T tokens: exact generation of the missing form arrives last, after prompt control shifts and accuracy improves.The reported developmental sequence is parts, then control, then selection, then generation.Source: §5 · p. 5
  • In OLMo-3 and Comma, prompt control shifts hundreds of billions of tokens before their first above-chance checkpoints; successful series generally shift before or by the time accuracy crosses chance.Runs that fail to compose do not sustain requested-value control and keep the copied answer; one SmolLM3 series later regains closest-example influence.Source: §5 · pp. 5–6
  • The control shift is distributed across heads: five heads carry part of the blocking effect, and silencing them moves the balance without creating composition.Blocking a prompt span moves the margin far more than silencing either selected head set.Source: §5 · p. 6, Figure 5 · p. 6
  • Under direct task fine-tuning, the blocking crossing and accuracy rise happen together, or accuracy moves first, unlike their ordering during pretraining.This makes the reported ordering specific to pretraining rather than an invariant of the task.Source: §5 · p. 6

6 Training teaches what it practices

Different training signals change different forms of prompt use: exposure or copy suppression alone does not create the missing answer, while synthetic query practice shifts the diagnostic and definition exercises improve prompt-assigned label use. Transfer follows the use of the prompt that training rewards.

6 Training teaches what it practices

  • Training effects separate suppression, construction, and transfer: synthetic query practice shifts the controlled diagnostic, while definition exercises improve label use and leave the morphology task unchanged.Which use of the prompt the training text repeatedly rewards is the distinguishing factor described by the authors.Source: §6 · p. 6, §6.3 · p. 8

6.1 Practice, not exposure or suppression, moves control

  • After 1B additional tokens, mined passages containing definitions leave SmolLM2-1.7B controlled by the closest ending, with D positive and exact generation at plain-continuation levels.Penalizing attention from the answer position to selected copy-favoring heads also leaves D positive and exact generation at zero; random-head penalties leave both at plain-continuation levels.Source: §6.1 · p. 6, Figure 7 · p. 7

6.2 Synthetic documents reverse copying and transfer what they practice

  • 3% synthetic mixture: SmolLM2-1.7B shifts from copy-dominant to query-dominant, with exact generation rising from near zero to near ceiling.By 50M tokens, P(copy) falls from .82 to near zero and exact generation rises to .56; held-out losses move by less than .001 while D moves by more than twenty nats.Source: §6.2 · p. 7, Figure 7 · p. 7
  • Held-out and all-shown synthetic documents both turn D negative by the first checkpoint and produce the missing form on most prompts, so practice on an absent cell is not required.The shift also holds under full fine-tuning, and LoRA takes five 7–8B bases to or near the diagnostic ceiling.Source: §6.2 · p. 7
  • Synthetic-training gains stay with the practiced form: external composition evaluations do not meet the joint-improvement criterion, and some SCAN-ICL scores decline.At 1.7B, external-evaluation differences are descriptive under the pre-registered qualification rule; the 7–8B adapter transfer boundary applies to bases above floor on SCAN-ICL and ElementalTask-ICL.Source: §6.2 · p. 7, Figure 8 · p. 7

6.3 Transfer depends on the supervision

  • Made-up-label accuracy rises 17.6 points on TREC, 17.2 on DBpedia, and 3.7 on Yahoo after fine-tuning on definition exercises.The exercises teach following prompt-assigned labels rather than prompt-dependent tasks in general; the morphology diagnostic remains near floor.Source: §6.3 · p. 8

7 Discussion

The discussion separates prompt control, answer preference, construction, and transfer into distinct developmental events. Models can become query-controlled before accuracy improves, while copying and composition may coexist.

7 Discussion

  • The requested value controls answers before accuracy rises above chance, and Figure 9 separates this shift from preference, construction, and transfer.Accuracy registers only the candidate-preference flip, not the earlier change in prompt control.Source: Figure 9 · p. 8, §7 · p. 8
  • Copying and composition coexist: strengthening the requested value’s influence does not erase the closest example’s influence, and the missing form appears only when training supplies a way to build it.The all-shown control reaches comparable exact generation at 360M, placing the shared factor in repeated resolution through temporary mappings rather than rehearsal of the absent cell.Source: §7 · p. 8
  • A wrong answer can mark a shift toward query-based control, while a correct multiple-choice answer does not establish that the model can construct the form without candidates.The four events are prompt-source control, candidate preference, construction without choices, and transfer to held-out tasks.Source: §7 · p. 8

8 Conclusion

The conclusion treats in-context learning as several capabilities developing on different schedules. Its diagnostic distinguishes prompt control, answer preference, construction, and transfer rather than relying on accuracy alone.

8 Conclusion

  • Prompt control can shift to the requested value before the correct answer wins, so accuracy alone misses an earlier change in how the prompt guides predictions.Suppressing copying moves the candidate margin, while training that repeatedly resolves queries can produce the missing form.Source: §8 · p. 8
  • The paper tracks four distinct aspects of in-context learning: prompt-source control, answer preference, construction without candidates, and transfer to other evaluations.The conclusion frames these as capabilities with different developmental schedules.Source: §8 · pp. 8–9
  • Across models, checkpoint series, and training experiments, the four aspects develop on different schedules rather than forming one uniform capability.Source: §8 · p. 9

Limitations

The study’s diagnostic and training experiments cover a bounded task and training format.

Limitations

  • The diagnostic is a single controlled task, and training uses models up to 8B parameters with synthetic documents sharing its format, so demonstrated transfer is format-bound.Larger models and more varied training data remain future tests.Source: Limitations · p. 10

Ethical Considerations

The appendix describes task construction and conventions alongside supporting analyses, while the ethical considerations note the limits of content screening for mined documents.

Ethical Considerations

  • The mined documents come from public web, scientific, mathematical, encyclopedia, forum, and legal collections; no additional personal or offensive-content screening was applied beyond source filtering.The source collections may therefore contain such material.Source: Ethical Considerations · p. 10

Contents of the Appendix

  • Appendix contents cover task construction, checkpoint and training analyses, intervention checks, and changes to the analysis plan.Source: Contents of the Appendix · p. 15

A Task construction and conventions

  • Prompts use fresh stems and shuffled demonstrations to query an unshown feature combination; suffix and code schemes provide distinct encodings.The prompts draw feature-value pairs and endings from pools, and the queried stem does not appear in demonstrations.Source: A Task construction and conventions · p. 15
  • The task vocabulary labels the rule-consistent candidate “correct” and the candidate repeating the closest example’s second ending “copied.”Intervention conventions define activation patching at the requested-value position before generation and block all tokenizer tokens overlapping selected spans.Source: A Task construction and conventions · p. 15
  • Endings can span multiple tokens, so scoring supplies preceding tokens only when the hyphen and ending remain separate tokens.Source: A Task construction and conventions · p. 15

B Threshold crossings during training

In OLMo-2-7B, component behaviors cross their thresholds before or during the held-out-form D sign-change interval, while held-out accuracy and generation cross later. Final models differ in copying preference and in how that preference responds to prompt variants.

B Threshold crossings during training

  • Threshold times are crossings of fitted logistic trajectories, not first significant differences; intervals use 300 item-resampling fits and censor fits whose asymptote stays below threshold.The mean trajectories are fit over log10 tokens, or training steps for Pythia.Source: B Threshold crossings during training · pp. 15–16
  • In OLMo-2-7B, every component behavior crossed by 338B before or during D’s 336B–630B sign-change interval; held-out accuracy and exact generation crossed at 1,464B and 3,586B.OLMo-2-13B shows the same order, though its exact-generation fit extrapolates beyond the observed run.Source: B Threshold crossings during training · p. 16
  • For Pythia-1.4B, in the code scheme held-out accuracy and generation cross at 143B and 131B, before shown-combination generation at 815B; component-before-selection order is established only for the suffix scheme.Source: B Threshold crossings during training · p. 16

C Final scores and prompt-variant controls

  • The correct form is generated on .00–.05 of prompts in the high-copy group and .30–.93 in the low-copy group.OLMo-2-1B (.58), Qwen3-0.6B (.50), and Qwen2.5-1.5B (.49) fall between the stated thresholds.Source: C Final scores and prompt-variant… · p. 16
  • Pythia’s P(copy) stays at .66–.95 under variants retaining the closest example, but falls to .32–.46 with single-feature lines only.Qwen3-1.7B and released OLMo-2-7B rank the correct form first on at least .45 of prompts under every variant except combined-form lines only; Qwen3-8B reaches 1.00 on the reworded template.Source: C Final scores and prompt-variant… · pp. 16–17, Figure 10 · p. 17

D Blocking, deletion, and scoring checks

Blocking and deletion tests distinguish the effects of nearby examples from requested-value information, while scoring and precision checks assess the stability of those results.

D Blocking, deletion, and scoring checks

  • Blocking closest endings raises m by +1.00 to +5.34 nats, while blocking requested values lowers it by −1.59 to −10.51; D is positive for Pythia and negative for low-copy models.Blocking the closest ending moves Pythia-2.8B’s m from −2.1 to +2.6 nats, while blocking the requested value moves Qwen3-8B’s from +8.9 to −1.5.Source: D Blocking, deletion, and scoring checks · p. 17, Table 3 · p. 18
  • Deleting the closest example makes the correct answer preferred in Pythia-1.4B, Pythia-2.8B, and Qwen3-0.6B by +3.20, +3.30, and +3.52 nats, respectively.Deleting correct-ending lines lowers m everywhere (−0.53 to −1.52).Source: D Blocking, deletion, and scoring checks · p. 17, Table 4 · p. 18
  • Restricting scores to single-token endings, first differing tokens, or length-normalized scores leaves P(copy) within .01 and changes neither D’s sign nor model group.Across variants, item-level scores correlate at .84–1.0 for m and .90–.98 for D.Source: D Blocking, deletion, and scoring checks · p. 17
  • Float32 re-scoring leaves OLMo P(copy) changes at most .005 and D changes at most .25 nats, with pooled signs unchanged; bfloat16 overstates Pythia’s closest-ending changes and D.The body and Figure 3 use float32 re-test scores.Source: D Blocking, deletion, and scoring checks · pp. 17–18

E Training trajectories of seventeen runs

This section examines how training trajectories and interventions affect prioritizing task-relevant information, producing answers, and transferring learning across prompt-defined tasks. Outcomes vary across methods, layers, models, and tasks: synthetic documents shift responses toward composition, exercise fine-tuning improves several tasks, and other interventions show limited or mixed effects.

E Training trajectories of seventeen runs

  • In every above-chance run, negative D first appears by the sampled accuracy crossing, while no below-chance run sustains negative D; the new-item retest preserves trajectory outcomes.OLMo-7B-0424 is an exception to the broader ordering pattern: its D first turns negative at 838B but hovers around zero until 2.3T, after fitted accuracy crosses at 1.6T. The retest reproduces the OLMo-2-7B sign-change interval in .999 of joint resamples and places its fitted 1.79T accuracy crossing after the sign change in every resample.Source: E Training trajectories of seventeen… · pp. 18, 20, Figure 11 · p. 20, J Re-test on new items · p. 22

F Code and Quechua versions

  • In code-word prompts, six models retain the suffix-scheme ordering; in Quechua, separate suffix tokens distinguish copy-preferring Pythia from low-copy Qwen models, while standard spelling degrades scores.With separate suffix tokens, Pythia has P(copy) = .77–.84 and correct-ranks-first .04–.06, while four Qwen models have P(copy) = .00–.14 and correct-ranks-first .83–1.00; standard spelling raises P(copy) to .37–.93 for eight models.Source: F Code and Quechua versions · p. 21

G Activation patching

  • Never-shown shares rise as P(copy) falls in low-copy models, but counting only the requested ending lets high-copy Pythia produce it without constructing the full form.For low-copy models, never-shown shares range from .26 to .91 while shown shares are .85–1.00; when only the requested ending is counted, Pythia's never-shown shares rise to .11, .31, and .19.Source: G Activation patching · p. 21

H Stage-2 OLMo-2-7B models

  • Stage-2 OLMo-2-7B training improves correct-ranks-first and exact generation before parameter averaging, but the comparison cannot separate further training from the data-mixture change.In the one run with intermediate checkpoints, the measures rise from .53 and .17 to .64 and .22 within 5B additional tokens; the released average reaches .69 and .30.Source: H Stage-2 OLMo-2-7B models · p. 22

I Fine-tuning a copying model

  • Fine-tuning does not reproduce the pretraining order: target runs cross both thresholds at step 250, and the all-shown run learns the held-out form despite initially favoring copies.In the all-shown run, D is +21 at step 250, turns negative by step 500, and correct-ranks-first reaches .70–.78 from step 1,000; the requested-value effect grows while the closest-ending effect remains present.Source: I Fine-tuning a copying model · p. 22

K Selected attention heads

  • Selected heads contribute more correct-answer logit mass than MLP blocks, but copy-favoring heads remain influential after control switches; induction-head replacement weakens copying without robust held-out generation.In low-copy models, selected heads receive .14–.64 of all head contributions toward the correct answer. Induction-head replacement raises correct-next-ending accuracy in Pythia, but held-out exact generation increases only from .03 to .07 and .05 to .11.Source: K Selected attention heads · p. 23

L Continued pretraining with synthetic documents and attention penalties (360M)

  • At 360M, both synthetic-document conditions cross both thresholds at step 32, while original documents and attention-penalty conditions do not cross within 1B tokens.The one-unshown condition meets transfer and loss criteria versus original documents but not versus all-shown documents; the selected-head penalty sharply reduces penalized attention without producing composition.Source: L Continued pretraining with synthetic… · p. 24

M Continued pretraining with synthetic documents and mined passages (1.7B)

  • At 1.7B, synthetic-document training reaches P(copy) = .006 and exact generation .994, while plain and mined-passage runs retain P(copy) near .84 and exact generation .013.Synthetic documents cross both thresholds at 50.3 million tokens; the other conditions never cross within 1B tokens. No external task changes by more than .03, and the whole-condition comparison does not isolate which document property matters.Source: M Continued pretraining with synthetic… · p. 25

N Layer sweep, head penalties, adapters, and exercises

  • Activation-patching peaks vary by model, and neither selected-head interventions nor head-derived fine-tuning penalties outperform their controls.At SmolLM2-360M, an attention penalty reduces selected-head attention from .310/.341 to .008/.006 while D remains positive at +1.01/+1.60; lower attention alone does not meet the transfer criterion.Source: N Layer sweep, head penalties,… · p. 25
  • Adapters raise exact generation from base rates of .04–.94 to .91–1.00, but none meets the joint transfer criterion on three external tasks.Only all-shown Llama-3.1 gains on ElementalTask (+.12), with a SCAN loss (−.07); OLMo-2 loses .17–.22 on SCAN and its Pile loss rises +.035 nats. The adapters train on 20M synthetic and 20M original-text tokens across five released 7–8B bases (Hu et al., 2022).Source: N Layer sweep, head penalties,… · p. 25
  • Answer-position attention to the defining ending separates model groups with AUC .814 [.59, 1.00], below the .90 threshold, and none of three attention predictions is met.No trajectory shows the predicted increase, and the attention measure is correlational.Source: N Layer sweep, head penalties,… · p. 25
  • Synthetic documents change none of ten standard benchmarks at 1.7B; at 360M, MMLU loses .4 points and LAMBADA gains .6 (Hendrycks et al., 2021; Paperno et al., 2016).With label meanings exchanged, 1.7B loses 3.2 points on SST-2 and 1.5 on AG News; at 360M, SST-2 loses 2.1, AG News gains 1.2, and two other benchmarks are unchanged (Zhang et al., 2015).Source: N Layer sweep, head penalties,… · p. 25
  • Every rewritten-document run stops at its first scheduled evaluation, and neither shows a full-window gain over original-document training.The 1.7B run stops at 120 steps (250M tokens) after bAbI-20 falls 2.8 points (Holm p = .04) (Weston et al., 2015); an extension to 360 steps stops again when List Functions falls 2.4 points (Srivastava et al., 2023).Source: N Layer sweep, head penalties,… · p. 25

P Changes to the analysis plans

N Layer sweep, head penalties, adapters, and exercises

  • Figure 17 reports exercise-training gains of 17.6–20.4 points on TREC-6, 17.2–19.0 on DBpedia-7, and 3.7–6.3 on Yahoo-5 for made-up labels on unseen datasets, plus 3.6–13.6 for exchanged labels.At 3B, TREC-6 and Yahoo-5 gain 14.4 and 6.1 points, with no qualified unseen-label evaluation losing. Seen-format results include SST-2 exchanged labels +23.4 and Subj +17.4 in the mixture, alongside bAbI-1 −5.3 and CLUTRR −2.7; farther-task results include List Functions +3.3–+3.6 and bAbI-20 −3.6 at 1.7B and −6.3 at 3B. The morphology diagnostic is unchanged (Weston et al., 2015; Sinha et al., 2019; Srivastava et al., 2023).Source: N Layer sweep, head penalties,… · pp. 25–26, Figure 17 · p. 29

Related papers

Loading 2610.19113v1…