Larger language models do in-context learning differently

• The study distinguishes two routes to ICL: semantic priors can predict labels, while input–label mappings can be learned from exemplars, and regular ICL permits both.Flipped-label ICL tests whether mappings override priors, whereas SUL-ICL removes semantic label information and therefore requires mappings.Source: §1 · pp. 1–2, Figure 1 · p. 2
• Large models can override semantic priors and follow flipped in-context labels, whereas small models largely retain their original predictions; this ability emerges with scale.Flipped-label ICL makes semantic priors and exemplar mappings disagree, with below-50% binary accuracy indicating successful override.Source: §1 · p. 2
• Large language models can perform semantically-unrelated-label ICL without label semantics, while small models suffer large performance drops when semantic priors are removed.For some datasets, substantial scaling was required to exceed random performance, with only PaLM-540B doing so.Source: §1 · p. 2
• Instruction tuning improves SUL-ICL performance but makes models more reluctant to follow flipped labels, increasing reliance on semantic priors more than mapping learning.The study compares instruction-tuned and pretraining-only models across these settings.Source: §1 · p. 2

2.1 EVALUATION TASKS

• Experiments cover seven widely used NLP tasks spanning sentiment, subjectivity, questions, duplicate questions, entailment, financial sentiment, and hate speech.The tasks are SST-2, SUBJ, TREC, QQP, RTE, FP, and ETHOS.Source: §2.1 · p. 3

2.2 MODELS

• The study compares GPT-3, InstructGPT, Codex, PaLM, and Flan-PaLM, including PaLM sizes from 8B to 540B for controlled scaling analysis.PaLM models share training data and protocol and differ only in model size; GPT-3 sizes range from 350M to 175B.Source: §2.2 · p. 3

2.3 ADDITIONAL EXPERIMENTAL DETAILS

• By default, each example uses 16 few-shot exemplars per class, an Input/Output prompt with Foo/Bar targets, and 100 randomly sampled evaluation examples per dataset.The study varies exemplar count, input format, and target type in ablations; different exemplars are used for each inference example.Source: §2.3 · p. 3
• Question–Answering and Coreference Resolution are excluded because even the largest models showed weak preliminary performance in many settings.The omitted tasks are QNLI and WSC.Source: §2.3 · p. 3

Experimental setup

• With 100% flipped labels and unchanged evaluation labels, a model that perfectly learns the in-context mapping should achieve 0% accuracy.The experiment progressively flips exemplar labels; 50% flipped labels are equivalent to random labels in the binary classification datasets, while TREC is excluded because it has six classes.Source: §3 · p. 4

Results

• Large models override semantic priors by following flipped in-context labels, while small models’ predictions remain relatively stable and often do not fall below random guessing.For text-davinci-002, performance falls from 90.3% with 0% flipped labels to 22.5% with 100% flipped labels; GPT-3 models remove priors but do not override them.Source: §3 · p. 4, Figure 2 · p. 4

Interpretation

• The ability to override prior knowledge with in-context input–label mappings appears only in large models, indicating an emergent ability unlocked by scaling.Small models rely on semantic priors and do not flip their predictions, consistent with Min et al. (2022b), whereas the scaling interpretation follows Wei et al. (2022b).Source: §3 · p. 4

• SUL-ICL swaps natural-language targets such as “negative” and “positive” for unrelated labels such as “foo” and “bar” to test mapping learning.The setup uses semantically unrelated targets by default and also examines alternative remappings, prompt formats, target types, and out-of-distribution datasets in Appendix B.Source: §4 · p. 5
• Large models show much smaller accuracy drops than small models when natural-language targets are replaced with semantically unrelated labels, indicating emergent in-context mapping ability.Small models primarily rely on target semantics, whereas large models can learn presented input–label mappings without semantic priors.Source: §4 · p. 5
• Increasing exemplars improves large-model performance more than small-model performance across three tested model families, showing greater use of additional input–label mappings.These results are reported in Figure 4, with accuracy averaged across datasets and 100 evaluation examples per dataset.Source: §4 · p. 6, Figure 4 · p. 6
• SUL-ICL performance emerges sharply on some tasks: RTE rises above random only for PaLM-540B, while RTE and ETHOS exceed 80% for code-davinci-002.ETHOS emerges earlier for PaLM, spiking from PaLM-8B to PaLM-62B; other datasets such as SST-2, TREC, SUBJ, and FP can exceed random with small models.Source: §4 · p. 6, Figure 5 · p. 6
• The experiments exclude InstructGPT and davinci models because running the large volume of experiments was too costly.Source: §4 · p. 6

Experimental design

• The comparison uses PaLM and instruction-tuned Flan-PaLM models under the same experiments as Sections 3 and 4.InstructGPT and GPT-3 are not compared because the equivalence of their base models cannot be established.Source: §5 · p. 7

Semantically-unrelated label ICL

• Flan-PaLM outperforms PaLM in SUL-ICL, with Flan-PaLM-8B exceeding PaLM-8B by 9.6% and nearly matching PaLM-62B.The improvement is most prominent in smaller models, supporting stronger learning of input–label mappings after instruction tuning.Source: §5 · p. 7, Figure 6 · p. 7

Instruction tuning effects

• Instruction tuning improves input–label mapping learning but strengthens semantic-prior reliance, making models less able to follow flipped labels in-context.Flan-PaLM could not override semantic priors beyond random guessing with 100% flipped labels, whereas PaLM reached 31% accuracy.Source: Figure 7 · p. 7, §5 · p. 7, Figure 6 · p. 7

Method

• Models classify N-dimensional points above or below an undisclosed threshold using k exemplars per class in the SUL-ICL setup.Coordinates are random integers from 1 to 1000, and the dataset-generation procedure is specified in Algorithm 1.Source: §6 · p. 8

Results

• The largest Codex model outperforms random guessing by 19% on 16-dimensional linear classification, whereas smaller models exceed it by no more than 9%.Accuracy is measured over 100 evaluation examples with k = 16 in-context exemplars per class.Source: Figure 8 · p. 8, §6 · p. 8

Motivation

• These results suggest that sufficiently large language models can perform high-dimensional linear classification, extending scale-based input–label learning beyond natural-language tasks.The study investigates whether greater capacity to learn input–label mappings also holds for non-natural-language tasks.Source: §6 · p. 8

7.1 IN-CONTEXT DEMONSTRATIONS PROVIDE SEMANTIC PRIOR KNOWLEDGE

• Earlier ICL studies attribute performance largely to semantic priors, sequence format, input distributions, and intermediate-token generation rather than exemplar labels.This paper is consistent with Min et al. (2022b) through davinci but finds input–label learning emerging in larger models such as PaLM-540B, text-davinci-002, and code-davinci-002.Source: §7.1 · p. 8

7.2 LEARNING INPUT–LABEL MAPPINGS

• Prior work shows transformers can learn input–label mappings through linear regression and standard learning algorithms, whereas this work observes the ability in frozen pretrained transformers without additional learning.Related natural-language work also finds that irrelevant or misleading prompts do not prevent learning during finetuning or prompt-tuning.Source: §7.2 · pp. 8–9

7.3 EMERGENT PHENOMENA IN LARGE LANGUAGE MODELS

• Scaling enables models to learn input–label mappings even when mappings contradict label semantics, extending emergent large-model behavior to symbolic ICL.The mechanism behind why scaling unlocks these behaviors remains an open question.Source: §7.3 · p. 9

8 CONCLUSIONS

• Model scale enables overriding semantic priors with flipped labels and learning semantically unrelated input–label mappings through SUL-ICL.These abilities emerge when models learn mappings that contradict prior knowledge or use labels without semantic meaning.Source: §8 · p. 9
• Instruction tuning improves input–label mapping but also strengthens semantic priors.Source: §8 · p. 9
• High-dimensional linear classification emerges with model scale, supporting broader in-context mapping across label types.The paper characterizes this as a form of true symbolic reasoning for arbitrary symbols.Source: §8 · p. 9
• Overall, in-context learning behavior changes with language-model scale because models increasingly learn input–label mappings rather than relying only on pretrained semantic priors.Source: §8 · p. 9
• The conclusions distinguish pretrained prior knowledge from input–label mappings presented in context as the two sources of language-model in-context learning.Source: §8 · p. 9

Appendix

• Figure 9 shows prompt formatting for all datasets using varying numbers of in-context exemplars per class.For conciseness, the figure displays one exemplar per class.Source: Figure 9 · p. 15

A DATASET CREATION

• Datasets use HuggingFace versions, with random training-set exemplars and validation-set evaluation examples; datasets without splits receive random 80/20 train/validation partitions.This follows Min et al. (2022b).Source: A DATASET CREATION · p. 15
• The FP dataset uses sentences_allagree, ETHOS uses its binary subset, and TREC uses six coarse labels.Source: A DATASET CREATION · p. 15

Evaluation setup

• The comparison uses per-dataset SUL-ICL results and flipped-label outputs evaluated with 100% flipped labels, with evaluation examples flipped to align the accuracy interpretation.Both settings use k = 16 in-context exemplars per class; accuracy is calculated over 100 evaluation examples.Source: §B.1 · pp. 15–16, Figure 10 · p. 16

Comparison of SUL-ICL and flipped-label ICL

• Performance is almost always higher in SUL-ICL than flipped-label ICL, with medium-sized models differing by up to 74% on SST-2.Small and large models also show significant but smaller performance drops with flipped labels.Source: §B.1 · p. 16, Figure 10 · p. 16
• SUL-ICL is easier because flipped-label ICL requires overriding semantic priors, whereas SUL-ICL primarily requires learning input-label mappings.The results suggest this distinction explains why medium-sized models are particularly affected, while small models often do not outperform guessing and large models better override priors.Source: §B.1 · pp. 15–16

Evaluation limitation

• Accuracy is not always exactly 100% minus the Section 3 accuracy because models, especially small ones, sometimes produce outputs outside the provided labels.This limitation affects the direct interpretation of the flipped-label comparison.Source: §B.1 · p. 16

B.2 REMAPPING INPUTS HURTS PERFORMANCE

• The experiment remaps a per-prompt proportion of input words, ranging from unchanged inputs at 0% to essentially meaningless concatenations of randomly selected words at 100%.The procedure randomly maps selected input words to other words drawn from the prompt’s input vocabulary, reducing semantic meaningfulness.Source: §B.2 · p. 17
• At 100% remapped input words, code-davinci-002 and PaLM-540B cannot beat random guessing on almost all datasets.Small models generally reach guessing performance at 25%–50% remapping, whereas large models show approximately linear declines and usually reach guessing at 75%–100%.Source: §B.2 · p. 17
• Larger models are more robust to input noise, but cannot reliably learn mappings needed to unscramble words after sufficiently extensive remapping.At 100% remapping, solving reliably would require reconstructing most original words, which is difficult because each prompt contains many inputs.Source: §B.2 · p. 17
• TREC is an exception: large models can outperform random guessing with 100% remapped inputs, although the reason remains unclear.Fully remapped TREC inputs should be equivalent to completely scrambled inputs, making this result unexplained.Source: §B.2 · p. 17

B.3 MANY TARGET TYPES WORK

• The target-type comparison tests numerical, alphabetical, fruit, and reversed labels against natural-language targets in SUL-ICL.The reversal tests whether labels can be interchanged within each label set.Source: §B.3 · p. 18
• Large models learn input–label mappings across numerical, alphabetical, fruit, and Foo/Bar targets, with performance generally stable across target types.Reversing labels does not consistently cause the performance degradation observed in flipped-label experiments.Source: §B.3 · pp. 18–19
• QNLI and WSC remain near chance across target types, so the authors remove them from subsequent experiments.These tasks are too difficult for the model regardless of whether natural-language or semantically unrelated targets are used.Source: §B.3 · p. 19

B.4 PROMPT TEMPLATES SHOWING INPUT–LABEL RELATIONSHIPS WORK

• The prompt-format experiment compares Input/Output with arrow, comma-separated, Question/Answer, Student/Teacher, and Q/A templates.Each format explicitly pairs an input placeholder with a label placeholder.Source: §B.4 · p. 19
• 77.9%–87.7% mean accuracy across prompt formats stays within ±6.3% of the 84.2% mean, with no format significantly better than another.These results support using simple templates that clearly present each input with its corresponding label.Source: §B.4 · p. 19

B.5 SEMANTIC PROMPT TEMPLATES YIELD VARYING RESULTS DEPENDING ON MODEL SIZE

• Switching to semantically relevant templates lowers performance for code-cushman-001 across SST-2, TREC, and RTE, leaves code-davinci-002 relatively unaffected, and yields mixed changes for code-davinci-001.The comparison uses Codex models and datasets overlapping with Zhao et al. (2021).Source: §B.5 · p. 20, Figure 15 · p. 20
• Semantic prompt wording may steer small models toward semantic priors instead of in-context mappings, whereas large models remain able to learn unrelated mappings despite the wording.The experiment directly tests whether semantic information changes reliance on priors versus input–label mappings.Source: §B.5 · p. 20

B.6 LARGE MODELS ARE ROBUST TO OUT-OF-DISTRIBUTION DATASETS

• The OOD experiment combines SST-2 and Rotten Tomatoes inputs, prompting on one sentiment dataset and evaluating on the other.It tests whether scale-related OOD robustness also appears when InstructGPT models must use SUL-ICL mappings across input distributions.Source: §B.6 · p. 20
• Large models remain robust to out-of-distribution datasets in SUL-ICL, whereas small models lose up to 36% accuracy.text-curie-001 loses 4% accuracy and text-davinci-001 shows no significant change, suggesting OOD robustness emerges with scale even without semantic targets.Source: §B.6 · p. 20, Table 2 · p. 21

C.1 THE FLIPPED LABELS SETTING

• The per-dataset flipped-label analysis excludes PaLM on RTE because k = 16 exemplars per class exceed the maximum context length.Accuracy is plotted against the proportion of flipped labels for each dataset and model family.Source: §C.1 · p. 21
• In flipped-label experiments, large models initially outperform small models but suffer larger performance drops as more labels are flipped.This pattern is strongest for InstructGPT and weaker for Codex and PaLM; base GPT-3 largely lacks it because its largest models often perform near guessing.Source: §C.1 · p. 21
• The flipped-label trend appears on FP, RTE, and ETHOS, while SUBJ performance declines across all models and families.SUBJ may make it easier to follow flipped labels, though the reason for this task-specific behavior is unclear.Source: §C.1 · p. 21

C.2 THE SUL-ICL SETTING

• Base GPT-3 does not consistently follow the SUL-ICL scaling trend: davinci often suffers the largest drops with semantically unrelated targets.This behavior is unexplained but is consistent with davinci resembling small models in its limited use of input–label mappings.Source: §C.2 · p. 21
• Across many datasets and model families, larger models use more in-context exemplars effectively in SUL-ICL than smaller models.The experiments omit InstructGPT and davinci for cost reasons, and some curves stop before k = 32 because of context-length limits.Source: §C.2 · p. 21
• For InstructGPT, Codex, and PaLM, larger models lose less performance than smaller models when natural-language labels are replaced by semantically unrelated targets.This indicates greater capacity to learn input–label mappings when semantic priors are unavailable.Source: §C.2 · p. 21, Figure 17 · p. 23

C.3 INSTRUCTION TUNING

• Instruction-tuned Flan-PaLM models outperform corresponding PaLM models on SUL-ICL for SST-2, QQP, RTE, and ETHOS, although exceptions include SUBJ.On SST-2, Flan-PaLM-8B exceeds PaLM-8B by 28% and PaLM-62B by 2%, whereas PaLM-8B exceeds Flan-PaLM-8B on SUBJ by 23%.Source: §C.3 · p. 23, Figure 19 · p. 24
• In SUL-ICL, Flan-PaLM can score close to 0% with one exemplar per class, but this issue generally disappears when two or more exemplars per class are provided.The authors suggest that one-shot Flan-PaLM responses may fall outside the target label set, such as “Foo” or “Bar.”Source: §C.3 · p. 24
• Instruction-tuned models are worse than pretraining-only models at overriding semantic priors when labels are flipped in-context.Across all datasets and model sizes, Flan-PaLM performs better than PaLM in the flipped-label setting, while PaLM accuracy declines less as more labels are flipped.Source: §C.3 · pp. 24–25, Figure 20 · p. 25

C.4 LINEAR CLASSIFICATION

• The largest Codex model performs linear classification through N = 64 dimensions, whereas smaller Codex models reach guessing performance around N = 16.All PaLM models reach at least N = 8 with little scale-related difference, and neither model family outperforms the polynomial SVM baseline.Source: §C.4 · p. 26, Figure 21 · p. 26
• PaLM-540B does not outperform PaLM-8B or PaLM-62B on high-dimensional linear classification, indicating that model size alone does not explain the Codex result.The scaling factor enabling Codex’s ability is unknown and left for future work.Source: §C.4 · p. 26

D FULL PROMPT EXAMPLES

• The appendix provides full few-shot prompts for seven datasets using 16 exemplars per class, the default Input/Output template, and natural-language targets.SUL-ICL and flipped-label prompts are produced by replacing the labels, and the authors warn that ETHOS examples may contain offensive language drawn directly from the datasets.Source: D FULL PROMPT EXAMPLES · p. 27
• The linear-classification prompt uses the same default setup but assigns “Foo” to negative examples and “Bar” to positive examples.This task is presented only in the SUL-ICL setting, with the generation procedure specified in Algorithm 1.Source: §C.3 · p. 26, D FULL PROMPT EXAMPLES · p. 27

D.2 SUBJ

• D.2 SUBJ examples pair review-like or descriptive inputs with “Subjective Sentence” or “Objective Sentence” outputs.Subjective labels accompany evaluative statements, while objective labels accompany factual or plot-oriented descriptions in the supplied examples.Source: §D.2 · p. 30
• Other D.2 SUBJ examples classify factual or narrative descriptions as “Objective Sentence,” including historical events, character actions, and plot summaries.Examples include submarine-launched buzz bombs, characters pursuing quests, and documentary footage about a rapper.Source: §D.2 · pp. 30–31
• The D.2 SUBJ examples include subjective evaluations such as calling a romantic comedy “predictable” and recommending it “should get a pink slip.”That input is followed by the output “Subjective Sentence.”Source: §D.2 · p. 31

D.3 TREC

• D.3 TREC examples present fact-seeking questions spanning people, places, definitions, history, science, and popular culture.The supplied prompts ask about Madonna’s real name, HIV, volcanoes, Queen Victoria, LASER, and many other topics.Source: §D.3 · pp. 31–33
• Some D.3 TREC prompts are followed by the label “Description and Abstract Concept.”This output appears after grouped questions concerning pronunciation, causes, origins, locations, and web-page design.Source: §D.3 · pp. 31–33
• The displayed D.3 TREC material also includes an isolated question about the abbreviated form of the National Bureau of Investigation.No corresponding output is shown in the supplied passage.Source: §D.3 · p. 33
• The TREC examples contain diverse factual questions, including location, health, abbreviations, definitions, history, and science prompts.Examples ask about McCarren Airport, tuberculosis mortality, IOC, HTML, the Vietnam War, and fluorescent-bulb gas.Source: §D.3 · pp. 34, 35, 37
• Several TREC inputs receive the output label “Description and Abstract Concept.”This output appears after groups of questions about topics such as airliners, hydroelectric dams, children’s rights, and Colin Powell.Source: §D.3 · p. 35
• The TREC appendix also includes questions about people, places, objects, cultural references, and general knowledge.Examples concern Colin Powell, Mediterranean islands, camcorders in Malaysia, the Sun’s distance from Earth, Frosted Flakes, and the Tarot deck.Source: §D.3 · pp. 35–36

D.4 QQP

• The QQP examples present pairs of questions about potentially equivalent topics, such as Indian currency, gift cards, petroleum engineering, Minecraft, and employment.Each pair is shown as an input question followed by a related question, including questions about HR jobs, Python learning, and career opportunities after a B.Tech.Source: §D.4 · p. 37
• Additional QQP pairs cover caste or surname classification, Kashmir, online advertising, sadness, dieting, and graduate apprenticeships.The examples include questions about the Saini and Bera surnames, burning schools in Kashmir, Chrome ads, feeling sad, losing 10 pounds, and HAL opportunities.Source: §D.4 · pp. 37–38
• The later QQP examples include topical and personal questions about black money, Jawaharlal Nehru, PM Modi, loneliness, dreams, relationships, clothing, and taste.These examples range from demonetization and political questions to loneliness, dreaming, breakup advice, fashion, and taste.Source: §D.4 · pp. 38–39
• QQP examples pair questions about topics including cricket, heuristics, suicide, freelancing, photography, weight loss, history, technology, and prices.The examples include both closely related and apparently different question formulations.Source: §D.4 · p. 39
• QQP also includes morally or socially complex question pairs, such as theft for charitable purposes and helping a child afford medicine.These prompts extend beyond straightforward factual or paraphrase comparisons.Source: §D.4 · pp. 39–40

D.5 FP

• FP examples consist largely of financial news statements reporting changes in profits, sales, operating income, earnings, cash flow, prices, and orders.The statements include both increases and decreases, often with comparison periods and numerical values.Source: §D.5 · pp. 40–41
• Additional FP inputs describe company outlooks, market movements, acquisitions, airline traffic, subscriptions, and other business developments.The examples cover varied corporate and market contexts rather than a single financial relation.Source: §D.5 · pp. 41–42

D.6 ETHOS

• ETHOS examples contain toxic, hateful, threatening, demeaning, and discriminatory comments targeting people, religions, nationalities, genders, sexual orientations, and disabilities.The inputs range from short insults to longer ideological or political statements.Source: §D.6 · pp. 42–44
• The cited inputs include sympathy about hardship, praise for an Asian woman’s intelligence, upbeat self-expression, and extended reflection on anxiety, depression, mental illness, awareness, and influence.Together, these passages show varied social and emotional language, including personal distress, appreciation, enthusiasm, and discussion of mental-health discourse.Source: §D.6 · pp. 43–45

D.7 RTE

• The RTE examples provide a long news premise about tornado deaths and rescue efforts followed by a shorter statement about the governor visiting the affected cities.The pair uses disaster reporting and a related factual statement as the entailment inputs.Source: §D.7 · p. 45
• The examples include supported entailments, such as glue-sniffing being common among youngsters and Neil Armstrong landing on the Moon.These candidate statements restate or summarize information explicitly given in their inputs.Source: §D.7 · p. 45
• Several pairs test entailment through named people and events, including Anna Politkovskaya’s murder, Boris Becker’s Wimbledon title, and Patrick Nottingham’s scandal involvement.The inputs provide the relevant biographical or event information needed for each statement.Source: §D.7 · pp. 45, 47, 48
• Additional pairs cover political, military, and international affairs, including luxury taxes, Iraq’s interim government, North Korean weapons, and renewed soldier immunity.These examples require distinguishing stated actions or policies from altered participants, outcomes, or implications.Source: §D.7 · pp. 46–47
• Other examples test whether geographic and institutional claims follow from the input, including Duke University’s location, the WTO headquarters, and the UCLA/Hammer Museum’s director.The candidate statements vary in whether they preserve, narrow, or alter the input’s stated relationships.Source: §D.7 · p. 46
• The remaining examples include scientific, commercial, and everyday claims about rabies, maglev trains, Vodafone subscribers, and rising paper costs.Together they broaden the prompts beyond named entities to diseases, transport, business trends, and ordinary price changes.Source: §D.7 · pp. 46, 47, 49
• The set also contains contradiction or unsupported-claim cases, such as saying Ian Tomlinson was shot, Wehkamp was based in the UK, or Babe Ruth hit 755 home runs.Each candidate introduces information that conflicts with or is not established by its corresponding input.Source: §D.7 · pp. 46–47
• The examples further test temporal, numerical, and comparative facts, including the 23-day spaceflight record, the ECB meeting date, and the world’s highest skyscraper being built in Dubai.These statements depend on preserving specific dates, quantities, or rankings from the input.Source: §D.7 · pp. 48–49