Source-linked AI summary
Language Models are Few-Shot Learners
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, Dario Amodei
TL;DR
Task-specific fine-tuning requires large labeled datasets, limiting few-shot language-task performance. GPT-3 tests in-context learning without fine-tuning and performs strongly across tasks, sometimes matching or surpassing fine-tuned systems.
Problem
Strong NLP performance typically requires task-specific fine-tuning datasets containing thousands to hundreds of thousands of examples.
Method
GPT-3 is evaluated on over two dozen datasets and novel tasks using zero-, one-, and few-shot in-context learning without weight updates.
Results
GPT-3 performs strongly across NLP and rapid-adaptation tasks, with 71.2% accuracy on TriviaQA in the few-shot setting.
Takeaways & Limitations
Very large language models may be an important ingredient in adaptable, general language systems.
Takeaways & Limitations
GPT-3 still exhibits weaknesses including repetition, incoherence, contradictions, and non-sequitur passages in text synthesis.
Abstract
from arXiv · showhide
Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task. While typically task-agnostic in architecture, this method still requires task-specific fine-tuning datasets of thousands or tens of thousands of examples. By contrast, humans can generally perform a new language task from only a few examples or from simple instructions - something which current NLP systems still largely struggle to do. Here we show that scaling up language models greatly improves task-agnostic, few-shot performance, sometimes even reaching competitiveness with prior state-of-the-art fine-tuning approaches. Specifically, we train GPT-3, an autoregressive language model with 175 billion parameters, 10x more than any previous non-sparse language model, and test its performance in the few-shot setting. For all tasks, GPT-3 is applied without any gradient updates or fine-tuning, with tasks and few-shot demonstrations specified purely via text interaction with the model. GPT-3 achieves strong performance on many NLP datasets, including translation, question-answering, and cloze tasks, as well as several tasks that require on-the-fly reasoning or domain adaptation, such as unscrambling words, using a novel word in a sentence, or performing 3-digit arithmetic. At the same time, we also identify some datasets where GPT-3's few-shot learning still struggles, as well as some datasets where GPT-3 faces methodological issues related to training on large web corpora. Finally, we find that GPT-3 can generate samples of news articles which human evaluators have difficulty distinguishing from articles written by humans. We discuss broader societal impacts of this finding and of GPT-3 in general.
1 Introduction
GPT-3 tests whether scaling language models enables task-agnostic in-context learning from instructions and a few demonstrations, without gradient updates or fine-tuning. It achieves promising and sometimes state-of-the-art-level results while retaining important weaknesses and raising broader societal concerns.
- Motivation: The paper motivates in-context learning as a response to task-specific data requirements, narrow fine-tuning distributions, and the gap between current systems’ adaptability and human task learning.Humans can often learn from a brief directive or tiny number of demonstrations, whereas existing approaches require task-specific labeled datasets.
- Evaluation: GPT-3 is evaluated as a 175-billion-parameter autoregressive language model across over two dozen NLP datasets and novel rapid-adaptation tasks under zero-, one-, and few-shot conditions.The evaluation uses in-context demonstrations rather than gradient updates or fine-tuning.
- Results: GPT-3 achieves promising zero-shot and one-shot results and is sometimes competitive with or surpasses state-of-the-art fine-tuned models in few-shot settings.On CoQA, GPT-3 reaches 81.5 F1 zero-shot, 84.0 F1 one-shot, and 85.0 F1 few-shot.
- Capabilities: GPT-3 demonstrates one-shot and few-shot proficiency on unscrambling words, arithmetic, novel-word usage, and generating news articles that evaluators struggle to distinguish from human-written articles.These tasks test rapid adaptation or on-the-fly reasoning beyond standard NLP benchmarks.
- Limitations: Few-shot performance still struggles on some tasks, including ANLI natural-language inference and reading-comprehension datasets such as RACE and QuAC.The paper presents these limitations alongside GPT-3’s strengths to identify areas needing further progress.
- Broader impacts: The paper also discusses bias, fairness, and broader societal impacts associated with GPT-3’s broad capabilities.It includes a preliminary analysis of GPT-3’s characteristics in these areas.
2 Approach
GPT-3 uses a scaled-up GPT-2-style autoregressive architecture, curated and reweighted training data, and evaluation settings that provide zero, one, or few demonstrations without weight updates. The approach also addresses contamination risks from pretraining on broad internet data.
- In-context learning settings: Few-shot evaluation conditions the model on K demonstrations without updating weights, while one-shot uses one demonstration and zero-shot uses only a natural-language task instruction.All three settings require only forward passes at test time and trade off benchmark performance against sample efficiency.
- Training data: Training data combines filtered Common Crawl with curated WebText, Books1, Books2, and English Wikipedia, while sampling weights intentionally favor higher-quality datasets over proportional size.The resulting mixture includes Common Crawl collected from 2016–2019 and is trained for 300 billion tokens.
- Contamination control: The authors search for and attempt to remove overlaps between pretraining data and benchmark development or test sets to reduce downstream contamination.They identify contamination as a major concern because large language models can memorize vast amounts of broad internet content.
3 Results
GPT-3’s performance generally improves smoothly with model size and in-context examples, achieving strong or competitive results across language modeling, question answering, translation, and reasoning tasks. Gains are substantial but uneven, with persistent gaps, methodological concerns, and limited in-context improvement on some datasets.
- Language modeling: GPT-3 achieves 76% on LAMBADA in zero-shot evaluation, an 8% gain over the previous state of the art.The result is presented as evidence that continued scaling remains promising.
- Reasoning and classification: GPT-3 reaches 86.4% accuracy in a few-shot setting, improving over the previous state of the art by over 18%, while model size strongly affects performance.The same setting decreases the smallest model’s performance by almost 20% but improves GPT-3’s accuracy by 10%.
- Limitations and uneven gains: Results remain uneven: GPT-3 reaches 79.3% on HellaSwag and 87.7% on StoryCloze, but remains below the cited fine-tuned or overall state-of-the-art systems on both tasks.Winograd shows no clear in-context learning, while Winogrande improves from 70.2% zero-shot to 77.7% few-shot; PIQA is flagged for possible contamination.
- Question answering: On open-domain question answering, few-shot GPT-3 reaches 71.2% on TriviaQA, 41.5% on WebQuestions, and 29.9% on Natural Questions, approaching or matching strong fine-tuned systems on some datasets.One-shot GPT-3 matches the open-domain fine-tuning state of the art on one dataset, while few-shot performance approaches closed-book state-of-the-art results on the other two.
- Scaling and in-context learning: Across tasks, performance scales smoothly with model capacity, while few-shot demonstrations often improve results substantially over zero-shot performance.This trend is reported across question answering, translation, and multiple reasoning datasets.
- Translation: In translation, one example improves performance by over 7 BLEU and the full few-shot setting adds another 4 BLEU, reaching similar average performance to prior unsupervised NMT.Performance is stronger when translating into English and weaker in the reverse direction.
4 Measuring and Preventing Memorization Of Benchmarks
GPT-3’s internet-scale training creates substantial potential for benchmark contamination, motivating conservative cleaning and comparison of original versus uncontaminated scores. Most benchmark results changed negligibly, but several datasets required review or exclusion, and the analysis has important distributional limitations.
- Contamination analysis: Most benchmark scores changed negligibly after evaluation on conservatively cleaned subsets, though some benchmarks were flagged for further review.The cleaned subsets removed examples with potential 13-gram overlap or whole-example overlap for shorter examples.
- Reading comprehension: More than 90% of QuAC, SQuAD2, and DROP examples were flagged as potentially contaminated, but inspected overlaps contained source text without the associated question-answer pairs.This meant the training data provided background information rather than memorized answers to specific questions.
- Language modeling: The four Wikipedia language-modeling benchmarks and Children’s Book Test were almost entirely contained in training data, so GPT-3 results were not reported for them.Penn Tree Bank was unaffected by contamination and became the chief language-modeling benchmark.
- Contamination effects: LAMBADA showed substantial genuine contamination, yet its clean-subset score remained within 0.5% of the original.Other highly contaminated datasets often contained false positives or contamination that did not reveal task answers.
- Limitations: The analysis cannot guarantee that clean subsets match the original data distribution, so memorization could theoretically be offset by statistical differences that make the clean subset easier.The authors nevertheless argue that the many near-zero score shifts make this explanation unlikely.
5 Limitations
GPT-3 shows strong improvements but retains weaknesses in text synthesis and several NLP tasks, while its few-shot behavior, training objective, efficiency, interpretability, calibration, and biases pose important limitations. These limitations motivate future work on broader architectures, more efficient pretraining and inference, and safer, more capable language systems.
- GPT-3 still repeats or contradicts itself in long text, loses coherence, and performs near chance on some comparison and reading-comprehension tasks.
- The study excludes bidirectional architectures and denoising objectives, while equal token weighting and prediction-based task specification may limit the general approach.
- GPT-3 remains far less sample-efficient during pretraining than humans, despite moving toward human-like one-shot or zero-shot test-time efficiency.
- Few-shot performance may reflect recognizing tasks encountered during training rather than learning entirely new tasks at inference time.
- Models at GPT-3’s scale are expensive and inconvenient to run, difficult to interpret, potentially miscalibrated, and prone to reproducing training-data biases.
6 Broader Impacts
GPT-3’s increasingly human-like text generation raises risks of misuse, while its internet-scale training data produces substantial biases, including gender- and race-related stereotypes. Misuse was not yet widely deployed in observed cases, but greater consistency and steerability could increase future interest among malicious actors.
- Misuse potential: Language models can lower barriers to misinformation, spam, phishing, fraud, social engineering, and other harmful activities by generating high-quality text.The misuse potential increases as text synthesis improves, and people found several paragraphs of GPT-3-generated content difficult to distinguish from human-written text.
- Misuse potential: Observed misuse discussions after GPT-2 produced fewer experiments and no successful deployments, while threat analysts found no discernible operational change attributable to language models.Researchers nevertheless expect sufficiently consistent and steerable models to become more attractive to malicious actors.
- Bias analysis: The bias assessment was preliminary and focused on gender, race, and religion rather than exhaustively characterizing all model biases.The analysis broadly indicates that internet-trained models reflect stereotypes present in their training data.
- Bias analysis: GPT-3 175B achieved 64.17% accuracy on the Winogender pronoun-resolution task while retaining associations between occupations and male identities.The analysis found that language models learned societal biases, including tendencies to associate female pronouns more with participant positions than male pronouns.
- Bias analysis: ‘Black’ ranked lowest in sentiment in 5 of 7 analyzed models, whereas ‘Asian’ ranked first in 3 of 7, though differences narrowed marginally with larger models.The analysis used prompts explicitly focused on race, and sentiment based on word co-occurrences may reflect socio-historical factors.
7 Related Work
Related work spans scaling language models, compressing them, developing harder evaluations, and improving transfer through question answering, metalearning, multitask learning, and architectural innovation. The paper positions its in-context learning approach as complementary to these directions while emphasizing learning without weight updates.
- Scaling and efficiency: Language-model research has scaled parameter counts and computation substantially, with smooth power-law improvements in loss and generally increasing downstream-task performance.Scaling has progressed from billion-parameter LSTMs to successively larger transformer models, while the reported trends extend across three orders of magnitude.
- Scaling and efficiency: Model compression work, including ALBERT and general or task-specific distillation, seeks strong performance from smaller models and could reduce giant-model latency and memory use.These approaches are described as potentially complementary to scaling large models.
- Evaluation and question answering: As fine-tuned models approached human performance on standard benchmarks, researchers developed harder question-answering, reading-comprehension, and adversarial datasets, many of which this work evaluates.Prior question-answering efforts include fine-tuning an 11-billion-parameter model and attending over large test-time corpora.
- Learning and transfer: Prior metalearning, gradient-based few-shot learning, semi-supervised learning, and natural-language task instructions pursue related goals, whereas this approach uses context examples without weight updates.Natural-language task presentation had previously been studied in supervised or multitask fine-tuning settings rather than purely in-context learning.
- Learning and transfer: Related transfer approaches include multitask learning and numerous architectural, data, and training innovations that improve language-model generality, efficiency, or downstream performance.The cited innovations include bidirectional denoising, prefixLM, encoder-decoder architectures, random training permutations, more efficient sampling, improved procedures, and embedding-parameter efficiency.
8 Conclusion
GPT-3 demonstrates strong zero-, one-, and few-shot performance across many NLP tasks, sometimes nearly matching fine-tuned state-of-the-art systems, while revealing predictable scaling trends and important limitations and social impacts.
- 8 Conclusion: GPT-3 shows strong performance across many NLP tasks and benchmarks without fine-tuning, in some cases nearly matching state-of-the-art fine-tuned systems.The model also generates high-quality samples and performs strongly on tasks defined on-the-fly.
- 8 Conclusion: Performance scales in roughly predictable ways as language models grow, even without fine-tuning.
- 8 Conclusion: Despite limitations and weaknesses, very large language models may help develop adaptable, general language systems.The paper also discusses the social impacts of this class of model.
Contributions
The project combined leadership, scaling analysis, model and infrastructure engineering, data preparation, task implementation, and evaluations of fairness, human judgments, and malicious-use risks. Contributors also developed key training and model-parallel techniques and wrote the paper.
- The team implemented the large-scale models, training infrastructure, model-parallel strategies, an early codebase, and memory optimizations for fully half-precision training.
- Researchers conducted pre-training experiments, prepared and analyzed training data, implemented downstream tasks and synthetic tasks, and developed sampling without replacement during training.
- The project’s scaling and in-context learning work predicted continued gains from larger models and studied scaling laws, learning curves, prompting, and evaluation methods.
- The team evaluated fairness and representation, human assessments, broader ethical impacts, and threats from malicious use, while advising and editing the paper’s policy analysis.
- Project leadership included designing and leading the research, NLP guidance and task suggestions, early advocacy for scaling, and writing the paper.
A Details of Common Crawl Filtering
The authors improved Common Crawl quality by classifier-based re-sampling and fuzzy deduplication, then partially removed text overlapping benchmark datasets. Classifier re-weighting improved out-of-distribution generative-text loss, while deduplication reduced dataset size by 10% on average.
- A Details of Common Crawl Filtering: A classifier trained on WebText versus raw Common Crawl re-sampled documents by prioritizing those predicted to be higher quality.The classifier used logistic regression with Spark features.
- A Details of Common Crawl Filtering: Setting α = 9 favored highly scored documents while retaining some out-of-distribution examples and improved loss on out-of-distribution generative-text samples.The value matched the classifier-score distribution on WebText.
- A Details of Common Crawl Filtering: Fuzzy deduplication removed highly overlapping documents within each dataset and removed WebText from Common Crawl using Spark MinHashLSH with 10 hashes.The procedure was intended to improve model quality and prevent overfitting as model capacity increased.
- A Details of Common Crawl Filtering: Deduplication decreased dataset size by an average of 10%, after which text occurring in benchmark datasets was also partially removed.Benchmark-text removal is described in Appendix C.
B Details of Model Training
GPT-3 training uses Adam with gradient clipping, a warmup and cosine-decay learning-rate schedule, and a gradually increasing batch size. Training always uses full 2048-token sequences, packing shorter documents together with end-of-text delimiters for efficiency and context separation.
- B Details of Model Training: GPT-3 training uses Adam, clips global gradient norms at 1.0, and applies warmup followed by cosine learning-rate decay.The learning rate decays to 10% over 260 billion tokens, after which training continues at that rate.
- B Details of Model Training: The batch size increases linearly from 32k tokens to its full value during the first 4–12 billion tokens.
- B Details of Model Training: Training always fills the 2048-token context window, packing shorter documents together and separating them with end-of-text tokens.Sequences containing multiple documents are not specially masked; the delimiter indicates that their contexts are unrelated.
C Details of Test Set Contamination Studies
The contamination study defines dataset examples as dirty when they share an N-gram with training data, then compares clean-only with overall performance. Despite methodological limitations and false positives, Figure 4.2 finds no apparent performance bias as contamination increases, suggesting relative insensitivity.
- Overlap methodology: Examples are classified as dirty when any N-gram overlaps a training document and clean when no such collision exists.The overlap length N is dataset-dependent, bounded between 8 and 13 for non-synthetic tasks, and the resulting values and dirty-data amounts appear in Table C.1.
- Limitations: A filtering bug caused near-complete overlap in several language-modeling benchmarks and the Children’s Book Test, so those datasets were excluded rather than retrained on corrected data.The bug affected long documents such as books, and retraining was infeasible because of cost.
- Overlap results: The analysis evaluates clean-only versus original scores and uses their relative change to assess whether exposure to overlapping examples may have caused overfitting.A clean score more than 1% or 2% below the overall score is treated as suggestive of overfitting.
- Limitations: The overlap metric can produce false positives for web-derived background information and short examples, and it gives weak signal for DROP, where 94% of examples are dirty.These limitations complicate interpreting contamination effects for some datasets.
- Overlap results: As contamination increases, the variance of the clean/all performance fraction rises, but there is no apparent bias toward improved or degraded performance.This suggests GPT-3 is relatively insensitive to contamination.
D Total Compute Used to Train Language Models
This appendix estimates the compute used to train the language models in Figure 2.2 by ignoring attention, which typically contributes less than 10% of total compute. It calculates total training flops from model parameters, training tokens, and per-token forward/backward operations, reporting results in flops and petaflop/s-days.
- Assumptions: Attention is omitted as a simplifying assumption because it typically uses less than 10% of total compute for the analyzed models.
- Compute calculation: Training compute is estimated by multiplying total parameters, training tokens, and flops per parameter per token.The calculation accounts for active parameters, per-token additions and multiplications, and a 3x multiplier for backward computation.
- Reporting units: The appendix reports training compute in both flops and petaflop/s-days, with one petaflop/s-day equal to 8.64e+19 flops.
E Human Quality Assessment of Synthetic News Articles
Human participants found larger GPT-3 models’ news articles harder to distinguish from human writing, even while spending more time making judgments. A preliminary study also examined longer, approximately 500-word articles generated by GPT-3 175B and a 160M control model.
- Procedure and design: The study compared human-written and model-generated articles across models ranging from 125M to GPT-3’s 200B parameters using paired quizzes.Participants evaluated shuffled sets of 25 articles, half human-written and half model-generated, with complementary article assignments across quizzes.
- Duration statistics: As model size increased, participants spent more time identifying whether news articles were machine-generated, while their accuracy decreased.The increased time investment alongside lower accuracy supports the conclusion that larger models generated harder-to-distinguish articles.
- Preliminary investigation of ∼500 word articles: A preliminary investigation recruited 160 US-based participants to assess approximately 500-word articles generated by GPT-3 175B and a 160M control model.The study used 12 Reuters world news articles and generated completions from titles and locations, with one additional Reuters article providing GPT-3 context.
F Additional Samples from GPT-3
GPT-3 adapts to creative writing prompts, producing four uncurated poem completions in the style of Wallace Stevens without additional editing or selection.
- F Additional Samples from GPT-3: Four uncurated completions followed a prompt to compose a poem titled ‘Shadows on the Way’ in the style of Wallace Stevens.The samples were generated with temperature 1 and nucleus sampling with P = 0.9.
G Details of Task Phrasing and Specifications
This section documents the text formats and task specifications used throughout the evaluation suite. The examples cover diverse reasoning, comprehension, translation, arithmetic, and completion tasks, with task-specific scoring conventions noted where applicable.
- Task formatting: The appendix provides formatted dataset examples spanning multiple-choice reasoning, reading comprehension, commonsense inference, completion, translation, and arithmetic tasks.Examples are shown for tasks including RACE, ANLI, PIQA, COPA, ReCoRD, ARC, Winograd, MultiRC, CoQA, DROP, TriviaQA, and several others.
- Scoring specifications: Several multiple-choice tasks normalize each answer by its unconditional probability during prediction.This convention is stated for RACE-h, RACE-m, OpenBookQA, ARC, and related formatted examples.
- Scoring specifications: ReCoRD treats the displayed context as a single problem because that matches the dataset presentation and evaluation script.This specification determines how the task is grouped and scored.
- Scoring specifications: Winograd and Winogrande use a partial evaluation method comparing completion probabilities under correct and incorrect contexts.The method compares the probability of the completion given each alternative context.
- Scoring specifications: MultiRC is evaluated at the question level, counting a question as correct only when all of its answers are labeled correctly.The appendix uses K for the number of questions shown within the context.
- Translation: For one- and few-shot translation, inputs are presented as language-specific prompts followed by a target completion, while zero-shot prompts explicitly request the translation.The German-to-English example pairs a source sentence with the completion “In no case may they be used for commercial purposes.”
H Results on All Tasks for All Model Sizes
This section presents scores for every investigated task, setting, and model, with task-specific figures covering the full evaluation suite.
- Table H.1 reports scores for every task, setting, and model investigated in the paper.
- The results include SuperGLUE, SAT, Winograd, and Arithmetic tasks.
- Additional figures cover Cloze and Completion, Common Sense Reasoning, QA, Reading Comprehension, ANLI, Scramble, and Translation tasks.