Language Models are Few-Shot Learners

1 Introduction

• Thousands to hundreds of thousands of task-specific examples are typically needed for strong fine-tuned performance, limiting applicability when labeled datasets are difficult to collect.Humans often achieve reasonable competence from a brief instruction or a few demonstrations, enabling flexible switching and mixing of language tasks.Source: §1 · pp. 3–4
• Large models fine-tuned on narrow task distributions can exploit spurious correlations, making benchmark performance overstate underlying task competence [GSL+18] [NK19].Larger models do not necessarily generalize better out of distribution [HLW+20]; evidence links poor generalization to excessive specificity to training distributions [YdC+19] [MPL19].Source: §1 · p. 3
• 4% on Natural Questions and 55 F1 on CoQa illustrate earlier in-context learning’s substantial shortfall relative to fine-tuning.The cited CoQa result was more than 35 points behind the state of the art, motivating substantial improvements to make this approach practical.Source: §1 · p. 4
• Larger models generally perform better and gain more from in-context demonstrations, with few-shot performance improving faster than zero-shot performance.Comparisons include smaller models ranging from 125 million to 13 billion parameters; task descriptions and additional demonstrations also improve performance on the illustrated symbol-removal task. Figure 1.3 aggregates 42 accuracy-denominated benchmarks, but the authors caution that this aggregate is not itself a rigorous or meaningful benchmark.Source: Figure 1.3 · p. 5, §1 · pp. 5–6
• 175 billion parameters underpin GPT-3, evaluated on over two dozen NLP datasets and novel tasks through zero-, one-, and few-shot prompting rather than fine-tuning.In-context learning conditions predictions on instructions and demonstrations without gradient updates; few-shot prompts typically contain 10 to 100 demonstrations. These labels describe inference-time demonstration counts, without deciding whether the model learns new tasks or recognizes training-time patterns.Source: §1 · pp. 4–5
• 71.2% few-shot accuracy on closed-book TriviaQA establishes state-of-the-art performance relative to fine-tuned models; GPT-3 sometimes matches or surpasses fine-tuned systems.TriviaQA accuracy is 64.3% zero-shot and 68.0% one-shot. CoQA F1 is 81.5 zero-shot, 84.0 one-shot, and 85.0 few-shot.Source: §1 · p. 5
• GPT-3 demonstrates rapid adaptation on word unscrambling, arithmetic, and novel-word use, and generates few-shot news articles that human evaluators struggle to distinguish from human writing.The paper also discusses bias, fairness, and broader societal impacts, offering a preliminary analysis of GPT-3’s characteristics.Source: §1 · pp. 5–6
• Few-shot GPT-3 still struggles on ANLI, RACE, and QuAC, while training-data contamination potentially inflates results on some datasets.Systematic contamination analysis finds minimal effects on most datasets. Depending on severity, potentially affected results are withheld or marked with an asterisk.Source: §1 · pp. 5–6

2 Approach

• One-shot supplies one demonstration plus a task description; zero-shot supplies only instructions, potentially leaving the task’s format ambiguous.One-shot resembles how tasks are sometimes communicated to humans; zero-shot offers maximum convenience and potential robustness but can be unfairly difficult without examples. The authors consider one-shot, and sometimes zero-shot, the fairest comparisons to human performance.Source: §2 · pp. 6–7
• GPT-3 compares zero-, one-, and few-shot inference as performance–sample-efficiency trade-offs, highlighting many few-shot results only slightly behind state-of-the-art fine-tuned models.Few-shot inference typically conditions on 10 to 100 demonstrations within a 2048-token context without weight updates. Fine-tuning instead updates weights using thousands to hundreds of thousands of labels, risking poor out-of-distribution generalization [MPL19] and exploitation of spurious features [GSL+18] [NK19].Source: §2 · pp. 6–7, Figure 2.1 · p. 7

2.1 Model and Architectures

• Eight models span 125 million to 175 billion parameters, adapting GPT-2 with alternating dense and locally banded sparse attention similar to Sparse Transformer.The architecture retains GPT-2’s modified initialization, pre-normalization, and reversible tokenization; every model uses a 2048-token context. Multiple sizes allow testing predicted power-law validation-loss scaling and scaling on downstream language tasks.Source: §2.1 · p. 8

2.2 Training Dataset

• Training combines quality-filtered, document-deduplicated Common Crawl with curated corpora, sampling higher-quality datasets more frequently rather than proportionally to dataset size.Curated sources include expanded WebText, first described in, Books1, Books2, and English-language Wikipedia. Training uses 300 billion tokens, seeing some datasets up to 3.4 times but Common Crawl and Books2 less than once, accepting some overfitting for higher-quality data.Source: §2.2 · p. 8, Table 2.2 · p. 9
• A filtering bug left some benchmark development- and test-set overlaps in pre-training data despite attempted removal, creating a contamination concern.Training costs made retraining infeasible; Section 4 characterizes the impact of remaining overlaps.Source: §2.2 · p. 9

2.3 Training Process

• Training uses gradient-noise-guided batch sizes and model parallelism within matrix multiplications and across layers on Microsoft’s high-bandwidth V100 GPU cluster.Larger models generally permit larger batches but require smaller learning rates. Partitioning models across GPUs along depth and width minimizes inter-node data transfers.Source: §2.1 · p. 8, §2.3 · p. 9

2.4 Evaluation

• Evaluation randomly samples demonstrations for each example, tunes their count on development data when possible, and reports test results where accessible.LAMBADA and Storycloze use development-set demonstrations and test-set evaluation; original Winograd draws demonstrations from its sole dataset. Private-test tasks generally use development results because of server model-size limits, with successful few-shot test submissions for SuperGLUE, TriviaQA, and PiQa.Source: §2.4 · p. 10
• Multiple-choice tasks compare completion likelihoods, binary tasks use meaningful option labels, and free-form tasks use beam search scored with dataset-standard metrics.Most multiple-choice tasks normalize per token; ARC, OpenBookQA, and RACE benefit from normalization by unconditional answer likelihood. Free-form decoding uses beam width 4 and length penalty 0.6, following, with F1 similarity score, BLEU, or exact match.Source: §2.4 · p. 10

3 Results / 3.1.1 Language Modeling

• 20.50 zero-shot perplexity on Penn Tree Bank sets a new state-of-the-art, improving the previous result by 15 points.Cross-entropy validation loss extends the compute power-law observed in by two orders of magnitude with only slight deviations, alongside broad downstream gains. Penn Tree Bank is evaluated only zero-shot; Wikipedia-derived benchmarks and the one-billion-word benchmark are excluded because of training-data overlap.Source: §3 · p. 10, Table 3.1 · p. 11, §3.1.1 · p. 11

3.1.2 LAMBADA / 3.1.3 HellaSwag / 3.1.4 StoryCloze

• 86.4% few-shot accuracy on LAMBADA exceeds the previous state-of-the-art by over 18%, with gains increasing strongly with model size.Fill-in-the-blank demonstrations specify single-word completion, but one-shot formatting underperforms zero-shot; detected training contamination appears to have negligible impact. Few-shot HellaSwag accuracy reaches 79.3%, above the fine-tuned baseline’s 75.4% [ZHR+19] but below ALUM’s 85.6%, while StoryCloze reaches 87.7% with K = 70, remaining 4.1% below fine-tuned SOTA [LDL19].Source: Table 3.2 · p. 12, §3.1.2 · pp. 12–13, §3.1.3 · p. 13

3.2 Closed Book Question Answering

• 71.2% few-shot performance on TriviaQA exceeds the open-domain fine-tuning state-of-the-art that GPT-3’s 68.0% one-shot result matches [LPP+20].Few-shot WebQuestions reaches 41.5%, versus 44.7% for fine-tuned T5-11B+SSM; Natural Questions reaches 29.9%, versus 36.6% for fine-tuned T5 11B+SSM. Performance scales smoothly across all three datasets without fine-tuning, while larger demonstration gains on WebQuestions and Natural Questions suggest adaptation to distribution differences.Source: Table 3.3 · p. 13, §3.2 · pp. 13–14

3.3 Translation

• Few-shot translation achieves similar average performance to prior unsupervised NMT, outperforming it into English but underperforming it in the reverse direction.Zero-shot performance trails prior unsupervised NMT; one demonstration adds over 7 BLEU and full few-shot prompting adds another 4 BLEU, although using paired examples prevents strict comparability with unsupervised methods. Performance improves smoothly with capacity, but English-to-Romanian remains over 10 BLEU behind prior unsupervised work, possibly reflecting the English-oriented tokenizer; Romanian-to-English finishes within 0.5 BLEU of overall SOTA [LHCG19b].Source: §3.3 · pp. 14, 16, Table 3.4 · p. 15

3.4 Winograd-Style Tasks

• 77.7% few-shot accuracy on Winogrande improves over 70.2% zero-shot, but remains below fine-tuned RoBERTa’s 79% and T5’s 84.6% state-of-the-art.Winogrande human performance is 94.0%. Original Winograd scores are 88.3%, 89.7%, and 88.6% in zero-, one-, and few-shot settings, showing no clear in-context gains; detected contamination appears to have only a small effect.Source: Table 3.5 · p. 16, §3.4 · p. 17

3.5 Common Sense Reasoning

• 82.8% few-shot accuracy on PIQA exceeds the prior 79.4% fine-tuned RoBERTa state-of-the-art, although potential contamination qualifies this result.PIQA sets SOTA in all evaluation settings, but in-context gains are small and inconsistent on both PIQA and ARC; few-shot ARC Challenge and Easy accuracies are 51.5% and 70.1%, respectively, trailing UnifiedQA [KKS+20] by 27% and 22%. OpenBookQA improves substantially with demonstrations but remains over 20 points below overall SOTA, roughly matching a fine-tuned BERT Large baseline.Source: Table 3.6 · p. 17, §3.5 · pp. 17–18

3.6 Reading Comprehension

• 69.8 F1 on few-shot SQuAD 2.0 improves almost 10 F1 over zero-shot, but reading-comprehension performance varies sharply across datasets.CoQA performance is within 3 points of the human baseline, whereas QuAC falls 13 F1 below an ELMo baseline. DROP exceeds the original fine-tuned BERT baseline but remains far below humans and symbolic-augmented SOTA [RLL+19], while RACE remains 45% behind SOTA.Source: Table 3.7 · p. 18, §3.6 · p. 18

3.7 SuperGLUE

• GPT-3 approaches state-of-the-art on SuperGLUE’s COPA and ReCoRD, while roughly matching fine-tuned BERT-Large on BoolQ, MultiRC, and RTE.Evaluation uses 32 examples per task without gradient updates; COPA ranks second behind fine-tuned T5. Few-shot scores reach 80.1% on WSC and 75.6% on CB, illustrating substantial variation across tasks.Source: Table 3.8 · p. 19, §3.7 · p. 20
• Fewer than eight examples per task let GPT-3 outperform fine-tuned BERT-Large on overall SuperGLUE, although WiC few-shot accuracy remained at chance (49.4%).Scores improved with model size and example count, tested up to 32 examples per task. GPT-3 outperformed fine-tuned BERT-large on four of eight tasks, but struggled with comparing sentences or snippets.Source: §3.7 · p. 20

3.8 NLI

• Few-shot GPT-3 175B closed almost half the gap between random chance and state-of-the-art performance on ANLI [NWD+19] Round 3, while smaller models remained near chance.Only the largest model convincingly exceeded RTE’s random baseline (56%), with few-shot performance similar to single-task fine-tuned BERT Large; smaller models scored approximately 33% on ANLI. ANLI Round 3 evaluation used 1,500 development examples, with an estimated standard deviation of 1.2%.Source: Figure 3.9 · p. 21, §3.8 · p. 21

3.9.1 Arithmetic

• 100% accuracy on two-digit addition, 98.9% on two-digit subtraction, 80.2% on three-digit addition, and 94.2% on three-digit subtraction demonstrated GPT-3’s few-shot arithmetic proficiency.Accuracy fell to 25–26% on four-digit operations and 9–10% on five-digit operations; two-digit multiplication reached 29.2%, and single-digit combined operations reached 21.3%. The 13-billion-parameter model solved two-digit addition and subtraction only half the time and other operations less than 10% of the time.Source: Figure 3.10 · p. 22, §3.9.1 · p. 22
• 0.8% of tested three-digit addition problems and 0.1% of subtraction problems matched searched training-data patterns, suggesting memorization explained only a trivial fraction of correct answers.Errors such as failing to carry a “1” also suggested attempted computation. Arithmetic performance strengthened from zero-shot to one-shot to few-shot, yet full GPT-3’s zero-shot performance exceeded every smaller model’s few-shot performance.Source: §3.9.1 · pp. 22–23, Table 3.9 · p. 23

3.9.2 Word Scrambling and Manipulation Tasks

• 66.9% few-shot accuracy on removing random insertions contrasted with 38.6% on cycling letters, 40.2% on easier anagrams, and 15.1% on harder anagrams; no model reversed words.One-shot performance dropped by half or more, and zero-shot performance was rare, suggesting—but not proving—test-time learning. These tasks require manipulating subword-token structure, while ambiguous cycling and anagram tasks additionally require search.Source: §3.9.2 · pp. 23–24, Table 3.10 · p. 23, Figure 3.11 · p. 24

3.9.3 SAT Analogies

• 65.2% few-shot accuracy on 374 SAT analogies exceeded college applicants’ 57% average; one-shot and zero-shot accuracies were 59.1% and 53.7%.The full 175-billion-parameter model improved by over 10% compared with the 13-billion-parameter model.Source: §3.9.3 · p. 24, Figure 3.12 · p. 25

3.9.4 News Article Generation

• 52% human identification accuracy for GPT-3’s approximately 200- and 500-word news articles approached chance, versus 86% and 88%, respectively, for deliberately bad control articles.Three preceding articles conditioned generation from a proposed title and subtitle. Factual inaccuracies, repetition, non sequiturs, and unusual phrasing provided possible detection cues, but were often subtle.Source: §3.9.4 · pp. 25–26, Table 3.11 · p. 26, Table 3.12 · p. 27

3.9.5 Learning and Using Novel Words

• GPT-3 was qualitatively tested on using a nonexistent word in a sentence after receiving its definition and one to five preceding examples.Representative completions used only the displayed in-context conditioning, with no additional task-specific information.Source: §3.9.5 · p. 26, Figure 3.16 · p. 29
• GPT-3’s generated sentences appeared to use the newly defined words correctly or plausibly in every case.It produced the plausible conjugation “screeghed” for “screeg,” although its use in the sentence was slightly awkward.Source: §3.9.5 · p. 29
• The novel-word task was few-shot for the broad task but one-shot for each specific word.The six examples used human-generated definitions and an initial human-generated answer, followed by GPT-3 answers. Generation proceeded continuously in one sitting without omitted examples or repeated prompts.Source: §3.9.5 · p. 29

3.9.6 Correcting English Grammar

• One human-generated correction conditioned GPT-3 to correct 5 additional English sentences, without omissions or repeated attempts.Prompts paired “Poor English Input” with “Good English Output”; the passage reports the evaluation procedure but does not describe its outcomes.Source: §3.9.6 · p. 29

Detection method

• A removal bug left benchmark overlaps in internet-sourced training data, prompting comparisons against clean subsets filtered using approximately 13-gram matches.Shorter examples were checked for whole-example overlap, and retraining was infeasible because of cost. Despite dataset and model sizes about two orders of magnitude larger than GPT-2’s, GPT-3 showed no significant overfitting against deduplicated validation data.Source: §4 · pp. 29, 31

Overall analysis

• Over 50% potential contamination was flagged for a quarter of benchmarks, yet most clean-subset scores changed negligibly, without correlation between contamination levels and performance differences.The authors conclude that their conservative detection method either substantially overestimated contamination or that contamination had little effect on performance.Source: §4 · p. 31, Figure 4.2 · p. 32

Other flagged tasks

• >90% of QuAC, SQuAD2, and DROP examples were flagged, but inspected overlaps contained source text rather than question/answer pairs.For WMT16 German-English, 25% were flagged with an associated 1-2 BLEU total effect size, but inspected matches were monolingual rather than paired translations. Reversal/anagram filtering used 2-grams and mostly flagged palindromes or trivial unscramblings, making cleaned subsets harder; symbol insertion produced spurious matches because overlap detection ignored non-letter characters.Source: §4 · p. 32

PIQA

• 3 percentage points of absolute PIQA performance were lost on the clean subset, a 4% relative decrease; 29% of examples were flagged.A similar decrease in a 25x smaller model suggested statistical bias rather than memorization, but this explanation could not be proved. Because source webpages appeared in training data, PIQA results received an asterisk for potential contamination.Source: §4 · p. 32

Winograd

• 2.6% lower Winograd performance on the clean subset accompanied 45% flagged examples; manual inspection confirmed 132 training-set schemas, prompting an asterisk on reported results.The schemas appeared in a different format from the evaluation prompts.Source: §4 · p. 32

Language modeling

• 4 Wikipedia language-modeling benchmarks and the Children’s Book Test were almost entirely contained in training data, preventing reliable clean-subset extraction and prompting omission of their results.Penn Tree Bank was unaffected because of its age and became the chief language-modeling benchmark.Source: §4 · p. 33

LAMBADA

• Within 0.5% of the full-dataset score, LAMBADA’s clean-subset score showed little performance impact despite substantial genuine contamination.The fill-in-the-blank format precludes the simplest form of memorization. Potential contamination was nevertheless noted alongside the results because the reported gains were large.Source: §4 · p. 33

Limitations

• Clean-subset distribution differences could mask memorization effects, although numerous near-zero changes and similar shifts in small models argue against this explanation.The authors describe contamination measurement as a best effort and call for further work in benchmark design and model training.Source: §4 · p. 33

5 Limitations

• GPT-3 can repeat itself, lose coherence, contradict itself, and produce non-sequiturs, while few-shot performance remains near chance on some comparison and reading-comprehension tasks.Comparison failures include WIC and ANLI; informal observations also suggest difficulty with common-sense physics despite success on PIQA [BZB+19].Source: §5 · p. 33
• Autoregressive-only experiments omit bidirectional architectures and denoising objectives, potentially disadvantaging comparison, fill-in-the-blank, and reading-comprehension tasks.Prior literature reports improved fine-tuning with these approaches over standard language models. Exploring GPT-3-scale bidirectional models and their few- or zero-shot learning is proposed as future work.Source: §5 · p. 33
• Scaling self-supervised prediction may hit limits because the objective weights tokens equally, forces tasks into prediction problems, and lacks grounding in other experience.Entity-focused prediction has shown benefits [RRS20], while absent physical or video grounding limits world context [BHT+20]. Proposed alternatives include learning objectives from humans, reinforcement-learning fine-tuning, and adding images [CLY+19].Source: §5 · pp. 33–34
• GPT-3 remains sample-inefficient during pretraining, consuming more text than humans encounter in a lifetime despite improved test-time sample efficiency [Lin20].Physical grounding and algorithmic improvements are suggested routes toward better pretraining sample efficiency.Source: §5 · p. 34
• Few-shot performance does not establish whether GPT-3 learns new tasks during inference or recognizes and adapts tasks encountered during pretraining.The balance may vary by task: word unscrambling and nonsense-word definitions seem plausible examples of new learning, whereas translation must have been learned during pretraining.Source: §5 · p. 34
• GPT-3-scale inference is expensive and inconvenient, limiting practical applicability; task-specific distillation could reduce model size [HVD15].Its broad collection of skills suggests aggressive task-specific compression may be possible, but distillation has not been tested at hundreds-of-billions-parameter scale.Source: §5 · p. 34
• GPT-3's decisions are difficult to interpret, its predictions on novel inputs are not necessarily well-calibrated, and inherited training-data biases can produce stereotyped or prejudiced content.The calibration concern is associated with greater benchmark-performance variance than humans; biased outputs are highlighted as a particular societal concern.Source: §5 · p. 34

6.1.1 Potential Misuse Applications

• Human-like text generation could lower barriers to misinformation, phishing, fraud, and other text-based harms.GPT-3’s hard-to-distinguish multi-paragraph outputs motivate assessing applications, actors, and incentives through security risk assessment. Beneficial uses include writing assistance and question answering; focusing on harms encourages mitigation rather than asserting that harms outweigh benefits.Source: §6 · p. 34, §6.1.1 · p. 35, §6.1 · p. 35

6.1.2 Threat Actor Analysis

• Monitoring found no successful low/mid-skill deployments, while expert consultations found no discernible APT operational changes after GPT-2; improved reliability could alter risks.Observed misuse discussions correlated with media coverage. Analysts cited no convincing advantage over existing text-generation methods and immature content-targeting and control techniques as reasons against substantial investment.Source: §6.1.2 · p. 35

6.1.3 External Incentive Structures

• Human filtering of stochastic outputs limits malicious scalability despite potential reductions in deployment costs.Economic incentives favor scalable, easily deployed tactics such as phishing. The authors anticipate greater malicious interest in consistent, steerable models and propose mitigation research, prototyping, and coordination with developers.Source: §6.1.3 · p. 35

6.2.1 Gender

• 83% of 388 tested occupations were more likely to receive male than female identifiers from GPT-3.Binary-pronoun tests found stronger male associations under competence prompts and more appearance-oriented descriptions of women. On Winogender [RNLVD18], GPT-3 175B achieved the highest tested accuracy (64.17%), with female versus male Occupant-sentence accuracies of 81.7% versus 76.7%, offering preliminary evidence of greater robustness in larger models.Source: §6.2.1 · pp. 36–37, §6.2.2 · p. 37

6.2.2 Race

• 5 of 7 models ranked ‘Black’ lowest in sentiment; ‘Asian’ ranked highest in 3 of 7, with differences narrowing marginally at larger sizes.These results concern explicitly race-primed generation, not unprompted behavior. Senti WordNet scores on co-occurring words can reflect socio-historical context, including negative sentiment in discussions of slavery.Source: §6.2.2 · p. 37, Figure 6.1 · p. 38

6.2.3 Religion

• GPT-3 associated Islam more strongly than other religions with ‘violent’, ‘terrorism’, and ‘terrorist’.These terms appeared among Islam’s top 40 most favored words; ordinary religious terms such as ramadan, prophet, and mosque also co-occurred disproportionately. The analysis generated completions for prompts concerning six religious categories.Source: §6.2.3 · p. 38

6.2.4 Future Bias and Fairness Challenges

• Bias mitigation should combine normative, technical, and empirical perspectives rather than pursue metric-only bias removal.The gender, race, and religion probes are preliminary, non-exhaustive, subjective signposting, inspired by Model Cards for Model Reporting. Future interventions should engage affected communities’ lived experience [BBDIW20] and address blind spots in metric-driven approaches [GG19] [NvNvdG19].Source: §6.2 · p. 36, §6.2.4 · p. 39

6.3 Energy Usage

• 0.4 kW-hr can generate approximately 100 pages with GPT-3 175B, despite pretraining requiring several thousand petaflop/s-days of compute.GPT-2 1.5B required tens of petaflop/s-days; training costs should be considered across a model’s lifetime of uses. Distillation [LHCG19a] and algorithmic progress could further improve efficiency.Source: §6.3 · p. 39

7 Related Work

• 10x larger than previous models scaling parameters and computation together, GPT-3 follows straightforward model enlargement rather than conditional-computation or computation-only approaches.Conditional computation [BLC13] increases parameter capacity without increased computation, whereas adaptive computation time [Gra16] and the universal transformer increase computation without increasing parameters.Source: §7 · p. 39
• 3 orders of magnitude of scaling produce relatively smooth gains on many, but not all, downstream tasks, alongside broadly continuing power-law loss trends.Earlier studies reported smooth power-law loss trends [HNA+17]; the present work suggests these largely continue, although a slight bend may be detectable.Source: §7 · p. 39
• Compression approaches, including ALBERT [LCG+19] and distillation, could complement scaling by reducing giant models’ latency and memory footprint.The related approaches include general language-model distillation [HVD15] and task-specific distillation [SDCW19] [JYS+19] [KR16].Source: §7 · pp. 39–40
• GPT-3 tests harder benchmarks and approaches question answering through in-context learning rather than fine-tuning or test-time corpus attention.Benchmarks include question answering [CCE+18] [MCKS18], reading comprehension [CHI+18] [RCM19], and adversarial datasets [NWD+19]. Its approach differs from fine-tuning an 11-billion-parameter model [RRS20] and test-time corpus attention [GLT+20], but could be combined with [GLT+20] [LPP+20].Source: §7 · p. 40
• In-context learning resembles RL2 and: adaptation occurs through activations without weight updates, while language-model pre-training supplies the outer loop.Earlier language-model meta-learning had more limited results and lacked systematic study. Gradient-based few-shot learning [SS20] and semi-supervised approaches such as UDA instead explore fine-tuning with little labeled data.Source: §7 · p. 40
• Large-scale pre-training appears to supply diverse tasks implicitly, contrasting with multi-task fine-tuning, which requires manually curated datasets and training curricula.Natural-language instructions appeared in supervised learning [MKXS18], language modeling, and multi-task fine-tuning, rather than weight-update-free in-context learning. Broader explicit task distributions might use procedural generation, human interaction, or active learning.Source: §7 · p. 40
• GPT-3 retains pure autoregressive modeling to focus on in-context learning and simplify implementation; incorporating algorithmic advances could improve downstream performance, especially with fine-tuning.Potential additions include denoising-based bidirectionality [DCLT18], encoder-decoder architectures [LLG+19], random training permutations [YDY+19], and improved data and training procedures [LOG+19].Source: §7 · p. 40

8 Conclusion

• 175 billion parameters: the model performs strongly across zero-shot, one-shot, and few-shot NLP tasks, sometimes nearly matching state-of-the-art fine-tuned systems.These results span many NLP tasks and benchmarks.Source: §8 · pp. 40–41
• Performance follows roughly predictable scaling trends without fine-tuning.Source: §8 · p. 41
• The model generates high-quality samples and demonstrates strong qualitative performance on tasks defined on-the-fly.Source: §8 · p. 41
• The authors discuss social impacts and suggest that, despite limitations and weaknesses, very large language models may contribute to adaptable, general language systems.Source: §8 · p. 41

Contributions

• Tom Brown, Ben Mann, Prafulla Dhariwal, Dario Amodei, Nick Ryder, Daniel M Ziegler, and Jeffrey Wu implemented large-scale models, training infrastructure, and model-parallel strategies.Prafulla Dhariwal also implemented an early codebase and memory optimizations for fully half-precision training. Rewon Child and Mark Chen developed an early model-parallel strategy; Rewon Child and Scott Gray contributed the sparse transformer.Source: Contributions · p. 42
• Tom Brown, Dario Amodei, Ben Mann, and Nick Ryder conducted pre-training experiments; Ben Mann and Alec Radford prepared and analyzed training data.Mann and Radford collected, filtered, deduplicated, and analyzed overlap in the training data. Radford also provided NLP advice, suggested tasks, contextualized results, and demonstrated the training benefit of weight decay.Source: Contributions · p. 42
• Melanie Subbiah and colleagues implemented downstream tasks, supporting software, and synthetic tasks; additional contributors implemented and tested beam search and worked on SuperGLUE.The downstream implementation team also included Ben Mann, Dario Amodei, Jared Kaplan, Sam McCandlish, Tom Brown, Tom Henighan, and Girish Sastry. Subbiah and Arvind Neelakantan implemented, experimented with, and tested beam search; Pranav Shyam worked on SuperGLUE and connections to few-shot learning and meta-learning literature.Source: Contributions · p. 42
• Jared Kaplan and Sam McCandlish used scaling laws to guide model and data scaling and showed that larger models learn more quickly in-context.They studied in-context learning curves, prompting, and evaluation; McCandlish led scaling analysis and advised Tom Henighan and Kaplan. Alec Radford originally demonstrated few-shot learning in language models; Ilya Sutskever advocated scaling large generative likelihood models and advised contributors.Source: Contributions · p. 42
• Cluster optimization and fast GPU kernels supported efficient training of the largest models.Benjamin Chess, Clemens Winter, Eric Sigler, Christopher Hesse, Mateusz Litwin, and Christopher Berner optimized the clusters; Scott Gray developed fast training kernels. Aditya Ramesh experimented with pretraining loss scaling, and Ben Mann implemented sampling without replacement.Source: Contributions · p. 42
• Dario Amodei designed and led the research, while Jack Clark led ethical-impact analysis covering fairness, human assessments, and broader impacts.Sandhini Agarwal analyzed fairness and representation; Girish Sastry and Amanda Askell conducted human evaluations; Ariel Herbert-Voss analyzed malicious-use threats; Gretchen Krueger edited and red-teamed policy sections. Amodei, Alec Radford, Tom Brown, Sam McCandlish, Nick Ryder, Jared Kaplan, Agarwal, Askell, Sastry, and Clark wrote the paper.Source: Contributions · p. 42

A Details of Common Crawl Filtering

• A logistic-regression classifier prioritized higher-quality Common Crawl documents using curated text as positive examples and unfiltered Common Crawl as negative examples.Positive examples included WebText, Wikipedia, and the web books corpus; features came from Spark’s standard tokenizer and HashingTF.Source: A Details of Common Crawl Filtering · p. 43
• α = 9 favored highly scored documents while retaining some out-of-distribution documents; reweighting improved quality measured by loss on out-of-distribution generative text samples.The parameter was chosen to match the classifier-score distribution on WebText.Source: A Details of Common Crawl Filtering · p. 43
• 10% average dataset-size reduction resulted from fuzzy deduplication within datasets and fuzzy removal of WebText from Common Crawl.To improve quality and prevent overfitting, Spark’s MinHashLSH used 10 hashes and the classifier’s features to remove highly overlapping documents. After quality filtering and deduplication, benchmark text was also partially removed.Source: A Details of Common Crawl Filtering · p. 43

B Details of Model Training

• All GPT-3 versions use Adam with β1 = 0.9, β2 = 0.95, and ϵ = 10^-8.The global gradient norm is clipped at 1.0, and weight decay of 0.1 provides a small amount of regularization [LH17].Source: B Details of Model Training · p. 43
• Learning rates follow cosine decay to 10% of their original values over 260 billion tokens, remaining at that level afterward.A linear learning-rate warmup covers the first 375 million tokens.Source: B Details of Model Training · p. 43
• Batch sizes increase linearly from 32k tokens to full size over the first 4–12 billion tokens, depending on model size; training always uses full 2048-token sequences.Shorter documents are packed together with end-of-text delimiters indicating unrelated context, enabling efficient training without special sequence-specific masking. Data are sampled without replacement until an epoch boundary to minimize overfitting.Source: B Details of Model Training · p. 43

Initial training set filtering

• Initial benchmark filtering removed matching 13-grams and surrounding 200-character windows rather than automatically discarding whole documents, reducing false positives affecting long documents.Grams were lowercase, punctuation-free, whitespace-delimited words; fragments under 200 characters and documents split into more than 10 pieces were discarded. Matches occurring in more than 10 training documents were ignored because most represented common phrases or boilerplate.Source: C Details of Test Set Contamination… · pp. 43–44

Overlap methodology

• Exact Apache Spark collisions defined dirty examples by any training-corpus N-gram overlap, with N chosen separately for each dataset.N was the fifth-percentile example length, ignoring punctuation, whitespace, and case, with a minimum of eight for non-synthetic tasks and a maximum of 13. Analysis covered the full training corpus, although training used only 40% of filtered Common Crawl documents.Source: C Details of Test Set Contamination… · p. 44
• A filtering bug on long documents caused almost complete overlap in several language-modeling benchmarks and the Children’s Book Test, which were excluded.Retraining on corrected data was infeasible because of cost. Test and validation splits showed similar contamination levels despite some test splits being unlabeled.Source: C Details of Test Set Contamination… · p. 44

Overlap results

• Clean-only evaluation measured relative score changes against all examples to assess whether overlap inflated downstream performance.A clean score more than 1% or 2% worse suggested possible overfitting; a substantially better clean score could indicate that easier examples were preferentially marked dirty.Source: C Details of Test Set Contamination… · p. 44
• 94% of DROP examples were marked dirty, but matching training documents contained source passages rather than questions or answers.The authors considered a distribution difference in the remaining 6% a more likely explanation for reduced clean-only performance than cheating. The overlap metric also produced many false positives for web-derived background information and examples shorter than eight words.Source: C Details of Test Set Contamination… · p. 44
• GPT-3 appeared relatively insensitive to contamination: increasing contamination raised clean/all performance-ratio variance without an apparent bias toward improved or degraded performance.This conclusion came from comparing performance on clean examples with performance on all examples.Source: C Details of Test Set Contamination… · p. 44

D Total Compute Used to Train Language Models

• Less than 10% of total compute typically comes from attention in the analyzed models, so the estimates omit attention operations.These approximate calculations underpin the training-compute estimates in Figure 2.2.Source: D Total Compute Used to Train Language… · p. 46
• 3x the forward-pass compute accounts for forward and backward passes; total training flops multiply per-parameter, per-token cost by parameter and training-token counts.The forward pass requires one addition and one multiplication per active parameter per token. Totals are reported in flops and petaflop/s-day, with one petaflop/s-day equaling 8.64e+19 flops.Source: Table D.1 · p. 46, D Total Compute Used to Train Language… · p. 46
• Half of T5's parameters are active for each token during a forward or backward pass because T5 uses an encoder-decoder architecture.The calculation adjusts the active parameter count accordingly.Source: Table D.1 · p. 46

Participants and quiz design

• 621 participants remained after 97 exclusions across six experiments, with randomized, complementary quizzes containing approximately equal numbers of human-written and model-generated articles.Participants were US-based Positly recruits who could participate only once. Each quiz contained 25 articles, shuffled independently for each participant.Source: E Human Quality Assessment of… · pp. 46–47

Procedure and design

• 25 early-2020 Newser articles supplied titles and subtitles for model-generated completions, with output selection controlling for article length.Each model generated five outputs per article; the completion closest to the human-written article’s word count was selected automatically.Source: E Human Quality Assessment of… · p. 46

Duration statistics

• Larger models produced harder-to-distinguish news articles, with human detection accuracy decreasing despite increased time spent judging the articles.Figure E.1 shows judgment time increasing with model size.Source: Figure E.1 · p. 47, E Human Quality Assessment of… · pp. 47–48

Statistical Tests

• Independent-group two-sample t-tests compared each model with the control, while a power law modeled average participant accuracy against model size.The 95% confidence intervals were estimated from the t-distribution of the sample mean.Source: E Human Quality Assessment of… · p. 47

Preliminary investigation of ∼500 word articles

• 160 US-based participants evaluated approximately 500-word articles in two preliminary experiments comparing GPT-3 175B with a 160M control model.Prompts used titles and locations from 12 Reuters articles, with a separate Reuters article providing GPT-3’s context. Balanced 12-question quizzes included comprehension questions and revealed articles in three stages at 30-second intervals.Source: E Human Quality Assessment of… · p. 48

Additional Samples from GPT-3

• The context includes poems attributed to C. P. Cavafy and John Ashbery before the requested Wallace Stevens title and author heading.Cavafy’s “The City” is identified as translated by Edmund Keeley; Ashbery’s contribution is titled “Some Trees.”Source: F Additional Samples from GPT-3 · p. 49
• The generated poems feature recurring imagery of shadows, woods, light, and solitude.Examples include an isolated small town, a traveler facing desolate mountains, and a speaker describing colors as companions.Source: F Additional Samples from GPT-3 · p. 49
• Four uncurated completions demonstrate GPT-3’s response to a prompt requesting a poem titled “Shadows on the Way” in Wallace Stevens’s style.After experimenting with prompts, the authors generated these samples without further editing or selection, using temperature 1 and nucleus sampling [HBFC19] with P = 0.9. Completions were truncated when a new title and author heading or prose commentary began.Source: F Additional Samples from GPT-3 · p. 48, Figure F.1 · p. 49

• The figures illustrate formatting and phrasing for all tasks using ground-truth dataset examples, with no GPT-3 samples.Source: G Details of Task Phrasing and… · p. 50
• RACE-h, RACE-m, OpenBookQA, ARC Challenge, and ARC Easy predictions normalize by each answer’s unconditional probability.Source: Figure G.1 · p. 50, Figure G.3 · p. 51, Figure G.8 · p. 53
• ReCoRD treats the displayed context as a single problem, matching the dataset’s presentation and evaluation script.Source: Figure G.6 · p. 52
• Winograd and Winogrande use partial evaluation, comparing completion probabilities given correct and incorrect contexts.Source: Figure G.13 · p. 53, Figure G.14 · p. 54
• MultiRC scores a question correct only when every answer is labeled correctly, and K counts questions shown in context.Its three organizational levels are passages, questions, and answers.Source: Figure G.15 · p. 54
• TriviaQA allows multiple valid completions.Source: Figure G.34 · p. 59
• Translation examples illustrate one- and few-shot formatting, while zero-shot prompts use an explicit translation question and answer.The zero-shot template is “Q: What is the {language} translation of {sentence} A: {translation}.”Source: Figure G.36

H Results on All Tasks for All Model Sizes

• Table H.1 collects scores for every task, setting, and model investigated in the paper.The supplied caption establishes coverage but provides no numerical scores or model comparisons.Source: Table H.1 · p. 63
• Dedicated figures present results for SuperGLUE, Winograd, common sense reasoning, and all ANLI rounds.These results appear in Figures H.1, H.3, H.6, and H.9, respectively.Source: Figure H.1 · p. 64, Figure H.3 · p. 64, Figure H.6 · p. 66
• Additional figures present results for SAT, arithmetic, and Scramble tasks.These results appear in Figures H.2, H.4, and H.10, respectively.Source: Figure H.2 · p. 64, Figure H.4 · p. 65, Figure H.10 · p. 67
• Separate figures cover question answering, reading comprehension, cloze and completion, and translation tasks.These results appear in Figures H.7, H.8, H.5, and H.11, respectively.Source: Figure H.5 · p. 65, Figure H.7 · p. 66, Figure H.8 · p. 66