Language Models are Few-Shot Learners

Motivation

• Task-specific fine-tuning requires thousands to hundreds of thousands of labeled examples, limiting applicability and potentially encouraging narrow, spurious generalization.Humans often learn tasks from brief instructions or a few demonstrations, motivating more fluid and general language systems.Source: §1 · pp. 3–4
• Scaling transformer language models is proposed as a route to stronger in-context learning, after earlier meta-learning remained far below fine-tuning, including 4% on Natural Questions and 55 F1 on CoQA.The paper hypothesizes that larger models can absorb more skills and tasks in their parameters and use them at inference time.Source: §1 · p. 4

Evaluation setup

• GPT-3 is evaluated without fine-tuning under zero-, one-, and few-shot conditions, using natural-language instructions and up to typically 10–100 demonstrations in context.Increasing demonstrations and model size improves performance without gradient updates.Source: §1 · p. 5

Results

• 85.0 F1 on CoQA and 71.2% accuracy on TriviaQA are GPT-3’s few-shot results, with TriviaQA reaching state-of-the-art among comparable closed-book fine-tuned models.GPT-3 also scores 81.5/84.0 F1 on CoQA in zero-/one-shot settings and 64.3%/68.0% accuracy on TriviaQA.Source: §1 · p. 5
• GPT-3 demonstrates one- and few-shot ability on word unscrambling, arithmetic, novel-word usage, and synthetic news generation that evaluators struggle to distinguish from human writing.These tasks test rapid adaptation or on-the-fly reasoning beyond standard NLP benchmarks.Source: §1 · p. 5

Scope and limitations

• The paper also studies data contamination, model-size scaling, and societal impacts, finding contamination usually has minimal effect but may inflate results on some datasets and identifying weaknesses on ANLI, RACE, and QuAC.Smaller-model comparisons generally show smooth scaling, while zero-/one-/few-shot gaps often widen with capacity.Source: §1 · pp. 5–6

2 Approach

• The study compares fine-tuning, few-shot, one-shot, and zero-shot settings as distinct trade-offs between benchmark performance, sample efficiency, and resemblance to human task communication.Fine-tuning updates weights using typically thousands to hundreds of thousands of labeled examples, whereas in-context settings use only forward passes; few-shot typically uses 10–100 demonstrations within a 2048-token context window.Source: §2 · pp. 6–7, Figure 2.1 · p. 7

2.1 Model and Architectures

• GPT-3 uses a GPT-2 architecture with alternating dense and locally banded sparse attention, and spans eight models from 125 million to 175 billion parameters.All models use a 2048-token context window; the range enables testing how validation loss and downstream performance depend on model size.Source: §2.1 · p. 8

2.2 Training Dataset

• The training corpus combines filtered, deduplicated Common Crawl with curated WebText, Books1, Books2, and Wikipedia, sampling higher-quality datasets more frequently than their sizes warrant.This mixture uses roughly 400 billion filtered Common Crawl byte-pair-encoded tokens, while accepting limited overfitting to increase data quality.Source: §2.2 · p. 8, Table 2.2 · p. 9
• A filtering bug left some benchmark development or test examples overlapping pre-training data, so the authors characterize residual contamination and acknowledge retraining was infeasible.They attempted to remove overlaps and planned more aggressive filtering in future work.Source: §2.2 · p. 9

2.3 Training Process

• Training uses gradient-noise-scale-informed batch sizes, smaller learning rates for larger models, and combined intra-layer and inter-layer model parallelism on V100 GPUs.These choices allow larger models to train within memory constraints; detailed settings are provided in Appendix B.Source: §2.3 · p. 9

2.4 Evaluation

• The evaluation samples conditioning examples for each test item, varies K from zero to the context limit, and selects K using development data when available.Multiple-choice tasks use length-normalized completion likelihoods, some tasks additionally use unconditional-probability normalization, and free-form tasks use beam search scored by F1, BLEU, or exact match.Source: §2.4 · p. 10

3 Results

• Cross-entropy validation loss follows a power-law with training compute across two additional orders of magnitude, with only small deviations from the predicted curve.The results indicate that lower loss is associated with consistent gains across a broad range of natural-language tasks.Source: §3 · p. 10, Figure 3.1 · p. 11

3.1.1 Language Modeling

• GPT-3 sets a new PTB language-modeling state of the art by 15 perplexity points, achieving 20.50 in the zero-shot setting.Only zero-shot evaluation is used because PTB lacks a clear separation of examples for one-shot or few-shot evaluation.Source: §3.1.1 · p. 11

3.1.2 LAMBADA

• On LAMBADA, GPT-3 achieves 76% zero-shot and 86.4% few-shot accuracy, improving over prior state of the art by 8% and over 18%, respectively.Few-shot fill-in-the-blank framing lets the model infer that exactly one word should complete the passage; one-shot formatting performs worse than zero-shot.Source: §3.1.2 · p. 12, Figure 3.2 · p. 12

3.1.3 HellaSwag / 3.1.4 StoryCloze

• GPT-3 exceeds a fine-tuned 1.5B model on HellaSwag but remains below ALUM, while StoryCloze reaches 87.7% few-shot accuracy, 4.1% below fine-tuned state of the art.HellaSwag scores are 78.1% one-shot and 79.3% few-shot; StoryCloze improves previous zero-shot results by roughly 10%.Source: §3.1.3 · p. 13, §3.1.4 · p. 13

3.2 Closed Book Question Answering

• In closed-book QA, GPT-3’s one-shot TriviaQA matches open-domain fine-tuning state of the art, while few-shot results approach closed-book state of the art across all three datasets.Performance scales smoothly with model size; few-shot scores reach 71.2% on TriviaQA, 41.5% on WebQuestions, and 29.9% on Natural Questions.Source: §3.2 · pp. 13–14, Figure 3.3 · p. 14

3.3 Translation

• Few-shot translation improves each task by over 7 BLEU and adds another 4 BLEU in the full few-shot setting, reaching similar average performance to prior unsupervised NMT.Performance improves smoothly with model capacity; GPT-3 outperforms prior unsupervised NMT when translating into English but underperforms in the other direction, with En-Ro over 10 BLEU worse.Source: §3.3 · p. 16

3.4 Winograd-Style Tasks

• GPT-3 reaches 77.7% few-shot accuracy on Winogrande, below fine-tuned RoBERTA at 79% and T5 state of the art at 84.6%, while original Winograd remains near state of the art.Original Winograd scores are 88.3% zero-shot, 89.7% one-shot, and 88.6% few-shot; Winogrande improves from 70.2% zero-shot to 77.7% few-shot.Source: Figure 3.5 · p. 16, §3.4 · p. 17

3.5 Common Sense Reasoning

• Commonsense results are mixed: GPT-3 sets SOTA on PIQA, approaches fine-tuned RoBERTa on ARC Challenge, exceeds it on ARC Easy, and remains over 20 points short of overall SOTA on OpenBookQA.PIQA remains over 10% worse than human performance. ARC few-shot accuracy is 51.5% on Challenge and 70.1% on Easy.Source: §3.5 · p. 18

3.7 SuperGLUE

• GPT-3 approaches state-of-the-art on COPA and ReCoRD, while matching fine-tuned BERT-Large on several SuperGLUE tasks and surpassing it on four of eight tasks.It achieves 80.1% on WSC and 75.6% on CB in few-shot evaluation; COPA ranks second behind fine-tuned T5.Source: §3.7 · p. 20
• SuperGLUE performance improves with model size and context examples, with fewer than eight examples per task enough to outperform fine-tuned BERT-Large overall.The experiments scale to 32 examples per task, beyond which examples do not reliably fit in context.Source: §3.7 · p. 20

3.8 NLI

• NLI remains difficult: smaller models stay near chance on RTE and ANLI, while GPT-3 175B only begins to improve, closing almost half the gap from chance to state of the art on ANLI Round 3.On RTE, GPT-3 performs similarly to single-task fine-tuned BERT-Large in the few-shot setting, but only the largest model exceeds random performance convincingly.Source: §3.8 · p. 21, Figure 3.9 · p. 21

3.9 Synthetic and Qualitative Tasks

• The synthetic and qualitative evaluations test few-shot computation, novel symbolic manipulation, analogy solving, new-word use, grammar correction, and news generation without task-specific training.The authors release the synthetic datasets to support further study of test-time language-model behavior.Source: §3.9 · p. 21

3.9.1 Arithmetic

• 100% accuracy on 2-digit addition and 98.9% on 2-digit subtraction show strong few-shot arithmetic, which declines to 9–10% on five-digit operations but extends to multiplication and composite expressions.GPT-3 175B achieves 80.2% on 3-digit addition, 94.2% on 3-digit subtraction, 29.2% on 2-digit multiplication, and 21.3% on single-digit composite operations.Source: §3.9.1 · p. 22, Figure 3.10 · p. 22

3.9.2 Word Scrambling and Manipulation Tasks

• Few-shot GPT-3 learns artificial word transformations as scale increases, reaching 66.9% on random insertion, but none of the models reverses words and zero-shot performance is rarely successful.The sharp one-shot and zero-shot degradation, together with minimal training-set matches, suggests test-time task learning rather than memorization; the tasks require character-level manipulation and search over non-bijective transformations.Source: §3.9.1 · p. 23, §3.9.2 · p. 24

3.9.3 SAT Analogies

• GPT-3 175B scores 65.2% on SAT analogies in few-shot evaluation, exceeding the 57% average among college applicants and improving by over 10% over GPT-3 13B.Performance is 59.1% one-shot and 53.7% zero-shot, compared with 20% for random guessing.Source: §3.9.3 · p. 24, Figure 3.12 · p. 25

3.9.4 News Article Generation

• For ~500-word news articles, mean human detection accuracy was ~52% for GPT-3 175B, barely above chance, versus ~88% for intentionally bad control articles.Factual inaccuracies, repetition, non sequiturs, and unusual phrasings can indicate model-generated text, though these indicators are often subtle.Source: §3.9.4 · p. 26

3.9.5 Learning and Using Novel Words

• GPT-3 generated correct or plausible uses for all six novel-word examples, though one conjugation and usage were slightly awkward.The task used human-generated definitions and one human-generated conditioning answer; subsequent answers were generated continuously by GPT-3 without omissions or repeated prompts.Source: §3.9.5 · p. 29
• The novel-word task defined each nonexistent word and requested a sentence using it, making the task few-shot for the broad task and one-shot for each word.Examples included “whatpu,” “farduddle,” and “yalubalu,” each paired with a definition and usage sentence.Source: §3.9.5 · p. 29

3.9.6 Correcting English Grammar

• GPT-3 was tested on English grammar correction by receiving one human-generated correction and producing five additional corrections from prompts pairing poor and good English.The results are presented in Figure 3.17, but the supplied passage does not report the figure’s outcomes.Source: §3.9.6 · p. 29

Contamination measurement

• Clean benchmarks conservatively removed examples with 13-gram overlaps, or whole-example overlaps for shorter items, then compared clean-subset scores with original scores.A training-data removal bug prevented complete proactive deduplication, so the authors evaluated remaining overlaps instead of retraining GPT-3.Source: §4 · p. 31

Overall contamination results

• Most benchmark scores changed negligibly after conservative cleaning, with no correlation between contamination level and performance difference, although PIQA and Winograd showed evidence warranting marked results.Potential contamination exceeded 50% for a quarter of benchmarks, but the authors conclude it was either overestimated or usually had little effect.Source: §4 · p. 31, Figure 4.2 · p. 32

Detailed investigations

• The analysis flagged six benchmark groups for review: word scrambling, reading comprehension, PIQA, Winograd, language modeling, and German-to-English translation.The overlap procedure was intentionally conservative and could produce false positives, motivating manual inspection of flagged groups.Source: §4 · pp. 31–32
• PIQA performance fell 3 percentage points, or 4% relative, on its clean subset, while Winograd performance fell 2.6%; both results were marked with asterisks.Manual inspection found 132 Winograd schemas in training data, whereas the PIQA shift also appeared in a 25x smaller model and may reflect statistical bias rather than memorization.Source: §4 · p. 32
• Language-modeling benchmarks were withheld because four GPT-2 Wikipedia datasets and Children’s Book Test were almost entirely contained in training, while Penn Tree Bank was unaffected.German translation had 25% potentially contaminated examples with a 1-2 BLEU effect, but inspection found no paired-sentence leakage; reading-comprehension overlaps exposed source text, not answers.Source: §4 · pp. 32–33

Limitations and implications

• The contamination analysis cannot ensure that clean subsets match the original distributions, but many near-zero shifts and similar shifts for small models make offsetting bias unlikely.LAMBADA contained substantial apparent contamination yet its clean subset scored within 0.5% of the full dataset; the authors documented or removed problematic results while noting broader methodological work remains.Source: §4 · p. 33

• GPT-3 still repeats itself, loses coherence over long passages, contradicts itself, and produces occasional non-sequiturs, while struggling with common-sense physics and several benchmark tasks.It performs near chance on some one-shot or few-shot comparison tasks, including WIC and ANLI, and on a subset of reading-comprehension tasks despite strong performance elsewhere.Source: §5 · p. 33
• Using an autoregressive architecture excludes bidirectional objectives and may hurt tasks requiring comparison, re-reading, or long-context understanding, including WIC, ANLI, QuAC, and RACE.The authors suggest that bidirectional models could improve fine-tuning and potentially enable stronger few-shot or zero-shot performance.Source: §5 · p. 33
• Scaling self-supervised prediction may hit fundamental limits because all tokens are weighted equally, task goals are reduced to prediction, and models lack grounding in physical or multimodal experience.Proposed remedies include customizing prediction to entities of interest [RRS20], learning objectives from humans, reinforcement learning, and adding images for grounding [CLY+19].Source: §5 · pp. 33–34
• GPT-3 is more sample-efficient at inference than during pre-training, yet it still consumes far more text than a human lifetime and may not learn few-shot tasks entirely from scratch.Few-shot behavior may range from recognizing training tasks to adapting formats or learning skills de novo, and its mechanism remains an important open question.Source: §5 · p. 34
• Models at GPT-3’s scale are expensive and inconvenient to run, motivating aggressive task-specific distillation despite unresolved challenges at hundreds-of-billions-parameter scale [HVD15] [LHCG19a].Because large models contain many skills unnecessary for a given task, the authors argue that substantial compression may be possible.Source: §5 · p. 34
• GPT-3’s decisions are difficult to interpret, predictions can be poorly calibrated on novel inputs, and training-data biases may produce stereotyped or prejudiced content.The authors identify bias as a particular societal concern and defer its broader discussion to Section 6.Source: §5 · p. 34

6 Broader Impacts

• GPT-3’s improved text quality and adaptability could advance applications ranging from writing assistance and search to misinformation, phishing, fraud, and social engineering.The authors focus on potential harms to stimulate mitigation efforts, while acknowledging that language models also have broad societal benefits.Source: §6 · p. 34

6.1 Misuse of Language Models

• High-quality text generation could lower barriers to misinformation, spam, phishing, fraud, and other socially harmful activities that previously depended on human writing.The authors frame misuse assessment around applications, threat actors, and external incentives.Source: §6.1.1 · p. 35, §6.1 · p. 35

6.1.2 Threat Actor Analysis

• Threat from low- and mid-skill actors was assessed as not immediate but potentially significant if reliability improves, while APTs lacked convincing incentives to adopt current models.Monitoring found discussion but no successful deployments after GPT-2; analysts cited weak advantages and immature content targeting or control.Source: §6.1.2 · p. 35

6.1.3 External Incentive Structures

• Stochastic outputs and required human filtering currently limit scalability, although more consistent and steerable models could reduce deployment costs and increase malicious interest.The authors expect future capability improvements to create challenges requiring mitigation research, prototyping, and coordination.Source: §6.1.3 · p. 35

6.2.4 Future Bias and Fairness Challenges

• The bias analysis is preliminary and subjective, so mitigation requires shared normative, technical, and empirical vocabulary, community engagement, and holistic approaches beyond metric-driven bias removal.The study examines only selected gender, race, and religion dimensions and is intended to motivate continued methodological and intervention research [QMZH19] [HZJ+19] [BBDIW20] [GG19] [NvNvdG19].Source: §6.2 · p. 36, §6.2.4 · p. 39

6.2.1 Gender

• Internet-trained models reflect internet-scale stereotypes: 83% of 388 tested occupations were male-leaning, while GPT-3 175B achieved 64.17% Winogender accuracy.Occupation bias averaged −1.11 for neutral, −2.14 for competent, and −1.15 for incompetent prompts; female descriptions more often emphasized appearance.Source: §6.2.1 · pp. 36–37, §6.2 · p. 36

6.3 Energy Usage

• GPT-3 175B training consumed several thousand petaflop/s-days versus tens for GPT-2 1.5B, but generating 100 pages after training costs about 0.4 kW-hr.Distillation [LHCG19a] and algorithmic progress may improve lifecycle efficiency despite substantial upfront computation.Source: §6.3 · p. 39

Scaling language models

• GPT-3 follows straightforward scaling of parameters and computation, reaching 10x the size of prior models using this strategy.Earlier approaches either increased parameter count without proportional computation through mixture-of-experts models or increased computation without parameters through adaptive computation methods.Source: §7 · p. 39

Scaling laws

• Language-model loss follows a smooth power law with scale, alongside generally smooth downstream-task improvements across three orders of magnitude, though not universally.Prior studies reported power-law loss trends, and the authors observe continued scaling trends with possible slight curvature in the loss curve.Source: §7 · p. 39

Compression and distillation

• Model compression and distillation seek strong performance in smaller language models and could complement GPT-3 by reducing giant models’ latency and memory footprint.Related approaches include ALBERT and general or task-specific language-model distillation.Source: §7 · pp. 39–40

Evaluation and question answering

• Researchers have developed harder question-answering, reading-comprehension, and adversarial datasets as fine-tuned models approach human performance, and GPT-3 evaluates many of them.Question-answering work includes fine-tuned 11-billion-parameter models and attending over large corpora at test time, whereas GPT-3 emphasizes in-context learning that could later be combined with these methods.Source: §7 · p. 40

Metalearning and few-shot learning

• GPT-3’s few-shot method relates to metalearning and prior few-shot or semi-supervised approaches, but adapts through activations across timesteps without updating weights.Its structure most closely resembles RL2, while related work includes few-shot density estimation, low-resource translation, gradient-based adaptation, and semi-supervised fine-tuning.Source: §7 · p. 40

Instructions, multitask learning, and architecture

• Natural-language task instructions and multi-task learning pursue broader transfer, but existing methods require supervised training, curated datasets, or curricula; GPT-3 instead obtains task breadth from large-scale text pre-training.The authors also retain a pure autoregressive architecture for implementation simplicity, while noting that denoising, encoder-decoder, sampling, data, and embedding advances could improve GPT-3, especially for fine-tuning.Source: §7 · p. 40

Conclusion

• The 175-billion-parameter model shows strong zero-shot, one-shot, and few-shot performance across many NLP tasks and benchmarks, sometimes nearly matching state-of-the-art fine-tuned systems.It also generates high-quality samples and performs strongly on tasks defined on the fly.Source: §8 · pp. 40–41
• Performance scales roughly predictably without fine-tuning, although the model retains important limitations and weaknesses.The conclusion also discusses the social impacts of this class of model.Source: §8 · p. 41
• These results suggest that very large language models may be an important ingredient in developing adaptable, general language systems.Source: §8 · p. 41

Model and infrastructure

• Large-scale models, training infrastructure, model parallelism, half-precision optimization, and cluster efficiency were developed to support the research.Contributors implemented the models and infrastructure, developed early model-parallel strategies, optimized clusters, and created fast GPU kernels.Source: Contributions · p. 42

Training and data

• Training data were collected, filtered, deduplicated, and analyzed for overlap, while pre-training experiments and sampling-without-replacement were implemented.The team also experimented with loss scaling and demonstrated the benefit of weight decay for training.Source: Contributions · p. 42

Downstream evaluation

• The team implemented downstream tasks and their software framework, including synthetic tasks, beam search, and task-specific evaluation support.These contributions covered both benchmark implementation and supporting methods for generating and evaluating task outputs.Source: Contributions · p. 42

Scaling and few-shot learning

• Scaling-law analysis guided model and data scaling, while researchers studied in-context learning, prompting, evaluation methods, and the prediction that larger models would continue improving.The work also connected few-shot learning to language-model and meta-learning research and included early demonstrations of few-shot learning.Source: Contributions · p. 42

Ethical and societal analysis

• Researchers assessed fairness, representation, human judgments, malicious-use threats, policy implications, and broader ethical impacts of the model.The ethical-impact analysis included fairness and representation, human assessments, broader impacts, threat analysis, and policy red-teaming.Source: Contributions · p. 42

Research leadership and writing

• Dario Amodei designed and led the research, contributors advised the project from technical and NLP perspectives, and a broader author group wrote the paper.Ilya Sutskever advised several contributors, while Alec Radford advised the project and suggested tasks and context for the results.Source: Contributions · p. 42

Automatic filtering

• A classifier trained on curated corpora versus unfiltered Common Crawl re-sampled documents by prioritizing those predicted to be higher quality.The logistic regression classifier used Spark tokenizer and HashingTF 10 features, with WebText, Wikipedia, and web books as positive examples.Source: A Details of Common Crawl Filtering · p. 43
• α = 9 emphasized highly scored documents while retaining some out-of-distribution documents, matching the classifier-score distribution observed on WebText.This re-weighting increased quality as measured by loss on a range of out-of-distribution generative text samples.Source: A Details of Common Crawl Filtering · p. 43

Fuzzy deduplication

• Fuzzy deduplication removed documents with high overlap within each dataset and also removed WebText from Common Crawl, decreasing dataset size by an average of 10%.The procedure used Spark’s MinHashLSH implementation with 10 hashes and the same features used for classification, partly to prevent overfitting as model capacity increased.Source: A Details of Common Crawl Filtering · p. 43

Benchmark overlap removal

• After quality filtering and deduplication, text occurring in benchmark datasets was partially removed.Source: A Details of Common Crawl Filtering · p. 43

Optimization and regularization

• All GPT-3 versions use Adam, gradient clipping, cosine learning-rate decay with warmup, gradual batch-size growth, sampling without replacement, and weight decay of 0.1.Training spans 260 billion tokens before continuing at 10% of the original learning rate; the learning rate warms up over 375 million tokens, while batch size reaches its full value over 4–12 billion tokens depending on model size.Source: B Details of Model Training · p. 43

Context construction

• Training always uses the full nctx = 2048-token context window, packing shorter documents and delimiting them with end-of-text tokens instead of special masking.This packing strategy increases computational efficiency while allowing the model to infer that contexts separated by the delimiter are unrelated.Source: B Details of Model Training · p. 43

Initial training set filtering

• The filtering procedure removed colliding N-grams and surrounding 200-character windows, discarded short pieces, and removed documents split into more than 10 pieces.It ignored highly frequent matches found in more than 10 training documents to avoid removing common phrases, legal boilerplate, and culturally widespread text.Source: C Details of Test Set Contamination… · pp. 43–44

Overlap methodology

• Exact Apache Spark searches measured dataset overlap using N-gram lengths from the fifth-percentile example length, bounded between 8 and 13 words for non-synthetic tasks.The analysis compared every validation and test set with the full training corpus, despite training on only 40% of filtered Common Crawl documents.Source: C Details of Test Set Contamination… · p. 44

Overlap results

• Examples were labeled dirty if any N-gram collision occurred with a training document and clean otherwise, then evaluated separately to compare clean and original scores.A clean score more than 1–2% below the overall score could indicate overfitting, whereas a higher clean score could indicate easier examples were preferentially marked dirty.Source: C Details of Test Set Contamination… · p. 44

Limitations

• Contamination estimates produced false positives for web-derived context and short examples, while DROP’s 94% dirty rate likely reflected distribution shift rather than cheating.A filtering bug left long documents such as books insufficiently filtered, so several language-modeling benchmarks and Children’s Book Test were excluded because of near-complete overlap.Source: C Details of Test Set Contamination… · p. 44

Overlap results

• GPT-3 showed no apparent performance bias as contamination increased, suggesting relative insensitivity to contamination despite greater variance in clean/all performance.Figure 4.2 indicates neither systematically improved nor degraded performance with increasing contamination.Source: C Details of Test Set Contamination… · p. 44

• Attention is omitted from the compute estimate because it typically accounts for less than 10% of total compute for the analyzed models.Source: D Total Compute Used to Train Language… · p. 46
• For T5, only half of the parameters are active per token because it uses an encoder-decoder architecture.Source: Table D.1 · p. 46
• The method combines one addition and one multiply per active parameter in the forward pass with a 3x multiplier for backward computation, then multiplies by training tokens and total parameters to obtain total flops.Results are reported in both flops and petaflop/s-days, with one petaflop/s-day equal to 8.64e+19 flops.Source: Table D.1 · p. 46, D Total Compute Used to Train Language… · p. 46

Participants

• The study analyzed 621 participants recruited for six experiments after excluding 97 participants who failed an internet-check question.Participants were US-based, recruited through Positly, and could participate in only one experiment.Source: E Human Quality Assessment of… · p. 46, Table E.1 · p. 47

Procedure and design

• The approximately 200-word experiments compared outputs from models ranging from 125M to GPT-3 200B parameters on matched human and model-generated quizzes.Twenty-five newser.com articles were used; each model generated five outputs per prompt, and the completion closest in length to the human article was selected.Source: E Human Quality Assessment of… · pp. 46–47

Duration statistics

• Larger models produced news articles that were harder to distinguish from human writing: accuracy decreased despite participants spending more time on the judgments.Average time spent on question sets increased with model size, while lower accuracy supported the conclusion that larger models generated harder-to-distinguish articles.Source: Figure E.1 · p. 47, E Human Quality Assessment of… · pp. 47–48

Preliminary investigation of ∼500 word articles

• A preliminary study recruited 160 US-based participants to assess approximately 500-word Reuters-style articles generated by GPT-3 175B and a 160M control model.Participants completed two 12-question quizzes per model, with comprehension questions and staged article presentation encouraging closer reading.Source: E Human Quality Assessment of… · p. 48, Table E.2 · p. 48

Task formatting

• The appendix provides formatted examples for the paper’s task datasets, using ground-truth data rather than GPT-3-generated samples.Examples span reading comprehension, reasoning, classification, completion, knowledge, and other task formats.Source: G Details of Task Phrasing and… · p. 50, Figure G.1 · p. 50, Figure G.2 · p. 51

Evaluation conventions

• Several multiple-choice tasks normalize each answer by its unconditional probability, including RACE, OpenBookQA, ARC, and related datasets.The appendix also specifies partial evaluation for Winograd and Winogrande and question-level correctness for MultiRC.Source: Figure G.1 · p. 50, Figure G.3 · p. 51, Figure G.8 · p. 53

Language tasks

• Translation tasks use source-context and target-completion prompts, while zero-shot translation is phrased as a direct question requesting the target-language translation.The figures cover German–English, English–German, English–French, French–English, English–Romanian, and Romanian–English formats.Source: G Details of Task Phrasing and… · p. 60, Figure G.37 · p. 60, Figure G.38 · p. 60

Arithmetic tasks

• Arithmetic task examples specify distinct prompt formats for one- through five-digit operations, including subtraction, addition, and multiplication variants.These formats are illustrated from Arithmetic 1DC through the Arithmetic 5D− and 5D+ tasks.Source: Figure G.42 · p. 61, Figure G.43 · p. 61, Figure G.44 · p. 61

• Table H.1 reports scores for every task, setting, and model investigated in the paper.Source: Table H.1 · p. 63
• Figures H.1–H.4 organize results for SuperGLUE, SAT, Winograd, and arithmetic tasks.Source: Figure H.1 · p. 64, Figure H.2 · p. 64, Figure H.3 · p. 64
• Figures H.5–H.8 organize results for cloze and completion, common-sense reasoning, QA, and reading comprehension tasks.Source: Figure H.5 · p. 65, Figure H.6 · p. 66, Figure H.7 · p. 66
• Figures H.9–H.11 organize results for all ANLI rounds, scramble tasks, and translation tasks.Source: Figure H.9 · p. 66, Figure H.10 · p. 67, Figure H.11 · p. 67