Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling

Introduction

• Pythia spans 70M–12B parameters, trains models on the same data in the same order, and publicly releases data and intermediate checkpoints for study.It includes eight model sizes trained on both the Pile and a deduplicated version.Source: §1 · p. 1
• Counterfactual retraining that feminized most pronouns in the final 7% or 21% of training reduced bias measures on a targeted benchmark.The study modified the grammatical gender of pronouns in the pretraining data to test effects on downstream behavior and bias.Source: §1 · p. 2
• Training-data sequence location does not affect memorization likelihood, and a Poisson point process models the occurrence of memorized sequences over training.The study uses Pythia’s reproducible dataloader setup to examine memorization across training.Source: §1 · p. 2
• After 65,000 steps (45% through training), models with at least 2.8B parameters begin correlating task accuracy with relevant-term frequency, unlike earlier checkpoints and mostly smaller models.The section examines how the influence of pretraining term frequencies changes during training.Source: §1 · p. 2

2.1. Requirements for a Scientific Suite of LLMs

• The suite targets scientific research through public models and data, linked intermediate checkpoints and data ordering, documented training choices, and consistent model designs across multiple scales.The authors prioritize consistency and control over maximizing each model’s performance.Source: §2 · p. 2, §2.1 · p. 3

2.2. Training Data

• Pythia uses the public English-language Pile and its dedicated BPE tokenizer, and trains original and near-deduplicated versions of each model.The near-deduplicated Pile contains approximately 207B tokens versus 300B in the original; the authors cite the lack of a comparably comprehensive multilingual evaluation framework among their reasons for not using multilingual data.Source: §2.2 · p. 3

2.3. Architecture

• The architecture uses dense attention, Flash Attention, rotary embeddings, parallel attention and feedforward layers, and untied embedding and unembedding matrices.The authors selected parallel layers for efficiency and because they expected them not to harm performance, including at smaller scales.Source: §2 · p. 2, §2.3 · p. 4

2.4. Training

• Using batch sizes 4× to 8× standard for sub-billion-parameter models caused no convergence issues and enabled up to 10× wall-clock speedups on smaller models.All models use batches of 1,024 samples at sequence length 2,048; training runs for approximately 300B tokens, with 154 checkpoints saved per model.Source: §2.4 · pp. 4–5

2.5. Evaluation

• On eight common language-modeling benchmarks, Pythia and its deduplicated version performed very similarly to OPT and BLOOM.The evaluation used the Language Model Evaluation Harness, and the authors ran the evaluations themselves.Source: §2.5 · p. 5

2.6. Novel Observations in Evaluation

• Deduplication showed no clear language-modeling benefit, and Pythia matched OPT at equi-token and equi-parameter scales despite using parallel attention and MLP sublayers at every model size.The paper notes this runs counter to claims that this architecture regresses performance below 6B parameters.Source: §2.6 · p. 5
• BLOOM underperformed on LAMBADA, PIQA, and WSC, but not on five other listed benchmarks, suggesting a minimal and inconsistent curse of multilinguality.The authors argue that claims about multilinguality may need reassessment with more diverse evaluation benchmarks.Source: §2.6 · p. 5

2.7. Public Release and Reproducibility

• The authors publicly released models and checkpoints under Apache 2.0, along with evaluation code, raw benchmark scores, and tools to reconstruct the exact training dataloader.Researchers can inspect or save the contents of each batch at each training step.Source: §2.7 · p. 5

3.1. How Does Data Bias Influence Learned Behaviors?

• Pythia’s controlled data order and retrainability let researchers isolate pronoun-frequency effects and motivate further study of training-data interventions and the reliability of bias measures.The authors also suggest using its fine-grained training data to study the influence of specific samples on encoded bias.Source: §3.1 · pp. 6–7
• Gender-swapping interventions reduced measured bias across model sizes; on WinoBias, the 6.9B model shifted from pro-stereotypical to anti-stereotypical bias.CrowS-Pairs showed the strongest reductions for larger models, which were also more biased initially; the interventions caused only a marginal decrease in LAMBADA perplexity, though CrowS-Pairs progression may be noisy.Source: Figure 2 · p. 6, §3.1 · pp. 6–7, Figure 1 · p. 6

3.2. Does Training Order Influence Memorization?

• A Poisson model fit the memorized-sequence data well, indicating training order has little impact and memorized sequences occur at roughly uniform rates across training.The result implies that placing undesirable sequences at the beginning or end will not reduce memorization, though placing them early may reveal undesirable behavior sooner in a partially trained model.Source: §3.2 · pp. 7–8, Figure 3 · p. 8

3.3. Do Pretraining Term Frequencies Influence Task Performance Throughout Training?

• For arithmetic and QA, the correlation between term frequency and average performance depended on model size, emerging in larger models; models below 1B rarely performed well, even with up to 16 few-shot examples.Performance gains as training progressed occurred mainly in larger models, and the multiplication performance gap between the most- and least-frequent operand groups widened during training.Source: §3.3 · pp. 8–9
• Tracking term frequencies across checkpoints and model sizes can help estimate whether a model of a given size is likely to retain and recall information from its training data.The controlled suite removes confounds such as differences in architecture, pretraining datasets, and training hyperparameters.Source: §3.3 · p. 9

4. Conclusion

• Pythia provides a model suite spanning multiple orders of magnitude, trained with consistent data ordering and architecture.The authors present it as a framework for detailed experiments on LLMs.Source: §4 · p. 9
• The case studies report novel analyses of gender debiasing, memorization, and term frequency effects, and the authors recommend the suite for further LLM experiments.They hope follow-up work will examine how pretraining data drives capability acquisition and emergence on more complex tasks.Source: §4 · p. 9

A. Author Contributions

• Stella Biderman, Hailey Schoelkopf, and Quentin Anthony led project design, model training and implementation, experiments, and paper writing.Biderman conceived and organized the project and designed the memorization and pretraining-frequency studies; Schoelkopf trained the models and planned the gender-bias study; Anthony optimized implementation and advised on hyperparameters.Source: A. Author Contributions · p. 17
• Herbie Bradley, Kyle O’Brien, and Eric Hallahan contributed gender-bias analysis, zero- and five-shot benchmark evaluations, and paper plots.Bradley carried out the WinoBias analysis; Hallahan evaluated standard NLP benchmarks and authored most plots.Source: A. Author Contributions · p. 17
• Mohammad Aflah Khan, Shivanshu Purohit, and USVSN Sai Prashanth contributed evaluation implementation, model optimization, and memorization and benchmark analyses.Khan helped implement CrowS-Pairs and analyze results; Purohit optimized the model implementation and advised on hyperparameters; Sai Prashanth conducted the memorization study and evaluated standard NLP benchmarks.Source: A. Author Contributions · p. 17
• Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal contributed project advice, documentation, pretraining-frequency experiments, and gender-bias study work.Raff advised on the project; Skowron wrote model-suite and analysis documentation; Sutawika conducted experiments and wrote the pretraining-frequencies section; van der Wal helped with CrowS-Pairs and gender-bias write-up.Source: A. Author Contributions · p. 17

B. Corrections and Updates

• All model sizes now use a uniform batch size of 2M tokens; 160M, 410M, and 1.4B models previously used 4M.The authors found during initial training that a uniform batch size was feasible across model sizes.Source: B. Corrections and Updates · p. 17
• The suite adds checkpoints at initialization and steps 1, 2, 4, 8, 16, 32, 64, 128, 256, and 512, as well as every 1000 steps.These checkpoints support study of early training dynamics and emergent behaviors, and provide access to random initializations.Source: B. Corrections and Updates · p. 17
• Flash Attention’s fused kernel greatly increased per-device throughput during the second set of training runs (Dao et al., 2022).Source: B. Corrections and Updates · p. 17
• All models now decay learning rates to a minimum of 0.1× their maximum, correcting the prior difference between models up to 2.8B and the 6.9B and 12B models.Previously, the smaller models decayed to 10% of their starting rate, while the two largest decayed to zero.Source: B. Corrections and Updates · p. 18
• The preliminary v0 suite was overwritten on March 31, 2023, but remains available for replicability; the authors adopted semantic versioning for future fixes.Source: B. Corrections and Updates · p. 18
• On January 20, 2023, Pythia’s naming was updated to count embedding and unembedding parameters, with future work advised to use counts that include embeddings.The authors said this convention better reflects on-device memory requirements and reduces documentation debt across model releases.Source: B. Corrections and Updates · p. 18

C.1. Gender Bias Interventions

• For WinoBias, the study prompts models on pro-stereotypical examples with both pronouns and measures stereotype accuracy from the higher log probability.A score of 1 indicates perfectly predicting stereotypes, while 0.5 represents random accuracy or no bias; this differs from the original benchmark setup.Source: §C.1 · p. 18
• For CrowS-Pairs (N´ev´eol et al., 2022), the study measures how often models assign lower perplexity to stereotyped than less-stereotyped sentences, using only English gender examples.The evaluation adapts a benchmark originally designed for masked language models (Nangia et al., 2020); the English gender subset matches the monolingual models and pronoun interventions.Source: §C.1 · pp. 18–19
• Swapping gendered pronouns reduces bias without major LAMBADA accuracy dips, preserving the models’ text-understanding capabilities.Figure 6 compares Pythia models and their intervened counterparts over training.Source: §C.1 · p. 19, Figure 6 · p. 19

C.2. Pretraining Term Frequency

• Table 4 reports arithmetic multiplication performance gaps across model sizes, shot counts, and checkpoints.The caption describes the comparison scope but provides no values or direction of effects.Source: Table 4 · p. 19
• Figure 7 plots 16-shot arithmetic addition accuracy by binned term frequency across model sizes and intermediate checkpoints.Each point represents average accuracy for a frequency bin; checkpoints are distinguished by color and step number.Source: Figure 7 · p. 20

D. Training Hardware and GPU hours

• The table reports GPUs used and GPU-hours by model size, with GPU-hours calculated from iteration time, iteration count, and GPU count; all GPUs were 40GB A100s.The authors also describe reporting the accelerator counts and total GPU-hours at the training throughputs they achieved.Source: Table 5 · p. 21, D. Training Hardware and GPU hours · p. 21
• 544,280 A100-hours were required to train the model suites for this paper.The authors trained two models of each size—one on the Pile and one on the deduplicated Pile—and retrained both suites once more.Source: D. Training Hardware and GPU hours · p. 21
• “Total” in the table means training one model of each size in the Pythia suite.The paper trained two models of each size and retrained both suites an additional time.Source: D. Training Hardware and GPU hours · p. 21, Table 5 · p. 21

E. Full Configuration Details

• Pythia’s exact model configuration files are publicly available alongside the full training configuration details in Table 6.Some configuration values differ by model; Table 1 gives model dimensions, and the repository contains per-model GPU counts, microbatch sizes, and gradient-accumulation steps.Source: E. Full Configuration Details · p. 22, Table 6 · p. 22
• Most models used model-parallel-size = 1, compared with 2 for 6.9b models and 4 for 12b models.The 6.9b and 12b models used zero-optimization bucket sizes of 1260000000; other models used 500000000.Source: E. Full Configuration Details · p. 22

F.1. Assessment of Existing Suites

• The assessment of existing suites asked whether any met researchers’ requirements and expectations for rigorous scientific study of language models.The supplied passage states the assessment’s aim but does not enumerate those requirements.Source: §F.1 · p. 23
• GPT-Neo models were commonly used as a consistent suite despite differences in architecture implementation, training codebase, tokenizer, and training data setup and order [Black et al., 2022].The passage notes that, strictly speaking, these models do not form a suite.Source: §F.1 · p. 23
• OPT’s training data was not public, while fewer than 10 OPT checkpoints were available, only for the 2.7b, 6.7b, and 13b models [Zhang et al., 2022].T5’s original paper did not release its training data, although it released code later run by Dodge et al. (2021).Source: §F.1 · p. 23
• BLOOM’s training data was available to researchers by application, and its models had small, undocumented differences in training composition and order after divergence handling [Scao et al., 2022; Jernite et al., 2022; McMillan-Major et al., 2022].GPT-3’s API was publicly accessible but paid and substantially limited the research users could conduct; the API models were not the same models as those in Brown et al. (2020), with sizes estimated by Gao (2021).Source: §F.1 · p. 23

F.2. Contrast with Multiply Trained Models

• The authors chose one copy of each model rather than multiple random-seed runs because rigorous 25-seed training would cost approximately 10 million USD in compute [Sellam et al., 2021].They describe seed variation as a dual question about how stable analysis of a particular model is when the random seed varies, and judged one model per size to have greater impact given available resources.Source: §F.2 · p. 23

G. Evaluations

• Final trained models are evaluated on standard NLP tasks, with raw scores reported in Appendices G.1 and G.2.The appendices also report scores for model suites before deduplicated Pythia models start their second training-data epoch.Source: G. Evaluations · p. 24
• Appendix G.3 compares the Pythia models with baseline model suites on standard NLP tasks.Source: G. Evaluations · p. 24
• Appendix G.4 plots benchmark performance over training; broader evaluation data and additional task results are available in the Pythia repository.Source: G. Evaluations · p. 24

• Tables 7 and 8 report fully trained zero-shot results for the Pythia and Pythia (Deduplicated) suites, respectively.Source: Table 7 · p. 24, Table 8 · p. 25
• Tables 9 and 10 report zero-shot results for the Pythia and Pythia (Deduplicated) suites, respectively, after 93k pretraining steps.Source: Table 9 · p. 25, Table 10 · p. 26
• The 93k-step results are from the closest measured step before Pythia (Deduplicated) enters a second epoch at the 207B-token mark.Source: Table 9 · p. 25, Table 10 · p. 26

G.2. Raw Benchmark Scores - Five Shot

• Tables 11 and 12 report five-shot results for the fully trained Pythia and Pythia (Deduplicated) suites on selected NLP benchmarks.Source: Table 11 · p. 26, Table 12 · p. 27
• Tables 13 and 14 report five-shot results for both suites after 93k pretraining steps, before the deduplicated suite’s second epoch at 207B tokens.Source: Table 13 · p. 27, Table 14 · p. 28

G.3. Comparison to Baseline Models

• Tables 15 and 16 provide zero-shot benchmark results for BLOOM and OPT, respectively, for comparison with Pythia.Source: Table 15 · p. 28, Table 16 · p. 29

G.4. Graphs

• Figure 8 plots zero-shot evaluations of final Pythia checkpoints against OPT and BLOOM.Source: Figure 8 · p. 30
• Figure 9 plots zero-shot evaluations for deduplicated models’ last checkpoints before their second epoch.Source: Figure 9 · p. 31
• Figures 10–12 track LAMBADA (OpenAI), Winogrand Schema Challenge, and Winogrande during training on standard and deduplicated Pile data.Each graph marks when the deduplicated Pile began its second epoch.Source: Figure 10 · p. 32, Figure 11 · p. 32, Figure 12 · p. 33
• Figures 13–15 show AI2 Reasoning Challenge—Easy Set, SciQ, and LogiQA over training; captions identify standard Pile on the left and deduplicated Pile on the right.Figures 14 and 15 mark where the deduplicated Pile began its second epoch with a dashed line.Source: Figure 13 · p. 33, Figure 14 · p. 34, Figure 15 · p. 34