Source-linked AI summary
Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling
Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, Oskar van der Wal
TL;DR
Despite major successes, little is known about how large language models develop and why they succeed. Pythia addresses this gap with a controlled suite of models trained on the same data in the same order, enabling detailed analyses of training dynamics, including memorization, term-frequency effects, and gender debiasing.
Problem
Despite large language models’ successes across applications, very little is known about how and why they succeed.
Method
Pythia introduces decoder-only autoregressive language models ranging from 70M to 12B parameters, with all models trained on the same data in the same order.
Results
Pythia supports detailed analyses of gender debiasing, memorization, and term-frequency effects, yielding novel results across these areas.
Takeaways & Limitations
The authors recommend Pythia as a framework for novel experimental setups and follow-up research on how pretraining data drives capability acquisition and emergence.
Takeaways & Limitations
The suite lacks a multilingual evaluation framework nearly as comprehensive as Gao et al. (2021).
Abstract
from arXiv · showhide
How do large language models (LLMs) develop and evolve over the course of training? How do these patterns change as models scale? To answer these questions, we introduce \textit{Pythia}, a suite of 16 LLMs all trained on public data seen in the exact same order and ranging in size from 70M to 12B parameters. We provide public access to 154 checkpoints for each one of the 16 models, alongside tools to download and reconstruct their exact training dataloaders for further study. We intend \textit{Pythia} to facilitate research in many areas, and we present several case studies including novel results in memorization, term frequency effects on few-shot performance, and reducing gender bias. We demonstrate that this highly controlled setup can be used to yield novel insights toward LLMs and their training dynamics. Trained models, analysis code, training code, and training data can be found at \url{https://github.com/EleutherAI/pythia}.
1. Introduction
Pythia addresses limited understanding of LLM training and scaling by providing a controlled, public suite spanning multiple model sizes and shared data order. Its case studies examine how training data and scale relate to memorization, term-frequency effects, and gender bias.
- Very little is known about how and why transformers succeed, especially across training dynamics and model scaling.
- Pythia is a suite of decoder-only autoregressive language models ranging from 70M to 12B parameters for scientific research.
- The models span several orders of magnitude, use the same data in the same order, and provide public data and intermediate checkpoints.
- Pythia trains eight model sizes on both the Pile and its deduplicated version, providing two comparable copies of the suite.
- The suite enables studies of how precise training data and model scale affect gender bias, memorization, and few-shot learning.
- A memorization study finds no effect of sequence location and models memorized-sequence occurrences with a Poisson point process.
- After 65,000 steps, models with at least 2.8B parameters correlate task accuracy with task-relevant term occurrence, unlike earlier checkpoints and mostly smaller models.
2. The Pythia Suite
Pythia is designed as a controlled, publicly accessible suite for studying language-model training and scaling. Its consistent architectures, data provenance, checkpoints, and reproducibility tools support comparisons across model sizes and training conditions.
- Design goals: Pythia prioritizes consistency and control across model design and training rather than maximizing performance for each model.The authors state that this design choice supports scientific research on large language models.
- Suite requirements: The suite compares models across multiple orders of magnitude while providing publicly released models, public training data, intermediate checkpoints, and documented training procedures.These criteria include public access, training provenance, and consistency across scale.
- Training data: Pythia trains two identical-architecture suites: one on the Pile and one on a near-deduplicated version of the Pile.The deduplicated corpus uses MinHashLSH with a threshold of 0.87.
- Training efficiency: Large batch sizes produce wall-clock speed-ups of up to 10× compared with standard batch sizes on smaller models.The authors attribute these speed-ups to GPT-NeoX scaling linearly with the number of GPUs.
- Training checkpoints: 154 log-spaced checkpoints are saved for every model, enabling analyses of training dynamics from early training through later stages.The checkpoints include iterations 1 through 512 on a logarithmic schedule.
- Evaluation findings: Pythia performs on par with equi-parameter OPT models across scales, while deduplication shows no clear language-modeling benefit and multilingual effects are minimal and inconsistent in the cited comparison.The multilingual comparison reports underperformance for BLOOM on some benchmarks but not others.
- Public release: The public release includes models, checkpoints, evaluation code, benchmark scores, training code, and tools for reconstructing the exact training dataloader.Researchers can inspect or save the contents of each batch at each training step.
3. Case Studies
Pythia enables controlled case studies of how training data, training order, and model scale shape gender bias, memorization, and few-shot task performance. Across these studies, interventions and reproducible training records reveal dynamics that prior model suites could not isolate.
- Overview: Pythia enables three case studies that use public training-data order and controlled comparisons to study language-model dynamics across scale.The studies address gender bias, memorization, and term-frequency effects on few-shot performance.
- How Does Data Bias Influence Learned Behaviors?: WinoBias stereotypical accuracy decreases after intervention across model scales, and the 6.9B model changes from pro-stereotypical to anti-stereotypical bias during intervention training.The authors hypothesize that larger models learn more complex occupation–pronoun relationships.
- How Does Data Bias Influence Learned Behaviors?: Swapping gendered pronouns during the final 7% or 21% of training reduces CrowS-Pairs bias across model sizes, most prominently in larger models.The intervention produces only a marginal decrease in LAMBADA perplexity, while the reliability of CrowS-Pairs progression remains open.
- Does Training Order Influence Memorization?: A Poisson model fits memorized-sequence occurrences extremely well, indicating that training order has little impact on memorization.Memorized sequences are not denser at the beginning or end of training, and roughly equal numbers occur between checkpoints.
- Do Pretraining Term Frequencies Influence Task Performance Throughout Training?: Term-frequency effects on arithmetic and question-answering performance emerge more clearly in larger models, while models below 1 billion parameters rarely perform accurately despite up to 16 few-shot examples.For multiplication, the performance gap between the 10% most frequent and least frequent operands widens during training.
- Do Pretraining Term Frequencies Influence Task Performance Throughout Training?: Pythia’s controlled access to training data and checkpoints removes confounding differences in architecture, datasets, and hyperparameters when studying when term frequencies affect task performance.This setup supports finer-grained analysis across model size and intermediate training stages.
4. Conclusion
Pythia provides a controlled, publicly released suite for studying how model scale and training data relate to language-model behavior. The paper demonstrates this framework through analyses of gender debiasing, memorization, and term-frequency effects.
- Pythia enables detailed experiments across multiple orders of model scale using consistent data ordering and model architecture.
- The paper presents novel analyses of gender debiasing, memorization, and pretraining term-frequency effects.
- The authors position the suite and its dataset tooling as resources for future research on how pretraining data relates to capability acquisition and emergence.
B. Corrections and Updates
Pythia v1 was retrained with more consistent hyperparameters and denser early checkpoints to improve control, transparency, and usefulness for studying training dynamics. The authors reran analyses and report that the changes did not significantly affect the paper’s findings.
- Pythia v1 uses a uniform 2M-token batch size across all model sizes.
- Additional checkpoints at initialization and early training steps enable analysis of early training dynamics and emergent behaviors.
- All models now use learning-rate schedules decaying to 0.1× their maximum learning rate, correcting an inconsistency in the original suite.
- The authors reran all analyses and evaluations on the updated suite and found no significant impact on experimental findings.
- The preliminary v0 suite remains available for replicability, while future fixes will use semantic versioning.
C.1. Gender Bias Interventions
The gender-bias case study adapts WinoBias and CrowS-Pairs for autoregressive language models and evaluates interventions involving gendered pronouns. The reported interventions reduce bias without major losses in LAMBADA accuracy.
- The gender-bias benchmarks require modified evaluation setups because they were not originally designed for autoregressive language models or text generation.
- WinoBias is reformulated as a prompted multiple-choice task using log probabilities to measure stereotype accuracy.A score of 1 represents perfect stereotype prediction, while 0.5 represents random accuracy or no bias.
- CrowS-Pairs is evaluated by comparing model perplexities for stereotyped and less-stereotyping sentence versions.
- Intervening by swapping gendered pronouns reduces bias while causing no major dips in LAMBADA accuracy.
C.2. Pretraining Term Frequency
The arithmetic multiplication table and arithmetic addition figure examine how few-shot accuracy varies with model size, training checkpoint, and term frequency. The figure plots binned term frequency against average accuracy across intermediate checkpoints.
- Arithmetic multiplication performance is compared across model sizes, shot counts, and training checkpoints.
- Arithmetic addition accuracy is plotted against binned term frequency for multiple model sizes and intermediate checkpoints.Colors distinguish checkpoints and their training steps; accuracy is shown on the y-axis and term frequency on the x-axis.
D. Training Hardware and GPU hours
The paper reports the hardware and compute used to train each Pythia model, including accelerator counts, GPU-hours, and an aggregate total across training runs.
- The paper also reports accelerator counts and total GPU-hours at the achieved training throughputs.
- Table 5 lists each Pythia model size alongside its GPU count and total A100 GPU-hours.GPU-hours are calculated from iteration time, iteration count, and GPU count.
- 544,280 A100-hours were required for all model training conducted in this paper.This includes two model suites and an additional retraining of both suites.
E. Full Configuration Details
The paper provides complete Pythia training configurations and makes exact configuration files available, while explaining how model-specific settings and parallelism were handled.
- Full Configuration Details: Table 6 contains the full configuration details for training the Pythia models, with exact files released in the GitHub repository.
- Full Configuration Details: The listed configuration includes zero dropout, checkpoint activations, FP16 training, a global batch size of 1024, and gradient clipping of 1.0.
- Full Configuration Details: Most models use model-parallel-size = 1, while the 6.9b and 12b models use sizes 2 and 4, respectively.These larger models also use specified ZeRO bucket-size settings for parallel training.
- Existing Model Suites: The authors assessed existing suites against requirements for rigorous language-model research before developing Pythia.
- Existing Model Suites: Existing suites differed in public access, training-data availability, training-order consistency, or architectural and implementation consistency.The discussion covers GPT-3, GPT-Neo, OPT, T5, and BLOOM, alongside suites focused on random-seed stability.
G. Evaluations
The paper evaluates Pythia throughout training and reports final-model scores, baseline comparisons, and evaluations before deduplicated models begin a second data epoch.
- Evaluations: Evaluation scores and plots track selected benchmarks over the course of training.
- Evaluations: The appendices report raw final scores and comparisons with baseline model suites across standard NLP tasks.
- Evaluations: The paper also reports scores for both model suites before the deduplicated Pythia models start their second training-data epoch.The corresponding transition occurs at 207B tokens.
G.1. Raw Benchmark Scores - Zero Shot
The paper presents zero-shot results on selected NLP benchmarks for fully trained and partially trained Pythia suites, including deduplicated and non-deduplicated models.
- Raw Benchmark Scores - Zero Shot: The evaluation materials include measurements before Pythia (Deduplicated) enters a second epoch at the 207B-token mark.
G.2. Raw Benchmark Scores - Five Shot
This section presents five-shot benchmark results for the fully trained Pythia and Pythia (Deduplicated) suites, alongside corresponding results after 93k pretraining steps. It also includes zero-shot benchmark evaluations for Pythia, BLOOM, and OPT, plus training-course figures across several tasks.
- Tables 11 and 12 report five-shot results on selected NLP benchmarks for the fully trained Pythia and Pythia (Deduplicated) suites.
- Tables 13 and 14 report five-shot benchmark results after 93k pretraining steps for the Pythia and Pythia (Deduplicated) suites.
- Tables 15 and 16 provide zero-shot results for BLOOM and OPT as comparison points for Pythia’s performance.
- Figures 8 and 9 show zero-shot evaluations of final Pythia checkpoints and the last checkpoints before deduplicated models entered their second epoch.The second epoch began at the 207B-token mark for the deduplicated setup.
- Figures 10–15 track LAMBADA, Winogrand Schema Challenge, Winogrande, AI2 Reasoning Challenge—Easy Set, SciQ, and LogiQA over training on standard and deduplicated Pile data.The figures mark where the deduplicated Pile began its second epoch.