Source-linked AI summary
GPT-NeoX-20B: An Open-Source Autoregressive Language Model
Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, Michael Pieler, USVSN Sai Prashanth, Shivanshu Purohit, Laria Reynolds, Jonathan Tow, Ben Wang, Samuel Weinbach
TL;DR
The paper addresses the limited public availability of large, capable dense autoregressive language models. It introduces and evaluates GPT-NeoX-20B, a 20-billion-parameter model trained on the Pile, and releases its weights and code openly. GPT-NeoX-20B is reported as a particularly powerful few-shot learner, with substantially larger gains from five-shot evaluation than comparable GPT-3 and FairSeq models.
Problem
Large dense autoregressive language models are generally proprietary, gated, or unavailable for external research, limiting access to models at this scale.
Method
The paper develops GPT-NeoX-20B, describes its architecture and training, and evaluates it across language-understanding, mathematics, and knowledge-based tasks.
Results
GPT-NeoX-20B is reported to be a particularly powerful few-shot learner, gaining substantially more from five-shot evaluation than similarly sized GPT-3 and FairSeq models.
Takeaways & Limitations
The open release of GPT-NeoX-20B, its weights, code, and training checkpoints is intended to facilitate research on language-model training dynamics, safety, and interpretability.
Takeaways & Limitations
The model was not evaluated on coding benchmarks because those evaluations were underestimated in difficulty and cost.
Abstract
from arXiv · showhide
We introduce GPT-NeoX-20B, a 20 billion parameter autoregressive language model trained on the Pile, whose weights will be made freely and openly available to the public through a permissive license. It is, to the best of our knowledge, the largest dense autoregressive model that has publicly available weights at the time of submission. In this work, we describe \model{}'s architecture and training and evaluate its performance on a range of language-understanding, mathematics, and knowledge-based tasks. We find that GPT-NeoX-20B is a particularly powerful few-shot reasoner and gains far more in performance when evaluated five-shot than similarly sized GPT-3 and FairSeq models. We open-source the training and evaluation code, as well as the model weights, at https://github.com/EleutherAI/gpt-neox.
1 Introduction
GPT-NeoX-20B addresses the limited public availability of large dense autoregressive language models by releasing a 20-billion-parameter model and its checkpoints openly. The study reports strong few-shot learning and examines architecture, training, evaluation, and release implications.
- Large language models have scaled predictably, but most of the largest models remain proprietary, gated, or unavailable for external use.
- GPT-NeoX-20B is presented as an openly released large dense autoregressive language model with freely available training artifacts.The authors specifically plan to make checkpoints throughout training available.
- GPT-NeoX-20B shows no performance loss from duplicated training data across more than one epoch.
- Few-shot prompting improves GPT-NeoX-20B and FairSeq models, contrary to a finding reported for GPT-3.GPT-NeoX-20B receives a larger few-shot performance boost than similarly sized GPT-3 and FairSeq models.
- The paper documents GPT-NeoX-20B’s architecture, training, evaluation, dataset, tokenization, training difficulties, release impacts, cost, and carbon emissions.
2 Model Design and Implementation
GPT-NeoX-20B is a 20-billion-parameter dense autoregressive transformer broadly based on GPT-3 but modified for positional encoding, computation, initialization, and implementation efficiency. Several design choices prioritize computational efficiency, while an untied-layer-normalization mistake was retained transparently after small-scale tests found no performance difference.
- Model architecture: 20 billion parameters, 44 layers, hidden dimension 6144, and 64 attention heads define GPT-NeoX-20B’s core architecture.
- Rotary positional embeddings: Rotary embeddings replace learned positional embeddings and make attention depend linearly on the relative position difference m − n.
- Rotary positional embeddings: Rotary embeddings are applied only to the first 25% of embedding dimensions to balance performance and computational efficiency.
- Parallel attention and FF layers: Attention and feed-forward layers are computed in parallel and summed, reducing distributed communication to one all-reduce instead of two per residual addition.A related implementation reported a 15% throughput increase with comparable early-training loss curves.
- Layer normalization: An implementation oversight used independent Layer Norms instead of tied normalization, but small-scale experiments indicated no performance difference.
- Implementation: The model uses dense layers exclusively and a Megatron- and DeepSpeed-based codebase for large-scale training.
3 Training
GPT-NeoX-20B was trained on the diverse Pile dataset using distributed optimization and parallelism, with tokenizer changes intended for general-purpose text. The authors trained beyond one epoch because validation loss continued falling and observed no test-validation-loss drop after crossing that boundary.
- Training setup: Hyperparameters were selected by interpolating published GPT-3 values and using smaller-scale experiments because full-scale sweeps were infeasible.The selected learning rate was 0.97E−5 and weight decay was 0.01.
- Distributed training: Optimizer states and model computation are distributed across GPUs using ZeRO, tensor parallelism, and pipeline parallelism.
- Training data: The Pile contains over 825 GiB of raw text from diverse sources, up-sampled in places to create a more balanced distribution.
- Tokenizer: GPT-NeoX-20B uses a new Pile-trained BPE tokenizer with consistent space delimitation and the same 50,257-token vocabulary size as GPT-2.The design particularly improves whitespace handling for source code.
- Training duration: Validation loss continued falling into the beginning of the second epoch, motivating training beyond one epoch.
- Data duplication: No drop in test validation loss appeared after crossing the one-epoch boundary, although the authors note independent reasons to deduplicate production training data.
4 Performance Evaluations
The evaluation compares GPT-NeoX-20B with publicly accessible autoregressive models across natural-language, knowledge-based, and mathematical tasks using zero- and five-shot settings where feasible. The design includes open evaluation tooling, multiple baselines, confidence intervals, and representative results with additional findings deferred to the appendix.
- Evaluation design: GPT-NeoX-20B is evaluated against GPT-3, FairSeq dense models, and GPT-J-6B using the open-source EleutherAI evaluation harness.
- Reporting: Where available, plots report human and random baselines together with error bars representing two standard errors and 95% confidence intervals.
- Task coverage: The evaluation covers natural-language, advanced knowledge-based, and mathematical task categories.
- Evaluation settings: GPT-J-6B, GPT-NeoX-20B, and FairSeq models are evaluated zero-shot and five-shot, while GPT-3 models are evaluated only zero-shot because of financial constraints.
- Reporting: The paper presents a representative subset of results in the main text, with remaining evaluations reported in Appendix D.
5 Discussion
GPT-NeoX-20B performs strongly across several evaluation settings, especially when given few-shot examples, while important limitations remain in training optimization, coding evaluation, and data deduplication. Its release is framed as a contribution to broader access for research.
- Performance Results: Across 32 natural-language evaluations, GPT-NeoX-20B outperformed FairSeq 13B on 22 tasks, underperformed on four, and fell within the margin of error on six.Its weakest result was HellaSwag, where it scored four standard deviations below FairSeq 13B in both zero- and five-shot evaluations.
- Performance Results: GPT-J and GPT-NeoX consistently outperformed GPT-3 and FairSeq models on arithmetic tasks, although the authors caution that this may reflect training-data frequency rather than general out-of-distribution reasoning.The authors could not investigate this effect for FairSeq and GPT-3 because their training data is unavailable.
- Performance Results: GPT-NeoX and FairSeq models showed dominant five-shot MMMLU performance over GPT-3, while their zero-shot performance was much closer.The authors argue that benchmark conclusions should not be based on a single model class.
- Powerful Few-Shot Learning: GPT-NeoX-20B improved by 0.0598 from zero-shot to five-shot evaluation, compared with improvements of 0.0051 and 0.0183 for FairSeq 6.7B and 13B.The result was statistically significant and robust to prompting perturbations.
- Limitations: Full-scale hyperparameter tuning was infeasible, so the authors interpolated settings from prior work despite substantial differences in architecture, data, and tokenizer.They conclude that the chosen hyperparameters were almost certainly no longer optimal and potentially never were.
- Limitations: The paper did not evaluate coding performance, and the lack of dataset deduplication may have affected downstream performance despite no observed training signal.The authors underestimated the difficulty and cost of coding benchmarks and note conflicting evidence about deduplication.
- Releasing a 20B Parameter LLM: The public release is presented as the paper’s most important contribution, intended to expand access for researchers while acknowledging that inference and fine-tuning require substantial computing infrastructure.The authors plan to reduce inference costs and help researchers access needed infrastructure.
6 Summary
GPT-NeoX-20B is a 20 billion parameter autoregressive Transformer trained on the Pile and evaluated against other publicly available models. The paper details its architecture, evaluations, and open-source release.
- GPT-NeoX-20B is a 20 billion parameter autoregressive Transformer language model trained on the Pile.
- Its main differences from GPT-3 include a new tokenizer, Rotary Positional Embeddings, parallel attention and feed-forward computation, and different initialization and hyperparameters.
- The authors evaluate GPT-NeoX-20B extensively on natural-language and factual-knowledge tasks against other publicly available models.
- The project includes contributions spanning development leadership, training infrastructure, scaling experiments, positional embeddings, tokenization, evaluations, alignment implications, and environmental impact.
Scientific Experimentation
The paper frames GPT-NeoX-20B’s public release as a response to centralized access to large language models, enabling independent ethics, alignment, and interpretability research. It argues that release risks are acceptable in this specific context while acknowledging important limitations.
- The full training configuration is provided in a YAML file usable with gpt-neox, with documentation for each parameter.
- The release argument is conditional: the authors believe benefits outweigh risks in these particular circumstances, while urging future actors to consider what, how, and when they release.
- Centralized LLM training and restricted access make meaningful independent evaluation difficult, especially when larger models are not freely usable and their training data is private.
- GPT-NeoX-20B is presented as a publicly available platform for experiments whose results might generalize to future alignment work.
- The authors identify mechanistic interpretability and LLM-based reward modeling as promising directions for further investigation.
- The paper notes that value learning has fundamental theoretical limitations, although it may still be robust enough to align weaker superhuman AIs.
C.4 Environmental Impact
The authors report energy use and carbon emissions for GPT-NeoX-20B’s development and training, contextualizing the result against a widely cited estimate while noting comparability limits.
- 31.73 metric tons of CO2 were emitted during GPT-NeoX-20B’s development and training, roughly 10% of Strubell et al.’s estimate.
- The reported electricity mix included coal, gas, hydroelectric, nuclear, solar, wind, and other renewables.
- Environmental impact figures are presented in metric tons to align with standard reporting.
- 66.24 MWh of electricity were consumed across training, scaling, testing, and evaluation.
- The authors caution that the comparison estimate comes from a neural architecture search paper and is not directly comparable to GPT-NeoX-20B.
D Full Evaluation Results
The evaluation appendix organizes GPT-NeoX-20B comparisons across natural-language understanding, arithmetic, MATH, and Hendrycks knowledge-based tasks. Results are reported under zero-shot and five-shot settings across model families and scales.
- Natural-language understanding results are reported in Tables 2–5 for GPT-J, GPT-NeoX, GPT-3, and FairSeq models.
- All evaluations used Evaluation Harness version 0 to support reproducibility as task implementations change.
- The appendix distinguishes zero-shot and five-shot evaluations, with GPT-3 omitted from several five-shot tables because of financial limitations.
- Basic arithmetic and MATH results are reported in Tables 6–9 for GPT-J, GPT-NeoX, GPT-3, and FairSeq models.
- Hendrycks-task results are divided across Tables 10–13, covering GPT-J, GPT-NeoX, GPT-3, and FairSeq models.
E Tokenizer Analysis
The GPT-NeoX-20B tokenizer generally represents the Pile with fewer or comparable tokens than GPT-2, with especially large savings in several whitespace-heavy components. Its vocabulary also differs qualitatively, containing many scientific terms among its longest tokens.
- The tokenizers share 36938 of 50257 tokens, approximately 73.5% overlap.
- Approximately 10% fewer tokens represent the Pile validation set with GPT-NeoX-20B than with GPT-2.
- More than 20% token savings occur in EuroParl, GitHub, and PubMed Central.
- The GPT-NeoX-20B tokenizer uses fewer or closely comparable tokens across all Pile components.
- Excluding whitespace, GPT-NeoX-20B still uses 5% fewer tokens than GPT-2.
- GPT-NeoX-20B’s longest tokens are mostly scientific terms, whereas GPT-2’s often reflect artifacts of tokenizer training data.
E.1.2 Worst Case Word Tokenization Comparison
The worst-case comparison examines words whose tokenization lengths differ substantially between GPT-NeoX-20B and GPT-2, using frequent component-level words and representative Pile-CC examples.
- The analysis identifies words for which one tokenizer requires many more tokens than the other.
- Words are defined as contiguous strings delimited by whitespace or punctuation and must occur at least 10 times within a component.
- Table 18 presents representative worst-case word-tokenization examples from the Pile-CC corpus.
F Tokenization Examples
The paper illustrates tokenizer behavior with document examples from multiple Pile components, contrasting GPT-2 and GPT-NeoX-20B tokenizations. It also reports vocabulary and worst-case word-tokenization examples.
- Tokenized document examples compare GPT-2 with GPT-NeoX-20B across Pile components.
- Table 17 lists the ten longest tokens in each tokenizer vocabulary, excluding tokens mainly comprising symbols, numbers, and spaces.
- The examples include arXiv, BookCorpus2, DM Mathematics, GitHub, and OpenWebText2 documents.
- The tokenization examples also include PubMed Abstracts documents and worst-case tokenization comparisons.