PaperlayerView exact revision on arXiv

Source-linked AI summary

GPT-NeoX-20B: An Open-Source Autoregressive Language Model

Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, Michael Pieler, USVSN Sai Prashanth, Shivanshu Purohit, Laria Reynolds, Jonathan Tow, Ben Wang, Samuel Weinbach

arXiv:2204.06745v1cs.CL

TL;DR

Although many large language models existed, most were protected or otherwise unavailable for outsider use. The paper presents GPT-NeoX-20B and finds that it gains more from few-shot examples than similarly sized GPT-3 and FairSeq models.

  • Problem

    Most large language models were protected or otherwise unavailable for outsider use, while few larger dense autoregressive models had freely public weights.

    Source: §1 · p. 1

  • Method

    The paper presents a 20-billion-parameter autoregressive transformer, describes its architecture and training, and evaluates it on language, mathematics, and knowledge-based tasks.

    Source: §2 · p. 2, §1 · p. 2, §5.1 · p. 7

  • Results

    GPT-NeoX-20B receives a much larger few-shot performance boost than comparably sized GPT-3 and FairSeq models.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The freely available training checkpoints are intended to facilitate research on LLM training dynamics, AI safety, and interpretability.

    Source: §1 · p. 2

    The model was not evaluated on coding tasks, despite design choices intended to improve coding performance.

    Source: §5.3 · p. 10

Abstract

from arXiv · show

We introduce GPT-NeoX-20B, a 20 billion parameter autoregressive language model trained on the Pile, whose weights will be made freely and openly available to the public through a permissive license. It is, to the best of our knowledge, the largest dense autoregressive model that has publicly available weights at the time of submission. In this work, we describe \model{}'s architecture and training and evaluate its performance on a range of language-understanding, mathematics, and knowledge-based tasks. We find that GPT-NeoX-20B is a particularly powerful few-shot reasoner and gains far more in performance when evaluated five-shot than similarly sized GPT-3 and FairSeq models. We open-source the training and evaluation code, as well as the model weights, at https://github.com/EleutherAI/gpt-neox.

1 Introduction

GPT-NeoX-20B is presented as a large, openly available autoregressive language model, with the authors reporting notable few-shot learning gains and no observed performance loss from repeated training data. Its release is intended to support research on LLM capabilities, training dynamics, safety, and interpretability.

1 Introduction

  • The paper introduces GPT-NeoX-20B, a 20-billion-parameter autoregressive model whose weights are freely available to the public under a permissive license.The authors motivate open access as critical to research, particularly in AI safety, mechanistic interpretability, and studying how LLM capabilities scale.Source: §1 · p. 1
  • The authors release partially trained checkpoints at evenly spaced 1,000-step intervals to facilitate research on LLM training dynamics, safety, and interpretability.Source: §1 · p. 2
  • GPT-NeoX-20B shows no performance loss from duplicate training data beyond one epoch and gains more from few-shot examples than comparable-sized GPT-3 and FairSeq models.The authors report that the finding that few-shot prompting does not improve performance on one task is specific to GPT-3; GPT-J-6B shows similar gains, which they hypothesize may reflect shared training data.Source: §1 · p. 2
  • The paper outlines GPT-NeoX-20B’s architecture, training, hardware and software setup, dataset and tokenization choices, training challenges, broader impacts, costs, and carbon emissions.Source: §1 · p. 2

2 Model Design and Implementation

GPT-NeoX-20B is a 20-billion-parameter autoregressive transformer decoder with 44 layers, a 6144-dimensional hidden state, and 64 heads. Its design follows GPT-3 closely, with changes to positional embeddings, layer computation, initialization, and layer density.

2 Model Design and Implementation

  • GPT-NeoX-20B has 20 billion parameters, including 19.9 billion non-embedding parameters, 44 layers, a hidden dimension of 6144, and 64 attention heads.The authors identify the non-embedding parameter count as the appropriate number for scaling-law analysis, following Kaplan et al. (2020).Source: §2 · p. 2

2.1 Model Architecture

  • GPT-NeoX-20B largely follows GPT-3, while its architecture is almost identical to GPT-J.The authors use GPT-3 as their reference because they say GPT-J has no canonical published design reference.Source: §2.1 · p. 2

2.1.1 Rotary Positional Embeddings

  • Rotary embeddings replace learned positional embeddings, and applying them to only the first 25% of dimensions balances performance and computational efficiency in initial experiments.Rotary embeddings make attention between tokens depend linearly on their relative positions.Source: §2.1.1 · pp. 2–3

2.1.2 Parallel Attention + FF Layers

  • Attention and feed-forward layers run in parallel and their outputs are summed, allowing local reduction before a single all-reduce; Mesh Transformer JAX reported a 15% throughput increase with comparable early-training loss curves.The efficiency rationale concerns the all-reduce operations otherwise required for residual additions in op-sharded training.Source: §2.1.2 · p. 3
  • An implementation oversight left the two Layer Norms untied, and small-scale experiments found no performance difference; the issue was discovered too late to restart training.The implemented form uses LN1 for attention and LN2 for the feed-forward layer.Source: §2.1.2 · p. 3

2.1.3 Initialization

  • Feed-forward output layers use an initialization scheme intended to prevent activations growing with depth and width, with a factor of 2 to account for parallel feed-forward layers.The other layers use the small initialization scheme cited in the passage.Source: §2.1.3 · p. 3

2.1.4 All Dense Layers

  • Unlike GPT-3’s alternating dense and sparse layers, GPT-NeoX-20B uses only dense layers to reduce implementation complexity.The cited sparse-layer technique is Child et al. (2019).Source: §2.1.4 · p. 3

2.2 Software Libraries

  • Training builds on Megatron and DeepSpeed, using PyTorch v1.10.0 compiled with CUDA 11.1 and bundled NCCL 2.10.3 for distributed communication.Source: §2.2 · p. 3

3 Training

GPT-NeoX-20B was trained on the diverse Pile using selected hyperparameters and distributed training, with a Pile-based tokenizer designed to handle whitespace consistently. The authors used the Pile without deduplication and observed no drop in test validation loss after crossing one epoch.

3 Training

  • Training used a 0.97E−5 learning rate, 0.01 weight decay, approximately 3.15M tokens per batch, and 150,000 steps with cosine decay to 10% of the initial rate.The setup used AdamW with beta values 0.9 and 0.95 and epsilon 1.0E−8, extended with ZeRO; tensor parallelism of 2 and pipeline parallelism of 4 achieved 117 teraFLOPS per GPU.Source: §3 · pp. 3–4

3.1 Training Data

  • The Pile combines 22 sources across five broad categories and over 825 GiB of raw text, with some components up-sampled for balance.Its diversity supports a general-purpose model, but makes training-data differences a major unknown when comparing GPT-NeoX-20B with GPT-3; the Pile also includes StackExchange Q/A data, whose formulation appears to influence code generation (Biderman and Raff, 2022).Source: §3.1 · pp. 4–5

3.2 Tokenization

  • The tokenizer uses a 50,257-token BPE vocabulary trained on the Pile, consistent space delimitation, and repeated-space tokens up to 24 spaces.Repeated-space tokens reduce token counts for whitespace-heavy inputs such as source code and arXiv LaTeX files.Source: §3.2 · p. 5

3.3 Data Duplication

  • Beyond one epoch, the authors observed no drop in test validation loss and used the Pile as-is, while noting production deduplication can reduce training-data leakage.They cite the lack of large-scale ablations and noticeable smaller-scale impact as reasons for this choice; prior deduplication results were difficult to replicate because trained models were not released (Lee et al., 2021).Source: §3.3 · p. 5

4 Performance Evaluations

The evaluation compares GPT-NeoX-20B with publicly accessible autoregressive language models and tests it across natural-language, mathematical, and advanced knowledge-based tasks.

4 Performance Evaluations

  • The comparison targets publicly accessible English language models with at least 10B parameters; named comparators include GPT-3 API models, FairSeq dense models, and GPT-J-6B.T5 and its derivatives were excluded because the evaluation methodology assumes autoregressive models. Mixture-of-experts models were excluded because no public MoE model achieves performance comparable to a 10B-parameter dense model; the released Megatron-11B code was non-functional.Source: §4 · p. 6
  • Plots report two-standard-error bars for 95% confidence intervals, alongside human-level and random baselines where available.Some confidence intervals are too small to be visible.Source: §4 · p. 6

4.1 Tasks Evaluated

  • Evaluation covers natural-language, advanced knowledge-based, and mathematical tasks; GPT-J-6B, GPT-NeoX-20B, and FairSeq models are tested zero- and five-shot, while GPT-3 models are zero-shot only.Only a representative subset of results appears in the paper, with the rest in Appendix D.Source: §4.1 · p. 6
  • Natural-language evaluation uses a diverse collection of standard language-model datasets, including ANLI, ARC, HellaSwag, TriviaQA, and Winogrande.The collection also includes HeadQA, LAMBDADA, LogiQA, OpenBookQA, PiQA, PROST, QA4MRE, SciQ, and WSC.Source: §4.1 · p. 7
  • Mathematical evaluation uses the MATH test dataset and Brown et al. (2020) numerical arithmetic problems, with zero- and five-shot testing because of computational limits.MATH is generally used for fine-tuning, but the models are evaluated without fine-tuning here.Source: §4.1 · p. 7
  • Advanced knowledge-based evaluation uses multiple-choice questions across Humanities, Social Sciences, STEM, and Miscellaneous, reporting five-shot performance for comparison with prior work.The five-shot GPT-3 values are taken from Hendrycks et al. (2021a).Source: §4.1 · p. 7

5 Discussion

The discussion reports broad but uneven task performance, especially strong gains from few-shot evaluation, alongside limitations in tuning and evaluation. It presents public release as the paper’s central contribution while recognizing that wider access also requires computing infrastructure.

5.1 Performance Results

  • 22 of 32 evaluations favored GPT-NeoX-20B over FairSeq 13B, four underperformed, and six fell within the margin of error; HellaSwag was the major weakness.On HellaSwag, GPT-NeoX-20B scored four standard deviations below FairSeq 13B in both zero- and five-shot evaluations; the authors say they cannot explain the loss.Source: §5.1 · p. 7
  • GPT-J and GPT-NeoX consistently outperform GPT-3 and FairSeq models on arithmetic tasks, but the authors caution that this does not establish better out-of-distribution reasoning.They conjecture that equations in the Pile contribute; Razeghi et al. (2022) report a correlation between equation frequency and GPT-J performance, and the authors could not investigate the effect in FairSeq or GPT-3 because their training data are unreleased.Source: §5.1 · p. 7
  • On MMMLU, GPT-NeoX and FairSeq models dominate GPT-3 in five-shot evaluation, while their zero-shot performance is much closer.The authors find that GPT-NeoX and FairSeq improve substantially with as few as five examples, and recommend using multiple model classes when developing benchmarks.Source: §5.1 · p. 7, Figure 7 · p. 10

5.2 Powerful Few-Shot Learning

  • 0.0598 and 0.0526 were the 0-to-5-shot improvements for GPT-NeoX-20B and GPT-J-6B, versus 0.0183 and 0.0051 for FairSeq 13B and 6.7B.The improvement was statistically significant and robust to prompting perturbations; the authors lacked systematic five-shot GPT-3 evaluations but cite further support from Tables 10–13 and Figure 7.Source: §5.2 · pp. 7, 10

5.3 Limitations

  • Hyperparameters may be suboptimal because tuning relied on smaller-scale experiments and Brown et al. (2020) settings despite differences in architecture, data, and tokenizer.The authors could not evaluate coding performance because of benchmark difficulty and cost, and note that absent deduplication may have affected downstream performance despite no sign of it in their experiments.Source: §5.3 · p. 10

5.4 Releasing a 20B Parameter LLM

  • To the authors’ knowledge, GPT-NeoX-20B was the largest and most performant dense language model to be publicly released, and they describe its release as the paper’s most important contribution.They argue that public access can benefit researchers who otherwise lack access to LLMs and believe those benefits outweigh the risks.Source: §5.4 · p. 11
  • Wider access requires computing infrastructure as well as model access: inference is most economical on two RTX 3090 Tis or one A6000, and fine-tuning requires significantly more compute.The authors plan to reduce inference costs and work with researchers to provide infrastructure access.Source: §5.4 · p. 11

6 Summary · A Individual Contributions

GPT-NeoX-20B is a 20-billion-parameter autoregressive Transformer trained on the Pile, with architectural differences from GPT-3 and strong performance on knowledge-based and mathematical tasks. The paper also describes open-sourced code and weights and credits the project leads.

6 Summary

  • GPT-NeoX-20B is a 20-billion-parameter autoregressive Transformer language model trained on the Pile.Source: §6 · p. 11
  • Compared with GPT-3, it changes the tokenizer, adds Rotary Positional Embeddings, parallelizes attention and feed-forward computation, and uses different initialization and hyperparameters.Source: §6 · p. 11
  • Evaluations on natural-language and factual-knowledge tasks found that GPT-NeoX-20B performs particularly well on knowledge-based and mathematical tasks versus other publicly available models.Source: §6 · p. 11
  • The project open-sourced its training and evaluation code at the GPT-NeoX repository.Source: §6 · p. 11
  • The repository provides a link to download model weights across the whole training run.Source: §6 · p. 11

A Individual Contributions

Implementation and Engineering … C.3 Differential Impact on Access

The paper presents GPT-NeoX-20B’s public release as its central contribution and argues that broader access can support ethics and alignment research. It weighs that benefit against capability-related risks, emphasizing that the case depends on this release’s circumstances.

Implementation and Engineering

C Broader Impacts

  • GPT-NeoX-20B is described as the largest publicly released dense language model, and its public release is presented as the paper’s most important contribution despite concerns about potential harm.The paper notes a several-way tie for second place at 13 billion parameters (Artetxe et al., 2021; Xue et al., 2020, 2022).Source: C Broader Impacts · p. 20
  • The authors argue that release can enable ethics and alignment research, while acknowledging capability-acceleration risks and asserting that companies and governments may be better positioned than individuals to cause large-scale harm.They cite prior work enabled by earlier public releases as evidence of benefits, while maintaining that the benefits outweigh the risks.Source: C Broader Impacts · p. 20

C.2 Impact on Ethics and Alignment Research

  • The authors argue that greater access can empower both AI ethics and AI alignment researchers, whose work focuses respectively on current technologies and future systems with much greater capabilities.They note that the two research communities use different tools, methods, and ideas.Source: §C.2 · p. 20

C.2.1 The Necessity of Model Access for AI Ethics

  • Centralized LLM training and restrictive access can prevent independent evaluation, while GPT-3 is costly to use and its training data is private.The passage says that only GPT-3 was publicly accessible among models larger than GPT-NeoX-20B at the time of writing.Source: §C.2.1 · pp. 20–21

C.2.2 The Usefulness of Large Language Models in Alignment

  • Although GPT-NeoX-20B is not itself a system to align, the authors propose it as a public platform for experiments in interpretability, reward modeling, transparent prompting, simulated agents, and tool-assisted alignment research.These are presented as potential directions for investigation, not established solutions.Source: §C.2.2 · pp. 21–22

C.3 Differential Impact on Access

  • Large-model engineering and capital requirements put powerful models out of reach for small labs and independent researchers, while ethics and alignment researchers are fewer at large organizations.The passage cites examples of large organizations’ access to the latest generation of models (Brown et al., 2020; Rae et al., 2022; Fedus et al., 2021; Lieber et al., 2021; Tang, 2021).Source: §C.3 · p. 22
  • The authors judge GPT-NeoX-20B’s capability-research contribution likely limited because leading organizations already have more powerful models and scaling knowledge matters more than fungible implementation.They identify this differential impact—primarily benefiting people with less funding and infrastructure—as key to the release decision, while urging other actors to assess releases case by case.Source: §C.3 · p. 22

C.4 Environmental Impact

The authors tracked GPT-NeoX-20B’s energy use and emissions, finding that development and training emitted about 10% of Strubell et al. (2019)’s estimate. They report broader energy accounting while cautioning that the comparison is not direct and advocating more systematic reporting.

C.4 Environmental Impact

  • 31.73 metric tons of CO2 from GPT-NeoX-20B development and training was almost exactly 10% of Strubell et al. (2019)’s 284.01-metric-ton estimate.The authors equate this amount to the yearly emissions of an average American or 35 round-trip flights between New York City and San Francisco.Source: §C.4 · p. 22
  • The Illinois electricity mix was 30.40% coal and 31.30% gas, with an average emissions factor of 0.47905 tCO2/MWh.The listed emissions factors were 0.95 tCO2/MWh for coal and 0.6078 tCO2/MWh for gas.Source: §C.4 · pp. 22–23
  • The remaining listed electricity sources were hydroelectric 1.30%, nuclear 17.40%, solar 0.30%, wind 18.10%, and other renewables 1.30%, each listed at 0 tCO2/MWh.Source: §C.4 · pp. 22–23
  • 66.24 MWh total consumption over training, scaling, testing, and evaluation produced just under 35 metric tons of CO2.Training used 43.92 MWh over 1,830 hours; scaling, testing, and evaluation added 920 system-hours.Source: §C.4 · p. 23
  • Strubell et al. (2019)’s estimate came from a neural architecture search paper and is not directly comparable; the authors advocate systematic, comprehensive emissions reporting.Source: §C.4 · p. 23

D Full Evaluation Results

The section reports zero- and five-shot evaluations across natural-language understanding, arithmetic and MATH, and Hendrycks tasks, with model coverage varying by table.

D Full Evaluation Results

E Tokenizer Analysis · E.1 Tokenizer Comparisons · E.1.1 Longest Tokens

On the Pile validation set, GPT-NeoX-20B’s tokenizer uses approximately 10% fewer tokens than GPT-2’s, while its longest tokens suggest different training-data patterns in the two vocabularies.

E Tokenizer Analysis

  • The GPT-NeoX-20B and GPT-2 tokenizers share 36,938 of 50,257 tokens, approximately 73.5% overlap.Source: E Tokenizer Analysis · p. 36
  • GPT-NeoX-20B’s tokenizer represents the Pile validation set with approximately 10% fewer tokens than GPT-2’s.Source: E Tokenizer Analysis · p. 36
  • EuroParl, GitHub, and PubMed Central show the largest token-count improvements, each with more than 20% savings; arXiv, GitHub, and StackExchange also benefit notably.Source: E Tokenizer Analysis · p. 36
  • Excluding whitespace, GPT-NeoX-20B still uses 5% fewer tokens on the Pile validation set, while GitHub and StackExchange token ratios move closer to even.Source: E Tokenizer Analysis · p. 36
  • On the AllenAI replication of C4, GPT-NeoX-20B’s tokenizer uses approximately the same number of tokens as GPT-2’s.The comparison uses the C4 validation set as an out-of-domain corpus [15].Source: Table 14 · p. 36, E Tokenizer Analysis · p. 36
  • Excluding whitespace, GPT-NeoX-20B requires approximately 1% more tokens than GPT-2 to represent the C4 validation set [15].Source: E Tokenizer Analysis · p. 36

E.1.1 Longest Tokens

  • Among the 10 longest non-symbol, non-whitespace tokens, GPT-2’s often appear to reflect overrepresented websites or web-scrapes, whereas GPT-NeoX-20B’s are mostly scientific terms.The scientific terms likely arise from the Pile’s PubMed components.Source: §E.1.1 · p. 36

E.1.2 Worst Case Word Tokenization Comparison

This subsection examines words with the largest differences in token counts between the two tokenizers, analyzing them within Pile components. It also reports that GPT-NeoX-20B uses fewer tokens overall, with its largest gains in whitespace-heavy datasets.

E.1.2 Worst Case Word Tokenization Comparison

  • The analysis targets words with the greatest token-length discrepancy, where one tokenizer uses many tokens and the other relatively few.A word is defined as a contiguous string delimited by whitespace or punctuation, and only words occurring at least 10 times in a component are considered.Source: §E.1.2 · pp. 36–37
  • GPT-NeoX-20B uses fewer tokens to represent the Pile overall, with the biggest gains in whitespace-heavy datasets such as arXiv, GitHub, and StackExchange.A separate table reports token counts excluding whitespace tokens, but no cell values are provided here.Source: Table 15 · p. 37, Table 16 · p. 37
  • The analysis is performed at the component level and shows a representative example from the Pile-CC corpus.Source: §E.1.2 · p. 37

F Tokenization Examples

This section compares GPT-2 and GPT-NeoX-20B tokenizations of Pile documents and presents examples of worst-case word tokenization.

F Tokenization Examples

Related papers

Loading 2204.06745v1…