PaperlayerView exact revision on arXiv

Source-linked AI summary

OPT: Open Pre-trained Transformer Language Models

Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, Luke Zettlemoyer

arXiv:2205.01068v4cs.CLcs.LG

TL;DR

Researchers lack full access to large language models, limiting their study. OPT presents openly shared decoder-only transformers from 125M to 175B parameters, with performance parity on standard GPT-3 evaluation datasets and one-seventh GPT-3’s carbon footprint for OPT-175B development.

  • Problem

    Full access to large language models is limited to a few highly resourced labs, restricting researchers’ ability to study how and why these models work.

  • Method

    OPT develops eight decoder-only transformer language models spanning 125M to 175B parameters, broadly following GPT-3 architecture and training practices.

  • Results

    OPT showed parity on standard GPT-3 evaluation datasets, while developing OPT-175B required only 1/7th GPT-3’s carbon footprint.

  • Takeaways & Limitations

    Broad access to OPT is intended to enable reproducible and responsible research on large language models.

  • Takeaways & Limitations

    The authors consider the technology premature for commercial deployment and call for greater scrutiny of training data and more consistent evaluation setups.

Abstract

from arXiv · show

Large language models, which are often trained for hundreds of thousands of compute days, have shown remarkable capabilities for zero- and few-shot learning. Given their computational cost, these models are difficult to replicate without significant capital. For the few that are available through APIs, no access is granted to the full model weights, making them difficult to study. We present Open Pre-trained Transformers (OPT), a suite of decoder-only pre-trained transformers ranging from 125M to 175B parameters, which we aim to fully and responsibly share with interested researchers. We show that OPT-175B is comparable to GPT-3, while requiring only 1/7th the carbon footprint to develop. We are also releasing our logbook detailing the infrastructure challenges we faced, along with code for experimenting with all of the released models.

1 Introduction

Large language models demonstrate strong generation and zero- and few-shot learning, but restricted access to full model weights limits research on robustness, bias, and toxicity. OPT addresses this gap with openly shared decoder-only models, research access, training documentation, and code intended to support reproducible and responsible research.

  • Motivation: LLMs show emergent text-generation and zero- and few-shot learning capabilities, while full model access remains limited to a few highly resourced labs.Paid APIs provide public interaction in some cases, but not full model access.
  • Motivation: Restricted access limits researchers’ ability to study and improve challenges including robustness, bias, and toxicity.
  • Contribution: 125M to 175B parameters: OPT is a suite of decoder-only pre-trained transformers designed for full and responsible sharing with interested researchers.The models are trained to roughly match the performance and sizes of GPT-3-class models while applying current data-collection and efficient-training practices.
  • Contribution: 125M to 66B parameters: OPT models are released, while full research access to OPT-175B is provided upon request to eligible researchers and organizations.Eligibility includes academic, government, civil-society, and industry research affiliations.
  • Contribution: 992 80GB A100 GPUs and 147 TFLOP/s utilization per GPU: the released logbook and metaseq codebase document and enable OPT-175B training.The broader release aims to give researchers access needed for reproducibility and collective work on responsible AI and ethical considerations.

2 Method

OPT trains eight decoder-only Transformer models spanning 125 million to 175 billion parameters, largely following GPT-3 settings while using a curated 180B-token corpus. OPT-175B training used large-scale sharding and required checkpoint restarts and optimization changes to manage hardware failures and instabilities.

  • Model architecture: Eight models span 125 million to 175 billion parameters and largely follow Brown et al. (2020), with batch-size variations for computational efficiency.The models use ReLU activation, sequence length 2048, and architectural details reported in Table 1.
  • Optimization: The models use AdamW with β1=0.9, β2=0.95, weight decay 0.1, dropout 0.1, gradient clipping, and a linear warmup-and-decay learning-rate schedule.Warmup lasts 2000 steps for OPT-175B or 375M tokens for smaller baselines; learning rates decay to 10% over 300B tokens.
  • Pre-training data: The corpus combines RoBERTa, the Pile, and PushShift.io Reddit, is deduplicated at Jaccard similarity ≥.95, uses GPT-2 byte-level BPE, and contains roughly 180B tokens.The predominantly English corpus retains a small amount of non-English CommonCrawl data.
  • Training infrastructure: OPT-175B was trained on 992 80GB A100 GPUs using Fully Sharded Data Parallel with Megatron-LM Tensor Parallelism, reaching up to 147 TFLOP/s per GPU.Adam state was kept in FP32, model weights in FP16, and dynamic loss scaling was used to avoid underflows.
  • Training challenges: At least 35 manual restarts and over 100 host cycles occurred over 2 months because of hardware failures during OPT-175B training.Flagged nodes were cordoned off, diagnostics were run, and training resumed from the last saved checkpoint.
  • Training challenges: Loss divergences were mitigated by lowering the learning rate, restarting from earlier checkpoints, and reducing gradient clipping from 1.0 to 0.3.Recovery checkpoints were selected when the dynamic loss scalar remained ≥1.0 and final-layer activation norms trended downward.

3 Evaluations

OPT is evaluated across standard NLP benchmarks, where its average zero-, one-, and few-shot performance generally tracks GPT-3 but varies substantially by task. On open-source dialogue tasks, OPT-175B outperforms an unsupervised Reddit 2.7B model and competes with supervised BlenderBot 1, while raising possible ConvAI2 data-leakage concerns.

  • NLP evaluations: OPT is evaluated on 16 standard NLP tasks using GPT-3 prompts and experimental settings, with accuracy as the primary metric.MultiRC and ReCoRD F1 are omitted for consistency, and WSC is formulated as multiple choice.
  • Zero-shot: Zero-shot average performance across 14 tasks follows GPT-3’s trend, but performance varies radically across tasks.MultiRC and WIC are excluded because they appear to favor GPT-3 or OPT disproportionately.
  • Zero-shot: OPT roughly matches GPT-3 on 10 tasks, underperforms on ARC Challenge and MultiRC, and shows scale-inconsistent behavior on CB, BoolQ, and WSC.The validation sets for CB, BoolQ, and WSC contain 56, 277, and 104 examples, respectively; OPT always outperforms GPT-3 on WIC.
  • One-shot and Few-shot: One- and few-shot average performance is similar between OPT and GPT-3, but OPT consistently underperforms GPT-3 on MultiRC.Task-level results remain inconsistent with model size for BoolQ, CB, WSC, and RTE, and the authors hypothesize setup differences from GPT-3’s evaluation.
  • Dialogue evaluations: OPT-175B significantly outperforms the unsupervised Reddit 2.7B model on all dialogue tasks and performs competitively with fully supervised BlenderBot 1.On Wizard-of-Internet, OPT-175B has the lowest perplexity but lower UF1 than models supervised on Wizard-of-Wikipedia.
  • Dialogue evaluations: OPT-175B’s competitive unsupervised ConvAI2 results may indicate dataset leakage, although searches found no overlap with the first ConvAI2 conversation.The authors additionally evaluated OPT-175B on ConvAI2’s hidden test set to address leakage concerns.

4 Bias & Toxicity Evaluations

OPT-175B shows strong hate-speech detection but exhibits greater stereotypical bias and toxic-generation propensity than comparison models in several evaluations. Its dialogue safety is roughly comparable to Reddit 2.7B, while curated-data fine-tuning is identified as a route to improving safety.

  • Hate speech detection: OPT175B considerably outperforms Davinci in hate speech detection across one-shot through few-shot configurations.The evaluation asks whether statements are racist or sexist, using binary and multiclass responses.
  • Stereotypical bias: OPT175B exhibits more stereotypical bias than Davinci in almost all CrowS-Pairs categories except religion.The authors associate this difference with the higher incidence of stereotypes and discriminatory text in the Pushshift.io Reddit corpus, a primary OPT-175B data source.
  • Stereotypical bias: Davinci and OPT-175B have similar aggregate StereoSet scores, with OPT-175B better on stereotype score and Davinci generally better on language-modeling score.Davinci performs better in profession and race, while OPT-175B performs better in gender and religion.
  • Toxic content generation: OPT-175B has a higher toxicity rate than either PaLM or Davinci, and all three models become more likely to generate toxic continuations as prompt toxicity increases.The experiment samples 25 generations of 20 tokens with nucleus sampling at p = 0.9 for each of 10,000 randomly sampled prompts.
  • Dialogue safety: OPT-175B performs similarly to Reddit 2.7B across SaferDialogues and Unit Tests, performing marginally better in Safe and Adversarial settings but worse in Unsafe.The authors conclude that explicit fine-tuning on curated dialogue datasets should be used to improve OPT-175B’s safety profile.

5 Limitations

OPT models show largely comparable performance on standard GPT-3 evaluation datasets and safety, bias, and inclusion evaluations, but OPT-175B retains important limitations shared by other large language models. These include instruction-following failures, repetition, factual errors, toxic language, harmful stereotypes, and insufficient readiness for commercial deployment.

  • Evaluation scope: Evaluations found performance parity on standard GPT-3 datasets and largely comparable safety, bias, and inclusion results, with variations in toxicity and hate speech detection.The authors caution that these evaluations may not fully characterize the models’ complete limitations.
  • Behavioral limitations: OPT-175B often simulates a dialogue instead of executing declarative instructions or point-blank interrogatives, and it can become repetitive or stuck in loops.Instruction learning may alleviate these limitations, while sampling can reduce the incidence of repetition.
  • Factuality: OPT-175B can produce factually incorrect statements, posing particular risks in applications where information accuracy is critical, including healthcare and scientific discovery.The passage notes that retrieval-augmented models may improve factual correctness.
  • Safety and bias: OPT-175B has a high propensity to generate toxic language and reinforce harmful stereotypes, even from relatively innocuous prompts, while adversarial prompts are trivial to find.The authors note substantial ongoing work on mitigating toxicity and bias.
  • Deployment readiness: The authors consider the technology premature for commercial deployment and call for greater scrutiny of training data through additional characterization and selection criteria.They argue that current practice relies on feeding models as much data as possible with minimal dataset selection.

6 Considerations for Release

OPT’s release emphasizes transparency, researcher access, and responsible study of large language models. The authors disclose training and implementation details, provide OPT-175B weights and smaller baselines, and restrict access to research use while documenting substantial but comparatively lower reported carbon emissions.

  • Transparency and accountability: OPT discloses OPT-175B’s training details through its logbook and code, provides model weights and smaller baselines, and emphasizes accountability across development.The release follows Partnership on AI recommendations and NIST governance guidance.
  • Transparency and accountability: Detailed day-to-day records expose compute usage and human overhead when infrastructure or training becomes unstable at scale.The authors note that such details are generally omitted from prior publications because mid-training changes are difficult to ablate without greatly increasing compute.
  • Research access: Researchers receive direct access to OPT-175B parameters to enable responsible AI research and reduce the environmental impact of reproducing models at this scale.The access rationale is explicitly twofold: support responsible research and address environmental costs.
  • Research access: Non-commercial research access prioritizes quantifying LLM limitations and risks before broader commercial deployment.The restriction responds to documented ethical and social risks from language models with emergent capabilities at scale.
  • Environmental considerations: 75 tons was OPT-175B’s estimated CO2eq footprint, compared with 500 tons for GPT-3 and 380 tons for Gopher.The authors caution that estimates and accounting methods are not standardized, and that experimentation and downstream inference also contribute to total AI-system energy use.
  • Baselines and scaling: Baselines across many scales enable research on scaling effects and model limitations, while additional training data may further improve their performance.The authors connect this possibility to evidence that some large language models may have been under-trained relative to their training-data amounts.

7 Related Work

Related work traces modern NLP’s shift toward self-supervised, increasingly large language models and identifies prompting as the field’s primary evaluation paradigm. It also highlights instruction-oriented fine-tuning and continuing challenges in prompt engineering.

  • Scaling and Pre-training: Self-supervised language modeling has reshaped NLP, with T5 and Megatron-LM showing improvements through scaling parameters and training data.The passage attributes scaling gains to both model size and the amount and quality of training data.
  • Autoregressive Language Models: Autoregressive models grew from 117M to over 500B parameters, with GPT-2 establishing major gains in generative fluency and GPT-3 and later models improving them further.The passage frames autoregressive language models as experiencing the largest growth in model size.
  • Prompting: Since GPT-3, prompt-based evaluation has become the primary criterion for LLMs because it conveniently covers many tasks without specialized fine-tuning.The passage connects modern prompting to earlier cloze evaluations and notes its use for evaluating many tasks.
  • Prompting: Instruction-style fine-tuning improves direct responses to prompts, but effective prompt engineering remains an open challenge with results that vary significantly.The cited passage lists several instruction-tuning efforts and states that prompt-engineering outcomes vary.

8 Conclusion

The report introduces OPT, a 125M-to-175B-parameter autoregressive language-model collection intended to replicate GPT-3-class model performance and sizes while applying improved data curation and training efficiency. It details training, evaluates NLP and dialogue performance, and characterizes bias, toxicity, and hate-speech behaviors.

  • 8 Conclusion: 125M to 175B parameters: OPT is a collection of autoregressive language models introduced in the report.The stated collection spans models from 125M to 175B parameters.
  • 8 Conclusion: OPT aims to replicate the performance and sizes of GPT-3-class models while applying best practices in data curation and training efficiency.
  • 8 Conclusion: The report describes training details, evaluates performance in NLP and dialogue settings, and characterizes bias, toxicity, and hate-speech behaviors.

A Additional Evaluations … Evaluations

Across zero-shot and multishot evaluations, GPT-3 and OPT perform similarly on most NLP tasks, although some tasks behave erratically. The paper credits contributors for planning, infrastructure, training, data work, evaluations, writing, code release, and responsible AI conduct.

  • A Additional Evaluations: Across all 16 zero-shot NLP tasks, GPT-3 and OPT models perform similarly on most tasks, with some showing highly erratic behavior.
  • A Additional Evaluations: Across all 16 multishot NLP tasks, OPT performance is roughly similar to GPT-3’s reported performance for most tasks, with some erratic results.
  • Pre-training: Initial planning was led by Susan Zhang.
  • Pre-training: Training infrastructure and initial ablations were handled by Naman Goyal, Myle Ott, Stephen Roller, Sam Shleifer, and Susan Zhang.
  • Pre-training: Training efficiency was handled by Naman Goyal, Myle Ott, and Sam Shleifer, while data curation and deduplication were handled by Shuhoi Chen, Myle Ott, and Stephen Roller.
  • Pre-training: Training and monitoring OPT-175B involved Mikel Artetxe, Moya Chen, Naman Goyal, Punit Singh Koura, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Stephen Roller, and Susan Zhang.
  • Evaluations: NLP, dialogue, and responsible AI evaluations were attributed to Xian Li, Xi Victoria Lin, Todor Mihaylov, Stephen Roller, Anjali Sridhar, Tianlu Wang, and Punit Singh Koura.
  • Evaluations: Paper writing, code release preparation, and responsible AI conduct were credited to Moya Chen, Stephen Roller, Luke Zettlemoyer, Susan Zhang, Christopher Dewan, and Mona Diab.

C Datasheet … D.2 Intended Use

The OPT dataset datasheet describes a 180B-token, 800GB corpus of publicly available textual documents used to pre-train OPT models, with documented collection, preprocessing, risks, and maintenance limitations. The model card positions OPT-175B for responsible-AI language-model research by researchers, excludes production deployment, and provides it under a non-commercial license.

  • C Datasheet: The dataset documentation covers its creation, composition, collection, preprocessing, uses, distribution, and maintenance, following Gebru et al. (2021).A data card is provided for the dataset used to train the OPT models.
  • C.2 Composition: A random validation set of approximately 200MB is held out proportionally to each dataset’s size, while naturally occurring cross-dataset duplication may remain.The authors report no other added redundancies, errors, or sources of noise.
  • C.2 Composition; C.5 Uses: The dataset includes public Common Crawl and Reddit material that may contain offensive, threatening, or anxiety-inducing sentences, and some documents relate to people without explicit subpopulation identification.No currently known tasks are prohibited, according to the datasheet.
  • C.3 Collection Process: The data was downloaded from the internet, mined, filtered, and sampled by machines; CC-News includes English articles crawled between September 2016 and September 2021.Some toxicity and bias evaluations were performed, with details referred to the main document and model card.
  • C.4 Preprocessing/cleaning/labeling; C.5 Uses: Component datasets underwent standard cleaning and re-formatting, including removal of repetitive or non-informative text, while raw components remain publicly available at their original locations.The dataset was used to pre-train OPT models and can support pre-training language models for many language tasks.
  • C.6 Distribution; C.7 Maintenance: The dataset was not being distributed to third parties at the time, has no stated license or regulatory terms, and had no current update or contribution mechanism.Meta AI supports, hosts, and maintains the dataset.

D.3 Data, Limitations, and Recommendations

OPT-175B’s training data was selected for breadth and availability, while evaluations were chosen for comparability and included Responsible AI assessments. The model retains limitations in bias, safety, generation diversity, and hallucination, but its release aims to expand open science and research access.

  • Data: Training data for OPT-175B was selected based on breadth and availability.Further information is provided in the Data Card in Appendix C.
  • Data: Evaluations were selected to enable comparable performance assessments against similar-scale models and included Responsible AI evaluations.The Responsible AI evaluations addressed community concerns about safety and fairness in large language models.
  • Limitations: OPT-175B has limitations involving bias, safety, generation diversity, and hallucination.The passage links downstream quality impacts to the diversity or lack thereof of training data and states that OPT-175B is not immune to broader large-language-model issues.
  • Recommendations for future work: The release aims to enable research avenues and increase open science around large language models, natural language processing, and deep learning.Section 6 discusses considerations for release and potential research enabled by opening the model to more researchers.

E Sample Model Outputs

Sample outputs intentionally showcase both successes and failures of OPT-175B across creative writing, conversation, translation, paper writing, arithmetic, and programming. The examples demonstrate useful capabilities alongside limitations in form, repetition, multilinguality, arithmetic, and output stability.

  • Sample selection: The samples intentionally highlight both successes and failures of OPT-175B, with prompts shown separately from their continuations.Initial prompts are presented in bold, followed by the model-generated continuations.
  • Poetry generation: OPT-175B can write entertaining poetry about dodos, samosas, and performance reviews, but struggles to follow rhyme or meter.The observed poetic outputs were engaging despite weaknesses in formal poetic constraints.
  • Conversation generation: When prompted as the Statue of Liberty, OPT-175B adopts a patriotic personality but later produces simple and linguistically repetitive conversation.The degradation appears further into the conversation.
  • Few-shot translation: Although not intentionally trained to be multilingual, OPT shows limited anecdotal success with simple translations in German, Spanish, French, and Chinese.The result is described as a basic few-shot translation capability rather than deliberate multilingual training.
  • Paper writing: Prompting with “1. Introduction” generally yields more interesting paper-writing outputs than prompting with “Abstract.”The prompt was inspired by the first sentence of the seminal ResNet work.
  • Arithmetic and programming: OPT-175B makes mistakes when extending arithmetic from addition to other operations, and changing a Python variable name can alter its generated output.These examples expose limitations in arithmetic generalization and sensitivity in programming generation.
Loading 2205.01068v4…