Source-linked AI summary
OLMo: Accelerating the Science of Language Models
Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Harsh Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, Shane Arora, David Atkinson, Russell Authur, Khyathi Raghavi Chandu, Arman Cohan, Jennifer Dumas, Yanai Elazar, Yuling Gu, Jack Hessel, Tushar Khot, William Merrill, Jacob Morrison, Niklas Muennighoff, Aakanksha Naik, Crystal Nam, Matthew E. Peters, Valentina Pyatkin, Abhilasha Ravichander, Dustin Schwenk, Saurabh Shah, Will Smith, Emma Strubell, Nishant Subramani, Mitchell Wortsman, Pradeep Dasigi, Nathan Lambert, Kyle Richardson, Luke Zettlemoyer, Jesse Dodge, Kyle Lo, Luca Soldaini, Noah A. Smith, Hannaneh Hajishirzi
TL;DR
Powerful language models increasingly conceal the data and development details needed for scientific study. OLMo responds with an open model and framework spanning data, training, evaluation, checkpoints, and adaptation, while acknowledging that its English-focused data and model behavior retain important limitations.
Problem
Commercially valuable language models increasingly hide training data, architectures, and development details needed to study their capabilities, biases, and risks.
Method
OLMo releases a language-model framework including open models, exact training data, training and evaluation code, checkpoints, logs, and adaptation resources.
Results
OLMo is presented as a state-of-the-art, truly open language model and framework for building and studying language modeling.
Takeaways & Limitations
The open release supports fuller and more controlled scientific evaluation and enables practitioners to build on the models without training from scratch.
Takeaways & Limitations
The work focuses on English pretraining data, which may contain toxic language, personal information, and copyrighted text despite mitigation efforts.
Abstract
from arXiv · showhide
Language models (LMs) have become ubiquitous in both NLP research and in commercial product offerings. As their commercial importance has surged, the most powerful models have become closed off, gated behind proprietary interfaces, with important details of their training data, architectures, and development undisclosed. Given the importance of these details in scientifically studying these models, including their biases and potential risks, we believe it is essential for the research community to have access to powerful, truly open LMs. To this end, we have built OLMo, a competitive, truly Open Language Model, to enable the scientific study of language models. Unlike most prior efforts that have only released model weights and inference code, we release OLMo alongside open training data and training and evaluation code. We hope this release will empower the open research community and inspire a new wave of innovation.
1 Introduction
OLMo addresses the growing closure of powerful language models by releasing a competitive model with the data, code, checkpoints, and logs needed for scientific study. Its framework is intended to support research into model capabilities, risks, and training decisions.
- As language models became commercially valuable, the largest systems increasingly hid training data, architectures, and development details behind proprietary interfaces.
- OLMo releases a powerful open language model together with training data, training and evaluation code, checkpoints, and training logs.
- OLMo narrows the gap between openly released models and the state-of-the-art capabilities of models such as Llama 2.
- The framework includes four 7B variants and one 1B model, all trained on at least 2T tokens, with hundreds of intermediate checkpoints.
- OLMo releases exact datasets, data-construction and analysis tools, evaluation frameworks, adaptation tools, and code and weights under Apache 2.0.
- The release is designed to catalyze research on how pretraining data, design choices, hyperparameters, and optimization methods affect language-model training and capabilities.
2 OLMo Framework
The OLMo framework combines open model architectures, the Dolma pretraining dataset, adaptation tools, and evaluation pipelines. It emphasizes reproducible, controlled study through released data, checkpoints, decontamination, and in-loop evaluation.
- OLMo Model and Architecture: OLMo uses a decoder-only transformer and provides 1B and 7B variants with architectural changes targeting stability and throughput.
- OLMo Model and Architecture: The architecture removes bias terms, uses non-parametric layer normalization, SwiGLU activations, and rotary positional embeddings.
- Pretraining Data: Dolma: Dolma is an openly released pretraining dataset built through language, quality, and content filtering, deduplication, source mixing, and tokenization.
- Pretraining Data: Dolma: The release includes high-performance data-curation tools and keeps documents from each source separate during curation and in the final dataset.
- Adaptation: OLMo adaptation uses instruction finetuning followed by Direct Preference Optimization on distilled preference data under the TÜLU setup.
- Evaluation: Evaluation combines in-loop downstream assessments with offline Catwalk task evaluation and Paloma perplexity evaluation against public models.
- Evaluation: Paloma covers 585 domains from 18 sources, while explicit decontamination and released checkpoints enable more controlled perplexity comparisons.
3 Training OLMo
OLMo’s training setup uses distributed, mixed-precision optimization and a reproducible 2T-token data pipeline across NVIDIA and AMD hardware. The two hardware runs achieved nearly identical evaluation performance.
- Distributed Training: OLMo’s pretraining setup uses ZeRO through PyTorch FSDP to shard model weights and optimizer state across GPUs.
- Distributed Training: Mixed-precision training uses bfloat16 for most operations while retaining full precision for selected stability-critical computations and gradient reduction.
- Optimization: The models use AdamW with a 5000-step learning-rate warmup, linear decay, and gradient clipping at a total l2-norm of 1.0.
- Data Preparation: The training data is a 2T-token Dolma sample, concatenated with EOS markers and grouped into shuffled 2048-token instances.
- Data Preparation: All released models train for at least 2T tokens, with some continuing into a second epoch using a different shuffle order.
- Hardware: The codebase was tested on LUMI AMD MI250X and MosaicML NVIDIA A100 clusters, using up to 256 nodes and 27 nodes respectively.
- Hardware: By 2T tokens, the NVIDIA and AMD runs produced nearly identical performance despite minor batch-size differences for throughput.
4 Results
OLMo-7B is competitive with comparable publicly available models on core zero-shot tasks and follows similar intrinsic scaling trends. Performance improves with final learning-rate decay, while sample efficiency varies with training–evaluation distribution similarity and adaptation substantially improves downstream performance and safety.
- Downstream evaluation: OLMo-7B is competitive against comparable publicly available models across 8 core zero-shot evaluation tasks.The reported checkpoint was trained on 2.46T tokens.
- Downstream evaluation: All core tasks except OBQA show upward accuracy trends as OLMo-7B is trained on more tokens.A sharp increase between the final two checkpoints corresponds to linearly reducing the learning rate to 0 over the final 1000 steps.
- Intrinsic language modeling evaluation: OLMo-7B follows similar aggregate bits-per-byte scaling trends to six comparably sized models on 11 decontaminated Paloma sources.Intermediate-checkpoint curves are influenced by checkpoint position in the learning-rate schedule.
- Intrinsic language modeling evaluation: Sample efficiency varies with training–evaluation distribution similarity: OLMo-7B performs well on Common Crawl-heavy sources but less efficiently on WikiText-103, M2D2 S2ORC, and M2D2 Wikipedia.The fine-grained analysis reports bits per byte separately for each of 11 sources.
- Adaptation Evaluation: Instruction tuning considerably improves OLMo-7B’s performance and safety, with additional gains especially after DPO training.The improvements include MMLU, ToxiGen, and TruthfulQA, and OLMo variants outperform most other chat variants after adaptation.
- Adaptation Evaluation: OLMo-7B still trails TÜLU 2, whose models apply the TÜLU mix to Llama 2.The authors suggest test-set contamination in Llama 2 and the TÜLU mix’s Llama-oriented design may contribute to the gap.
5 Artifacts Released
OLMo releases artifacts spanning pretraining, data, adaptation, and evaluation to support reproduction and detailed study of language-model development.
- Data: OLMo releases the full Dolma pretraining corpus and tools for reproducing data order, inspecting training data exposure, recreating the dataset, and analyzing it.The release also includes data-curation tools for further experimentation and reproduction.
- Adaptation: The artifacts cover adaptation training code and data alongside the broader framework’s released components.The supplied release description identifies adaptation as a separate artifact category.
6 Conclusion and Future Work
The paper presents OLMo as a truly open language model and framework for studying language-model science. The authors plan continued support and expansion, reporting later improvements and future work across model sizes, modalities, datasets, safety, and evaluation.
- Conclusion: OLMo releases training data, training and evaluation code, detailed training metrics, adapted models, and adaptation code and data.This whole-framework release differs from efforts that primarily provide model weights and inference code.
- Future Work: The authors intend to continuously support and extend OLMo and its framework for the open research community.They also aim to bring different model sizes, modalities, datasets, safety measures, and evaluations into future releases.
- Future Work: A later OLMo release improved MMLU by 24 points to 52%.This result is reported as an example of improvements made after the original release described in the paper.
Limitations
OLMo’s limitations span its English-focused data, incomplete training documentation, adaptation risks, and imperfect evaluation coverage. The authors also acknowledge broader societal limitations beyond the paper’s scope.
- Scope: The authors acknowledge significant societal limitations beyond what this section can cover.They connect this boundary to the broad societal reach of AI systems.
- Data: The pretraining data focuses on English and may contain toxic language, personal information, and copyrighted text despite mitigation efforts.The authors state that no perfect approach currently removes all such content.
- Training: The release does not provide extensive logs for training runs that diverged or failed to learn.The authors attribute this omission partly to limited page count.
- Adaptation: OLMo’s pretrained models retain bias, toxicity, and hallucination issues, while adapted models reduce but do not eliminate these generations.The adaptation data mixture was largely designed for the TÜLU/Llama model family and may not fit OLMo’s strengths and weaknesses.
- Evaluation: Many downstream evaluations are not representative of chatbot use, and automatic language-model evaluations remain noisy and imperfect.The authors therefore caution that comparisons should be interpreted carefully.
Ethics Statement
The authors argue that openness is essential for scientific understanding and broad participation, while acknowledging that open models can also be used harmfully. They favor openness because research on potential harms may also be accelerated.
- Ethics Statement: Increased openness is presented as essential for studying language-model abilities and limitations and enabling broad participation in development.The authors specifically extend this rationale to training on open data.
- Ethics Statement: Open releases let practitioners build on existing models instead of retraining from scratch, reducing repeated resource use and associated environmental impact.The passage frames this as a practical benefit of openness.
- Ethics Statement: Openness carries a risk that models may be used in unintended harmful ways, but the authors believe openness can accelerate research to understand and mitigate those harms.They state that stricter licensing would not remove the field’s overall risk.
A Training Settings
This section describes how OLMo estimates pretraining energy and carbon emissions using measured power, data-center efficiency, and local grid carbon intensity. The reported emissions are explicitly lower bounds.
- Carbon accounting: OLMo estimates emissions by multiplying total training power consumption by the carbon intensity of the training location’s power grid.This follows prior work and uses operational emissions rather than all development-related sources.
- Power measurement: Power consumption is estimated from 25ms node measurements averaged across training, multiplied by node count, and adjusted with a PUE factor of 1.1.The PUE factor represents a conservative 10% energy overhead for energy-efficient data centers.
- Reported emissions: 69.78 tCO2eq is the estimated total pretraining emissions for the LUMI-trained model under the official LUMI calculation.An alternative hydroelectric intensity estimate would imply 3.54 tCO2eq, but the authors use the official LUMI data.
- Accounting boundary: The emissions estimates are lower bounds because they exclude debugging, hyperparameter tuning, and downtime.The authors present open model releases as a possible way to reduce future duplicated pretraining emissions.
C Additional Evaluation
Additional evaluations show strong but qualified results: OLMo-7B is notably ahead on some perplexity sources and outperforms other models in aggregate on six extra tasks. The authors warn that contamination and task instability complicate interpretation.
- Additional perplexity results: OLMo-7B is much farther ahead on Dolma 100 Programming Languages than the other evaluated models.The apparent advantage may partly reflect contamination underestimation because Paloma cannot decontaminate code data.
- Architecture comparison: Table 5 compares language-model architecture and optimizer choices at the 7–8B scale, including parametric versus non-parametric layer normalization.The layer-normalization distinction concerns the presence or absence of adaptive gain and bias.
- Additional perplexity results: Code-data processing alignment may contribute to OLMo-7B’s 100 PLs result, while other models’ differently processed code data can perform worse despite similar contamination risk.The passage also notes possible in-distribution and contamination effects in Pile evaluation.
- Additional perplexity results: Perplexity results on five targeted sources should be interpreted carefully because superficial features can dominate the metric.TwitterAAE and Gab are highlighted as having short documents associated with unusually high bits per byte.
- Additional end-task results: The authors caution against relying heavily on the additional end-tasks because performance can be random, chance-driven, or inflated by spurious predictions and class imbalance.This limits their usefulness for comparing models throughout training.
- Additional end-task results: OLMo-7B outperforms the other models in aggregate across six additional zero-shot end-tasks.These tasks nevertheless provided limited and unstable signal during development.
D Adaptation Training Details
The adaptation setup specifies sequence length, data mixtures, and learning-rate schedules for instruction tuning and subsequent DPO training. Additional figures document Paloma sources and unstable signal from six end-tasks.
- Instruction tuning: Instruction-tuning hyperparameters were selected through small pilot experiments.
- Instruction tuning: Instruction tuning uses 2048-token sequences and the TÜLU V2 SFT mix, resplit so long conversations become 2048-token chunks.The released dataset is publicly available.
- DPO training: DPO training follows Ivison et al. (2023) with linear warmup for 10% of training and linear cooldown to zero thereafter.
- Additional evaluations: Figure 4 reports unstable performance and limited development signal for OLMo-7B across six additional end-tasks.
- DPO training: The adaptation data includes a modified UltraFeedback fixed variant with TruthfulQA prompts removed and GPT-generated aspect scores determining chosen and rejected pairs.
E Adaptation Evaluation and Model details
The evaluation compares canonical adapted models and TÜLU-based variants across capability, chat, toxicity, and truthfulness settings. The suite uses standardized zero-shot or generation-based procedures, with Table 8 distinguishing metric directions.
- Models compared: The comparison includes canonical best adapted models, TÜLU 2, and OLMo variants with SFT and DPO.OLMo+SFT uses the TÜLU 2 data, while OLMo+SFT+DPO adds DPO on UltraFeedback.
- Evaluation settings: MMLU uses the official script with zero few-shot examples and reports average accuracy across test examples.
- Evaluation settings: ToxiGen evaluates 500 hateful prompts per group and reports the percentage of generated text classified as toxic by a RoBERTa-large detector.
- Evaluation settings: TruthfulQA uses 818 questions, six in-context examples, greedy decoding, and LLaMA 2-based judges for truthfulness and informativeness.The primary reported metric is the percentage of responses that are informative and truthful.
- Evaluation settings: Table 8 states that lower is better for ToxiGen, while higher is better for the other evaluation metrics.
- Evaluation settings: AlpacaEval generates responses for 805 prompts and reports the percentage preferred over Davinci-003 by the alpaca_eval_gpt4 annotator.