OLMoE: Open Mixture-of-Experts Language Models

• The OLMoE model is openly released at hf.co/allenai/OLMoE-1B-7B-0924.Source: OLMoE · p. 1
• The project also provides the OLMoE-mix dataset, source code, and training logs.Source: OLMoE · p. 1
• OLMoE-1B-7B performs best among models with similar active parameter counts and is the most open MoE.Figure 1 compares performance, cost, and openness across open MoE and dense language models, highlighting MMLU as an overall-performance summary.Source: Figure 1 · p. 1

Motivation

• Sparsely activated MoEs improve efficiency by activating only a subset of layer experts per input, addressing the performance-cost trade-off that limits access to high-performing language models.Dense models activate all parameters for every input, whereas MoEs can provide greater efficiency with similar total parameter counts; frontier systems include Gemini-1.5 and reportedly GPT-4.Source: §1 · p. 3

Contributions

• OLMoE-1B-7B significantly outperforms open 1B models and matches Llama2-13B on MMLU at roughly one-tenth the inference cost, while OLMoE-1B-7B-Instruct exceeds larger instruct models on common benchmarks.The instruction-tuned model surpasses Llama2-13B-Chat, OLMo-7B-Instruct (0724), and DeepSeekMoE-16B [42] on benchmarks including MMLU, GSM8k, and HumanEval.Source: §1 · p. 3
• OLMoE-1B-7B uses 6.9B total parameters, activates 1.3B per token, and is pretrained on 5.1 trillion tokens, yielding inference costs similar to dense 1B models but higher memory requirements.The experiments report that MoEs train approximately twice as fast as dense models with equivalent active parameters.Source: §1 · p. 3

MoE design experiments

• Controlled experiments find that 64 granular experts with 8 active, dropless token-based routing, and no shared experts are effective design choices, while sparse upcycling helps mainly under small compute budgets.Dropless token-based routing outperforms expert-based routing, and the study reports limited benefits from sparsely upcycling pretrained dense models into MoEs [85].Source: §1 · p. 3

Routing analysis

• Routing saturates early during pretraining, experts are rarely co-activated, and experts specialize by domain and vocabulary.These analyses characterize how routing behavior develops and how specialization appears in OLMoE-1B-7B.Source: §1 · p. 3

Open release

• The project releases model-development resources including training code, intermediate checkpoints, logs, and training data under open-source licenses.The release is intended to facilitate further research and analysis of open MoE models.Source: §1 · p. 3

Pretraining architecture

• OLMoE replaces dense feedforward networks with MoE layers whose router selects k experts per token and combines their probability-weighted outputs.The router is a learned linear layer followed by softmax, and the selected experts’ outputs are summed across each layer [58].Source: §2 · pp. 3, 5
• The final design uses 1.3B active parameters out of 6.9B, activates 8 of 64 experts per layer, and applies dropless token-choice routing.OLMoE is trained from scratch with cross-entropy, load-balancing, and router z-losses weighted by α and β.Source: §2 · p. 5

Pretraining data

• OLMoE-MIX combines DCLM and Dolma 1.7 sources, including filtered Common Crawl, StarCoder, Algebraic Stack, arXiv, peS2o, and Wikipedia [90].The filtering removes documents containing long repeated n-gram sequences and applies additional quality filters to StarCoder data.Source: §2 · p. 6
• Pretraining runs for 5.133T tokens over 1.3 epochs, with dataset reshuffling and linear learning-rate decay to zero during the final 100B-token annealing phase [121] [65] [90].The pretraining configuration and dataset statistics are provided in the paper’s appendices and Table 2.Source: §2 · p. 6

Adaptation

• OLMoE-1B-7B-INSTRUCT follows instruction tuning with preference tuning and adds code, mathematics, No Robots, and Daring Anteater data for capability and diversity.The adaptation recipe builds on prior open models and reports datasets in Table 3 and hyperparameters in Appendix B [118, 190, 149, 156, 206, 31, 15, 138, 54, 184, 76, 188].Source: §2 · p. 6

During pretraining

• OLMoE-1B-7B matches or outperforms OLMo-7B at the end of pretraining while using less than half the training FLOPs and only 1B active parameters.Across all evaluated tasks, it reaches better performance with less compute than the dense OLMo models.Source: §3 · pp. 6–7

After pretraining

• OLMoE-1B-7B is the state of the art in its cost regime, performing best among models with fewer than 2B active parameters.It uses approximately 6–7× less compute per forward pass yet outperforms some 7B dense models, including Llama2-7B, while falling short of Llama3.1-8B.Source: §3 · p. 7

After adaptation

• SFT improves OLMoE-1B-7B on every measured task, including a greater than 10× gain on GSM8k, while DPO improves most tasks, especially AlpacaEval.The GSM8k gain is attributed to additional math data compensating for limited math data during pretraining.Source: §3 · p. 7
• OLMoE-1B-7B-Instruct has the highest average among all benchmarked models and outperforms Qwen1.5-3B-14B despite Qwen using more than twice as many parameters.Its 84% AlpacaEval score also exceeds much larger dense models such as Llama2-13B-Chat.Source: §3 · p. 8, Table 5 · p. 8

4.1.1 Mixture-of-Experts vs. Dense

• ∼3× fewer tokens and ∼2× faster training let the controlled MoE match dense-model performance, motivating 6.9B total and 1.3B active parameters.The MoE processes 23,600 tokens per second per GPU versus 37,500 for the dense model, limiting the speedup from the token reduction.Source: Figure 4 · p. 9, §4.1.1 · p. 9

4.1.2 Expert Granularity

• Around 10% improvement on HellaSwag and MMLU favors more granular experts, but the 64-expert configuration yields only 1–2% further downstream gains.The 32-expert setup uses 4 active experts, whereas the 64-expert setup uses 8; OLMoE-1B-7B therefore uses 64 experts because returns diminish.Source: §4.1.2 · pp. 9–10

4.1.3 Shared Experts

• Shared experts perform slightly worse than two routed experts because converting one routed expert removes almost 90% of possible expert combinations.OLMoE-1B-7B therefore omits shared experts, although learned frequently activated experts remain a possible alternative if load-balancing constraints are relaxed.Source: Figure 6 · p. 10, §4.1.3 · pp. 10–11

4.1.4 Expert Choice vs. Token Choice

• Token choice outperforms expert choice across depicted tasks, while expert choice is around 20% faster at 29,400 versus 24,400 tokens per second per device.The release uses token choice because its performance advantage outweighs expert choice’s throughput benefit in this dropless, load-balanced setup.Source: Figure 7 · p. 11, §4.1.4 · p. 11

4.1.5 Sparse Upcycling

• An MoE trained from scratch catches up with sparse upcycling after 500B tokens and begins outperforming it around 600B tokens, requiring 25% of the original dense model’s compute budget.Upcycling is not used because it constrains initialization and normalization choices, while OLMoE-1B-7B trains for 5T rather than 2T tokens.Source: Figure 8 · p. 12, §4.1.5 · p. 12

4.1.6 Load Balancing Loss

• The auxiliary load-balancing loss improves training and validation losses within a few billion tokens; without it, most experts remain unused, so OLMoE uses weight 0.01.The loss may constrain specialization, but removing it is left for future work.Source: Figure 9 · p. 13, Figure 10 · p. 13, §4.1.6 · p. 13

4.1.7 Router Z-loss

• Router z-loss improves stability and quality across training loss, validation loss, and downstream performance, despite reducing throughput by ∼2%; OLMoE uses weight 0.001.The z-loss penalizes large router logits that can cause numerical overflow in MoE matrix multiplications.Source: §4.1.7 · pp. 13–14, Figure 11 · p. 14

4.2.1 Dataset Experiments; 4.2.2 Initialization; 4.2.3 RMSNorm

• OLMOE-MIX, truncated-normal initialization, and RMSNorm improve their respective comparisons, leading to the final choices of DCLM-Dolma mixing, truncated initialization, and RMSNorm.OLMOE-MIX improves all three downstream metrics; truncated initialization becomes better and more stable around 450B tokens; RMSNorm improves performance but lowers throughput by 15%.Source: Figure 12 · p. 14, §4.2.1 · p. 14, Figure 13 · p. 15

4.2.4 Decaying Embedding Parameters

• The final model decays all parameters, including embeddings and RMSNorm, because RMSNorm decay is slightly better and embedding decay has only a minor performance impact.Embedding decay is also slightly better, supporting the simpler all-parameter weight-decay configuration.Source: MMLU Var (Acc %) · p. 16, §4.2.4 · p. 16, Figure 15 · p. 16

4.2.5 QK-Norm

• QK-Norm improves stability and performance, including slightly better training loss and prevention of a large gradient-norm spike, but reduces throughput by almost 10%.The comparison was also replicated with RMS layer normalization to rule out an artifact of using non-parametric normalization.Source: §4.2.5 · p. 16

4.2.6 AdamW Epsilon

• Decreasing AdamW epsilon from 1E-05 to 1E-08 significantly improves performance while remaining stable, so the final run uses 1E-08.The 1E-08 value is the recommended default cited in the experiment.Source: §4.2.6 · p. 17, Figure 19 · p. 17

4.3 Adaptation Settings

• Not using load balancing during adaptation performs better: 54.0 vs. 52.8 after SFT and 57.7 vs. 57.1 after DPO.The load-balancing loss decreases slightly during SFT, from 12.22 to 12.16, and expert activation distributions remain approximately unchanged.Source: §4.3 · pp. 17–18, Table 6 · p. 17
• The post-annealing checkpoint performs better than the pre-annealing checkpoint, scoring 54.0 vs. 53.8 after SFT and 57.7 vs. 56.3 after DPO.Accordingly, adaptation uses the checkpoint after the additional 100B tokens of annealing.Source: §4.3 · p. 18, Table 7 · p. 18
• KTO matches DPO in the reported setup, but DPO is selected for OLMOE-1B-7B-INSTRUCT because it scores higher on AlpacaEval.Both models are released.Source: §4.3 · p. 18, Table 7 · p. 18

5 MoE Analysis

• The analysis uses released intermediate checkpoints, data, and code to characterize four MoE properties: router saturation, expert co-activation, domain specialization, and vocabulary specialization.These properties are analyzed in Sections 5.1–5.4.Source: §5 · p. 18

5.1 Router Saturation

• At 1% of pretraining, up to ∼60% of top-8 routing has saturated, rising to ∼80% at 40%, while top-1 routing saturates more slowly and layer 0 is an outlier.Later layers saturate earlier, and slower first-layer saturation may contribute to changing assignments and impaired load balancing.Source: §5.1 · p. 19

5.2 Expert Co-activation

• Within-layer expert co-activation is generally weak, with only a few exceptions, although layers 7 and 15 contain several groups of two or three experts that activate together.The limited co-activation may indicate little redundancy among experts.Source: §5.2 · p. 20, Figure 21 · p. 20

5.3 Domain Specialization

• OLMoE exhibits strong domain specialization: arXiv routes nearly 100% to layer-0 expert 1, whereas generic C4 routing is much more balanced; Mixtral shows little specialization.The comparison uses OLMoE’s 64 experts with k = 8 and Mixtral’s 8 experts with k = 2 against uniform-routing baselines.Source: §5.3 · pp. 20, 22, Figure 22 · p. 21

5.4 Vocabulary Specialization

• Vocabulary specialization increases in later layers, focuses more on predicted output tokens than input tokens, and peaks at ∼90% for expert 27.Expert 27 specializes in many non-alphabetic tokens, while other experts specialize in geographic, connector, measurement, or family terms, linking vocabulary and domain specialization.Source: §5.4 · pp. 22, 24, Figure 23 · p. 22, Table 8 · p. 23

Advances in MoEs

• OLMoE-1B-7B is trained on 5T tokens, providing a testbed for studying MoE performance saturation versus dense models.The paper states that no prior MoE had been overtrained to this extent to its knowledge.Source: §6 · p. 24
• MoE research has improved routing, expert segmentation, stability, and efficiency, yet most language models, including Llama 3, still use dense architectures.The broader language-model landscape remains largely transformer-based, with architectural changes such as decoder-only training, SwiGLU, RoPE, MQA/GQA, and RMSNorm widely adopted.Source: §6 · p. 24

Open LMs

• OLMoE-1B-7B is presented as the first state-of-the-art fully open-source MoE LM, releasing model weights, training data, code, and logs.Earlier fully open-source releases included these components, but OpenMoE was the only fully open-source MoE and its poor performance limited its usefulness.Source: §6 · p. 24

7 Conclusion

• At 1B active and 7B total parameters, OLMoE achieves state-of-the-art performance among models with similar active-parameter counts.The models also outperform larger models including DeepSeekMoE-16B and Llama2-13B-Chat.Source: §7 · p. 24
• OLMoE-1B-7B and OLMoE-1B-7B-Instruct are released with their models, data, code, and logs.Source: §7 · p. 24
• The work shares training experiments and analyzes router saturation, expert co-activation, and domain and vocabulary specialization.Source: §7 · p. 24
• The fully open release is intended to help the field build better mixture-of-experts models.Source: §7 · p. 24
• The authors identify further OLMoE iterations as a path toward narrowing the gap between frontier and fully open models.Source: §7 · p. 24

A Artifacts

• Table 9 catalogs all artifacts released and used in the work.It links artifact names used in the paper, such as Figure 1, to URLs where they can be obtained.Source: Table 9 · p. 44

Pretraining

• Pretraining uses a sparsely activated decoder-only transformer with dropless MoE, 64 small experts of FFN dimension 1,024, router z-loss, load-balancing loss, and truncated-normal initialization with standard deviation 0.02.Source: B Training Configuration · p. 44
• 5T-token pretraining exceeds one epoch, uses AdamW epsilon 1.0E-08, and linearly decays learning rate from 5.0E-04 to 0 over the final 100B tokens.The dataset is shuffled before the second epoch, and these settings are experimentally evaluated in §4.Source: B Training Configuration · p. 45

Adaptation

• SFT trains for 2 epochs at learning rate 2.0E-5, while DPO trains for 3 epochs at 5.0E-7 with beta 0.1; KTO uses 1.3 epochs and RMSProp.SFT uses token-level loss aggregation and global batch size 128; DPO uses global batch size 32, and load-balancing loss is included in both.Source: B Training Configuration · p. 45

Hardware

• Pretraining uses 256 H100 GPUs for approximately 10 days, while instruction tuning uses 32 H100 GPUs for 33 hours and DPO preference tuning for 14 hours.KTO adaptation uses 8 H100 GPUs for 30 hours.Source: B Training Configuration · p. 45

C Evaluation Setup

• During pretraining, evaluation combines mostly 0-shot CF scoring with optional character normalization, MMLU 0–5-shot variation and MCF, plus Paloma perplexity.MCF performance for OLMoE-1B-7B begins rising around 1T training tokens, and evaluation code is released.Source: C Evaluation Setup · p. 47, Table 11 · p. 47
• After pretraining, the authors follow OLMES and DCLM protocols precisely, including their task formulations, prompting, normalization, and Core-versus-Extended task definitions.After adaptation, evaluations cover coding, reasoning, knowledge, instruction following, and safety using HumanEval, BBH, GSM8k, MMLU, AlpacaEval, IFEval, and XSTest.Source: C Evaluation Setup · pp. 47–48

D Openness of Models

• The openness comparison excludes Switch Transformers [56] because its older encoder-decoder MLM design differs substantially from recent MoE models.The comparison catalogs model licenses, data availability, and code release status across contemporary MoE systems.Source: D Openness of Models · p. 48

OLMOE-1B-7B

• OLMoE-1B-7B releases its model, data, code, and logs under open-source licenses, including Apache 2.0 for the model, code, and logs and ODC-By 1.0 for data.The logs use the same Apache 2.0 license as the code.Source: OLMOE-1B-7B · p. 50

E Additional Evaluation

• Additional evaluation includes training losses for OLMoE-1B-7B across Books, Reddit, and Stack datasets from Dolma 1.7 via Paloma.The losses are accompanied by released results, logs, and configurations.Source: Figure 24 · p. 50
• Figure 25 compares OLMoE-1B-7B with the best current OLMo models during pretraining and marks each run’s annealing interval.A corresponding version uses training FLOPs rather than training progress as the x-axis.Source: Figure 25 · p. 51
• Additional tables extend OLMES results and report DCLM metrics for Core and Extended subsets, distinguishing the annealed final checkpoint from the non-annealing checkpoint.The annealed checkpoint was derived from step 1,200,000, while the non-annealing run continued to step 1,220,000.Source: Table 12 · p. 52, Table 13 · p. 53

F Additional Experiments

• Adding Reddit or FLAN subsets to the pretraining mix produces no consistent gains, so neither is included in the final data mix.The comparison is reported in Figure 26 using Reddit or FLAN subsets from Dolma 1.7.Source: Figure 26 · p. 54, F Additional Experiments · p. 54
• Computing the load-balancing loss in FP32 does not reduce stability spikes, so the model retains BF16 computation.The experiment tests whether full-precision routing-related loss computation improves stability.Source: F Additional Experiments · p. 54, Figure 27 · p. 55
• After 700 billion tokens, no-noise upcycling performs slightly better, although both variants appear to converge to the same performance.The noisy variant randomly replaces 50% of each MLP with Gaussian samples of standard deviation 0.02.Source: F Additional Experiments · pp. 54–55
• Layer-shared MoE performs similarly to regular dense models, retains a small validation-loss and HellaSwag advantage, but reduces training throughput by around 20%.Its model-level load balancing permits expert deactivation and dense-model emulation, while shared experts could improve inference-time batching.Source: F Additional Experiments · pp. 55–56, Figure 29 · p. 55
• KTO adaptation uses the RMS optimizer and the 5,000-step checkpoint, with 5,000 and 10,000 steps corresponding to 1.3 and 2.6 dataset epochs.The choice is based on experiments varying optimizer and training duration; KTO denotes [54] and Adam denotes.Source: F Additional Experiments · p. 56, Table 14 · p. 56

G Additional Analysis

• OLMoE-1B-7B shows strong vocabulary and domain specialization, with routing probabilities concentrated at 0% or 100%, especially for GitHub and arXiv.Figure 32 compares OLMoE-1B-7B with Mixtral-8x7B across domains, using all activated experts.Source: Figure 30 · p. 56, Figure 31 · p. 56, Figure 32 · p. 57
• Despite omitting load-balancing loss during SFT and DPO, instruction adaptation does not substantially change the load distribution across active experts.The analysis considers 8 active experts among 64 total experts.Source: Figure 33 · p. 58
• Top-1 routing reveals domain-specific expert assignments and some cross-layer coordination, including substantial shared processing of Wikipedia tokens by experts 27 and 57.Figures 34–36 compare OLMoE-1B-7B and Mixtral-8x7B routing across domains and layers.Source: Figure 34 · p. 59, Figure 35 · p. 60, Figure 36 · p. 61

H Limitations and Future Work

• The authors identify four key limitations for this release and plan to address them in future OLMOE iterations.The limitations concern parameter capacity, data scale, modality, and multilingual coverage.Source: H Limitations and Future Work · p. 62
• Using only 1B active parameters limits OLMOE-1B-7B relative to models using more than seven times as many parameters, motivating larger allocations, recursion, agentic workflows, or rebalanced parameterization.The model has 7B total parameters but activates 1B per token; Figure 1 shows its efficiency advantage over more expensive models.Source: H Limitations and Future Work · p. 62
• OLMOE-1B-7B is trained on 5T tokens with a token multiplier of about 5,000, but longer training may help and its effectiveness for MoEs remains uncertain [57] [7].Llama 3 is cited as a dense model trained on 15T tokens.Source: H Limitations and Future Work · p. 62
• As a text-only model trained and evaluated predominantly in English, OLMOE-1B-7B lacks multimodal inputs or outputs and may have limited non-English usefulness [119] [136] [14] [47] [108] [53].Future work proposes multimodal versions and more non-English data [125] [98] [95] [112] [121].Source: H Limitations and Future Work · p. 62

J Change log

• OLMOE-1B-7B-0125 was released in January 2025 with models and datasets freely available on the Hugging Face hub, while the change log records parameter, learning-rate, and benchmark-table corrections and additions.The documented additions include references to OLMOE-1B-7B-0125, Zamba2, Llama3.2, and DeepSeekV2.Source: I OLMOE-1B-7B-0125 · pp. 62–63, J Change log · p. 63

I OLMOE-1B-7B-0125

• OLMOE-1B-7B-0125 retains the first-stage data mix and anneals on 100B curated DOLMINO tokens containing web, academic, question-answering, instruction, and math data.The DOLMINO composition and sampling distribution are summarized in Table 15.Source: I OLMOE-1B-7B-0125 · p. 62, Table 15 · p. 62
• +1.6 average improvement and +2.1 on MMLU: OLMOE-1B-7B-0125 outperforms the previous OLMOE iteration.The comparison appears in Table 16 on OLMES.Source: Table 16 · p. 63, I OLMOE-1B-7B-0125 · p. 63
• 10-point benchmark-average gain: post-training OLMOE-1B-7B-0125 is significantly better than OLMOE-1B-7B under the T¨ulu evaluation setup.The updated recipe uses improved SFT and DPO data plus PPO with verifier-based rewards.Source: Table 17 · p. 63, I OLMOE-1B-7B-0125 · p. 63