Source-linked AI summary
Mixtral of Experts
Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang, Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang, Timothée Lacroix, William El Sayed
TL;DR
Mixtral addresses how to increase language-model capacity without applying all parameters to every token. It uses sparse routing to select two experts per token, and reports performance matching or exceeding Llama 2 70B and GPT-3.5 across evaluated benchmarks, while its Instruct version surpasses several chat-model baselines on human evaluations.
Problem
Mixtral targets increasing model parameter capacity while controlling the computation and latency used for each token.
Method
Mixtral replaces feed-forward blocks with sparse mixture-of-experts layers whose router selects two experts per token and combines their outputs.
Results
Mixtral matches or exceeds Llama 2 70B and GPT-3.5 across several benchmarks, with particularly strong performance in mathematics, code generation, and multilingual tasks; its Instruct model surpasses several chat-model baselines on human evaluations.
Takeaways & Limitations
Mixtral provides an openly licensed model that combines sparse per-token computation with performance reported against substantially larger and leading chat-model baselines.
Takeaways & Limitations
Inference comparisons based on active parameters omit memory costs and hardware utilization, while routing adds overhead and is better suited to batched workloads.
Abstract
from arXiv · showhide
We introduce Mixtral 8x7B, a Sparse Mixture of Experts (SMoE) language model. Mixtral has the same architecture as Mistral 7B, with the difference that each layer is composed of 8 feedforward blocks (i.e. experts). For every token, at each layer, a router network selects two experts to process the current state and combine their outputs. Even though each token only sees two experts, the selected experts can be different at each timestep. As a result, each token has access to 47B parameters, but only uses 13B active parameters during inference. Mixtral was trained with a context size of 32k tokens and it outperforms or matches Llama 2 70B and GPT-3.5 across all evaluated benchmarks. In particular, Mixtral vastly outperforms Llama 2 70B on mathematics, code generation, and multilingual benchmarks. We also provide a model fine-tuned to follow instructions, Mixtral 8x7B - Instruct, that surpasses GPT-3.5 Turbo, Claude-2.1, Gemini Pro, and Llama 2 70B - chat model on human benchmarks. Both the base and instruct models are released under the Apache 2.0 license.
1 Introduction
Mixtral 8x7B is an openly licensed sparse mixture-of-experts model that routes each token through two of eight feedforward experts, increasing capacity while controlling per-token computation. It matches or exceeds larger reference models across benchmarks, with especially strong results in mathematics, code, and multilingual tasks.
- Mixtral 8x7B is a sparse mixture-of-experts model with open weights under the Apache 2.0 license.
- Mixtral uses only a subset of its parameters for each token, enabling faster inference at low batch sizes and higher throughput at large batch sizes.
- At every layer, a router selects two of eight feedforward experts for each token and combines their outputs additively.The selected experts can vary across timesteps.
- Mixtral matches or exceeds Llama 2 70B and GPT-3.5 across several benchmarks.It was pretrained on multilingual data with a 32k-token context size.
- Mixtral significantly outperforms Llama 2 70B in mathematics, code generation, and multilingual-understanding tasks.Experiments also report successful retrieval from the 32k-token context window across sequence lengths and information locations.
- Mixtral 8x7B-Instruct is fine-tuned with supervised fine-tuning and Direct Preference Optimization to follow instructions.The models are released under the Apache 2.0 license for academic and commercial usage.
2 Architectural details
Mixtral replaces transformer feed-forward sub-blocks with sparse mixture-of-experts layers, using top-K routing to select a small number of expert networks per token. With K = 2, it increases total parameter capacity while keeping per-token computation comparatively controlled.
- Mixtral uses a transformer architecture with dense 32k-token context and replaces feed-forward blocks with mixture-of-experts layers.
- Sparse Mixture of Experts: An MoE layer computes a weighted sum of expert outputs, with weights supplied by a gating network.
- Sparse Mixture of Experts: The gating network can apply softmax to the logits of the top-K experts, while zero-gated experts need not be computed.
- Sparse Mixture of Experts: Increasing the number of experts while keeping K fixed raises total parameter count while keeping computational cost effectively constant.Total sparse parameters and per-token active parameters are distinct quantities.
- Sparse Mixture of Experts: MoE layers can use specialized sparse-matrix kernels and expert or model parallelism for efficient execution across GPUs.
- Sparse Mixture of Experts: Mixtral sets K = 2 and routes each token to two SwiGLU sub-blocks with different weight sets.The output uses the top-two gating weights to combine the two expert outputs.
3 Results
Mixtral is evaluated across broad benchmark categories against Llama models using a common evaluation pipeline. It generally matches or surpasses Llama 2 70B, with especially strong results in code, mathematics, multilingual tasks, and long-context retrieval, while efficiency comparisons exclude memory and hardware utilization.
- The evaluation re-runs a wide range of benchmarks with the authors’ pipeline, with protocol differences for MBPP and TriviaQA relative to the Llama 2 paper.
- Mixtral surpasses Llama 2 70B across most reported metrics, especially in code and mathematics benchmarks.
- Size and Efficiency: With 5x lower active parameters, Mixtral outperforms Llama 2 70B across most benchmark categories.Mixtral uses 13B active parameters per token; the analysis focuses on active parameters as proportional to inference compute cost.
- Multilingual benchmarks: Mixtral significantly outperforms Llama 2 70B in French, German, Spanish, and Italian multilingual benchmarks.
- Long range performance: Mixtral achieves 100% passkey retrieval accuracy regardless of context length or passkey position, while proof-pile perplexity decreases as context length increases.
- Bias Benchmarks: Mixtral presents less bias than Llama 2 on BBQ and more positive sentiment with similar within-group variances on BOLD.The reported BBQ values are 56.0% for Mixtral versus 51.5% for Llama 2.
4 Instruction Fine-tuning
Mixtral-Instruct is trained to follow instructions through supervised fine-tuning and Direct Preference Optimization. It outperforms several leading proprietary and open chat models in human evaluations and reaches an MT-Bench score of 8.30.
- 8.30 on MT-Bench makes Mixtral-Instruct the best open-weights model as of December 2023.
- Independent LMSys human evaluation shows Mixtral-Instruct outperforming GPT-3.5-Turbo, Gemini Pro, Claude-2.1, and Llama 2 70B chat.
5 Routing analysis
The routing analysis finds little domain specialization but reveals structured syntactic and temporal patterns in expert assignment. Higher-layer locality has implications for parallel execution and caching.
- Domain specialization: Expert assignment shows no obvious topic-based specialization across ArXiv, biology, and philosophy documents.The distributions are very similar across these domains at layers 0, 15, and 31.
- Domain specialization: DM Mathematics has a marginally different expert distribution, especially at the first and last layers.The paper attributes this divergence likely to the dataset’s synthetic nature and limited natural-language coverage.
- Syntactic structure: Expert choices exhibit structured syntactic behavior, routing tokens with similar syntactic roles through the same experts across domains.Examples include Python self, English Question, and code indentation tokens, particularly at the first and last layers.
- Temporal locality: 46% of consecutive tokens share an expert among their first or second choices, compared with 12.5% for repeated first choices.The reported repetition proportions indicate substantial temporal locality in expert selection.
- Temporal locality: Repetitions are close to random at the first layer but significantly higher at layers 15 and 31.This higher-layer locality can cause expert oversubscription under Expert Parallelism but may also support caching.
6 Conclusion
Mixtral 8x7B achieves state-of-the-art performance among open-source models, while its Instruct variant surpasses several competing chat models on human evaluations. The models are publicly released under the Apache 2.0 license.
- Conclusion: Mixtral 8x7B reaches state-of-the-art performance among open-source models.The conclusion identifies it as the first mixture-of-experts network to reach this level.
- Conclusion: Mixtral 8x7B Instruct outperforms Claude-2.1, Gemini Pro, and GPT-3.5 Turbo on human evaluation benchmarks.The conclusion also reports that it uses 13B active parameters per token while outperforming Llama 2 70B.
- Conclusion: Both trained and fine-tuned models are publicly available under the Apache 2.0 license.The release is intended to facilitate development of new techniques and applications across industries and domains.