Source-linked AI summary
Mistral 7B
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, William El Sayed
TL;DR
Larger language models can improve performance but raise computational costs and inference latency, motivating efficient models that retain strong capabilities. Mistral 7B combines grouped-query and sliding-window attention, and surpasses Llama 2 13B across all metrics while outperforming Llama 1 34B on most benchmarks.
Problem
Increasing model size can improve performance but also raises computational costs and inference latency, motivating efficient models with strong capabilities.
Method
Mistral 7B combines grouped-query attention for faster, lower-memory inference with sliding-window attention for longer sequences at reduced computational cost.
Results
Mistral 7B surpasses Llama 2 13B across all metrics and outperforms Llama 1 34B on most benchmarks, especially in code, mathematics, and reasoning.
Takeaways & Limitations
Mistral 7B suggests that language models can compress knowledge more than previously thought, shifting attention toward balancing capability, training cost, and inference cost.
Takeaways & Limitations
Some benchmark results use evaluation protocols that differ from those reported in the Llama 2 paper, including MBPP and TriviaQA procedures.
Abstract
from arXiv · showhide
We introduce Mistral 7B v0.1, a 7-billion-parameter language model engineered for superior performance and efficiency. Mistral 7B outperforms Llama 2 13B across all evaluated benchmarks, and Llama 1 34B in reasoning, mathematics, and code generation. Our model leverages grouped-query attention (GQA) for faster inference, coupled with sliding window attention (SWA) to effectively handle sequences of arbitrary length with a reduced inference cost. We also provide a model fine-tuned to follow instructions, Mistral 7B -- Instruct, that surpasses the Llama 2 13B -- Chat model both on human and automated benchmarks. Our models are released under the Apache 2.0 license.
1 Introduction
Mistral 7B targets the tension between high model performance and the computational cost and latency of scaling language models. It combines GQA and SWA for more efficient inference and longer-sequence handling, and supports practical deployment and fine-tuning under the Apache 2.0 license.
- Motivation: Mistral 7B addresses the deployment barriers created by larger models’ computational costs and inference latency.The introduction frames balanced models that combine high performance with efficiency as critically important for real-world use.
- Method: GQA accelerates inference, reduces decoding memory requirements, and enables higher batch sizes and throughput.These benefits are identified as important for real-time applications.
- Method: SWA handles longer sequences more effectively at reduced computational cost, alleviating a common limitation in large language models.SWA is presented alongside GQA as a core efficiency mechanism.
- Contribution: Mistral 7B is released under the Apache 2.0 license with deployment support for local and cloud platforms.The release includes a reference implementation, vLLM and SkyPilot support, and streamlined Hugging Face integration.
- Contribution: The model is designed for fine-tuning across many tasks and aims to enable affordable, efficient, high-performing language models for real-world applications.The paper demonstrates this adaptability with a chat model.
2 Architectural details
Mistral 7B uses sliding window attention to reduce the cost of long-context inference while retaining access to information beyond the local window. Its architecture also includes rolling-buffer caching and chunked prompt pre-fill to bound memory usage.
- Sliding Window Attention: Sliding window attention limits each token to W previous-layer tokens, reducing vanilla attention’s quadratic sequence-length operations and linearly growing memory.Tokens outside the window can still influence later predictions through stacked transformer layers.
- Sliding Window Attention: With W = 4096, stacked layers provide a theoretical attention span of approximately 131K tokens.At layer k, position i attends to positions i − W through i, enabling access up to W × k input tokens.
- Rolling Buffer Cache: 8x lower cache memory usage on a sequence length of 32k tokens is achieved with a rolling buffer cache of fixed size W.Keys and values are stored at position i mod W, so older entries are overwritten once positions exceed W.
- Pre-fill and Chunking: Large prompts are chunked using the window size as the chunk size, allowing the (k, v) cache to be pre-filled incrementally while limiting memory usage.Each chunk attends to the cache and itself, with causal masking within the chunk.
3 Results
Mistral 7B surpasses Llama 2 13B across all evaluated metrics and is especially strong in code, mathematics, and reasoning. Its efficiency matches performance expected from Llama 2 models more than three times larger on several categories.
- Benchmark Results: The evaluation covers commonsense reasoning, world knowledge, reading comprehension, mathematics, code, and aggregated benchmarks using specified few-shot settings.Tasks include GSM8K with maj@8, MATH with maj@4, Humaneval, MBPP, MMLU, BBH, and AGI Eval.
- Evaluation Limitations: The comparison reports Llama 1 34B because Llama 2 34B was not open-sourced.This constrains the larger-model comparison to Llama 1 34B.
- Size and Efficiency: More than 3x: Mistral 7B mirrors reasoning, comprehension, and STEM reasoning performance expected from a Llama 2 model over three times larger.On knowledge benchmarks, its compression rate is 1.9x.
- Evaluation Limitations: Evaluation protocols differ from the Llama 2 paper: MBPP uses the hand-verified subset, while TriviaQA omits Wikipedia contexts.These differences affect direct comparability on those benchmarks.
4 Instruction Finetuning
Mistral 7B – Instruct was produced through straightforward fine-tuning on publicly available Hugging Face instruction datasets, without proprietary data or training tricks. It outperformed all evaluated 7B chat models on MT-Bench and was comparable to 13B chat models, while Mistral 7B outputs received more human preferences than Llama 2 13B.
- Instruction fine-tuning: Mistral 7B – Instruct used publicly available Hugging Face instruction datasets without proprietary data or training tricks.The authors describe it as a simple, preliminary demonstration that the base model can be fine-tuned for good performance.
- Instruction fine-tuning: Mistral 7B – Instruct outperformed all 7B models on MT-Bench and was comparable to 13B chat models.Table 3 summarizes the comparison against chat models.
- Human evaluation: 5020 human preferences favored Mistral 7B outputs, compared with 4143 for Llama 2 13B as of October 6, 2023.Participants selected their preferred response between anonymous model outputs.
5 Adding guardrails for front-facing applications
This section presents system prompting as an optional way to enforce output guardrails and shows that Mistral 7B can perform fine-grained content moderation. The model can decline harmful requests while preserving useful answers in some cases and classify unsafe content with high precision and recall.
- System prompting: System prompting guides Mistral 7B – Instruct to generate answers within specified guardrails, allowing users to trade off model utility against guardrail enforcement.The approach is described as similar to prior work with Llama 2.
- System prompting: 100% of 175 harmful questions were properly declined by the model with the recommended system prompt.The safety evaluation used a set of 175 unsafe prompts.
- Content moderation: Mistral 7B – Instruct classifies prompts or generated answers as acceptable or as illegal, hateful, harassing, violent, or unqualified advice content.The listed categories include terrorism, child abuse, fraud, discrimination, self-harm, bullying, and legal, medical, or financial advice.
- Content moderation: The self-reflection prompt achieves 99.4% precision and 95.6% recall when classifying acceptable prompts as positives.Evaluation used a manually curated and balanced dataset of adversarial and standard prompts.
- Content moderation: Moderation supports use cases including social-media or forum comment moderation and internet brand monitoring, with users selecting which categories to filter.Category filtering can be tailored to the particular use case.
6 Conclusion
Mistral 7B demonstrates that language models can compress knowledge more than previously thought, motivating a three-dimensional view of capabilities, training cost, and inference cost. The authors identify further work toward achieving the best performance with the smallest possible model.
- Mistral 7B demonstrates that language models may compress knowledge more than previously thought.
- The paper argues that model evaluation should consider three dimensions: capabilities, training cost, and inference cost.This extends the field’s emphasis beyond two-dimensional scaling laws linking capabilities directly to training cost.
- The authors identify substantial room to obtain the best performance with the smallest possible model.