PaperlayerView exact revision on arXiv

Source-linked AI summary

Ministral 3

Alexander H. Liu, Kartik Khandelwal, Sandeep Subramanian, Victor Jouault, Abhinav Rastogi, Adrien Sadé, Alan Jeffares, Albert Jiang, Alexandre Cahill, Alexandre Gavaudan, Alexandre Sablayrolles, Amélie Héliou, Amos You, Andy Ehrenberg, Andy Lo, Anton Eliseev, Antonia Calvi, Avinash Sooriyarachchi, Baptiste Bout, Baptiste Rozière, Baudouin De Monicault, Clémence Lanfranchi, Corentin Barreau, Cyprien Courtot, Daniele Grattarola, Darius Dabert, Diego de las Casas, Elliot Chane-Sane, Faruk Ahmed, Gabrielle Berrada, Gaëtan Ecrepont, Gauthier Guinet, Georgii Novikov, Guillaume Kunsch, Guillaume Lample, Guillaume Martin, Gunshi Gupta, Jan Ludziejewski, Jason Rute, Joachim Studnia, Jonas Amar, Joséphine Delas, Josselin Somerville Roberts, Karmesh Yadav, Khyathi Chandu, Kush Jain, Laurence Aitchison, Laurent Fainsin, Léonard Blier, Lingxiao Zhao, Louis Martin, Lucile Saulnier, Luyu Gao, Maarten Buyl, Margaret Jennings, Marie Pellat, Mark Prins, Mathieu Poirée, Mathilde Guillaumin, Matthieu Dinot, Matthieu Futeral, Maxime Darrin, Maximilian Augustin, Mia Chiquier, Michel Schimpf, Nathan Grinsztajn, Neha Gupta, Nikhil Raghuraman, Olivier Bousquet, Olivier Duchenne, Patricia Wang, Patrick von Platen, Paul Jacob, Paul Wambergue, Paula Kurylowicz, Pavankumar Reddy Muddireddy, Philomène Chagniot, Pierre Stock, Pravesh Agrawal, Quentin Torroba, Romain Sauvestre, Roman Soletskyi, Rupert Menneer, Sagar Vaze, Samuel Barry, Sanchit Gandhi, Siddhant Waghjale, Siddharth Gandhi, Soham Ghosh, Srijan Mishra, Sumukh Aithal, Szymon Antoniak, Teven Le Scao, Théo Cachet, Theo Simon Sorg, Thibaut Lavril, Thiziri Nait Saada, Thomas Chabal, Thomas Foubert, Thomas Robert, Thomas Wang, Tim Lawson, Tom Bewley, Tom Bewley, Tom Edwards, Umar Jamil, Umberto Tomasini, Valeriia Nemychnikova, Van Phung, Vincent Maladière, Virgile Richard, Wassim Bouaziz, Wen-Ding Li, William Marshall, Xinghui Li, Xinyu Yang, Yassine El Ouahidi, Yihan Wang, Yunhao Tang, Zaccharie Ramzi

arXiv:2601.08584v1cs.CL

TL;DR

Ministral 3 addresses the cost of producing capable compact language models by deriving smaller children from a strong pretrained parent through iterative pruning and distillation. The resulting 3B, 8B, and 14B families include base, instruction, and reasoning variants, and report competitive performance against similarly sized open-weight models. The 14B Base model closely matches Mistral Small 3.1 Base while being more than 40% smaller and trained on a much shorter horizon.

  • Problem

    Ministral 3 targets efficient dense language models for resource-constrained environments, where training compact models from scratch requires substantial compute and data.

  • Method

    The paper uses Cascade Distillation, iteratively pruning a pretrained parent and distilling its logits into smaller child models before post-training base, instruction, and reasoning variants.

  • Results

    More than 40% smaller, Ministral 3 14B Base closely matches Mistral Small 3.1 Base while using a much shorter training horizon.

  • Takeaways & Limitations

    Ministral 3 provides 3B, 8B, and 14B open-weight model families with base, instruction, and reasoning variants, image understanding, and contexts up to 256k tokens.

  • Takeaways & Limitations

    The 3B base model is more sensitive than the 14B and 8B models to hyper-parameter choices during fine-tuning.

Abstract

from arXiv · show

We introduce the Ministral 3 series, a family of parameter-efficient dense language models designed for compute and memory constrained applications, available in three model sizes: 3B, 8B, and 14B parameters. For each model size, we release three variants: a pretrained base model for general-purpose use, an instruction finetuned, and a reasoning model for complex problem-solving. In addition, we present our recipe to derive the Ministral 3 models through Cascade Distillation, an iterative pruning and continued training with distillation technique. Each model comes with image understanding capabilities, all under the Apache 2.0 license.

1 Introduction

Ministral 3 is a family of compact dense models trained through Cascade Distillation, which combines iterative shrinking and distillation from a pretrained parent. The family spans three sizes and three variants, with competitive performance and post-training results against similarly sized open-weight models.

  • Training approach: Cascade Distillation progressively transfers pretrained knowledge from a large parent model into compact child models through iterative pruning and distillation.The recipe uses Mistral Small 3.1 as the parent model and trains children with a shorter horizon.
  • Efficiency and performance: More than 40% smaller, Ministral 3 14B Base closely matches Mistral Small 3.1 Base while using a much shorter training horizon.This comparison illustrates the efficiency of the proposed recipe relative to the parent model.
  • Efficiency and performance: After post-training, Ministral 3 achieves competitive results with similarly sized open-weight models including Gemma 3, Qwen 3, and Mistral Small 3.2 2506.The comparison is reported at the family level after post-training.
  • Training approach: The training recipe begins by pruning Mistral Small 3.1, then applies short- and long-context distillation before post-training base models into instruction-following and reasoning variants.Instruction-following uses SFT and ODPO, while reasoning uses SFT with chain-of-thought data, GRPO, and ODPO.
  • Model family: The family contains 9 dense models across 3B, 8B, and 14B scales, with base, instruction-finetuned, and reasoning variants.All models are open-weight under the Apache 2.0 license.

2 Model Architecture

Ministral 3 uses a shared decoder-only Transformer foundation with size-specific scaling across 3B, 8B, and 14B models. The models include long-context and image-understanding components, with a frozen vision encoder and a newly trained projection.

  • Architecture: The family uses a decoder-only Transformer architecture with 3B, 8B, and 14B parameter configurations.The configurations use 26, 34, and 40 layers respectively.
  • Architecture: All models use Grouped Query Attention, RoPE positional embeddings, SwiGLU activation, and RMSNorm.They share 32 query heads and 8 key-value heads.
  • Long context: Long-context extension uses YaRN and position-based softmax temperature scaling in the attention layer.These choices support the family’s extended context configuration.
  • Vision: A frozen 410M-parameter ViT provides image understanding, while each language model receives a newly trained projection from vision to language space.The vision encoder is copied from Mistral Small 3.1 Base and follows the Pixtral architecture.

3 Training Recipe

Ministral 3 training consists of pretraining followed by two distinct post-training paths. These paths produce instruction-finetuned and reasoning variants from each base model.

  • Training pipeline: The training pipeline first performs pretraining and then uses separate post-training phases to create instruction-finetuned and reasoning variants.The two paths are illustrated in Figure 1.

3.1 Pretraining

Cascade Distillation constructs progressively smaller models by repeatedly pruning a pretrained parent and distilling its logits into each child. Ministral 3 pretraining uses short-context and long-context stages, while pruning preserves important model components and avoids data repetition.

  • Cascade Distillation: Cascade Distillation repeats pruning, distillation, and further shrinking to derive children of decreasing target sizes from a larger pretrained parent.The process starts from Mistral Small 3.1 Base and uses the parent as teacher across stages.
  • Efficiency: Compared with training each small model from scratch, Cascade Distillation is significantly more FLOP efficient and processes the data mix in a single run without repetition.The end-to-end procedure can also be viewed as continual pretraining with weight pruning.
  • Pruning: The pruning stage takes a pretrained model and target size configuration, using activation norms to score layers and PCA-based reductions for hidden dimensions.Layer importance is estimated from the ratio of output to input activation norms.
  • Distillation: Forward KL logit distillation on mixed text and interleaved text-image data outperforms tuning a weighted combination with next-token prediction.The parent model serves as teacher for all stages and model sizes.
  • Pretraining stages: Short-context distillation uses a 16,384-token window, after which the resulting model initializes pruning for the next child model.The short-context output is reused in the cascade.
  • Pretraining stages: Long-context distillation extends the context window from 16,384 to 262,144 tokens using YaRN and position-based temperature scaling.This is the second pretraining stage for each final base model.

3.2 Post-Training: Ministral Instruct

Ministral Instruct models are produced through supervised fine-tuning and online preference optimization on curated multimodal and text-only instruction data. Online preference optimization improves alignment with human preferences over both SFT and offline variants.

  • Training pipeline: Instruction tuning uses curated multimodal and text-only data through supervised fine-tuning followed by Online Direct Preference Optimization.The models are fine-tuned in two stages to impart instruction-following capabilities.
  • Training pipeline: ODPO samples two responses from the current policy and uses a text-based reward model to rank them.The online variant uses temperature T=0.7 for response sampling.
  • Preference optimization: The Pairwise Reward Model replaces hard winner/loser labels with probability-weighted response preferences in the refined DPO loss.The reward model is trained on structured pairwise data containing a conversation history and two candidate responses.
  • Preference optimization: Online preference optimization mitigates model-induced artifacts such as infinite generations and improves tool-use performance when tool execution is enabled.Responses exhibiting infinite loops are automatically treated as losers during sampling.
  • Results: Online preference optimization significantly improves alignment with human preferences over both SFT and offline variants.The resulting models are released as Ministral 3-14B/8B/3B Instruct.

3.3 Post-Training: Ministral Reasoning

Ministral Reasoning models are created from long-context pretrained checkpoints through a three-stage SFT, GRPO, and ODPO pipeline. The training combines reasoning traces with STEM and general reinforcement learning, while longer generations and targeted distillation address reasoning and stability challenges.

  • Post-training pipeline: Reasoning models start from long-context pretrained checkpoints and undergo SFT, GRPO, and ODPO for inference-time scaling.The released reasoning-oriented models are Ministral 3 14B/8B/3B Reasoning.
  • Supervised fine-tuning: SFT uses short and long chain-of-thought samples spanning mathematics, coding, dialogue, instruction following, multilingual tasks, tool use, and visual reasoning.The data is lightly filtered for formatting, repetition, and undesirable language switching.
  • Supervised fine-tuning: The 3B model uses logit distillation from Magistral Small 1.2 because vanilla SFT produced verbosity, repetition, and infinite generations.This distillation reduced verbosity and stabilized subsequent reinforcement-learning training.
  • Reinforcement learning: GRPO trains first on math, code, and visual reasoning, then on general chat, instruction-following, and open-ended reasoning tasks.The general stage sets rewards from the fraction of satisfied rubric heuristics judged by an LLM.
  • Reinforcement learning: Increasing maximum generation length from 32K to 80K reduced truncated generations and produced additional gains on challenging problems.The longer limit allowed reasoning to finish for difficult tasks.
  • Post-RL alignment: ODPO is applied after RL to align outputs with user preferences and polish conversational and instructional behavior.Thinking chunks are removed before generations are scored by the reward model.

4 Results

Ministral 3 is evaluated across general, math and code, multimodal, and post-training benchmarks against open-weight models using a common evaluation pipeline. Across sizes, the results show competitive performance, including strong parameter efficiency relative to larger Gemma models.

  • Evaluation setup: Evaluations cover general, math and code, multimodal, and post-training benchmarks, with external models rerun through the authors’ pipeline for fair comparison.The benchmark suite includes MMLU, MATH, GPQA Diamond, MBPP, MMMU, MathVista, and Arena Hard, among others.
  • Base-model results: At 14B, Ministral 3 outperforms Qwen 3 14B on TriviaQA and MATH while remaining competitive on other benchmarks.The comparison uses similarly sized open-weight models.
  • Base-model results: At 3B, the overall trend persists while performance gaps between models become more pronounced.Additional pretraining results include comparisons with the teacher model.
  • Reasoning-model results: Reasoning models are compared with size-matched Qwen 3 counterparts using pass@16, except LiveCodeBench, which uses pass@5.The evaluation covers mathematics, science, and code benchmarks.

5 Discussions

The discussions examine how teacher choice, training data, verbosity, and ODPO affect Ministral 3 models. Results show that teacher strength benefits different stages differently, longer reasoning improves STEM performance but can reduce conversational quality, and ODPO helps larger reasoning variants more reliably than the 3B model.

  • 5.1 Choice of Teacher Model for Distillation: Distilling from Mistral Small 3.1 outperformed the much stronger Mistral Medium 3 during pretraining, whereas post-training benefited from the more capable teacher.The pretraining result held even without FLOP matching, while post-training showed the opposite teacher preference.
  • 5.1 Choice of Teacher Model for Distillation: Post-trained teachers improved pretraining results, especially on MATH and code, with smaller multimodal gains and negligible effects on knowledge metrics.The comparison was between instruct and pretrained teacher versions.
  • 5.1 Choice of Teacher Model for Distillation: Preference-tuned checkpoints were substantially better SFT teachers than SFT checkpoints, and their gains persisted after the student’s own preference tuning.
  • 5.2 Model Verbosity: Increasing long-CoT data improved STEM benchmark performance but also produced excessive reflection, internal monologues, and backtracking.The experiment combined varying proportions of long chain-of-thought traces with curated system prompts.
  • 5.3 ODPO for Ministral 3 Reasoning: ODPO substantially improved chat benchmarks for the 14B and 8B reasoning models, while the 3B model showed no significant public-benchmark gains but performed better in internal human evaluations.The ODPO checkpoint was nevertheless selected as the 3B release candidate.
  • 5.3 ODPO for Ministral 3 Reasoning: The 3B base model was more sensitive than the 14B and 8B models to fine-tuning hyper-parameter choices.

6 Conclusion

Ministral 3 is presented as an efficient dense-model family for resource-constrained environments. It spans three sizes and three variants, uses iterative distillation from larger teachers, supports vision and long contexts, and is intended to contribute to open-source development.

  • 6 Conclusion: Ministral 3 comprises efficient dense language models designed for resource-constrained environments.
  • 6 Conclusion: The family includes 14B, 8B, and 3B models, each available in base, instruction-following, and reasoning-enhanced variants.
  • 6 Conclusion: The models are created through iterative distillation from Mistral Small 3.1 and Mistral Medium 3 and support vision capabilities with contexts up to 256K tokens.
  • 6 Conclusion: The authors frame Ministral 3 as supporting open-source initiatives and contributing to a stronger open-source ecosystem.
Loading 2601.08584v1…