Source-linked AI summary
SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Model
Loubna Ben Allal, Anton Lozhkov, Elie Bakouch, Gabriel Martín Blázquez, Guilherme Penedo, Lewis Tunstall, Andrés Marafioti, Hynek Kydlíček, Agustín Piqueres Lajarín, Vaibhav Srivastav, Joshua Lochner, Caleb Fahlgren, Xuan-Son Nguyen, Clémentine Fourrier, Ben Burtenshaw, Hugo Larcher, Haojun Zhao, Cyril Zakka, Mathieu Morlon, Colin Raffel, Leandro von Werra, Thomas Wolf
TL;DR
Large language models are costly to train and deploy, motivating strong models that fit resource-constrained settings. SmolLM2 addresses this through data curation, multi-stage training, and specialized datasets, and is reported to advance the state of the art for open small language models.
Problem
Large language models’ computational cost makes deployment difficult in resource-constrained settings.
Method
SmolLM2 combines dataset evaluation, multi-stage training, manual source rebalancing, and newly created specialized datasets.
Results
SmolLM2 advances the state of the art for open small language models through careful dataset curation and multi-stage training.
Takeaways & Limitations
The released model, datasets, and training code provide resources for research and applications involving performant small language models.
Takeaways & Limitations
A noticeable loss spike during dataset integration remained unexplained, although most evaluation metrics recovered by the end of the stage.
Abstract
from arXiv · showhide
While large language models have facilitated breakthroughs in many applications of artificial intelligence, their inherent largeness makes them computationally expensive and challenging to deploy in resource-constrained settings. In this paper, we document the development of SmolLM2, a state-of-the-art "small" (1.7 billion parameter) language model (LM). To attain strong performance, we overtrain SmolLM2 on ~11 trillion tokens of data using a multi-stage training process that mixes web text with specialized math, code, and instruction-following data. We additionally introduce new specialized datasets (FineMath, Stack-Edu, and SmolTalk) at stages where we found existing datasets to be problematically small or low-quality. To inform our design decisions, we perform both small-scale ablations as well as a manual refinement process that updates the dataset mixing rates at each stage based on the performance at the previous stage. Ultimately, we demonstrate that SmolLM2 outperforms other recent small LMs including Qwen2.5-1.5B and Llama3.2-1B. To facilitate future research on LM development as well as applications of small LMs, we release both SmolLM2 as well as all of the datasets we prepared in the course of this project.
1. Introduction
SmolLM2 targets resource-constrained deployment by combining small-model efficiency with careful, specialized data curation. The paper evaluates datasets and manually rebalances sources across training stages, reporting state-of-the-art performance among similarly sized models.
- Small language models reduce computational costs and can run on more devices while retaining satisfactory performance on important tasks.
- Data curation is especially influential for small models because limited capacity must prioritize core knowledge and capabilities over incidental memorization.
- The paper evaluates web, code, math, and instruction-following datasets to guide SmolLM2’s training-data design.
- SmolLM2 uses multi-stage manual rebalancing of data sources, alongside newly created FineMath, Stack-Edu, and SmolTalk datasets.The datasets address cases where existing data were too small or low-quality.
- Both base and instruction-tuned SmolLM2 variants are reported as state-of-the-art among similarly sized models.
2. Background
The background frames language-model development as a sequence from large-scale pretraining through instruction tuning and preference learning. It emphasizes that web data require quality control and that specialized data can improve reasoning and broader capabilities.
- Pretraining uses trillions of unstructured-text tokens to model language structure and store factual knowledge.
- Raw web scrapes can produce poorly performing language models, motivating large-scale filtering and curation.
- Specialized code and mathematics data can improve performance on reasoning and world-knowledge tasks beyond their source domains.
- Instruction tuning trains models on instruction–response pairs so they answer queries helpfully rather than merely continuing input text.
- Preference learning further aligns models by training them to distinguish helpful from unhelpful responses, typically using reinforcement learning.
3. Pretraining datasets
The pretraining-data study compares web, math, and code sources through controlled ablations and develops larger, more targeted datasets where existing sources are insufficient. FineMath notably improves mathematical benchmark performance over prior datasets.
- Motivation: Pretraining data curation and source mixing are especially important for small models, which are more sensitive to training-data noise.
- Ablation setup: Dataset ablations train identical 1.7B-parameter models on controlled token budgets to compare source quality and mixtures.The setup uses 350B tokens for English web-dataset ablations.
- Web data: A 60% FineWeb-Edu and 40% DCLM mixture nearly matches FineWeb-Edu on MMLU, ARC, and OpenBookQA while aligning with DCLM on HellaSwag.The mixture provides 5.1T tokens of English text.
- Math data: InfiMM-WebMath reaches 14% peak GSM8K accuracy versus 10% for OpenWebMath, while OpenWebMath slightly outperforms it on MATH.Both datasets showed limitations involving size, reasoning focus, and overrepresented advanced academic content.
- Math data: FineMath was created as a dataset of up to 54B tokens focused on mathematical deduction and reasoning through classifier-based filtering.
- Math data: All FineMath subsets outperform OpenWebMath and InfiMM-WebMath across GSM8K, MATH, and MMLU-STEM; FineMath4+ doubles GSM8K and sextuples MATH performance versus InfiMM-WebMath.
4. Pretraining
SmolLM2 uses 11 trillion tokens and a multi-stage pretraining strategy that manually rebalances web, code, and math data in response to observed capability gaps. The staged interventions improved coding and mathematics, while final evaluation showed competitive generalization against similarly sized base models.
- Training strategy: 11 trillion tokens were used across four pretraining phases, with dataset mixtures adjusted between stages rather than fixed throughout training.The training used approximately two epochs on the collected datasets, and the four phase mixtures are documented in Figure 2.
- Stage 1: After 6T tokens, coding and mathematics performance were generally poor despite knowledge and reasoning matching expectations.This evaluation motivated later mixture changes targeting code and math.
- Stage 2: Stage 2 increased code data to 20% and added 5% math data, improving code performance across most languages.OWM was added at 5%, but it had no significant effect on mathematics performance.
- Stage 3: Stage 3 replaced StarCoderData with Stack-Edu, raised math data to approximately 10%, and shifted the FineWeb-Edu/DCLM ratio to 40/60.Jupyter Notebooks were also added to provide contextual code examples and explanations.
- Stage 4: Stage 4 introduced the highest-quality mathematical datasets during learning-rate decay, producing substantial gains in coding and especially mathematics.All benchmark tasks improved after this stage.
- Base-model evaluation: SmolLM2 outperformed Qwen2.5-1.5B on HellaSwag and ARC, exceeded it by nearly 6 percentage points on MMLU-Pro, and outperformed Llama3.2-1B on GSM8K, MATH, and HumanEval.It remained competitive on math and coding benchmarks and showed next to no degradation after context-length extension.
5. Post-training
SmolLM2’s post-training combines supervised instruction tuning with preference learning, supported by newly constructed conversational and task-specific datasets. The resulting instruct model shows strong instruction following and competitive performance across conversation, rewriting, and mathematics benchmarks.
- Post-training: Post-training combines supervised instruction tuning and preference learning, using existing datasets plus the new SmolTalk dataset.Preference learning uses Direct Preference Optimization.
- Conversational data: MagPie-Ultra contains 1M filtered three-turn conversations generated with Llama-3.1-405B-Instruct-FP8 and quality-scored and deduplicated for training use.Smaller Llama models provided quality and safety filtering, while ArmoRM and gte-large-env1.5 supported filtering and deduplication.
- Conversational data: MagPie-Ultra outperformed MagPie-Pro on most evaluated benchmarks and largely surpassed OpenHermes2.5 and UltraChat on IFEval and MT-Bench.The evaluation covered instruction following, conversation, reasoning, knowledge, and mathematics.
- Specialized instruction data: SmolTalk includes task-specific datasets for detailed constraints, summarization, rewriting, code generation, system prompts, function calling, and long-context tasks.These datasets were constructed or selected to broaden instruction-following capabilities.
- Preference learning: UltraFeedback was the most consistently effective preference dataset, improving MT-Bench, MMLU-Pro, and MATH.The model was trained with Direct Preference Optimization for two epochs.
- Instruct-model evaluation: SmolLM2-Instruct strongly outperformed Qwen2.5-1.5B-Instruct on IFEval and remained competitive on MT-Bench, OpenRewrite-Eval, GSM8K, and MATH.The results support performance across instruction following, rewriting, and mathematical capabilities.
6. SmolLM2 135M and 360M
The project also trains 360M- and 135M-parameter SmolLM2 models with reduced token budgets. These smaller models use a single-stage, consistently high-quality data strategy rather than the 1.7B model’s multi-stage approach.
- Model scale: SmolLM2-360M was trained on 4T tokens and SmolLM2-135M on 2T tokens, with both described as state-of-the-art for their size classes.The models contain 360M and 135M parameters, respectively.
- Training strategy: Unlike SmolLM2-1.7B, the smaller models benefited from single-stage training with consistently high-quality data.Their mixtures included Stack-Edu, InfiMM-WebMath, FineMath, and Cosmopedia from the start.
7. Conclusion
SmolLM2 advances open small language models through careful dataset curation and multi-stage training, with new datasets addressing shortcomings in existing resources.
- SmolLM2 advances the state-of-the-art for open small LMs through careful dataset curation and multi-stage training.The paper introduces FineMath, Stack-Edu, and SmolTalk to improve reasoning, mathematics, and instruction-following capabilities.
A. Training setup
The training setup uses a 1.7B-parameter architecture with WSD scheduling and evaluates alternative web-data mixtures during training.
- SmolLM2 uses a 1.7B architecture, with Table 6 documenting its model configuration before context extension.
- The WSD scheduler governs learning-rate progression during SmolLM2 training.The setup includes 2000 warmup steps and a learning rate of 5.0 × 10−4.
- A 60/40 FineWeb-Edu/DCLM mixture achieves balanced performance across tasks after 350B-token training.FineWeb-Edu excels at knowledge and reasoning, whereas DCLM is stronger on commonsense reasoning benchmarks.
C.1. Public datasets comparison
The section describes dataset-quality filtering and annotation procedures for FineMath and Stack-Edu, alongside ablations showing that existing math data remained insufficient for strong performance. FineMath uses Llama3-based educational scoring, while Stack-Edu applies language-specific educational filtering across programming languages.
- Math-data ablations: OWM and InfiMM-WebMath both remain far below state-of-the-art LLM performance after training on 60B math tokens, motivating a new math dataset.InfiMM-WebMath consistently outperforms OWM on GSM8K, whereas OWM has a slight advantage on MATH.
- FineMath filtering: FineMath assigns educational scores using Llama3 prompts that evaluate mathematical content, reasoning, appropriate difficulty, and step-by-step deductions.The 3-point and 5-point schemes accumulate points for increasingly useful mathematical and educational properties.
- FineMath filtering: FineMath’s scoring prompts accept question-answer formats and instruct evaluators to judge extracts by overall content despite formatting errors or missing equations.The prompts also require a brief justification and a standardized final score.
- Stack-Edu filtering: Stack-Edu evaluates Python programs with a 5-point additive rubric covering valid code, practical concepts, educational suitability, self-contained exercises, and exceptional teaching value.The rubric emphasizes well-structured code, comments, tutorials, course sections, and school-exercise-like material.
- Stack-Edu filtering: Stack-Edu uses analogous prompts for 14 additional programming languages, adapting examples to language-specific topics such as asynchronous programming in JavaScript.HTML was initially included, but poor classifier performance led the authors to retain StarCoder2Data instead.
E.1. Evaluation after each training stage
Evaluation is tracked across training stages using benchmark tables and MMLU curves, with scores rising during the stable phase and MMLU CF later plateauing.
- Table 8 reports per-benchmark SmolLM2 performance at the end of each training stage.The evaluation includes additional generative tasks such as CoQA, DROP, Jeopardy, and SQuAD v2.
- MMLU scores are tracked throughout the stable phase using progression curves.
- After 6T tokens, MMLU CF appears to plateau.
F. Post-training
SmolTalk is a 1.1M-pair instruction dataset assembled from multiple data sources. The post-training analysis compares instruction datasets and specialized-math mixtures before defining the final supervised fine-tuning checkpoint.
- 1.1M instruction-response pairs comprise the SmolTalk dataset, drawn from different data sources.
- Table 10 compares fine-tuning SmolLM2 base on different instruction datasets and evaluates adding 20% specialized math data to an 80% MagPie-Ultra+ mixture.
- All instruction-dataset comparisons fine-tune SmolLM2 base for 1 epoch, while SmolLM2-SFT is trained for 2 epochs on SmolTalk.
- MagPie-Ultra+ combines MagPie-Ultra with Smol-Constraints, Smol-Rewrite, and Smol-Summarization; MagPie-Pro-MT is multi-turn and MagPie-Pro is single-turn.
G. Long context evaluations
The paper evaluates SmolLM2's long-context behavior with the Needle in the Haystack and HELMET benchmarks. These evaluations use an 8192 context length for Needle in the Haystack and an 8k maximum input length for HELMET.
- Figure 7 reports SmolLM2's evaluation results on the Needle in the Haystack benchmark.
- Needle in the Haystack evaluates SmolLM2 with an 8192 context length.
- Table 11 reports evaluation results on the HELMET benchmark.
- The HELMET evaluation uses an 8k maximum input length for base models.