Source-linked AI summary
EuroLLM: Multilingual Language Models for Europe
Pedro Henrique Martins, Patrick Fernandes, João Alves, Nuno M. Guerreiro, Ricardo Rei, Duarte M. Alves, José Pombal, Amin Farajian, Manuel Faysse, Mateusz Klimaszewski, Pierre Colombo, Barry Haddow, José G. C. de Souza, Alexandra Birch, André F. T. Martins
TL;DR
Open-weight LLMs remain predominantly focused on English, while many European languages are underserved. EuroLLM develops multilingual models through data and tokenizer design, training, and instruction tuning; its initial models show multilingual benchmark and translation performance, including strong comparisons with the cited baselines.
Problem
Open-weight LLMs are predominantly limited to English and a few high-resource languages, leaving many European languages out.
Source: §1 · p. 1Method
EuroLLM develops multilingual models using collected and filtered data, a multilingual tokenizer, pre-training, and instruction fine-tuning.
Source: §1 · pp. 1–2, §2.1 · p. 3Results
EuroLLM-1.7B matches or outperforms Gemma-2b and TinyLlama on translated Hellaswag outside English, while EuroLLM-1.7B-Instruct outperforms Gemma-2b-Instruct across all evaluated language pairs and datasets and is competitive with Gemma-7b-Instruct.
Source: §6.1 · p. 7, §6.2 · p. 8Takeaways & Limitations
Abstract
from arXiv · showhide
The quality of open-weight LLMs has seen significant improvement, yet they remain predominantly focused on English. In this paper, we introduce the EuroLLM project, aimed at developing a suite of open-weight multilingual LLMs capable of understanding and generating text in all official European Union languages, as well as several additional relevant languages. We outline the progress made to date, detailing our data collection and filtering process, the development of scaling laws, the creation of our multilingual tokenizer, and the data mix and modeling configurations. Additionally, we release our initial models: EuroLLM-1.7B and EuroLLM-1.7B-Instruct and report their performance on multilingual general benchmarks and machine translation.
1 INTRODUCTION
Open-weight LLMs remain predominantly English-focused, leaving many European languages underserved. EuroLLM is developing multilingual models for all EU official languages and additional relevant languages, with initial models and a pipeline spanning data, tokenization, training, instruction tuning, and evaluation.
1 INTRODUCTION
- Open-weight LLMs are predominantly limited to English and a few high-resource languages, leaving many European languages out.EuroLLM targets all official EU languages and additional relevant languages.Source: §1 · p. 1
- The project explored methodologies for training multilingual LLMs and developed initial models EuroLLM-1.7B and EuroLLM-1.7B-Instruct, involving data collection and filtering, corpus-mixture choices, tokenizer development, pre-training, and instruction fine-tuning.Source: §1 · pp. 1–2
- The project reports that it evaluates model performance on multilingual benchmarks and machine translation.Evaluation is identified as part of the project’s work, with results reported in §6.Source: §1 · p. 2
2 DATA
EuroLLM builds its corpus from web, parallel, code/math, and high-quality data, then uses scaling-law experiments and data availability to guide the mixture. The section describes collection and filtering, parallel-data and repetition choices, and language allocations across training phases.
2.1 DATA COLLECTION AND FILTERING
- The corpus combines web, parallel, code/math, and high-quality data, with Figure 1 showing their shares in the first and annealing phases.The collection process includes filtering web data and quality-filtering parallel sentence pairs.Source: §2 · p. 2, Figure 1 · p. 2, §2.1 · p. 2
- Code/math sources include The Stack, Algebraic-stack, and Open-web-math; high-quality data includes Wikipedia for all languages and Arxiv, Books, and Apollo for English.Annealing adds educationally filtered monolingual data and additional high-quality sources, including document-level parallel data from Europarl and ParaDocs.Source: §2.1 · pp. 2–3
2.2.1 PARALLEL DATA
- Adding parallel data improved parallel-domain performance without reducing web or Wikipedia performance, while increasing its share from 25% to 37.5% gave diminishing returns.The authors consequently use 20% parallel data for each language in the final corpus; Figure 2 presents the scaling laws for the parallel-data proportions.Source: Figure 2 · p. 3, §2.2.1 · p. 3
2.2 DATA MIXTURE
- The mixture design considers the amount of parallel data, whether to repeat high-quality data, and how to allocate tokens across languages.The authors base these choices on scaling laws and data availability for each language.Source: §2.2 · p. 3
2.2.2 REPEATING HIGH QUALITY DATA
- Repeating Wikipedia improved Wikipedia test performance without degrading web test performance, leading the authors to repeat high-quality data.Figure 3 presents scaling laws for web and Wikipedia test sets with and without repeated Wikipedia.Source: Figure 3 · p. 4, §2.2.2 · p. 4
2.2.3 DIVISION BETWEEN LANGUAGES
- English receives 50% of first-phase tokens and 32.5% during annealing, while code/math rises from 5% to 7%.The remaining first-phase 45% is distributed across other languages according to data obtained after collection and filtering; annealing redistributes the English share across those languages.Source: Figure 4 · p. 4, §2.2.3 · p. 4
3 TOKENIZER
EuroLLM uses a multilingual BPE tokenizer with byte-fallback and a 128,000-piece vocabulary selected as a trade-off. Its fertility comparisons show different tokenizer strengths across English, other European languages, and Asian languages.
3 TOKENIZER
- A 128,000-piece vocabulary provides the authors’ best trade-off for the multilingual tokenizer.The tokenizer uses BPE with byte-fallback and is trained with SentencePiece; a larger vocabulary supports efficiency across many languages but increases embedding parameters.Source: §3 · pp. 4–5
- Compared with Mistral, EuroLLM’s larger vocabulary yields significantly lower fertility; LLaMa-3 has the lowest English fertility but higher fertility for most other languages.Gemma seems better for Asian languages and has fertility very similar to EuroLLM for European languages.Source: §3 · p. 5, Figure 5 · p. 5
4 MODELING
EuroLLM uses a dense Transformer and pretrains on 4 trillion tokens; its scheduler comparison favors the trapezoid option across multilingual benchmarks and machine translation.
4 MODELING
- EuroLLM uses a standard dense Transformer with grouped-query attention, pre-layer normalization, RMSNorm, SwiGLU, and rotary positional embeddings.The design uses 8 key-value heads for grouped-query attention.Source: §4 · p. 5
- EuroLLM-1.7B is pretrained on 4 trillion tokens, with a higher proportion of high-quality data during the final 10%.Training uses 256 Nvidia H100 GPUs and a constant batch size of 3,072 sequences, or approximately 12 million tokens.Source: §4 · p. 5
4.1 LEARNING RATE SCHEDULER
- The trapezoid scheduler scores consistently better than cosine on multilingual benchmarks and machine translation.It warms up for 10% of steps, holds a constant rate, then decays during the final 10%, which uses higher-quality annealing data.Source: §4.1 · p. 6, Table 2 · p. 6
5 POST TRAINING
EuroBlocks provides multilingual instruction and translation examples for supervised fine-tuning EuroLLM-1.7B into the conversational EuroLLM-1.7B-Instruct model.
5 POST TRAINING
- EuroBlocks contains 1 million human-written and synthetic samples across all supported languages and a variety of tasks.Sources include OpenHermes-2.5 and Aya instruction conversations, plus machine-translation examples from NTREX-128, FLORES-200-DEV, WMT-21, and WMT-22.Source: §5 · p. 6
- Supervised fine-tuning on EuroBlocks turns EuroLLM-1.7B into the instruction-following conversational EuroLLM-1.7B-Instruct.Training uses target-token cross-entropy loss for 4 epochs, at a learning rate of 7 · 10−6 over around 6K steps.Source: §5 · p. 6
- Chat control tokens separate user and model turns, and the template may include a system prompt for added functionality and steerability.The tokens are included in the original tokenizer, so fine-tuning adds none and instead overwrites the end-of-sequence token with <|im end|>.Source: §5 · p. 6
6 RESULTS
EuroLLM-1.7B-Instruct outperforms Gemma-2b-Instruct across the reported machine-translation language pairs and datasets, while general-benchmark results vary by benchmark and language.
6 RESULTS
- General benchmarks assess whether EuroLLM-1.7B acquired world knowledge, while machine translation assesses its ability to understand and generate text in supported languages.Source: §6 · p. 6
- The evaluation uses Hellaswag for commonsense natural-language inference and Arc Challenge for science-exam questions; translation compares EuroLLM-1.7B-Instruct with Gemma-2b and Gemma-7b on three datasets, including FLORES-200-TEST and WMT-23.Source: §6.1 · p. 6, §6.2 · p. 7
6.1 GENERAL BENCHMARKS
- On Hellaswag, EuroLLM-1.7B matches or beats Gemma-2b and TinyLlama outside English; on Arc Challenge, it beats TinyLlama but trails Gemma-2b.EuroLLM-1.7B-Instruct is slightly better than EuroLLM-1.7B on both benchmarks.Source: §6.1 · p. 7, Figure 6 · p. 7
6.2 MACHINE TRANSLATION
- EuroLLM-1.7B-Instruct clearly outperforms Gemma-2b-Instruct on all language pairs and datasets, and is competitive with Gemma-7b-Instruct despite fewer parameters.The comparisons use COMET-22 on FLORES-200-TEST, WMT-23, and WMT-24, with the Gemma models also fine-tuned on EuroBlocks.Source: §6.2 · p. 8, Figure 7 · p. 8, Figure 8 · p. 8
7 CONCLUSIONS AND FUTURE WORK
The paper reports EuroLLM project progress, releases its initial models, and identifies scaling model parameters and improving data quality as future work.
7 CONCLUSIONS AND FUTURE WORK
- The project reports data collection and filtering, multilingual tokenizer development, data-mixture and modeling configurations, and releases EuroLLM-1.7B and EuroLLM-1.7B-Instruct.It also reports performance on multilingual general benchmarks and machine translation.Source: §7 · p. 8
- Future work will continue training multilingual LLMs for Europe, scaling model parameters and further improving data quality.Source: §7 · p. 8