Source-linked AI summary
mmBERT: A Modern Multilingual Encoder with Annealed Language Learning
Marc Marone, Orion Weller, William Fleshman, Eugene Yang, Dawn Lawrie, Benjamin Van Durme
TL;DR
Recent large-scale multilingual encoder pretraining has lagged despite encoder-only models remaining important for classification and retrieval. The paper introduces mmBERT, trained on 3T multilingual tokens with staged language expansion, annealed sampling, and inverse masking. It improves over XLM-R across benchmarks and surpasses larger language models on some low-resource evaluations.
Problem
Recent multilingual encoder pretraining has lagged, with XLM-R still state of the art despite encoder-only models remaining important for classification, clustering, and retrieval.
Method
MMBERT is pretrained on 3T multilingual tokens using inverse masking, annealed language sampling, and language expansion from 60 to 110 to 1833 languages.
Results
MMBERT improves over XLM-R across the board, including 72.8 versus 70.4 average XTREME performance and 77.1 versus 74.6 XNLI accuracy.
Takeaways & Limitations
Adding low-resource languages only during decay enables rapid learning from limited data and reaches performance beyond OpenAI’s o3 and Google’s Gemini 2.5 Pro on some evaluations.
Takeaways & Limitations
Many languages still have very little or no data, especially high-quality filtered data, limiting further low-resource-language improvements.
Abstract
from arXiv · showhide
Encoder-only languages models are frequently used for a variety of standard machine learning tasks, including classification and retrieval. However, there has been a lack of recent research for encoder models, especially with respect to multilingual models. We introduce mmBERT, an encoder-only language model pretrained on 3T tokens of multilingual text in over 1800 languages. To build mmBERT we introduce several novel elements, including an inverse mask ratio schedule and an inverse temperature sampling ratio. We add over 1700 low-resource languages to the data mix only during the decay phase, showing that it boosts performance dramatically and maximizes the gains from the relatively small amount of training data. Despite only including these low-resource languages in the short decay phase we achieve similar classification performance to models like OpenAI's o3 and Google's Gemini 2.5 Pro. Overall, we show that mmBERT significantly outperforms the previous generation of models on classification and retrieval tasks -- on both high and low-resource languages.
1 Introduction
mmBERT addresses the lack of recent large-scale multilingual encoder pretraining by combining a 3T-token model with an annealed multilingual training recipe. It improves over XLM-R broadly and reaches beyond larger decoder-only models on low-resource languages.
- Encoder-only models remain widely used for classification, clustering, and retrieval despite decoder-only models dominating recent language-model development.
- XLM-R remained state of the art in massively multilingual encoder modeling, motivating a newer improved model.
- MMBERT is pretrained on 3T multilingual tokens using inverse masking, annealed language sampling, and staged expansion from 60 to 110 to 1833 languages.
- MMBERT improves over XLM-R across tasks and beats OpenAI’s o3 and Google’s Gemini 2.5 Pro on low-resource languages.
- Adding low-resource languages during decay boosts their performance roughly 2x despite using only 100B tokens for that phase.
2 Related Work
Related work established encoder-only models for efficient language understanding, while recent scaling largely favored decoder-only systems. Available multilingual encoders remain limited in language coverage and modernization.
- Encoder-only models are still predominant for classification and retrieval when inference speed matters, whereas decoder-only models have scaled to trillion-parameter sizes.
- Modern encoder research revived efficient pretraining, with ModernBERT improving performance and newer models targeting English or limited language sets.
- Massively multilingual encoder options include mBERT with 104 languages, XLM-R with 100, and mGTE with 74.
- Multilingual language models increasingly use decoder-only architectures, but many releases do not disclose their pretraining data or training recipes.
3 Training Details
MMBERT uses a ModernBERT-inspired architecture, public multilingual and curated English data, and staged training that progressively increases language coverage while lowering masking and learning rates. Its decay variants are merged to combine strengths across language mixtures.
- Architecture: MMBERT uses a ModernBERT architecture with the Gemma 2 tokenizer and 307M total parameters for base versus 140M for small.
- Training Data: Training uses 2.3T pretraining tokens, 600B mid-training tokens, and 100B decay tokens, with later stages using higher-quality data.
- Training Data: The data mix combines multilingual FineWeb2 and FineWeb2-HQ with curated English corpora from Dolma, Dolmino, and ProLong.
- Training Data: Cascading Annealed Language Learning changes sampling temperature and iteratively adds languages, moving from 60 to 110 to 1833 languages with temperatures 0.7 to 0.5 to 0.3.
- Training Recipe: The inverse masking schedule progressively lowers mask rates across training stages, reaching 5% during the 100B-token decay phase.
- Training Recipe: Decay variants focus on English, 110 languages, or all 1833 languages, after which model merging combines selected checkpoints for the base model.
4 Results
MMBERT improves multilingual encoder performance across NLU and retrieval benchmarks, while extending context length and substantially increasing throughput. Its strongest gains appear in multilingual and low-resource settings, though code retrieval remains behind EuroBERT.
- NLU: 84.7 average GLUE score makes MMBERT small stronger than MiniLM’s 78.3, while MMBERT base reaches 86.3 versus ModernBERT’s 87.4.MMBERT base outperforms the other multilingual models despite using a majority of non-English data.
- NLU: MMBERT base leads multilingual XTREME overall, averaging 72.8 versus XLM-R’s 70.4, with especially strong classification and question-answering results.It reaches 77.1 XNLI accuracy versus 74.6 for XLM-R and 74.5 TyDiQA F1 versus 70.5.
- Retrieval: 54.1 average multilingual MTEB score for MMBERT base exceeds XLM-R’s 52.4, with both MMBERT sizes outperforming similarly sized counterparts.On English MTEB, MMBERT averages 53.9 versus mGTE’s 52.7 and performs similarly to ModernBERT’s 53.8.
- Retrieval: 42.2 CoIR average for MMBERT base exceeds XLM-R’s 33.6 but trails EuroBERT’s 45.3.The paper attributes EuroBERT’s advantage likely to its higher-quality Stack v2 training corpus, which was unavailable for MMBERT.
- Efficiency: MMBERT base is more than 2x faster on variable sequences and roughly 4x faster at long contexts, supporting sequences up to 8192 tokens.Previous multilingual models such as MiniLM and XLM-R cannot exceed 512 tokens, whereas MMBERT maintains comparable speed at that length.
5 Conclusion
MMBERT is a modern multilingual encoder trained on 3T tokens and 1833 languages, with training innovations that improve over XLM-R and rapidly learn low-resource languages during decay.
- MMBERT is trained on 3T tokens spanning 1833 languages.
- Its training introduces inverse masking and cascading annealed language-learning schedules for multilingual data.
- MMBERT improves over XLM-R as a drop-in replacement across multilingual encoder evaluations.
- Adding low-resource languages only during decay rapidly raises their performance to levels beyond OpenAI’s o3 and Google’s Gemini 2.5 Pro.
6 Limitations
Despite improvements on low-resource languages, many languages still have very little or no data, particularly high-quality edu-style filtered data. The authors leave this gap to future work.
- Many languages still have very small amounts of data or none at all.This limitation remains despite mmBERT’s significant improvements on low-resource languages.
- High-quality data, including edu-style filtered data, is especially limited for these languages.
- Future work targeting data availability could further improve low-resource language scores.
A Architecture Details
MMBERT’s configurations differ across model sizes and include distinct warmup settings. The small model also required a later reduction in learning rate and weight decay after early plateauing.
- Warmup settings: MMBERT small used 4B-token learning-rate warmup and 100B-token batch-size warmup.
- Warmup settings: MMBERT base used 3B-token learning-rate warmup and 60B-token batch-size warmup.
- Training adjustment: MMBERT small reduced its learning rate and weight decay to half the initial value after 1.2T tokens because training plateaued early.
B Hyperparameter and Compute Details
The paper reports substantial training compute and task-specific hyperparameter sweeps for NLU, embedding, FoQA, and TiQuAD evaluations. These procedures select settings using task or benchmark performance.
- Compute: Training used 8×H100s for roughly 10 days for the small model and 40 days for the base model.L40s were used mainly for smaller-model inference, and each evaluation experiment took roughly 1–2 hours per setting.
- NLU sweep: NLU evaluation swept seven learning rates and four epoch options with batch size 32 and warmup ratio 0.06.The best result was selected for each model and task in an oracle fashion.
- Embedding sweep: Embedding evaluation swept four learning rates, trained on 1.25M hard triplets for one epoch, and selected the best rate using MTEB.The best learning rate was 1e-4 in all cases except MiniLM, according to the passage.
- FoQA and TiQuAD: FoQA and TiQuAD used a partial sweep over the two best NLU learning rates, 2e-5 and 3e-5, while varying epochs.
C Comparison with DeBERTa
RTD-trained models are excluded from the main analysis because their embedding performance is substantially worse. On multilingual MTEB, MMBERT outperforms mDeBERTa by more than 11 points.
- RTD-trained models are excluded from the main analysis because comparable models show significantly worse embedding-task performance.The exclusion applies to Ettin and ModernBERT.
- 54.1 on multilingual MTEB, compared with mDeBERTa’s 42.5 average, gives MMBERT a more-than-11-point advantage.
- 48.6 on English MTEB v2 leaves mDeBERTa below every comparable model, including MMBERT small.
- RTD-trained models may perform well on classification at the expense of embedding tasks, which are a main use case for encoder-only models.
D Language Distributions
The language-distribution data uses inverse temperature sampling and reports stage-specific values and percentages for the first 90 included languages. The complete dataset is provided separately in CSV form.
- 0.7, 0.5, and 0.3 are the inverse temperature sampling values used for language data.
- The full language-distribution data is available as a CSV file on GitHub because more than 90 rows would not fit on the page.
- The table organizes language data across Pretrain, Mid-Training, Decay-Eng, Decay-Cont, and Decay-All stages.
- Language rows report stage values, percentages, and totals, with the displayed table limited to the first 90 included languages.