Source-linked AI summary
Scaling Multilingual Pretraining of Pixel Language Models
Ilker Kesen, Jenny Kunz, Kasra Malihi, Desmond Elliott
TL;DR
The paper examines whether larger-scale pretraining improves pixel language models’ cross-lingual transfer, a capability whose scaling behavior is not well understood. It trains MPIXEL models varying languages, scripts, and capacity under fixed per-language compute, finding that gains are localized and that increased capacity mitigates multilinguality costs.
Problem
How increasing writing-system diversity, pretraining languages, or model capacity affects pixel language models’ cross-lingual transfer, including transfer to unseen scripts, remains an open question.
Source: §1 · pp. 1–2Method
The paper trains MPIXEL models with up to sixteen languages and 307M parameters, varying language and script coverage and capacity under fixed per-language compute.
Source: §1 · pp. 1–2, §7 · p. 8Results
Takeaways & Limitations
The findings characterize multilingual scaling in pixel models as localized: added languages and scripts benefit most, while unseen-script transfer does not continue improving with scale.
Source: §7 · p. 8The scaling claims are limited to four language counts and two model sizes, with one pretraining run per variant and less pretraining data than XLM-R.
Source: Limitations · p. 9
Abstract
from arXiv · showhide
Pixel language models process text as rendered images instead of a finite vocabulary, which gives them strong cross-lingual transfer abilities. However, not much is known about how these abilities are influenced by larger-scale pretraining. In this paper, we train a family of MPIXEL models along three axes of variation: more languages, more scripts, and larger model capacity under fixed per-language compute. In evaluations on five tasks covering 83 languages, we find that scaling acts locally: each axis improves the added languages and scripts, while increased capacity mitigates the curse of multilinguality from pretraining on more languages. Transfer to unseen scripts improves with multilingual pretraining, but does not further improve with scaling. On typologically distant low-resource languages, our models outperform subword and character-level baselines across many writing systems. We also show that larger pixel models reserve more layers for visual processing, and that cross-lingual alignment weakens as more languages are added. Overall, our findings characterize how multilingual scaling improves pixel language models.
1 Introduction
The paper asks whether multilingual pretraining and model-capacity scaling improve pixel models’ cross-lingual transfer, including transfer to unseen scripts. Its controlled study finds that benefits are largely localized to newly covered languages and scripts, while greater capacity mitigates multilinguality costs.
1 Introduction
- The study scales MPIXEL along language count, script diversity, and capacity, evaluating five tasks across 83 languages.The models include up to sixteen pretraining languages and 307M parameters.Source: §1 · pp. 1–2
- Adding languages and scripts mainly benefits newly covered languages and scripts, while transfer to unseen scripts does not improve with further scaling.On unseen scripts, additional languages and model capacity yield no consistent gains.Source: Figure 1 · p. 1, §1 · p. 2
- Increased model capacity mitigates the curse of multilinguality that pixel models experience as pretraining languages are added.Source: §1 · p. 2
- MPIXEL outperforms subword and character-level baselines on low-resource languages typologically distant from baseline pretraining languages.The introduction also reports this advantage for languages in multiple writing systems.Source: §1 · p. 2
- The analysis finds that larger models reserve more layers for visual processing, while cross-lingual alignment weakens as languages are added.Source: §1 · p. 2
2 Pretraining Setup
The pretraining setup varies language diversity, script diversity, and model capacity while controlling exposure per language. It also adapts text rendering to script-specific character and word structure.
2 Pretraining Setup
- The model variants compare language diversity within a seen script, the addition of unseen scripts, both data expansions together, and increased capacity.Matched language-count expansions distinguish script diversity from adding languages alone.Source: §2 · p. 2
- Pretraining language sets extend a four-language base with Latin-script languages, languages in unseen scripts, or both expansions plus four further languages.The added languages are selected to diversify language families and writing systems.Source: §2 · pp. 2–3
- Rendering groups graphemes into bigrams by default, uses one character per patch for CJK, and renders Arabic and Hebrew words contiguously in logical order.These choices address separated diacritics, multi-patch wide characters, and disrupted ligatures or word structure.Source: §2 · p. 3
- Each language receives about 7.5 billion patches, and batch size increases with language count to preserve roughly 4.3 epochs of per-language exposure.The eight- and sixteen-language variants use doubled and quadrupled effective batch sizes, respectively.Source: §2 · p. 3
3 Experimental Setup
The experiments compare MPIXEL with subword, character-level, and pixel-model baselines across transfer settings. Evaluation spans multilingual classification and structured prediction, alongside English GLUE evaluation of multilinguality effects.
3 Experimental Setup
- Baselines include XLM-R and mBERT with subword tokenizers, CANINE with character-level hash embeddings, and PIXEL† and MPIXEL4S pixel models.The authors pretrained PIXEL† and MPIXEL4S to control for data quality, data size, and rendering differences.Source: §3 · p. 3
- Evaluation covers SIB-200 sentence classification in 77 languages, dependency parsing, named entity recognition in 12 languages, XNLI, and English GLUE.Transfer settings include pretrained languages, unseen languages in seen scripts, and languages in unseen scripts.Source: §3 · p. 3
4 Results
MPIXEL transfers well to typologically distant low-resource languages, while multilingual scaling mainly benefits the languages and scripts added. More capacity improves performance within pretraining languages and scripts and mitigates the performance loss from multilingual pretraining, but effects on unseen scripts vary by task.
Baseline comparisons
- MPIXEL outperforms CANINE-S on every task except XNLI and scores better than XLM-R on unseen scripts in UDP, largely driven by Coptic.MPIXEL also outperforms CANINE-S on Latin and Bengali; XLM-R performs close to random on four scripts unseen by all models.Source: §4 · p. 4
Low-resource cross-lingual transfer
- MPIXEL outperforms XLM-R on typologically distant low-resource languages across all writing systems in SIB-200.The evaluation selects languages typologically distant from every model’s pretraining data.Source: §4 · p. 4, Figure 2 · p. 4
Language and script scaling
- Adding four languages in seen or unseen scripts mainly improves the added languages and other languages sharing their scripts, with no consistent advantage on unseen scripts.The eight-language variants show smaller gains on seen-script SIB-200 and UDP groups and mixed benefits overall; NER and XNLI have too few seen-script languages for reliable comparison.Source: §4 · p. 4
- The 16-language model improves overall averages on SIB-200, NER, and UDP, with gains concentrated on seen languages and task-dependent effects on unseen scripts.Unseen scripts improve with each extension on UDP, where MPIXEL outperforms all other models including XLM-R, but MPIXEL has the lowest score among multilingual variants on SIB-200.Source: §4 · p. 5
Capacity scaling
- Increasing capacity improves seen-language performance on every task, while its effects on unseen scripts are mixed across tasks.MPIXELLARGE loses the smaller multilingual variants’ UDP advantage on unseen scripts and does not exceed MPIXEL4S on SIB-200.Source: §4 · p. 5
Curse of multilinguality
- On the four shared pretraining languages, performance declines across all five benchmarks as languages are added, but MPIXELLARGE recovers the losses and achieves the best result on every benchmark.MPIXEL8L and MPIXEL8S hurt the initial languages to a similar extent; increased capacity therefore mitigates the curse of multilinguality in these comparisons.Source: §4 · pp. 5–6
5 Further Analysis
Probing reveals that adding languages and scripts weakens cross-lingual semantic alignment, while larger models extend visual processing into deeper layers. Linguistic information emerges with depth, though performance and alignment patterns vary by task and language.
5.1 ABX-style Discrimination
- Meaning Discrimination rises with depth and peaks in the final layer, while Language Discrimination drops sharply from the embedding layer before partially recovering in middle layers.The results indicate increasing semantic alignment at higher layers and diminishing reliance on strong script-level cues after the embedding layer.Source: §5.1 · p. 6
- Final-layer semantic alignment decreases in the order MPIXEL4S > MPIXEL8L > MPIXEL8S > MPIXEL as languages and scripts are added.This mirrors the downstream curse of multilinguality reported in the paper.Source: §5.1 · p. 6
- MPIXELLARGE has higher intermediate-layer Meaning Discrimination scores, but its scores decline in the final layers until MPIXEL matches or surpasses it.The decline is driven by nonmonotonic Meaning Discrimination for same-script language pairs.Source: §5.1 · p. 6
- Related unseen languages sharing a script—and usually a family—with a seen language show the highest Meaning Discrimination and lowest Language Discrimination at every depth.This suggests their alignment reflects surface similarity rather than exposure to the unseen language.Source: §5.1 · p. 7
5.2 Sentence-level Probing
- On SentLen and WC, 12-layer models plateau at layers 3–4, while MPIXELLARGE continues improving until layer 9.MPIXELLARGE also achieves the highest peak accuracy on all surface-semantic tasks, but both MPIXEL models lag monolingual PIXEL† on the complex-semantics SOMO task.Source: §5.2 · p. 7
5.3 Word-level Probing
- On word-level probes, MPIXEL variants start low and improve with depth; they perform similarly to XLM-R on Spanish and Turkish in deeper layers but worse on Arabic.MPIXELLARGE closes the XLM-R gap in deeper layers; the authors suggest Arabic’s word-level rendering may contribute to the difference, but do not test this explanation.Source: §5.3 · pp. 7–8
6 Related Work
Prior work established visually rendered text processing for translation and pixel-based language modeling, alongside vocabulary-free character- and byte-level alternatives. This paper’s related-work distinction is its controlled study of multilingual pixel-model scaling across languages, scripts, and model size under matched per-language compute.
Visually rendered text models
- Salesky et al. (2021, 2023) applied visually rendered text to machine translation, and Rust et al. (2023) introduced PIXEL, a masked-autoencoder language model pretrained on rendered English.PIXEL outperformed its tokenizer-based BERT equivalent on languages with novel writing systems and showed increased robustness to orthographic perturbations.Source: §6 · p. 8
Multilingual pixel models
- Kesen et al. (2025) introduced PIXEL-M4, pretrained on four languages each using a distinct writing system, and reported strong cross-lingual transfer learning performance to unseen scripts.Hu et al. (2026) subsequently extended the patch-level autoregressive visual text generation model PIXAR.Source: §6 · p. 8
Distinction from prior work
- Unlike these prior multilingual pixel-model studies, this work controls scaling across languages, scripts, and model size under matched per-language compute.Character- and byte-level models also avoid fixed vocabularies, but lack access to visual text form and orthographic similarities.Source: §6 · p. 8
7 Conclusion
MPIXEL scales multilingual pretraining across languages, scripts, and model capacity under fixed per-language compute. The results show local scaling gains, capacity mitigation of multilinguality costs, and strong performance on typologically distant low-resource languages.
7 Conclusion
- Each scaling axis mainly improves the languages and scripts it adds, while increased capacity mitigates the curse of multilinguality.Transfer to unseen scripts is established by the first multilingual step and does not improve with further scaling.Source: §7 · p. 8
- On low-resource languages typologically distant from all baselines’ pretraining languages, MPIXEL outperforms subword and character-level baselines on many writing systems.Source: §7 · p. 8
- Larger models reserve more layers for visual processing, while cross-lingual alignment weakens with more languages but generalizes through shared scripts.Source: §7 · p. 8
Limitations
The study’s conclusions are bounded by a small number of scaling settings, single pretraining runs, and a fixed-per-language-compute design. Its language choices and rendering strategy also limit what the experiments establish.
Limitations
- The study covers only four language counts and two model sizes, and its claims hold only within that range.Each variant has one pretraining run, so small differences may fall within unquantified run-to-run variance.Source: Limitations · p. 9
- Fixed per-language compute gives variants with more languages more total data, so results under a fixed total-compute budget may differ.Source: Limitations · p. 9
- Each scaling axis uses one language set, and the first multilingual step adds languages and scripts together, preventing separation of their individual effects.Source: Limitations · p. 9
- MPIXEL is encoder-only and lacks generation capability, and the study does not establish whether ADAPTIVE is the optimal rendering strategy.Source: Limitations · p. 9
Ethical Considerations
The authors note that web-crawled pretraining data may retain social biases, offensive content, and personal information, and that performance is strongest in pretraining languages.
Ethical Considerations
- Web-crawled corpora may still contain social biases, offensive content, and personal information, while models perform best on downstream tasks in their pretraining languages.The authors recommend careful evaluation before use in other languages.Source: Ethical Considerations · p. 9
A.1 Use of Frontier LLMs
The authors used LLMs in preparing the work and state that they reviewed the code and written material.
A.1 Use of Frontier LLMs
- LLMs helped write and debug code, produce results tables and plots, and revise text; authors reviewed and tested the code and checked all written material.Source: §A.1 · p. 12
A.2 Data Statistics
The appendix documents the pretraining and evaluation data and implementation, then provides full benchmark results and representation analyses. Those analyses report how language-pair alignment varies with script sharing and how script identity is represented across model layers.
A.2 Data Statistics
- Evaluation covers SIB-200, a 200-language parallel text-classification benchmark with 701/99/204 training, validation, and test examples per language, plus dependency parsing and NER; selected languages had all available same-script distances above T = 0.50 or used unseen scripts.Source: §A.2 · p. 13
A.3 Implementation Details
- Pretraining uses Flash Attention 2 and torch.compile; the reported runs take approximately 1350 GPU hours for MPIXEL and 2500 for MPIXELLARGE on NVIDIA H100s.The appendix also lists shared and model-size-specific pretraining hyperparameters and task-specific fine-tuning settings.Source: §A.3 · p. 13, Table 8 · p. 16, Table 9 · p. 17
A.4 Full Results
- The appendix provides complete per-language results for SIB-200, NER, XNLI, GLUE, and Universal Dependencies parsing.The accompanying table captions identify the reported metrics as test F1, accuracy, or LAS, depending on the task.Source: §A.4 · pp. 13–14, Table 10 · p. 19, Table 11 · p. 20
A.6.1 Generalization to unseen languages.
- Related seen–unseen language pairs sharing a script have the highest, flattest MD and lowest LD; unseen–unseen pairs have the lowest MD and unstable LD.The difference between seen–seen pairs and unrelated seen–unseen pairs is small, so joint pretraining inclusion does not increase alignment.Source: §A.6.1 · p. 22
A.6.2 Same- versus cross-script pairs.
- Adding languages changes the language-pair mix: the 16-language set has more same-script pairs, which lack easy visual language cues but are easier to align in meaning.For MPIXELBASE, embedding-layer LD falls from 1.00 on the four PIXEL-M4 languages to 0.95 on all 16.Source: §A.6.2 · p. 22
- Script sharing separates the alignment trajectories: MPIXELBASE same-script LD recovers above cross-script LD in middle layers, while MPIXELLARGE same-script LD stays lower throughout.Same-script MD is mildly affected by script sharing in MPIXELBASE but is more dynamic and peaks higher than cross-script MD in MPIXELLARGE.Source: §A.6.2 · p. 22
A.7 Script ID Probes
- Script-ID probes test layer representations with linear classifiers on mean-pooled outputs from 13 layers, using iid and out-of-distribution language settings.The out-of-distribution setting uses 32 unseen languages with scripts represented during training.Source: §A.7 · p. 22
- Script identity is nearly perfectly recovered in the iid setting; out-of-distribution accuracy stays near-perfect for MPIXELBASE and MPIXEL8S but declines late for PIXEL-M4 and MPIXEL8L.The late-layer errors are largely Cyrillic misclassified as Latin, despite each MPIXEL variant having one Cyrillic-script pretraining language.Source: §A.7 · pp. 22–23