Source-linked AI summary
The Curse of Multilinguality in Lexical Normalization
Saman Rahbar
TL;DR
Scarce labelled data makes multilingual training attractive for lexical normalization, but it is unknown how many languages a compact model should share. The paper varies the language count for one fixed-capacity character-level model across twelve benchmark languages. Accuracy peaks with only a few partners and then falls, while typological distance does not reliably identify the best language count.
Problem
With labelled normalization corpora small and available for few languages, it is unclear whether joint training helps compact models or whether languages compete for their fixed capacity.
Source: §1 · p. 1Method
The study trains one fixed-size character-level model on subsets of one to twelve MULTILEXNORM languages and measures per-language accuracy.
Source: §1 · p. 1Results
Takeaways & Limitations
For compact multilingual normalizers in this setting, training on fewer languages can outperform pooling all available languages.
Source: §5 · p. 5, §6 · p. 6The study fixes one small model size and one from-scratch character-level Transformer, does not map how the peak shifts with capacity, and does not separate mechanisms behind the decline.
Source: Limitations · p. 6
Abstract
from arXiv · showhide
Lexical normalization rewrites the noisy, non-standard words that fill user-generated text (tmrw, u, gr8) into their standard forms. Because labelled data is scarce for most languages, a popular shortcut is to train a single model on many languages at once. We ask a simple question: how many languages should such a model be trained on? Using one fixed-capacity character-level model and twelve languages from a standard benchmark, we vary the number of jointly trained languages from one to twelve and measure per-language accuracy. We find a clear curse of multilinguality: accuracy is highest when a language is trained with only a few others, often just one to four, and then falls steadily and substantially, dropping by about forty percent as the rest are piled on. A control that holds the total amount of training data constant makes the decline arrive sooner and fall further, which points to competition among the languages for one fixed-size model rather than to how much data is available. We also test whether a language's typological distance from the others predicts its ideal number of co-training languages, and find no dependable rule: any apparent relationship rests on a couple of languages and does not hold up. For compact normalization models, less can be more: a few languages beat pooling everything into a single model.
1 Introduction
Because labelled normalization data is scarce, multilingual training offers a way to pool examples, but it is unclear whether adding languages helps a small character-level model. This study varies the number of jointly trained languages and examines accuracy, data volume, and typological distance.
1 Introduction
- The study asks whether adding languages helps lexical normalization, where small labelled corpora motivate joint training but fixed model capacity may create competition.The same trade-off has been observed in other multilingual settings, but its relevance to small character-level normalization models was unknown.Source: §1 · p. 1
- A single small character-level model is trained on every language count from one to twelve, with per-language normalization accuracy measured.Languages are drawn from the MULTILEXNORM benchmark.Source: §1 · p. 1
- Per-language accuracy peaks with a few languages, then declines steadily as more languages share the model.The paper identifies this as a curse of multilinguality for lexical normalization.Source: §1 · pp. 1–2
- A control holding total training data fixed shows that the decline is not explained by data volume, but by many languages sharing one fixed-size model.This result treats capacity competition as the explanation without isolating its underlying mechanisms.Source: §1 · p. 2
- Typological isolation does not reliably predict a language’s ideal number of co-training partners; apparent effects rest on a couple of languages.Source: §1 · p. 2
2 Related Work
Prior work established multilingual normalization benchmarks and documented multilinguality costs in larger language and translation models. This paper examines whether those patterns and typological distance help explain language pooling in lexical normalization.
2 Related Work
- MULTILEXNORM aligned twelve normalization datasets under one metric, but its language count had not been treated as a variable.Its strongest entry fine-tuned BYT5 jointly on all languages.Source: §2 · p. 2
- Earlier multilinguality studies covered large models and word- or subword-level tasks, leaving the curse’s relevance to tiny character-level normalization unresolved.Proposed explanations include negative interference from limited shared capacity.Source: §2 · p. 2
- The study uses URIEL typological distance to test whether a language’s isolation indicates how many co-training partners it should have.Source: §2 · p. 2
3 Experimental Setup
The experiments use the twelve MULTILEXNORM datasets and one fixed, from-scratch character-level model. They sweep language-subset sizes, control total training tokens, and measure typological isolation.
3 Experimental Setup
- The twelve MULTILEXNORM datasets vary from about 6k to 57k training tokens and from 6.6% to 37% changed tokens.The study retains official train/test splits and creates development data for five languages without a development split.Source: §3 · p. 2, Table 1 · p. 3
- ERR measures normalization performance relative to copying the input: 1 is perfect, 0 matches leave-as-is, and below 0 means more corruption than correction.Source: §3 · p. 2
- Every condition uses the same character-level encoder–decoder Transformer, with 1.49M parameters and a shared character vocabulary.Training from scratch avoids inheriting unknown multilingual capacity from a pretrained model.Source: §3 · pp. 2–3, Table 2 · p. 3
- For each language count from one to twelve, random subsets are trained across three seeds; each language’s mean ERR by subset size defines its frontier and optimum k.The sweep comprises 390 runs and estimates test ERR on up to 2,000 tokens per language.Source: §3 · p. 3
- A data-controlled ablation caps total training tokens so a twelve-language model sees no more data than a one-language model.Any decline that remains cannot be attributed to data volume alone.Source: §3 · p. 3
- Typological isolation is each language’s mean URIEL syntactic distance to the other eleven; Turkish and Indonesian–English are clear outliers.The European languages cluster tightly.Source: §3 · p. 3
4 Results
Across the ten languages the model normalizes effectively, ERR peaks with few partners and then declines; a fixed-data control makes the decline steeper, while typological isolation yields no dependable rule.
4.1 Normalization has a curse of multilinguality
- Four languages peak alone; the others’ optima range from k = 2 for Danish and Slovenian to k = 6 for Turkish, while English falls from ERR 0.384 to 0.088 at k = 12.Six of the ten languages peak at an interior k*; the table reports ERR at one language, the optimum, and all twelve.Source: §4.1 · p. 4, Table 3 · p. 4
- Spanish and Italian have negative ERR at every k and improve with partners; their k* marks least harm, while the proposed absolute-evidence account is not a measured mechanism.The authors note that the fixed-capacity model may be below the level needed to serve these two datasets.Source: §4.1 · p. 4
- The mapped frontier is specific to this model size; a larger model might serve Spanish and Italian successfully and peak later.Source: §4.1 · p. 4
- 0.316 mean ERR at k = 2 (versus 0.312 at k = 1) falls to 0.191 at k = 12, about 40% below peak, across ten languages.Every one of the ten ends below its own best result at k = 12.Source: §4.1 · pp. 3–4, Figure 1 · p. 5
4.2 The decline is not a data-volume effect
- About 52% drop at k = 12 under fixed total tokens, versus 40% with growing data; mean k* shifts earlier, from 2.5 to 2.1.The authors attribute the decline to more languages sharing one fixed-size model, but do not distinguish possible capacity mechanisms.Source: Figure 1 · p. 5, §4.2 · p. 5
4.3 Typology does not predict the sweet spot
- The positive isolation–k* association (r = 0.69, p = 0.04) rests on Turkish and Indonesian–English and disappears across all twelve, so no typological rule is supported.The eight European languages form a tight cluster with no internal trend.Source: §4.3 · p. 5
5 Discussion
For compact normalizers, using only a few languages—often one to four—outperforms pooling all twelve on average. Typological isolation gives no dependable rule for choosing the number of co-training languages, and the observed optimum is specific to the model capacity studied.
5 Discussion
- For most languages, the best model used one to four languages, and pooling all twelve was worse on average.Some languages were best off training alone.Source: §5 · p. 5
- Five of ten languages did best with one to three partners, while Turkish did best with five.The authors suggest that some noise patterns may carry over between languages.Source: §5 · p. 5
- The apparent positive correlation between typological isolation and optimum k* rests on two languages and disappears when all twelve are included.The authors draw no dependable typological rule for selecting the number of co-training languages.Source: Figure 3 · p. 6, §5 · p. 6
- The observed peak is a compact-model finding: its dependence on model size was not measured, and the authors expect it to shift with capacity.Larger models and lightweight per-language adapters are proposed as directions to explore.Source: §5 · p. 6
- How well a language is served alone comes closest to tracking its optimum, but the association at ten languages is not a rule (r = −0.58, p = 0.08).Source: §5 · p. 6
6 Conclusion
The conclusion is that a fixed-size multilingual normalizer benefits from adding languages only briefly: per-language accuracy peaks with a handful and then falls. Holding data fixed points to shared model capacity rather than data volume, while typological distance does not reliably locate a language’s best setting.
6 Conclusion
- With a single fixed-size model, per-language accuracy peaks at a handful of languages and then falls.Source: §6 · p. 6
- A control that holds data fixed shows that the decline is not a matter of data volume but of many languages sharing one fixed model.Source: §6 · p. 6
- For compact multilingual normalizers, fewer languages are preferable to pooling everything, and typological distance does not reliably identify the sweet spot.Source: §6 · p. 6
Limitations
The study’s findings are bounded by its single small model size and model family, and its design does not isolate the mechanisms behind the decline. Its typology analysis also covers a small, mostly European sample with specific measurement and code-switching choices.
Limitations
- The study fixes one small model size, so it does not establish how the peak changes with capacity or a general optimum for other model sizes.The authors expect the peak to shift with capacity but do not map that dependence.Source: Limitations · p. 6
- The design does not separate reduced effective parameters, reduced training exposure, and language interference as explanations for the decline.These factors move together as languages are added and are treated jointly as capacity.Source: Limitations · p. 6
- Results use one character-level Transformer family trained from scratch and may differ for large pretrained byte-level models.Source: Limitations · p. 6
- The sweep estimates test ERR on fixed token samples and averages random language subsets, so it cannot identify which partners help.Per-subset records were not retained.Source: Limitations · p. 6
- The typology analysis covers only twelve languages, mostly European, and uses syntactic URIEL vectors while mapping code-switched datasets to one matrix language.The authors treat the question as open rather than settled given the small, clustered sample.Source: Limitations · pp. 6–7