PaperlayerView exact revision on arXiv

Source-linked AI summary

One Form to Transfer Them All: Pretraining Multilingual Language Models Beyond Native Orthography

Muge Zhang, Aaron Jencks, Krishna Badikela, Yulia Tsvetkov, Sachin Kumar

arXiv:2608.25904v1cs.CL

TL;DR

Script differences reduce vocabulary overlap and hinder transfer between related languages. The paper compares text, IPA, and romanization in controlled autoregressive multilingual pretraining, finding that romanized pretraining gives the strongest transfer across seen and unseen languages, while downstream romanization finetuning can hurt covered languages.

  • Problem

    Different writing systems limit shared subword vocabulary, while direct comparisons of romanization and IPA in autoregressive multilingual pretraining remain limited.

  • Method

    The paper pretrains autoregressive multilingual models from scratch with orthographic text, IPA, or romanization under matched conditions across three scales and eight languages.

  • Results

    Romanized pretraining yields the strongest cross-lingual transfer across seen and unseen languages, while IPA usually improves over text but trails romanization.

  • Takeaways & Limitations

    Romanization should be treated as a pretraining design choice when transfer to unseen scripts is a priority, rather than primarily as a downstream fix.

  • Takeaways & Limitations

    The study relies on Phonemizer for speed and language coverage, while simple heuristic transcription handling may limit robustness.

Abstract

from arXiv · show

Multilingual language models transfer knowledge across languages through shared subword vocabulary, a mechanism that breaks down when related languages use different writing systems. Prior work addresses this via script equalization (romanization or IPA transcription), but direct comparisons are rare; the focus has been on encoder-only models, with most work adapting existing pretrained models. We systematically compare different input representations in autoregressive multilingual pretraining, comparing orthographic text, IPA, and romanization in a controlled setup across three scales (467M, 709M, and 1.03B) on eight languages in four typologically motivated pairs. Across a wide range of downstream tasks on seen and unseen languages, romanized pretraining yields the strongest cross-lingual transfer, and the advantage over text widens with scale. IPA improves over text in most settings but trails romanization. Surprisingly, finetuning a text-pretrained model on romanized data hurts performance on languages already covered by the base model, only marginally helping when the model lacks script coverage. Our results indicate that for multilingual models spanning typologically diverse scripts, to obtain maximum benefits, romanization should be treated as a core design choice applied at pretraining rather than a post hoc fix.

1 Introduction

Different writing systems limit vocabulary overlap, weakening transfer even between related languages. This study compares orthographic text, IPA, and romanization in controlled autoregressive multilingual pretraining across languages, scales, and downstream settings.

  • Different writing systems largely prevent shared vocabulary overlap, limiting cross-lingual transfer even between linguistically close languages.
  • The study compares orthographic text, IPA, and romanization in autoregressive multilingual pretraining across eight languages organized into four typologically motivated pairs.The pairs are English–Spanish, Russian–Polish, Hindi–Urdu, and Tamil–Malayalam.
  • Each representation is trained from scratch at 467M, 709M, and 1.03B parameters under matched architecture, data, vocabulary, and training conditions.
  • Romanized pretraining is strongest in every evaluation regime and model size, with its advantage over text widening as parameters scale.
  • IPA improves over orthographic text in most settings and matches romanization for Hindi–Urdu, but trails romanization on unseen-language transfer.Hindi–Urdu is both phonologically aligned and orthographically disjoint in the corpus.
  • Finetuning a text-pretrained model on romanized data regresses performance on covered languages and helps only when the base model lacks script coverage.

2 Related Work

Prior research identifies script mismatch and missing vocabulary overlap as major barriers to multilingual transfer. It explores romanization and phonemic representations, increasingly extending script equalization from encoder-only models to decoder-only language models.

  • Multilingual models transfer well across languages sharing scripts but show little improvement or may degrade across script mismatches.
  • Prior studies identify script mismatch, and therefore limited vocabulary overlap, as a dominant failure mode for transfer to unseen languages.
  • Romanization exposes lexical overlap hidden by original orthography and has benefited multilingual learning, especially for low-resource languages.
  • Recent work extends romanization from encoder-only models to decoder-only language models through continued pretraining and instruction tuning.
  • Phonemic representations provide another route for bridging scripts, with prior studies reporting reduced cross-lingual performance gaps and low meaning-level-task cost.

3 Experimental Setup

The experiments isolate input representation in matched multilingual pretraining, using typologically varied language pairs, three representations, shared tokenization, and models trained from scratch.

  • Language pairs: The corpus contains eight languages arranged into four pairs spanning varied orthographic and phonological relationships.English–Spanish shares Latin script; Russian–Polish, Hindi–Urdu, and Tamil–Malayalam use different scripts.
  • Language pairs: Hindi–Urdu tests phonemic overlap despite minimal orthographic overlap, while English–Spanish controls for naturally shared subword tokens.
  • Data: The four bilingual corpora contain approximately 21.7B words, with word-count ratios preserved within pairs and budgets equalized across pairs.
  • Representations: Orthographic text, IPA, and Uroman romanization are compared, alongside Text→Rom finetuning to test whether downstream adaptation can recover romanization benefits.
  • Representations: IPA removes selected diacritics so the representation retains phonemic segments while increasing cross-lingual overlap.Retaining diacritics produced roughly 50% of the stripped version’s weighted-Jaccard overlap in early experiments.
  • Tokenization: Each representation uses a jointly trained 100K-vocabulary Byte-Level BPE tokenizer, keeping tokenization capacity constant across conditions.
  • Analysis: The study measures tokenizer overlap and sequence length because representation affects lexical sharing, compute, latency, API cost, and context-window usage.
  • Pre-training: Causal language models are pretrained from scratch at three scales under a fixed compute budget and shared optimization procedure.The token budget is below the compute-optimal point for each parameter count, but remains constant for controlled comparison.

4 Results and Discussion

Across representation properties and downstream evaluations, romanization consistently supports the strongest multilingual transfer, while IPA generally improves over text but remains behind romanization. Romanized fine-tuning instead helps mainly when the base model lacks script coverage and can harm languages already covered.

  • 4.1 Properties of the Three Representations: Romanization lifts Russian–Polish and Tamil–Malayalam subword overlap toward the English–Spanish range, but does not close Hindi–Urdu’s gap.
  • 4.1 Properties of the Three Representations: Hindi, Tamil, and Malayalam require roughly three times the inference latency and API costs under text, while IPA and romanization close this gap.
  • 4.2 Downstream Performance: Romanized pretraining produces the strongest cross-lingual transfer across zero-shot, few-shot, supervised fine-tuning, summarization, and unseen-language evaluations at all three scales.The ranking holds across XStoryCloze, XCOPA, NLI, MASSIVE, XL-Sum, and XNLI, with the gap over text widening as models scale.
  • 4.2 Downstream Performance: IPA improves over orthographic text in most settings but trails romanization outside Hindi–Urdu, especially for transfer to unseen languages.IPA matches romanization only for Hindi–Urdu, where phonology is nearly identical despite disjoint orthographies.
  • 4.2 Downstream Performance: Text→Rom fine-tuning causes large regressions on pretraining-seen languages but helps unseen languages when the base model lacks script coverage, with smaller gains than romanized pretraining.For unseen MASSIVE, the benefit appears at the smallest scale but disappears at Medium and Large.
  • 4.2 Downstream Performance: Fine-tuning an English-only text-pretrained model on romanized data recovers prior transfer gains, concentrated in languages with absent or rare scripts.Hindi, Urdu, Tamil, and Malayalam gain 20–30 F1 on MASSIVE at larger scales and move from near-zero to 10–15 on XL-Sum.

5 Conclusion

The paper systematically compares orthographic text, IPA, and Uroman romanization in controlled autoregressive multilingual pretraining. Romanized pretraining provides the strongest transfer across seen and unseen languages, while romanized fine-tuning helps only when script coverage is absent.

  • 5 Conclusion: The study treats input representation as a controlled variable, comparing orthographic text, IPA, and Uroman romanization across three scales and eight languages.The models are evaluated under matched pretraining conditions.
  • 5 Conclusion: Romanized pretraining yields the strongest cross-lingual transfer on both seen and unseen languages.This conclusion is reported across the evaluated downstream settings.
  • 5 Conclusion: IPA improves over text in most settings but matches romanization only for Hindi–Urdu, where phonology is shared and orthography is disjoint.The result identifies Hindi–Urdu as the narrow case where IPA reaches romanization.
  • 5 Conclusion: Romanized fine-tuning of a text-pretrained model regresses performance on covered languages and helps only when the base model lacks script coverage.The supplied table caption describes Text and Romanized conditions for English-pretrained checkpoints on MASSIVE and XL-Sum.
  • 5 Conclusion: The paper recommends treating input representation as a deliberate pretraining design choice and adopting romanization when transfer to unseen scripts is a priority.This recommendation is stated alongside data mixture and tokenizer choices.

6 Limitations

The paper’s limitations concern scale, language coverage, transcription quality, IPA encoding, heuristic preprocessing, and untested conversion back to orthographic text. These boundaries leave larger models, broader scripts, improved phonemization, and generative reconstruction for future work.

  • 6 Limitations: Models reach approximately 1B parameters, substantially below contemporary multilingual models ranging from 7B to over 100B parameters.The authors identify larger-scale evaluation as future work.
  • 6 Limitations: The language set covers four pairs using mainly alphabetic or abugida-based systems, excluding logographic systems such as Chinese.The authors also call for studying additional linguistic relatedness and shared inheritance.
  • 6 Limitations: Phonemizer was selected for practical transcription quality and throughput, but a comprehensive evaluation of transcription cost and alternative G2P tools remains undone.The authors state that improving phonemization could change the reported IPA trends.
  • 6 Limitations: Incomplete language support and Unicode handling may affect IPA transcription consistency, especially for low-resource or under-supported languages.The preprocessing stripped some diacritics and stress markers, while compound-symbol handling motivated future encoding work.
  • 6 Limitations: The IPA stripping and numerical-preservation procedure relies on a relatively simple heuristic-based system.More sophisticated approaches may improve robustness.
  • 6 Limitations: The study did not evaluate the performance or practicality of phoneme-to-grapheme conversion for returning IPA or romanized outputs to orthographic text.This constrains conclusions about usefulness in generative settings.
  • 6 Limitations: Romanization and phonemization are lossy and may not be fully reversible, while many reconstruction approaches remain language-specific.The supplied passage frames reversibility and generalization as unresolved concerns.

A Tokenizer Overlap Computation Details

The tokenizer-overlap analysis measures shared vocabulary between language pairs using raw and adjusted Jaccard-style quantities. Adjustments address contamination, token frequency, and corpus-size imbalance before comparing representations.

  • A Tokenizer Overlap Computation Details: For each language pair, the analysis tokenizes separate monolingual corpora and defines T_Li as the unique tokens observed in language L_i.The multilingual tokenizer is held fixed while corpora are processed separately.
  • A Tokenizer Overlap Computation Details: The raw shared-vocabulary measure is the Jaccard index over the two languages’ token sets.Higher values indicate greater shared vocabulary and potential transfer through shared embeddings.
  • A Tokenizer Overlap Computation Details: Raw Jaccard is confounded by rare tokens, punctuation and whitespace artifacts, code-switching, and unequal corpus sizes.The adjusted variant is introduced to address these effects.
  • A Tokenizer Overlap Computation Details: The adjusted analysis filters documents by language tags and removes code-switched or noisy lines.These corrections target contamination observed during preprocessing.
  • A Tokenizer Overlap Computation Details: The frequency-aware variant weights each token by its actual usage in the two corpora, downweighting tokens rare in one language.The count c_Li(t) denotes token t’s corpus frequency for language L_i.
  • A Tokenizer Overlap Computation Details: Matched token budgets and repeated random sampling control corpus-size imbalance when averaging overlap.This makes unique-token comparisons less dependent on unequal corpus sizes.

B Pretraining details

The experiments use matched training and evaluation procedures across representations, while sequence lengths differ substantially by script. Text requires many more tokens for several non-Latin languages, reducing document coverage under a fixed token budget.

  • B Pretraining details: Each model trains for three epochs with AdamW for embedding, head, and scalar parameters and Muon for hidden-layer matrix parameters.Training uses 524,288 tokens per step across eight NVIDIA H100 GPUs and selects the lowest-validation-loss checkpoint.
  • B Pretraining details: Pretraining BPC values are not directly comparable across representations because their character denominators differ.The paper therefore uses downstream task performance as the operational measure of pretraining quality.
  • B Pretraining details: Figure 5 reports average tokens per document by language and representation on the FineWeb-2 pretraining corpus.The figure organizes sequence-length statistics by language and input representation.
  • B Pretraining details: 2.5–3.5 times: Hindi, Urdu, Tamil, and Malayalam text tokens exceed their romanized counterparts.IPA and romanization compress these languages into the same token-length range as the other languages.
  • B Pretraining details: 2.5–3.5 times fewer documents per step are covered by text models than by IPA or romanized models for those four languages under a fixed token budget.Romanization and IPA reduce this sequence-length disparity.
  • B Pretraining details: Fine-tuning searches over four learning rates and two batch sizes, using validation loss for selection.Seen languages are fine-tuned jointly; unseen languages use a separate model per language.

C.2 Bilingual Downstream Results

Bilingual small-scale results informed the multilingual experiments, with romanization or IPA outperforming text in seven of eight dataset–pair settings.

  • Table 11 reports bilingual cross-lingual transfer F1 at the small-model scale and informed the multilingual experiment design.
  • Romanization or IPA outperforms text on seven of eight (dataset, pair) settings.
  • Text wins only on MASSIVE Eng–Spa, the one pair that already shares Latin script.
  • The largest gaps occur for related-language pairs with disjoint scripts, where text can collapse to near-zero F1 while equalized representations recover transfer.
  • Romanization is the most consistent winner, while IPA’s largest gains concentrate on Hindi–Urdu.

D.1 Full k-shot results

Across scales and shot counts, representation choice remains more consequential than added demonstrations: romanization is strongest, IPA trails, and the ordering appears early during pretraining.

  • Full k-shot results: Few-shot demonstrations provide little reliable improvement over zero-shot, with small and often non-monotonic within-scale variation across k.
  • Full k-shot results: Increasing model scale yields consistent gains for representations trained from scratch in the target surface form.
  • Full k-shot results: IPA’s gap relative to text and romanization persists across k values, indicating a representation-level ICL deficit rather than a zero-shot artifact.
  • Full k-shot results: Text-pretrained models evaluated on romanized inputs generally underperform the same models evaluated on text.
  • Progression during pretraining: Romanized pretraining ranks above text and IPA at nearly every intermediate checkpoint, and the gaps do not close during training.

E Impact of the Tokenizer Integer-Overflow Bug

A 32-bit BPE frequency accumulator could overflow during tokenizer training, so the authors patched it to 64-bit and reassessed affected evaluations; representation conclusions remained unchanged.

  • BPE pair counts above 2^31 could overflow the signed 32-bit accumulator and be dropped or mis-ranked.
  • The authors patched the accumulator to 64-bit, verified identical output below the threshold, retrained tokenizers and affected models, and reevaluated MASSIVE and zero-shot prompting.
  • The correction study did not rerun XNLI or XL-Sum because doing so exceeded the available compute budget.
  • The tokenizer correction changed MASSIVE macro-F1 by at most 0.46 points and zero-shot average accuracy by at most 0.007.
  • Every pairwise ordering of the three representations was preserved at every scale, with changes smaller than the representation gaps.
  • Frequency-weighted subword overlap changed by at most 0.012, and the authors conclude that their results are unchanged.
Loading 2608.25904v1…