PaperlayerView exact revision on arXiv

Source-linked AI summary

How to Use Parallel Data for Multilingual Pretraining

Zhihong Huang, Yao Lu, Derek F. Wong, Hamid Alinejad-Rokny, Min Yang

arXiv:2610.28838v1cs.CLcs.AI

TL;DR

Standard multilingual pretraining samples source documents and translations independently, leaving unclear how to use their parallel structure at pretraining scale. This paper concatenates translation pairs during PPT, then tests continuing with SPT; at matched budgets, the two-stage approach improves language understanding and translation over SPT in both resource settings.

  • Problem

    Although parallel structure is available at pretraining scale, it remains unclear how to use it effectively because standard multilingual pretraining samples source documents and translations independently.

    Source: §1 · p. 1

  • Method

    PPT concatenates each document with its translation in one training sequence, and the two-stage approach continues PPT with SPT to adapt to standard language modeling.

    Source: §1 · p. 1, §7 · p. 8

  • Results

    At matched budgets, PPT→SPT improves understanding and translation over SPT in both resource settings, with BLEU gains over SPT of 77% on English–French.

    Source: §1 · p. 2, §7 · p. 8

  • Takeaways & Limitations

    PPT→SPT improves monolingual language modeling while retaining greater use of corresponding cross-lingual content than SPT.

    Source: §5 · p. 8

    The experiments use 1.3B-parameter models and two language pairs; larger models and more pairs would broaden understanding of the approach.

    Source: Limitations · p. 9

Abstract

from arXiv · show

High-quality human-translated parallel corpora can improve cross-lingual transfer, but are typically too small for large-scale pretraining. Machine-translated web corpora help overcome this limitation, making parallel data available at pretraining scale. Standard multilingual pretraining, however, samples source documents and their translations independently, without explicitly using their parallel structure. We investigate how to use the parallel structure of the data to improve multilingual pretraining by concatenating each document with its translation in the same training sequence. We compare this parallel pretraining (PPT) approach with standard pretraining (SPT), training models from scratch at matched token budgets. Surprisingly, despite the benefits of parallel data reported in prior cross-lingual work, PPT alone does not consistently improve language understanding over SPT. Our further analysis suggests that although PPT builds a strong cross-lingual foundation, as shown by its better use of shared information across languages, the mismatch between parallel training contexts and single-language task inputs limits its gains in general language understanding. This finding motivates a two-stage approach following the pretraining-fine-tuning paradigm: we first use PPT to build a cross-lingual foundation, then continue with SPT to adapt it to standard language modeling. In both high-resource and low-resource settings, the two-stage approach consistently improves general language understanding and cross-lingual task performance over standard pretraining alone at matched training budgets.

1 Introduction

The paper asks how to use document-level parallel structure for multilingual pretraining at scale, where machine-translated web corpora make parallel data available but standard pretraining samples translations independently. It introduces PPT and finds that PPT alone does not consistently improve language understanding, while PPT followed by SPT improves understanding and translation at matched budgets.

1 Introduction

  • The paper investigates how to use parallel structure available at pretraining scale, since standard pretraining samples source documents and translations independently.Machine translation makes high-quality English web corpora available in a wide range of languages, while human-translated corpora are typically too small for large-scale pretraining.Source: §1 · p. 1
  • PPT concatenates each document with its translation in the same training sequence, and the comparison trains models from scratch to isolate this sequence-construction difference.SPT treats original and translated documents as independent examples; the methods use the same training setup and hyperparameters.Source: §1 · pp. 1–2
  • PPT alone does not consistently improve general language understanding over SPT, although PPT models benefit more from cross-lingual context.The authors suggest that parallel training contexts mismatch the single-language inputs typical of language-understanding tasks.Source: §1 · p. 2
  • At matched budgets, PPT→SPT improves average general language understanding in both language settings and achieves the best translation performance among four strategies.BLEU improves over SPT by 77% on English–French and 45% on English–Swahili.Source: §1 · p. 2
  • Behavioral analyses find that continuing PPT with SPT improves monolingual language modeling while retaining greater use of cross-lingual context than SPT alone.The two-stage approach uses PPT to develop cross-lingual capabilities, then SPT to adapt to standard language modeling.Source: §1 · p. 2

2 Pretraining with Parallel Data

PPT pairs each document with its translation in one training sequence, making corresponding cross-lingual content available during prediction. PPT→SPT then adapts the model to independently sampled documents, and the paper also compares the reverse order.

2.1 Independent and paired-document sequences

  • PPT joins translation pairs before sequence packing, so corresponding cross-lingual content can be available when predicting the later document.Both PPT and SPT use the same bilingual corpus and autoregressive next-token prediction; document order alternates to provide context in both directions.Source: §2.1 · p. 3

2.2 From parallel to standard pretraining

  • PPT→SPT continues parallel-pretrained models with independently sampled documents, adapting them to standard language modeling without systematically supplying each document’s translation.The paper also tests SPT→PPT to examine how training order affects understanding, translation, and cross-lingual context use.Source: §2.2 · p. 3

3 Experiments

The experiments compare SPT, PPT, and both training orders on English–French and English–Swahili tasks at matched token budgets. PPT→SPT consistently improves language understanding and translation over SPT across both language settings.

3.1 Experimental setup

  • The experiments use 1.3B-parameter models trained on English–French and English–Swahili data, with language-understanding accuracy and translation BLEU as evaluation measures.The English–French runs train to approximately 200B tokens, and staged continuations compare matched total budgets.Source: §3.1 · p. 3

3.2.1 SPT and PPT throughout pretraining

  • SPT and PPT have similar average language-understanding accuracy throughout pretraining, with bilingual averages differing by at most 1.64 percentage points.Translation rankings vary: at 200B tokens, SPT exceeds PPT by 12.74 BLEU.Source: §3.2.1 · p. 4

3.3 English–Swahili results

  • In English–Swahili, PPT→SPT achieves the highest Swahili understanding mean and improves translation over SPT from 9.07 to 13.18 BLEU.SPT→PPT gives the highest English mean, while both orders reach a bilingual understanding average of 50.11%.Source: §3.3 · pp. 4–5, Table 2 · p. 5

3.2.2 Combining PPT and SPT

  • PPT→SPT improves language understanding and translation over SPT at matched budgets in both language settings.For English–Swahili, both orders reach a bilingual understanding average of 50.11%, and PPT→SPT raises translation from SPT’s 9.07 to 13.18 BLEU.Source: §3.3 · pp. 4–5, §3.2.2 · p. 4, Figure 2 · p. 4

4 Analysis of Prediction and Context Use

The analyses show how the training order changes monolingual prediction and use of corresponding cross-lingual context. They also examine whether those changes appear in single-language reasoning and property-knowledge tasks.

4.1 Language modeling and cross-lingual context use

  • At matched budgets, PPT→SPT has lower target-only loss than SPT and PPT while retaining greater corresponding-source benefit than SPT; PPT alone has greater benefit but higher loss than SPT.SPT→PPT also has lower target-only loss and greater corresponding-source benefit than SPT.Source: §4.1 · pp. 5–6
  • During the 20B-token SPT stage after PPT, target-only loss falls by 0.038–0.061 BpB as corresponding-source benefit decreases, but remains above SPT; in the reverse order, loss falls by 0.020–0.029 BpB as benefit increases.These ranges cover three stages and two languages for PPT→SPT, and all six comparisons for SPT→PPT.Source: §4.1 · p. 6
  • In English–Swahili at 120B, both staged models have lower target-only loss and greater corresponding-source benefit than SPT; PPT→SPT retains this combination after SPT continuation.During SPT after PPT, target-only loss falls while corresponding-source benefit decreases from the PPT starting point.Source: §4.1 · p. 6

4.2 Using corresponding content

  • For target sentences 4–8, adjusted source-extension gains are 1.00–1.32 nats per token for PPT and staged models versus 0.72–0.87 for SPT; sentences 2–3 show −0.024 to 0.030 across models and languages.The adjustment subtracts the loss reduction from extending a length-matched unrelated source.Source: §4.2 · p. 6, Figure 4 · p. 7
  • Blocking source attention in layers 9–16 raises loss more for PPT and staged models than SPT: 0.62–0.73 versus 0.40 nats per token for French, and 0.87–0.95 versus 0.54 for English.PPT→SPT retains greater sensitivity to corresponding content than SPT on both measures; SPT→PPT also exceeds SPT.Source: Figure 4 · p. 7, §4.2 · p. 7
  • In English–Swahili, adjusted extension benefits on Swahili target sentences 4–8 are 0.62–0.69 nats per token for PPT and staged models versus 0.37 for SPT; both languages show greater extension benefits and larger layer-9–16 blocking effects for these models.Source: §4.2 · p. 7

4.3 Commonsense reasoning and property knowledge

  • From 100B to 120B, both training orders retain more correct answers than continued use of the initial method on both tasks; on XStoryCloze, lost answers fall by 0.79 percentage points for PPT→SPT and 1.16 for SPT→PPT, with higher accuracy in each language.On HellaSwag, both orders also correct more errors in English and French.Source: §4.3 · p. 7
  • On English COMPS-BASE, PPT→SPT reaches 75.15% accuracy, up from PPT’s 67.96% and close to SPT’s 75.52%; SPT→PPT reaches 77.27%, and both orders improve over continued initial-method training in all five property categories.Source: §4.3 · p. 7

5 Discussion

PPT→SPT improves understanding and translation at matched budgets in both language settings, with analyses showing better monolingual prediction alongside retained cross-lingual context use. The reverse order also has benefits, but PPT→SPT has the highest translation average across tested stages and language pairs.

5 Discussion

  • PPT→SPT improves understanding and translation over SPT at matched budgets in both language settings, while combining lower monolingual prediction loss with greater context benefit than SPT.Source: §5 · p. 8
  • Greater retention of correct answers accompanies commonsense-reasoning gains in both language pairs, while subsequent SPT improves PPT’s concept–property judgments.These task-level gains accompany improved monolingual prediction.Source: §5 · p. 8
  • SPT→PPT also improves monolingual prediction and raises corresponding-context benefit, but PPT→SPT has the highest translation average across all tested stages and both language pairs.SPT→PPT has the highest understanding average at 80B and 180B in English–French; both orders reach the same understanding average in English–Swahili.Source: §5 · p. 8

6 Related Work

Related work covers multilingual web translation, parallel-data training objectives, and the use of related documents as context. The paper distinguishes its study by testing document-scale parallel pretraining and both training orders on the same bilingual corpus.

6 Related Work

  • FineTranslations translates non-English FineWeb2 documents into English, producing over one trillion translated tokens (Penedo et al., 2025, 2026); TransWebLLM uses independently sampled English-web translations and reports competitive multilingual performance, including in Swahili (Wang et al., 2025).Translation into other languages also supports pretraining (Doshi et al., 2024; Wang et al., 2025).Source: §6 · p. 8
  • Prior parallel-corpus work includes translation language modeling (Lample and Conneau, 2019), document-level encoder–decoder objectives (Lee et al., 2022), and decoder-only studies of sentence ordering and data composition (Kondo et al., 2024; Lin et al., 2025).Qorib et al. (2025) report their strongest translation results after a short parallel-sentence stage following predominantly English pretraining.Source: §6 · p. 8
  • Related-document grouping improves language modeling and context use (Shi et al., 2024; Staniszewski et al., 2025); this paper holds target content fixed to measure corresponding cross-lingual context’s predictive benefit.Other cited work studies random cross-document context and title-matched multilingual Wikipedia articles.Source: §6 · p. 8

7 Conclusion

The conclusion presents PPT→SPT as a way to use document correspondence from machine-translated corpora while adapting to standard language modeling. The approach improves understanding and translation over standard pretraining at matched budgets in high- and low-resource settings.

7 Conclusion

  • PPT uses document correspondence in machine-translated corpora to improve cross-lingual context use, and following it with SPT improves language understanding and translation over standard pretraining at matched budgets in both resource settings.Behavioral analyses show improved monolingual language modeling while retaining an advantage over SPT in cross-lingual context use.Source: §7 · p. 8

Limitations

The experiments use 1.3B-parameter models and two language pairs; the authors say larger models and more pairs would broaden understanding of the approach. The appendix provides training, evaluation, analysis, and language- and task-level results.

Limitations

  • The study is limited to 1.3B-parameter models and English–French and English–Swahili.The authors identify larger models and additional language pairs as ways to broaden understanding of the two-stage approach.Source: Limitations · p. 9
  • Extending the study to larger models and more language pairs would broaden understanding of how two-stage parallel-data use benefits multilingual pretraining.Source: Limitations · p. 9

Appendix

  • The appendix provides training and evaluation details, prediction and context-use analyses, and results by language and task.Source: Appendix · p. 12

A Model and Training Settings

The model and training settings specify the architecture and optimization setup, checkpointing and continuation protocol, and construction of the SPT and PPT training sequences.

A Model and Training Settings

  • The models use the TransWebLLM architecture and Llama-2 tokenizer, and training uses Megatron-LM with FlashAttention-2.Table 3 lists model and optimization settings; its global batch size counts sequences and sequence length is in tokens.Source: A Model and Training Settings · p. 12, Table 3 · p. 12
  • SPT and PPT start from the same initial weights within each language pair, with base checkpoints saved about every 20B tokens.Staged endpoints follow 20B additional tokens; continued pretraining loads model weights without restoring optimizer or random-number-generator state.Source: A Model and Training Settings · p. 12
  • SPT samples documents independently, whereas PPT pairs documents by shared UUID and separates them with language prefixes and a newline.Training uses 2,048-token windows that can cross document boundaries, with causal attention and position IDs continuing across boundaries.Source: A Model and Training Settings · p. 12

B.1 Language-understanding tasks

These sections specify the evaluation tasks and protocols, paired prediction and context-use analyses, and answer-change and concept–property judgments. The reported analyses compare models and interventions using matched inputs or training budgets.

B.1 Language-understanding tasks; B.2 Translation

  • Language-understanding evaluations cover the listed task families, while translation is evaluated in both directions with corpus BLEU on FLORES-200 and, for English–French, WMT14.Task protocols specify language-specific demonstrations and benchmark averaging; translation averages datasets, directions, and prompts equally within each language pair.Source: §B.1 · p. 12, §B.2 · pp. 12–13

C.1 Prediction conditions and scoring

  • Prediction analyses score identical target continuations under target-only, corresponding-source, and unrelated-source contexts, with paired document comparisons and bootstrap intervals.The English–French analysis uses 173 WMT14 document pairs; English–Swahili uses all 61 AfriDocMT health documents.Source: §C.1 · pp. 13–14

C.2 Paired model comparisons

  • During PPT→SPT, source benefit decreases by 0.005–0.051 BpB, while during SPT→PPT it increases by 0.076–0.134 BpB on 173 WMT14 documents.The staged comparisons use matched token budgets and compare each endpoint with its starting model.Source: §C.2 · p. 14

C.3 Source extension and attention blocking

  • Blocking attention in layers 9–16 or 17–24 produces larger adjusted loss increases for PPT and both combined training orders than for SPT in both languages.Effects from blocking layers 1–8 are smaller; the intervention blocks attention to corresponding source sentences while leaving the full input present.Source: §C.3 · pp. 14–15, Figure 5 · p. 15

C.4 Corresponding-content use in English–Swahili

  • On English–Swahili documents, source extension benefits and layer 9–16 blocking effects are larger for PPT and both combined orders than for SPT.The interventions compare corresponding-source effects against unrelated-source effects on 61 AfriDocMT health documents.Source: §C.4 · p. 15, Table 8 · p. 15

D Judgments on Monolingual Inputs; D.1 HellaSwag answer changes

  • The HellaSwag answer-change analysis rescored the same questions across models and reports corrected and lost answers as shares of all questions.Figure 7's corrected-minus-lost difference gives the accuracy change from 100B to 120B.Source: §D.1 · p. 15, Figure 7 · p. 16

D.2 English–Swahili answer changes

  • The English–Swahili XStoryCloze analysis compares answer changes on all 1,511 paired stories, with corrected and lost answers measured against all questions.The same target items are scored across models; corrected plus retained correct answers gives the 120B accuracy in this scoring run.Source: §D.2 · pp. 15–16

D.3 Concept–property judgments

  • At 120B, COMPS-BASE accuracy is 77.27% for SPT→PPT versus 75.52% for SPT, and 75.15% for PPT→SPT versus 67.96% for PPT.The English minimal-pair evaluation covers 49,340 comparisons across five property categories.Source: §D.3 · p. 16, Table 10 · p. 16

E Detailed Benchmark Results

Tables 11–14 provide the task- and translation-score details underlying the paper’s main result tables, with captions specifying the evaluation scope and averaging.

E Detailed Benchmark Results

Loading 2610.28838v1…