PaperlayerView exact revision on arXiv

Source-linked AI summary

Language Contamination Helps Explain the Cross-lingual Capabilities of English Pretrained Models

Terra Blevins, Luke Zettlemoyer

arXiv:2204.08110v4cs.CL

TL;DR

The paper investigates how much foreign-language data appears in English pretraining corpora and whether it helps explain cross-lingual transfer. It estimates corpus language composition and evaluates models across languages, finding non-English exposure and transfer that is not truly zero-shot. The authors note that classifier-based data estimates and POS-only downstream evaluation constrain the findings.

  • Problem

    Because pretraining corpora are large and often unavailable, the amount of foreign-language data in nominally monolingual corpora is unknown, despite reported cross-language transfer.

    Source: §1 · p. 1, §2.1 · p. 2

  • Method

    The authors estimate corpus language composition with automatic and manual analyses, then test models on masked language modeling and POS tagging across 50 languages.

    Source: §1 · p. 1, §2.2 · p. 3

  • Results

    English pretrained models encounter considerable non-English data, which acts as a significant signal for cross-lingual transfer; the transfer is not zero-shot.

    Source: §1 · pp. 1–2, §4 · pp. 4–5

  • Takeaways & Limitations

    Cross-lingual evaluation should account for foreign-language exposure during pretraining, since these models are not truly monolingual or zero-shot.

    Source: §1 · p. 2, §4 · p. 5

    Non-English data quantities are classifier-based estimates, and downstream evaluation uses POS tagging, so behavior on other NLP tasks may differ.

    Source: §5 · p. 5

Abstract

from arXiv · show

English pretrained language models, which make up the backbone of many modern NLP systems, require huge amounts of unlabeled training data. These models are generally presented as being trained only on English text but have been found to transfer surprisingly well to other languages. We investigate this phenomenon and find that common English pretraining corpora actually contain significant amounts of non-English text: even when less than 1% of data is not English (well within the error rate of strong language classifiers), this leads to hundreds of millions of foreign language tokens in large-scale datasets. We then demonstrate that even these small percentages of non-English data facilitate cross-lingual transfer for models trained on them, with target language performance strongly correlated to the amount of in-language data seen during pretraining. In light of these findings, we argue that no model is truly monolingual when pretrained at scale, which should be considered when evaluating cross-lingual transfer.

1 Introduction

The extent of foreign-language text in monolingual pretraining corpora, and its role in cross-lingual transfer, are not well established. The paper estimates corpus language composition and tests model performance across languages, finding that small amounts of non-English pretraining data accompany transfer that is not truly zero-shot.

1 Introduction

  • The paper finds that English pretrained models encounter non-English data and that this leakage can support cross-lingual transfer.The authors question whether fully removing non-English text from pretraining at scale is practically possible.Source: §1 · pp. 1–2
  • The study estimates language composition in common English pretraining corpora and evaluates models on masked language modeling and POS tagging across 50 languages.It combines large-scale automatic evaluation with a smaller manual analysis.Source: §1 · p. 1
  • Small percentages of non-English text can amount to substantial quantities, particularly in web-crawled corpora.The paper reports that English T5 outperforms mBERT on POS tagging in multiple languages without finetuning.Source: §1 · pp. 1–2, Figure 1 · p. 1
  • The authors conclude that cross-lingual transfer from these models is not zero-shot because their pretraining data includes other languages.They argue that truly monolingual pretraining at scale may be practically difficult.Source: §1 · p. 2

2 Pretraining Data Composition

The paper estimates foreign-language content in commonly used English pretraining corpora through automatic identification and manual review. It finds that leakage varies with corpus size and collection method, and that classifier predictions include both genuine foreign-language material and errors.

2 Pretraining Data Composition

  • The authors estimate non-English content using automatic language identification and manually review a sample of lines classified as non-English.The corpora include English Wikipedia, BookCorpus, Stories, OpenWebText, CC-NEWS, and C4.EN.Source: §2 · p. 2

2.1 Automatic Evaluation of Language Composition

  • 300k to 406M non-English tokens were estimated across the corpora, with smaller corpora having relatively less non-English data.The estimates use FastText with a 0.6 confidence threshold; C4 was subsampled to its first 50M examples.Source: §2.1 · p. 2
  • Web-crawled datasets contain higher percentages of non-English text, while manually curated Wikipedia is less prone to it.The authors note that imperfect language identification makes fully removing non-English text from web-crawled data at scale likely impossible.Source: §2.1 · p. 2
  • Non-English text remains a small share of total data, but large corpora can contain millions of tokens even at those percentages.The largest individual languages after English account for 0.01% of BERT, 0.15% of RoBERTa, and 0.05% of T5 training data.Source: §2.1 · p. 2

2.2 Qualitative Analysis of Non-English Texts

  • Most sampled lines classified as non-English contain only foreign-language text; bilingual and parallel material, English text with foreign entities, and classifier errors also occur.The manual review coded 200 predicted non-English lines per corpus into six categories.Source: Table 1 · p. 2, §2.2 · p. 3
  • The classifier misclassified a quarter of sampled Stories lines as non-English even though they contained only English text.Errors also tended to involve short, non-standard English, while noisy lines could lack an appropriate language label.Source: §2.2 · p. 3

3 Cross-lingual Transfer of English Pretrained Models

The section evaluates English pretrained models on language modeling and POS tasks across more than 50 languages. Transfer varies by model and task, but target-language pretraining data is more strongly associated with performance than language similarity.

3.1 Non-English MLM Evaluation

  • RoBERTa reduces the gap with multilingual models from 2.51 BPC to 0.87 BPC when modeling other languages.Bits per character allows comparison across models with different tokenizations.Source: §3.1 · p. 3

3.2 POS Performance Across Languages

  • English models underperform multilingual models on average in POS probing, while RoBERTa beats BERT and T5 despite C4's larger absolute foreign-text quantity.RoBERTa's relative non-English exposure is 0.78% versus T5's 0.22%; its subword vocabulary also avoids UNK tokens, unlike T5 and BERT for some non-Latin languages.Source: §3.2 · p. 4
  • 2.65 points worse than XLM-R after fine-tuning, RoBERTa's average gap is smaller than the 12.5-point gap when probing.The fine-tuning evaluation covers BERTbase, RoBERTabase, mBERT, and XLMRbase.Source: §3.2 · p. 4

3.3 Potential Reasons for Cross-lingual Generalization

  • Across tasks, RoBERTa performance is most strongly correlated with the amount of target-language data seen during pretraining.For T5, controlling for languages not written with Latin script raises the correlation to ρ = 0.313; language similarity is generally less correlated with performance.Source: §3.3 · p. 4

4 Discussion

The discussion connects foreign-language exposure in English pretraining corpora with cross-lingual transfer, especially for models trained on large web-crawled datasets. It cautions that this transfer should not be treated as zero-shot and that controlling pretraining language is difficult at scale.

4 Discussion

  • English pretrained models encounter considerable non-English data, particularly recent models trained on larger web-crawled corpora, and that text is a significant transfer signal.The authors state that foreign-language data correlates directly with cross-lingual transfer.Source: §4 · p. 4
  • Prior studies report cross-lingual transfer by monolingual models, but do not examine foreign-language leakage as a signal; de Souza et al. (2021) note leakage but assume small amounts do not affect performance.The authors report that their findings show foreign-language amounts directly correlate with transfer.Source: §4 · p. 5
  • Retraining on text verified to contain only English is proposed as a follow-up to confirm the effect of leaked non-English data.Source: §4 · p. 5
  • Classifier filtering is imperfect and manual filtering is infeasible at data scale, making language-controlled pretraining nearly impossible.Source: §4 · p. 5
  • Foreign-language pretraining data is not inherently problematic, but cross-lingual performance should not be treated as zero-shot.The authors note that models trained on these datasets perform well on their target languages and generalize to other languages better than expected.Source: §4 · p. 5

5 Limitations

The reported foreign-language quantities are classifier estimates, downstream evaluation is limited to POS tagging, and the experiments cover English pretrained models only.

5 Limitations

  • Non-English data quantities are estimates from a language classifier that likely misclassified some examples.Source: §5 · p. 5
  • Downstream performance is evaluated via POS tagging, and models may behave differently on other NLP tasks.Source: §5 · p. 5
  • The study considers English pretrained models only; the extent of similar effects in other-language monolingual models is unclear.The authors note that many evaluated resources, such as Wikipedia, are also used to pretrain non-English monolingual models, so similar effects would likely be observed.Source: §5 · p. 5

A Details of Transfer Experiments

The transfer experiments measure zero-shot masked-language-modeling performance and evaluate POS tagging with both frozen representations and finetuned encoders.

A Details of Transfer Experiments

  • Whole-word masking of 15% of Wiki40B test-set words is used to calculate BPC without further model training.The language-modeling experiment is zero-shot.Source: A Details of Transfer Experiments · p. 6
  • POS probing uses a linear classifier on each encoder’s final layer, with averaged subword representations for words split into multiple units.The probe has m*l parameters, where m is the encoder output dimension and l is the 17-label POS set.Source: A Details of Transfer Experiments · p. 6
  • For finetuned POS tagging, the encoder weights are unfrozen, and models are trained and evaluated on Universal Dependencies treebanks.The experiments use different batch sizes and learning rates for frozen and finetuned settings, without tuning those parameters.Source: A Details of Transfer Experiments · p. 7
  • Experiments run on a single Nvidia V100 GPU, with frozen probes taking under 1–8 minutes and finetuned probes taking 5 minutes to 7.5 hours.Training time varies with dataset size and early stopping epoch.Source: A Details of Transfer Experiments · p. 7

B The Effect of Tokenization

The section compares how model tokenizers encode languages and explains how unknown-token rates can affect reported language-modeling and downstream performance.

B The Effect of Tokenization

  • All considered models use more subword tokens per word for non-English languages; T5 is more efficient on French, German, and Romanian than other monolingual models.The paper attributes this likely to T5’s tokenizer training on English mixed with those languages.Source: B The Effect of Tokenization · p. 7
  • BERT and T5 have UNK rates above 10% for many non-Latin-script languages, while byte-level BPE produces no UNK tokens for Unicode text.SentencePiece tokenizers used by BERT and T5 mark tokens absent from tokenizer training as UNK; other tokenizer decisions may also affect these rates.Source: B The Effect of Tokenization · p. 7
  • Japanese has many more subword tokens per whitespace-separated word because Japanese words are not separated by whitespace.Source: B The Effect of Tokenization · p. 7
  • High UNK rates likely degrade downstream performance, including POS tagging, and can make language modeling artificially easy by lowering BPC.The paper marks BPC results with an asterisk when more than 10% of the considered data is UNKed.Source: B The Effect of Tokenization · p. 7

C Full Results of the Automatic Language Identity Analysis

The section provides the complete automatic language-composition analysis and specifies which languages are included in its reported results.

C Full Results of the Automatic Language Identity Analysis

D Full Results of Transfer Experiments

The appendix directs readers to full tables covering language composition, tokenization, masked language modeling, and POS tagging results.

D Full Results of Transfer Experiments

Related papers

Loading 2204.08110v4…