PaperlayerView exact revision on arXiv

Source-linked AI summary

Language ID in the Wild: Unexpected Challenges on the Path to a Thousand-Language Web Text Corpus

Isaac Caswell, Theresa Breiner, Daan van Esch, Ankur Bapna

arXiv:2010.14571v2cs.CLcs.LG

TL;DR

LangID is often treated as solved, yet reliable language identification remains a challenge for building multilingual web corpora from mostly unlabeled pages. The paper analyzes real-crawl failures and develops wordlist filters and semi-supervised neural models; even high-F1 systems were nearly unusable for many low-resource languages, while the proposed improvements produced usable corpora across hundreds of languages.

  • Problem

    High-scoring LangID systems may still fail to produce useful web corpora, where severe class imbalance makes balanced-set precision and F1 misleading.

    Source: §1 · p. 2, §3.1 · p. 4

  • Method

    The authors develop tunable-precision wordlist filtering and semi-supervised neural LangID models, alongside evaluation that accounts for web-mining conditions.

    Source: §1 · p. 2

  • Results

    Models covering up to 1,629 languages with F1 above 95 were nearly unusable in practice for low-resource languages, while the improved system yielded usable monolingual corpora across hundreds of languages.

    Source: §8 · p. 9

  • Takeaways & Limitations

    The authors report that their approaches create cleaner monolingual corpora across hundreds of languages, which hold promise for bringing MT and ASR to more languages.

    Source: §8 · p. 9

    Percent-Threshold Wordlist Filtering does not address mislabeling between languages with high lexical overlap, such as Nigerian Pidgin and English.

    Source: TF-IDF based filtering · p. 6

Abstract

from arXiv · show

Large text corpora are increasingly important for a wide variety of Natural Language Processing (NLP) tasks, and automatic language identification (LangID) is a core technology needed to collect such datasets in a multilingual context. LangID is largely treated as solved in the literature, with models reported that achieve over 90% average F1 on as many as 1,366 languages. We train LangID models on up to 1,629 languages with comparable quality on held-out test sets, but find that human-judged LangID accuracy for web-crawl text corpora created using these models is only around 5% for many lower-resource languages, suggesting a need for more robust evaluation. Further analysis revealed a variety of error modes, arising from domain mismatch, class imbalance, language similarity, and insufficiently expressive models. We propose two classes of techniques to mitigate these errors: wordlist-based tunable-precision filters (for which we release curated lists in about 500 languages) and transformer-based semi-supervised LangID models, which increase median dataset precision from 5.5% to 71.2%. These techniques enable us to create an initial data set covering 100K or more relatively clean sentences in each of 500+ languages, paving the way towards a 1,000-language web text corpus.

1 Introduction

Monolingual text can support language technologies, but building multilingual web corpora depends on reliable LangID because most web pages lack language labels. The paper finds that high-coverage LangID models still fail in real-world crawls and proposes improved evaluation, filtering, and modeling approaches.

1 Introduction

  • Monolingual text is more commonly produced than parallel data and can support language technologies, while parallel resources remain scarce for most languages.Cross-lingual approaches also make monolingual corpora a prerequisite for further work on technologies such as MT and ASR.Source: §1 · pp. 1–2
  • Because most web pages lack language labels, previous efforts have relied on selected URLs and domains or LangID models, despite evidence that web text exists in thousands of languages.The authors therefore trained a large-coverage model and used it in a web crawl, but found it did not deliver useful results in practice.Source: §1 · p. 2
  • The paper argues that LangID is less solved than commonly believed and that popular n-gram methods used for existing web-crawl corpora have serious problems.It categorizes common failure modes and presents tunable-precision wordlist filtering and semi-supervised Transformer models as improvements.Source: §1 · p. 2
  • The work focuses on monolingual corpora, while stating that its problems also apply to parallel texts and that its improvements are straightforward to extend to parallel-data crawling.Source: §1 · p. 2

2 LangID Approaches for Web Corpora

Existing web corpora use varied approaches to language selection, while prior LangID systems report high scores across many languages. The authors build a broad-coverage n-gram model and balance training data to support recall for less common languages, while noting that balanced evaluation can overstate web precision.

2.1 Previous Implementations

  • Earlier web corpora mined Common Crawl for 166 and 174 language varieties, while the authors say depth of mining and LangID robustness may limit their quality.Their final filtered dataset is reported as approximately 20 times larger than CCNet and 180 times larger than OSCAR for shared low-resource languages.Source: §2.1 · p. 3
  • Prior systems reported F1 above 0.95 for 464 languages and accuracy above 99% for as many as 1,366 varieties, but these scores do not ensure useful real-world web corpora.Source: §2.1 · p. 3

2.2 Our LangID Implementation

  • The authors trained a CLD3 n-gram model on aggregated corpora covering 1,629 varieties, with an average of 800K tokens per language.They balanced training data across languages to preserve recall for scarce languages, while warning that balanced evaluation overestimates precision on the imbalanced web.Source: §2.2 · p. 3

3 Failure Modes of LangID Models on Web Text

A large crawl exposed severe noise and unexpected misclassifications despite strong held-out performance. The section traces failures to class imbalance, internet noise, n-gram artifacts, language similarity, unsupported related languages, and unrepresentative training data.

3 Failure Modes of LangID Models on Web Text

  • Even after document-consistency filtering, labels were so noisy that corpora for most crawled languages were unusable for practical NLP tasks.The authors ran a full crawl on approximately 100B documents with a 224-language model after a limited crawl showed high noise; that model had median held-out F1 of 96.8.Source: §3 · pp. 3–4

3.1 Massive Class Imbalances: 99% Accuracy Is Not Enough

  • A model with 99% precision, 99% recall, and 0.01% FPR can yield a web-crawled dataset with just under a tenth of a percent of target-language sentences when only 10,000 of 100 billion pages are in that language.Precision and F1 can mislead when development-set class balance differs from the web; the authors recommend reporting FPR alongside precision and recall.Source: §3.1 · p. 4

3.2 General Internet Noise and Creativity

  • Web-scale noise can overwhelm crawled corpora, with classifiers assigning distinct classes of oddities to languages, including malformed PDFs, invalid HTML, URLs, and garbled JavaScript.The authors also note that script-unique languages can still attract decorative or repurposed characters, although some yield high-precision corpora.Source: §3.2 · p. 5

3.3 Artifacts from Character N-gram Modeling

  • N-gram models can over-trigger on frequent overlapping character sequences: 50% of the crawled Oromo dataset was English sentences containing “essay” at least three times.The section also describes arbitrary n-gram repetition raising class probability and space-separated characters confusing standard n-gram and SentencePiece models.Source: §3.3 · p. 5

3.4 Languages with High-Prevalence Cousins

  • A low-resource language closely related to a prevalent language can be misclassified because of high false-positive rates; English’s prevalence makes high-precision Nigerian Pidgin corpora especially difficult.Source: §3.4 · pp. 5–6

3.5 Languages with Out-of-Model Cousins

  • Languages unsupported by the model can interfere with supported relatives; much of the Uyghur crawl was actually Kazakh and Kyrgyz written in Arabic script.The model had been trained to recognize Kazakh and Kyrgyz only in Cyrillic.Source: §3.5 · p. 6

3.6 Unrepresentative Training Data

  • Training data may be too clean or noisy for web text, and templated or biased data can skew n-gram token distributions away from normal text.Chechen training data represented the Cyrillic letter Palochka with the ASCII digit “1,” so the model may not handle the correct code point or other substitutes well.Source: §3.6 · p. 6

4 Improving LangID Precision on Web Text

Wordlist filters and semi-supervised LangID address different sources of web-text labeling errors, while the semi-supervised models also improve on clean evaluation. The approaches offer ways to raise corpus precision, though wordlist precision comes with a recall trade-off.

4 Improving LangID Precision on Web Text

  • Wordlists can tune corpus cleanliness to downstream noise tolerance, while monolingual web-text corpora affected by LangID errors may be unusable for practical purposes.Token-based filtering can complement character n-gram models and avoid some of their failure modes.Source: §4 · p. 6, §4.1 · p. 6

Percent-Threshold filtering

  • Percent-Threshold Wordlist Filtering removes sentences below a chosen percentage of tokens found in a curated language wordlist.The in-house lists had a median of approximately 15K words per language; the threshold tunes the precision-recall trade-off.Source: Percent-Threshold filtering · p. 6

TF-IDF based filtering

  • Percent-Threshold filtering can retain mislabeled text from lexically similar languages, such as English text labeled Nigerian Pidgin, because shared words satisfy the filter.The authors propose Disjunctive Wordlist Filtering, which keeps a sentence if it contains at least one distinctive language token.Source: TF-IDF based filtering · pp. 6–7
  • TF-IDF treats documents equally, so the authors propose TF-IIF, using token frequency in a language corpus relative to its frequency across the internet.They report slightly improved performance when accounting for both IDF and IIF, producing TF-IDF-IIF scores; released wordlists cover the top 100 tokens for about 500 languages.Source: TF-IDF based filtering · p. 7

4.2 Semi-Supervised LangID

  • SS-LID uses crawled data from 212 languages in a text-to-text self-supervised task co-trained with LangID; a LangID-only Transformer serves as a capacity control.The comparison includes models trained on 212 or 624 languages and scores averaged over the shared 212 languages.Source: §4.2 · p. 7, Table 3 · p. 7
  • 40% reduction in FPR: SS-LID improves on Transformer LangID on the clean evaluation set, despite training its additional objective on noisy web-crawl text.The authors suspect gains may be greater on web-type data, with this partly validated by their web-text evaluation.Source: §4.2 · pp. 7–8, Table 3 · p. 7

5 Evaluating LangID Filtering Methods on Web-Text

The authors recommend evaluating LangID on noisy, representative web data and assessing crawled corpora with human judgments. Their evaluation finds that filtering greatly improves in-language precision, but performance varies by language and some cases remain difficult.

5.1 Evaluation Methodology: Principles and Suggestions

  • The authors recommend human evaluation of crawled text because representative large, noisy LangID test sets are not available.For automatic evaluation, they suggest emphasizing false positive rate and recall, and comparing models by relative error reduction.Source: §5.1 · p. 8

5.2 Evaluating our Systems

  • The evaluation sampled 100–1,000 sentences per language from filtered crawls and asked fluent or closely related-language speakers whether each sentence matched the target language.Wordlist filtering used top-N TF-IDF-IIF words chosen to retain at least 90% held-out recall; percent-threshold filtering used a 20% threshold.Source: §5.2 · p. 8
  • Filtering raised the median share of in-language sentences from 5% to as high as 99%, while maintaining high recall; the best method varied widely by language.The neural SS-LID model performed best for Bhojpuri and Swiss German, but worse than wordlist approaches for Oromo and Cherokee.Source: §5.2 · pp. 8–9
  • The reported Table 4 thresholds were illustrative, and wordlist precision can be increased at the cost of recall according to downstream noise tolerance.The authors note that the chosen thresholds were somewhat arbitrary.Source: §5.2 · p. 9
  • For Guinea-Bissau Creole, none of the filtering methods worked effectively when Portuguese and Papiamentu were both related-language confounders.Swiss German in the same situation barely scraped by; the authors call for additional techniques and note preliminary promise from a hand-curated TF-IDF-IIF list for Nigerian Pidgin.Source: §5.2 · p. 9

6 Web-crawled Dataset and Comparison with other Public Datasets

The authors describe a deep web crawl and a staged filtering pipeline that produced a larger, cleaner collection of monolingual text than similar corpora.

6 Web-crawled Dataset and Comparison with other Public Datasets

  • A crawl of over 100 billion webpages with a 600-language LangID model yielded a recall-focused dataset through percent-threshold filtering, then a larger, cleaner set through SS-LID post-filtering.The paper points to appendices for comparisons with OSCAR and CCNet.Source: §6 · p. 9

7 Future Work

The authors identify broader crawl coverage and additional online domains as directions for future work. They highlight document formats, scripts, and informal orthographies that current crawling and identification approaches may not fully capture.

7 Future Work

  • The authors plan to scale corpus coverage using multilingual LangID models, including a 1,629-language model, after finding usable monolingual text corpora in about 600 languages.Internal user research suggests the web may contain some monolingual text in thousands of languages.Source: §7 · p. 9
  • Broader web coverage may require handling PDF text, non-Unicode fonts, text in images, and informal transliterated orthographies.The authors mention OCR and separate detection and conversion for fonts such as Kruti Dev; their models identify common transliterations but could cover more.Source: §7 · p. 9
  • The work focused on web crawls, while many new internet users use languages online through social media and chat messages.The authors suggest their techniques should also help LangID accuracy in those challenging domains.Source: §7 · p. 9

8 Conclusion

The paper concludes that LangID remains unreliable in real-world web crawling, especially for low-resource languages, despite high held-out scores. It presents tunable-precision filters and semi-supervised neural models as ways to build cleaner monolingual corpora across hundreds of languages.

8 Conclusion

  • Models with over 95 F1 were nearly unusable in practice for low-resource languages, amid natural-language noise, limited n-gram expressiveness, related-language confusion, and class imbalance.The authors note that even 99% F1 can be insufficient under massive class imbalance.Source: §8 · p. 9
  • The authors developed tunable-precision filters, releasing wordlists in about 500 languages, and semi-supervised neural models to address LangID errors.They report creating usable monolingual text corpora across hundreds of languages, with more and cleaner data per language than previously published approaches.Source: §8 · p. 9
  • The authors argue that these corpora hold promise for bringing technologies such as MT and ASR to more languages.They believe the outlined approaches could support monolingual corpora in many more languages.Source: §8 · p. 9

A Complete human evaluation results

The section reports human-judged crawl precision alongside held-out recall and filtering coverage, then illustrates how class imbalance can undermine crawl precision despite strong model metrics.

A Complete human evaluation results

  • Table 5 reports human-judged crawl precision, held-out-evaluation recall, and the percentage of the crawl removed for each filtering method.Precision and crawl removal are measured on the noisy crawl; recall is measured on held-out evaluation data.Source: Table 5 · p. 14, A Complete human evaluation results · p. 14

B Massive Class Imbalance: Worked Example

  • With 99% recall and a 0.01% false-positive rate across an estimated 100 billion web pages, the worked example yields about 0.1% crawl precision.The example assumes 10,000 pages in the target language.Source: B Massive Class Imbalance: Worked… · p. 14
  • The example shows that a very low false-positive rate can still produce many false positives when almost all pages are outside the target language.It estimates 10 million false positives versus 9.9 thousand true positives.Source: B Massive Class Imbalance: Worked… · p. 14

C Statistics on languages most affected by different types of noise

This section quantifies selected web-crawl noise phenomena with simple heuristics, while noting that several other error types are difficult to measure without additional classifiers or human ratings.

C Statistics on languages most affected by different types of noise

  • Some noise types require specialized classifiers or human ratings to quantify, including misrendered PDFs, non-Unicode fonts, creative Unicode use, and out-of-model cousins.The authors describe the listed heuristics as approximations for noise types that can be quantified more readily.Source: C Statistics on languages most… · p. 15
  • The misrendered-PDF heuristic uses characteristic bigrams, implemented as a simple classifier built from known misrendered PDFs.This is one of the section’s specific approximation methods.Source: C Statistics on languages most… · p. 15
  • Table 6 quantifies selected noise phenomena and identifies the languages most affected in the web crawl.The listed heuristics include spaced letters, repeated n-grams, HTML-like strings, URLs, title case, Oromo-specific “essay” matches, and misrendered-PDF patterns.Source: Table 6 · p. 15, C Statistics on languages most… · p. 15

D Details on the web-mined datasets

The section compares the two mined dataset versions with public corpora and explains how the statistics are stratified to better represent low-resource languages.

D Details on the web-mined datasets

  • Table 7 compares the recall-focused and precision-focused mined datasets with CCNet and OSCAR.The precision-focused version is labeled sslid(recall) in the table.Source: D Details on the web-mined datasets · p. 15, Table 7 · p. 16
  • Statistics omit the roughly 100 highest-resource languages in one view and also report languages shared across all three datasets, to better approximate low-resource coverage.The crawl mined a smaller portion of the internet for the highest-resource languages and did not filter them.Source: D Details on the web-mined datasets · p. 15
  • The datasets are difficult to compare with public benchmarks because they cover a wider swath of the internet and are more highly multilingual.The table’s sources should not be treated as evidence about the paper’s filtering methods.Source: D Details on the web-mined datasets · pp. 15–16
  • For low-resource subsets, the public datasets contain 20x to 200x less data than this crawl identified, despite similar-looking full-dataset statistics.This comparison excludes the 100 highest-resource languages or uses languages shared across the datasets.Source: Table 7 · p. 16

E Comparison with OSCAR Corpus

The OSCAR analysis finds wide variation in low-resource corpus precision and recurring n-gram noise, while later sections describe wordlist adaptations, language-specific errors, filtering trends, and practical dataset-building steps.

E Comparison with OSCAR Corpus

  • Repeated n-grams and spaced-letter noise were common in OSCAR; in the cited cases, affected datasets contained only noise and no in-language content.Direct comparison with this paper’s dataset is confounded by different web portions, language subsets, and FastText’s tendency toward lower recall and higher precision.Source: E Comparison with OSCAR Corpus · p. 16, Table 8 · p. 16, Figure 1 · p. 17
  • 0%–100% in-language content across OSCAR’s 30 lowest-resource languages averaged just over 89%, while median dataset size was only 37 sentences.Wordlist-filtered precision correlated with human-judged precision at Pearson’s R of 87.3%.Source: E Comparison with OSCAR Corpus · p. 16, Table 9 · p. 17

F Notes on Curated Wordlist Approaches

  • For unsegmented scripts, curated wordlists include valid characters as a fallback, which can retain reasonable text in another language using the same script.Word segmentation would otherwise be needed to check word-level entries.Source: F Notes on Curated Wordlist Approaches · p. 18

G Wordlist-based Language ID

  • Word-Based LangID counts known words, weighting words unique to one language; it underperforms n-gram LangID on most tested languages but may suit languages lacking sentence-level training data.The authors did not use its outputs as a filter in this work.Source: G Wordlist-based Language ID · p. 18, Table 10 · p. 18

H Illustration of the High-Prevalence-Cousin problem

  • Formal Nigerian Pidgin can resemble English closely enough that LangID and frequent-wordlist filters may select English instead of the target language.The examples are more distinguishable in informal registers.Source: Table 11 · p. 18, H Illustration of the… · p. 18

I Oromo: A Case Study in Unfortunate N-gram Overlap

  • An English 4-gram, “essa,” caused the Oromo model to overtrigger on essay-related English; half of the crawl’s sentences contained “essay” at least three times.The other common crawl n-grams largely reflected English words in sentences about essays.Source: Table 12 · p. 19, I Oromo: A Case Study in Unfortunate… · p. 19

J Correlation of filtering precision with relevant variables

  • With only 17 languages analyzed, both wordlist filters appeared similar, while SS-LID was noticeably better for more confusable languages and possibly slightly worse for larger datasets.The authors present these as trends rather than strong conclusions.Source: J Correlation of filtering precision… · p. 19, Table 13 · p. 20

K Complete Recipe

  • The recipe recommends inspecting language samples and codes, balancing training data, considering transformer LangID and curated wordlists, then applying consistency filtering, deduplication, and precision-tuned filtering.Transformer models are recommended especially when co-trained with a self-supervised objective on in-domain text.Source: K Complete Recipe · pp. 20–21

Related papers

Loading 2010.14571v2…