PaperlayerView exact revision on arXiv

Source-linked AI summary

CommonLID: Re-evaluating State-of-the-Art Language Identification Performance on Web Data

Pedro Ortiz Suarez, Laurie Burchell, Catherine Arnett, Rafael Mosquera-Gómez, Sara Hincapie-Monsalve, Thom Vaughan, Damian Stewart, Malte Ostendorff, Idris Abdulmumin, Vukosi Marivate, Shamsuddeen Hassan Muhammad, Atnafu Lambebo Tonja, Hend Al-Khalifa, Nadia Ghezaiel Hammouda, Verrah Otiende, Tack Hwa Wong, Jakhongir Saydaliev, Melika Nobakhtian, Muhammad Ravi Shulthan Habibi, Chalamalasetti Kranti, Carol Muchemi, Khang Nguyen, Faisal Muhammad Adam, Luis Frentzen Salim, Reem Alqifari, Cynthia Amol, Joseph Marvin Imperial, Ilker Kesen, Ahmad Mustafid, Pavel Stepachev, Leshem Choshen, David Anugraha, Hamada Nayel, Seid Muhie Yimam, Vallerie Alexandra Putra, My Chiffon Nguyen, Azmine Toushik Wasi, Gouthami Vadithya, Rob van der Goot, Lanwenn ar C'horr, Karan Dua, Andrew Yates, Mithil Bangera, Yeshil Bangera, Hitesh Laxmichand Patel, Shu Okabe, Fenal Ashokbhai Ilasariya, Dmitry Gaynullin, Genta Indra Winata, Yiyuan Li, Juan Pablo Martínez, Amit Agarwal, Ikhlasul Akmal Hanif, Raia Abu Ahmad, Esther Adenuga, Filbert Aurelian Tjiaranata, Weerayut Buaphet, Michael Anugraha, Sowmya Vajjala, Benjamin Rice, Azril Hafizi Amirudin, Jesujoba O. Alabi, Srikant Panda, Yassine Toughrai, Bruhan Kyomuhendo, Daniel Ruffinelli, Akshata A, Manuel Goulão, Ej Zhou, Ingrid Gabriela Franco Ramirez, Cristina Aggazzotti, Konstantin Dobler, Jun Kevin, Quentin Pagès, Nicholas Andrews, Nuhu Ibrahim, Mattes Ruckdeschel, Amr Keleg, Mike Zhang, Casper Muziri, Saron Samuel, Sotaro Takeshita, Kun Kerdthaisong, Luca Foppiano, Rasul Dent, Tommaso Green, Ahmad Mustapha Wali, Kamohelo Makaaka, Vicky Feliren, Inshirah Idris, Hande Celikkanat, Abdulhamid Abubakar, Jean Maillard, Benoît Sagot, Thibault Clérice, Kenton Murray, Sarah Luger

arXiv:2601.18026v2cs.CL

TL;DR

LID remains poorly supported for many languages and noisy web data, limiting multilingual corpus curation. The paper introduces CommonLID, a human-annotated web benchmark, and evaluates eight popular models across multiple datasets. No model achieves >75% F1 across all evaluation datasets, showing that LID remains challenging, especially in the long tail.

  • Problem

    LID remains unreliable for many languages and web data, while under-served languages often lack suitable high-quality training resources.

  • Method

    The paper creates CommonLID from human-annotated Common Crawl web text and uses it with other datasets to evaluate eight popular LID models.

  • Results

    >75% F1 is not achieved by any model across all evaluation datasets, even when language coverage is considered.

  • Takeaways & Limitations

    CommonLID supports selecting and developing LID models intended to work better on web data and across under-served languages.

  • Takeaways & Limitations

    CommonLID is biased toward languages, genres, and registers supported by the models and dataset used to select samples.

Abstract

from arXiv · show

Language identification (LID) is a fundamental step in curating multilingual corpora. However, LID models still perform poorly for many languages, especially on the noisy and heterogeneous web data often used to train multilingual language models. In this paper, we introduce CommonLID, a community-driven, human-annotated LID benchmark for the web domain, covering 109 languages. Many of the included languages have been previously under-served, making CommonLID a key resource for developing more representative high-quality text corpora. We show CommonLID's value by using it, alongside five other common evaluation sets, to test eight popular LID models. We analyse our results to situate our contribution and to provide an overview of the state of the art. In particular, we highlight that existing evaluations overestimate LID accuracy for many languages in the web domain. We make CommonLID and the code used to create it available under an open, permissive license.

1 Introduction

LID remains unreliable for many languages and for noisy web data, limiting the creation of representative multilingual datasets. CommonLID addresses this gap with a human-annotated web benchmark and a broad model comparison.

  • Motivation: LID performance remains low for many languages, including on web data and non-standard English text.These conditions are common in web-crawl-based pre-training data.
  • Motivation: Under-served languages are often represented by religious training text, leaving LID systems poorly prepared for other domains and registers.This limits practitioners’ ability to use web data for language technology development.
  • Contribution: CommonLID provides native-speaker, line-level annotations for 109 language varieties from Common Crawl web text.Seventy-eight varieties contain at least 100 lines, and the dataset was created with over eighty annotators.
  • Contribution: CommonLID is designed to reveal where existing evaluations overestimate performance in low-resource, heterogeneous, and noisy web contexts.The dataset and code are released under an open, permissive license.
  • Contribution: Eight popular LID models are evaluated across multiple domains, including web data, to assess current performance.The comparison is intended to support an overview of the state of the art.

2 Issues Affecting LID datasets

Existing LID datasets and training resources have recurring coverage, domain, verification, availability, and size limitations. These mismatches make evaluations less representative of noisy, heterogeneous web data and motivate CommonLID.

  • Training datasets: Popular LID systems often lack open training data, especially for long-tail language coverage.Some systems release data, but open resources for training state-of-the-art models remain limited.
  • Training datasets: Many LID models support no more than approximately 200 languages, reflecting limited coverage in widely used multilingual datasets.Newer global and region-specific corpora seek to improve representation for under-served languages.
  • Training datasets: Religious text is the primary training source for many low-resource languages, limiting domain generalisation and biasing classification of heterogeneous web data.GlotLID, for example, has been reported to favor religious text when classifying heterogeneous web data.
  • Training datasets: Many corpora omit web data because it is noisy, relying instead on cleaner sources whose styles and registers differ from the web.Professionally translated web data is a notable exception.
  • Training datasets: Automatically assigned language labels in large datasets are often not human verified, raising reliability concerns.Prior work found such labels to be inaccurate in many large-scale datasets.
  • Evaluation datasets: Evaluation sets commonly use clean, high-quality data that lacks web artifacts and may not indicate performance on web text.FLORES and UDHR are examples of professionally translated, non-fiction or legal-domain datasets.
  • Evaluation datasets: Some evaluation datasets are unavailable or have lost permission for use, increasing the importance of open resources for replicable research.Examples include the AfroLID test split, JW300, and Twitter.
  • Evaluation datasets: Key datasets can contain few items per language, while larger Bible datasets provide coverage from a limited domain and often overlap with training data.UDHR contains approximately 90 lines per language.

3 Annotation Collection/Dataset Creation

CommonLID was assembled through community-driven sampling, recruitment, and annotation of multilingual web data. Existing LID systems guided sample selection, enabling representation of under-served languages but introducing a documented selection bias.

  • Collection process: The collection process sampled multilingual web data, recruited annotators, and gathered annotations through a custom interface and hosted hackathons.The work used recent filtered Common Crawl crawls and MADLAD-400.
  • Web sampling: Common Crawl samples were selected with fastText, OpenLID, and GlotLID, using WET files from two recent crawls.Up to 1,000 documents per language were sampled from each crawl for each model.
  • Web sampling: MADLAD-400 clean and noisy splits were added to increase data for less-represented languages and diversify the web-data LID architecture used for selection.The sampling produced 4,000 documents per language.
  • Limitation: Selecting samples with existing LID systems means that already-recognized content is more likely to enter CommonLID.The authors describe this as a necessary compromise for representing under-represented languages at web scale.
  • Recruitment: NLP-community recruitment and regional partnerships were used to engage contributors with relevant language expertise.The project used social media, NLP Discord servers, grass-roots organizations, and virtual hackathons.
  • Recruitment: Community contributors were reported to produce higher-quality annotations than might be expected from crowd-sourced workers.The authors attribute this to contributors’ understanding of the value of annotated data in their languages.
  • Annotation: Annotators selected languages, highlighted text written in them, and could annotate at line or character level when complete-line identification was impossible.Large documents were truncated to keep the interface responsive.
  • Participation: Authorship incentives recognized contributors’ intellectual contributions and supported the project’s participatory-design goals.Contributors generally annotated 100–200 documents, while some annotated more than 1,000.

4 CommonLID dataset

CommonLID was cleaned into a line-level web corpus while retaining difficult multilingual cases and documenting annotation disagreement. The final dataset contains 373,230 lines across 109 language varieties, with uneven class sizes and limited multi-annotator coverage.

  • Dataset preparation: The corpus was split on newlines, deduplicated at the label-line level, and filtered using length, English-prediction, and manual quality checks.Levenshtein distance supported fuzzy splitting, while three spurious language classes were removed after auditing.
  • Dataset preparation: Multiple valid labels were retained for lines with macro- and micro-language labels or without a dominant language, creating a key limitation of the labeling scheme.These cases were kept as separate annotated copies rather than reconciled into one label.
  • Inter-annotator agreement: Only 12.9% of dataset lines had more than one annotator, limiting the scope of direct agreement estimates.Agreement statistics were calculated after filtering short spans but before applying later quality filters.
  • Inter-annotator agreement: 2.3% of multiply annotated lines remained genuine annotator disagreement after filtering mislabeled English boilerplate.This corresponds to 1,524 lines; Arabic label distinctions accounted for many disagreements.
  • Dataset description: 373,230 lines span 109 language varieties, with 78 classes containing more than 100 examples and four classes containing one line each.The largest class contains 43,189 lines, while the macro-average is 3424.1 lines per class.

5 Evaluation

The evaluation compares eight LID models across multiple datasets, using harmonized labels and macro-averaged metrics to address unequal language coverage. It shows that coverage affects apparent performance and that lower-coverage models can sometimes be preferable when accuracy on supported languages matters.

  • Evaluation setup: Eight widely used LID models were evaluated on CommonLID and comparable datasets, with raw scores and analysis code released for reproducibility.The models include AfroLID, CLD2, fasttext, FUN-LangID, pyFranc, CLD3, GlotLID v4, and OpenLID-v2.
  • Evaluation setup: The evaluation includes openly available datasets plus restricted Bible and social-media datasets selected for language coverage and colloquial web-like text.The social-media set contributes colloquial text, while the Bible set provides unusually broad language coverage.
  • Evaluation setup: 300 lines were sampled per remaining class in three imbalanced datasets after discarding classes with fewer than 300 examples.This reduced inference-time effects from large classes and avoided unrepresentative results for the smallest classes.
  • Evaluation setup: Labels were normalized toward ISO 639-3, and unresolved or non-single-language outputs were discarded before comparison.Automatic normalization can miss comparable macro- and micro-language codes, but manual reconciliation across schemes was infeasible.
  • Metrics and comparison: Macro-averaged F1 and false positive rate give less-resourced language varieties equal weight in the reported averages.Table 1 reports both whole-dataset scores, where uncovered languages score zero, and covered-subset scores.
  • Metrics and comparison: Higher language coverage can inflate simple average scores, while low accuracy on less-resourced languages can produce unusable labeled data downstream.The paper therefore notes that a more reliable lower-coverage model may be preferable for some applications.

6 Results

The evaluation compares LID models across datasets, coverage, inference speed, and language groups. CommonLID is challenging, while apparent strengths on other datasets may reflect training–evaluation overlap.

  • CommonLID results: Most models achieve F1 scores in the 60s on CommonLID, averaged over the language varieties they cover.The authors interpret these low scores as evidence that CommonLID is a challenging web-domain evaluation dataset.
  • Evaluation caveats: Higher scores on some datasets may reflect overlap between model training data and evaluation data, especially for long-tail languages.The paper notes examples involving UDHR, Bible data, and shared development teams, while exact training data are unknown for FUN-LangID and SmolSent.
  • Evaluation caveats: High scores do not necessarily reflect real-world performance, motivating more diverse and independent evaluation data for long-tail languages.
  • Model trade-offs: GlotLID combines strong performance and broad language coverage with slower inference, while CLD2 and GlotLID lie on the compute–performance Pareto frontier.Figure 2 compares inference speed and F1 on 76 shared core languages across combined datasets.
  • Comparison setup: The comparison evaluates models on shared language subsets, including high-coverage, African-language, and 76-language core comparisons.The paper cautions that only metrics in the same table column should be compared.
  • LLMs for LID: GPT models are outperformed by GlotLID, with GPT-5 trailing by 1.8% F1 on core languages and 30% F1 on African languages.The LLM evaluation uses 15k down-sampled test samples and compares GPT models with GlotLID as the baseline.

7 Discussion

The discussion presents CommonLID as an expanded web-domain resource and examines persistent challenges in comparing and evaluating LID models. It concludes that long-tail performance remains difficult to assess reliably and that no model is clearly best across settings.

  • Dataset contribution: CommonLID expands web-domain LID evaluation data through native-speaker annotations of texts originally written in the target language.The dataset especially increases evaluation availability for low-resource languages and complements efforts using translated texts.
  • Dataset limitations: The annotation process was constrained by existing model and dataset coverage, making unsupported languages difficult to collect.Some contributors had few documents available during hackathons, limiting the amount they could annotate.
  • Comparison challenges: Fair model comparison is difficult because systems support different language sets and language-variety boundaries can be contentious.
  • Evaluation limitations: For many long-tail languages, the absence of distinct test data makes meaningful evaluation impossible and can overestimate performance.Training data are often sparse and limited to religious text, leaving no independent test split.
  • State of the art: CLD2 and GlotLID perform best overall when coverage and inference speed are considered, but CLD2 lacks lower-resource language support.GlotLID performs best on FLORES and Bible datasets, where training–evaluation similarity may inflate long-tail results.
  • State of the art: No model achieves >75% F1 across all evaluation datasets, and there is no clear top-performing model under meaningful comparison.The paper identifies CommonLID as useful for selecting and developing models intended for web data.

8 Conclusion

The paper presents CommonLID as a web-domain LID evaluation dataset created through collaborative annotation and evaluates it against popular models and existing benchmarks. Its data, analysis, and code are released openly to support further research.

  • CommonLID is a web-domain LID evaluation dataset built through highly collaborative annotation involving multiple language communities.
  • The authors test eight popular LID models on CommonLID and other common evaluation datasets to assess the state of the art.
  • The dataset and code are released under an open, permissive license for community use.

Limitations

The paper’s limitations concern dataset-selection bias, unresolved ambiguity in language labels, potential harmful web content, and limited inter-annotator agreement. These constraints reflect the breadth of the target languages and the use of volunteer annotation.

  • Selecting samples with existing LID systems biases CommonLID toward languages, genres, and registers recognized by those systems.
  • Multiple valid labels and macro- versus microlanguage distinctions can make language annotation ambiguous even for native speakers.
  • Common Crawl data may contain personally identifiable information or harmful content despite heuristic filtering and participant reporting mechanisms.
  • Only 12.9% of dataset lines were labeled by more than one annotator, preventing a full inter-annotator agreement study.

C Examples of CommonLID Errors

Table 5 presents examples from CommonLID that GlotLID misclassified, illustrating concrete model errors on the benchmark.

  • Table 5 lists CommonLID examples that GlotLID misclassified.
  • The examples provide concrete instances of model errors on the CommonLID dataset.
  • The table focuses on error examples rather than aggregate evaluation scores.

D Additional Model Details

This section supplies model summaries, coverage comparisons, and inference-speed measurements, alongside annotation-interface details and contributor instructions. Together, these materials document both evaluation models and the data-collection workflow.

  • Additional Model Details: Table 6 summarizes the LID models used in the evaluation.
  • Additional Model Details: Table 7 reports mutual language coverage between LID models, while Table 8 reports their inference speed on FLORES+.
  • Annotation Instructions: Annotators select a proficient language, inspect processed web passages, and highlight spans written in that language.
  • Annotation Instructions: The interface supports optional tags, language switching, and reporting of incorrectly labeled samples.
  • Annotation Instructions: Contributors completing at least 100 annotations are invited to become co-authors of the scientific paper.

F Evaluation Dataset Details

The evaluation sets span formal multilingual benchmarks, web-derived translated sentences, religious text, and challenging social-media data. Their coverage and training-data overlap create important differences in what LID performance measures.

  • FLORES+: FLORES+ contains 997 parallel dev sentences translated into 222 language varieties, but its text is relatively formal.It is sourced from Wikitext articles and is widely used because of its quality and coverage.
  • UHDR-LID: UHDR-LID covers 374 languages with cleaned Universal Declaration of Human Rights translations, but its use in multilingual training data can inflate evaluation scores.Its high coverage and open availability make it attractive as an evaluation set while also creating training-test overlap.
  • SmolSent: SmolSent uses 863 English sentences professionally translated into 88 under-served languages, with vocabulary coverage and web-domain content suited to evaluation.The source sentences come from Common Crawl and target common English-token coverage.
  • Social media: The social-media dataset contains 169,019 lines across 97 language varieties and includes emoji, URLs, non-standard orthography, and hashtags.These phenomena make it a challenging test of robustness on informal web text.
Loading 2601.18026v2…