Source-linked AI summary
OpenLID-v3: Improving the Precision of Closely Related Language Identification -- An Experience Report
Mariia Fedorova, Nikolay Arefyev, Maja Buljan, Jindřich Helcl, Stephan Oepen, Egil Rønningstad, Yves Scherrer
TL;DR
LID systems must handle noisy web data and closely related languages without contaminating language-specific datasets. The paper develops OpenLID-v3 through expanded data, revised language labels, and a noise class, finding comparable or better precision and best precision with top-1 ensembling. Evaluation also shows that language-specific benchmarks are needed, while the available evaluation data do not fully match intended web use.
Problem
Existing LID systems face challenges with closely related languages and non-language content, risking contamination of language-specific datasets, especially for low-resource languages.
Method
The authors extend OpenLID with additional training data, revised language groupings, a not-a-language class, and evaluations using single-label and multilabel metrics.
Results
OpenLID-v3 performs on par or better in precision than OpenLID-v2 and GlotLID, while top-1 ensembling achieves the best precision.
Takeaways & Limitations
Reliable LID evaluation for similar languages requires benchmarks specific to those languages, and short web texts motivate multilabel training data and benchmarks.
Takeaways & Limitations
The evaluation data did not fully match the intended web-text use because suitable large-scale web LID data were unavailable during the experiment.
Abstract
from arXiv · showhide
Language identification (LID) is an essential step in building high-quality multilingual datasets from web data. Existing LID tools (such as OpenLID or GlotLID) often struggle to identify closely related languages and to distinguish valid natural language from noise, which contaminates language-specific subsets, especially for low-resource languages. In this work we extend the OpenLID classifier by adding more training data, merging problematic language variant clusters, and introducing a special label for marking noise. We call this extended system OpenLID-v3 and evaluate it against GlotLID on multiple benchmarks. During development, we focus on three groups of closely related languages (Bosnian, Croatian, and Serbian; Romance varieties of Northern Italy and Southern France; and Scandinavian languages) and contribute new evaluation datasets where existing ones are inadequate. We find that ensemble approaches improve precision but also substantially reduce coverage for low-resource languages. OpenLID-v3 is available on https://huggingface.co/HPLT/OpenLID-v3.
1 Introduction
The paper examines language identification challenges in noisy web documents, emphasizing closely related languages, and introduces OpenLID-v3 with expanded coverage, new evaluation resources, and a noise label.
- The study focuses on LID performance and challenges in noisy web documents, particularly for selected groups of closely related languages.
- OpenLID-v3 extends the open-source system to 194 languages plus a ‘not-alanguage’ class.
- The authors evaluate OpenLID-v3 on mainstream and closely related-language benchmarks, including new datasets for BCMS and Norwegian Bokmål and Nynorsk.
- The paper reports negative results for a two-step coarse-to-fine classification approach.
2 Related Work
Related work establishes LID as essential for multilingual data quality, reviews existing systems and benchmarks, and motivates evaluations that better reflect noisy web text and closely related languages.
- Accurate LID is essential because incorrect assignments can severely contaminate language-specific datasets, especially for low-resource languages.
- FastText-based classifiers are widely used in large-scale multilingual corpus processing because they combine efficiency with strong cross-language performance.
- OpenLID and GlotLID are fastText-based multilingual systems that differ mainly in language coverage and training-data selection.
- FLORES+ and UDHR are widely adopted benchmarks, while FastSpell specifically targets discrimination between closely related languages in web documents.
- Because real web crawls have severe language imbalance, precision and F1 can be misleading; recall and FPR are less susceptible to class imbalance.
- The evaluation reports FPR, precision, recall, and F1 for single-label benchmarks, plus loose and exact-match metrics for multilabel benchmarks.
3 OpenLID-v3
OpenLID-v3 addresses inventory, training-data, and closely related-language problems through targeted label and data changes, then compares thresholding and ensemble strategies across benchmarks.
- The system restores support for Latin, which had been present in OpenLID-v1 but removed in v2.
- OpenLID-v2 misclassified Serbian written in Latin script as Bosnian or Croatian, and manual inspection found half of HPLT 3.0 Bosnian was Serbian in Latin script.
- OpenLID-v3 introduces zxx_Zxxx for not-a-language content, addressing the prior tendency to assign code, broken encoding, and out-of-inventory language to existing classes.
- The development process merged several closely related variants, including eight Arabic dialects, two Persian varieties, and mutually intelligible Bambara and Dyula.
- OpenLID-v3 adds training data, including Latin and Serbian in Latin script, while removing a possibly noisy training subset and adding recent Wikipedia dumps.
- OpenLID-v3 is on par with OpenLID-v2 and GlotLID, with thresholding slightly favoring OpenLID-v3 and no thresholding slightly favoring GlotLID.
- Top-1 ensembling produced the lowest FPR across all datasets and was used to produce HPLT 4.0 datasets.
4 Case Studies on Related languages
Across closely related language groups, model performance depends strongly on benchmark design and the trade-off between precision and coverage. Error analyses identify ambiguity, shared lexical or historical forms, named entities, nonstandard syntax, and short noisy sequences as recurring challenges.
- Bosnian, Croatian and Serbian: OpenLID-v3 performed best for Bosnian and Serbian on Twitter, while GlotLID and OpenLID-v3 consistently disagreed.OpenLID-v2 achieved slightly higher Croatian precision but lacked a Serbian class label; OpenLID-v3 also performed best on reannotated HPLT-LID, where Bosnian–Serbian confusion remained high.
- Bosnian, Croatian and Serbian: Common roots and shared lexemes can outweigh Serbian jat orthography as a classification signal.Bosnian and Serbian documents were often confused despite clear e/ije indicators when they also contained overlapping roots and inflections.
- Bosnian, Croatian and Serbian: Historic Serbo-Croatian forms frequently caused Croatian documents to be mislabeled as Bosnian or Serbian.This occurred especially when older speakers or colloquial language used lexemes absent from current standard Croatian, despite otherwise unambiguous grammatical markers.
- Bosnian, Croatian and Serbian: Nonstandard syntax was often labeled not-a-language when documents lacked distinguishing grammatical or lexical markers.The relevant texts included run-on sentences, missing punctuation, incorrect syntax, interjections, and highly colloquial wording.
- Romance languages of Italy and France: 31% of UDHR samples annotated as Occitan were predicted as Ligurian because four of seven translations were actually Francoprovençal, absent from OpenLID-v2.This case shows why multilingual benchmarks can be inadequate for closely related, unstandardized, and low-resource languages.
- Romance languages of Italy and France: Ensembling achieved the best precision and lowest FPR for the Italian Romance evaluation, but removed 45% of Venetian samples.GlotLID alone had higher precision and recall than OpenLID-v3, while the ensemble retained roughly similar Ligurian and Friulian counts.
- Scandinavian languages: For Scandinavian identification, GlotLID provided the best recall, OpenLID-v3 better FPR and precision, and ensembling further improved precision and FPR.Bokmål and Nynorsk were the most easily confused varieties; the authors recommend a dedicated model for separating them.
- Scandinavian languages: Short web sequences were especially difficult because named entities, dates, and foreign words often lacked language-specific markers.The authors recommend OpenLID-v3 when Scandinavian precision matters, GlotLID when recall matters, and a specialized Bokmål–Nynorsk model for that distinction.
5 Conclusion
The study evaluates three LID models across four multilingual benchmarks, finding that OpenLID-v3 is at least as precise as its predecessors while top-1 ensembling with GlotLID yields the highest precision. It also shows that reliable evaluation of similar languages requires language-specific benchmarks.
- Three LID models were evaluated on four multilingual benchmarks spanning clean text and web documents.
- OpenLID-v3 performs on par or better in precision than OpenLID-v2 and GlotLID.
- Top-1 ensembling with GlotLID achieves the best precision.
- Reliable evaluation of LID models for similar languages requires benchmarks specific to those languages, even when aggregate metrics are high.
Limitations
The evaluation does not fully match the intended use on web text, and contamination could not be controlled for the heavily preprocessed Nordic DSL dataset. Across models, unavailable fully parallel data may also encourage overfitting to language-specific concepts and named entities.
- The evaluation does not fully match the intended use because sufficiently high-quality large-scale web LID data were unavailable.
- Training-test contamination could not be controlled for Nordic DSL because its Wikipedia data were heavily preprocessed.
- All evaluated models lack fully parallel datasets for every language of interest, which might cause overfitting to language-specific concepts and named entities.
Ethical Considerations
The authors report that their new annotations were produced voluntarily without monetary compensation. They also did not check OpenLID-v3’s training data for inappropriate or biased content, leaving possible prediction bias when representations are used semantically.
- The authors created the new data annotations voluntarily and without monetary compensation.
- OpenLID-v3’s training data were not checked for inappropriate or biased content.
- The authors consider possible prediction bias particularly relevant if OpenLID-v3 is used as a source of semantic representations rather than only as a classifier.
- Instruction-tuned generative-model research may risk losing data from low-resource varieties and valuable cultural knowledge.
A Multilingual Benchmarks
The benchmark evidence combines multilingual datasets with a cleaned and preprocessed HPLT-LID resource. OpenLID-v3 shows no performance regression against OpenLID-v2 on HPLT-LID, but HPLT-LID cannot evaluate noisy web data directly.
- Evaluation metrics were computed only for languages present in every compared model, using GlotLID-v3 as its latest available version.
- HPLT-LID accepts samples excluding unnatural language, pornography, web artifacts, and incorrect LID.
- HPLT-LID contains web texts but cannot benchmark noisy web data because the texts were heavily cleaned and preprocessed before annotation.
- OpenLID-v3 shows no regression compared with OpenLID-v2 on HPLT-LID.
B Additional Data Sources
OpenLID-v3 expands its training-data design beyond the previous inventory, while adding resources for languages and non-linguistic material. The section also documents constraints on interpreting these sources and benchmarks.
- OpenLID-v3 compares its training-data sources with OpenLID-v2 and considers languages outside the earlier class inventory.
- HPLT-LID is not a noisy-web benchmark because its documents were heavily cleaned and preprocessed before annotation.
- 366 languages remained after unreliable sources were removed, including 150 with more samples than OpenLID-v2’s smallest language, Yiddish.Yiddish had 923 samples; Low German had 117,676 samples.
- The authors argue that additional nonstandard, historical, and constructed languages may merit inclusion despite uncertain prevalence in web crawls.
- The data design distinguishes random script sequences from web-collected non-linguistic noise in different scripts.
C Bosnian, Croatian, Montenegrin and Serbian
The BCMS evaluation exposes annotation and script-related difficulties in distinguishing closely related varieties. The authors therefore rely on carefully qualified benchmark use and reannotation evidence.
- SETimes contains roughly 9,000 parallel documents in each South-Eastern European language, but OpenLID evaluation was excluded because deduplication failed.GlotLID results were reported instead; longer documents generally supported better classification.
- Heritage BCS was unsuitable for meaningful LID evaluation because its bilingual-speaker transcripts contain too much second-language interference.
- Some documents labeled Montenegrin or Chakavian by GlotLID exhibited nonstandard syntax and repetitive conjunctions or adjectives.The latter patterns were associated with Chakavian and Montenegrin poetry in the training data.
- OpenLID-v2 predictions labeled 201 samples as Serbian Cyrillic; 151 were missing annotations rather than false positives, while 21 were tagged translationese.Three samples were valid only in Montenegrin and received the cnr_Latn label.
- The appendix documents URLs for Francoprovençal and Occitan UDHR translations used to examine language-label confusion.
E Scandinavian Languages
The Scandinavian-language evaluation is organized around ensemble and thresholding comparisons across multiple test datasets.
- The experiments compare GlotLID and OpenLID-v3 ensembles, alongside different OpenLID versions, across Scandinavian and other test sets.Comparisons include SLIDE, Nordic DSL, FLORES+ devtest, and UDHR.
F Hierarchical Models
The hierarchical-model experiments specialize classifiers for related language groups and replace base-model predictions within those groups. They find no significant performance advantage over OpenLID-v3 on FLORES+ or UDHR.
- Hierarchical Models: Hierarchical LID decomposes classification from broad language groups to individual languages or variants, motivated by the difficulty of distinguishing related languages.
- Hierarchical Models: The specialized models cover Scandinavian, South Slavic, Arabic, and Persian language groups using language-specific subsets of OpenLID-v3 training data.
- Hierarchical Models: The inference procedure replaces a base-model prediction with the corresponding specialized classifier whenever the predicted language belongs to one of the four groups.
- Results: No significant performance difference was found between OpenLID-v3 and hierarchical models for related language groups on FLORES+ or UDHR.
- Results: The authors hypothesize that the large model is not capacity-saturated and recommend larger, higher-quality annotated data instead of subset-based specialization.