PaperlayerView exact revision on arXiv

Source-linked AI summary

Automatic Language Identification in Texts: A Survey

Tommi Jauhiainen, Marco Lui, Marcos Zampieri, Timothy Baldwin, Krister Lindén

arXiv:1804.08186v2cs.CL

TL;DR

Language identification is important because downstream text-processing methods generally assume the input language is known, yet the field’s literature is fragmented across research communities. This article surveys LI’s history, features, methods, evaluation, applications, systems, open issues, and future directions, concluding that LI remains complex rather than solved.

  • Problem

    Language identification must accurately determine a text’s language because many downstream text-processing techniques assume that the input language is known.

  • Method

    The article provides a comprehensive survey that organizes LI research across features, methods, evaluation approaches, applications, systems, open issues, and future directions.

  • Results

    The survey concludes that LI is a rich, complex, multi-faceted problem: near-perfect accuracy is possible under controlled conditions, but broader settings make it much harder.

  • Takeaways & Limitations

    LI remains a productive area for future work across supervised learning, representation learning, multitask learning, domain adaptation, and multi-label classification.

  • Takeaways & Limitations

    The data requirements for accurately modeling low-resource languages for LI remain unclear, and existing work does not establish a simple answer.

Abstract

from arXiv · show

Language identification (LI) is the problem of determining the natural language that a document or part thereof is written in. Automatic LI has been extensively researched for over fifty years. Today, LI is a key part of many text processing pipelines, as text processing techniques generally assume that the language of the input text is known. Research in this area has recently been especially active. This article provides a brief history of LI research, and an extensive survey of the features and methods used so far in the LI literature. For describing the features and methods we introduce a unified notation. We discuss evaluation methods, applications of LI, as well as off-the-shelf LI systems that do not require training by the end user. Finally, we identify open issues, survey the work to date on each issue, and propose future directions for research in LI.

1. Introduction

Language identification determines which natural language a document or passage uses and supports downstream processing, translation, and information access. Although controlled settings can yield very high accuracy, the survey treats LI as a broad, challenging field with fragmented literature.

  • Scope and motivation: Language identification determines the natural language used in a document or part of a document.The survey illustrates this task with excerpts from Wikipedia articles written in different languages.
  • Scope and motivation: Automatic LI develops algorithms and indexing structures to identify languages without human intervention, potentially covering thousands of human languages.The survey contrasts this goal with the more limited number of languages most people or trained linguists can recognize.
  • Scope and motivation: The survey limits its discussion to LI for digitally encoded written text, excluding speech, sign language, and handwritten text.In the broad sense, LI applies to those modalities and to both digital and non-digital information storage.
  • Applications: LI enables text-processing pipelines to operate on real-world data when the input language is unknown.Many NLP and information-retrieval techniques presuppose a known input language or a common language across documents.
  • Applications: LI identifies unknown document languages for applications including web-collection processing and routing documents to suitable machine-translation systems.Determining the source language is generally necessary before translating a document into a target language.
  • Research landscape: The literature is fragmented across NLP, information retrieval, machine learning, data mining, social-media analysis, education, and systems science because LI lacks a home discipline.The survey identifies partial knowledge of related work and the proliferation of systems and datasets as consequences for the field.

2. LI as Text Categorization

LI can be framed as text categorization, but its language-independent labels, tokenization problem, multimodal classes, and mutually exclusive document assignments create distinctive challenges. The survey therefore synthesizes LI research separately while relating it to broader categorization work.

  • Relation to text categorization: LI maps documents onto language classes, making it a special case of the broad text-categorization task.The survey contrasts LI with topic classification, such as assigning Reuters articles to subject labels.
  • Distinctive characteristics: LI cannot rely universally on word-frequency statistics because some languages do not use whitespace to mark word boundaries.Choosing an appropriate tokenization strategy would itself presuppose the document’s language.
  • Distinctive characteristics: Unlike dataset-specific categorization labels, LI labels refer to languages across domains such as biomedical articles, microblogs, and newspapers.The survey presents language as a domain-independent classification concept.
  • Distinctive characteristics: LI classes can be multimodal because one language may appear in different orthographies and encodings while remaining one class.This distinguishes LI from simpler representations that assume a single surface form for each category.
  • Distinctive characteristics: LI labels are mutually exclusive for monolingual segments, while multilingual documents can be divided into segments associated with different languages.This differs from multilabel categorization, where labels generally cannot be assigned to specific document segments.
  • Research landscape: These characteristics have led LI research to proceed largely independently from general text-categorization research.The survey presents its review as a synthesis of common themes and ideas across LI work, including representation, algorithms, evaluation, applications, systems, and open issues.

3. A Brief History of LI

Automatic LI developed from manual language-detection needs into statistical and profile-based computational methods. Early work established character and word-frequency approaches, while Cavnar and Trenkle’s character n-gram method became a major benchmark and widely used implementation.

  • Early computational work: Before computational LI, translators used simple manual methods to identify documents in particular languages.Mustonen’s 1965 work is described as the earliest known functional LI program for text.
  • Early computational work: Nakamura’s early-1970s identifier distinguished 25 Latin-alphabet languages using character and word occurrence rates plus binary presence features.The account is based on the available abstract and later reports.
  • Statistical methods: Rau’s 1974 statistical LI work used relative frequencies of characters and character bigrams, with continued interest partly connected to military intelligence applications.The passage cites later military-intelligence research as evidence of that continuing need.
  • Speech and speech technology: Early LI research also addressed spoken language and speech technology, including statistical models of phonetic information and language identification for pronunciation.Church’s 1985 method estimated language probabilities from relative character-trigram frequencies in a Bayesian model.
  • Character n-gram milestone: Cavnar and Trenkle’s 1994 method classified documents by rank-order similarity between per-document and per-language profiles.It achieved 99.8% best overall accuracy on 3,478 documents in eight languages from USENET newsgroups.
  • Character n-gram milestone: TextCat popularized Cavnar and Trenkle’s character n-gram approach through an off-the-shelf implementation with pretrained language models.The method remains a benchmark for automatic LI.

4. On Notation

The survey introduces unified notation to make LI methods comparable and potentially reproducible. It formalizes corpora, segments, features, n-grams, language models, and language-specific scores used in identification.

  • Unified notation: The survey translates notation from prior LI papers into a unified system so similarities and differences among methods are easier to inspect.The formulas are intended to support implementation and reevaluation of earlier studies.
  • Corpora and segments: A corpus C is a finite sequence of tokens, which may be bytes, characters, or words, and it can be divided into non-overlapping segments.The total token count is l_C, and the number of segments is l_S.
  • Features and counts: A feature is a countable corpus characteristic, with c(C, f) denoting its count and U(C) denoting the set of unique features.The notation also represents feature sets and their cardinality.
  • Features and counts: Corpus feature counts equal the sum of counts across segments, but segmentation changes counts because features cannot cross segment boundaries.The survey explicitly notes this segmentation dependence.
  • N-grams: An n-gram is a contiguous sequence of n tokens within one corpus segment, with C_n denoting all n-grams of length n.When n = 1, the feature is an individual token.
  • Language models and scoring: For candidate language g and unknown corpus C, a language model O(C_g) supplies feature values and the identifier calculates a score R(g, C).The notation distinguishes the language set G, language-specific corpora, model domains, and test documents.

2. A model for each language is derived from a training corpus of labelled documents

A language model is derived from a training corpus of labelled documents.

  • The model-building stage derives a language model from a labelled training corpus.
  • The resulting model represents one language for later document comparison.
  • Training therefore establishes language-specific models before prediction occurs.

4. The language of a document is predicted based on the highest-scoring model

Language prediction uses a similarity function to compare a document with language models. The theoretical description leaves room for interpretation in implementation.

  • A similarity function determines how closely a document matches each language model.
  • The method description does not uniquely determine how that similarity function is implemented.
  • Different implementations may therefore realize the same theoretical method differently.

5. Features

LI represents text with diverse features, most prominently character and word sequences, while accounting for encoding and tokenization choices. The survey compares frequencies, probabilities, weighting schemes, and smoothing across these representations.

  • Bytes and encodings: Encoding detection can transcode text before LI, but some systems instead build per-language models spanning multiple encodings.Transcoding is computationally expensive, motivating approaches that ignore encoding and model multiple encodings simultaneously.
  • Feature types: The survey covers features based on alphabets, capitalization, word length, character frequencies, and morphological analysis.A spelling-checker and morphological-analyzer system achieved 97.9% accuracy on one-line texts across 11 official South African languages.
  • Character features: Character n-grams generally outperform character unigrams, and combining character n-grams with words can improve identification.Bigrams and trigrams generally performed better than unigrams; one system reached 99.8% average recall using character n-grams combined with words.
  • Feature weighting: Feature values may use raw or relative frequencies, probabilities, IDF, mutual information, and other weighting schemes.Artemenko and Shramko calculated trigram IDF values per language, while Zamora et al. found mutual-information weighting best in their evaluations.
  • Word features: Word and character features can complement each other: words may outperform character bigrams, but character 3–5-grams and combined representations perform better.Prager reported 95.6% accuracy for word and character 4-grams together when choosing between 13 languages.
  • Feature smoothing: Smoothing choices affect language-model performance: additive smoothing with 0.1 performed best in the reported evaluations.Good-Turing was better than Witten-Bell and Kneser-Ney, while Witten-Bell was clearly better than Kneser-Ney.

6. Methods

LI methods represent text with features such as characters, n-grams, words, or linguistic annotations, then score or classify candidate languages. The survey covers rule-based, tree-based, probabilistic, and frequency-based approaches, including smoothing and combined feature systems.

  • Decision Rules: Decision rules identify a language from features present in its corpus or eliminate languages lacking features found in the test document.Positive rules can use unique features; negative rules omit incompatible languages.
  • Decision Rules: Decision rules tend not to scale well to many languages or very short documents, so they are often combined with other methods or used in decision trees.
  • Simple Scoring: 99.8% accuracy was achieved by simple scoring when identifying six language groups, while 100% success was reported with character 4-grams across ten languages in relatively large documents.Simple scoring awards points to languages containing test-document features; the highest- or lowest-scoring language wins, depending on the variant.
  • Sum or Average of Values: Sum or average methods combine feature values across a test document, selecting the language with the highest score and allowing normalization across document lengths.Reported applications include word frequencies, graphones, n-grams, and weighted relative frequencies.
  • Product of Values: Product-of-values methods select the language with the highest product of feature values, while smoothing is usually required to avoid multiplication by zero.Summing log frequencies can avoid underflow while preserving the relative ordering of products.
  • Naive Bayes: Naive Bayes multiplies feature probabilities for each language and selects the language with the highest probability, although dependent features can improve accuracy in practice.

6.6 Similarity Measures

Similarity measures compare test-document representations with language models using ranks, distances, probabilities, or entropy. The surveyed methods differ in scoring direction and feature representation, with performance depending on document length, language set, and n-gram order.

  • Rank and Distance Measures: The out-of-place method compares ranked n-gram profiles and selects the language with the lowest rank-distance score.It achieved an F-score of 95% for 35-character test documents across 285 languages, ranking fourth in that evaluation.
  • Rank and Distance Measures: Local Rank Distance sums separations between identical units within a predetermined local window and has been combined with RBF kernels, KDA, and KRR.
  • Vector and Distance Methods: 96.0% accuracy was obtained when SLWE classified ten-word sentences among three languages by updating a feature vector word by word and choosing the closest language distribution.
  • Rank and Distance Measures: Out-of-place distance was the most accurate among several distance measures compared by Abainia et al. (2016).The comparison included Manhattan, Bhattacharyya, chi-squared, Canberra, Bray Curtis, histogram intersection, and correlation distances.
  • Entropy Measures: Relative-entropy methods can be competitive for character bigrams, whereas products of relative frequencies achieved clearly higher recall with higher-order n-grams.
  • Entropy Measures: 98.51% precision was reported for relative-entropy measures with character n-grams on 100-character texts spanning 53 language-encoding pairs.

6.7 Discriminant Functions

Discriminant-function methods map test documents into a feature space using functions that distinguish languages, then select the nearest known language. Studies compared regression, discriminant analysis, SVMs, and related classifiers.

  • Discriminant Functions: Discriminant functions map a test document into an n-dimensional space and select the nearest language according to distance from known language representations.
  • Comparative Evaluations: Linear regression, Naive Bayes, and logistic regression had very similar precision, with linear regression ranking second behind logistic regression.
  • Comparative Evaluations: Multiple discriminant analysis separated Finnish from English and Swedish, then separated English and Swedish from each other using two functions.
  • Comparative Evaluations: SVM outperformed discriminant analysis and neural networks in a comparison using character features.
  • Comparative Evaluations: The Winnow 2 algorithm was outperformed by other methods tested for language identification.

6.8 Support Vector Machines (“SVMs”)

SVMs are widely used in language identification, with linear and kernelized variants extending binary separation to multiclass decisions. The surveyed results also cover neural architectures, whose effectiveness varies by model, features, and evaluation setting.

  • Support Vector Machines: SVMs learn separating hyperplanes that maximize the margin, with one-vs-rest classifiers extending the approach to multiple languages.Kernels allow nonlinear projections of the original feature space.
  • Linear kernel SVMs: Linear-kernel SVMs have been widely used for LI and have achieved success across a range of shared tasks.Kim and Park (2007) were the first reported users of linear-kernel SVMs in LI.
  • Sigmoid kernel SVMs: Sigmoid-kernel SVMs were reported to outperform NB, CART, and summed relative frequencies in one LI evaluation.Bhargava and Kondrak (2010) first used sigmoid-kernel SVMs for LI; Majliˇs (2012) reported the comparison.
  • Kernel comparisons: SVM-based approaches were the top-performing systems in the 2014 and 2015 DSL shared tasks.The comparison covered three SVM approaches evaluated on datasets from different DSL shared tasks.
  • Neural Networks: Neural-network results varied: early RNNs underperformed character n-gram counts, while later CNN, LSTM, GRU, and embedding-based models achieved stronger results.Later work incorporated character n-grams, byte-level sentence representations, adversarial learning, and word embeddings.

6.10 Other Methods

The survey covers diverse LI methods beyond SVMs, including decision trees, Markov models, clustering, boosting, compression, topic features, Gaussian processes, passive-aggressive classifiers, and ensembles. Results differ by method and evaluation setting, with ensemble gains depending on the rule and track.

  • Decision-tree methods: Bagging with decision trees achieved 73.86% accuracy in English variety detection, slightly exceeding decision-table classification at 73.07%.These were the best and second-best methods tested by Simaki et al. (2017).
  • Markov models: Aggregate Markov models achieved 74% accuracy for six-language identification with text length of ten characters.The result was reported by Binas (2005).
  • Boosting and classifiers: Results among boosting methods conflicted: gradient boosting outperformed random forests in one study, whereas logistic regression outperformed gradient boosting in another.The studies evaluated XGBoost or gradient boosting in LI tasks.
  • Boosting and classifiers: A passive-aggressive classifier performed almost as well as SVMs when five machine-learning algorithms were compared.The comparison was reported for language identification across five languages.
  • Ensemble methods: Ensemble performance depended on the aggregation rule and evaluation track: voting reached 0.9025 weighted F1-score versus 0.9013 in the closed track, but underperformed without ensembling in the open track.The open-track scores were 0.9016 with the ensemble and 0.9065 without it.
  • Ensemble methods: Mean probability ensembles outperformed plurality voting and median probability ensembles in one evaluation, while product-rule ensembles produced only a small average-accuracy increase over majority voting.A separate comparison found mean ensembling better than median ensembling, at 68% versus 67% accuracy.

7. Empirical Evaluation

LI evaluation commonly uses document-level accuracy and per-language or collection-level summaries, but comparisons are constrained by short texts, feature choices, and substantial variation in datasets and labeling schemes.

  • Evaluation metrics: Document-level accuracy is the most frequently reported metric, calculated as the proportion of evaluation documents assigned their correct labels.The corresponding error rate is 1 −accuracy.
  • Evaluation metrics: Per-language evaluation distinguishes precision for predicted-language groups from recall for actual-language groups, often also reporting their harmonic-mean F-score.Under the monolingual assumption, precision and recall are closely linked, but both are increasingly reported for clarity.
  • Evaluation metrics: As the number of languages grows, full confusion matrices and per-language results become harder to present and interpret, encouraging collection-level summary statistics.Whole-collection summaries can weight documents equally or weight classes equally through macro-averaging.
  • Comparability and standardization: 99.8% accuracy on eight European languages did not transfer consistently across evaluations using 67 or 285 languages, illustrating the limits of cross-publication comparisons.Evaluation outcomes vary with language set, test size, and hyper-parameters.
  • Experimental settings: Longer text samples are generally easier to classify, while short-text LI remains important for microblog messages, search queries, and one-word inputs.These challenges have motivated dedicated research on LI for shorter segments.
  • Experimental settings: 300–400 features per language were sufficient for one method, whereas 20,000 features per language produced the best evaluation results for the same method in another study.Feature count affects memory requirements and classification rate, and the accuracy-feature relationship varies with method and data.
  • Comparability and standardization: Published results are difficult to compare because evaluation datasets vary in size and source, while language codes, orthographies, encodings, and transliterations complicate standardization.Correctly labeled data is costly to obtain, and automatically filtered corpora can be biased toward documents that systems identify correctly.
  • Comparability and standardization: Shared-task datasets provide standardized evaluation data, but existing tasks generally target specific LI sub-problems rather than broad-coverage language identification.Examples include tasks on French varieties and Arabic dialect or broadcast identification.

8. Application Areas

Automatic language identification supports translation, multilingual retrieval, and language-specific NLP processing. Its applications also include linguistic research, while multilingual documents motivate segment- and word-level identification.

  • Translation: LI routes documents to suitable human translators or machine-translation models by determining their source language.
  • NLP processing: LI helps route documents to language-specific NLP components and can identify foreign inclusions in predominantly monolingual text.POS taggers and parsers commonly assume monolingual input, while foreign-inclusion detection handles embedded words and phrases.
  • Retrieval: Multilingual retrieval systems apply LI to both indexed documents and user queries to reduce ambiguity from words valid in multiple languages.The passage uses German gift, meaning “poison,” as an example of a false friend.
  • Research: Automatic LI supports linguistic research and web-based corpus construction by locating pages and gathering language-specific resources.
  • Multilingual text: Multilingual documents challenge standard LI because they violate the assumption that each document uses one language.This challenge has prompted research on segmenting text by language and identifying language at the word level.

9. Off-the-Shelf Language Identifiers

Off-the-shelf language identifiers provide pre-trained models so users can apply LI without supplying training data. The surveyed systems vary in algorithms, language coverage, and target domains, and comparisons favor an ensemble of leading systems on Twitter messages.

  • Overview: Off-the-shelf identifiers distribute pre-trained models, allowing users to apply LI without providing training data.This setup targets end-users primarily interested in using language-identification output rather than developing the underlying technique.
  • Survey scope: The survey summarizes available open-source or free systems and records their key characteristics and update dates as of October 2018.
  • General-purpose systems: whatlang combines weighted character n-gram features, discriminative feature selection, and sentence-level smoothing, with pre-training for 1400 languages.The smoothing exploits locality by assuming adjacent sentences are likely to share a language.
  • System characteristics: Coverage varies substantially across systems, including 176 languages for whatthelang, 122 for YALI, 19 for LDIG, 83 for ChromeCLD, and 97 for langid.py.LDIG targets Twitter messages, whereas YALI uses Wikipedia data and langid.py combines data from five text sources.
  • Methods: The surveyed systems use varied classifiers and representations, including FastText, linear models, tries with logistic regression, and Naïve Bayes over character n-grams.
  • Evaluation: On Twitter messages, ChromeCLD, langid.py, and LangDetect were the best individual systems, while a voting ensemble of the three achieved slightly higher accuracy.Accuracy comparisons are difficult because systems support different language subsets, so the study compared performance over the full language set.

10. Research Directions and Open Issues in LI

The survey synthesizes open issues in LI, including preprocessing, multilingual and short-text identification, low-resource support, and discrimination among similar languages and varieties. It highlights domain, script, data, and representation challenges while proposing directions such as language-aware extraction and more abstract linguistic features.

  • The survey consolidates previously catalogued LI issues and adds new open problems into a unified research-directions section.Text representation and algorithm choice are covered earlier, while remaining issues are synthesized here.
  • 10.1 Text Preprocessing: Normalization transforms training and test text to remove variation that may confound language classification, while cleaning applies only to training data.Case folding and range compression are examples of normalization; range compression reduces sparsity in ideographic character sets.
  • 10.1 Text Preprocessing: Language-aware extraction can isolate language-bearing text from binary files more effectively than conventional heuristic approaches, motivating data-driven domain-independent representations.The survey proposes using language models to segment documents into textual and non-textual components.
  • 10.2 Scripts and Encodings: Identifying languages across scripts and encodings remains an open problem, including recognizing a shared language or recovering the source language of transliterated text.The survey suggests exploiting consistent alphabet-order patterns across encodings as one possible approach.
  • 10.3 Supporting Low-Resource Languages: Low-resource LI lacks a clear answer to how much data is needed to model a language accurately, making it a major future-work area.Prior work reports weak support for low-resource languages and difficulties constructing highly multilingual datasets.
  • 10.7 Short Texts: Short-text evaluations often use narrow, well-formed domains, while LI results do not trivially generalize to search queries, microblogs, or other domains.Twitter research broadened evaluation to 65 languages, but a 1300-language model performed relatively poorly in the target domain because it over-predicted low-resource languages.
  • 10.8 Similar Languages, Language Varieties, and Dialects: Similar-language identification remains difficult: TextCat reached only 45% accuracy on a three-way Bosnian/Serbian/Croatian dataset, and handcrafted rules were needed for Malay versus Indonesian.The survey also notes that word-level models can outperform character n-grams for some Spanish varieties, whereas character n-grams perform better for Portuguese and French.
  • 10.8 Similar Languages, Language Varieties, and Dialects: Robust discrimination among similar languages and varieties requires features capturing abstract systemic differences beyond lexical content.Suggested representations include de-lexicalized text, part-of-speech tags, morphosyntactic information, and adversarially modified representations.

11. Conclusions

The survey concludes that language identification is a complex problem whose difficulty increases substantially on real-world data, despite near-perfect accuracy under controlled conditions. It synthesizes prior work and identifies open issues and future research directions, while treating its best practices as application-dependent guidelines.

  • Near-perfect accuracy is possible for small language sets and long, grammatical, structured documents under controlled conditions.The survey contrasts these conditions with the difficulties of real-world data.
  • Very short documents, non-linguistic noise, non-standard language, and mixed-language content make language identification much harder.Examples include search queries, HTML markup, social media data, and multilingual forum posts.
  • The survey synthesizes prior results into language-identification best-practice guidelines, but their use depends on the broader context of the target application.
  • Open issues show that language identification remains a learning problem with opportunities for supervised, representation, multitask, domain-adaptation, and multi-label learning.
  • The paper aims to provide a reference point for future work by explaining existing research and identifying aspects that merit further investigation.
Loading 1804.08186v2…