MixLID: Token-Level Language Identification in Mixed-Language Text

1 Introduction

• A single input-level label cannot distinguish languages used across different spans of code-switched text, motivating token-level identification.Token-level identification complements identifying the predominant language of an input.Source: §1 · p. 1
• Prior token-level approaches use naturally occurring code-switched data, while synthetic data have also been created by mixing phrases from different languages.Training for mixed-language inputs must account for both how languages are combined and how monolingual and mixed-language data are used.Source: §1 · p. 1
• MixLID supports 111 language–script combinations and is trained first on monolingual examples, then on mixed-language data.The monolingual labels supervise tokens, while the later stage combines code-switched examples with word-substitution augmentation using English source words and corresponding loanwords.Source: §1 · pp. 1–2
• MixLID outperforms baselines in accuracy and macro F1 on four monolingual benchmarks and in token-level identification on every evaluated language pair.Monolingual comparisons use only language–script combinations shared with each baseline.Source: §1 · p. 2

2 MixLID

• MixLID first learns from monolingual examples and then continues training on existing and synthetic examples containing multiple languages.The same objective is used in both stages.Source: §2 · p. 2, §2.2 · p. 3

2.1 Training Data Construction

• Monolingual sentence labels are assigned to words, with separate labels for numbers, punctuation, and symbols and heuristics for abbreviations and non-Latin-script examples.English-list abbreviations receive the English label, while abbreviations found only in the source-language list retain that language's label.Source: §2.1 · p. 2
• Mixed-language training concatenates labeled ACS examples and augments Smol and Belebele text by replacing English words with corresponding loanwords, or vice versa.Replacements receive the selected target-language label; remaining expressions default to the original text's language label.Source: §2.1 · p. 2

2.2 Learning Objective

• Character offsets align word labels to subwords; split-word subwords share a label, and both stages minimize cross-entropy over labeled positions.A multilingual encoder produces contextual subword representations, and a classification head predicts each label.Source: §2.2 · pp. 2–3

3.1 Setup

• Sentence-level evaluation covers FLORES+, WiLI-2018, UDHR, and CommonLID; token-level evaluation uses ACS and the LID subset of CodeMixBench.Sentence-level comparisons are restricted to languages jointly supported by MixLID and each baseline.Source: §3.1 · p. 3

3.2 Results

• MixLID outperforms all eight baselines in accuracy and macro F1 across four sentence-level benchmarks.The comparison includes sentence-level identification systems and Polyglot Tagger, which supports token-level identification.Source: §3.2 · pp. 3–4
• 93.23% accuracy and 88.78% macro F1 on ACS, and 94.50% accuracy and 87.29% macro F1 on CodeMixBench, accompany MixLID's lead over all three baselines across evaluated subsets.The reported token-level comparisons are with MaskLID, Polyglot Tagger, and CLD3.Source: Table 3 · p. 4, §3.2 · p. 4

4.1 Language-wise Performance

• 97.38% mean F1 on 36 shared languages exceeds GlotLID-v3 by 0.99 percentage points.The comparison spans all four benchmarks and all models’ shared language coverage.Source: §4.1 · p. 4
• MixLID achieves the highest F1 on 26 of 36 shared languages and scores above 95% on 32.It outperforms GlotLID-v3 and Polyglot Tagger on 33 and 31 languages, respectively.Source: §4.1 · p. 4

4.2 Effect of Training Stages

• Stage 1+2 outperforms either individual stage on ACS and CodeMixBench, while exceeding Stage 1 on sentence-level identification except for a slight WiLI-2018 decrease.Stage 1 is stronger on all four sentence-level benchmarks, whereas Stage 2 performs better at token identification in mixed-language inputs.Source: §4.2 · p. 4, Table 4 · p. 4

• MixLID covers 111 language labels and is designed to distinguish languages within mixed-language inputs.Source: §5 · p. 4
• Trained on monolingual data followed by existing and augmented mixed-language data, MixLID outperforms baselines in both monolingual and mixed-language identification.Source: §5 · p. 4
• Mixed-language training improves individual token identification while largely preserving performance on monolingual inputs.Source: §5 · pp. 4–5

• Evaluation covers a limited set of code-switching combinations, so generalization to a wider range requires further validation.Coverage is limited to 111 language labels; dictionary-based substitution and automatic labeling may also miss contextual usage or linguistic ambiguity, potentially introducing unnatural expressions or labeling errors.Source: Limitations · p. 5
• The study covers 111 language labels and does not encompass the full range of linguistic diversity.Source: Limitations · p. 5

Ethical Considerations

• Loanword-based augmented data are for training token-level identification and do not represent the actual language use or code-switching practices of a particular community.The study documents resource sources, data construction and preprocessing, and evaluation procedures for transparency.Source: Ethical Considerations · p. 5

A Related Work

• Prior sentence- and document-level LID work includes statistical and neural methods, broader language coverage, and approaches to noise and domain differences.The cited work includes cross-domain generalization challenges and supervised contrastive learning for low-resource languages.Source: A Related Work · p. 7
• Mixed-language LID research includes sequence labeling, synthetic data, lexicon and contextual methods, and transfer across language pairs; MixLID combines two-stage training with existing and loanword-augmented data.Source: A Related Work · pp. 7–8

B.1 Language Inventory

• The 111 language labels and three additional labels—NUM, PUNCT, and SYM—form 114 output classes shared across both training stages.The additional labels apply to numbers, punctuation, and symbols rather than language labels.Source: §B.1 · p. 8

B.2 Shared Labeling and Token Alignment

• Unicode categories guide labeling of numbers, punctuation, and symbols, with decimal points and digit-grouping commas retained inside numbers.For example, 3.14 and 1,000 each receive one NUM label, while 3,14 is split into NUM, PUNCT, and NUM.Source: §B.2 · p. 8
• Labeled spans are aligned to subwords by character offsets, assigning overlaps to the span with greatest character coverage and resolving ties toward the earlier span.Ordinary spaces are excluded from overlap calculations; apostrophes within words inherit the word label, while standalone quotation marks are PUNCT.Source: §B.2 · p. 8

B.3 Monolingual Data Preparation

• For non-Latin-script monolingual examples, sequences of three or more English letters trigger exclusion, except for expressions in abbreviation lists.The filter is intended to reduce assignment of a sentence’s language label to expressions from other languages.Source: §B.3 · p. 8
• The English-letter filter is not applied to Latin-script examples or to mixed-language data.Source: §B.3 · p. 8
• Abbreviation-list expressions are exempt from exclusion under the filter, even when they contain three or more English letters.Source: §B.3 · p. 8

B.4 Mixed-Language Data Preparation

• ACS examples with token-level labels are cleaned and concatenated in shuffled groups of up to four, separated by single spaces and used once each.Anonymized mentions are removed; unlabeled tokens are relabeled when possible, and examples with remaining unrelabeled tokens are excluded.Source: §B.4 · p. 8
• Loanword augmentation substitutes longest non-overlapping word-boundary matches independently with 70% probability, choosing among candidate languages uniformly at random.Substitution uses borrowing relationships between English source words and loanwords; the method does not disambiguate word sense or infer inflected forms, and retains examples even when no substitution occurs.Source: §B.4 · p. 8

B.5 Training Configuration

• Stage two starts from the stage-one model, reinitializes the optimizer and learning-rate scheduler, and updates all parameters using four NVIDIA RTX A6000 GPUs.Source: §B.5 · p. 8

C.1 Baseline Models

• The comparison includes whole-input language identifiers and token- or span-level predictors, including CLD3, MaskLID, and Polyglot Tagger.CLD3 provides both whole-input and span predictions; MaskLID applies iterative masking without additional training, and Polyglot Tagger predicts tokens directly.Source: §C.1 · pp. 8–10
• Other listed baselines include OpenLID-v3, GlotLID-v3, NLLB-LID, fastText lid.176, and CommonLingua.Their stated label inventories range from 176 for fastText lid.176 to 2,102 for GlotLID-v3.Source: §C.1 · p. 9
• CONLID is another baseline, described as using contrastive learning and providing 2,099 output labels in the experiments.Source: §C.1 · p. 9

C.2 Evaluation Benchmarks

• Monolingual evaluation treats each sentence, paragraph, or line as one input across FLORES+, WiLI-2018, UDHR, and CommonLID.The benchmarks use, respectively, translated texts, Wikipedia paragraphs, multilingual human-rights translations, and human-annotated web text.Source: §C.2 · p. 10
• Mixed-language evaluation uses observed and manipulated ACS examples across 11 language pairs, plus German–English, Spanish–English, and Chinese–English in CodeMixBench.The CodeMixBench pairs are supported by all models in the comparison.Source: §C.2 · p. 10

C.3 Evaluation Protocol and Metrics

• Comparisons include only languages and scripts jointly supported by the benchmark and models; token-level subsets require support from MixLID, MaskLID, Polyglot Tagger, and CLD3.Arabic and Persian varieties are mapped to macrolanguage labels only when a model provides them; support for one variety does not imply support for others.Source: §C.3 · p. 10
• Token predictions are aligned to reference tokens by character overlap, and non-language, mixed, ambiguous, and whitespace-or-symbol-only reference tokens are excluded.Undetermined predictions and predictions outside the language pair count as errors for eligible tokens.Source: §C.3 · pp. 10–11
• Sentence-level results report accuracy and macro F1, with macro F1 averaging per-language F1 scores over the shared evaluation languages.Accuracy is the proportion of correctly predicted evaluation examples.Source: §C.3 · p. 11
• For token-level evaluation, accuracy and macro F1 are computed per example and averaged across examples within each language pair.The main-table Avg is the arithmetic mean of language-pair scores within a benchmark, and scores are percentages.Source: §C.3 · p. 11