Source-linked AI summary
Improving Language Identification for Code-Switched Utterances with Integer Linear Programming
Joanna Radoła, Josep Maria Crego, François Yvon
TL;DR
Code-switched utterances challenge LID systems because multiple languages occur within one sentence, while MaskLID depends heavily on unreliable word-level associations. The paper improves the backbone LID, reformulates assignment as an Integer Linear Program, and reports strong gains across 10 languages and code-switched benchmarks.
Problem
Intra-sentential code-switching remains difficult for LID systems because several languages occur in the same sentence.
Method
The paper improves MaskLID’s backbone LID and reformulates its language-assignment algorithm as an Integer Linear Program with interpretable constraints.
Results
+100% accuracy boosts are observed on several code-switched benchmarks in experiments involving 10 diverse languages.
Takeaways & Limitations
Improved word-level scores and ILP optimization substantially increase code-switched identification rates, including very large gains for some language pairs.
Takeaways & Limitations
The approach simplifies code-switched utterances to two languages and is constrained by ambiguous short segments.
Abstract
from arXiv · showhide
Automatic identification of code-switched (CS) utterances remains a challenge for language identification (LID) systems, causing such texts to be underrepresented in the training data of Large Language Models. In this paper, we revisit MaskLID, a state-of-the art approach for CS identification, which requires no training and detects arbitrary language combinations. We make three main contributions: (a) we reveal, and address, a major issue of MaskLID: its overreliance on word-level language association scores; (b) we reformulate the underlying optimization algorithm as an Integer Linear Program, enabling us to experiment with a large set of clear and interpretable constraints; (c) each of these improvements vastly improves the baseline system, as we illustrate in experiments involving 10~diverse languages, where we observe a strong boost in performance on CS benchmarks. We release our code and data for reproducibility.
1 Introduction
Language identification supports multilingual data collection but struggles with intra-sentential code-switching, where multiple languages occur in one sentence. The paper revisits MaskLID and improves its word-level scoring and optimization.
- Modern LIDs recognize hundreds of languages, but identifying multiple languages within one sentence remains difficult.
- Intra-sentential code-switching is widespread in informal communication and is a central unresolved challenge for LID systems.
- MaskLID provides a training-free approach that can recognize arbitrary language combinations using a generic LID and post-processing.
- The paper addresses MaskLID’s reliance on inaccurate word-level language association scores and reformulates its assignment algorithm as an Integer Linear Program.
- +100% accuracy boosts are observed on several code-switched benchmarks across experiments involving 10 diverse languages.
2 Background
MaskLID combines sentence-level and word-level language evidence to iteratively assign language labels to fragments, but its word-level associations can be unreliable. The paper recasts this constrained assignment problem as an Integer Linear Program to seek better global solutions.
- FastText-based LIDs: FastText-based LIDs represent text with word and n-gram embeddings and compute language posterior probabilities with a softmax classifier.
- MaskLID: MaskLID first predicts the dominant sentence language, then evaluates each word and masks it when that language is among the word’s most likely labels.
- MaskLID: The masking step prevents words compatible with the dominant language from being reassigned to closely related languages in later iterations.
- MaskLID: MaskLID repeatedly analyzes the remaining text and returns additional languages when the residual fragment is sufficiently long.
- Reanalysis: Because MaskLID relies heavily on word-level scores produced by sentence-level classifiers, those associations may be unreliable for selecting masked words.
- ILP formulation: The paper formulates word-to-language assignment as an Integer Linear Program with global and length constraints, replacing greedy optimization with a search for better global solutions.
3 Improving MaskLID assignments with better word-level scores
The authors diagnose brittle word-level predictions in GlotLID and improve them by training LiteLID variants with word-level examples. These improved scores strengthen MaskLID across code-switched and monolingual evaluations.
- Datasets and metrics: The evaluation combines code-switched corpora across multiple language pairs with balanced monolingual data from the 10 languages used in those pairs.
- Datasets and metrics: The benchmarks vary in source, annotation scheme, script, and inserted-span length, requiring filtering and preprocessing before evaluation.
- Datasets and metrics: Exact match measures whether the predicted language set exactly matches the reference, while precision, recall, and F1 are macro-averaged across languages or pairs.
- GlotLID scores: GlotLID can assign unexpected languages to short segments, such as labeling an English phrase as Norwegian because of individual words.
- GlotLID scores: Figure 1 shows near-perfect sentence-level probabilities but many near-zero word-level probabilities for gold labels, undermining MaskLID’s word-selection assumption.
- LiteLID: LiteLID-v2, trained on sentences and words, improves short-segment prediction without reducing performance on longer inputs.
- LiteLID: 0.14 absolute F1 points are gained by LiteLID-v2 over GlotLID on sequences of length 6-10.
- Improved MaskLID results: Improved word-level probabilities increase performance for both code-switched and monolingual test sets across all tested length parameters.
4 Computing better assignments with ILP
The paper reformulates MaskLID’s language-assignment problem as an Integer Linear Program, making its objective explicit and enabling interpretable constraints. Expanded objectives and constraints yield configurations that improve the MaskLID baseline on development data.
- The ILP objective maximizes word-language association scores while enforcing at most one language per word and at most K languages per sentence.These are the core C1 and C2 constraints.
- ILP4LID formalizes code-switched language identification as constrained optimization over word-language assignments, returning languages assigned at least once.Binary variables represent word-language assignments and whether each language appears.
- The extended model adds language-ranking variables, a penalty for each predicted language, score-based ranking weights, and supplementary constraints.Supplementary constraints include minimum language length, top-M assignment rules, and a limit on language changes.
- Three configurations optimize exact match on code-switched data, monolingual data, or their average, using different objectives and constraint settings.The CS configuration uses the core objective with C1 and C2; MONO and AVG use extended objectives and additional constraints.
- 0.08 absolute points: AVG increases development-set EM from 0.68 to 0.76 over the MaskLID baseline.The AVG configuration uses α1 = 1, α2 = 0.75, P = 15, τ = 5, and M = 10.
5 Experiments on the testset
Test results confirm that both the improved backbone LID and ILP optimization outperform the MaskLID baseline across a larger, more diverse language set. Their combination raises average exact match substantially, while error analysis highlights trade-offs among configurations and remaining difficulty on noisy short texts.
- 0.17 absolute EM: replacing MaskLID’s greedy optimizer with ILP4LID improves the AVG configuration, while the combined improvements raise average EM from 0.54 to 0.71.The AVG configuration also surpasses MaskLID globally by +0.04.
- +0.13 absolute EM: replacing the MaskLID backbone with LiteLID-v2 produces a large test-set improvement, while LiteLID-v2 outperforms GlotLID broadly.The reported gains are especially large on code-switched data.
- The three ILP4LID configurations obtain significantly better test-set EM than MaskLID, although the CS configuration’s difference is borderline significant.McNemar’s test reports p-values of 1.3e-302, 1.3e-18, and 0.0498 for AVG, MONO, and CS.
- The AVG configuration offers the best trade-off between code-switched detection and monolingual identification, while MONO has 0.96 average precision but only 0.69 CS recall.The CS configuration has 0.99 monolingual recall but 0.04 monolingual EM because it predicts two languages in most cases.
- 97% of test words receive ILP4LID AVG assignments versus 77% for MaskLID, yielding a +0.1 increase in code-switched recall.The analysis attributes MaskLID’s unassigned words to greedy early decisions that leave fragments too short for later assignment.
- Very short, noisy, ambiguous social-media sequences remain the biggest challenge, despite preliminary filtering.The authors recommend AVG generally, and CS or MONO when the number of languages is known beforehand.
6 Related work
Prior work frames code-switching identification through word-level, character-level, subword-level, and unsupervised approaches. These alternatives address mixed-language structure or unknown language sets, but often require annotated data or simplify the language setting.
- Word-level code-switching identification is difficult because of extralexical terms, lexical overlap, and intra-word code-switching.Consequently, much supervised work focuses on two predefined languages.
- Subword models require annotated data that are not available at large scale, whereas word-level methods remain difficult beyond small predefined language sets.These constraints motivate approaches that can operate across broader language combinations.
- Infra-word LID predicts languages for characters or subwords, enabling mixed-language words to be split and tagged by language.This approach can represent language changes inside a word.
- Unsupervised approaches avoid predefined language lists, including methods using sliding windows or language-specific word lists.Examples include estimating code-switched text on Twitter and identifying code-switching in multilingual documents.
7 Conclusion and Outlook
The paper reanalyzes MaskLID for code-switched language identification by improving its word-level LID and replacing greedy optimization with ILP. These changes improve CS identification, but future work targets calibration, language coverage, confidence reporting, and runtime.
- The authors identify unreliable word-level scores as a major MaskLID issue and mitigate it with a better backbone LID.
- The ILP formulation expresses constraints more flexibly and finds better global solutions than the original optimization approach.
- The combined improvements yield very large gains for some language pairs and support better processing of informal code-switched or minority-language text.
- Future work includes better-calibrated posterior probabilities, confidence scores, broader language coverage, and faster ILP-based CS identification.
Limitations
The approach is bounded by a two-language framing, restricted language and domain coverage, difficult romanization settings, and substantial ILP runtime costs. Its MaskLID analysis also depends on implementation details and word-level scores that can be unreliable.
- The formulation assumes two languages and can fail on more than two languages, ambiguous assignments, very short spans, and intra-word code-switching.
- Experiments cover only a handful of mostly Latin-script language pairs and informal social-media or transcribed-conversation data.
- Less-standard romanization such as Arabizi remains challenging because high-quality LIDs for mixed-language texts are difficult to find.
- Runtime grows linearly with the number of language labels across CS and AVG configurations and sentence-length buckets.
- This runtime is approximately 10 times higher than MaskLID for short segments and 20 times higher for longer segments, although CPU parallelization is possible.
- MaskLID's masking procedure uses parameters and word-association lists whose behavior varies with the underlying LID scores.
D Training a Word-Level LID
The training study develops LiteLID, a FastText-based alternative restricted to 125 Latin-script labels, and augments sentence data with isolated words to improve word-level predictions. The sentence-and-word model improves short-segment performance without harming longer-segment performance.
- LiteLID trains FastText-based LIDs on 125 Latin-script labels to obtain more reliable word-level language predictions.
- The main augmentation splits sentences into unigrams, with up to one million running words per label, to make word-level scores more reliable.
- Trigram enrichment provides no clear benefits, while training a model takes around 8 hours on a single CPU.
- The ILP formulation uses language-word association scores, top-M membership indicators, word-assignment variables Y, and sentence-level language variables U.
- The core ILP imposes dependencies between Y and U, at most one language per word, and at most K languages per sentence.
E.2 The Extended Model
The extended ILP model adds rank-aware assignments and constraints to control language hierarchy, switching, and span length. A rank-weighted objective and per-language penalty provide a tunable trade-off between monolingual and bilingual solutions.
- The extended model addresses spurious code-switch predictions that arise when assigning additional languages increases the global objective.
- Rank-indexed variables record whether each word is assigned to a language at rank k, while U records globally assigned languages by rank.
- Rank weights α1 > ··· > αK ensure that the language with the largest global score receives rank 1.
- A fixed penalty P for each new language controls the trade-off between monolingual and bilingual assignments.
- The extended constraints preserve one-language-per-word and language-count limits while adding rank restrictions and auxiliary variables.
- Additional constraints impose minimum span length, top-M eligibility, rank-1 assignment, and a maximum number of language switches.
- Relaxing the switching constraint into a fixed per-language penalty performs better and is more efficient in practice.
- The extended model has O(K × T × L) variables, approximately doubling the core formulation, while constraints grow linearly with T, L, and K.
F Ablation of ILP4LID configurations
The ablation shows that ILP4LID performance depends strongly on constraint selection, language weights, penalties, and hyperparameters, with code-switching performance prioritized over monolingual performance.
- ILP4LID configurations vary in exact match and F1 according to constraint selection, language weights, penalties, and hyperparameters M, τ, and S.These settings are evaluated on development data in Table 9.
- A high language penalty reaches monolingual Exact Match of 0.99, but identifies significantly fewer code-switching cases; the selected MONO configuration uses 0.98 instead.The authors prioritize code-switching performance over the higher monolingual score.
- Adding constraints C3 and C5 with a handpicked penalty drastically increases monolingual performance, while slightly reducing code-switched EM.
- The maximal-number-of-switches constraint C6 has limited impact on CS EM, minimally harms performance, and doubles runtime.
- Constraint C4 provides no significant improvement.
G Detailed Results per Language
Table 10 reports detailed monolingual and mixed-language performance by language, comparing GlotLID and LiteLID-v2 as base language identification models.
- Table 10 reports per-language performance on monolingual and mixed-language corpora using GlotLID and LiteLID-v2 as base LIDs.Best performance for each code-switched language pair is bold, and results improved by a factor of two are underlined.