Source-linked AI summary
IndicTriMix: Developing Language Identification Datasets and Models for Tri-Language Code-Mixing
Pruthwik Mishra, Rudra Trivedi, Avi Patel, Ashok Urlana, Shrikant Malviya
TL;DR
The paper addresses token-level language identification for multilingual code-mixed text, extending research beyond predominantly bilingual settings. It fine-tunes MuRIL and XLM-RoBERTa on three-language configurations and finds strong performance from language-specific and combined-data training, while noting label-imbalance and subword-alignment limitations.
Problem
Existing token-level code-mixed language-identification work is largely restricted to bilingual settings, despite multilingual social-media text requiring fine-grained token classification.
Method
The study builds trilingual datasets and fine-tunes MuRIL and XLM-RoBERTa for token classification across combined, ENG-HIN-BEN, and ENG-HIN-GUJ configurations.
Results
Language-pair-specific models perform strongly on corresponding evaluation sets, while combined multilingual training remains competitive across language pairs.
Takeaways & Limitations
Contextual word embeddings provide a robust foundation for handling sequence boundaries and structural changes in multilingual code-mixed sentences.
Takeaways & Limitations
Performance estimates become less stable for languages with limited representation in imbalanced evaluation subsets, and first-subword-only supervision leaves later subwords without direct labels.
Abstract
from arXiv · showhide
Language identification in code-mixed text, largely observed in social media, is highly essential when users frequently switch between multiple languages within a single utterance. Accurately identifying the languages of code-mixed tokens becomes an urgent necessity. Traditional language identification models, designed for monolingual text, are not well suited for token-level language identification in code-mixed settings. We formulate the task as a sequence labeling problem and fine-tune contextual transformer-based models MuRIL and XLM-RoBERTa best suited for Indian languages. We evaluate these systems on three different data configurations (Hindi, Gujarati, and Bengali) to predict language labels for individual tokens. We release a benchmark for language identification in code-mixed tokens with manually annotated test sets. We propose two approaches of code-mixed generation using parallel sentences of three languages. The trained models demonstrate the effectiveness of contextual embeddings for token-level language identification in multilingual social media text. For reproducibility and to facilitate future research, we publicly release our fine-tuned models.
1 Introduction
The paper targets token-level language identification in multilingual code-mixed social media text, where conventional monolingual or sentence-level tools are inadequate. It develops multilingual transformer models and releases associated resources for three-language settings.
- The work provides publicly released fine-tuned models, datasets, and code for reproducibility.
- Token-level language identification is formulated as sequence labeling because conventional tools fail on code-mixed user-generated content.
- Prior work largely focuses on bilingual code-mixing, motivating language-agnostic techniques for three languages using 3-way parallel corpora.
- MuRIL and XLM-RoBERTa are fine-tuned to assign linguistic tags to every token in multilingual code-mixed social media sentences.
2 Related Work
Prior work addressed bilingual or multilingual code-mixed language identification, but the paper targets the previously unaddressed three-language setting. It contributes rule-based and LLM-generated trilingual datasets with expert annotations for ENG-HIN-GUJ and ENG-HIN-BEN.
- 2 Related Work: MuRIL and XLM-RoBERTa are compared under combined multilingual, ENG-HIN-BEN, and ENG-HIN-GUJ training configurations.
- 2 Related Work: Three-language code-mixed identification was not previously addressed, as earlier studies largely focused on bilingual settings.
- 2 Related Work: The benchmark combines rule-based and LLM-based datasets, with expert human annotations for the generated trilingual code-mixed sentences.
- 2 Related Work: The task covers ENG-HIN-GUJ and ENG-HIN-BEN, using three-letter language tags and a UNI tag for symbols and punctuation.
- 2 Related Work: LLM generation uses parallel English, Hindi, Gujarati, and Bengali sentences while varying language order to approximate natural multilingual mixing.
4 Assessing Quality of Generated Code-Mixed Sentences
The paper assesses generated trilingual code-mixed sentences by comparing them with corresponding language sentences using multilingual semantic embeddings. Both generation approaches exceed 0.85 average similarity across the evaluation methods.
- 4 Assessing Quality of Generated Code-Mixed Sentences: BertScore, Sentence BERT, and LaBSE embeddings measure semantic similarity between each generated sentence and its corresponding language sentence.
- 4 Assessing Quality of Generated Code-Mixed Sentences: Parallel sentences serve as the main pivot for the code-mixed generation approaches.
- 4 Assessing Quality of Generated Code-Mixed Sentences: 0.85 average similarity is exceeded by sentences generated with both approaches across the evaluation methods.The result is interpreted as indicating high fluency and faithfulness.
5 Methodology
The methodology formulates token-level language identification as sequence labeling and fine-tunes transformer representations with a token classification layer. It aligns word-level annotations to subwords while dynamically padding batches for training.
- 5 Methodology: Token-level language identification maps each token sequence S = (w1, w2, . . . , wn) to language labels Y = (y1, y2, . . . , yn).The label set includes BEN, ENG, GUJ, HIN, and UNI.
- 5 Methodology: Sequences are parsed from CoNLL-style data and dynamically padded within batches to match the longest sequence.
- 5 Methodology: Word-to-subword alignment assigns each word label to its first subword and masks later subwords with ignore index -100 during loss computation.
- 5 Methodology: Contextual transformer hidden vectors feed a linear token classification layer that estimates crossentropy distributions over target tags.
6 Model Architectures
The paper evaluates MuRIL and XLM-RoBERTa for token-level language identification in multilingual code-mixed text. MuRIL targets Indian-language linguistic and code-mixing nuances, while XLM-RoBERTa provides broad cross-lingual representations.
- Both architectures are evaluated for token-level language identification in code-mixed text.
- 6.1 MuRIL: MuRIL is designed for Indian languages and code-mixed variations, including morphological and phonetic complexity across native scripts and Latin transliteration.
- 6.2 XLM-RoBERTa: XLM-RoBERTa-base is a cross-lingual transformer trained on 100+ languages and used as a multilingual comparison baseline.
7 Experimental Setup
The experiments fine-tune both models across combined and language-specific trilingual data configurations. Models are trained for 10 epochs, with the best validation macro F1-score checkpoint retained for evaluation.
- The implementation uses PyTorch, Hugging Face Transformers, and mixed-precision fp16 optimization, with foundational settings listed in Table 5.
- Training uses combined multilingual data, ENG-HIN-BEN data, and ENG-HIN-GUJ data as three fine-tuning configurations.
- Models are fine-tuned for 10 epochs, evaluated after each epoch, and selected by the best validation macro F1-score.
8 Results and Evaluation
The systems show strong performance across combined and language-specific trilingual configurations. Specialized training performs strongly on its corresponding language pair, combined training generalizes competitively, and MuRIL is generally slightly better while XLM-RoBERTa remains comparable in several settings.
- Specialized training achieves consistently strong performance on its corresponding language-pair evaluation sets, indicating capture of language-specific characteristics.
- Combined multilingual training produces competitive performance across both language pairs, showing that one model can learn shared representations while maintaining strong overall performance.
- MuRIL generally performs slightly better than XLM-RoBERTa, particularly with combined training, although XLM-RoBERTa is comparable in several configurations and splits.
- Rule-based development sets achieve consistently high F_1-scores, whereas LLM-generated sets vary more, especially for HIN; UNI is recognized with near-perfect F_1-scores.
9 Conclusion
The study compares token-classification frameworks based on MuRIL and XLM-RoBERTa across three multilingual data configurations. Its results indicate that contextual word embeddings provide a robust foundation for handling sequence boundaries and structural changes in multilingual code-mixed sentences.
- MuRIL and XLM-RoBERTa were trained as token-classification frameworks under combined multilingual, ENG-HIN-BEN, and ENG-HIN-GUJ configurations.The models were evaluated on corresponding development and test sets to compare combined-data and language-pair-specific training.
- Contextual word embeddings provide a robust foundation for managing sequence boundaries and structural changes in multilingual code-mixed sentences.
10 Limitations
The paper identifies evaluation instability from uneven label distributions and challenges from informal, transliterated code-mixed text. Word-to-subword alignment also limits direct supervision for non-initial subwords.
- Uneven label distributions make F1-score estimates less stable for languages with limited representation in evaluation subsets.With few supporting examples, precision and recall variations can disproportionately affect performance estimates.
- Abbreviations, slang, formatting anomalies, and similar Romanized phonetics make language classification more challenging.The paper suggests incorporating additional lexical or phonetic information as future work.
- Assigning labels only to first subwords leaves subsequent subword representations without direct supervision from word-level annotations.The implementation assigns subsequent subwords the ignore index -100, and the paper suggests more sophisticated alignment strategies.
A Appendix
The generated code-mixed sentences show high semantic similarity to their corresponding English and romanized Indic-language sentences. These scores indicate that the generated sentences are both faithful and fluent with their language-specific counterparts.
- High semantic similarity scores indicate that generated code-mixed sentences are faithful and fluent relative to English and romanized Hindi, Gujarati, and Bengali counterparts.The comparison uses semantic similarity scores reported in Table 9.