Source-linked AI summary
GLUECoS : An Evaluation Benchmark for Code-Switched NLP
Simran Khanuja, Sandipan Dandapat, Anirudh Srinivasan, Sunayana Sitaram, Monojit Choudhury
TL;DR
Code-switched NLP lacks standardized resources and comprehensive evidence across tasks. GLUECoS introduces a six-task benchmark for English-Hindi and English-Spanish, compares cross-lingual and multilingual models, and finds that multilingual models—especially after code-switched fine-tuning—perform best in most cases.
Problem
Code-switched NLP has scarce standardized resources and lacks comprehensive evaluation of embedding models across different task types.
Method
GLUECoS evaluates cross-lingual and multilingual models on six code-switched NLP tasks in English-Hindi and English-Spanish, including a new conversational NLI dataset.
Results
Multilingual BERT outperforms cross-lingual embedding techniques on all tasks, while a version fine-tuned on synthetic and limited real code-switched data performs best for most datasets.
Takeaways & Limitations
Multilingual models can be further improved for code-switched NLP by using synthetic and real code-switched data.
Takeaways & Limitations
The mBERT and cross-lingual model results are not directly comparable because their training data differ substantially in scale, sources, and language coverage.
Abstract
from arXiv · showhide
Code-switching is the use of more than one language in the same conversation or utterance. Recently, multilingual contextual embedding models, trained on multiple monolingual corpora, have shown promising results on cross-lingual and multilingual tasks. We present an evaluation benchmark, GLUECoS, for code-switched languages, that spans several NLP tasks in English-Hindi and English-Spanish. Specifically, our evaluation benchmark includes Language Identification from text, POS tagging, Named Entity Recognition, Sentiment Analysis, Question Answering and a new task for code-switching, Natural Language Inference. We present results on all these tasks using cross-lingual word embedding models and multilingual models. In addition, we fine-tune multilingual models on artificially generated code-switched data. Although multilingual models perform significantly better than cross-lingual models, our results show that in most tasks, across both language pairs, multilingual models fine-tuned on code-switched data perform best, showing that multilingual models can be further optimized for code-switching tasks.
1 Introduction
GLUECoS addresses limited standardized resources and comprehensive evaluation for code-switched NLP by benchmarking six tasks across English-Hindi and English-Spanish. It compares cross-lingual and multilingual models and finds further gains from fine-tuning multilingual models on synthetic code-switched data.
- Benchmark and motivation: Code-switched NLP lacks standardized datasets and comprehensive evaluation across different task types.Existing resources are scarce and prior evaluations generally focus on individual tasks or limited language pairs.
- Benchmark and motivation: GLUECoS proposes a benchmark for six code-switched NLP tasks across English-Hindi and English-Spanish.The tasks span language identification, POS tagging, named entity recognition, sentiment analysis, question answering, and natural language inference.
- Benchmark and motivation: The benchmark introduces Natural Language Inference as a new task for code-switched data.The proposed benchmark combines five previously evaluated tasks with a sixth task, Natural Language Inference.
- Models and findings: Pre-trained multilingual embeddings significantly outperform cross-lingual embeddings across the evaluated tasks.The comparison evaluates both model families on the benchmark tasks.
- Models and findings: Fine-tuning pre-trained multilingual models on linguistically motivated synthetic code-switched data improves performance in most cases.These results indicate that multilingual models can be further optimized for code-switched settings.
2 Relation to prior work
Prior code-switching shared tasks have advanced individual NLP problems, but the field lacked a benchmark comparing models across tasks, languages, and datasets. GLUECoS is designed to measure cross-task generalization and identify models that perform best across these dimensions.
- Existing evaluations: Earlier code-switching shared tasks focused on individual problems such as language identification, named entity recognition, POS tagging, sentiment analysis, or question answering.These tasks were conducted across workshops and evaluation efforts for several language pairs and settings.
- Evaluation gap: Prior work had not evaluated models across several code-switched NLP tasks.GLUECoS addresses this gap through a unified evaluation framework.
- Evaluation gap: GLUECoS aims to determine which models generalize best across different tasks, languages, and datasets.Its benchmark design is inspired by GLUE's multi-task evaluation objective, although GLUECoS does not evaluate models in a multi-task setting.
3 Tasks and Datasets
GLUECoS evaluates code-switched NLP across varied tasks, datasets, language pairs, scripts, and switching patterns. It includes standardized code-switching statistics and addresses dataset comparability limitations when contextualizing results.
- Benchmark scope: GLUECoS spans word-level, syntactic, semantic, and inference tasks across English-Hindi and English-Spanish datasets.The benchmark includes LID, NER, POS tagging, Sentiment Analysis, Question Answering, and NLI.
- Dataset characteristics: English-Hindi adds script variation because Hindi appears in Roman or Devanagari, whereas English-Spanish uses the Roman script.
- Benchmark scope: The benchmark includes multiple datasets per language pair where available, enabling comparisons across datasets for the same task.
- Dataset construction: The benchmark uses author-created train-test-validation splits for some tasks because standardized code-switched datasets are lacking.An off-the-shelf transliterator and language detector are also used where necessary.
- Dataset characteristics: Code-switching characteristics are summarized with CMI, average switch-points, M-index, I-index, burstiness, language entropy, and span entropy.These metrics quantify language-tag distributions, switching frequency, switching regularity, and language-span distributions.
- Task definitions: NLI uses conversations as premises, making it a conversational code-switched inference task with entailed or contradicted premise-hypothesis relationships.The dataset is new, so the authors report their number as the task’s SOTA.
4 Experimental Setup
The experiments compare representative cross-lingual embeddings with multilingual BERT-based models across the benchmark tasks. They also test embeddings and mBERT adapted using synthetically generated code-switched data, with only a small real-data component in adaptation.
- Experimental design: The experimental setup uses standard task architectures together with pretrained mBERT and representative embedding baselines.
- Models: The study compares several cross-lingual word embeddings with Multilingual BERT across all benchmark tasks.The cross-lingual methods vary in the amount of supervision used during training.
- Models: Cross-lingual embeddings are trained with supervised, unsupervised, parallel-data, or bilingual-objective approaches.MUSE supports supervised and unsupervised mappings, while BiCVM and BiVec use parallel corpora or alignments.
- Code-switched adaptation: Synthetic code-switched data is used to train skip-gram embeddings for English-Spanish and English-Hindi.The English-Spanish data contains 8M synthetic code-switched sentences, while English-Hindi data is generated from the IITB parallel corpus.
- Code-switched adaptation: mBERT is fine-tuned with synthetically generated code-switched data and a small amount of real code-switched data using masked language modeling.The real code-mixed data accounts for less than 5% of the total code-mixed data used.
5 Results and Analysis
Across GLUECoS tasks, mBERT substantially outperforms cross-lingual techniques, while modified mBERT usually performs best after fine-tuning on synthetic code-switched data. Results vary by task and language pair, and code-switched NLP remains far from solved beyond LID.
- mBERT provides a large increase in accuracy over cross-lingual techniques across the evaluated tasks.
- Modified mBERT performs best in most cases, including NLI, where it outperforms base mBERT.The paper attributes this pattern to code-switched distributions differing from monolingual language distributions.
- Direct comparison between mBERT and cross-lingual models is limited because mBERT uses substantially more training data and languages.mBERT is trained on 104 languages and massive wiki data, whereas cross-lingual embeddings use more restricted Twitter and query-log data.
- English-Spanish results are generally better than English-Hindi results, possibly because the language pairs differ in similarity and word-order preservation.English and Spanish mostly retain individual word order during code-switching, unlike English and Hindi.
- Modified mBERT does not significantly outperform base mBERT for English-Hindi overall, with small differences on QA En-Hi and FG En-Hi.The paper suggests transliterator errors or corpus differences may contribute to these exceptions.
- Except for LID, code-switched NLP is far from solved, especially for Sentiment and NLI, where modified mBERT performs only a little above chance.
6 Conclusion
GLUECoS introduces a code-switching benchmark spanning six NLP tasks in English-Hindi and English-Spanish. Multilingual models outperform cross-lingual techniques, while fine-tuning with synthetic and real code-switched data improves performance on most datasets.
- Benchmark scope: GLUECoS contains six tasks across English-Hindi and English-Spanish, including a new code-switched Natural Language Inference task.The tasks cover LID, POS tagging, NER, Sentiment Analysis, Question Answering, and NLI.
- Findings: Multilingual BERT outperforms cross-lingual embedding techniques on all evaluated tasks and datasets.
- Findings: A modified mBERT fine-tuned on synthetically generated code-switched data with some real code-switched data performs best for most datasets.
- Task difficulty: LID and POS tagging are relatively easier, whereas QA and NLI have low accuracies.
- Resources: The benchmark’s datasets are publicly available, and the authors plan to release the NLI dataset for research use.
A Additional Dataset Details
The benchmark uses custom balanced splits when standard partitions are unavailable and adds language tags or script conversion for English-Hindi data. These preprocessing choices address dataset-format constraints.
- Dataset splits: Datasets without predefined training, development, and test partitions receive balanced custom splits in an 8:1:1 ratio.
- English-Hindi preprocessing: For untagged Roman-script English-Hindi corpora, language tags are obtained with an LID tool.
- English-Hindi preprocessing: Roman Hindi words are converted to Devanagari with an off-the-shelf transliterator when language tags are available.
B Additional Training Details
Experiments are repeated across five random seeds, and the reported results are averaged across those runs.
- Evaluation procedure: Each experiment is conducted with 5 random seed values.
- Evaluation procedure: The evaluation therefore reports aggregate results over repeated seeded experiments.
- Evaluation procedure: Results are averaged across the runs obtained from the random seeds.
B.1 Word-Level Tasks
Word-level tasks use a sequence-labeling architecture, while mBERT experiments use the transformers library with the reported optimizer settings and five training epochs.
- Word-level architecture: Language Identification, Named Entity Recognition, and POS tagging use a sequence labeler.
- Word-level architecture: The sequence labeler implements a BiLSTM with a CRF layer.
- mBERT training: mBERT experiments use the transformers library, AdamW, a 5e-5 learning rate, and 5 training epochs.
B.2 Sentence-Level Tasks
Sentence-level sentiment classification uses a one-layer BiLSTM with specified regularization, optimization, batching, and early-stopping settings, alongside mBERT experiments.
- Sentiment classification uses a one-hidden-layer BiLSTM with hidden dimension 256 and 300-dimensional word embeddings.
- The BiLSTM uses dropout 0.5, Adam with learning rate 0.001 and epsilon 1e-8, batch size 64, and up to 15 epochs.
- Training stops when validation accuracy continually drops for 3 epochs.
- The mBERT experiments use AdamW with learning rate 5e-5, epsilon 1e-8, batch size 32, and 5 training epochs.
B.3 Sentence-Pair Tasks
The sentence-pair evaluations use BiDAF for embedding-based question answering and Transformers training scripts for mBERT, with mBERT additionally fine-tuned through a generated-to-real code-mixed data curriculum.
- Question Answering: Embedding-based question answering uses the BiDAF architecture with default settings including a 400-token context and 50-token question limit.The configuration also uses learning rate 0.5, batch size 1, and 5 training epochs.
- mBERT Experiments: The mBERT experiments use the SQuAD and XNLI training scripts from the Transformers library with default hyperparameters.
- Code-Mixed Fine-Tuning: The multilingual BERT model is fine-tuned for masked language modeling on generated and real code-mixed datasets.
- Code-Mixed Fine-Tuning: Training follows a curriculum of 10 epochs on generated code-mixed data followed by 10 epochs on real code-mixed data.