Source-linked AI summary
Overview for the Second Shared Task on Language Identification in Code-Switched Data
Giovanni Molina, Fahad AlGhamdi, Mahmoud Ghoneim, Abdelati Hawwari, Nicolas Rey-Villamizar, Mona Diab, Thamar Solorio
TL;DR
The paper addresses language identification in code-switched text through a shared task covering SPA-ENG and MSA-DA Twitter data. Nine teams participated, and evaluation found stronger overall systems than in the previous task, while closely related languages remained more difficult.
Problem
Language identification in code-switched text remains difficult and increasingly relevant as social media produces more such data.
Method
The shared task evaluates systems labeling code-switched data from SPA-ENG and MSA-DA, with nine teams participating overall.
Results
Systems achieved higher overall performance than in the previous shared task, with all SPA-ENG teams and four MSA-DA teams submitting systems.
Takeaways & Limitations
The shared task provides a competitive setting supporting further development of code-switched language-identification systems and corpora.
Takeaways & Limitations
Crowd-sourcing remained noisy, especially for MSA-DA homographs and named entities.
Abstract
from arXiv · showhide
We present an overview of the second shared task on language identification in code-switched data. For the shared task, we had code-switched data from two different language pairs: Modern Standard Arabic-Dialectal Arabic (MSA-DA) and Spanish-English (SPA-ENG). We had a total of nine participating teams, with all teams submitting a system for SPA-ENG and four submitting for MSA-DA. Through evaluation, we found that once again language identification is more difficult for the language pair that is more closely related. We also found that this year's systems performed better overall than the systems from the previous shared task indicating overall progress in the state of the art for this task.
1 Introduction
The shared task addresses automatic language identification in increasingly common code-switched text, aiming to advance annotation resources, universal labeling, and research. It covers Twitter data from SPA-ENG and MSA-DA language pairs.
- Code-switching uses two or more languages interchangeably in spoken or written form.
- Improving code-switched text processing can contribute to downstream NLP tasks such as part-of-speech tagging, parsing, and machine translation.
- The shared task uses Twitter data from Spanish-English and Modern Standard Arabic-Dialect Arabic language pairs.
- Participants predict the correct language label for each token in unseen test sets.
2 Task Description
The task labels each token in code-switched test data with one of eight categories, including language, mixed, unknown, ambiguous, other, and named entities. The annotation scheme was expanded and refined to support consistent labeling across language pairs.
- Each token is assigned one of 8 labels: lang1, lang2, fw, mixed, unk, ambiguous, other, or named entities (ne).
- The fw and unk labels were added in this edition by subdividing tokens previously labeled as other.
- Named entities are difficult to annotate because they typically remain unchanged when languages switch.
- The annotation guidelines aim to provide a universal label set that minimizes errors and are updated using findings from prior work and new research.
3 Related Work
Prior research on code-switched language data spans parsing, language and dialect identification, tagging, corpus construction, sociolinguistic analysis, and machine translation. This shared task builds on the first shared task and subsequent studies across language pairs and NLP applications.
- Research on code-switched data began with parsing work and continued through the first shared task on language identification at EMNLP 2014.
- Studies have addressed language and dialect identification, intra-word switching, bilingual parsing, and POS tagging across Arabic, Dutch, Hindi-English, and related varieties.
- Sociolinguistic work examined code-switching among Egyptian Twitter users across Modern Standard Arabic, Egyptian Colloquial Arabic, English, and Arabizi.
- New corpora cover Arabic-Moroccan Darija, Turkish-German, Modern Standard Arabic-Egyptian Arabic, and Spanish-English social media data.
4 Data Sets
The shared-task datasets were collected and annotated from Twitter for SPA-ENG and MSA-DA, combining prior corpora with newly crawled test data. The organizers applied relabeling, quality checks, filtering, and participant-availability constraints to prepare the evaluation sets.
- Twitter was selected for its large multilingual user base and accessible API, although collection volume and data sharing were restricted.
- Dataset statistics reflect availability at release, while parenthesized counts record the maximum tweets shared by all participants after test submission.
- SPA-ENG reused the earlier training and test corpora as training and development data, then remapped former other tokens to fw, unk, or other.
- The organizers corrected inconsistent word-type labels and tokenization errors, including stray whitespace tokens and improperly tokenized emoticons and symbols.
- SPA-ENG: SPA-ENG test construction selected bilingual areas, used an SVM for pre-annotation, applied regular-expression matching, and manually verified the 1,000 most frequent word types.
- SPA-ENG: SPA-ENG evaluation statistics include only tweets that every participant successfully crawled, not the complete dataset.
- MSA-DA: For MSA-DA, Egyptian Arabic was used as the dialect, and earlier Train, Test-1, and Test-2 corpora were combined into new training and development sets.
- MSA-DA: MSA-DA data were re-crawled and validated, with missing or whitespace-containing tweets removed before publication.
5 Survey of Shared Task Systems
The shared task attracted diverse systems, mostly machine-learning based, with CRFs dominant and deep learning newly introduced by two participants.
- System participation: Nine teams submitted systems, with architectures ranging from rule-based approaches to complex machine-learning implementations.Most systems were designed without changing implementation between language pairs, suggesting interest in language-independent systems.
- Machine-learning architectures: Most participants used Conditional Random Fields, which suit the task's sequence-labeling formulation.The authors connect this choice to CRFs' high performance in the previous shared task.
- Deep learning: Two participants introduced deep-learning systems using CNNs, bidirectional LSTMs, or LSTMs combined with CRFs and postprocessing.The paper describes these systems as more complex than traditional machine-learning algorithms, with a performance trade-off evident in the results.
- External resources and features: Most participants incorporated external resources such as monolingual corpora, dictionaries, taggers, word embeddings, or named-entity recognizers.Additional features included language models, word case, affixes, and surrounding context.
- System overview: Table 5 summarizes the architectures of the submitted systems.The table is presented as an overview of participant system designs.
6 Results
The evaluation compared systems with a lexicon-based baseline at tweet and token levels across SPA-ENG and MSA-DA. SPA-ENG systems generally performed better, while minority classes remained difficult to predict.
- Evaluation: Accuracy, Precision, Recall, and F-measure were used, with regular F-measure ranking tweets and weighted average F-measure ranking tokens.The weighted token-level metric accounts for imbalanced label distributions.
- Cross-language-pair comparison: SPA-ENG systems outperformed MSA-DA systems, indicating greater difficulty for the closely related MSA-DA language pair.The comparison is reported at the level of the best-performing systems.
- Class-level results: The other class was easiest, with most systems exceeding 97% F-measure, while minority classes often received 0%.The minority classes were ambiguous, mixed, fw, and unk; MSA-DA had no fw or unk samples.
- SPA-ENG results: All SPA-ENG systems beat the tweet-level baseline by at least 16%, while Shirvani et al. achieved 91.3% Avg-F-measure.Shirvani et al. also led token-level SPA-ENG performance with 97.3%, 0.4% above IIIT Hyderabad.
- MSA-DA results: All MSA-DA systems beat the tweet-level baseline by at least 20%, with Samih et al. achieving the highest tweet-level Avg-F-measure of 83%.At token level, Samih et al. also led with 87.6%, followed by 85.1% and 82.8% for the next systems.
- Winner interpretation: Overall winners cannot be determined because not all participants submitted systems for both language pairs.Within each dataset, Shirvani et al. led SPA-ENG and Samih et al. led MSA-DA at both evaluation levels.
7 Lessons Learned
The task faced persistent Twitter data constraints and noisy annotation, especially for MSA-DA, while proposing simpler future annotation guidance and more informative code-switch evaluation.
- Twitter data remained constrained by data loss and sharing restrictions, despite being retained because alternative sources were harder to search for code-switched text.
- Crowd-sourced annotation produced substantial noise, worsened in MSA-DA by homographs and by named-entity errors from limited contextual judgment.
- Future tasks may reduce annotation errors by offering crowd-sourcing annotators fewer choices and simpler guidelines with more examples.
- Future evaluation may assess predicted code-switch positions rather than only classifying tweets as code-switched or monolingual.
8 Conclusion
The second shared task attracted more participation and stronger systems than its predecessor, while producing more consistent rankings across token- and tweet-level evaluation. It also broadened system approaches and supported continued development of code-switched data and language-identification methods.
- 9 teams submitted systems, compared with 7 previously, and overall performance was higher than in the previous shared task.
- Most systems used CRF sequence labeling, while deep-learning submissions appeared alongside a weaker rule-based system.
- Token- and tweet-level results were more consistent, with the same teams ranking first at both levels for each language pair.
- The task provided a competitive setting intended to advance systems and corpora for improving understanding of code-switching.