LinCE: A Centralized Benchmark for Linguistic Code-switching Evaluation

1. Introduction

• Existing code-switching methods are often evaluated on few language pairs and domains, leaving their generalizability unclear and comparisons across datasets slow and difficult.The paper attributes these problems to the lack of an official benchmark for direct multi-task comparisons and the limited evaluation of methods across datasets with different language pairs.Source: §1 · p. 1
• LinCE combines ten datasets across four tasks, selected to span language pairs, code-mixing indexes, typological diversity, and code-switching domains.The benchmark includes four LID datasets, two POS tagging datasets, three NER datasets, and one sentiment analysis dataset; its language-selection criteria also consider geolocation and code-mixing index (Gamb¨ack and Das, 2014).Source: §1 · p. 1
• LinCE proposes new, fairer task splits for nine of its ten datasets and provides LSTM, ELMo, and multilingual BERT baselines.The authors report inspecting the original datasets for five major issues and describe analysis of the best model and outstanding benchmark challenges.Source: §1 · p. 1
• The benchmark is publicly available online and is intended to grow with additional low-resource languages, tasks, and language pairs.The paper describes LinCE as a continuous effort.Source: §1 · p. 1

• Code-switching has been studied in LID, POS, NER, parsing, sentiment, and question answering, with datasets from FIRE, CALCS, and other efforts spanning core and additional tasks.Despite this growth, comparison across language pairs, domains, and general language-processing tasks remains unclear.Source: §2 · pp. 1–2
• Token-level LID methods often exceed 90% accuracy, but model rankings and trade-offs remain unclear; diverse POS approaches are effective on their datasets yet difficult to compare across languages.Approaches include CRFs with hand-crafted features, LSTMs with word and character embeddings, code-mixed embeddings, transfer learning, monolingual linguistic resources, traditional ML, and joint POS-LID models.Source: §2 · p. 2
• NER remained data-scarce despite recent corpora; CALCS explored neural, attention, and multitask models with external resources, yet cross-language-pair superiority remained unresolved.Participants could submit predictions for one or both language-pair competitions.Source: §2 · p. 2
• Code-switched sentiment analysis remained underexplored: prior Spanish-English work compared bag-of-words logistic regression, while SemEval-2020 launched a Spanish-English/Hindi-English competition and LinCE adopted its Spanish-English corpus.The authors argue that consolidating language pairs and tasks can support steadier progress beyond language-specific advances.Source: §2 · p. 2

Linguistic Challenges

• The benchmark uses CMI to quantify alternation complexity: higher scores indicate more language alternations across datasets.Table 2 reports CMI scores and token counts across corpora; the MSA-EA NER corpus lacks CMI information because it has no language-identification labels.Source: §3 · p. 2, Table 2 · p. 3

Spanish-English

• Spanish-English lexical overlap increases ambiguity for character-based systems, while Spanish-inflected English verbs raise OOV rates for non-contextualized embeddings.The overlap reflects English words borrowed from Latin and French that resemble Spanish words; this within-word switching is called Spanglish.Source: §3 · p. 2

Hindi-English

• Hindi-English lacks standardized Hindi transliteration in the Roman alphabet, and Hindi’s flexible word order adds difficulty despite its loose SOV structure.Ad-hoc phonological transliteration makes Hindi resources available only in Devanagari difficult to use for Romanized text.Source: §3 · pp. 2–3

Nepali-English

• Nepali-English combines arbitrary Romanized Nepali with SOV–SVO grammatical differences, encouraging fine-grained alternations and producing one of the largest averaged CMI scores.The authors report a large code-switching rate despite the corpus being of middle size; switching preserves each language’s grammatical structure.Source: §3 · p. 3

Modern Standard Arabic-Egyptian Arabic

• MSA-EA diglossia enables code-switching, while substantial word overlap with differing meanings, rich morphology, multiple word orders, and morpheme-level inflection increase semantic complexity.The passage characterizes Arabic diglossia as combining dialects with Modern Standard Arabic in one community.Source: §3 · p. 3

4. Tasks

• LinCE comprises four tasks and four language pairs across eleven datasets; Sections 4.1–4.4 discuss task datasets, Section 4.5 justifies modifications to nine of ten datasets for official splits, and Section 4.6 explains leaderboard evaluation criteria.Source: §4 · p. 3

4.1. Language Identification (LID)

• LID uses preexisting corpora for four language pairs and the CALCS label scheme, including language, mixed, ambiguous, foreign-word, named-entity, other, and unknown labels.The benchmark proposes new splits for some LID corpora, as described in the stratification section.Source: §4.1 · p. 3

4.2. Parts-of-Speech (POS) Tagging

• POS tagging covers Hindi-English and Spanish-English, using universal POS tags with corpus-specific additions and benchmark splits stratified for fair evaluation.The Hindi-English corpus adds PART NEG and PRON WH; the Spanish-English corpus includes POS and language-identification labels.Source: §4.2 · p. 4

4.3. Named Entity Recognition (NER)

• NER covers Spanish-English, Modern Standard Arabic-Egyptian Arabic, and Hindi-English; all corpora use BIO labels, but only the Arabic-English corpus lacks language-identification labels.New splits are proposed for the Spanish-English and Hindi-English datasets, while the Arabic-English organizer-provided splits are retained.Source: §4.3 · pp. 4–5, Figure 3 · p. 5

4.4. Sentiment Analysis (SA)

• Sentiment analysis uses Spanish-English SentiMix posts labeled positive, negative, or neutral, with token-level LID labels and newly proposed partitions.The corpus contains 18,789 tweets and 286,810 tokens; the new partitions address label-distribution issues in the original splits.Source: §4.4 · p. 5

4.5. Stratification

• New splits are proposed for nine of ten datasets when classes are missing, label or sentence-length distributions differ, LID labels were overlooked, or official partitions are unavailable.The NER Modern Standard Arabic-Egyptian Arabic dataset is not repartitioned because it lacks LID labels needed to balance code-switching behavior.Source: §4.5 · pp. 5–6
• For affected datasets, iterative stratification jointly considers LID labels, task labels, and sentence-length categories, and the proposed splits show less KL-divergence from full-corpus label distributions.Posts are treated as samples with grouped labels; lengths are categorized as small (≤10 tokens), medium (>10 and ≤20), or large (>20).Source: §4.5 · p. 6

4.6. Evaluation

• LinCE’s platform accepts test-set predictions, publishes per-task public leaderboards, and ranks submissions by the average of task scores.The evaluation model is described as similar to SemEval, Kaggle, and GLUE.Source: §4.6 · p. 6

5. Baseline Models

• LinCE tests whether a single model architecture can generalize across all proposed tasks using three baselines: BiLSTM, ELMo, and multilingual BERT.The architectures were selected because they had demonstrated strong performance across NLP tasks.Source: §5 · p. 7
• BiLSTM uses tuned random word embeddings; it predicts token labels from concatenated bidirectional states and sentiment from max-pooled states.It is the simplest baseline; token labeling covers LID, POS, and NER.Source: §5 · p. 7
• ELMo is fine-tuned for sequence labeling using high-level word representations and predicts sentiment from the mean of token representations (Peters et al., 2018).Sequence labeling covers LID, POS, and NER.Source: §5 · p. 7
• Multilingual BERT, pretrained on 104 languages, receives a task-specific prediction layer and is fine-tuned as a whole (Devlin et al., 2018).The passage identifies BERT as a Transformer-based pretrained language model (Vaswani et al., 2017).Source: §5 · p. 7

5.1. Implementation and Training

• Across tasks, model architectures stay the same except for the prediction layer; BiLSTM and ELMo use SGD, BERT uses AdamW, and all train with batch size 32 for at most 50 epochs.Training stops after 10 epochs without validation improvement, and choices were tuned on validation sets. Learning rates are 0.1 for BiLSTM, 0.01 for ELMo, and 5e−5 for BERT; β = 0.9 and ϵ = 1e−8 are also specified.Source: §5.1 · p. 7

6. Results and Analysis

• Pretrained language models outperform BiLSTM on almost all tasks, and ML-BERT performs best on every task across all available language pairs.ELMo is close to ML-BERT on most LID and POS tasks (≈1-4%), but the reported gap is larger for NER (≈810%) and sentiment analysis (≈18%); the average ELMo–BERT gap is ≈6%.Source: §6 · p. 7
• The authors attribute ML-BERT’s improvement over ELMo to its larger parameter set (110M vs 13.6M) and pretraining data, including document-level BERT data versus sentence-level ELMo data.BERT used Book Corpus and Wikipedia, while ELMo used the Billion Words Benchmark.Source: §6 · p. 7
• NER and sentiment analysis appear harder than LID and POS, which the authors link to semantic understanding being more challenging than syntactic analysis in code-switched text.Source: §6 · p. 7
• LID is harder for MSA-EA than for the other language pairs, possibly because the languages overlap substantially, complicating annotation.The passage cites Molina et al. (2016) regarding the annotation effect.Source: §6 · p. 7

• LinCE introduces a centralized benchmark for evaluating code-switching technology across multiple tasks and language pairs, built from ten publicly available datasets.Source: §7 · p. 7
• The authors identify issues in existing dataset partitions and propose splits stratified by up to three criteria, preserving full-corpus distributions that original partitions may not preserve.Criteria include language-identification labels, task-specific labels, and sentence lengths.Source: §7 · pp. 7–8
• The benchmark reports strong baselines using state-of-the-art models, including BERT and ELMo, on monolingual datasets.Source: §7 · p. 8
• The authors plan to continue evolving the LinCE platform by incorporating more tasks and language pairs.Source: §7 · p. 8

A.1. LID Label Scheme

• For LID, lang1 denotes English in English-paired datasets and Modern Standard Arabic for MSA-EA, while lang2 denotes the paired language: Spanish, Hindi, Nepali, or Egyptian Arabic.Source: §A.1 · p. 11
• The LID scheme also labels partly cross-language words as mixed, uncertain-language words as ambiguous, third-language words as fw, named entities as ne, unrecognized tokens as unk, and symbols, punctuation, and emoticons as other.Source: §A.1 · p. 11

A.2. POS Label Scheme

• POS uses Petrov et al.’s (2012) universal tagset, adding PRON WH for interrogative pronouns and UNK for undetermined syntactic categories, following Singh et al. (2018b) and Soto and Hirschberg (2017).Source: §A.2 · p. 11
• The POS scheme uses X for other categories, including abbreviations and foreign words.Source: §A.2 · p. 11

A.3. NER Label Scheme

• NER follows Aguilar et al.’s (2018) BIO scheme, with entity types person, organization, location, group, and product.Source: §A.3 · p. 11
• NER also distinguishes titles as a fine-grained product category, events from holidays, periodic dates and holidays as time, and other entities that fit no prior type.Source: §A.3 · p. 11