PaperlayerView exact revision on arXiv

Source-linked AI summary

LinCE: A Centralized Benchmark for Linguistic Code-switching Evaluation

Gustavo Aguilar, Sudipta Kar, Thamar Solorio

arXiv:2005.04322v1cs.CL

TL;DR

Code-switching research lacks broad, centralized evidence for comparing methods across language pairs, domains, and tasks. LinCE consolidates ten datasets into a benchmark with standardized splits, baseline models, and an online platform, reporting multilingual BERT as the strongest model across available tasks and language pairs.

  • Problem

    Code-switching methods are difficult to compare and their generalizability across language pairs, domains, and tasks remains unclear because no official centralized benchmark exists.

  • Method

    LinCE combines ten datasets across four language pairs and four tasks, introduces stratified splits, and provides an online evaluation platform with LSTM, ELMo, and multilingual BERT baselines.

  • Results

    Multilingual BERT demonstrates superior performance for each task across all available language pairs, while ELMo is close on most language-identification and part-of-speech tasks.

  • Takeaways & Limitations

    LinCE provides a centralized basis for comparing code-switching systems and is intended to expand with additional tasks and language pairs.

  • Takeaways & Limitations

    The benchmark focuses on language pairs, and the MSA-EA NER dataset was not repartitioned because it lacks language-identification labels needed for balanced stratification.

Abstract

from arXiv · show

Recent trends in NLP research have raised an interest in linguistic code-switching (CS); modern approaches have been proposed to solve a wide range of NLP tasks on multiple language pairs. Unfortunately, these proposed methods are hardly generalizable to different code-switched languages. In addition, it is unclear whether a model architecture is applicable for a different task while still being compatible with the code-switching setting. This is mainly because of the lack of a centralized benchmark and the sparse corpora that researchers employ based on their specific needs and interests. To facilitate research in this direction, we propose a centralized benchmark for Linguistic Code-switching Evaluation (LinCE) that combines ten corpora covering four different code-switched language pairs (i.e., Spanish-English, Nepali-English, Hindi-English, and Modern Standard Arabic-Egyptian Arabic) and four tasks (i.e., language identification, named entity recognition, part-of-speech tagging, and sentiment analysis). As part of the benchmark centralization effort, we provide an online platform at ritual.uh.edu/lince, where researchers can submit their results while comparing with others in real-time. In addition, we provide the scores of different popular models, including LSTM, ELMo, and multilingual BERT so that the NLP community can compare against state-of-the-art systems. LinCE is a continuous effort, and we will expand it with more low-resource languages and tasks.

1. Introduction

Code-switching research lacks a centralized benchmark for comparing models across tasks, domains, and language pairs. LinCE addresses this gap by consolidating datasets, standardized splits, baseline models, and an online comparison platform.

  • Existing code-switching methods are often evaluated on few language pairs and specific domains, leaving their broader generalizability unclear.
  • The benchmark focuses on language pairs and accounts for geolocation when considering code-switched languages.
  • LinCE combines ten datasets spanning four language pairs and four NLP tasks.The benchmark covers language identification, named entity recognition, part-of-speech tagging, and sentiment analysis.
  • A new stratification process proposes fair or official splits for nine of the ten datasets after identifying problems in the original partitions.
  • LinCE provides strong baselines using LSTMs, ELMo, and multilingual BERT, alongside an online platform for real-time result comparison.

2. Related Work

Code-switching datasets and methods now span several NLP tasks, but comparisons across language pairs, domains, and tasks remain unclear. Sentiment analysis has received comparatively limited attention, with early work using small Spanish-English data and a newer multilingual competition.

  • Prior code-switching research covers language identification, part-of-speech tagging, named entity recognition, parsing, sentiment analysis, and question answering.
  • Shared-task datasets from FIRE and CALCS have focused mostly on core NLP tasks, while other datasets address dialect recognition, humor detection, and sub-word code-switching detection.
  • Despite growing dataset availability, model comparison across language pairs, domains, and general language-processing tasks remains unclear.
  • Sentiment analysis on code-switched data has been explored less extensively than several other tasks.
  • Early Spanish-English sentiment work used polarity annotations from a small section of a CALCS language-identification corpus and compared bag-of-words features with regularized logistic regression.

3. Linguistic Challenges

LinCE examines paired-language code-switching, whose complexity varies across corpora and language pairs. Shared alphabets, transliteration practices, lexical overlap, morphology, and differing grammatical structures create distinct processing challenges.

  • The benchmark focuses on language pairs and uses the CMI index to quantify code-switching complexity.Higher CMI indicates more language alternations and more complex code-switching behavior.
  • Spanish-English: Spanish-English lexical overlap increases ambiguity for character-based language identification, while within-word inflection raises out-of-vocabulary rates for non-contextualized embeddings.
  • Hindi-English: Hindi-English code-switching commonly uses nonstandard transliteration into the English alphabet, limiting the usefulness of resources built for Hindi in Devanagari.
  • Hindi-English: Hindi-English also poses challenges through Hindi’s flexible word order, which differs from English’s subject-verb-object structure.
  • Nepali-English: Nepali-English combines arbitrary romanization with contrasting subject-object-verb and subject-verb-object structures, encouraging finer-grained alternations.
  • Modern Standard Arabic-Egyptian Arabic: Modern Standard Arabic-Egyptian Arabic features substantial word overlap whose meanings can vary by language within a diglossic setting.

4. Tasks

LinCE organizes four code-switching tasks across multiple language pairs and datasets, while revising most dataset partitions to support more appropriate evaluation. The tasks cover language identification, POS tagging, NER, and sentiment analysis.

  • Benchmark scope: LinCE covers four tasks across four language pairs, using ten datasets and official benchmark partitions.The tasks are language identification, POS tagging, named entity recognition, and sentiment analysis.
  • Language Identification: Language identification uses Spanish-English, Hindi-English, Nepali-English, and Modern Standard Arabic-Egyptian Arabic corpora.The benchmark uses preexisting datasets and proposes revised partitions for the covered corpora.
  • Part-of-Speech Tagging: POS tagging covers Hindi-English and Spanish-English corpora with universal POS tags and language-identification annotations.The Hindi-English corpus adds PART NEG and PRON WH labels, while the Spanish-English corpus uses bilingual conversational data.
  • Named Entity Recognition: NER covers Spanish-English, Modern Standard Arabic-Egyptian Arabic, and Hindi-English, with BIO-style entity annotations and language labels where available.The MSA-EA test set has NER annotations but no language-identification analysis, whereas HIN-ENG includes person, location, and organization entities.
  • Sentiment Analysis: Sentiment analysis predicts positive, negative, or neutral labels for Spanish-English posts and includes token-level language-identification annotations.The SentiMix corpus contains 18,789 tweets and receives new partitions to correct the original label distribution.
  • Stratification: New splits are proposed for nine of ten datasets to address distribution, label-coverage, and sentence-length problems.The stratification process considers language-identification labels, task-specific labels, and sentence lengths when available.

5. Baseline Models

LinCE evaluates whether a common model architecture can generalize across its code-switching tasks. Its baselines are bidirectional LSTM, ELMo, and multilingual BERT systems with task-specific prediction layers.

  • Modeling goal: LinCE motivates a single model architecture that can generalize across the proposed code-switching tasks.The experiments compare architectures previously shown to perform well across NLP tasks.
  • Bidirectional LSTM: The bidirectional LSTM predicts sequence labels from concatenated forward and backward token representations and pools token states for sentiment analysis.Sequence labeling uses a linear prediction layer, while sentiment classification uses max pooling over hidden representations.
  • ELMo: ELMo is fine-tuned for sequence labeling using high-level word representations and for sentiment analysis using mean-pooled token representations.The model combines character-level convolution with word-level LSTM sequence modeling.
  • Multilingual BERT: The multilingual BERT baseline uses a pretrained model trained on 104 languages with a task-specific prediction layer.The system is designed to test a multilingual pretrained architecture across the benchmark tasks.
  • Training setup: Across tasks, the model architecture remains unchanged except for the prediction layer.Training uses SGD for BiLSTM and ELMo, AdamW for BERT, batch size 32, and early stopping based on validation performance.

6. Results and Analysis

Pre-trained language models generally outperform the simple BiLSTM baseline on LinCE, with ML-BERT leading across tasks and language pairs. Semantic tasks are harder than syntactic ones, while ELMo remains close to ML-BERT on LID and POS.

  • ML-BERT demonstrates superior performance in each task for all available language pairs.
  • Pre-trained language models generally outperform the simple BiLSTM model across almost all tasks.
  • ELMo is approximately 1-4% behind ML-BERT on most LID and POS tasks, but the gap is approximately 8-10% for NER and 18% for SA.
  • The average performance gap between ELMo and BERT is approximately 6%.
  • NER and SA appear harder than LID and POS, indicating greater difficulty for semantic understanding than syntactic analysis in code-switched texts.

7. Conclusion

LinCE centralizes evaluation of code-switching technology across multiple tasks and language pairs using ten public datasets and revised stratified splits. The benchmark preserves corpus distributions more consistently than original partitions and provides strong-model baselines through an evolving platform.

  • LinCE centralizes evaluation across multiple code-switching tasks and language pairs using ten publicly available datasets.
  • The authors identify dataset issues such as absent test labels and substantially different split distributions that undermine evaluation.
  • New splits use stratification criteria including LID labels, task-specific labels, and sentence lengths.
  • The proposed LinCE splits preserve the full corpus distribution, unlike some original partitions.
  • LinCE reports strong baselines from state-of-the-art monolingual models, including BERT and ELMo.
  • The platform is intended to evolve by incorporating additional tasks and language pairs.

A.1. LID Label Scheme

The benchmark adopts standardized label schemes for language identification, part-of-speech tagging, and named entity recognition. These schemes define language, syntactic, and entity categories for consistent annotation across datasets.

  • LID Label Scheme: The CALCS language-identification scheme labels tokens as lang1, lang2, mixed, ambiguous, fw, ne, unk, or other.
  • LID Label Scheme: In English-paired datasets, lang1 denotes English, while lang2 denotes Spanish, Hindi, or Nepali; for MSA-EA, they denote Modern Standard Arabic and Egyptian Arabic.
  • LID Label Scheme: The LID scheme marks mixed words, uncertain language membership, foreign words, named entities, unknown tokens, and symbols or emoticons.
  • POS Tag Scheme: POS tagging uses the universal UPOS tagset with PRON WH and UNK extensions.
  • POS Tag Scheme: The POS inventory includes categories such as NUM, PART, PRON, PROPN, PUNCT, SCONJ, VERB, ADJ, ADP, ADV, AUX, CONJ, DET, INTJ, NOUN, and X.
  • NER Label Scheme: Named entity recognition uses CALCS BIO labels covering person, organization, location, group, product, title, event, time, and other.
Loading 2005.04322v1…