PaperlayerView exact revision on arXiv

Source-linked AI summary

Code-Switching Information Retrieval: Benchmarks, Analysis, and the Limits of Current Retrievers

Qingcheng Zeng, Yuheng Lu, Zeqi Zhou, Heli Qi, Puxuan Yu, Fuheng Zhao, Hitomi Yanaka, Weihao Xuan, Naoto Yokoya

arXiv:2604.17632v1cs.IR

TL;DR

Code-switching is common but lacks systematic IR evaluation. The paper builds human-annotated and broad task benchmarks, analyzes retrieval and embedding behavior, and tests vocabulary expansion; results show substantial, incompletely mitigated performance degradation.

  • Problem

    Code-switching remains critically understudied in IR despite its prevalence in communication and search, leaving system capabilities without systematic evaluation.

  • Method

    The paper constructs human-annotated CSR-L and CS-MTEB benchmarks, evaluates multiple retrieval paradigms, analyzes embedding-space shifts, and tests lexicon-based vocabulary expansion.

  • Results

    Across CSR-L and CS-MTEB, code-switching degrades retrieval and embedding-model performance, with drops of up to 27%, while vocabulary expansion does not restore monolingual performance.

  • Takeaways & Limitations

    Code-switching is a persistent robustness gap involving more than vocabulary coverage and remains unresolved by surface-level adaptation.

  • Takeaways & Limitations

    The benchmarks focus on natural query-level mixing between English and a small set of partner languages, excluding broader phenomena such as romanization, transliteration, and mixed-language documents.

Abstract

from arXiv · show

Code-switching is a pervasive linguistic phenomenon in global communication, yet modern information retrieval systems remain predominantly designed for, and evaluated within, monolingual contexts. To bridge this critical disconnect, we present a holistic study dedicated to code-switching IR. We introduce CSR-L (Code-Switching Retrieval benchmark-Lite), constructing a dataset via human annotation to capture the authentic naturalness of mixed-language queries. Our evaluation across statistical, dense, and late-interaction paradigms reveals that code-switching acts as a fundamental performance bottleneck, degrading the effectiveness of even robust multilingual models. We demonstrate that this failure stems from substantial divergence in the embedding space between pure and code-switched text. Scaling this investigation, we propose CS-MTEB, a comprehensive benchmark covering 11 diverse tasks, where we observe performance declines of up to 27%. Finally, we show that standard multilingual techniques like vocabulary expansion are insufficient to resolve these deficits completely. These findings underscore the fragility of current systems and establish code-switching as a crucial frontier for future IR optimization.

1 Introduction

IR has expanded from lexical and dense methods toward multilingual systems, but code-switching remains critically understudied despite its prevalence in communication and search. This paper introduces a holistic evaluation spanning natural-query benchmarking, broad task evaluation, and vocabulary adaptation.

  • Motivation: Code-switching is understudied in IR despite widespread bilingual communication and substantial mixed-language search traffic.The paper cites code-switching in approximately 70% of the world’s population, more than 15 occurrences every 10 minutes in three speech communities, and around 27% of entertainment-domain Bing traffic.
  • Contributions: The study builds CSR-L, a human-annotated benchmark for natural code-switched queries and evaluates statistical, dense, cross-encoder, and late-interaction retrievers.CSR-L captures natural mixed-language queries while comparing multiple retrieval paradigms.
  • Findings: Query-side code-switching substantially degrades retrieval quality, including for strong multilingual retrievers, and accompanies a shift in embedding space.The findings connect retrieval degradation with representational divergence between original and mixed-language queries.
  • Contributions: CS-MTEB extends evaluation to 11 tasks across 7 task types, where advanced embedding models show performance drops of up to 27%.The benchmark broadens the investigation beyond standard retrieval.
  • Findings: Lexicon-based vocabulary expansion improves English-centric retrievers but does not restore monolingual performance.The intervention tests whether vocabulary adaptation can mitigate the observed embedding-space divergence.

2 Related Work

Related work frames IR as evolving from lexical matching toward semantic and multilingual retrieval, while existing benchmarks still leave code-switching insufficiently evaluated. Recent resources broaden language and reasoning coverage but do not directly resolve this gap.

  • IR and Embedding Models: Modern IR commonly combines efficient retrieval with reranking, while dense bi-encoders encode queries and documents into independent vector spaces.The field has shifted from lexical matching toward semantic representation in retrieve-then-rerank pipelines.
  • IR and Embedding Models: BRIGHT evaluates highly challenging realistic queries that require deep reasoning, extending assessment beyond conventional embedding capabilities.The benchmark is presented as a test of difficult, reasoning-intensive retrieval queries.
  • Multilingual and Cross-lingual Retrieval: Multilingual resources such as MMTEB cover hundreds of languages and tasks, yet code-switching remains a crucial linguistic phenomenon in need of systematic evaluation.Prior work addresses multilingual, cross-lingual, and cross-dialect retrieval without centering mixed-language queries.

3 Code-Switching Retrieval Benchmark-Lite (CSR-L)

CSR-L is constructed through human rewriting and validation to capture natural code-switching, then evaluated across several retrieval families on original and mixed-language queries. Its setup spans multiple IR domains and uses task-specific relevance metrics.

  • Building CSR-L: Human annotation is used because no standard automatic metric reliably judges code-switching naturalness.This design prioritizes data quality and supports more reliable benchmark evaluation.
  • Building CSR-L: CSR-L rewrites English queries from four IR datasets into English–Mandarin Chinese and English–Japanese code-switched forms.The source datasets cover argument, code, biomedical, and instruction-following retrieval.
  • Building CSR-L: Each rewrite is produced by one annotator and validated by a second, who may edit or discard it.The annotators were native Chinese speakers professionally proficient in English and Japanese.
  • Evaluation Setup: The benchmark evaluates BM25, bi-encoder retrievers, cross-encoder rerankers, and late-interaction methods.The model set includes multilingual and English-centric systems across these retrieval families.
  • Evaluation Setup: Performance is compared between original and code-switched queries using nDCG@10, except for FollowIR, which uses pairwise-MRR.Cross-encoder CSR-L results use direct full-corpus scoring rather than reranking a top-k candidate pool.
  • Evaluation Setup: Table 2 reports nDCG@10 and pairwise-MRR across four English–Chinese IR benchmarks, with macro-average drops computed as Avg(CSR-L) - Avg(Orig).Negative delta values indicate degradation under code-switching.

4 CSR-L Results

Across CSR-L, code-switching reduces retrieval performance for English-centric and multilingual systems, while multilingual training offers only partial robustness. The corresponding embedding-space shifts and cross-paradigm consistency indicate a broad challenge rather than a narrowly architectural one.

  • 4.1 General Results: Query-side code-switching substantially degrades performance even when documents remain unchanged.The decline is especially large on Touché 2020 and TRECCOVID, while HumanEval shows a milder decrease.
  • 4.1 General Results: English-centric bi-encoders lose roughly 15 points on the two general retrieval datasets, while Qwen3-Embedding drops exceed 8 points in some settings.The Qwen3 series is comparatively more robust, but even its 8B variant does not eliminate the gap.
  • 4.1 General Results: Multilingual retrievers generally show smaller relative declines than English-only systems, consistent with benefits from exposure to diverse languages during training.The paper describes this exposure as helping models interpret code-switching patterns and partially absorb linguistic disruption.
  • 4.1 General Results: No retrieval paradigm shows significant robustness variation, and cross-encoders are not more resistant despite their higher computational cost.The vulnerability also appears in statistical and late-interaction methods; cross-encoder absolute scores use direct full-corpus scoring.
  • 4.2 Embedding Space Analysis: Code-switching separates e5-large-v2’s original and mixed-language queries into disjoint dense embedding clusters.For Touché 2020, Qwen3-Embedding-0.6B has a smaller centroid distance than e5-large-v2, 0.20 versus 0.25.

5 CS-MTEB

CS-MTEB broadens evaluation beyond standard retrieval to test code-switching across diverse tasks, languages, and models. Performance declines persist across settings, with task demands shaping sensitivity.

  • 5 CS-MTEB: CS-MTEB expands evaluation across 11 tasks and 7 task types using an MTEB-style benchmark.The benchmark uses LLM-generated code-switched variants refined against human-authored CSR-L queries, with manual quality checks.
  • 5 CS-MTEB: Code-switching remains a broad and persistent bottleneck across models, tasks, and language pairs.The same qualitative pattern observed in CSR-L reappears in CS-MTEB.
  • 5 CS-MTEB: 10 to 15 points: e5-large-v2 loses approximately this much average score across four language mixtures.The degradation is similar for typologically distinct and closely related languages.
  • 5 CS-MTEB: Approximately 5 points: Arctic-Embed-m-v2.0 drops this much in Spanish-English, versus the ∼10 point drop for e5-large-v2.Multilingual training mitigates, but does not eliminate, the decline.
  • 5 CS-MTEB: Reranking is the most fragile task category, while pair classification is comparatively less sensitive to code-switching.The paper relates this difference to ranking objectives requiring precise ordering and alignment rather than coarse semantic cues.

6 Vocabulary Expansion for Retrieval

The paper tests lexicon-based vocabulary expansion as a low-cost way to transfer source-language semantics into target-language token embeddings. It improves robustness for English-only retrievers, but only partially closes the code-switching gap.

  • 6.1 Lexicon-Based Vocabulary Expansion: Vocabulary expansion projects source-language semantic capabilities onto target-language tokens using bilingual translation pairs.The approach leaves the retriever’s main body unchanged and initializes target embeddings from aligned source representations.
  • 6.1 Lexicon-Based Vocabulary Expansion: The method resolves word–subword granularity mismatch by tokenizing source translations and averaging their pretrained subword embeddings.Representations are then aggregated across valid translations for each target word.
  • 6.1 Lexicon-Based Vocabulary Expansion: Untranslated target tokens receive embeddings sampled from a standard normal distribution N(0, σ2).This is the fallback when the bilingual lexicon contains no translation for a target token.
  • 6.2 Experiments and Results: Lexicon-based vocabulary expansion consistently improves robustness for both evaluated English-only retrievers on CSR-L.The experiments adapt all-MiniLM-L12-v2 and e5-large-v2 for Chinese and Japanese.
  • 6.2 Experiments and Results: 30.09 to 37.73: all-MiniLM-L12-v2’s CSR-L-Chinese macro-average increases after adaptation, indicating partial recovery.For CSR-L-Japanese, the corresponding increase is 29.94 to 34.34.
  • 6.2 Experiments and Results: 35.32 to 43.50: e5-large-v2’s CSR-L-Chinese average improves after adaptation, while the code-switching deficit remains.The gains are driven primarily by general retrieval benchmarks, with smaller improvements on HumanEval.

7 Discussion

The discussion frames code-switching as a persistent bottleneck that is not fully resolved by multilingual training or vocabulary expansion. It distinguishes semantic alignment from retrieval relevance and points toward code-switching-specific training data.

  • 7 Discussion: Code-switching degrades modern IR across statistical, dense, and late-interaction architectures, while multilingual training only mitigates the severity.The resulting models still trail English-only settings.
  • 7 Discussion: Vocabulary expansion yields measurable improvements but does not eliminate the performance gap, indicating that vocabulary coverage alone is insufficient.The discussion characterizes the remaining challenge as semantic rather than merely a surface-level vocabulary problem.
  • 7 Discussion: Competitive multilingual semantic alignment can coexist with weaker retrieval relevance under code-switching.The paper contrasts synonym alignment in semantic tasks with more complex outcomes on CSR-L and CS-MTEB.
  • 7 Discussion: The findings support treating code-switching as a distinct linguistic modality in future training data.The discussion connects the failure of surface-level fixes with the need for dedicated handling of native mixed-language sequences.

Limitations

The study is limited by its controlled language and phenomenon coverage and by unavoidable annotation and generation noise. Its benchmarks focus on natural query-level mixing between English and a small set of partner languages.

  • Language and phenomenon coverage: The benchmarks focus on natural query-level mixing between English and a small set of partner languages.This controlled setting aligns with search behavior in which English technical terms appear inside otherwise non-English queries.
  • Language and phenomenon coverage: Code-switching beyond this setting includes romanization, transliteration, spelling variation, community-specific conventions, and mixed-language documents.These forms are not the primary focus and remain directions for future benchmark extensions.
  • Annotation and generation noise: Human judgment is required to determine whether a code-switched query is natural while preserving its original information need.The study uses bilingual annotators, validation checks, conservative rewrite guidelines, and manual spot checks to mitigate this issue.
  • Annotation and generation noise: Modest stylistic variation and occasional generation artifacts are difficult to eliminate entirely at scale.The authors therefore emphasize consistent trends across models and settings and release their resources for replication.
  • Annotation procedure: Annotators must preserve the query’s exact information need without adding or removing important details.Instructions require realistic bilingual phrasing, meaningful contributions from both languages, stable key terms, and roughly similar length.

B Additional Details on Evaluation

The evaluation follows standard MTEB settings with resource-driven adjustments, and each task type uses a specified metric. Batch size is reduced for most retrieval tasks, while FlashAttention 2 may cause slight leaderboard differences.

  • Evaluation settings: The evaluation generally follows the standard MTEB process, but retrieval-task batch size is reduced from 32 to 4 because of VRAM limitations.HumanEvalRetrieval is the exception.
  • Evaluation settings: FlashAttention 2 is enabled when supported to speed inference, which may produce slight differences from the public MTEB leaderboard.
  • Task metrics: Instruction reranking uses p-MRR, clustering uses V-measure, and classification uses Accuracy.
  • Task metrics: STS uses Cosine Spearman correlation, while Pair Classification uses mean average precision.

C CSR-L Results on Japanese

This section documents the Japanese CSR-L results and associated evaluation materials, including the Table 5 comparison of original and code-switched queries across four IR benchmarks. The appendix also provides prompt templates, rewriting examples, dataset statistics, and additional-language results.

  • Prompt templates: The Japanese CSR-L materials include prompt templates for English-Chinese and English-Japanese code-switching rewrites.The templates specify single mixed-language outputs, preserved intent, meaningful use of both languages, and realistic phrase-level mixing.
  • Japanese CSR-L results: Table 5 reports nDCG@10 and p-MRR for original and code-switched queries across four English-Japanese IR benchmarks.It also reports a macro-average and defines Drop Δ as Avg(CSR-L) - Avg(Orig), with negative values indicating degradation.
  • Prompt templates: The Chinese-oriented instructions retain product names, library names, frameworks, and proper nouns in English unless an extremely standard Chinese name exists.
  • Rewriting examples: Examples include Python data analysis 入门教程, best 游戏本for college students 性价比, and Notion personal knowledge management 入门教程.
  • Additional materials: Japanese-CSR-L statistics and rewritten-query examples are reported in Tables 10–14, while Table 15 reports CS-MTEB results for five additional languages.
  • Dataset statistics: The Japanese CSR-L dataset statistics describe query count, corpus size, average positive documents per query, and average token length.

H Additional Discussion on a Newly Curated Retrieval Benchmark

Additional evaluations show that code-switching degradation persists across newer benchmarks, standard two-stage retrieval, and non-English monolingual baselines. Manual inspection also finds that rewritten CS-MTEB queries generally preserve information needs and remain natural.

  • AILACaseDocs: On AILACaseDocs, both mE5-large and Qwen3-Embedding-0.6B degrade on Chinese code-switched queries, with a larger drop for mE5-large.The newer benchmark is outside the commonly used MMTEB leaderboard.
  • Two-stage reranking: A standard two-stage pipeline still degrades: Qwen3-Embedding-0.6B retrieval followed by either reranker performs worse on code-switched than original English queries.The setup reranks the top-100 retrieved candidates with jina-reranker-v3 or Qwen3-Reranker-0.6B.
  • Non-English baselines: In Chinese-centric retrieval, all three retrievers perform worse on Chinese–English code-switched queries than on original Chinese queries.The documents remain Chinese, separating code-switching from simply moving away from English.
  • Non-English baselines: The Chinese-centric comparison indicates that the degradation is not merely a consequence of moving away from English as the highest-resource language.The comparison is between monolingual Chinese and Chinese–English code-switching with unchanged documents.
  • Query quality: Manual inspection of 50 CS-MTEB rewrites evaluates naturalness and information preservation on independent 1–10 ratings from two raters.Information preservation is consistently close to the ceiling, while naturalness scores remain high.
  • Query quality: The quality check supports that rewritten queries largely retain the original search intent and use generally fluent, plausible language switches.The authors qualify this spot check as additional evidence rather than a replacement for full-scale human verification.
Loading 2604.17632v1…