Source-linked AI summary
CS-YODAS: A Mined Dataset of In-the-Wild Code-Switched Speech
Brian Yan, Qingzheng Wang, Matthew Wiesner, Anuj Diwan, Olga Iakovenko, Alexander Polok, Injy Hamed, Shuichiro Shimizu, Iris Emerman Thomas Hain, David R. Mortensen, Peter Viechnicki, Shinji Watanabe
TL;DR
Large-scale speech resources often exclude code-switching, and existing code-switched datasets are limited in scope. CS-YODAS mines spontaneous speech with human-guided LLM validation; adding it to training improves results on two natural-read language-pair subsets where synthetic-only training scored zero.
Problem
Large speech resources often exclude mixed-language utterances, while automatic detection in noisy, web-mined speech is difficult and prone to false positives.
Source: §1 · p. 1Method
The paper introduces CS-YODAS, using human feedback to guide LLM-based detection and validation of in-the-wild code-switching.
Source: §1 · pp. 1–2Results
On CS-FLEURS READ, adding CS-YODAS improved fra–eng from 0% to 51.1% and hin–eng from 0% to 19.9%, where synthetic-only training failed across languages.
Source: §4.3 · p. 6Takeaways & Limitations
The authors conclude that synthetic code-switched training data has not obviated the need to collect spontaneous speech for robust spoken LID.
Source: §5 · p. 7The scale of the data precludes exhaustive manual verification, and residual transcription and audio-segmentation errors remain.
Source: §6 · p. 7
Abstract
from arXiv · showhide
We present CS-YODAS, a Creative Commons-licensed dataset of in-the-wild code-switched speech mined from multilingual YouTube data. Code-switching (CS), or the alternation between languages within an utterance or conversation, is common in multilingual settings but remains underrepresented in existing CS speech resources, which are typically small, domain-specific, or artificially constructed. Building on the YODAS corpus, we develop a scalable, human-in-the-loop pipeline for identifying and validating naturally occurring code-switching. The resulting dataset, which totals 313 hours and spans 7 matrix languages, provides diverse, real-world examples of spontaneous code-switched speech. We further analyze the distribution and characteristics of code-switching in the wild, examining language-pair frequencies and switching patterns, and report baseline results for spoken language identification. We hope that CS-YODAS will encourage broader and more comprehensive research on code-switched speech. Dataset link: https://huggingface.co/datasets/byan/cs-yodas.
1. Introduction
Large multilingual speech resources often exclude code-switching, while existing code-switched speech datasets tend to be limited in size, domain, or spontaneity. CS-YODAS addresses this gap with a scalable human-in-the-loop mining pipeline and a 313-hour dataset spanning seven matrix languages.
1. Introduction
- Monolingual labels and mining pipelines that exclude mixed-language content leave code-switching largely absent from large-scale speech resources.Code-switching is the alternation between languages within an utterance or conversation.Source: §1 · p. 1
- Existing code-switched datasets tend to be small, domain-specific, or elicited, while synthetic speech may miss the spontaneity, prosody, and sociolinguistic nuance of natural speech.The paper frames these limitations as a motivation for in-the-wild data.Source: §1 · p. 1
- 313 hours of transcribed code-switched speech across 7 matrix languages form the CS-YODAS dataset.The paper also characterizes code-switching in the wild and reports baseline evaluations for CS-aware spoken language identification.Source: §1 · p. 1
- A scalable human-in-the-loop pipeline uses human validation to guide LLM-based detection, improving precision while retaining scalability across web-scale speech data.The dataset captures conversational, entertainment, and educational settings, including spontaneous and semi-scripted interactions.Source: §1 · pp. 1–2
2. Dataset Construction
CS-YODAS is mined from multilingual YouTube speech in YODAS through a two-stage process: LLM-based candidate detection followed by human-guided validation. The pipeline addresses common false positives from noisy transcripts and language-like words that do not represent genuine switching.
2.1. Source Corpus and Motivation
- CS-YODAS uses cleaned YODAS speech from publicly available YouTube content, whose transcripts were automatically produced and whose original data use single-language labels.Preliminary inspection of the corpus revealed code-switching, motivating systematic extraction.Source: §2.1 · p. 2
2.2. Overview
- The pipeline first uses text-based language identification to produce candidate segments, then applies human-in-the-loop validation to produce the final CS-YODAS set.Human feedback on 700 mine_iter0 utterances is used as in-context examples while the LLM validates the candidate set to produce mine_iter1.Source: §2.2 · p. 2, Figure 1 · p. 2
2.3. Mining of Candidate Segments
- A multilingual LLM infers each transcript line’s primary matrix language and other languages, retaining segments where at least two distinct languages are detected.This can capture transliteration, named entities, or embedded phrases that token-level LID may miss.Source: §2.3 · p. 2
2.4. Human-in-the-Loop Validation
- Noisy transcripts, borrowed words, and named entities can resemble code-switching without being genuine language alternation, so annotators validate candidate segments across seven matrix languages.The annotation procedure asks annotators five questions for each segment.Source: §2.3 · p. 2, §2.4 · p. 2, Table 1 · p. 2
1. Is the transcript correct?
Table 2 compares the precision of mine_iter0 and mine_iter1 samples against human evaluations.
1. Is the transcript correct?
- Table 2 reports precision for samples selected from mine_iter0 versus mine_iter1, measured by agreement with human evaluations.The supplied passage identifies the comparison and measure but gives no table values.Source: Table 2 · p. 3
5. Are all <lang2> words proper nouns?
Human-in-the-loop validation raises the precision of detected code-switching, though reliability varies across matrix languages. The final release captures contextual audio and intra-sentential switching across seven matrix languages, providing a spontaneous-speech resource distinct from synthetic collections.
2.5. Evaluating the Pipeline
- 70% precision was reached across 120 mine_iter1 segments, compared with 18% confirmed true code-switching among 700 mine_iter0 candidates.Table 3 breaks down the validation confusion matrix across the seven matrix languages.Source: §2.5 · p. 3, Table 3 · p. 3
- Hindi, Chinese, and Arabic produced the most true-positive code-switched samples, with precision rates of 82.9%, 76.9%, and 55.2%, respectively.Japanese and Russian yielded few true positives; proper-noun-only embedded-language samples made up a large portion of false negatives, indicating annotator disagreement about the Q5 = “No” filter.Source: §2.5 · p. 3
2.6. Capturing Code-Switching Context
- The 312.7 hours of contextual audio include 37.3 hours of intra-sentential code-switching, or 11.9%; Arabic and Hindi have higher code-switching rates.Contextual chunks pair target segments with surrounding audio.Source: §2.6 · p. 3
2.7. Dataset Statistics
- The final CS-YODAS release comprises 313 hours of speech across 7 matrix languages.Earlier efforts to broaden code-switched corpus coverage largely used synthetic generation, which may not capture natural switching dynamics.Source: §2.7 · p. 4
3. Dataset Analyses
The dataset analyses characterize code-switching across matrix languages, embedded-language and lexical patterns, and topical domains. The analyses use data estimated at 70% precision.
3.1. Overview
- The analyses add language-role, part-of-speech, and text or audio domain metadata, but are based on data estimated at 70% precision.This estimate qualifies the distributions reported in the section.Source: §3.1 · p. 4
3.2. Language Distributions
- Hindi had the highest mining yield at 6.87%, versus 0.01% for Russian; this yield is a lower-bound estimate of in-the-wild code-switching.The method prioritizes precision over recall, so the actual rate is expected to be somewhat higher than the yield.Source: §3.2 · p. 4, Figure 2 · p. 4
- English was the embedded language in 85.6% of examples, while non-English embedded languages were most frequent with Arabic matrix language (27.6%) and French (14.6%).Egyptian Arabic was the leading non-English embedded language for Arabic, and Arabic was the leading one for French.Source: §3.2 · p. 4, Figure 3 · p. 4
3.3. Lexical Distributions
- Both CS-YODAS and CS-FLEURS contained mostly content words and proper nouns among embedded English words, but CS-YODAS had nearly half the function-word proportion and included discourse markers.CS-FLEURS had no discourse markers; the authors suggest its synthetic text may have an unnatural lexical pattern.Source: §3.3 · p. 5, Figure 4 · p. 5
3.4. Domain Distributions
- Games, Internet and Telecom, and Computers and Electronics were the most over-represented domains, while Law and Government, Health, Science, and People and Society were under-represented.The comparison uses each domain's relative frequency in CS-YODAS divided by its relative frequency in YODAS; values above one indicate over-representation.Source: §3.4 · p. 5, Figure 5 · p. 5
4. Baseline Experiments
The experiments compare spoken language identification with and without CS-YODAS training data. Adding the in-the-wild data yields nonzero accuracy on two CS-FLEURS READ subsets, while the baseline fails across its languages.
4.1. Overview
- The experiments address spoken LID's monolingual training bias and report that synthetic code-switched training alone is insufficient for robust generalization, whereas spontaneous code-switching is essential.The paper frames the shortage of large-scale multilingual speech corpora with natural code-switching and reliable language labels as a data bottleneck.Source: §4.1 · pp. 5–6
4.2. Experimental Setup
- The comparison tests FLEURS plus XTTS-generated CS-FLEURS data against the same training data further augmented with CS-YODAS; both settings cover 102 monolingual languages and 16 code-switched pairs.Six of the 16 pairs are supplemented with CS-YODAS data.Source: §4.2 · p. 6
4.3. Results and Discussion
- Adding CS-YODAS raised CS-FLEURS READ accuracy from 0% to 51.1% for fra–eng and from 0% to 19.9% for hin–eng.Without CS-YODAS, the model failed across all languages on CS-FLEURS READ; the authors attribute this to a domain mismatch from lacking natural code-switched training data.Source: §4.3 · p. 6, Table 6 · p. 6
- Accuracy on CS-FLEURS READ did not rise above 0 until more than 5 hours of code-switched training data were available.The paper reports a clear trend between code-switched training-data duration and code-switched LID performance.Source: §4.3 · p. 6, Figure 6 · p. 7
5. Conclusion
The conclusion presents CS-YODAS as a large-scale dataset of spontaneous, naturally occurring code-switched speech.
5. Conclusion
- CS-YODAS provides large-scale spontaneous code-switched speech, and the paper concludes that synthetic code-switching neither fully mimics observed in-the-wild patterns nor removes the need to collect spontaneous speech.The authors hope the dataset will support future work on robust spoken LID for multilingual communication.Source: §5 · p. 7
6. Limitations
CS-YODAS is limited by its YouTube source, the inability to verify every example manually, and the language-specific expertise required for reliable evaluation.
6. Limitations
- The Creative Commons YouTube source skews CS-YODAS toward publicly available, broadcast-style material, while casual settings may be underrepresented.The dataset still captures code-switching in daily, informal contexts.Source: §6 · p. 7
- The dataset’s scale precludes exhaustive manual verification, and source transcription and audio segmentation errors leave residual noise.The human-in-the-loop pipeline was implemented to improve precision, not to verify every example.Source: §6 · p. 7
- Reliable evaluation requires skilled bilingual annotators, limiting language coverage; this release spans only 7 matrix languages.Source: §6 · p. 7
7. Ethics Statement
The authors report that the dataset is a mined subset of an already publicly released resource and do not foresee harm from its content.
7. Ethics Statement
- Because CS-YODAS is mined from an already publicly released dataset, the authors do not foresee harm arising from its content.Source: §7 · p. 7
- Project annotators volunteered and were informed about the mining approach.Source: §7 · p. 7
- Annotators agreed that their feedback could be used to refine the dataset.Source: §7 · p. 7
8. Example LLM Prompts
The appendix provides example prompts for text-based language identification and human-in-the-loop validation, supplementing the main description of the data-mining pipeline.
8.1. Overview
- The appendix’s prompt examples supplement the main description of the LLM-based data-mining pipeline.Source: §8.1 · p. 7
8.2. Text-based LID
- The text-based LID prompt asks the model to identify all languages in a segment, including cases with only one language.Source: §8.2 · p. 7
- The example pairs a Chinese-English target segment with a JSON response listing the languages in prevalence order.Source: §8.2 · p. 7
- The prompt defines code-mixing or code-switching as language alternation within a discourse, sentence, or constituent, and instructs word-by-word review.Source: §8.2 · p. 7
- Table 7 shows the in-context examples in blue and the target prompt and response in violet; the LID stage uses 3 examples to maximize throughput.Source: Table 7 · p. 7, §8.2 · p. 7
8.3. Human-in-the-Loop Validation
- Table 8 depicts in-context examples in blue and the target segment prompt and response in violet.Source: Table 8 · p. 8
- The validation prompt asks the LLM to replicate the human feedback pattern, using 100 examples from feedback on same-language samples in an attempt to maximize accuracy.Source: §8.3 · p. 8