PaperlayerView exact revision on arXiv

Source-linked AI summary

Code-Switching Spoken Language Identification as Multi-Label Set Prediction

Shunsuke Mitsumori, Matthew Wiesner, Shigeo Morishima, Shinji Watanabe

arXiv:2610.01450v1eess.AScs.CL

TL;DR

Monolingual LID filters can leave code-switched speech in multilingual corpora, mismatching labels and content. The paper predicts language sets directly and compares this approach with atomic-pair and score-based classifiers. Oracle Top-k remains the strongest baseline, while set generation avoids its cardinality assumption but does not match its accuracy.

  • Problem

    Code-switched speech can pass through monolingual LID filters, creating mismatched labels that risk biased sampling, contaminated evaluation, and unintended languages in conditioned ASR training.

    Source: §I · p. 1

  • Method

    The paper formulates utterance-level CS-LID as multi-label language-set prediction, generating the languages present without a supplied language count or score threshold.

    Source: §I · p. 1, §III · p. 2

  • Results

    Oracle Top-k is the strongest baseline, while set generation avoids oracle cardinality but does not match Top-k accuracy.

    Source: §VII · p. 6

  • Takeaways & Limitations

    Robust set prediction for real-world CS remains open, with all approaches struggling on unseen-pair, English-dominant, and three-language speech.

    Source: §VII · p. 6

    Most CS-trained models struggle when assumptions break, including transfer from TTS-based training data to human-read speech.

    Source: §VI · p. 6

Abstract

from arXiv · show

Code-switched (CS) speech leaks through the monolingual language identification (LID) filters used to curate massive speech corpora, calling for CS-aware LID (CS-LID). We formulate utterance-level CS-LID as multi-label language-set prediction and propose a set generator that directly outputs the languages in an utterance, comparing it against atomic-pair and score-based classification baselines. Oracle Top-k is the strongest baseline, but thresholding fails because no single threshold separates CS from monolingual speech. Our set generator predicts the correct language count on unseen pairs without assuming the number of languages, but underperforms oracle Top-k in exact set accuracy. Our analysis identifies the key obstacles to robust CS-LID: oracle cardinality, threshold instability, language bias in CS training data, and the synthetic-to-real gap.

I. INTRODUCTION

Monolingual LID filters can leave code-switched speech in multilingual corpora, creating mismatches between labels and spoken content. The paper compares language-set generation with atomic-pair and score-based approaches, finding that generation avoids key decoding assumptions but does not yet match oracle Top-k accuracy.

I. INTRODUCTION

  • Code-switched speech leaking through monolingual LID filters mismatches language labels with multilingual content, risking biased sampling, contaminated evaluation, and unintended languages in conditioned ASR training.Source: §I · p. 1
  • As multilingual CS corpora expand, possible language pairs grow quadratically, making it unrealistic to collect training data for every pair.Source: §I · p. 1
  • Atomic-pair classes cannot naturally handle pairs absent from training, while Top-k requires the utterance’s language count and thresholding depends on an operating point that varies across conditions.Source: §I · p. 1
  • The proposed multi-label language-set prediction represents unseen pairs as combinations of known languages without requiring oracle language count or threshold selection.Source: §I · p. 1
  • Atomic-pair classification performs well on seen pairs but cannot represent unseen pairs; oracle Top-k is strongest among score-based baselines, while set generation avoids its assumptions but does not match its accuracy.Thresholding also fails to match Top-k accuracy.Source: §I · p. 1

II. RELATED WORK

Prior code-switching identification work includes fine-grained methods, but this paper focuses on utterance-level LID because it better fits corpus filtering.

II. RELATED WORK

B. Utterance level LID at Code-Switching Speech

Utterance-level CS-LID has used atomic pair classes or sets derived from individual-language scores; the paper proposes language-set generation to address their constraints.

B. Utterance level LID at Code-Switching Speech

  • Atomic-pair classification treats a CS pair as its own class, tying prediction to pairs observed during training.Examples include separate Hindi, English, and Hindi–English classes [14], and distinguishing jpn from jpn-eng [9].Source: B. Utterance level LID at… · p. 2
  • Score-based approaches rank individual languages or fine-tune LID models with soft language-distribution targets.Source: B. Utterance level LID at… · p. 2
  • Deriving a language set from scores either requires the language count for Top-k or an operating point that varies across languages and domains for thresholding.Source: B. Utterance level LID at… · p. 2

III. METHOD

The method compares atomic-pair and score-based classifiers with a language-set generator that predicts the languages present without requiring a supplied language count or score threshold.

III. METHOD

  • Atomic-pair classification combines two constituent languages into one class, but cannot predict a test-time pair absent from training even when both languages were learned individually.Source: §III · p. 2
  • Top-k decoding returns the highest-scoring languages using the reference language count, so oracle decoding requires that count at inference.Source: §III · p. 2
  • The soft-target classifier adapts LID with a KL-divergence objective, using one-hot targets for monolingual speech and equal-mass targets over constituent languages for CS speech.Source: §III · p. 2
  • The sigmoid multi-label baseline predicts a score for each language and is trained with binary multi-hot targets and binary cross-entropy.Source: §III · p. 2
  • Thresholding removes the need for oracle cardinality but requires choosing a score threshold.Source: §III · p. 2
  • The set generator predicts how many and which languages are present, then converts a generated language-token sequence into an unordered set.The set’s size supplies the predicted language count; one language is monolingual and two or more indicate CS.Source: §III · p. 2

A. Datasets

The experiments use five speech datasets, with FLEURS and CS-FLEURS used for training and evaluation, CS-YODAS for training only, and two naturalistic corpora reserved for evaluation. CS-FLEURS supports seen- and unseen-pair evaluation across distinct test subsets.

A. Datasets

  • FLEURS and CS-FLEURS are used for both training and evaluation, while CS-YODAS is training-only and the Bangor Miami and Multilingual Soap Opera corpora are evaluation-only.FLEURS provides monolingual LID data and covers 102 languages.Source: A. Datasets · pp. 2–3
  • CS-FLEURS combines human-read and synthetic speech; its test subsets distinguish seen and unseen language pairs, with XTTS1 seen-only and XTTS2 unseen-only.READ and MMS each contain both seen and unseen pairs; training speech is synthesized entirely with XTTS.Source: A. Datasets · p. 3
  • CS-YODAS supplies in-the-wild English pairings with Arabic, Mandarin, French, Hindi, Japanese, and Russian; the evaluation corpora cover Spanish–English and three-language South African CS.The Bangor Miami Corpus is used to examine varying English proportions; the Multilingual Soap Opera Corpus tests English, isiXhosa, and isiZulu speech.Source: A. Datasets · p. 3

B. Models

The models share an MMS-1B frontend, while classifiers and a Transformer set generator differ in output design and training. The evaluation reports exact set accuracy, including oracle Top-k results with task-specific k.

B. Models

  • All systems update the shared 965M-parameter MMS-1B frontend, and the set generator has 23.55M trainable parameters beyond it versus 9.32–9.33M for each classifier.The largest difference is less than 2% of the parameters updated during training.Source: B. Models · p. 3
  • The three classifiers share an ECAPA-TDNN-based architecture and optimization, differing in output head and objective; hard-target, soft-target, and sigmoid models use distinct CS handling or targets.Hard-target training converts predicted atomic pairs to language sets when trained with CS data; soft targets assign 0.5 to each constituent language, while the sigmoid model uses positive-class weight 50.Source: B. Models · p. 3
  • The set generator uses a 24-layer Transformer encoder and 4-layer decoder to generate language tokens, training with decoder cross-entropy and the lower loss across both token orders for CS utterances.It does not use CTC loss.Source: B. Models · p. 3
  • The set generator is trained in two stages: first with MMS-1B frozen, then initialized from the best validation-loss checkpoint with MMS-1B unfrozen.Each stage runs for three epochs; the first updates about 1/40 as many parameters as the second.Source: B. Models · p. 3
  • Table III reports exact set accuracy on FLEURS and CS-FLEURS, with oracle Top-k using k = 1 for FLEURS and k = 2 for CS-FLEURS.The table caption identifies column maxima, including oracle results.Source: Table III · p. 4

C. Experimental Configurations

The experiments comprise nine configurations that vary model family and whether code-switched training datasets are included. They report exact set accuracy and distinguish seen from unseen CS-FLEURS pairs.

C. Experimental Configurations

V. RESULTS

On CS-FLEURS, score-based classifiers with oracle Top-2 are the strongest baselines, while the language-set generator improves over atomic-pair classification on unseen pairs but remains lower in exact set accuracy. Results also expose unstable thresholds, errors in predicting language count, and difficulties with domain shift and English-dominant or three-language speech.

a) Overall results

  • On FLEURS, H1-Top1, H2, and H3 reach 96.7%, 97.3%, and 97.4%, while G1 reaches 95.6%.S1-Top1 is lower at 91.4%; M1-Top1 reaches 96.5%, avoiding the softmax target conflict because its per-language scores are not normalized to sum to one.Source: §V · p. 4
  • 59.7% overall for M1-Top2 and 58.7% for M2-Top2 make score-based Top-2 the strongest CS-FLEURS baselines; G1 reaches 11.1% on unseen pairs versus 0.0% for H2.G1 remains below both score-based baselines in absolute accuracy, and Top-k comparisons use oracle cardinality (k = 2 on CS-FLEURS).Source: §V · p. 4

b) CS-FLEURS subset analysis

  • On MMS unseen pairs, M1-Top2 reaches 87.3% versus G1’s 28.5%, while XTTS2 unseen pairs remain near zero across models.On Read seen pairs, score-based Top-2 models reach 82.7–88.6% while H2 and G1 are near zero; Read unseen pairs remain difficult, with M2-Top2 at 19.0%.Source: §V · p. 5

c) Effect of adding CS-YODAS as training data

  • Adding CS-YODAS yields limited Read-seen gains, degrades S1 overall from 52.0% to 33.0%, and raises M1 Read-unseen accuracy from 1.2% to 19.0%.The authors attribute the limited benefit to differences in utterance length, noise, and domain between CS-YODAS and CS-FLEURS.Source: §V · p. 5

d) Prediction cardinality

  • On MMS unseen pairs, G1 predicts two languages for 96.8% of utterances versus H2’s 38.5%, but G1’s exact set accuracy is only 28.5%.Both models almost always predict one language on Read, explaining their near-zero exact set accuracy there.Source: §V · p. 5

e) Threshold sensitivity

  • Thresholded exact set accuracy varies substantially across subsets, and no single threshold recovers oracle Top-k accuracy across them.Soft-target decoding needs a threshold below 10^-4 to detect a second CS language, around 10^-8 on Read, while FLEURS accuracy improves toward 10^-1; sigmoid’s best FLEURS accuracy is about ten points below Top-1.Source: §V · pp. 5–6, Figure 1 · p. 6

f) Effect of English proportion

  • On Bangor Miami English–Spanish speech, H2 and G1 fail almost completely across English-proportion groups, while M1-Top2 is strongest in every group.Score-based accuracy degrades as English proportion rises; the authors link this pattern to CS-FLEURS training data containing English only as a low-proportion embedded language.Source: §V · p. 6

g) Three-language code-switching

  • All four models achieve 0.0% exact set accuracy on three-language English–Xhosa–Zulu speech, leaving extension beyond two-language mixtures an open challenge.The authors also note that naive mixing of CS-YODAS with FLEURS and CS-FLEURS may be ineffective under the current setup given the datasets’ length, noise, and domain mismatch.Source: §V · pp. 5–6

VI. DISCUSSION

The discussion separates language-count prediction from identifying the languages themselves: the set generator predicts cardinality accurately on MMS unseen pairs but identifies their constituent languages less often. Oracle cardinality and the synthetic-to-real gap remain central obstacles to robust CS-LID.

VI. DISCUSSION

  • Atomic-pair, threshold, and TTS-trained approaches falter when their assumptions break, and transfer from synthetic training to human-read speech remains challenging.The synthetic-to-real gap remains the most pressing issue; naively adding CS-YODAS does not consistently help.Source: §VI · p. 6
  • On MMS unseen pairs, the set generator predicts cardinality correctly 96.8% of the time but identifies constituent languages only 28.5%, separating counting from language identification.Score-based classifiers do the reverse, externalizing cardinality to an oracle k.Source: §VI · p. 6
  • Score-based classifiers externalize cardinality to an oracle k, while the set generator predicts how many languages are present.The discussion treats predicting how many languages and which languages as separable subproblems.Source: §VI · p. 6

VII. CONCLUSION

The paper formulates utterance-level CS-LID as language-set prediction and compares it with atomic-pair and score-based classifiers. Oracle Top-k remains the strongest baseline, while robust set prediction for real-world CS remains open.

VII. CONCLUSION

  • Oracle Top-k is the strongest baseline, but it requires the language count, and thresholding is not a viable substitute.The set generator removes both assumptions but does not yet match Top-k accuracy.Source: §VII · p. 6
  • The set generator predicts language sets without assuming the language count or requiring a threshold, but does not yet match Top-k accuracy.Source: §VII · p. 6
  • All approaches struggle on unseen-pair, English-dominant, and three-language CS speech, leaving robust set prediction for real-world CS an open challenge.Source: §VII · p. 6
Loading 2610.01450v1…