PaperlayerView exact revision on arXiv

Source-linked AI summary

CoSE-E: A Benchmark for Code-switched Speech Evaluation in Enterprise Settings

Shama Gupta, Hoang H Nguyen, Chelsea Huang, Lindsay Devon Brin, Fanny Riols

arXiv:2609.35645v1cs.SDcs.CLcs.LGeess.AS

TL;DR

Enterprise code-switched speech is poorly covered by transcript-only evaluation, despite the importance of downstream voice-agent tasks. The paper constructs CoSE-E and evaluates ASR with transcription and answer-based metrics. Code-switching often raises WER, but much less often changes downstream answers; the benchmark uses synthetic audio rather than human recordings.

  • Problem

    Enterprise code-switching remains under-evaluated, and transcript-only metrics do not measure downstream voice-agent task impact.

    Source: §1 · p. 1

  • Method

    The paper builds a synthetic enterprise CS-ASR benchmark across five language pairs and evaluates transcription and task fidelity with WER, SWER, and an LLM-based AER harness.

    Source: §1 · p. 1, §3.2 · p. 4

  • Results

    63% of model × language pair × baseline combinations show a significant WER penalty, versus 15% under AER, while roughly 80% of utterances show no downstream answer change.

    Source: §6 · p. 9

  • Takeaways & Limitations

    WER alone can mischaracterize task readiness: systems with moderate WER may preserve task-critical content, while lower-WER systems may fail on critical entities.

    Source: §6 · p. 9

    The benchmark uses synthesized rather than human-recorded audio and excludes telephony bandwidth, disfluencies, and background noise, so reported results are an upper bound on deployment quality.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Code-switching (CS), a seamless alternation between languages within a single utterance, remains a critical challenge in automatic speech recognition (ASR). While prior works focus on conversational CS-ASR, enterprise settings demand evaluation of operational impact beyond edit-distance errors: how code-switching transcription errors propagate to downstream voice agent task failures. In this work, we propose (1) a CS-ASR synthetic benchmark and multidimensional evaluation framework tailored to enterprise domains, (2) systematic evaluation of frontier ASR systems across 5 language pairs, (3) diagnostic analysis of the additional transcription errors that code-switching introduces across language pairs and models. We release COSE-E to support enterprise-focused CSASR evaluation for multilingual voice agents in enterprise deployment.

1 Introduction

Enterprise code-switching (CS) is a consequential but under-evaluated setting for ASR, and existing transcript-focused metrics do not capture downstream task impact. The paper introduces CoSE-E and an evaluation framework to assess transcription, meaning preservation, and code-switching-related error patterns.

1 Introduction

  • Enterprise-specific CS corpora are scarce, leaving ASR systems inadequately evaluated where transcription failures have operational consequences.The paper focuses on customer service, IT support, and business process automation, where speakers switch languages for domain-specific terminology and technical concepts.Source: §1 · p. 1
  • Standard WER, CER, and MER assess transcription in isolation and do not quantify downstream effects such as ticket-routing or policy-retrieval failures.The authors expect code-switching to amplify error propagation, but state that this remains untested for enterprise code-switching.Source: §1 · p. 1
  • CoSE-E is a synthetic, linguistically validated benchmark of 1,212 utterances across five language pairs, grounded in HR and IT Service Management workflows.The framework evaluates eight ASR models using WER, SWER, and AER to capture transcription accuracy and meaning preservation.Source: §1 · p. 1
  • The paper also analyzes the additional transcription cost of code-switching beyond monolingual baseline errors, identifying language-pair- and model-specific failure patterns.The benchmark and evaluation are intended to support enterprise-focused CS-ASR evaluation for multilingual voice agents.Source: §1 · pp. 1–2

2 Related Work

Prior CS-ASR benchmarks primarily evaluate conversational speech and transcript quality, while enterprise interaction and downstream task impact remain less covered. CoSE-E is presented as a benchmark designed for that enterprise setting.

Code-Switched Speech Corpora

  • Wide-coverage benchmarks such as SwitchLingua (Xie et al., 2026) and CS-FLEURS (Yan et al., 2025) span many language pairs, while pair-specific corpora include SEAME (Lyu et al., 2010) and ASCEND.These resources typically report per-language WER or CER on transcripts, partitioned by matrix and embedded language.Source: §2 · p. 2

Enterprise Code-Switching and Downstream Evaluation

  • Existing corpora evaluate transcripts in isolation, although ASR errors can propagate to spoken QA and code-switched LLM processing.Enterprise switches often introduce high-information English terms, including VPN, payroll, and ticket, that may lack settled equivalents.Source: §2 · p. 2
  • The closest prior benchmark, Abdoli et al. (2026), covers three language pairs but lacks Mandarin–English as a distinct-script pair, native-speaker review, and end-to-end downstream evaluation.Earlier corpora are generally drawn from social or conversational speech rather than enterprise interaction.Source: §2 · p. 2

3 Benchmark Design

CoSE-E builds synthetic enterprise code-switched speech through a staged data pipeline, then evaluates transcription and downstream meaning with complementary metrics. Its dataset enforces code-switching in every utterance and uses linguist review to validate generated data.

3.1 Data Pipeline

  • The pipeline filters parallel HR and ITSM utterances to 12–40 words, excludes utterances dominated by non-lexical entities, and requires at least three switchable content words.Entities remain in the benchmark; the entity filter removes cases where entity strings crowd out lexical material for studying switching.Source: §3.1 · p. 2
  • LLMs generate diverse code-switched text variants using unconstrained persona prompts, followed by normalization and verbalization before audio synthesis.The text variants are grounded in HR and ITSM workflows across five language pairs.Source: §3.1 · pp. 2–3
  • Audio uses ElevenLabs Multilingual V2 for the four European pairs and V3 for ZH/EN, with two native-speaker-verified voices per language and balanced gender representation.The normalization pass standardizes pronunciation of numbers, symbols, and special characters.Source: §3.1 · p. 3
  • Professionally trained linguists review utterances for natural switching and pronunciation, excluding flagged cases or returning them for regeneration and review.Reviewers are native-fluent in each matrix language and familiar with enterprise code-switching norms.Source: §3.1 · p. 3
  • CoSE-E contains 1,212 utterances across five language pairs, with zero monolingual records; European pairs have 4.9–6.9 switches per utterance, while ZH/EN has 2.86.ZH/EN also has lower CMI (0.10) and switch density (0.15), consistent with insertional switching; European pairs have CMI 0.26–0.33 and switch density 0.13–0.29.Source: §3.1 · pp. 3–4, Table 1 · p. 3

3.2 Evaluation Metrics

  • The evaluation reports WER as a surface baseline and SWER as a semantic metric alongside AER; ZH/EN additionally uses CER.AER is the primary metric, while the complementary metrics support comparison with prior CS- and multilingual-ASR work.Source: §3.2 · p. 4
  • AER measures whether transcription errors change answers to questions about critical entities, technical terms, and action-bearing tokens.Questions are generated from the reference transcript, checked by human annotators, answered from both reference and hypothesis, and compared by an alignment judge.Source: §3.2 · p. 4
  • AER scores answers rather than strings, avoiding normalization choices that destabilize WER and directly scoring information-dense content rather than averaging it uniformly.The authors describe AER as a task-aligned proxy, not a substitute for measured task completion.Source: §3.2 · p. 4

4 Results

Across models and language pairs, surface and meaning-aware metrics produce different rankings, showing that transcription error rates alone do not represent enterprise task fidelity. The results also show that language-pair difficulty depends on the metric used.

4.1 Model performance across metrics

  • WER, SWER, and AER separate eight models significantly, while the WER results place Scribe-v2, Gemini-3-Flash, and AssemblyAI at the top, a four-model middle tier, and Whisper far below.Whisper translates rather than transcribes code-switched audio under automatic language detection; Table 2 reports the model-by-language-pair metrics.Source: §4.1 · p. 5, Table 2 · p. 5
  • Qwen ranks sixth on ES/EN WER (0.042) but first on AER (0.027), ahead of every model, including Scribe-v2.The difference illustrates that a mid-pack WER model can rank first on AER.Source: §4.1 · p. 5, Table 2 · p. 5
  • The top four systems are mutually non-significant on AER after Holm correction, and the best system depends on the language pair and metric.Scribe leads almost everywhere on WER, while Qwen has the lowest AER on ES/EN and ZH/EN; Gemini has the second-lowest AER on ES/EN.Source: §4.1 · p. 5
  • All three metrics agree on a coarse three-tier model ranking (Kendall’s W = 0.963), but AER distinguishes models where WER and SWER rankings collapse.For Chinese, Gemini’s WER changes from 0.167 to 0.090 after Traditional-to-Simplified conversion, while its AER remains 0.059.Source: §4.1 · p. 5

4.2 Model performance across language pairs

  • Across Latin-script pairs, Friedman tests find significant differences for SWER (χ2(3)=13.95, p=.003, W=0.58), WER (χ2(3)=10.05, p=.018, W=0.42), and AER (χ2(3)=9.45, p=.024, W=0.39).The signal is sharpest on SWER, the most consistently detectable language-pair difference; Table 3 presents mean ranks.Source: §4.2 · p. 6, Table 3 · p. 6
  • SWER and AER rank DE/EN easiest and FR/EN hardest, whereas WER ranks ES/EN easiest and FR-CA/EN hardest.German’s morphology, compounding, and casing produce surface errors that rarely affect task entities, while French errors more often affect consequential content.Source: §4.2 · p. 6
  • In the five-model comparison, ZH/EN is hardest by WER (mean rank 5.0) and nearly hardest by SWER (4.6), but FR/EN is harder by AER (4.4 vs. 4.2).Chinese surface errors scatter across word forms, segmentation, and normalization rather than concentrating on consequential content.Source: §4.2 · p. 6

5 Discussion

Code-switching generally raises transcription error, but its measured cost shrinks from word-level error to downstream task failure. The remaining task failures are often tied to corrupted critical entities, while error patterns vary by language pair and model.

5.1 What additional cost does codeswitching add compared to plain monolingual speech?

  • The analysis pairs records across code-switched and monolingual runs, then tests per-utterance metric deltas; positive deltas mean code-switching increased error.Code-switched speech is transcribed language-agnostically, while monolingual speech uses its unambiguous language tag; tests use sign-flip permutations and Holm–Bonferroni correction.Source: §5.1 · p. 6

5.1.1 Code-switching raises word-level error

  • 63% of evaluated model × language pair × baseline combinations had a significant positive WER delta; strongest systems stayed near +0.02 to +0.05, while mid-tier systems reached +0.07 to +0.10 on harder pairs.The figure compares WER, SWER, and AER across two monolingual baselines, with models grouped by tier and language pairs distinguished by color.Source: §5.1.1 · pp. 6–7, Figure 2 · p. 7
  • Whisper’s language-agnostic decoding fails categorically: it translates part of mixed speech into English, with ZH/EN WER deltas reaching +2.5.The authors report this separately from incremental code-switching costs.Source: §5.1.1 · p. 7

5.1.2 Code-switching hurts some language pairs more than others

  • FR/EN had the most consistent cost, with significant penalties across WER, SWER, and AER; DE/EN penalties were largely surface-level, while ZH/EN changed little.FR/EN significance occurred in 10/14 WER, 13/14 SWER, and 4/14 AER combinations; DE/EN was significant in 10/12 WER, 2/12 SWER, and 0 AER combinations. The authors hypothesize ZH/EN’s smaller switch count and CMI explain its limited change.Source: §5.1.2 · p. 7

5.1.3 Code-switching leaves meaning largely intact—but the errors that survive are consequential

  • The significant penalty rate fell from 63% under WER to 50% under SWER and 15% under AER, while roughly 80% of utterances were unchanged under SWER and AER.Most introduced word errors left meaning or the agent’s answer intact, but significant AER penalties mark clusters of utterances where the downstream answer changes and the task likely fails.Source: §5.1.3 · pp. 7–8

5.1.4 Code-switching errors that reach AER are critical-entity failures

  • 57% of AER-reaching cases involved a corrupted critical entity, and an introduced entity error worsened AER about 40% of the time versus a roughly 5% base rate.That is an eightfold increase; reformatting an entity alone can leave the answer unchanged.Source: §5.1.4 · p. 8, Table 4 · p. 8
  • Against the non-English baseline, entity errors were introduced in 9.9% rather than 13.6% of records, but propagation risk ratios remained similar at 7.6 versus 8.3.Aggregate entity error fell by −0.02 against that baseline and rose by +0.05 against the English baseline; the entity-error link to task failure held for both.Source: §5.1.4 · p. 8

5.2 Do code-mixing index, switch count, and utterance length predict transcription errors?

  • CMI did not predict error occurrence, while switch count predicted onset robustly mainly in FR/EN; elsewhere its significant associations were sparse or absent.In FR/EN, 6 of 7 models showed elevated error odds per additional switch, controlling for length. Length dominated error magnitude partly because WER is normalized by utterance length; CMI and switch count showed no consistent severity relationship.Source: §5.2 · pp. 8–9

6 Conclusion

The conclusion presents CoSE-E as an enterprise-focused benchmark for evaluating code-switched ASR across five language pairs and three metrics. Its results show that many word-level penalties do not change downstream answers, while the task failures that remain are often associated with critical-entity corruption.

6 Conclusion

  • CoSE-E comprises 1,212 linguistically validated utterances across five language pairs and evaluates eight frontier ASR systems using three metrics.The benchmark is presented as enterprise-focused.Source: §6 · p. 9
  • 63% of model × language pair × baseline combinations had a significant WER penalty, compared with 15% under AER, where roughly 80% of utterances showed no downstream-answer change.The conclusion characterizes the cost as real but largely surface-level, with an eightfold increase in task-failure probability when a critical entity is destroyed.Source: §6 · p. 9
  • WER alone can mischaracterize enterprise CS-ASR readiness because transcription accuracy does not establish whether task-critical content is preserved.The paper supports CS-ASR evaluation for multilingual voice agents in enterprise settings through its release of CoSE-E.Source: §6 · p. 9

Limitations

The benchmark’s results are bounded by synthetic audio, limited language and domain coverage, and a zero-shot snapshot of off-the-shelf systems. Its AER measure is a task-success proxy rather than an end-to-end voice-agent evaluation.

Synthetic audio

  • Synthetic audio omits telephony bandwidth, disfluencies, and background noise, so reported results are an upper bound on deployment quality.The audio is synthesized rather than recorded from human speakers.Source: Limitations · p. 9

Single vendor bias

  • CoSE-E uses one TTS provider, and ElevenLabs Scribe V2 may benefit from familiarity with that provider’s TTS characteristics.The authors identify multiple TTS providers as future work to assess TTS-independent performance.Source: Limitations · p. 9

Coverage

  • Coverage is limited to five English-embedded language pairs and HR and ITSM; other high-volume pairs and enterprise verticals are out of scope.Only Mandarin contributes a non-Latin script.Source: Limitations · p. 9

Corpus scope

  • The corpus contains fully code-switched, intrasentential insertional switches, excludes alternational and inter-turn switching, and does not separate established English loanwords from embedded English.Mandarin–English has lower switch density, so cross-pair comparisons involving Mandarin conflate typology with model capability.Source: Limitations · p. 9

AER harness

  • AER is an LLM-based question-answering proxy rather than an end-to-end evaluation, and rankings may change with another judge configuration.The stochasticity study bounds noise at the reported scale but does not rule out systematic judge sensitivity; AER is more compute-intensive than WER and intended for model selection, not continuous monitoring.Source: Limitations · p. 9

Evaluation scope

  • Evaluation is zero-shot on off-the-shelf systems as of mid-2026; targeted fine-tuning and contextual biasing are not measured.Absolute scores are a point-in-time snapshot; the authors expect qualitative patterns to generalize.Source: Limitations · p. 9

A Prompts

The appendix documents prompts and preprocessing used to generate code-switched data and evaluate transcription meaning. It describes runtime-substituted prompt fields, text verbalization, and the question-answering stages of AER.

A Prompts

  • Curly-brace tokens such as {Utterance} mark fields substituted at runtime in the dataset-building prompts.The appendix presents prompts used to build the dataset.Source: A Prompts · p. 12

A.1 Data Generation

  • GPT-5 generates code-switched utterances from parallel matrix-language and embedded-language pairs, with the persona language swapped for each pair.The example prompt asks for natural workplace speech in HR and IT support contexts.Source: §A.1 · p. 12

A.2 Answer Error Rate (AER) Metric Prompts

  • AER uses three comprehension questions per utterance, validated for answerability; reference answers are compared with answers derived from transcribed text.Gemma4-31B IT generates questions and judges alignment, while GPT-4.1 generates reference answers.Source: §A.2 · p. 12
  • Question prompts require English questions answerable from explicit utterance content, excluding questions that seek information the speaker is asking for.The validation prompt checks that answers are fully and explicitly stated in the utterance.Source: §A.2 · pp. 12–13
  • Answer-generation prompts require one concise, English answer per question using only utterance information, with “N/A” when a question cannot be answered.The alignment prompt treats semantically equivalent answer pairs as matches and returns a Boolean per question.Source: §A.2 · p. 13

A.3 Verbalization Prompts

  • Before synthesis, a deterministic script normalizes numbers and an LLM verbalizes abbreviations, symbols, and context-sensitive identifiers for speech.The multilingual prompts preserve code-switching and use English separator words for technical strings; monolingual prompts also define spoken forms for identifiers and quantities.Source: §A.3 · pp. 13–14
  • Verbalization prompts expand abbreviations and render symbols in the surrounding language while preserving punctuation and product names.The English prompt leaves acronyms untouched and specifies spoken forms for numbers, symbols, and identifiers.Source: §A.3 · pp. 14–15

B.1 Judge Stochasticity Experiment

The stochasticity experiment finds that AER judgments are nearly deterministic when only the judge is rerun, with modestly more variation when answer generation is also resampled. Human review also finds high answerability and inter-annotator agreement for generated questions.

B.1 Judge Stochasticity Experiment

  • The experiment fixes questions and ground-truth answers, reruns the judge alone, then resamples the answerer to separate judge noise from answerer variation.The first experiment freezes LLM2 answers; the second passes newly generated answers through LLM3.Source: §B.1 · p. 15
  • 0.7–1.0% of records were non-unanimous when LLM2 was resampled, versus 0.0%–0.5% with the judge alone; answer variation was the primary instability source.German had zero judge-only disagreements and the highest resampled flip rate, 1.0%.Source: §B.1 · p. 15, Table 5 · p. 16
  • 12 of 19 flips newly accepted a match and 7 newly rejected one, suggesting resampling more often produced a different but semantically adequate answer.This is the authors’ interpretation of the direction of observed flips.Source: §B.1 · p. 15

B.2 Human Validation of Questions

  • 97.5–99.8% of generated questions passed individual answerability review, and inter-annotator agreement was AC1 = 0.978–0.986 across languages.Three annotators reviewed each language, except ZH/EN, which used two; no language was an outlier in validation or agreement.Source: §B.2 · pp. 15–16, Table 6 · p. 16

C Evaluation Details

Evaluation details cover normalization, synthetic speech preparation, annotation, and scoring choices, then report metric- and model-specific diagnostics. The results show that surface-form differences and translation-mode transcription can materially affect interpretation of scores.

C.1 Normalization; C.2 TTS Synthesis

  • Text preparation uses deterministic number rules followed by LLM verbalization, then synthesizes European pairs with ElevenLabs V2 and ZH/EN with V3.Normalization covers digit runs and phone-like patterns; verbalization expands abbreviations, converts symbols, and renders identifiers and measurements for speech.Source: §C.1 · p. 16

C.2.1 Annotation Procedures

  • Native bilingual reviewers judged audio naturalness, utterance naturalness, and fidelity; utterances not theoretically transcribable with perfect accuracy after alternate rendering were excluded.The review process sought to isolate ASR performance from audio-quality limitations.Source: §C.2.1 · p. 17

C.3 Evaluated ASR System Settings; C.3.1 Judge Settings

  • Models used default settings for unspecified parameters; SWER used Gemma 4, and AER reference answers used GPT-4.1 with fixed decoding settings.The tested systems’ configurations are listed in Table 8; judge settings are specified for reproducibility.Source: §C.3 · p. 17, §C.3.1 · p. 17, Table 8 · p. 18

D.1 Statistical Significance and Concordance Tests

  • Friedman tests found model differences on every metric, while within-metric ordering agreement declined from WER to SWER to AER; cross-metric agreement was 0.963.Confidence intervals were bootstrapped over clips, and the reduced SWER sample reflects complete-case analysis after judge-failure drops.Source: §D.1 · p. 17, Table 9 · p. 18

D.3 Whisper Language Parameter Behavior

  • Whisper translates the remainder of a mixed utterance into English when no language is specified; on ZH/EN its deltas reach +2.5 WER, +0.45 SWER, and +0.43 AER.The paper excludes Whisper from the main analysis because this failure mode is categorically different from other systems.Source: §D.3 · p. 17

D.2 Normalization Effect on ZH Results

  • WER drops 46% (0.167 → 0.090) after Traditional-to-Simplified conversion for Gemini-3-Flash on ZH/EN, while SWER and AER are unchanged.The script variants otherwise count as substitutions under jieba word-level WER.Source: §D.2 · p. 17, Table 10 · p. 18

D.4 Model performance on Language Pair Including Chinese

  • Table 11 reports that ZH/EN ranks last (hardest) on every metric, with mean ranks averaged across the five models having ZH/EN coverage; its WER uses jieba word-level segmentation.Cross-pair WER comparisons involving Chinese are approximate.Source: Table 11 · p. 18, §D.4 · p. 18

D.5 Per-Model Code-Switching Deltas

  • 1 of 35 non-English AER model–pair cells was significant: Qwen on ZH/EN, where code-switching was easier (∆= −0.037); significance thinned from WER through SWER to AER.The per-model pattern supports metric attenuation beyond the aggregate result.Source: §D.5 · p. 19, Figure 4 · p. 19

D.7 Dataset composition

Dataset composition differs by language pair, error base rates shape the available evidence for predictor analysis, and predictor collinearity is generally limited. ZH/EN is the main structural exception, with stronger overlap between switch count and CMI.

D.7 Dataset composition

  • ZH/EN averages fewer than three switches per utterance and CMI of roughly 10, while the other four pairs average at least five switches and CMI above 25.Utterance lengths are comparable across pairs.Source: §D.7 · p. 19

D.8 Error base rates

  • 96–97% WER>0 base rates for Whisper on DE/EN and ZH/EN leave too few zero-error cases for reliable Part A predictor estimates.Lower-base-rate models, including Scribe v2, Gemini-3, and AssemblyAI, have more balanced splits and greater Part A statistical power.Source: §D.8 · p. 19

D.9 Predictor collinearity

  • All predictor VIFs are below 5, supporting interpretable partial coefficients, although ZH/EN has substantial CMI–switch-count correlation (r = 0.58).Across pairs, switch count and log(nwords) correlate moderately (r = 0.28–0.66); in ZH/EN, each additional switch shifts language balance more.Source: §D.9 · pp. 19–20

E Examples of Errors Captured by AER Specifically

The appendix examines whether code-switching-related entity errors correspond to downstream task degradation, using paired records and examples that distinguish destructive corruption from cosmetic edits. Its results also report entity-error patterns by category and baseline.

Method

  • Entity WER scores tagged critical-entity tokens, while propagation pairs same-ID code-switched and monolingual records to test new entity errors against AER worsening; Whisper is excluded.Source: E Examples of Errors Captured by AER… · p. 20

Worked examples

Entity error rates by category

  • Against English, entity WER rises +0.09–0.12 on IDs, emails, URLs, and phones; against non-English, the overall effect is −0.02.Names are unaffected in either comparison; dates and measurements are omitted from the non-English comparison because localization makes word counts incomparable.Source: E Examples of Errors Captured by AER… · p. 20, Table 19 · p. 23

Propagation contingency tables

Loading 2609.35645v1…