PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Word Error Rate: A Switch Aware Evaluation of ASR and Audio Language Models on English Yoruba Code-Switched Speech

Chibuzor Okocha, Christan Earl Grant

arXiv:2609.11786v1cs.CLcs.AI

TL;DR

Monolingual benchmarks provide limited evidence about ASR and audio-LM behavior on low-resource, diacritic-rich code-switched speech. This paper evaluates eleven systems on English–Yoruba speech with switch-aware metrics and finds that aggregate WER obscures severe Yoruba and switch-boundary failures, while some audio LMs add prompt-sensitive generation errors.

  • Problem

    The behavior of ASR systems and audio language models on low-resource, diacritic-rich code-switched speech is poorly characterized beyond monolingual benchmarks and aggregate WER.

  • Method

    The paper evaluates six ASR systems and five audio LMs on a deterministic English–Yoruba set using shared alignment-based, language-specific, and switch-localized metrics.

  • Results

    Aggregate WER masks code-switching behavior: a leading audio LM ties the best ASR model on WER but significantly outperforms it at switch boundaries, while faithful systems show near-ceiling Yoruba error.

  • Takeaways & Limitations

    Switch-localized and language-specific metrics should accompany WER, and audio LMs require verbatim-transcription constraints and prompt control for fair evaluation.

  • Takeaways & Limitations

    This evaluation-only study uses a single English–Yoruba corpus, so results may not transfer to other dialects, accents, or recording conditions.

Abstract

from arXiv · show

Automatic speech recognition (ASR) systems and audio language models (audio LMs) now report low error rates on monolingual benchmarks, but their behavior on code switched speech in low resource, diacritic rich languages remains poorly characterized. We present a switch aware evaluation of eleven modern systems (six ASR models and five audio LMs) on English Yoruba code-switched speech, using a deterministic 2000 utterance evaluation set and a shared scoring pipeline. Beyond word error rate (WER), we report switch localized diagnostics: a switch entry token error rate (SETER), windowed switch point error rates, language specific error rates, and a diacritic insensitive WER. Our central finding is that aggregate WER hides code switching behavior. The best system by WER (an ASR model) is statistically indistinguishable from a leading audio LM on WER, yet the audio LM is significantly better on every switch localized metric. Across faithful systems, Yoruba token recognition collapses (error 0.97 for almost all systems) while English tokens are recognized far better, and errors concentrate sharply at switches into Yoruba. Several generative audio LMs fail as exact transcribers, producing translation, verbosity, and prompt leakage that are strongly prompt dependent. We release manifests, metric implementations, and evaluation scripts to support reproducible, switch aware benchmarking for African code switched speech.

I. INTRODUCTION

English–Yoruba code-switched speech exposes weaknesses that monolingual benchmarks and aggregate WER can miss. This paper therefore benchmarks eleven systems with deterministic switch-aware diagnostics, finding that WER rankings diverge from switch fidelity and revealing failures in Yoruba recognition and generative transcription.

  • Motivation and contribution: Code-switching degrades tokenization, language priors, and decoding at language boundaries, especially when Yoruba is low-resource and orthographically rich.English–Yoruba speech is comparatively underexplored and includes dense lexical mixing, tone diacritics, and sub-dot characters.
  • Motivation and contribution: Aggregate WER can hide switch failures because matrix-language tokens dominate utterances and dilute errors on embedded-language tokens and switch points.A system may therefore appear strong by WER while failing where code-switching occurs.
  • Motivation and contribution: The study evaluates six ASR systems and five audio LMs on a shared deterministic 2000-utterance set using diagnostics for switch boundaries, embedded-language tokens, and diacritics.The evaluation-first design avoids proposing a new model and instead tests current systems under a common protocol.
  • Motivation and contribution: WER rankings diverge from switch-aware performance: the best ASR model ties a leading audio LM on WER, while the audio LM performs significantly better at switch boundaries.Across systems, WER is essentially uncorrelated with Yoruba token error and only weakly correlated with switch-entry error.
  • Motivation and contribution: Across systems, Yoruba recognition collapses and errors show a consistent switch-direction asymmetry, while generative audio LMs additionally exhibit translation, verbosity, and prompt leakage.The paper also releases manifests, metric implementations, and evaluation scripts for reproducible benchmarking.

II. RELATED WORK

Prior code-switching work concentrates on a few language pairs and commonly uses aggregate error metrics, while audio LMs are mainly evaluated for free-form understanding rather than faithful low-resource transcription. African speech resources are expanding, but English–Yoruba code-switched evaluation remains underdeveloped.

  • Code-switching ASR and metrics: Code-switching ASR research has mainly targeted Mandarin–English and Arabic–English corpora, typically using WER, CER, or MER that aggregate errors across utterances.These metrics underweight embedded-language tokens and switch points when the matrix language dominates.
  • Code-switching ASR and metrics: Monolingual fine-tuning can improve classical WER on code-switched tests while degrading accuracy on the words that actually participate in switching.Prior work motivates metrics that isolate code-switched content rather than relying only on utterance-level aggregates.
  • Audio language models: Instruction-following audio LMs combine speech encoders with large language models and are usually benchmarked for audio understanding or question answering, not verbatim code-switched transcription.This leaves their reliability as faithful transcribers of low-resource speech insufficiently characterized.
  • Low-resource and African speech: African speech technology remains underserved, although new resources cover African-accented English, African languages, multilingual evaluation, dialogue, and entity-rich recognition.Yoruba has also motivated work on tone and sub-dot diacritic restoration.

III. CORPUS

The corpus uses a deterministic, tagged English–Yoruba evaluation setup and computes masked error rates over language- and switch-specific position sets.

  • III. CORPUS: The evaluation uses AFRICODESWITCH validation audio with a deterministic 2000-utterance cap, while excluding metadata rows without available audio.The corpus contains 9.54 hours of available audio, with short, densely mixed utterances.
  • III. CORPUS: Reference language tags identify switch points, language dominance, switch density, and English- or Yoruba-specific token sets.The manifest is balanced across dominance categories but gender-skewed toward female speakers.
  • IV. SWITCH-AWARE METRICS: All metrics derive from minimum-edit-distance alignment and compute mean error over selected reference-position sets.The alignment provides substitution, deletion, insertion, and per-token correctness information.
  • IV. SWITCH-AWARE METRICS: WER is computed after normalization, while diacritic-insensitive WER strips Yoruba tone and sub-dot marks to isolate diacritic-related errors.The standard normalization is NFC, case-folding, and punctuation stripping; default scoring remains diacritic-sensitive.
  • IV. SWITCH-AWARE METRICS: SETER measures errors on the first token of each new language run, while SPER@k expands scoring to windows around switch entries.SPER@0 equals SETER; the evaluation foregrounds SPER@1 and SPER@3.
  • IV. SWITCH-AWARE METRICS: Output-quality flags identify blank, truncated, repetitive, overlong, or prompt-leaking generations rather than scoring transcription accuracy.These heuristics characterize generative failure modes separately from the error metrics.

V. EXPERIMENTAL SETUP

The benchmark evaluates six ASR systems and five audio language models under a shared zero-shot transcription protocol with greedy decoding.

  • V. EXPERIMENTAL SETUP: Eleven systems are evaluated: six ASR systems and five audio language models.The ASR set includes Parakeet, Qwen3-ASR, Canary-Qwen, Whisper variants, and Granite-Speech; the audio-LM set includes Kimi-Audio, Audio Flamingo, Qwen2-Audio, and Qwen2.5-Omni.
  • V. EXPERIMENTAL SETUP: All systems run zero-shot with greedy decoding and receive the same primary transcription instruction in the main benchmark.The protocol is intended to keep scoring comparable across model families.
  • V. EXPERIMENTAL SETUP: The study follows the released implementation for exact normalization, alignment, and switch-window conventions.These conventions determine how the shared metrics are computed.

A. WER does not explain switch behavior

Aggregate WER fails to capture switch behavior: Parakeet and Kimi-Audio are effectively tied on WER, but Kimi-Audio is significantly better on switch-localized metrics.

  • A. WER does not explain switch behavior: 66.1% WER makes Parakeet the best aggregate system, but Kimi-Audio ties it at 66.2% while winning five of seven remaining benchmark columns.Kimi-Audio wins the diacritic-insensitive WER, English error rate, and all three switch-localized metrics.
  • A. WER does not explain switch behavior: Kimi-Audio’s WER disadvantage versus Parakeet is nonsignificant at Δ=+0.13 points, p=0.76, whereas its SETER, SPER@1, and SPER@3 advantages are significant at p<0.001.The paired bootstrap uses 1,000 utterance-level resamples over the shared manifest.
  • A. WER does not explain switch behavior: Across systems, WER correlates strongly with English error (ρ=0.74) but not Yoruba error (ρ=−0.10), and only moderately with SETER (ρ=0.67).This indicates that aggregate WER is dominated by matrix-language accuracy rather than switch-specific behavior.
  • A. WER does not explain switch behavior: SPER@0 exceeds SPER@3 for every system, including Parakeet’s 68.9% versus 65.2%, showing that the switch token is the hardest boundary region.Error eases only gradually as the window expands away from the switch.
  • A. WER does not explain switch behavior: Among eight faithful systems, English token error is 33%–46%, while Yoruba token error is at least 96.6% and at least 97.5% except for auto-language Whisper.Parakeet’s recomputed rates are YO-ER 0.9903 and EN-ER 0.3444.

C. Errors localize at switches into Yoruba

Errors concentrate at language boundaries, especially when switching into Yoruba, and worsen as Yoruba becomes more dominant or switching becomes denser.

  • 0.95–1.00 EN→YO switch-token error contrasts with 0.40–0.53 for YO→EN across systems.Figure 1 shows the boundary asymmetry directly.
  • ∼0.42 to ∼0.65 to ∼0.83 token error occurs from English-dominant to balanced to Yoruba-dominant utterances.Kimi-Audio is consistently lowest within each dominance stratum.
  • 71.2% WER and 72.1% SETER occur at high switching density, versus 66.9% and 66.7% at low density.SETER rises more steeply, indicating greater degradation in boundary fidelity than aggregate WER.

E. Diacritics are not yet the bottleneck

Diacritics contribute little to current faithful-system WER because Yoruba recognition is already poor, while domain and generation failures remain important distinctions.

  • Diacritic stripping changes WER by at most 0.30 points for Parakeet, Canary, Qwen3-ASR, and Whisper.Kimi-Audio is the exception, with a +1.85-point change consistent with producing more Yoruba content.
  • Qwen2-Audio leaks prompts in 32% of outputs, while Qwen2.5-Omni shows 3.1% repetition and 3.7% length expansion.These output-quality flags identify generative pathologies rather than transcription accuracy alone.
  • Faithful systems are substitution-dominated at 69–73% of errors, whereas Qwen2-Audio and Qwen2.5-Omni have insertion shares of 46% and 65%.The contrasting profiles separate recognition failure from over-generating behavior.

VII. QUALITATIVE ANALYSIS

Qualitative examples show that switch-aware metrics expose boundary recovery, bounded mis-transcription, degenerate generation, and translation behaviors that aggregate WER can obscure.

  • Landing the switch: Kimi-Audio recovers “nu chalkboard” with SETER = 0, while several systems miss the embedded word with SETER = 1.The example illustrates a concrete switch-boundary advantage.
  • WER wins, switch lost: On a Yoruba-heavy utterance, Parakeet has WER = 1.00 while Kimi-Audio loops with WER = 4.39, although neither recovers the switch.Lower WER can therefore coexist with failure at the code-switch boundary.
  • Translation and language confusion: Generative models may translate, summarize, or emit unrelated scripts instead of transcribing the spoken code-switched content.These behaviors are penalized by switch-localized metrics but can be partly hidden on matrix-heavy utterances.
  • Human annotation: A 100-utterance sample targets translation, Yoruba deletion or substitution, English hallucination, boundary collapse, and diacritic loss for human annotation.The human taxonomy complements automatic error flags for semantic categories.

VIII. DISCUSSION

The discussion shows that WER obscures failures concentrated in Yoruba recognition and switch boundaries. It also identifies prompt control and boundary-focused interventions as important for using audio LMs and improving code-switched transcription.

  • WER rank is essentially uncorrelated with Yoruba token error and only weakly correlated with switch-entry error, making WER a poor proxy for code-switching fidelity.WER tracks English error at ρ = 0.74 but is uncorrelated with Yoruba error at ρ = −0.10.
  • Yoruba token error remains near ceiling while diacritic stripping barely changes WER, indicating inadequate Yoruba coverage rather than mainly orthographic mistakes.The discussion therefore prioritizes Yoruba lexical and acoustic coverage before diacritic restoration.
  • Near-total error when switching into Yoruba, compared with roughly half that when switching back to English, suggests a dominant matrix-language prior at boundaries.The authors connect this asymmetry to language balance in decoding rather than acoustic difficulty alone.
  • Boundary-focused remedies include language-balanced or switch-aware decoding and supervision targeting embedded-language entry tokens rather than uniform WER.
  • Audio LMs can match the best ASR on WER and outperform it at switch boundaries, but prompt-dependent translation, verbosity, and leakage make verbatim constraints essential.A direct prompt reduced Qwen2-Audio leakage from 34% to near zero.

IX. LIMITATIONS AND ETHICS

The study is limited to evaluation on one English–Yoruba corpus without training or fine-tuning, so transferability is constrained. The authors nonetheless release resources to support reproducible African code-switched benchmarking and identify switch-aware adaptation as a next step.

  • The evaluation uses a single English–Yoruba corpus without training or fine-tuning, limiting transfer to other dialects, accents, and recording conditions.
  • Gender-skewed sampling, reference-side tags, automatic alignment, and ongoing human annotation constrain demographic claims and the precision of fine-grained diagnostics.The sample shows no large male–female error gap, but the skew limits the strength of per-gender conclusions.
  • Released manifests, metric implementations, and evaluation scripts provide a reproducible baseline, while switch-aware fine-tuning and adaptation remain next steps.
Loading 2609.11786v1…