Source-linked AI summary
AfriSwitch: A Benchmark for In-the-Wild African Code-Switched Speech Recognition
Gabrial Zencha Ashungafac, Busayo Awobade, Tobi Olatunji
TL;DR
African ASR is commonly evaluated on monolingual, curated speech despite pervasive real-world code-switching. AfriSwitch supplies a natural, switch-annotated benchmark across 16 African languages and evaluates five multilingual systems, finding high WER and distinct mixing patterns that require more than one scalar metric.
Problem
African code-switched speech is common, but ASR benchmarks and systems largely assume monolingual utterances, limiting evidence about deployment behaviour.
Method
AfriSwitch releases 61.36 hours of human-transcribed natural code-switched speech across 16 African languages and language varieties with switch-level English tags, CMI, and switch-point counts.
Results
Five zero-shot multilingual ASR systems produce WERs far above published monolingual figures, with the best system averaging 35.93% WER and no system below 24% on any language.
Takeaways & Limitations
Mixing behaviour varies along largely independent axes of alternation frequency and balance, while Africa-targeted training predicts performance better than model scale or nominal language coverage.
Takeaways & Limitations
Coverage favours relatively higher-resourced African languages, and seven languages have 5.00 hours while Afrikaans and Luganda have under 400 utterances each.
Abstract
from arXiv · showhide
Code-switching is pervasive in bilingual African conversation, yet most ASR systems assume monolingual input and are evaluated on curated monolingual benchmarks. We present AfriSwitch, a 61.36-hour human-transcribed benchmark of in-the-wild code-switched speech spanning 16 African languages and language varieties, released with switch-level English span tags, perutterance Code-Mixing Index (CMI), and switch-point counts. Corpus statistics show that mixing behaviour varies widely across African languages along two largely independent axes: how often speakers alternate, and how balanced the mixture is. No single scalar captures how code-switched a language is. Benchmarking five open and commercial multilingual ASR systems zero-shot yields word error rates far above published monolingual figures for the same languages, with the best system averaging 35.93% WER and no system falling below 24% on any language. Africa-targeted training, not model scale or nominal language coverage, best predicts performance.
1 Introduction
AfriSwitch addresses the mismatch between monolingual African ASR benchmarks and naturally code-switched speech by releasing a broad, richly annotated corpus and benchmarking current multilingual systems. The results show substantial recognition degradation and recurring failures that aggregate WER can obscure.
- Benchmark and motivation: Existing African ASR benchmarks assume linguistic consistency within utterances, leaving deployment behaviour on everyday multilingual speech poorly measured.Reported accuracy on curated monolingual test sets does not predict how systems behave in deployment.
- Benchmark and motivation: 61.36 hours of human-transcribed conversational code-switched speech span 16 African languages and language varieties in AfriSwitch.The benchmark includes switch-level English span tags, per-utterance CMI, and switch-point counts.
- Evaluation: Five open and commercial multilingual ASR systems are benchmarked zero-shot on natural code-switched speech.The evaluation uses per-language normalizers written by linguists and validated by native-speaker annotators.
- Evaluation: Every evaluated system degrades sharply relative to published monolingual results, and the degradation is not explained by model scale or nominal language coverage.This establishes naturally occurring conversational code-switching as a substantial failure mode.
- Error analysis: Embedded-language spans can be deleted or script-nativised even when aggregate WER does not reveal the underlying failure.These recurring errors motivate switch-level analysis beyond a single aggregate error rate.
2 Related Work
Prior African speech resources and code-switched benchmarks provide useful foundations, but existing code-switched ASR evidence remains geographically, linguistically, and evaluatively limited. AfriSwitch builds on this literature by targeting natural African conversational speech and switch-aware evaluation.
- African speech resources and benchmarks: African speech resources and evaluation suites have expanded rapidly across languages and benchmarks.Examples include AfriSpeech-200, Google Waxal, NCHLT, ALFFA, Common Voice, AfriVox, FLEURS, SimbaBench, and AfriSpeech-MultiBench.
- Code-switched speech corpora: African code-switched speech resources remain concentrated in a 14.3-hour South African soap-opera corpus covering four English–Bantu pairs.Its scripted broadcast dialogue differs from naturally occurring conversational speech.
- Code-switched benchmarks: LinCE and GLUECoS demonstrated the value of shared code-switched benchmarks, but both are text-only and cover no African language pair.Their scope does not provide an African speech evaluation resource.
- Modelling and evaluating code-switched ASR: Multilingual ASR research has explored synthetic mixing, language prompt fusion, attention-guided adaptation, and language-aware decoding.These approaches address systems trained or evaluated under predominantly monolingual assumptions.
- Modelling and evaluating code-switched ASR: Aggregate WER can under-weight embedded-language tokens, motivating point-of-interest scoring that requires switch-level annotation.AfriSwitch provides the annotation needed for this analysis on natural conversational speech.
3 AfriSwitch Dataset
AfriSwitch collects and annotates naturally code-switched African speech across multiple regions, then characterizes mixing with complementary per-utterance metrics. Its statistics show that alternation frequency and mixture balance vary independently across languages.
- Corpus construction: AfriSwitch covers West, East, and Southern Africa plus francophone contexts across 16 languages and language varieties.Audio was drawn from permissively licensed YouTube videos and podcasts selected by bilingual annotators for code-switching.
- Corpus construction: Bilingual native speakers produced verbatim transcriptions, followed by review and independent meta-review.Utterances were segmented with VAD and concatenated up to 40 seconds to increase the probability of capturing code-switches.
- Annotation and metrics: Token-level English labels form released spans, from which per-utterance CMI and switch-point counts are derived.The tagging is automatic and validated against human span annotations on a subset of clips in each language.
- Corpus construction: 61.36 hours across 18,861 utterances contain 78,333 code-switch events.The corpus is released with audio, verbatim transcriptions, tagged English spans, CMI, switch-point counts, and duration.
- Annotation and metrics: Switch points count tokens preceded by a different language tag, while CMI measures the balance of language contributions.CMI is 0 for monolingual utterances and increases with a more balanced distribution across languages.
- Mixing behaviour: Swahili has 10.29 switches per utterance with CMI 25.72, whereas Akan has CMI 28.20 with 2.89 switches.These contrasts show that alternation frequency and mixture balance capture distinct properties.
4 Benchmarking Multilingual ASR Systems
AfriSwitch evaluates five multilingual ASR systems zero-shot on naturally code-switched African speech under matched matrix-language conditioning. Performance gaps are large, with Africa-targeted training outperforming nominal language coverage and model scale, while aggregate WER can hide switch-specific failures.
- Evaluated Models and Protocol: Five systems are evaluated zero-shot on AfriSwitch, spanning open research models, production APIs, and Africa-focused systems.No system receives fine-tuning on AfriSwitch or code-switched data, and matrix-language conditioning is matched across systems.
- Evaluated Models and Protocol: Naming each utterance’s matrix language creates an optimistic upper bound because real deployments rarely provide per-utterance language labels.The supplied label resolves half of the language-identification problem before transcription.
- Results: 35.93% average WER makes Sahara V2.5 the strongest system, achieving the lowest WER on 10 of 12 languages.It outperforms Omnilingual LLM 7B at 51.46%, Gemini 3.6 at 55.05%, ElevenLabs at 56.48%, and Sahara V2 at 59.90%.
- Results: Omnilingual LLM 7B supports over 1,600 languages yet is outperformed on 11 of 12 languages by the smaller Africa-targeted system.It reaches 75.52% WER on Igbo and 85.33% on Luganda, showing that nominal coverage weakly predicts usable accuracy.
- Results: Luganda, Yoruba, and Igbo are difficult for every system, whereas Afrikaans, Pidgin, and Swahili are comparatively tractable.This ordering does not track AfriSwitch corpus size and is taken to reflect training-data representation.
- Discussion: Switch-boundary failures expose weaknesses that aggregate WER can obscure, including repeated deletion of embedded English in a fluent Swahili transcription.The deleted phrase changes corpus-level WER by less than the 39.22% versus 34.12% Swahili gap while removing nearly all switched content.
5 Conclusion
AfriSwitch provides a broad, human-transcribed benchmark for natural African code-switched speech and shows substantial gaps in zero-shot ASR performance. Its analyses indicate that mixing varies along distinct dimensions, while Africa-targeted training best predicts results.
- 61.36 hours of in-the-wild conversational speech across 16 African languages and varieties form the AfriSwitch benchmark.The release includes human transcriptions, per-utterance CMI, switch-point counts, and switch-level English span tags.
- Mixing behaviour varies across languages along largely independent axes of alternation frequency and mixing balance.Consequently, no single scalar characterises how code-switched a language is.
- 35.93% average WER was achieved by the best of five zero-shot multilingual ASR systems.No system fell below 24% WER on any language, and all systems performed far worse than published monolingual results for the same languages.
- Africa-targeted training was the strongest performance predictor, rather than model scale or nominal language coverage.
Limitations
AfriSwitch has uneven language coverage, reports only WER despite switch-specific annotations, and uses scoring conventions that limit direct comparison with published results. Its source media also over-represent public and broadcast registers.
- Seven languages have 5.00 hours, whereas Afrikaans has 0.68 hours and Luganda 1.21 hours from fewer than 400 utterances each.Per-language WER for these smaller subsets should therefore be read with wide error bars.
- WER can under-weight embedded-language tokens and mask systematic failures at switch boundaries.Although switch-level tags support point-of-interest metrics, the paper does not report those metrics.
- Diacritics-preserved WER is not directly comparable to published results that strip diacritics for Yoruba, Igbo, Wolof, Afrikaans, and Hausa.External comparisons for these languages should be treated as indicative rather than exact.
- YouTube and podcast sourcing over-represents broadcast, interview, and public-discourse registers relative to private conversation.The corpus also skews toward speakers with media presence.
Ethical Considerations
The benchmark uses bilingual native-speaker annotators and publicly available media, but ethical and representational constraints remain. Annotator demographics may shape transcription choices, and most African language communities are not represented.
- Transcription annotators were bilingual native speakers recruited through African crowdsourcing platforms and paid $10–$50 per hour.They were informed about the task, worked flexible hours, and were not exposed to harmful or sensitive content.
- The dataset contains processed audio segments and transcriptions without links to original sources, preventing direct tracing to content creators.The source audio came from publicly available YouTube videos and podcasts under permissive licenses.
- College-educated bilingual annotators aged 18–35 may introduce biases in handling disfluencies, informal registers, and non-standard orthography.No annotator demographic information is included in the released benchmark.
- Focusing on 16 languages and varieties implicitly prioritizes those communities over many unrepresented African language communities.The authors encourage extension to languages with fewer existing resources.