Source-linked AI summary
Benchmarking and Domain Adaptation of Automatic Speech Recognition (ASR) for Adolescent Health Communication in Ghanaian Languages
Stephen E. Moore, Akwasi Asare, Mich-Seth Owusu, Paul Azunre, Joel Budu, Lawrence A. Adu-Gyamfi
TL;DR
Ghanaian-language speakers have limited access to speech technology, and general-domain performance may not reflect accuracy in adolescent health communication. The paper benchmarks ASR systems, adapts a compact model, and validates a voice-first application; its results point to validated in-domain data as the principal constraint.
Problem
Method
The study benchmarks five models across Bible and ASRH data, fine-tunes Qwen3-ASR-0.6B on out-of-domain Bible speech with held-out in-domain evaluation, and validates KasaHealth with users.
Source: §1 · p. 2, §4.3 · p. 8Results
Validated in-domain data, rather than model capability or compute, is the principal constraint; Ewe WER fell 44.5 pp after fine-tuning.
Source: §4.4 · p. 10, §5 · p. 11Takeaways & Limitations
For these languages, prioritize in-domain data and validation, domain-adapt efficient deployable models, and use targeted engineering fixes rather than switching to larger architectures.
Source: §4.4 · pp. 10–11KasaHealth runs Ghana NLP’s production Khaya pipeline rather than the fine-tuned model, so its user results are complementary, not directly comparable, to fine-tuning evaluation.
Source: §4.3 · p. 8
Abstract
from arXiv · showhide
This paper presents an end-to-end study of automatic speech recognition (ASR) for adolescent health communication in three Ghanaian languages (Twi, Dagbani, and Ewe). The work proceeds in three connected stages; First, we benchmark five ASR systems (three language-specific Wav2Vec2 models and two multimodal LLMs, Gemma 3n and Gemma 4) on a general-domain Bible corpus and a Youth Adolescent Sexual and Reproductive Health (ASRH) Domain ASR dataset, using Character and Word Error Rate (CER, WER). Second, guided by the benchmark, we perform supervised domain adaptation: although Gemma 4 was the strongest zero-shot candidate, fine-tuning it proved computationally infeasible, so we pivoted to the compact Qwen3-ASR-0.6B, fine-tuned on a large Ghana Bible corpus (~90k samples) and evaluated strictly on held-out human-collected in-domain audio. Fine-tuning reduced WER on every language, most dramatically for Ewe (WER from 109.3% to 64.8%, a drop of 44.5 pp; CER from 65.1% to 24.9%). Third, we validate the work through KasaHealth, a live voice-first ASRH application deployed in all three languages, complemented by Senti-Check, a technical evaluation harness. KasaHealth was tested by 50 community respondents and achieved a 100% chat-approval rate, a 72% Good-or-Excellent translation rating, and a 92% would-recommend rate, while surfacing the domain gaps that most constrain real-world use. Across all three stages the evidence converges: for these languages the binding constraint is validated in-domain data, not model capability or computation.
1. INTRODUCTION
Native-language health information supports meaningful participation, yet Ghanaian-language speakers remain underserved by digital services and speech technology. This study benchmarks ASR, adapts a model for adolescent health communication, and validates a voice-first application with users.
1. INTRODUCTION
- Ghanaian-language speakers remain underserved by digital services, while ASRH transcription accuracy matters for trust in voice interfaces.Native-language health information is described as foundational to meaningful participation in health services, particularly for adolescents in multilingual communities.Source: §1 · pp. 1–2
- The study benchmarks five models across general and ASRH domains, adapts an ASR model, and validates KasaHealth with real users.The work also documents a pivot from an infeasible Gemma fine-tune to Qwen3-ASR-0.6B and introduces Senti-Check.Source: §1 · p. 2
- The authors release datasets, model weights, and tooling as an open-source pipeline for reuse by other West and Central African language communities.Source: §1 · p. 2
2. BACKGROUND AND RELATED WORK
Prior work describes persistent low-resource ASR challenges and recent self-supervised and multilingual approaches, but general benchmarks can miss health-domain errors. Ghanaian ASRH adds terminology and language-use differences that motivate domain adaptation.
ASR for low-resource languages
- Low-resource ASR faces scarce annotated speech, tonal phonology, rich morphology, and nonstandardized orthographies [7], while traditional systems required hundreds to thousands of transcribed hours.Source: §2 · p. 2
- Self-supervised wav2vec 2.0 learns acoustic representations from unlabeled audio and can be fine-tuned with as little as ten minutes of labels [10]; multilingual XLSR supports cross-lingual transfer [11].Health-domain errors on medical entities can remain higher than general WER, despite low general WER, and domain fine-tuning improved medical WER by 25–34% relative [12].Source: §2 · p. 2
Ghanaian-language NLP and zero-shot LLMs; The ASRH domain gap
- Ghana NLP provides local-language Wav2Vec2 models and datasets [2,4,5,14–16], while Gemma 4 is a newly evaluated zero-shot baseline whose competitiveness does not imply feasible fine-tuning [26].Gemma 3n performed poorly across the three languages [17]. ASRH domain mismatch includes unfamiliar vocabulary, speaking styles, prosody, and socially constructed euphemisms.Source: §2 · p. 2
3. METHODOLOGY
The methodology benchmarks five ASR systems on general-domain Bible and Youth ASRH data, then uses human-collected audio strictly as a held-out evaluation set for a tractable Qwen3-ASR-0.6B fine-tune.
3.1 Datasets
- The benchmark uses public Bible and Youth ASRH datasets, plus a synthetic multilingual ASRH text dataset, with 50-sample subsets per dataset per language.Bible data provides broader general-domain vocabulary, while Youth ASRH audio is paired with ASRH-relevant transcriptions.Source: §3.1 · pp. 2–3
3.2 Models and Benchmarking Methodology
- Five models were benchmarked: two language-fine-tuned Wav2Vec2-BERT models, a third with BERT language-model decoding, and two instruction-tuned Gemma models evaluated fully zero-shot; CER and WER were mean per-sample scores.Source: §3.2 · p. 3
3.3 Audio Collection and the Pivot from Gemma to Qwen3-ASR
- After Gemma 4 exceeded available GPU resources, the study fine-tuned Qwen3-ASR-0.6B on out-of-domain Bible data and reserved Phase-2 human audio strictly for evaluation.Phase 2 collected up to two hours per language; the separation avoids training overlap with the adolescent-domain evaluation audio.Source: §3.3 · pp. 3–4
4. RESULTS AND DISCUSSION
The results section is organized around three evaluation stages—benchmarking, domain adaptation, and user acceptance testing—before synthesizing implications and listing released resources.
4.1
- The section first reports the cross-model benchmark.Source: §4 · p. 4
4.2–4.3
- It next reports domain-adaptation fine-tuning and user acceptance testing as separate evaluation stages.Source: §4 · p. 4
4.4–4.5
- The section then synthesizes implications and lists the open-source resources released with the work.Source: §4 · p. 4
4.1 Benchmarking
Across Bible and Youth ASRH benchmarks, language-specific Wav2Vec2 models led on character error, while Gemma 4 was a competitive zero-shot baseline at the word level. Results also show a persistent domain gap, including for Gemma 4.
4.1 Benchmarking
- Gemma 4 beat Gemma 3n across all six conditions, with CER 44.3–55.4% and WER 92.5–98.8%, while Wav2Vec2 retained a clear CER advantage.The benchmark covers 50 samples per dataset.Source: Table 2 · p. 4, §4.1.1 · p. 6
4.1.1 Model-by-Model Analysis
- Native-language Wav2Vec2 models achieved CER of 17.0–25.2%, while their non-native performance degraded to CER 45–65% and WER consistently above 95%.Dagbani Wav2Vec2 had lower CER on Youth ASRH than Bible, while Ewe Wav2Vec2 had a roughly 1 pp CER domain gap.Source: §4.1.1 · p. 6
- The Bible-to-Youth-ASRH domain gap persisted even for Gemma 4, showing that general multimodal pretraining had not closed the shift between religious speech and health communication.The Bible corpus is the largest transcribed source for these languages, shaping Wav2Vec2 models’ vocabulary and acoustic profile.Source: §4.1.1 · p. 7
4.2 Domain Adaptation: Fine-Tuning
Fine-tuning Qwen3-ASR-0.6B on Bible audio improved performance on held-out human-collected Youth ASRH audio across all three languages. Ewe had the largest reported gains, though the evaluation reflects generalization from out-of-domain training.
4.2 Domain Adaptation: Fine-Tuning
- The evaluation used held-out human-collected Youth ASRH audio disjoint in domain from the Bible training corpus, measuring generalization rather than memorization.Figure 4 compares base and fine-tuned WER and CER; Table 3 reports their values.Source: §4.2 · p. 7, Figure 4 · p. 7, Table 3 · p. 7
- Fine-tuning reduced error in every language; Ewe WER fell 44.5 pp (109.3% to 64.8%) and CER fell 40.2 pp (65.1% to 24.9%).Twi and Dagbani each showed approximately 11 pp WER improvements.Source: §4.2 · p. 7, Table 3 · p. 7, Figure 4 · p. 7
- Residual error reflects low-resource ASR difficulty and the Bible-to-ASRH domain gap; with no in-domain training audio, these results are a conservative lower bound.The passage says a modest volume of in-domain training audio would be expected to improve results substantially.Source: §4.2 · p. 7
4.3 User Acceptance Testing
The user-facing KasaHealth evaluation and Senti-Check technical harness provide complementary validation; KasaHealth uses production Khaya models, so it is not a direct evaluation of the fine-tuned model.
4.3.1 KasaHealth: Voice-First ASRH Application
- KasaHealth delivers ASRH information in Twi, Dagbani, and Ewe through production Khaya ASR, LLM, and TTS models, with a pre-entry disclaimer that it is not a medical service.The interface supports voice and text input, language and voice selection, quick health topics, and transcribed chat.Source: §4.3.1 · p. 8, Figure 5 · p. 8
- 92% would recommend the application, and 75% rated answers Helpful or Very Helpful; 72% rated translation Good/Excellent (36/50), with zero Poor ratings.Translation was the strongest component; TTS received 58% Good/Excellent (29/50), with Twi vowel-elongation artefacts noted.Source: §4.3.1 · pp. 8–9, Figure 6 · p. 9
- All 26 health-question interactions received chat approval (100%); flagged translation reviews most often cited bad translation (18 of 29).Ewe drew the most negative feedback (1 up/16 down), while vocabulary gaps and a language-switching bug were also reported.Source: §4.3.1 · p. 9
4.3.2 Senti-Check: Technical Validation
- Fine-tuned Qwen3-ASR doubled blind emotion-label accuracy to 16/120 (13.3%) versus base 8/120 (6.7%), though absolute accuracy remained low.The 24-class proxy used six native-speaker testers and 120 judgements per model; improvement was strongest in Twi, while Ewe was stable pending a word-boundary fix.Source: §4.3.2 · p. 9, Figure 7 · p. 10
4.4 Discussion
The discussion identifies validated in-domain data as the principal constraint, ahead of models and compute, and argues for targeted collection across ASR, translation, and TTS.
4.4 Discussion
- The principal opportunity is validated in-domain data, followed by models and then compute; the limiting factor was data availability, not model capability or compute.The paper argues that smaller, carefully validated in-domain sets will usually outperform larger, noisier out-of-domain corpora.Source: §4.4 · p. 10
- The recommended data spans conversational health audio with verified transcripts, parallel health-terminology text, and targeted natural-speech recordings for TTS.An order of tens of training hours per language is presented as a direction, not a fixed prescription; speaker diversity and validation are emphasized.Source: §4.4 · p. 10
- KasaHealth demonstrates a usable production-grade local-language experience; the discussion favors efficient deployable models and targeted fixes over larger architectures.Compute was non-binding for the fine-tune on modest T4 hardware, while transcription and especially native-speaker validation were identified as time constraints.Source: §4.4 · p. 11
4.5 Open-Source Release
The work publicly releases its datasets, model weights, and tooling; Table 5 provides the resource inventory.
4.5 Open-Source Release
- All datasets, model weights, and tooling produced in the work are publicly released, with Table 5 listing the resources and their locations.Source: §4.5 · p. 11, Table 5 · p. 11
5. CONCLUSION
Across Twi, Dagbani, and Ewe, the conclusion reports language-specific Wav2Vec2 character-level strength, Qwen fine-tuning gains, and positive community response to KasaHealth. It identifies validated in-domain data as the binding constraint.
5. CONCLUSION
- 44.5 pp was the Ewe WER reduction after Qwen3-ASR-0.6B fine-tuning; WER fell in every language, with evaluation on strictly held-out in-domain audio.Fine-tuning followed an infeasible attempt to fine-tune Gemma 4 and used a large out-of-domain corpus.Source: §5 · p. 11
- Language-specific Wav2Vec2 models achieve a character-level advantage (CER 17–25%), while Gemma 4 is competitive zero-shot at word level; all models retain a domain gap.Source: §5 · p. 11
- KasaHealth received 100% chat approval, a 72% Good-or-Excellent translation rating, and a 92% would-recommend rate from community users.Source: §5 · p. 11
- Senti-Check shows that fine-tuning doubles downstream semantic accuracy.Source: §5 · p. 11
- Validated in-domain data is identified as the binding constraint; the released open-source pipeline provides a foundation for investment and adoption by other African language communities.Source: §5 · p. 11