Source-linked AI summary
Fairness Evaluation of Edge-AI Implementation for Cleft Lip and Palate Speech ASR
Susmita Bhattacharjee, Himashri Deka, H. S. Shekhawat, S. R. M. Prasanna
TL;DR
CLP speech variability makes ASR difficult, and severity-related fairness in pathological speech remains underexplored. This study fine-tunes Whisper-small across Normal and CLP severity mixes and evaluates it on Jetson. Fine-tuning lowers pooled error rates and runs faster than real time on-device.
Problem
Severity-dependent acoustic and articulatory variability challenges ASR, while severity-related performance disparities remain little explored in pathological speech.
Source: §1 · p. 2Method
The study fine-tunes Whisper-small using different combinations of Normal and CLP severity speech, then evaluates recognition and computational performance on an NVIDIA Jetson.
Source: §1 · pp. 3–4Results
Pooled WER fell from 62.46% for pretrained Whisper to 22.72% with NOMIMOSE; NOMIMO achieved the lowest pooled PER at 18.44%, and fine-tuned models had RTFs of 0.167–0.171.
Source: §4 · p. 7, §3.3 · p. 7Takeaways & Limitations
Severity-aware edge ASR supports on-device CLP speech interaction without continuous cloud access, while recognition accuracy and Normal–CLP disparity remain distinct objectives.
Source: §1 · p. 2, §3.1 · p. 6, §4 · p. 7
Abstract
from arXiv · showhide
Automatic speech recognition (ASR) remains challenging for individuals with cleft lip and palate (CLP) because of limited pathological speech data and large variations in speech characteristics across speakers and severity levels. These recognition difficulties can reduce the accessibility of voice-based human-computer interaction, particularly when cloud-based ASR services are unavailable or unreliable. This work investigates a severity-aware and edge-deployable ASR framework for improving recognition of CLP speech using Whisper-small. The model was fine-tuned using different combinations of normal and CLP speech representing mild, moderate, and severe conditions, together with a CLP-only training configuration, to examine how the inclusion of different severity levels influences recognition performance and fairness across speakers. The pretrained model produced pooled word error rate (WER) and phoneme error rate (PER) values of 62.46% and 52.72%, respectively. Severity-aware fine-tuning substantially improved performance, reducing the best pooled WER to 22.72% and the best pooled PER to 18.44%. Training with a broader representation of CLP severity levels also provided the best overall balance between recognition accuracy and performance consistency across severity groups. Deployment on an NVIDIA Jetson platform demonstrated real-time inference for all fine-tuned models, with real-time factors of 0.167-0.171 and peak GPU memory usage of approximately 566 MB. The results demonstrate that incorporating severity diversity during ASR adaptation can substantially improve recognition of CLP speech while reducing performance disparities across severity groups. The proposed approach further enables low-latency, Internet-independent speech interaction on edge devices, supporting more accessible and inclusive voice-based human-computer interaction for individuals with CLP.
1 Introduction
CLP speech varies across speakers and severity, challenging conventional and pretrained ASR; severity-related disparities in pathological speech remain underexplored. This work investigates severity-aware Whisper-small adaptation and Jetson deployment, assessing recognition and device performance.
1 Introduction
- Severity-dependent acoustic and articulatory variability challenges conventional and pretrained ASR, while severity-related disparities remain underexplored in pathological speech.The introduction notes that impairment severity can itself be an important source of disparity.Source: §1 · p. 2
- On-device ASR reduces dependence on cloud infrastructure, supports low-latency interaction, and avoids requiring continuous cloud access for assistive communication.Local processing also reduces external speech-data transfer.Source: §1 · p. 2
- The Jetson pipeline converts preprocessed speech to log-Mel spectrograms for transcription and reports recognition, fairness, and device-performance metrics.Models are fine-tuned off-device, averaged across five folds, and deployed for FP16 GPU inference.Source: Figure 1 · p. 3
- The study fine-tunes Whisper-small with different combinations of Normal and CLP severity groups, then evaluates severity-wise WER, phoneme error, and deployment performance on Jetson.The evaluation includes latency, real-time factor, throughput, and memory usage.Source: §1 · pp. 3–4
2 Dataset used
The study uses speaker-disjoint NMCPC partitions and fixed-size training configurations spanning Normal and CLP severity groups. Its Fairness Score jointly considers recognition error and disparity between Normal and CLP speech.
2 Dataset used
- NMCPC includes 41 CLP and 24 Normal speakers aged 9–13, with CLP speech labelled Mild, Moderate, or Severe.Speaker-disjoint training, development, and evaluation partitions were used; the evaluation set had 264 utterances, 66 per severity group.Source: §2 · p. 4
- Five training configurations each use 280 utterances: Normal-only, Normal-plus-Mild, Normal-plus-Mild-plus-Moderate, all four groups, and CLP-only.The group counts are 280; 140+140; 93+93+94; 70 per group; and 93+93+94, respectively.Source: §2 · p. 4
2.1 Fairness as a Metric
- The Fairness Score combines average error rate with Normal–CLP error disparity, with α and β weighting their relative importance.Values closer to zero indicate better overall performance and greater fairness.Source: §2.1 · p. 4
- Because severity groups have unequal file counts, the study uses pooled WER rather than averaged error for its fairness evaluation.The score ranges from −∞ to 0; lower average error and smaller disparity indicate better accuracy and more balanced performance.Source: §2.1 · p. 4
3 Experimental Setup
Severity-aware fine-tuning improved recognition while maintaining efficient Jetson inference; the results also distinguish aggregate accuracy from Normal–CLP performance disparity.
3 Experimental Setup
- Jetson comparisons used zero-shot pretrained Whisper-small and five-fold averaged proposed models, with identical evaluation utterances and decoding settings.Inference used FP16, greedy decoding, GPU acceleration, and utterance-wise processing.Source: §3 · pp. 4–5
3.1 Deployment Results for Pretrained and finetuned Whisper-small
- The pretrained baseline's RTF was 0.212; excluding cold-start and repetitive-decoder outliers yielded a steady-state RTF of 0.181.The baseline processed all 264 evaluation utterances, with mean and P95 latencies of 0.363 s and 0.385 s.Source: §3.1 · p. 5
3.3 Edge Deployment performance
- Pooled WER fell from 62.46% for pretrained Whisper-small to 22.72% with NOMIMOSE, while RTF fell from 0.212 to 0.167.The severity-wise recognition and Jetson fairness/edge-inference comparisons are summarized in Tables 2 and 3.Source: Table 2 · p. 6, Table 3 · p. 6, §3.3 · p. 7
3.2 WER Analysis and Phoneme Analysis
- 18.44% was the lowest pooled PER (NOMIMO); NOMIMOSE led Mild (3.01%) and Severe (47.32%), while CLP led Moderate PER (14.90%).Table 2 presents severity-wise recognition performance for pretrained and fine-tuned models.Source: Table 2 · p. 6, §3.2 · p. 6
3.1 Deployment Results for Pretrained and finetuned Whisper-small
- NOMIMOSE achieved 0.274 s median latency and 566.32 MB peak GPU memory; assistive HCI also requires accuracy and severity balance.Aggregate recognition error and Normal–CLP disparity are related but distinct objectives.Source: §3.1 · pp. 5–6, Table 3 · p. 6
4 Conclusion
The conclusion reports improved CLP speech recognition from severity-aware Whisper-small adaptation, with faster-than-real-time Jetson inference and persistent difficulty on severe speech.
4 Conclusion
- Fine-tuning reduced pooled WER from 62.46% to 22.72% with NOMIMOSE; NOMIMO achieved the lowest pooled PER at 18.44%.The conclusion also reports a fairness score for NOMIMO almost similar to the CLP-only model.Source: §4 · p. 7
- Fine-tuned models achieved RTFs of 0.167–0.171 with approximately 566 MB peak GPU memory on NVIDIA Jetson.The conclusion characterizes this as faster-than-real-time on-device inference without additional model-size overhead.Source: §4 · p. 7
- Severe speech remained the most challenging condition; the results support severity-aware edge ASR for CLP where continuous cloud connectivity cannot be guaranteed.Source: §4 · p. 7