Quantifying Bias in Automatic Speech Recognition

1. Introduction

• ASR is used in settings such as emergency response centers, voice assistants, and search engines, making its ability to handle speech variability important.The passage identifies speaker differences, demographics, speaking styles, and disability as sources of variation.Source: §1 · p. 1
• Prior studies report recognition differences associated with gender, age, impairment, race, and accents, while findings on gender differences are mixed.The cited studies include cases where female speech is better recognized, the reverse pattern, and no significant difference.Source: §1 · p. 1
• Potential sources of bias include training-data composition, transcript mismatch, articulation and speaking-rate differences, transcription practices, and system design.The authors also note that bias may enter during problem framing, data preparation, and collection.Source: §1 · p. 1
• The paper systematically investigates how a standard, state-of-the-art Dutch ASR system recognizes speech across speaker groups to quantify bias.The stated goal is to help work toward proactive bias mitigation.Source: §1 · p. 1

2.1.1. Dutch Spoken Corpus (CGN)

• The Dutch ASR system was trained on 483 hours of CGN recordings from speakers aged 18–65 across the Netherlands and Flanders, using only Netherlands data.The training material included 1,185 female and 1,678 male speakers.Source: §2.1.1 · p. 2

2.1.2. Jasmin-CGN corpus

• Jasmin-CGN evaluation covered native children, teenagers, and older adults across Dutch regions and Flanders, plus non-native children and adults; both read and HMI speech were used.The study examines gender, age, regional, and non-native accent dimensions.Source: §2.1.2 · p. 2

2.2. State-of-the-art ASR system for Dutch

• The ASR system uses a hybrid DNN-HMM with a TDNN-BLSTM acoustic model and an RNN language model, trained using Kaldi.Training included speed perturbation, reverberation, and noise augmentation of the CGN material.Source: §2.2 · p. 2

2.3. Experiments and Evaluation

• Potential bias was assessed through WER differences across speaker groups separately for read and HMI speech, with phoneme error rate analysis to examine error-prone pronunciations.For PER, word transcripts were converted to phoneme sequences using the CGN Dutch lexicon, aligned with Levenshtein distance, and compared.Source: §2.3 · pp. 2–3, Table 3 · p. 3

3.1. Baseline results

• 5.5% WER on the CGN BN set and 20.8% WER on the CTS set provide reference results because CGN lacked standard read and HMI test sets.On CTS, female speech had 17.9% WER and male speech 23.2%; on BN, the respective values were 5.5% and 5.4%.Source: §3.1 · p. 3

3.2. Word recognition results

• Female speech was generally better recognised than male speech across native and non-native groups and both styles, with the largest gender gap in older Dutch adults and the smallest in children.The largest-versus-smallest pattern held for both read and HMI speech.Source: §3.2 · p. 3
• Native Dutch children were worst recognised and teenagers best among native age groups, while non-native adult WER did not decrease with higher CEF proficiency.The per-speaker WER histograms of the two non-native groups also differed little across read and HMI speech.Source: §3.2 · p. 3
• Native Dutch speech was recognised much better than non-native speech; the worst native group, Dutch children, had read-speech WER around 20% absolutely better than the best non-native group, non-native children.The comparison is between the worst native and best non-native age groups, not all speakers in those groups.Source: §3.2 · p. 3
• HMI speech had worse WER than read speech in every group, with an overall absolute difference of around 13.7% for native speakers and 5.5% for non-native speakers.The HMI disadvantage was consistent across groups, though its overall size differed by native status.Source: §3.2 · p. 3
• Flemish speech had the worst WER except among older adults, for whom region S was worst; no Dutch region was consistently worst on read speech.Among Dutch speakers, regional WER differences were larger for older adults than for children or teenagers; region S was generally worst on HMI speech.Source: §3.2 · pp. 3–4

3.3. Error analyses

• Phoneme errors differed between non-native and older native speakers; non-native adults often struggled with Dutch vowels /œy/, /Y/, /y/, and /ø:/ that are difficult for second-language learners.Older native speakers’ listed errors instead included /h/, /x/, /E/, and /@/. The analysis reports top phonemes only when they occurred at least 50 times.Source: §3.3 · p. 4
• Children’s /S/ pronunciations confused the ASR, southern /x/ misrecognition was offered to explain older adults’ high WER in region S, and male speech had higher PER without a clear phoneme pattern.The authors cite sibilant substitution errors for children and regionally different production of /x/ in the southern Netherlands.Source: §3.3 · p. 4

• The Dutch ASR system showed bias across gender, age, regional accents, and non-native accents, and the authors conclude that ASR can perpetuate societal bias.Source: §4 · p. 4
• Teenagers’ speech was best recognised, followed by seniors over 65, while children’s speech was worst; female speech was better recognised than male speech.The authors attribute children’s poorer recognition to an acoustic-model mismatch and older adults’ poorer recognition, especially for those over 75, to less well-articulated speech.Source: §4 · p. 4
• Native Dutch speech was much better recognised than non-native speech irrespective of age, with no correlation between non-native speakers’ Dutch proficiency and ASR performance.The authors suggest accent mismatch as a possible reason for the native/non-native difference.Source: §4 · p. 4
• For native Dutch speakers, Flanders speech had the worst performance, while within the Netherlands older speakers’ regional accents were recognised worse than those of children and teenagers.The authors explain the Flanders result by the accent difference from the Netherlands speech used to train the system.Source: §4 · p. 4
• HMI speech was consistently worse recognised than read speech, which the authors link to less preparation and greater speaker-dependent articulation and word-use differences.Source: §4 · pp. 4–5
• The authors identify training-data composition as important to performance differences and recommend balanced speaker representation and diverse development teams, while noting that dataset bias cannot be fully removed.They describe balanced speaker representation as a direct mitigation strategy and diverse team composition as an indirect one; they also state that bias in datasets is impossible to remove.Source: §4 · p. 5