Careless Whisper: Speech-to-Text Hallucination Harms

1 INTRODUCTION

• Roughly 1% of Whisper transcriptions contained entire made-up sentences, including when no one was speaking in the audio.The study compares speech snippets with Whisper output to identify hallucinations.Source: §1 · p. 2
• The study treats hallucinations as a concern in their own right, beyond their prior use in speech-to-text error prediction or identification.The authors connect transcription bias to decisions based on transcripts, including surveillance and medical care.Source: §1 · pp. 1–2
• Nearly 40% of hallucinations were harmful or concerning rather than innocuous and random.The paper categorizes harms including violence, inaccurate associations, and potentially phishing website links.Source: §1 · p. 2
• The external evaluation ran 13,140 audio segments through Whisper’s default API, with English specified and no other parameter settings.The audio was sourced from TalkBank’s AphasiaBank and licensed for research use.Source: §1 · p. 2
• The paper examines whether hallucinations disproportionately affect people with aphasia and discusses ethical and legal ramifications of the disparities.It hypothesizes two underlying mechanisms and focuses on harms to people with speech impairments.Source: §1 · p. 2

2 METHODOLOGY

• The methodology focuses on potential disparities in transcription hallucinations for people with aphasia, who have reduced ability to express themselves through speech and voice.The authors note that speech-to-text tools tend to underperform for aphasic speakers without techniques tailored to them.Source: §2 · p. 2

2.1 Data & Participants

• The study used English AphasiaBank interviews from 437 participants with and without aphasia, alongside human transcripts and anonymized demographic information.The sample came from multiple U.S. institutions; interviews elicited spontaneous responses to standard topics.Source: §2.1 · pp. 2–3

2.2 Audio Segmentation

• Researchers segmented nearly 40 hours of audio into 13,140 roughly sentence-length utterances for Whisper, including 7,805 control and 5,335 aphasia segments.The audio averaged about 10 seconds per segment, with aphasia segments generally longer and containing fewer words.Source: §2.2 · p. 3

2.3 API Experiments

• The researchers ran AphasiaBank audio segments through Whisper and Google’s speech APIs, then tested a subset through Amazon Web Services, Microsoft Azure, AssemblyAI, and RevAI for comparison.The four additional APIs were tested on January 27, 2024. Data collection or storage was disabled for all APIs.Source: §2.3 · p. 3

2.4 Detecting Hallucinations

• Repeated Whisper runs flagged candidate hallucinations when transcripts changed by multiple tokens and contained more tokens than the ground truth; manual review confirmed 187 hallucinating segments.The approach relies on hallucinations varying across runs, while correctly transcribed portions remain more consistent; confirmed hallucinations often appeared as lengthy appended text.Source: §2.4 · p. 4

2.5 Categorizing Hallucinations

• The authors used thematic coding to group hallucinations into nine harm types across three categories: perpetuation of violence, inaccurate associations, and false authority.Examples include violence or stereotypes, fabricated names or health statuses, and language that falsely suggests a video-based authority or links to websites.Source: §2.5 · p. 4
• Many hallucinations fall outside the three categories yet generate more text than the audio and can confuse users as second-order threats.The passage contrasts these threats with the more direct harms of previously categorized hallucinations.Source: §2.5 · p. 7
• Hallucinations commonly repeat the same few words, a pattern that can confuse users but sometimes preserves the original meaning.The example contrasts the spoken Cinderella passage with Whisper transcriptions from April and May 2023 that repeat and extend it.Source: §2.5 · p. 7
• The May 2023 Cinderella transcription adds further dress-related lines after repeating the original passage.The audio ends with “head adornment,” while the transcription continues with lines about wearing a pretty dress.Source: §2.5 · p. 7

3 RESULTS

• 1.4% of transcriptions in the dataset yielded hallucinations, with 312 hallucinated transcriptions identified across April and May runs.The transcriptions came from audio segments that produced hallucinations in at least one run.Source: §3 · p. 7
• 38% of hallucinated transcriptions contained at least one harm: 19% violence, 13% inaccurate associations, and 8% false authority.These harms can include violent content that may trigger victims of abuse, representational harm from incorrect associations, and financial loss from fraudulent authority or phishing.Source: §3 · p. 7
• 12 of 187 previously hallucinating audio segments still produced hallucinations in December 2023 (9/92 aphasia; 3/95 control), despite many examples being resolved.The authors say the improvement was likely due to late-November 2023 Whisper updates.Source: §3 · p. 7
• 1 of 250 randomly sampled segments produced a hallucination in December 2023 that had not appeared in either the April or May run.The new example involved an aphasia speaker and a made-up name.Source: §3 · p. 7

3.1 Limitations

• The aphasia-control comparison used a relatively small speaker sample and standard interview questions; broader topics could widen the observed scope of hallucinations.Source: §3.1 · p. 8
• The hallucination count is likely an undercount because identical hallucinations in both April and May runs were excluded.The tabulation required different outcomes across the two runs.Source: §3.1 · p. 8
• Appearances of other languages were not tabulated because determining whether they qualified as hallucinations required additional resources.Source: §3.1 · p. 8
• English-target transcriptions can handle code-switching faithfully in some cases, but non-target languages may also be hallucinated.The paper contrasts a faithful Ukrainian example with a Chinese example showing repeated phrases and a false-authority ending.Source: §3.1 · p. 8

4.1 Generative AI in speech-to-text modeling

• Even with no prompts and default temperature 0, Whisper produced highly non-deterministic hallucinations, which the authors suggest may implicate its language modeling.The proposed connection is a hypothesis, not a demonstrated cause.Source: §4.1 · p. 9
• Across the same 187 identified audio segments, the authors found exactly 0 comparable hallucination concerns from Google products and exactly 0 from Amazon, Microsoft, AssemblyAI, and RevAI speech-to-text services.Google Text was tested in April 2023 and the latest Google Chirp model in December 2023; the other services were tested in January 2024.Source: §4.1 · p. 9

4.2 Speech patterns likely to yield hallucinations

• 1.7% of aphasia audio segments yielded hallucinations, versus 1.2% of control segments (p=0.019).In a propensity-matched subset of 6,046 segments, the rates were 1.8% for aphasia and 1.1% for controls.Source: §4.2 · p. 9
• Non-vocal shares were higher for aphasia speakers than controls (41% versus 15%) and for hallucinated versus non-hallucinated files (29% versus 26%).The four group means were 42.4%, 40.6%, 16.2%, and 15.4% for aphasia with and without hallucinations, then controls with and without hallucinations.Source: §4.2 · p. 10, Figure 3 · p. 10
• Longer non-vocal durations and more words were the strongest positive correlates of hallucinated transcriptions in the logistic regression.The association was robust to other regression specifications, including Mahalanobis-matched subsets.Source: §4.2 · p. 11, Table 2 · p. 11

• Hallucinations for people with aphasia include violent speech, inaccurate identification or association, and false video-based authority.The authors note that such outputs can affect hiring and interactions with government services.Source: §5 · p. 11
• The study’s findings concern a relatively small set of aphasia speakers and controls in an English-target setting.The authors also suggest hallucination bias could arise for other groups with speech impairments or disfluencies, including dysphonia, elderly, and non-native speakers.Source: §5 · p. 12
• Future work should examine intersectional hallucination bias and harms, and include speech-impaired people in generative-AI design and testing.The authors propose specific harm classifications that account for technology, application, and subpopulation to support targeted mitigation strategies.Source: §5 · p. 12
• Word Error Rate alone can conceal concrete harms because hallucinated quotes may affect impressions of speakers more viscerally than basic mistranscriptions.The authors give “orchestra violence” versus “orchestra violins” as an example of errors that could yield comparable WERs.Source: §5 · p. 12

• Whisper API users should be told that transcriptions can contain hallucinations, making outputs inaccurate or harmful.Source: §6 · p. 13
• OpenAI should include people with lived experience in underserved communities, including people with aphasia, in system design.Source: §6 · p. 13
• OpenAI should calibrate Whisper’s default settings to reduce transcription randomness.Source: §6 · p. 13
• The speech-to-text community should identify hallucination causes and examine disproportionate effects on subpopulations.The authors say this could inform model improvements or collection of speech acoustics that lead to hallucinations.Source: §6 · p. 13

• The researchers used TalkBank AphasiaBank data collected by contributors and relied on their ethics practices, including IRB adherence and confidentiality norms.Source: §7 · p. 13
• The team opted out of API data storage so TalkBank research data could not be repurposed for business use.Source: §7 · p. 13
• No researcher was an aphasic speaker, so the team could not draw on lived experience with Whisper hallucinations in aphasia.Four of the five researchers were non-native English speakers with experience of speech-to-text inaccuracies.Source: §7 · p. 13

A.1 Demographic Matching

• Mahalanobis matching on demographics used gender, age, race, education, and English as a first language, with a caliper of 0.20.The additional segment-attribute matching included average word speed and the share of segment duration that was non-vocal.Source: §A.1 · p. 15
• 1.82% of segments from aphasia speakers yielded hallucinations versus 1.16% for control speakers in the 6,046-segment demographic-matched subset.The difference was significant (p-value = 0.044).Source: §A.1 · p. 15
• 0.02% of segments from aphasia speakers yielded hallucinations versus 0.006% for control speakers in the 2,352-segment subset matched on demographics and segment attributes.The difference was significant (p-value = 0.0038).Source: §A.1 · p. 15

A.2 Voice Activity Detection

• Silero via PyTorch reproduced the non-vocal-duration patterns previously found with PyAnnote.The analysis compared aphasia speakers with control speakers and hallucination-yielding audio with audio not yielding hallucinations.Source: Figure 5 · p. 16
• Aphasia speakers had longer non-vocal durations than control speakers using Silero via PyTorch.Source: Figure 5 · p. 16
• Audio yielding hallucinations had longer non-vocal durations than audio not yielding hallucinations using Silero via PyTorch.Source: Figure 5 · p. 16

A.3 Regression Analysis

• Longer non-vocal periods, being aphasic, and more words or greater speed in the ground truth tended to correspond to higher likelihood of hallucinated Whisper transcriptions across regression specifications.The models conditioned on speaker demographics and audio-segment attributes, including matched subsets.Source: Table 3 · p. 16
• Matched datasets excluded English-as-a-first-language and vision-loss covariates because those subsets contained only participants whose first language was English and who had no vision loss.Source: Table 3 · p. 16
• The reported associations had varying levels of statistical significance across specifications.Source: Table 3 · p. 16