PaperlayerView exact revision on arXiv

Source-linked AI summary

How to Reduce Whisper Hallucination

Husein Zolkepli

arXiv:2609.32560v2cs.CL

TL;DR

Whisper can hallucinate text over silence, and filtering those examples from pseudo-label corpora can leave students without evidence of the failure. The paper benchmarks both suppression and speech recovery, then adds synthesized spoken versions of hallucinated phrases; matched fine-tunes generally improve both measures, with the best checkpoint reducing hallucination while raising phrase recovery.

  • Problem

    Whisper can emit text over silence, while pseudo-label filtering can remove examples of that failure and leave students without training examples whose correct transcript is empty.

    Source: §1 · p. 2, §2 · p. 4

  • Method

    The paper fine-tunes Whisper using blank-audio targets and synthesized spoken positives of hallucinated phrases, evaluating both suppression and speech recovery.

    Source: §3 · p. 5, §8.1 · p. 17

  • Results

    Adding spoken positives lowered word emission on real voice-free audio in 31 of 33 matched pairs and raised phrase recovery in 32; the best checkpoint reduced silence hallucination from 61.9% to 2.4% while raising phrase recovery from 69.8% to 82.7%.

    Source: §9 · p. 26

  • Takeaways & Limitations

    The benchmark, lexicon, synthetic corpus, and wild audio are released for subsequent attempts to be scored on both halves.

    Source: §9 · p. 27

    The silence arm has 42 clips, so differences of a few percentage points are noise.

    Source: §10 · p. 27

Abstract

from arXiv · show

Whisper is still what runs in production: one permissively licensed checkpoint, 99 languages, no per-language tuning. But it writes sentences nobody said. On 42 clips of pure room tone, whisper-large-v3 emits words on 61.9% of them and emits something on 100%. The usual response is to distil a student from Whisper pseudo-labels, filtering the hallucinations out of the corpus first, which deletes the evidence while leaving the behaviour in place: the student is fitted to the subset where the teacher was right, and inherits a failure mode absent from its own training data. The second response, adding non-speech audio so the model learns to stay quiet, already ships and works, but teaches suppression without discrimination. The checkpoint best on every non-speech arm here is also the one that recovers the fewest genuinely spoken phrases and deletes 58% of repeated speech. The teacher has to be fixed, with both halves of the signal. Our benchmark scores both at once: 11,852 clips over eight arms, 6,267 synthetic positives, 8,296 clips of real audio that made a production model fail, and FLEURS in 58 languages. We collect 40,891 hallucination phrases in 100 languages, choose a text-to-speech system by measurement, and synthesise those phrases as positives, so the model meets the same text both as something to suppress and as something to transcribe. Across 33 matched pairs of fine-tunes differing only by those positives, adding them lowers word emission on real voice-free audio in 31 pairs and raises phrase recovery in 32; among the 30 pairs that hold English accuracy it is 30 out of 30. The best checkpoint takes hallucination on silence from 61.9% to 2.4% and words over real voice-free audio from 99.9% to 47.8%, while raising phrase recovery from 69.8% to 82.7%. The benchmark, lexicon and synthetic corpus are released at https://huggingface.co/datasets/Scicom-intl/Whisper-Hallucination

1 Introduction

Whisper remains widely used for multilingual transcription, but it can emit fluent text over silence, a narrow failure with consequences for users and for models distilled from it. The paper motivates measuring hallucination alongside speech recognition rather than treating silence suppression as the only goal.

1 Introduction

  • Whisper’s broad language coverage, permissive licensing, code-switching behavior, and adoption help explain why it remains in production despite stronger reported accuracy elsewhere.Whisper covers 99 languages, and the paper reports roughly 4.6 million monthly Hugging Face downloads for whisper-large-v3.Source: §1 · pp. 1–2
  • A defect in Whisper can propagate to systems trained on its pseudo-labels, including Distil-Whisper [Gandhi et al., 2023], Kotoba-Whisper [Kotoba Technologies and Ushio, 2024], and Granary [Koluguri et al., 2025].The paper argues that their corpora can delete evidence of teacher failures while leaving the behavior in place.Source: §1 · p. 2
  • Whisper’s hallucinations can include harmful invented phrases, and rates were higher for speakers with longer non-vocal pauses, a symptom of aphasia.Koenecke et al. [2024] found roughly 1% of more than 13,000 AphasiaBank clips contained hallucinated phrases or sentences, and 38% of those hallucinations carried explicit harms.Source: §1 · p. 2
  • The benchmark targets two failures: emitting text over speech-free audio and repeating a known unit, with the decoder’s emission prior described as a common mechanism.The introduction also notes that production serving stacks may omit Whisper decoding controls, leaving the model itself as the remaining intervention point.Source: §1 · pp. 3–4
  • When acoustic evidence is weak or absent, Whisper’s decoder prior can produce fluent text because the model must decide to stop rather than emit a silence token.The paper frames this as the shared source of hallucination and repetition.Source: §1 · p. 3

2 Distilling from Whisper

Whisper distillation pipelines filter pseudo-labels to retain cleaner training data, but that can remove examples demonstrating the teacher’s hallucinations and leave students without examples whose correct output is silence. The paper also argues that negative-only training suppresses emissions without teaching discrimination.

2 Distilling from Whisper

  • Filtering pseudo-labels removes teacher hallucinations, so students are fitted to the teacher’s correct behavior without seeing examples that show those hallucinations are wrong.The paper says students inherit the teacher’s prior beyond clean speech.Source: §2 · p. 4
  • Pseudo-label pipelines may provide almost no examples whose correct transcript is empty, leaving students without training support for saying nothing at inference.Speech corpora are segmented to speech, and clips without voice may be dropped by filters or absent from the corpus.Source: §2 · p. 4
  • Speech segmentation can hide hallucination examples: mining GigaSpeech yields 0.41% of clips, while unsegmented AudioSet yields 54.70%.The paper attributes the difference to speech segmentation throwing away failures.Source: §2 · p. 4

2.1 NEGATIVE-ONLY TRAINING

  • Malaysian-turbo-v3 reaches 16.7% on silence against 61.9% for base large-v3 and 9.3% on non-speech against 89.9%, but recovers 41.1% of spoken phrases against 69.8%.It also deletes 58.0% of clips containing repeated speech and returns character error rate 1.26 on the typical FLEURS language.Source: §2.1 · pp. 4–5
  • Negative-only supervision teaches suppression without discrimination, motivating training examples where the phrases to suppress are also genuinely spoken and transcribed.The paper proposes applying the fix to the teacher with both halves of the signal.Source: §2.1 · p. 5

3 Requirements

The paper’s requirements are to evaluate both hallucination suppression and speech accuracy, enumerate the phrases Whisper hallucinates, and train on those same phrases as both absent and genuinely spoken.

3 Requirements

  • A benchmark should measure suppression and accuracy on the same run, since a model that emits nothing would otherwise appear perfect on non-speech audio.The benchmark must score both halves of the signal.Source: §3 · p. 5
  • Hallucinations are described as a small, language-dependent set of phrases that can be enumerated and converted into training data.Source: §3 · p. 5
  • Training data should pair empty-transcript clips with clips where the same phrases are genuinely spoken and must be transcribed.Without the spoken positives, suppression can become a blanket prior against emitting text.Source: §3 · p. 5

4 Dataset

The dataset combines non-speech arms, synthesized spoken positives, real failure clips, and multilingual accuracy testing. Its design measures both hallucination and whether a model still recognizes speech.

4.1 NON-SPEECH AUDIO

  • The benchmark includes three arms with the empty string as the correct output, making those arms direct tests of invented text.Each of the eight arms is a test split, while training data are separately built and disjoint by construction.Source: §4.1 · p. 5, Table 3 · p. 6

4.2 SPOKEN POSITIVES

  • The spoken-positive arm contains 29,112 synthesized clips in 83 languages, including 6,267 test clips, to test recovery when a benchmark phrase is genuinely spoken.Test clips are also scored as published, with digital-zero padding, and with real room tone padding.Source: §4.2 · pp. 5–6

4.3 WILD AUDIO

  • The wild pool contains real recordings that made a model fail, split by source recording to avoid near-neighbor leakage across train and test.It comprises 3,607 Earnings-22 clips with human span annotations, 4,689 mined clips without annotators, and 187 aphasia clips kept local.Source: §4.3 · p. 6
  • Mining flags token runs of six or more and words emitted where Silero voice activity detection finds no speech, rather than transcript-reference disagreement.GigaSpeech yields 0.41% because its segments are cut to speech.Source: §4.3 · pp. 6–7

4.4 MULTILINGUAL ACCURACY

  • FLEURS tests 20 clips in each of 58 languages and averages character error rate per language, so large languages cannot mask a collapsed one.Character error rate is used because several included languages do not separate words with spaces.Source: §4.4 · p. 7

5 Benchmark results

The benchmark shows that the five checkpoints trade word emission over non-speech audio against recovery of genuine phrases. Testing also exposes failures that a single English accuracy score misses, including on real voice-free audio and multilingual speech.

5 Benchmark results

  • No checkpoint is both quiet on noise and accurate on speech; pooled non-speech word emission and phrase recovery correlate at r = +0.63.The ordering holds across silence, music, and non-speech sound.Source: §5 · pp. 7–8
  • malaysian-whisper-v2 matches base large-v3 on LibriSpeech at 3.5%, yet has character error rate 1.05 on the typical FLEURS language, with only 2 of 58 languages under 0.15 versus 35 for large-v3.The English result does not reveal the multilingual damage.Source: §5 · p. 9
  • For malaysian-turbo-v3, dropping roughly 1% of clips with token runs of six or more changes genuine from 60.7% to 33.9% and speech_in_noise from 58.7% to 25.6%.That small share carries 27 points of word error rate; looping also separates turbo from large-v3 on FLEURS, at 4.7% versus 0.9% of clips.Source: §5 · p. 10

5.1 THE POSITIVE ARM

  • Malaysian-turbo-v3 recovers 41.1% of genuinely spoken phrases versus 69.8% for large-v3, and deletes repeated speech on 58.0% of clips.It wins every non-speech arm, illustrating why non-speech performance alone is insufficient.Source: Figure 3 · p. 8, §5.1 · p. 11
  • Room tone padded around the same speech raises mean character error rate by 0.38 for malaysian-v2, 2.24 for malaysian-turbo-v3, and 0.014 for large-v3.The mean is a tail statistic: 3% to 6% of clips run past twice the phrase length while the median moves by hundredths.Source: §5.1 · p. 11, Figure 7 · p. 11

5.2 RESULTS ON WILD AUDIO

  • On 4,421 real clips confirmed to contain no speech, every OpenAI checkpoint emits words on 100% of clips, while Malaysian-turbo-v3 does so on 8.5%.Of blank-clip outputs, 53.6% for large-v3 and 72.1% for turbo exactly match a lexicon phrase; a blocklist works only once audio is known to be blank.Source: §5.2 · p. 12, Figure 9 · p. 12

6 Lexicon

The lexicon collects recurring hallucination phrases and turns them into material for both identifying and training against the same failure. It is filtered to remove unusable artifacts and fragments before synthesis.

6 Lexicon

  • The merged lexicon contains 40,891 hallucination phrases across 100 languages, drawn from four public, permissively licensed sources.All four sources are vendored in the release for reproducibility.Source: §6 · p. 13, Table 7 · p. 13
  • The phrases NVIDIA’s pipeline deletes from pseudo-label corpora are reused here to generate training data.The examples removed as hallucinations are precisely the phrases a model needs to learn to handle.Source: §6 · p. 13
  • The lexicon serves as both a blocklist and a set of phrases the model must still transcribe when genuinely spoken.This links the suppression and transcription sides of the training signal.Source: §6 · p. 13
  • Entries that are a single repeated token or contain two or fewer distinct characters are removed because they distort character error rate comparisons.A Gujarati character repeated seven times produced a CER of 22.9.Source: §6 · p. 13
  • Only entries observed at least twice are synthesised, yielding 3,882 phrases across 85 languages with a median length of two words.This excludes one-off recognition fragments that fail text-to-speech round trips by construction.Source: §6 · p. 13

7 Synthesis

The synthesis pipeline turns lexicon phrases into spoken clips, selecting systems and conditioning methods by measured recognition, coverage, voice diversity, and duration. Its final corpus contains 29,112 clips in 83 languages.

7 Synthesis

  • Turning lexicon phrases into training data requires each phrase spoken aloud in its own language and in more than one voice.Text-to-speech system choice materially affects this process.Source: §7 · p. 13

7.1 CHOOSING A GENERATOR

  • OmniVoice, with enumerable coverage across 646 languages, drives the hundred-language sweep, while Multilingual-Expressive handles Malay, English, Chinese, and Tamil.Coverage and quality differ: ToucanTTS covers 7,233 languages but wins nothing; Multilingual-Expressive’s 0.221 is measured in speaker-name mode.Source: §7.1 · p. 14, Figure 10 · p. 14, Table 8 · p. 14

7.2 VOICE DIVERSITY

  • OmniVoice has no speaker argument, and 33 of its 74 languages have median pairwise speaker similarity of 0.75 or worse.The team topped up 26 languages with three named voices per phrase; 12,190 of 15,675 new clips passed the quality gate, and median similarity moved from 0.81 to 0.63.Source: §7.2 · p. 15, Figure 11 · p. 15

7.3 VOICE CONVERSION

  • Character error rate alone can select a converter that returns its input, so the comparison also measures voice movement relative to source and target.The reported conversion gap distinguishes actual voice conversion from unchanged input.Source: §7.3 · p. 16, Figure 12 · p. 16
  • Multilingual-Expressive’s speaker-name mode is the intelligibility winner, while its reference-clip cloning mode is the worst system measured because it over-generates.The cloning mode’s median output is 3.27 times source length, with 67.9% of clips exceeding twice the length and character error rate 1.740.Source: §7.3 · p. 16, Table 10 · p. 16

7.4 QUALITY GATE

  • The per-language quality gate uses the judge’s FLEURS character-error floor, excludes nine languages where that floor is too high, and yields 29,112 clips in 83 languages.The corpus contains 22,845 training clips and 6,267 test clips, totaling 15.8 hours and 92% non-English audio.Source: §7.4 · p. 17

8 Training sweep

The sweep compares paired fine-tunes with and without synthetic positives across four setup factors and evaluates both hallucination suppression and speech recovery. Synthetic positives yield better quiet-versus-recovery trade-offs, while adapter coverage and multilingual accuracy remain important constraints.

8.1 SETUP

  • The training mix uses empty transcripts for blank clips, masks prompt tokens from loss, and compares mixes differing only by synthetic positives.The paired mixes and 66-run grid are designed to isolate training-data and configuration effects.Source: §8.1 · p. 17, Table 12 · p. 18, §8.4 · p. 19

8.3 GRID

  • The 66-run grid pairs 33 configurations across two mixes, varying adapter method, rank, and learning rate while holding effective batch size constant.Each run trains for 1,000 steps and is scored on the complete benchmark, synthetic positives, wild test split, and FLEURS.Source: §8.3 · pp. 18–19, Table 13 · p. 18, Table 15 · p. 19

8.4 RESULTS

  • Without synthetic positives, accuracy-preserving runs reduce wild-audio word emission from 0.999 to 0.799–0.968, while phrase recovery falls from 0.698 to 0.511–0.602.FLEURS character error rate also rises from 0.305 to 0.522–0.809 in these runs.Source: §8.4 · p. 22
  • In accuracy-preserving pairs, synthetic positives lower wild-audio word emission in 30 of 30 configurations and raise phrase recovery in 29 of 30.The quieter no_synth configurations were those that failed the LibriSpeech accuracy condition.Source: §8.4 · p. 22
  • 61.9% to 2.4% silence hallucination and 69.8% to 82.7% phrase recovery mark the best checkpoint, with wild voice-free word emission falling from 99.9% to 47.8%.That checkpoint raises LibriSpeech word error rate from 0.035 to 0.041 and FLEURS character error rate from 0.305 to 0.398.Source: §8.4 · pp. 22–23
  • Training loss ranks the mixes misleadingly: 28 of 30 no_synth LoRA runs finish below every all-mix LoRA run, despite recovering fewer phrases.The blank-heavy no_synth mix makes empty transcripts cheap to predict; the mixes contain 77% versus 47% blank-labelled clips.Source: §8.2 · p. 18, §8.4 · p. 24, Figure 16 · p. 24
  • Adding feed-forward matrices alongside attention lowers wild-audio word emission in all 15 matched cells on the all mix, by a median of 0.133.Attention-only adapters remain at 0.762–0.911, while attention-plus-MLP configurations span 0.241–0.807; without synthetic positives, the module-choice advantage disappears.Source: §8.4 · p. 25, Figure 17 · p. 25
  • Every fine-tune worsens FLEURS relative to the base model, although synthetic positives reduce the median character error rate from 0.607 to 0.460.The best of 24 fine-tunes reaches 0.381 versus 0.305 for the base model; LibriSpeech word error rate stays near 0.034–0.037 across almost every run.Source: §8.4 · p. 26

9 Conclusion

Whisper’s hallucinations persist in its weights, while filtering them from pseudo-label corpora removes evidence of the failure and negative-only audio teaches suppression without discrimination. The paper synthesizes hallucination phrases as spoken positives; matched fine-tunes show improved voice-free suppression and phrase recovery, while adapter choice and FLEURS results reveal costs and constraints.

9 Conclusion

  • Filtering hallucinations from pseudo-labels removes failure examples while retaining the teacher’s behavior, leaving students fitted to the teacher’s already-correct cases.The paper argues that the teacher must instead be trained with both negative examples and genuinely spoken phrases.Source: §9 · p. 26
  • Negative-only supervision can make a model quiet without teaching discrimination, with the quietest run recovering only 29.2% of spoken phrases and tripling LibriSpeech word error rate.The checkpoint best on non-speech arms also deletes 58% of repeated speech.Source: §9 · p. 26
  • Across 33 matched fine-tune pairs differing only in synthetic positives, real voice-free-audio word emission fell in 31 pairs and phrase recovery rose in 32; among 30 pairs holding English accuracy, all 30 improved.The best checkpoint takes silence hallucination from 61.9% to 2.4% and words over real voice-free audio from 99.9% to 47.8%, while raising phrase recovery from 69.8% to 82.7%, for 0.005 LibriSpeech word error rate.Source: §9 · p. 26
  • Attention-only adapters do not learn to stop at any tested rank, while adding feed-forward matrices at rank 8 does so with a quarter of the parameters.Every fine-tune worsens FLEURS despite LibriSpeech remaining flat at 0.034–0.037, so English-only evaluation misses that cost.Source: §9 · pp. 26–27

10 Limitations

The evaluation has important scope and measurement boundaries, including synthetic-heavy positives, sparse language coverage, and limited or mismatched audio conditions. A filtered corpus also partly favors the checkpoint used to create it.

10 Limitations

  • 52% of the positive pool is synthetic text-to-speech, 23% read prose, and 25% real conversational speech from Sarawak dialect.The authors note that tuning on synthetic phrases may teach the model about the generator as well as the phrases.Source: §10 · p. 27
  • The evaluation has no telephony condition: all audio is 16 kHz, while much production audio is 8 kHz narrowband.Source: §10 · p. 27
  • Chinese and Tamil positives are nearly absent, with only 11 and 4 clips, respectively.Source: §10 · p. 27
  • The 42-clip silence arm resolves only 2.4 percentage points per clip; music may contain singing, so prefer label-verified nonspeech when results disagree.Source: §10 · p. 27
  • Synthetic speech-in-noise mixing sets the signal-to-noise ratio exactly but includes neither a Lombard effect nor a channel.Source: §10 · p. 27
  • Because whisper-large-v3 filtered the synthetic corpus with language forced, it is scored partly on clips it already agreed with, flattering it against large-v2 and turbo.This does not explain the 30-point gap to the Malay fine-tunes.Source: §10 · p. 27

11 Future work

Future work includes an unrun decoder-side ablation, longer training, improved language-specific judging floors, and a larger wild-audio pool for condition-level measurement.

11 Future work

  • The planned decoder-side ablation remains unrun: 34 configurations would test decoding knobs and their interaction with a fine-tuned checkpoint.The motivating question is whether a model trained to stop still needs those knobs.Source: §11 · p. 27
  • Stage two uses LoRA rank 64 at 5e-4 on the all mix for whisper-large-v3-turbo; each run is 1,000 steps, roughly one-third of an epoch.A longer run would test whether the FLEURS regression is transient undertraining or a real data cost.Source: §11 · p. 27
  • Sinhala and Malagasy lack FLEURS configurations and judge floors, so a flat gate accepts almost no phrases; another corpus is needed to measure their floors.Source: §11 · p. 27
  • Expanding the wild pool could enable hallucination measurement by acoustic condition rather than only in aggregate.AudioSet yields more than half of requested clips, making a tenfold collection increase a compute-and-storage cost.Source: §11 · p. 27

Related papers

Loading 2609.32560v2…