Reducing Hallucinated Transcripts in Whisper via Hallucination Space Projection

1 Introduction

• A reliable ASR system should transcribe speech accurately and remain empty when audio contains no speech.Non-speech inputs can include silence, music, environmental sounds, and background noise.Source: §1 · p. 1
• Whisper’s generative decoder can emit plausible hallucinated text for non-speech audio, including acknowledgments, fillers, and repeated fragments.Such outputs may be mistaken for real speech.Source: §1 · p. 1
• The method estimates a low-rank hallucination-associated direction from non-speech calibration examples and projects decoder hidden states away from it without parameter updates.The projection basis is estimated once and kept fixed.Source: §1 · p. 2
• Across non-speech and speech benchmarks, the compact projection reduces hallucinated transcripts while preserving ASR performance, without external VAD, phrase filtering, or fine-tuning.The introduction reports evaluation on ESC-50, UrbanSound8K, FSD50K, and LibriSpeech.Source: §1 · p. 2

2.1 Whisper as a Foundation Model for ASR

• Whisper’s autoregressive decoder combines transcription and no-speech estimation, but can generate fluent text when acoustic evidence is weak or absent.This behavior motivates decoder-level mitigation for hallucinated non-speech transcripts.Source: §2.1 · p. 2

2.2 Whisper Hallucinations on Non-Speech Audio

• For non-speech audio, the correct transcript is empty, so any emitted text counts as hallucinated rather than a standard WER error against a non-empty reference.The empty reference makes hallucination directly measurable by whether text is emitted.Source: §2.2 · p. 2
• Koenecke et al. (2024) report unsupported content in about 1% of Whisper transcriptions, with roughly 38% of those cases potentially harmful or misleading.They also observe higher hallucination rates with long pauses or weak vocal activity.Source: §2.2 · p. 2
• Baránski et al. (2025) report that Whisper large-v3 produces non-empty transcripts for 40.3% of non-speech inputs, with rates varying by duration.Reported outputs include acknowledgments, fillers, applause markers, and animal-sound onomatopoeia.Source: §2.2 · p. 2

2.3 Existing Mitigation Strategies

• VAD removes non-speech before decoding, post-processing filters generated phrases, and fine-tuning updates decoder components; each approach has stated dependencies or risks.VAD depends on external segmentation, post-processing may miss unseen hallucinations or remove valid speech, and fine-tuning may overfit.Source: §2.3 · p. 2
• The proposed method instead intervenes on decoder activations at inference time without external VAD, phrase filtering, or parameter updates.Existing no-speech, log-probability, and compression-ratio heuristics operate on output-level scores.Source: §2.3 · pp. 2–3

3 Proposed Method: Gated Low-Rank Decoder Projection

• A training-free method estimates a hallucination-associated subspace from calibration data and projects intermediate decoder activations away from it during generation.The projection is applied only when Whisper’s default no-speech probability exceeds a gate threshold.Source: §3 · p. 3

3.1 Calibration Activations

• Calibration separates non-speech examples that produce hallucinated transcripts from non-speech examples correctly rejected with no transcript.The hallucination-associated subspace is estimated from the calibration set and kept fixed.Source: §3.1 · p. 3

3.2 Low-Rank Subspace Estimation

• Activation differences between the two non-speech groups identify decoder directions associated with hallucination-prone behavior, and the top r right singular vectors form the projection basis.The rank r controls the dimensionality of the subspace removed at inference.Source: §3.2 · p. 3

3.3 Inference-Time Decoder Projection

• At a selected decoder layer, the method removes a controllable fraction of the hidden-state component aligned with the hallucination-associated subspace.The projection strength α controls the amount removed; the intended effect is to reduce hallucinations while preserving transcription information.Source: §3.3 · p. 3

3.4 No-Speech-Conditioned Gating

• Gating uses an initial unprojected pass and applies projection only when its no-speech probability exceeds γ; otherwise, the unprojected output is retained.The resulting no-speech probability is then compared with Whisper’s threshold τ for the final rejection decision.Source: §3.4 · p. 4

4 Experimental Setup

• Experiments use pretrained Whisper small, medium, and large-v3 models and run on an NVIDIA DGX A100 system.The implementation uses the official OpenAI Whisper implementation.Source: §4 · p. 4

4.1 Datasets

• Non-speech evaluation uses ESC-50, UrbanSound8K, and an FSD50K subset filtered to exclude speech, vocal, and music labels.ESC-50 folds 1–3 provide calibration and parameter selection; held-out folds 4–5, UrbanSound8K, and filtered FSD50K are used for testing.Source: §4.1 · p. 4

4.2 Evaluation Metrics and Parameter Selection

• The study measures non-speech hallucinations with HR and speech recognition with WER and speech FRR; each model’s parameters are selected on calibration and validation data then fixed for evaluation.LibriSpeech validation-clean controls degradation on valid speech during parameter selection.Source: §4.2 · p. 4

4.3 Evaluation Baselines

• Baselines include original Whisper, always-on and gated projection, external VAD, post-hoc phrase filtering, and no-speech threshold tuning.Gated projection applies the intervention when Whisper’s no-speech estimate exceeds γ and otherwise leaves the model unchanged.Source: §4.3 · p. 5

5.1 Offline Parameter Selection

• Middle-to-late decoder layers substantially reduce HR, whereas early-layer interventions have little effect; overly strong interventions can increase WER and FRR.For large-v3, the selected compact setting is ℓ=28 and r=4; α=1.0 and γ=0.05 are used for subsequent experiments.Source: §5.1 · p. 5
• Small projection ranks capture the main hallucination directions, but increasing rank does not consistently lower HR and can worsen WER and FRR.Rank variation shows saturation rather than steady improvement.Source: §5.1 · p. 5

5.2 Hallucination Reduction and Cross-Dataset Generalization

• 44.25% of held-out ESC-50 examples still receive non-empty transcripts from original Whisper; on UrbanSound8K, HR falls from 76.08% to 0.87% with always-on and 2.74% with gated projection.On filtered FSD50K, HR falls from 21.35% to 0.18% and 1.15%, respectively; transfer suggests a reusable decoder-level signature.Source: §5.2 · pp. 5–6, Table 1 · p. 6

5.3 Speech Recognition and False Rejection

• For large-v3, gated projection yields 6.17% and 6.57% WER on test-clean and test-other, versus 12.95% and 13.13% with always-on projection.Original Whisper obtains 4.06% and 5.87% WER on those splits.Source: §5.3 · p. 6, Table 2 · p. 6
• For large-v3, gated projection lowers FRR to 2.86% on test-clean and 1.40% on test-other, versus 10.50% and 11.47% with always-on projection.The probability distributions show non-speech examples shifting above the rejection threshold while clear-speech probabilities remain near zero.Source: §5.3 · p. 7, Table 3 · p. 7

5.4 Comparison with Baselines

• WhisperX VAD reaches 2.67% and 4.74% WER with 0% FRR on both LibriSpeech splits, but requires an external VAD and segmentation stage.Gated projection has higher WER and FRR, but requires no external detector.Source: §5.4 · pp. 7–8
• 4.09% average HR with gated projection beats WhisperX VAD’s 5.24%; always-on reaches 0.85% but at a larger WER and FRR cost.Gated projection achieves this trade-off without an external detector.Source: §5.4 · pp. 7–8, Figure 4 · p. 8

• The method estimates a compact hallucination-associated subspace and edits decoder representations at inference without parameter updates.Source: §6 · p. 8
• Across multiple datasets and Whisper scales, the method substantially reduces hallucinated transcripts while providing a controllable HR–WER–FRR trade-off.Source: §6 · p. 8
• The results indicate that hallucination-associated decoder directions can be manipulated through lightweight inference-time intervention.Source: §6 · p. 8

• The method is designed and evaluated for non-speech hallucinations; long-form, multilingual, and acoustically ambiguous scenarios remain future work.Source: Limitations · p. 9
• Gated projection reduces the speech degradation of always-on projection, but some operating points still increase WER and FRR.Where missed speech is costly, less aggressive settings may be preferred, and uncertain segments can be retained for re-transcription or verification.Source: Limitations · p. 9
• The subspace is estimated from non-speech calibration data, and the gated variant depends partly on Whisper’s internal no-speech probability.Broader calibration-to-deployment shifts remain to be evaluated.Source: Limitations · p. 9

Ethical Considerations

• The experiments use pretrained Whisper weights and public benchmarks, without collecting new audio, annotations, or human-subject data or redistributing datasets or derived audio.Source: Ethical Considerations · p. 9
• The benchmarks cover environmental and urban non-speech audio, a filtered FSD50K non-speech subset, and English read-speech ASR.The study reports sample counts and development/test splits.Source: Ethical Considerations · p. 9
• The study reports aggregate LibriSpeech metrics without attempting speaker identification or demographic inference, and removes speech, vocal, or music clips from the FSD50K non-speech subset.Source: Ethical Considerations · p. 9

A Gated Low-Rank Decoder Projection Algorithm

• The gated procedure applies projection only when Whisper’s no-speech probability exceeds γ; otherwise, decoding proceeds normally.The basis is estimated once and kept fixed across evaluation datasets; γ controls the gate and τ controls final no-speech filtering.Source: A Gated Low-Rank Decoder Projection… · p. 10

B Examples of Whisper Hallucinations on Non-Speech Audio

• Examples show that non-speech audio can trigger fluent but unrelated textual continuations despite containing no spoken content.Source: B Examples of Whisper Hallucinations… · p. 10

C Additional Results for Whisper Small and Medium

• With Whisper large-v3, projection effectiveness depends strongly on decoder layer and rank, with middle-to-late layers achieving lower HR than early layers.Source: C Additional Results for Whisper Small… · p. 10
• For Whisper small, the selected always-on setting uses layer ℓ = 10 and rank r = 1; the offline analysis compares development-set HR on ESC-50 folds 1–3 with WER on LibriSpeech validation-clean.Source: C Additional Results for Whisper Small… · p. 10, Figure 5 · p. 11
• For Whisper small, gated projection achieves 1.2% HR with 4.68% WER, compared with the selected non-gated setting’s HR reduction from 20.2% to 1.08% and WER increase from 4.28% to 5.07%.The gated sweep selects α = 1.0 and γ = 0.15; the non-gated setting uses layer ℓ = 10 and rank r = 1.Source: C Additional Results for Whisper Small… · p. 10
• For Whisper medium, gated projection achieves 2.8% HR with 3.90% WER, while the selected non-gated configuration reduces HR from 22.8% to 2.0% with 5.13% WER.The selected non-gated setting uses layer ℓ = 24 and rank r = 2; the gated setting uses α = 0.75 and γ = 0.10.Source: C Additional Results for Whisper Small… · p. 10

D.1 No-Speech Threshold Tuning with Gated Projection

• Because projection changes Whisper’s no-speech probabilities, the authors test whether varying τ improves gated projection’s operating point.They fix the projection configuration and compare original Whisper with gated projection at each threshold.Source: §D.1 · p. 11
• At τ = 0.4, average HR falls from 5.09% to 2.89%, while average WER rises from 8.58% to 11.49% and speech FRR from 7.03% to 10.91%.These comparisons are for original Whisper versus gated projection at the same threshold.Source: §D.1 · p. 11, Table 5 · p. 12
• At τ = 0.7, gated projection lowers average WER to 7.28% and speech FRR to 4.76%, while average HR rises to 7.85%.The default τ = 0.6 provides the best trade-off; lower or higher thresholds prioritize lower HR or lower speech rejection, respectively.Source: §D.1 · pp. 11–12, Table 5 · p. 12

D.2 Multi-Layer Gated Projection

• Multi-layer projection raises LibriSpeech WER from 5.48% to 10.62% on test-clean and from 11.68% to 15.71% on test-other.Speech FRR also rises from 4.07% to 11.07% on test-clean and from 9.97% to 13.75% on test-other.Source: §D.2 · p. 12, Table 6 · p. 13
• HR falls from 2.75% to 0.13% on ESC-50, from 0.66% to 0.07% on UrbanSound8K, and from 8.78% to 4.87% on FSD50K with multi-layer projection.The configuration uses gated projections at layers 20 and 24; the authors retain single-layer projection as the main setting because multi-layer projection increases speech degradation.Source: §D.2 · p. 12, Table 6 · p. 13

E Bag of Hallucinations Phrase List

• The BoH baseline removes frequent hallucinated phrases from decoded transcripts when they appear as exact normalized matches.The phrase list comes from Whisper outputs on the ESC-50 development split.Source: E Bag of Hallucinations Phrase List · p. 13, Table 7 · p. 13

F No-Speech Threshold Sweep

• Threshold-only tuning gives its lowest average HR at τ = 0.5, but improves only modestly over the default τ = 0.6.WER and FRR remain largely unchanged across thresholds, so threshold tuning alone is insufficient to substantially mitigate non-speech hallucinations.Source: F No-Speech Threshold Sweep · p. 13, Table 8 · p. 14