Source-linked AI summary
Whisper-AT: Noise-Robust Automatic Speech Recognizers are Also Strong General Audio Event Taggers
Yuan Gong, Sameer Khurana, Leonid Karlinsky, James Glass
TL;DR
Robust speech recognition and audio tagging are typically handled by separate systems, although applications may need both spoken text and acoustic scene information. The paper analyzes Whisper’s noise-variant representations and builds Whisper-AT, which adds audio tagging to speech recognition with less than 1% extra computational cost in a single forward pass.
Problem
ASR and audio-tagging models are typically run independently, making it computationally expensive to obtain both spoken text and acoustic scene analysis.
Source: §1 · p. 1Method
Results
With less than 1% extra computational cost, Whisper-AT recognizes audio events alongside spoken text in one forward pass; it achieves 41.5 mAP on AudioSet, slightly below standalone audio-tagging models but over 40× faster.
Source: §1 · p. 1Takeaways & Limitations
For systems already running ASR, Whisper-AT targets adding audio labels without the separate additional audio-tagging cost.
Source: §5.2 · p. 4The analysis is mostly based on Whisper and does not establish that noise-invariant representations fail to help robust ASR.
Source: §3 · p. 2
Abstract
from arXiv · showhide
In this paper, we focus on Whisper, a recent automatic speech recognition model trained with a massive 680k hour labeled speech corpus recorded in diverse conditions. We first show an interesting finding that while Whisper is very robust against real-world background sounds (e.g., music), its audio representation is actually not noise-invariant, but is instead highly correlated to non-speech sounds, indicating that Whisper recognizes speech conditioned on the noise type. With this finding, we build a unified audio tagging and speech recognition model Whisper-AT by freezing the backbone of Whisper, and training a lightweight audio tagging model on top of it. With <1% extra computational cost, Whisper-AT can recognize audio events, in addition to spoken text, in a single forward pass.
1. Introduction
Whisper’s diverse training data yields robust ASR, but its representations encode background sounds rather than removing them. The paper uses this finding to motivate a unified ASR and audio-tagging model.
1. Introduction
- Whisper was trained on 680K hours of labeled Internet speech from diverse environments and recording setups, unlike clean-speech public checkpoints.Those checkpoints’ real-world robustness is limited, while Whisper reports better robustness over existing ASR models.Source: §1 · p. 1
- Whisper’s noise robustness coincides with representations rich in non-speech background-sound information, rather than noise-invariant representations.Figure 1 reports a positive relationship between robustness and encoded background-sound information; Whisper retains that information even in its deepest layer.Source: §1 · p. 1, Figure 1 · p. 1
- Whisper-AT reaches 41.5 mAP on AudioSet, slightly below standalone audio-tagging models but over 40× faster.It adds audio-event recognition to spoken-text recognition in a single forward pass with less than 1% extra computational cost.Source: §1 · p. 1
- The authors report that robust ASR can learn noise-variant representations, contrasting with prior work emphasizing noise-invariant representations.They distinguish their target of large-vocabulary continuous speech recognition from keyword spotting [12] and note that prior ASR representations tested for audio tagging performed unsatisfactorily.Source: §1 · p. 1
2. Whisper Robust ASR Model
Whisper uses a standard Transformer encoder-decoder; the paper attributes its distinction from existing ASR models primarily to its unusually large and diverse training data and training scheme.
2. Whisper Robust ASR Model
- Whisper’s 680K-hour non-public training set pairs Internet audio with transcripts across diverse environments, recording setups, speakers, and languages.Compared with audiobook-based Librispeech (960 hours) and Libri-light (60K hours), it is larger and more diverse, but has noisy labels.Source: §2 · p. 2
- Whisper uses only text transcripts as supervision, with no audio-event labels during training.The authors identify its data and training scheme, rather than its standard Transformer architecture, as the main distinction from existing ASR models.Source: §2 · p. 2
- For audio tagging, the paper uses only Whisper-Large’s 32-layer audio encoder, whose Transformer layers have dimension 1280.Whisper itself has a Transformer encoder-decoder architecture.Source: §2 · p. 2
3. Noise-Robust ASR Learns Noise-Variant Representations
Whisper remains robust to environmental sounds while encoding them in its representations. The authors describe its robustness as noise-conditioned and qualify that this finding does not establish noise-invariance as ineffective.
3. Noise-Robust ASR Learns Noise-Variant Representations
- Whisper outperforms other state-of-the-art ASR models on ESC-50 sound classification probes, and unlike them retains background-sound information in deeper layers.The experiment freezes Whisper, extracts representations from each layer, and trains a linear classifier for 50 sound classes.Source: §3 · p. 2
- Whisper’s noise-robustness mechanism differs from noise-invariant ASR: it encodes background sound and transcribes speech conditioned on the noise type.The authors contrast this with wav2vec2-robust.Source: §3 · p. 2
- Class-wise sound-recognition ability positively correlates with robustness, but recognizing a sound is necessary, not sufficient, for robustness to it.Some recognizable sounds overlap heavily with speech and remain difficult to handle robustly.Source: Figure 2 · p. 2, §3 · p. 2
- The experiments do not show that noise-invariance fails to help robust ASR or that robust ASR must use noise-variant representations.The discussion is mostly based on Whisper; the authors describe noise-invariance as practical for self-supervised learning or small-data cases and note Whisper’s training requires industry-level resources.Source: §3 · p. 2
4. Unifying ASR and Audio Tagging Model
Whisper-AT preserves Whisper’s ASR weights while adding audio-tagging layers to produce spoken text and sound labels together. The section develops tagging heads from layer aggregation through temporal and layer-wise Transformers, with pooling and projection choices to reduce added computation.
4. Unifying ASR and Audio Tagging Model
- Whisper-AT adds audio-tagging layers atop Whisper without modifying its weights, producing spoken text and sound labels in a single forward pass.The added layers use Whisper’s intermediate representations.Source: §4 · p. 3
- Different ESC-50 sound classes achieve best F1 at different Whisper layers, motivating TL-Tr’s layer-wise Transformer after its temporal Transformer.Unlike weighted averaging, TL-Tr uses attention over layers so classes can use different layer weights.Source: §4 · p. 3, Figure 3 · p. 3
- The tagging-head progression is Last-MLP, WA-MLP, WA-Tr, and TL-Tr, moving from last-layer pooling to weighted layer aggregation and Transformer modeling.Last-MLP applies temporal mean pooling and a linear layer; WA-MLP learns weights across layers, and WA-Tr replaces its linear layer with a temporal Transformer.Source: §4 · p. 3
- Pooling cuts sequence length from 500 to 25; optional projection reduces dimension from 1280 to 512, and WA-Tr and TL-Tr each use one temporal Transformer.These design choices aim to minimize the extra audio-tagging cost.Source: §4 · p. 3, Figure 4 · p. 3
5. Experiments
Experiments evaluate audio tagging on AudioSet and ESC-50 while keeping Whisper’s ASR performance unchanged. Whisper-AT adds efficient tagging, with results spanning backbone comparisons, model designs, fine-tuning, and Whisper model sizes.
5.1. Experiment Settings
- Experiments preserve Whisper’s original ASR performance by freezing its weights and evaluate tagging on AudioSet and ESC-50 using standard protocols.AudioSet experiments use AS-20K and AS-2M; ESC-50 evaluation uses official 5-fold cross-validation.Source: §5 · p. 3, §5.1 · p. 3
5.2. Experiment Results
- 32.8 mAP, 41.5 mAP, and 91.7 accuracy are achieved by TL-Tr512 on AS-20K, AS-2M, and ESC-50, and it is 42 times faster and 11 times smaller than AST.The comparison concerns additional tagging cost, excluding the ASR backbone.Source: Table 1 · p. 4, §5.2 · p. 4
- Whisper-AT is significantly stronger than Hubert X-Large and wav2vec2-Large-Robust on audio tagging.The authors characterize Whisper as both the most robust ASR model and the strongest audio-tagging backbone.Source: §5.2 · p. 4, Table 1 · p. 4
- TL-Tr performs best among the four Whisper-AT models at higher computational cost, while TL-Tr512 approaches TL-Tr1280 with less than 1% of Whisper ASR FLOPs.TL-Tr128 also provides reasonably good audio-tagging performance at less than 0.1% of Whisper ASR FLOPs.Source: §5.2 · p. 4, Table 2 · p. 4
- After end-to-end fine-tuning, Whisper-Small performs similarly to self-supervised pretrained models SSAST and MAE-AST.The comparison uses a newly added Transformer layer and is described as not exactly fair in Table 1.Source: §5.2 · p. 4, Table 1 · p. 4
- Larger Whisper models are more robust to ESC-50-contaminated speech, while audio-tagging differences among Small, Medium, and Large are minor.Whisper-Base (74M parameters) is already more robust in ASR and stronger in audio tagging than Hubert-X-Large (964M parameters); Figure 5 presents AS-2M tagging and ASR robustness.Source: §5.2 · p. 4, Figure 5 · p. 4
6. Conclusion
The conclusion characterizes Whisper as a robust ASR model whose representation is noise-variant and encodes background-sound information. It presents Whisper-AT as a unified model that adds audio tagging to speech recognition with less than 1% extra cost in a single forward pass.
6. Conclusion
- Whisper’s audio representation is noise-variant and encodes rich background-sound information despite the model’s robustness.Source: §6 · p. 4
- Whisper-AT unifies audio tagging and ASR to recognize background sounds alongside spoken text.Source: §6 · p. 4
- Whisper-AT performs both tasks in a single forward pass with less than 1% additional computational cost.Source: §6 · p. 4