Source-linked AI summary
Does Audio Deepfake Detection Generalize?
Nicolas M. Müller, Pavel Czempin, Franziska Dieckmann, Adam Froghyar, Konstantin Böttinger
TL;DR
Audio deepfake detection lacks clear evidence about which architectural and preprocessing choices drive performance, and benchmark results may not reflect real-world generalization. The paper standardizes evaluation by reimplementing related architectures and testing them on ASVspoof and a new in-the-wild dataset. It finds that cqtspec or logspec features outperform melspec, while established models degrade sharply on real-world data.
Problem
Prior audio spoofing studies use inconsistent architectures, preprocessing, hyperparameters, and fine-tuning, while evaluation has relied exclusively on ASVspoof’s limited TTS and studio setting.
Method
The paper reimplements twelve popular architectures, systematically exchanges model and preprocessing components, and evaluates them on ASVspoof and a newly collected in-the-wild dataset.
Results
cqtspec or logspec features consistently outperform melspec, and established models’ EER deteriorates by about 200 to 1000 percent on in-the-wild data.
Takeaways & Limitations
The findings support considering cqtspec features and unabridged audio while indicating that many models’ in-the-wild generalization capabilities may have been overestimated.
Takeaways & Limitations
Prior evaluation was exclusively on ASVspoof, whose reported performance is based on a limited set of TTS synthesis algorithms.
Abstract
from arXiv · showhide
Current text-to-speech algorithms produce realistic fakes of human voices, making deepfake detection a much-needed area of research. While researchers have presented various techniques for detecting audio spoofs, it is often unclear exactly why these architectures are successful: Preprocessing steps, hyperparameter settings, and the degree of fine-tuning are not consistent across related work. Which factors contribute to success, and which are accidental? In this work, we address this problem: We systematize audio spoofing detection by re-implementing and uniformly evaluating architectures from related work. We identify overarching features for successful audio deepfake detection, such as using cqtspec or logspec features instead of melspec features, which improves performance by 37% EER on average, all other factors constant. Additionally, we evaluate generalization capabilities: We collect and publish a new dataset consisting of 37.9 hours of found audio recordings of celebrities and politicians, of which 17.2 hours are deepfakes. We find that related work performs poorly on such real-world data (performance degradation of up to one thousand percent). This may suggest that the community has tailored its solutions too closely to the prevailing ASVSpoof benchmark and that deepfakes are much harder to detect outside the lab than previously thought.
1. Introduction
Realistic TTS voices create societal risks, while audio spoof detection remains difficult to interpret and its real-world generalization is uncertain. The paper addresses these gaps through standardized evaluation and a new in-the-wild dataset.
- Realistic TTS can clone politicians’ voices, creating risks including the spread of misinformation.
- Audio spoof detection remains difficult to understand because prior studies vary in features, preprocessing, hyperparameters, and fine-tuning.
- ASVspoof-based evaluation relies on a limited set of TTS algorithms and studio-recorded professional speech.
- The paper reimplements twelve popular architectures, evaluates components under a common standard, and introduces a new real-world audio deepfake dataset.
- The authors report that established models generally perform poorly on real-world data, indicating weaker generalization than benchmark results suggest.
2. Related Work
The evaluated related work spans end-to-end waveform models, recurrent and convolutional architectures, residual networks, attention-based models, and hybrid designs. These models differ in how they represent audio and aggregate temporal information.
- Audio spoof detection research is driven largely by the ASVspoof challenges and datasets.
- LSTM-based models: The baseline LSTM uses three recurrent layers followed by a linear layer and averages outputs over time into one embedding.
- LCNN: LCNN variants combine convolutional layers with Max-Feature-Map activations, adding attention pooling or Bi-LSTM skip connections.
- MesoNet: MesoNet uses four convolutional layers with normalization, pooling, and a fully connected classifier, while MesoInception adds Inception blocks.
- ResNet18 uses convolutional layers and shortcut connections, whereas the Transformer uses self-attention layers, skip connections, and positional encodings.
- RawNet2, RawPC, RawGAT-ST, and CRNNSpoof learn from raw waveforms using Sinc layers, graph attention, or convolutional-recurrent combinations.
3. Datasets
The study evaluates spoof detection on ASVspoof 2019 and a newly collected in-the-wild dataset pairing authentic and spoofed recordings from public sources. The new dataset is designed to test cross-database performance in realistic conditions.
- ASVspoof 2019 LA contains real or fake speech, with spoofed files generated by 19 TTS synthesis algorithms.
- The new dataset contains 37.9 hours of audio, including 17.2 hours of fake clips and 20.7 hours of real clips.
- Spoofed clips were segmented from 219 public files advertising audio deepfakes, while genuine recordings were manually collected from the same speakers.
- The dataset covers 58 English-speaking celebrities and politicians using spoofed and bona-fide recordings from publicly available material.
- Clips were collected from social networks and video-sharing platforms and are intended for realistic cross-database evaluation.
4. Experimental Setup
The experiments compare twelve models across audio representations and input-length strategies under standardized training and evaluation procedures. Performance is measured on both in-domain ASVspoof data and unknown out-of-domain recordings.
- Models are trained with cross-entropy loss, Adam optimization, a 0.0001 initial learning rate, scheduling, and early stopping over 100 epochs.
- Training uses ASVspoof 2019 LA train and development splits, while evaluation uses its eval split and the separate in-the-wild dataset.
- ASVspoof evaluation reports EER and t-DCF, whereas the proposed dataset reports only EER because t-DCF costs are unavailable.
- Feature Extraction: Feature-based architectures are evaluated with cqtspec, logspec, and melspec representations, while other models receive raw waveforms.
- Table 1 reports averaged EER and t-DCF across three random-initialization trials and compares architectures, features, input lengths, and in-the-wild degradation.
- Input Length: Fixed-length four-second inputs perform significantly worse than variable-length inputs on both ASVspoof and in-the-wild data.
- Input Length: The study compares fixed four-second inputs with full variable-length audio, extending models with global averaging when needed.
5. Results
Across the common evaluation, input duration and feature preprocessing substantially affect spoof-detection performance, while models generalize poorly from ASVspoof to real-world audio. Full-length audio and non-mel spectrogram features are especially beneficial, but adding ASVspoof evaluation data does not improve out-of-domain results.
- Fixed vs. Variable Input Length: Average ASVspoof EER drops from 19.89% to 9.85% when models use full-length audio instead of four-second inputs.For all models, trimming the input to 4s decreases performance because the clip is insufficient for extracting useful information.
- Effects of Feature Extraction Techniques: Raw models reach up to 1.2% EER on ASVspoof and 33.9% EER on the ‘in-the-wild’ dataset, outperforming feature-based models.The reported raw-model results are achieved by RawGAT-ST and RawNet2.
- Effects of Feature Extraction Techniques: Replacing melspec with cqtspec increases average performance by 37%, with all other factors held constant.Among spectrogram-based models, melspec is consistently outperformed by cqtspec or logspec.
- Evaluation on ‘in-the-wild’ data: EER values deteriorate by about 200 to 1000 percent on the ‘in-the-wild’ dataset, and models often perform no better than random guessing.The comparison is between ASVspoof 2019 evaluation data and the real-world deepfake dataset.
- Evaluation on ‘in-the-wild’ data: Training RawNet2 on all ASVspoof 2019 splits yields 33.1 ± 0.2 % EER, providing no improvement over training on only the ‘train’ and ‘dev’ splits.Including the evaluation split does not appear to add information useful for real-world generalization because all splits are based on VCTK.
6. Conclusion
The paper standardizes evaluation across audio spoof-detection architectures and introduces an in-the-wild dataset to test real-world generalization. It finds that cqtspec or logspec features and unabridged input audio are promising design choices, while many models generalize poorly outside ASVspoof.
- Conclusion: The authors reimplement and evaluate related audio spoof-detection architectures using common standards and introduce a real-world deepfake dataset.The dataset is used to assess generalization beyond the prevailing benchmark.
- Conclusion: Cqtspec or logspec features consistently outperform melspec, and four seconds of audio often does not saturate performance compared with longer inputs.The authors therefore argue for considering cqtspec features and unabridged input audio in future architectures.
- Conclusion: Many models’ in-the-wild generalization capabilities may have been overestimated: performance drops sharply, with some models degenerating to random guessing.The authors suggest that solutions may be tailored too closely to ASVspoof and that deepfakes are harder to detect outside the lab.