PaperlayerView exact revision on arXiv

Source-linked AI summary

Audio Deepfake Detection: A Survey

Jiangyan Yi, Chenglong Wang, Jianhua Tao, Xiaohui Zhang, Chu Yuan Zhang, Yan Zhao

arXiv:2308.14970v1cs.SDeess.AS

TL;DR

Audio deepfake detection must distinguish increasingly realistic AI-manipulated audio, but prior research lacks a comprehensive survey with consistent analysis. This paper synthesizes audio types, competitions, datasets, features, classifiers, and evaluation, and compares representative methods across multiple datasets. The survey finds substantial generalization weaknesses and identifies large-scale in-the-wild data and interpretable detection as priorities.

  • Problem

    Prior audio deepfake detection reviews are scattered, and some collect classic-method results without consistent experimental analysis.

  • Method

    The paper systematically surveys audio deepfake types, competitions, datasets, features, classifiers, evaluation, end-to-end methods, and generalization approaches, then performs unified method comparisons.

  • Results

    Out-of-domain evaluation raises EER by 2%–52%, while pretrained-model and concatenated features are more consistent than handcrafted features across distribution shifts.

  • Takeaways & Limitations

    The survey supports developing robust and general detection algorithms evaluated on reliable samples for real-world audio deepfake detection.

  • Takeaways & Limitations

    Detection performance remains poor on mismatching datasets containing unseen attacks, acoustic conditions, or languages, motivating further robustness research.

Abstract

from arXiv · show

Audio deepfake detection is an emerging active topic. A growing number of literatures have aimed to study deepfake detection algorithms and achieved effective performance, the problem of which is far from being solved. Although there are some review literatures, there has been no comprehensive survey that provides researchers with a systematic overview of these developments with a unified evaluation. Accordingly, in this survey paper, we first highlight the key differences across various types of deepfake audio, then outline and analyse competitions, datasets, features, classifications, and evaluation of state-of-the-art approaches. For each aspect, the basic techniques, advanced developments and major challenges are discussed. In addition, we perform a unified comparison of representative features and classifiers on ASVspoof 2021, ADD 2023 and In-the-Wild datasets for audio deepfake detection, respectively. The survey shows that future research should address the lack of large scale datasets in the wild, poor generalization of existing detection methods to unknown fake attacks, as well as interpretability of detection results.

1 INTRODUCTION

Audio deepfake detection addresses increasingly realistic AI-generated or altered audio that can threaten social security and political economy. This survey organizes scattered prior work into a systematic overview of audio types, detection solutions, datasets, and evaluation.

  • Deepfake technologies can generate human-like speech that is difficult to distinguish from real audio, creating security and economic risks when misused.The introduction cites a fraudulent transfer induced by an AI-mimicked executive voice.
  • Audio deepfake detection is studied through pipeline systems and end-to-end detectors that jointly learn features and classification from raw waveforms.Pipeline systems use a front-end feature extractor and back-end classifier, whereas end-to-end methods optimize both jointly.
  • Deepfake audio includes text-to-speech, voice conversion, emotion fake, scene fake, and partially fake audio, whose characteristics differ across manipulation types.
  • Prior reviews are scattered and may collect classic-method results without consistent experimental analysis, leaving no comprehensive unified overview.
  • The survey addresses this gap by systematically reviewing discriminative features, computing methodologies, datasets, and unified comparisons of representative detection methods.It also organizes competitions, evaluation metrics, end-to-end methods, generalization methods, and future challenges across the paper.

2 OVERVIEW

The survey organizes audio deepfake detection around deepfake types, competitions, datasets, and evaluation metrics. It highlights expanding task complexity alongside persistent challenges in dataset diversity and detection generalization.

  • 2.1 Types of Deepfake Audio: Audio deepfakes include TTS, VC, emotion fake, scene fake, and partially fake audio, each manipulating different attributes while retaining perceived naturalness.TTS synthesizes speech from text, VC changes speaker timbre and prosody, emotion fake changes emotion, scene fake changes acoustic context, and partially fake alters selected words.
  • 2.2 Competitions: Competitions evolved from spoofing-focused ASV protection toward tasks covering low-quality, partial, game-based, localization, and source-identification challenges.ASVspoof includes logical access, physical access, and speech deepfake tasks, while ADD introduces broader real-world and manipulation-focused settings.
  • 2.2 Competitions: ADD 2022 results show that one model struggles across all fake types and that detection generalization remains unresolved.Its tasks include low-quality fake, partially fake, and audio fake game detection.
  • 2.3 Benchmark Datasets: Benchmark development progressed from proprietary or single-attack spoofing datasets toward public datasets containing multiple deepfake types and varied acoustic conditions.Representative datasets include TTS, VC, emotion, scene, and partially fake audio, but earlier attack diversity was insufficient for generalized studies.
  • 2.4 Evaluation Metrics: EER measures the threshold where false-alarm and miss rates are equal, while ADD final rankings combine round-specific EER values through weighted EER.The weighted metric uses EER from the first and second evaluation rounds with corresponding weights α and β.

3 DISCRIMINATIVE FEATURES

Feature extraction is a central pipeline component that captures discriminative artifacts from speech signals for audio deepfake detection.

  • 3 DISCRIMINATIVE FEATURES: Feature extraction learns discriminative representations by capturing fake-audio artifacts from speech signals and is a key module of pipeline detectors.The survey groups prior features into short-term spectral, long-term spectral, prosodic, and deep categories.

3.1 Short-term Spectral Features

Short-term spectral features characterize speech within short frames using magnitude- and phase-related representations. Their usefulness comes from spectral artifacts, but frame-wise processing limits temporal modeling.

  • 3.1 Short-term Spectral Features: The STFT represents each speech frame through magnitude and phase spectra, with power spectrum equal to the squared magnitude spectrum.Magnitude and phase support distinct feature families used in fake-audio detection.
  • 3.1.1 Short-term magnitude based features: Magnitude-based features include magnitude spectrum, power spectrum, log power spectrum, cepstrum, FBCC, APCC, and subband spectral features.Magnitude-spectrum processing can expose artifacts such as over-smoothed spectral envelopes introduced by parametric modeling.
  • 3.1.2 Short-term phase based features: Phase-based features exploit artifacts from simplified minimum-phase models, but post-processing is needed because raw phase patterns are unstable under phase warping.Examples include GD, MGD, MGDCC, APGD, IF, BPD, RPS, PSP, and CosPhase features.

3.2 Long-term Spectral Features

Long-term spectral features address the temporal limitations of short-term frame-wise analysis by modeling long-range speech information through several time-frequency transforms.

  • 3.2 Long-term Spectral Features: Long-term spectral features capture speech information over extended ranges because short-term frame-wise features miss temporal trajectories.The survey describes them as important for fake-speech detection.
  • 3.2.1 STFT based features: STFT-based long-term features include modulation spectrum, global modulation, shifted delta coefficients, FDLP, and local binary patterns.Modulation and SDC features model temporal information across multiple speech frames.
  • 3.2.2 CQT based features: CQT provides higher frequency resolution at low frequencies and higher temporal resolution at high frequencies than STFT.CQT-derived representations include CQT spectrum and CQCC.
  • 3.2.2 CQT based features: CQCC is derived by applying DCT to the CQT log power magnitude spectrum and has been used for detecting unit-selection TTS attacks.CQT spectrum is obtained by logarithmically transforming the raw CQT power magnitude spectrum.
  • 3.2 Long-term Spectral Features: Other long-term representations use Hilbert-transform, wavelet-transform, and cochlear-filter analyses, including MHEC, MWPC, CFCC, and CFCCIF.CFCCIF augments cochlear features with instantaneous frequency, whose phase information is described as useful.

3.3 Prosodic Features

Prosodic features capture longer-term speech characteristics such as pitch, duration, energy, speaking rate, and rhythm, complementing short-term spectral information. Their use is promising but constrained by unreliable pitch extraction in noise and substantial training-data requirements.

  • Prosody captures non-segmental speech information across phones, syllables, words, and utterances, including stress, intonation, rate, rhythm, F0, duration, and energy.
  • Pitch patterns differ between synthetic and natural speech because TTS and VC systems struggle to model human physiological and co-articulatory characteristics.
  • F0 complements magnitude- and phase-based features, while combining F0, phoneme duration, and energy has also been explored for fake-audio detection.
  • Prosodic features remain difficult to deploy because pitch extraction is unreliable in noisy environments and their sparsity demands relatively large training datasets.

3.4 Deep Features

Deep features address representation biases in hand-crafted spectral and prosodic features through learnable spectral representations, supervised embeddings, and self-supervised embeddings. Self-supervised models achieve strong reported results and can improve generalization, but supervised alternatives require abundant labeled data.

  • Deep features are motivated by biases in hand-crafted representations and comprise learnable spectral, supervised embedding, and self-supervised embedding features.
  • 3.4.1 Learnable spectral features: Learnable spectral methods train filterbanks partially from spectrograms or fully from raw waveforms, with learned FBCC especially effective for unseen attacks.Partially learnable methods operate on STFT spectrograms, whereas fully learnable methods approximate filtering directly from raw waveforms.
  • 3.4.2 Supervised embedding features: Supervised embeddings include spoof, emotion, speaker, and pronunciation representations learned from labeled tasks and used directly or alongside detection features.
  • 3.4.2 Supervised embedding features: Supervised embeddings can generalize to unknown conditions, but obtaining their required annotated speech and fake-utterance data is costly and technically demanding.
  • 3.4.3 Self-supervised embedding features: Wav2vec2.0 features achieved state-of-the-art reported results on ASVspoof 2021 LA and Deepfake datasets despite pretraining only on genuine speech.
  • 3.4.3 Self-supervised embedding features: XLS-R features ranked first in ADD 2022’s LF track and achieved the best PF-task results, including generalization to unknown partially fake voices.
  • 3.4.3 Self-supervised embedding features: Without target-data fine-tuning, antispoofing classifiers need depth, whereas fine-tuning permits average temporal pooling with a linear layer.

4 CLASSIFICATION ALGORITHMS

Audio deepfake classification uses traditional or deep-learning back ends to discriminate front-end features. Deep models generally outperform SVM and GMM baselines, while newer architectures target training stability, unseen attacks, channel relationships, efficiency, and partial-fake localization.

  • 4.1 Traditional Classification: Traditional back ends include LR, PLDA, RF, GBDT, ELM, KNN, SVM, and GMM, with SVM and GMM most widely used.
  • 4.1.2 GMM based classifiers: GMM serves as a generative baseline across ASVspoof 2017, 2019, and 2021 and ADD 2022 competitions.
  • 4.2 Deep Learning Classification: Deep-learning classifiers significantly outperform SVM and GMM classifiers, with CNN, ResNet, Res2Net, SENet, GNN, DARTS, and Transformer architectures represented.
  • 4.2.2 ResNet based classifiers: ResNet eases training of deeper networks through residual mapping, while variants target performance, parameter reduction, receptive fields, and unseen-attack generalization.
  • 4.2.3–4.2.6 Specialized classifiers: SENet models channel interdependencies, GNNs model relationships among temporal segments or spectral sub-bands, and DARTS automatically optimizes architecture operations.
  • 4.2.7 Transformer based classifiers: Transformer and ResNet-1D classifiers detect partially fake audio while locating fake regions and modeling local and global artifacts.

5 END-TO-END MODELS

End-to-end detectors jointly learn feature extraction and classification directly from raw waveforms, using CNN, RawNet, residual, graph, architecture-search, and Transformer designs. These models can achieve strong performance, but generalization remains sensitive to unseen telephony and cross-dataset conditions.

  • End-to-end methods jointly optimize feature extraction and classification from raw audio, avoiding a separately designed hand-crafted or learnable feature front end.
  • 5.1 CNN based models: CNN-based end-to-end models include simple CNN-MLP and raw-waveform CLDNN systems, with reported performance for VC and TTS attacks.
  • 5.2 RawNet2 based models: TO-RawNet improves on RawNet2 by using orthogonal convolution to reduce correlation between sinc-convolution filters, and its models observably outperform RawNet2-based models.
  • 5.4 GNN based models: RawGAT-ST models spectro-temporal relationships and outperforms RawNet2 and Res-TSSDNet on the ASVspoof 2019 LA evaluation set.
  • 5.4 GNN based models: AASIST uses heterogeneous attention to model artifacts across temporal and spectral segments, outperforming prior end-to-end models, while AASIST-L provides a lightweight variant.
  • End-to-end systems perform reliably under seen encoding and transmission conditions but unreliably in unknown telephony scenarios.
  • 5.7 Transformer based models: Rawformer generalizes better than AASIST on cross-dataset evaluation, and SE-Rawformer further improves Rawformer with squeeze-and-excitation operations.

6 GENERALIZATION METHODS

Audio deepfake detectors generalize poorly to unknown or out-of-domain attacks, motivating methods that learn more robust representations and adapt incrementally without forgetting earlier attacks.

  • Out-of-domain performance drops sharply despite impressive in-domain results, showing that generalization to unknown attacks remains a central challenge.
  • LMCL with online frequency masking improves robustness by encouraging more robust feature embeddings and augmenting training data.
  • Continual learning incrementally incorporates new fake attacks while aiming to preserve knowledge of previously learned attacks without accessing old data.
  • RAWM relaxes DFWF’s regularization constraint and modifies the adaptation direction, outperforming most typical continual-learning methods for audio deepfake detection.

7 PERFORMANCE COMPARISONS

The survey compares competition systems, features, and classifiers across in-domain and out-of-domain datasets, finding that strong benchmark performance does not ensure robust real-world generalization.

  • 7.1 Top-performing methods in typical competitions: Top-performing competition systems consistently outperform baselines and commonly combine augmentation, spectral or raw-waveform inputs, ResNet variants, and ensemble fusion.
  • 7.2 Evaluation of Features: EER increases by 2%–52% in out-of-domain evaluations, while pretrained and concatenated features are more consistent than many handcrafted features.
  • 7.2 Evaluation of Features: XLS-R, concatenated features, and LFCC achieve consistent and competitive performance across in-domain and out-of-domain evaluations.
  • 7.3 Evaluation of Classifiers: Classifier performance is relatively stable at EER of 14%–15% in some settings but degrades substantially when training and testing distributions differ.
  • 7.3 Evaluation of Classifiers: ADD 2023-trained models perform better on In-the-Wild data, possibly because differences in audio quality and perturbations create greater distribution differences.

8 FUTURE DIRECTIONS

Future work should improve dataset realism and linguistic coverage, model generalization to unseen attacks and conditions, adaptation to rapidly evolving fakes, interpretability, and evaluation metrics.

  • Collecting audio datasets in the wild: Most existing datasets are not collected in the wild, motivating realistic recordings across varied channels and conditions for practical assessment.Suggested sources include social media, the Internet, and telephone channels.
  • Designing large-scale multilingual datasets: Large-scale multilingual datasets are needed because predominantly English training data may make detection methods language dependent.The survey calls for cross-language and code-switching evaluation in realistic settings.
  • Improving generalization ability and robustness of detection models: Detection models generalize poorly to unseen fake attacks, acoustic conditions, and languages despite high performance in ASVspoof and ADD competitions.Effective loss functions and continual learning have been explored, but substantial room for improvement remains.
  • Dealing with rapid development of deepfake technologies: Rapidly improving generation technologies produce increasingly realistic audio that challenges current detectors and motivates stronger approaches for unseen attacks.ADD 2022 and 2023 used generation–detection rivalry games, though the survey describes these games as simple and insufficiently intelligent.
  • Improving the interpretability of detection results: Beyond binary real/fake decisions, interpretable systems should localize manipulated intervals and identify the tools or methods responsible for fake audio.These capabilities are relevant to audio forensics and attribution.
  • Exploring more reasonable evaluation metrics: Future evaluation should assess whether EER is appropriate while considering human detection capabilities and differences between human and machine detection.

9 CONCLUSIONS

The survey reviews audio deepfake detection research, compares existing state-of-the-art methods, and identifies challenges for developing robust, general algorithms applicable to real situations.

  • 9 CONCLUSIONS: The survey reviews current audio deepfake detection research, compares state-of-the-art methods, and highlights outstanding issues for future work.
Loading 2308.14970v1…