PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey on Speech Deepfake Detection

Menglu Li, Yasaman Ahmadiadli, Xiao-Ping Zhang

arXiv:2404.13914v2cs.SDcs.CRcs.MMeess.AS

TL;DR

Speech Deepfakes can generate realistic voices and spread misinformation, while existing surveys often focus on model architecture rather than the broader detection pipeline. This survey synthesizes more than 200 papers across detection components and application areas; its discussion includes performance patterns for speaker-aware and partially fake speech detection.

  • Problem

    Speech Deepfakes can generate realistic voices and spread misinformation, while many existing surveys focus on model architectures without covering the broader detection pipeline.

    Source: §1 · p. 1, §8 · p. 28

  • Method

    The survey analyzes more than 200 papers and reviews architectures, training techniques, datasets, metrics, generalizability, and open-source availability.

    Source: §1 · p. 3, §8 · p. 28

  • Results

    For speaker-aware detection, simple score-fusion and cascaded systems achieve better EER performance than a single integrated SASV system, while temporal information is emphasized in partially fake detection.

    Source: §5.2 · p. 24, §6.2 · p. 26

  • Takeaways & Limitations

    The survey provides open-source information on models and benchmark datasets as a guide to reproduction and proposes future research directions.

    Source: §1 · p. 3

    Detection models trained on vocoder-based speech may struggle to generalize to evolving neural-codec Deepfakes, motivating more diverse training data.

    Source: §7 · p. 27

Abstract

from arXiv · show

The availability of smart devices leads to an exponential increase in multimedia content. However, advancements in deep learning have also enabled the creation of highly sophisticated Deepfake content, including speech Deepfakes, which pose a serious threat by generating realistic voices and spreading misinformation. To combat this, numerous challenges have been organized to advance speech Deepfake detection techniques. In this survey, we systematically analyze more than 200 papers published up to March 2024. We provide a comprehensive review of each component in the detection pipeline, including model architectures, optimization techniques, generalizability, evaluation metrics, performance comparisons, available datasets, and open source availability. For each aspect, we assess recent progress and discuss ongoing challenges. In addition, we explore emerging topics such as partial Deepfake detection, cross-dataset evaluation, and defences against adversarial attacks, while suggesting promising research directions. This survey not only identifies the current state of the art to establish strong baselines for future experiments but also offers clear guidance for researchers aiming to enhance speech Deepfake detection systems.

1 INTRODUCTION

Speech Deepfakes can generate realistic voices that spread misinformation, while much prior Deepfake research and existing surveys emphasize visual media or narrower speech-detection topics. This survey reviews the speech-detection pipeline broadly, including partial Deepfakes, training optimization, datasets, and emerging research topics.

1 INTRODUCTION

  • Speech Deepfakes can manipulate speech or create synthetic voices, with potential to spread misinformation and mislead voters.The survey distinguishes these threats from other Deepfake modalities and focuses on speech detection.Source: §1 · p. 1
  • The survey focuses on TTS- and VC-generated speech attacks, excludes physical attacks, and extends beyond ASV systems.The authors distinguish replay attacks, which use physical recording and playback, from digitally altered or synthesized speech.Source: §1 · p. 2
  • The authors provide open-source information for models and datasets to support reproduction and discuss emerging topics including cross-dataset evaluation and adversarial-attack defence.They also identify challenges facing current models and propose future research directions.Source: §1 · p. 3
  • Earlier surveys often emphasize image and video, open challenges, or fully fake TTS/VC clips, with less systematic treatment of partial detection and training optimization.The survey identifies partial-Deepfake algorithms and optimization techniques as areas not systematically reviewed in prior work.Source: §1 · p. 3
  • The survey reviews detection architectures and training techniques across fully and partially Deepfake scenarios, including data augmentation, activation functions, and loss functions.It also addresses transfer learning and explainability as emerging research tasks.Source: §1 · p. 3

2 DATASETS AND EVALUATION METRICS FOR SPEECH DEEPFAKE DETECTION

The survey organizes speech Deepfake datasets into fully and partially fake speech and reviews metrics for detection and speaker-aware evaluation. It discusses dataset scope and characteristics, and metrics that address system costs or segment-level detection.

2.1 Datasets

  • The survey groups datasets into fully and partially Deepfake speech and presents their details in Table 1.The collection includes datasets for assessing generalization and varied synthesis conditions.Source: §2.1 · p. 3, Table 1 · p. 5

2.1.1 Fully Deepfake speech datasets.

  • Fully fake datasets vary in language, synthesis methods, and evaluation conditions; MLAAD spans 54 TTS models across 23 languages.ITW is designed to assess generalizability, while ADD2022-LF is inaccessible online.Source: §2.1.1 · pp. 4–5

2.1.2 Partially Deepfake speech datasets.

  • Partial-Deepfake datasets include utterances with replaced segments or words, and some provide fine-grained labels or manipulated-region localization.PartialSpoof supplies segment-level annotations, while ADD2023-PF focuses on locating manipulated regions.Source: §2.1.2 · p. 6

2.2 Evaluation metrics

  • EER is widely used for countermeasures, t-DCF evaluates cascaded ASV and countermeasure systems, and a-DCF supports architectures beyond cascaded systems.Range-based EER evaluates partially fake speech by measuring misclassified-region duration between reference and hypothesis, avoiding fixed-resolution segmentation.Source: §2.2 · pp. 6–7

3 FULLY DEEPFAKE DETECTION

This section reviews key components of speech Deepfake detection, including model architectures, training optimizations, and non-machine-learning methods. It contrasts feature-extraction pipelines with end-to-end designs and examines training choices for their detection-performance effects.

3 FULLY DEEPFAKE DETECTION

  • Most recent detection models use front-end feature extraction and back-end classification, while end-to-end architectures avoid information loss from predefined features.The section compares these architecture approaches as parts of the detection pipeline.Source: §3 · p. 7
  • The section assesses data augmentation, loss functions, and activation functions for their effectiveness in improving detection performance.Source: §3 · p. 7
  • The review also covers non-machine-learning detection methods and their distinctive approaches and advantages.Source: §3 · p. 7

3.1 Feature Engineering

The survey groups speech Deepfake features into hand-crafted spectral methods, deep-learning features, and other analysis-oriented approaches. It describes a shift toward pretrained SSL embeddings while retaining hand-crafted features for resource-limited or data-limited settings.

3.1.1 Hand-crafted spectral features.

  • Hand-crafted front ends span short-term magnitude coefficients, long-window CQT variants, phase and bispectral cues, and spectrogram representations often passed to CNN classifiers.These approaches capture different spectral, temporal, and phase characteristics of speech.Source: §3.1.1 · pp. 8–10

3.1.2 Deep-learning (DL) features.

  • Pretrained SSL embeddings improve detection over hand-crafted or SincNet features with matched back ends and augmentation, while multilingual models perform better on non-English Deepfakes than English-only SSL models.The survey also covers learnable filter-based features and deep embeddings derived from hand-crafted inputs.Source: §3.1.2 · pp. 10–11

3.1.3 Other analysis-oriented features.

  • Silence duration and content provide cues for TTS and VC detection, while low-frequency sub-bands can outperform full-band features under channel effects, codecs, and noise.Silence-related findings come from clean data; voiced-segment differences are concentrated at 0–4 kHz, whereas silence and unvoiced cues often lie at 4–8 kHz.Source: §3.1.3 · p. 11
  • Pooling before a detector can outperform fixed-length trimming or padding, and alternative features include pause energy, compressed-bitstream metadata, face embeddings, and dual-channel audio.These approaches address varied input lengths and seek cues beyond conventional speech features.Source: §3.1.3 · p. 12

3.1.4 Performance discussion on feature selection.

  • Feature engineering is shifting toward learnable embeddings, especially multilingual SSL, for cross-dataset and multilingual detection, but hand-crafted features remain useful when resources or training data are limited.Hand-crafted features are more interpretable and computationally lighter; the survey also recommends combining feature types.Source: §3.1.4 · p. 12

3.2 Classifier Architecture

The survey reviews classifier architectures ranging from traditional machine-learning baselines to deep-learning models, describing their structures, strengths, and limitations. It also discusses combining architectures to complement their different capabilities.

3.2 Classifier Architecture

  • Speech Deepfake classifiers have progressed from SVM, GMM, and random-forest baselines to CNNs, ResNets, GNNs, Transformers, and other deep architectures.GMM classifiers serve as a fundamental ASVspoof challenge baseline.Source: §3.2 · p. 12, §3.2.1 · p. 13

3.2.2 Convolutional Neural Network (CNN).

  • CNN variants use MFM for feature selection and smaller models, frequency-split processing to retain band-specific cues, or capsule routing to model hierarchical artifacts.ResNet variants add channel recalibration, MFM activations, or dense skip connections; DenseNets have fewer parameters than conventional ResNets.Source: §3.2.2 · p. 13, §3.2.3 · p. 13

3.2.3 Residual Network (ResNet).

  • Res2Net divides feature maps into channel segments and adds prior-segment convolution outputs to create multi-scale representations.Its bottleneck uses hierarchical residual-like connections.Source: §3.2.3 · p. 13

3.2.4 Graph Neural Network (GNN).

  • Graph detectors model time-frequency relationships through GCN or GAT aggregation; adding pre-emphasis and LayerNorm to RawGAT yields a reported 51% improvement on ASVspoof2019-LA.AASIST integrates spectral and temporal sub-graphs with heterogeneity-aware aggregation.Source: §3.2.4 · pp. 13–14

3.2.5 Transformer.

  • Transformer approaches combine attention with convolutional or ResNet features, while some adapt attention across temporal and frequency dimensions.CCT uses convolutions before Transformer encoders, OCT reduces overfitting with 1D convolutions and fewer encoders, and Rawformer combines SE-Res2Net with positional aggregation.Source: §3.2.5 · p. 14

3.2.6 Time-Delay Neural Network (TDNN).

  • TDNN variants such as ECAPA-TDNN are used for speech Deepfake detection, but their countermeasure performance remains below their effectiveness in speaker verification.Source: §3.2.6 · p. 14

3.2.7 Differentiable Architecture Search (DART).

  • DART uses validation performance to optimize network architecture and parameters; PC-DARTS reduces training time by 50% versus standard DARTS through random masking of partial channels.Light-DARTS includes MFM as a candidate operation, and a separate approach uses a Bi-LSTM embedding with a 4-qubit variational quantum classifier.Source: §3.2.7 · p. 15, §3.2.8 · p. 15

3.2.9 Pooling and Attention mechanism.

  • Pooling and attention emphasize informative frames or channel and spatial cues, while ensembling can pair CNN local-pattern strengths with Transformer long-range modeling.Deep-learning classifiers may overfit when training data are small or lack diversity.Source: §3.2.9 · p. 15, §3.2.10 · p. 15

3.3 End-to-End (E2E) Architecture

Two-stage detectors depend heavily on extracted features, and information lost during extraction may be unrecoverable. E2E approaches address separate feature-processing and classifier stages, though some models labeled E2E still rely on fixed spectral settings.

3.3 End-to-End (E2E) Architecture

  • Two-stage detectors depend on selected input features, and information lost during extraction—such as phase discarded by power-spectrum features—may be irretrievable.This dependence can cause classifier performance to vary with feature choice.Source: §3.3 · pp. 15–16
  • E2E models place the detection process in one network, removing separate front-end feature processing and back-end classifier engineering.The section notes that some nominally E2E models use learnable SincNet filters with fixed cutoff frequencies and Mel scales; learnable cutoffs may overfit, while predetermined settings limit how representative they are of E2E models.Source: §3.3 · p. 16
  • Genuinely E2E designs obtain embeddings directly from raw waveforms using convolutional architectures, including ResNet and Res2Net variants.Examples add skip-connection convolutions and batch normalization, larger first-layer kernels, genuinization-transformer layers, or channel attention.Source: §3.3 · p. 16

3.4 Training Optimization Techniques

The survey reviews data augmentation, loss functions, and activation functions as training optimization techniques for speech Deepfake detection. It highlights augmentation's dependence on features and datasets, and discusses promising augmentation and loss-function choices.

3.4.1 Data Augmentation (DA) techniques.

  • SpecMix replaces masked frequency or temporal regions with features from another sample and weights the new label by the masked-area proportions; omitting it or applying it uniformly can degrade performance.The cited experiments caution that the proportion of SpecMix used across the training set matters.Source: §3.4.1 · pp. 16–17
  • RawBoost combines convolutive effects and additive noise to improve reliability, particularly on ASVspoof2021-LA and under unknown encoding and transmission conditions.Codec augmentation simulates multimedia or transmission encoding, including codec changes, packet loss, and codec-mimicking filters.Source: §3.4.1 · p. 17
  • Noise addition and codec augmentation are identified as the most promising data-augmentation techniques for SOTA methods because they simulate real-world conditions.The discussion particularly notes unknown encoding and transmission scenarios.Source: §3.4.1 · p. 17
  • Augmentation effectiveness can depend on features or datasets; channel simulation helps wav2vec 2.0 models more than LFCC-based models, motivating adaptive strategies and evaluation across datasets and attacks.The survey identifies dynamically adjusting augmentation to model performance or dataset characteristics as a future research direction.Source: §3.4.1 · p. 18

3.4.2 Loss function.

  • CE with Softmax is the common objective, while margin-based losses extend it to encourage compact within-class embeddings and greater separation between classes.AM-Softmax places a margin in angular space; LMCL places it in cosine space, and OC-Softmax uses distinct margins for bona fide and Deepfake speech. One-class variants include speaker-specific bona fide clusters, dispersion loss, and thresholding for class imbalance.Source: §3.4.2 · pp. 18–19
  • MSE offers a hyperparameter-free alternative, focal loss addresses training imbalance, and Center loss minimizes embedding variability; PReLU and ARelu improved performance over non-learnable activations in tested E2E models.Center loss is combined with CE to complement its focus on inter-class differences.Source: §3.4.2 · p. 19, §3.4.3 · p. 19
  • OC-Softmax is highlighted for detection performance and generalization to unknown attacks, while hybrid losses such as Focal Loss with OC-Softmax may address underrepresented attack types.The survey attributes OC-Softmax's benefit to distinct bona fide and Deepfake margins.Source: §3.4.2 · p. 19

3.5 Non-Machine Learning-Based Detection Models

Non-machine-learning approaches use vocal-tract cues or speech-to-text transcription differences to detect Deepfakes. Their interpretability is a stated advantage, but their generalizability has not been rigorously established.

3.5 Non-Machine Learning-Based Detection Models

  • Vocal-tract reconstruction detects artifacts through frequency amplitudes at phoneme transitions, while Voicefox uses lower transcription accuracy for Deepfake speech as a cue.The vocal-tract approach showed high accuracy on Tacotron2-generated Deepfakes.Source: §3.5 · p. 20
  • These methods offer interpretable detection cues, but they have not been rigorously evaluated on diverse datasets and their cues' robustness across generation techniques and scenarios is unproven.The vocal-tract anomalies are grounded in anatomical knowledge, unlike the often-black-box operation of DL models.Source: §3.5 · p. 20

4 TRAINING AND ROBUSTNESS ADVANCEMENTS IN FULLY DEEPFAKE DETECTION

The survey covers training strategies, interpretability, adversarial defenses, attacker-source tracing, and cross-dataset robustness in fully Deepfake detection. It describes varied approaches while noting substantial challenges in generalization across datasets and attacks.

4.1 Training strategies

  • Training strategies include Siamese networks for compact representations without added model parameters, LoRA for task adaptation, and knowledge distillation to transfer knowledge to smaller models.The survey describes self-, offline, and online distillation variants.Source: §4.1 · p. 20

4.2 Interpretability of results

  • Interpretability work uses SHAP, Deep Taylor, LRP, Grad-CAM, and class activation maps to examine detector attributions and frequency regions.On ASVspoof2019-LA, SHAP analysis found attention to non-speech intervals and the 0.5–0.6 kHz low-frequency sub-band.Source: §4.2 · p. 21

4.3 Defense to adversarial attacks

  • Defense strategies include adversarial training with PGD- or FGSM-generated examples, frequency filtering and denoising, and knowledge distillation.The cited work proposes filtering and denoising against attacks focused beyond human perception; distillation is described as capturing more nuanced decision-boundary information.Source: §4.3 · p. 21

4.4 Attacker source tracing

  • Attacker-source tracing uses signal features and embeddings; neural embeddings, especially with RNNs, outperformed low-level features in clustering and classification tasks.Other approaches identify vocoder fingerprints or use embedding distances and class-centroid similarity for out-of-distribution source detection.Source: §4.4 · p. 21

4.5 Robustness on cross-dataset

  • Combining domains for co-training does not guarantee better generalization, while continual-learning and data-selection methods address domain mismatch and forgetting.Cross-dataset evaluation tests a model trained on one dataset on out-of-domain datasets to assess robustness and generalizability.Source: §4.5 · pp. 21–22
  • Models trained on ASVspoof2019-LA can show substantial performance decline, even random-guessing behavior, on ITW; the survey reports that knowledge distillation significantly improves student-model generalizability in cross-dataset evaluation.The discussion also describes a channel classifier with a Gradient Reversal Layer and a segment-level reliability estimator as approaches to the cross-dataset problem.Source: §4.5 · p. 22

5 INTEGRATION OF ASV TO DEEPFAKE COUNTERMEASURES

The survey examines integrating speaker verification (ASV) with speech Deepfake countermeasures (CMs) and reviews SASV system designs. It finds that simple fusion or cascaded systems outperform single integrated systems on EER, motivating joint optimization and new integrated designs.

5 INTEGRATION OF ASV TO DEEPFAKE COUNTERMEASURES

  • SASV combines detection of speech Deepfakes with speaker identity verification, using SASV-EER as its primary challenge metric.The challenge uses VoxCeleb2 and ASVspoof2019-LA for training; SV-EER and SPF-EER measure speaker verification and countermeasure performance, respectively.Source: §5 · pp. 22–23

5.1 SOTA for SASV models

  • SASV approaches include cascaded ASV-then-CM classifiers, score-level and embedding-level fusion, joint optimization, and integrated or end-to-end systems.In cascaded systems, only test speech labeled positive by the ASV classifier proceeds to the CM module; module order affects performance.Source: §5.1 · pp. 23–24

5.2 Performance discussion on SASV models

  • Simple score-fusion and cascaded systems outperform single integrated SASV systems on EER, while embedding concatenation reduces speaker-verification ability despite fewer false alarms than cascaded systems.The survey suggests a new latent space may be needed to represent speaker and Deepfake information effectively.Source: §5.2 · p. 24
  • The survey recommends joint optimization and developing integrated SASV systems to improve overall performance.This recommendation follows the reported performance comparisons among current SASV approaches.Source: §5.2 · pp. 24–25

6 PARTIALLY DEEPFAKE DETECTION

Partial Deepfakes alter one or more portions of otherwise bona fide speech, and detection research addresses frame labeling, multitask learning, and boundary localization. The survey highlights temporal modeling and SSL features, while noting that speaker-verification methods remain difficult to assess on current test datasets.

6 PARTIALLY DEEPFAKE DETECTION

  • Partial Deepfakes insert synthesized or voice-converted speech into bona fide utterances, and challenge tasks extend from detecting manipulated intervals to locating their boundaries.The survey distinguishes these technical manipulations from half-truths, which alter context rather than speech content.Source: §6 · p. 25

6.1 SOTA models for partially Deepfake detection

  • Detection approaches include frame-level labeling, joint frame- and utterance-level learning, and boundary detection to locate transitions between genuine and fake segments.Boundary-detection systems placed second in ADD 2022 PF and first in ADD 2023 PF; frame-level methods can struggle with isolated fake frames.Source: §6.1 · pp. 25–26

6.2 Performance discussion on partially Deepfake detection

  • RNNs are preferred for modeling temporal artifacts, and fine-tuned SSL features are widely used to capture temporal characteristics in partial-fake detection.Testing datasets do not yet include manipulated real segments from different speakers, limiting assessment of supplementary speaker-verification methods.Source: §6.2 · p. 26

7 CURRENT CHALLENGE AND FUTURE WORK

The survey identifies challenges in reproducibility, generalization, interpretability, robustness, multilingual coverage, and real-time deployment. It proposes research directions including broader datasets, improved defenses, efficient detection, privacy-preserving techniques, and multimodal integration.

Reproducibility of detection models

  • Only about 10% of published speech Deepfake detection papers provide source code, and missing training details such as loss functions and hyperparameters also hinder reproducibility.The survey argues that transparency supports building on prior work and checking errors, inconsistencies, or biases.Source: §7 · p. 27

Cross-dataset generalization; Interpretability of detection results

  • Cross-dataset generalization and interpretability remain challenges; transfer learning, domain adaptation, attention, and feature visualization are discussed as possible directions.The survey notes a challenge in balancing detection performance with interpretability during architecture design.Source: §7 · p. 27

Robustness to multilingual with multiple accents

  • Multilingual detection remains early-stage: most research focuses on English, only one dataset includes more than two languages, and cross-lingual robustness is rarely evaluated.The survey calls for larger multilingual datasets and exploration of transfer learning or domain adaptation across languages and accents.Source: §7 · p. 27

Robustness to adversarial attacks; Robustness to neural codec-based Deepfake speech

  • Adversarial defenses often rely on computationally demanding adversarial training, while codec-based Deepfakes may challenge detectors trained on vocoder-based techniques.The survey suggests domain-invariant attack learning, reducing reliance on signal regions, and more diverse codec-Deepfake training data as directions.Source: §7 · p. 27

Streaming / Real-time detection; Privacy preservation

  • Real-time detection research is limited, and deployment on smart devices could raise privacy concerns involving biometric data and model parameters.The survey names pruning and distributed computing for efficiency, and secure multiparty computation for restricting access.Source: §7 · p. 28

Integration of multi-modal Deepfake detection

  • The survey identifies integration of speech detection with image and video Deepfake detection as an emerging direction, including detecting speech-video synchronization issues.It also suggests adapting image and video techniques to identify anomalies in speech spectrograms.Source: §7 · p. 28

8 CONCLUSION

The survey reviews the speech Deepfake detection pipeline, comparing methods and examining performance and open-source information. It summarizes current challenges and proposes future research directions, including integrating speech detection into multimodal systems.

8 CONCLUSION

  • The survey covers speech Deepfake model architectures, training techniques, application scenarios, and available datasets across the detection pipeline.It presents comparisons of feature engineering and classifier design across applications.Source: §8 · p. 28
  • The survey assesses training optimization techniques, including data augmentation, activation functions, and loss functions, alongside state-of-the-art performance and open-source information.The authors intend these evaluations to help establish strong baselines for future experiments.Source: §8 · p. 28
  • The authors identify current challenges and future directions, and encourage integrating speech Deepfake detection into comprehensive multimodal detection systems.The proposed broader context is intended to address the full range of Deepfake threats.Source: §8 · p. 28
Loading 2404.13914v2…