Source-linked AI summary
Audio Deepfake Detection with Self-Supervised WavLM and Multi-Fusion Attentive Classifier
Yinlin Guo, Haofan Huang, Xi Chen, He Zhao, Yuehai Wang
TL;DR
Audio deepfakes threaten ASV because spoofed speech can be difficult to distinguish from genuine samples. The paper combines WavLM features with an MFA classifier, reporting the best performance on ASVspoof 2021 DF and competitive results on LA and ASVspoof 2019.
Problem
Speech synthesis and voice conversion produce spoofed samples that are difficult to distinguish from real speech and can challenge ASV systems.
Source: §1 · p. 1Method
The method uses WavLM as a feature extractor and an ASP-based MFA classifier that aggregates audio representations across time and layers.
Source: §1 · p. 1, §2.2.2 · p. 3, §5 · p. 4Results
The approach achieves the best performance on ASVspoof 2021 DF, competitive results on ASVspoof 2021 LA, and competitive results on ASVspoof 2019.
Source: §4.1 · p. 3, §5 · p. 4Takeaways & Limitations
Time-wise and layer-wise aggregation captures complementary audio-feature information, and the approach reports competitive results across the ASVspoof 2019 and 2021 datasets.
Source: §5 · p. 4
Abstract
from arXiv · showhide
With the rapid development of speech synthesis and voice conversion technologies, Audio Deepfake has become a serious threat to the Automatic Speaker Verification (ASV) system. Numerous countermeasures are proposed to detect this type of attack. In this paper, we report our efforts to combine the self-supervised WavLM model and Multi-Fusion Attentive classifier for audio deepfake detection. Our method exploits the WavLM model to extract features that are more conducive to spoofing detection for the first time. Then, we propose a novel Multi-Fusion Attentive (MFA) classifier based on the Attentive Statistics Pooling (ASP) layer. The MFA captures the complementary information of audio features at both time and layer levels. Experiments demonstrate that our methods achieve state-of-the-art results on the ASVspoof 2021 DF set and provide competitive results on the ASVspoof 2019 and 2021 LA set.
1. INTRODUCTION
Audio deepfakes challenge ASV systems as synthesis and voice conversion produce spoofed samples that are difficult to distinguish from real speech. The paper introduces WavLM features and an MFA classifier to address limits in speaker-related information captured by existing self-supervised features.
1. INTRODUCTION
- Spoofed audio is increasingly difficult to distinguish from real samples and can deceive both people and ASV systems.ASV is used for identity authentication, including access control, telephone banking, and forensic scenarios.Source: §1 · p. 1
- Prior work has used self-supervised speech models, including Wav2vec2, XLS-R, and other diverse-data-pretrained front ends, for spoofing countermeasures.The paper notes reported improvements from XLS-R.Source: §1 · p. 1
- Existing self-supervised features may capture audio content while neglecting speaker-related attributes relevant to deepfake detection.The paper links this limitation to masking-prediction pretraining and speaker-related artifacts in spoofed speech.Source: §1 · p. 1
- The paper introduces WavLM as a spoofing-detection front end and proposes MFA to aggregate features across layers and time steps.The authors associate WavLM’s masked speech prediction and denoising training with potential to learn speaker-related information and complex acoustic environments.Source: §1 · p. 1
2. PROPOSED METHOD
The proposed pipeline sends raw waveforms through WavLM to obtain frame embeddings, then uses an ASP-based Multi-Fusion Attentive classifier to predict. Its pooling combines time-level statistics with attention across WavLM layers.
2. PROPOSED METHOD
- Raw waveforms pass through WavLM for frame embeddings, then MFA produces the final prediction.The pipeline has WavLM as its front end and MFA as its classifier.Source: §2 · p. 2, Figure 1 · p. 2
2.1. WavLM Model
- WavLM uses masked speech denoising and prediction, training masked frames to predict pseudo-labels after noise or overlapping speech is added.The resulting features include speaker-related and acoustic-environment information, which is relevant because fake speech can contain speaker-related artefacts.Source: §2.1 · p. 2
2.2.1. Attentive Statistics Pooling
- Attentive Statistics Pooling computes attention-weighted means and standard deviations over a feature sequence.Frame attention weights are produced by linear layers and normalized with softmax.Source: §2.2.1 · p. 2
2.2.2. MFA Classifier
- MFA applies independent time-wise ASP layers to WavLM layer representations, concatenates each layer’s mean and standard deviation, then applies layer-wise ASP across them.The layer-wise output is passed to fully connected layers for the final prediction.Source: §2.2.2 · pp. 2–3
3. EXPERIMENTS
The experiments train on ASVspoof 2019 LA train and development data and evaluate on 2019 LA, 2021 LA, and 2021 DF. They describe the evaluation metrics and implementation setup.
3.1. Datasets and metrics
- Training uses ASVspoof 2019 LA train and development sets, with evaluation on ASVspoof 2019 LA, 2021 LA, and 2021 DF.The 2021 challenge provides no new training or development data.Source: §3.1 · p. 3
- The 2021 LA and DF evaluations include additional challenges such as transmission or encoding artefacts in LA and coding or compression artefacts in DF.The 2021 sets are described as more challenging than the 2019 LA data.Source: §3.1 · p. 3
- Evaluation uses min t-DCF for combined countermeasure and ASV performance, and EER for standalone spoofing detection.Source: §3.1 · p. 3
3.2. Implementation Details
- Audio is preemphasized with coefficient 0.97 and truncated or concatenated to about four seconds; no voice activity detection or normalization is applied.The fixed length is approximately 64,600 sample points.Source: §3.2 · p. 3
- WavLM and MFA are jointly trained for about 16,000 steps per configuration using Adam, with WavLM Base and WavLM Large evaluated.Experiments use four RTX 3090 GPUs; batch size and learning-rate settings differ when WavLM is fixed versus fine-tuned.Source: §3.2 · p. 3
4. RESULTS AND ANALYSIS
The WavLM-MFA approach achieves the best reported performance on ASVspoof 2021 DF and competitive results on 2021 LA and 2019 LA. Ablations report benefits from WavLM features and the ASP-based MFA classifier.
4.1. Results on ASVspoof 2021 LA and DF evaluation set
- On ASVspoof 2021 DF, WavLM with MFA achieves the best performance and, to the authors’ knowledge, the lowest reported EER; on 2021 LA, it achieves a competitive result.Source: §4.1 · p. 3
- WavLM's masked speech denoising and prediction training can learn speaker-related information and complex acoustic environments, which the authors identify as useful for detecting spoofing artifacts.They contrast this with Wav2vec2-style masked prediction representations that contain more audio-content than speaker information.Source: §4.1 · p. 3
- MFA attends to WavLM features across time and layer levels, leveraging complementary audio information without the heterogeneous graph neural network backend used by.The authors describe MFA as an ASP-based classifier built from fully connected layers.Source: §4.1 · p. 3
4.2. Results on ASVspoof 2019 LA evaluation set
- Pooled min t-DCF of 0.0126 and EER of 0.42% are reported on ASVspoof 2019 LA.The paper states that the model performs better than recent anti-spoofing systems.Source: §4.2 · pp. 3–4, Table 2 · p. 4
4.3. Ablation Study
- WavLM-L achieves an EER of 5.79% without fine-tuning, while performance deteriorates significantly with Wav2vec2.The authors attribute WavLM's EER reduction to resilience to speech variations in the DF dataset.Source: §4.3 · p. 4, Table 3 · p. 4
- The ASP-based MFA classifier outperforms GAP and TN + FCs, which show relative performance degradations of 136.6% and 15.0%, respectively.The ablation results identify ASP-based MFA as beneficial.Source: §4.3 · p. 4, Table 3 · p. 4
5. CONCLUSION
The paper uses self-supervised WavLM as a front-end and proposes an ASP-based Multi-Fusion Attentive classifier. Its time-wise and layer-wise aggregation captures complementary audio features, and joint training yields competitive results on ASVspoof 2019 and 2021.
5. CONCLUSION
- The proposed ASP-based Multi-Fusion Attentive classifier aggregates WavLM's multi-layer representations through time-wise and layer-wise ASP to capture complementary audio features.The paper jointly trains WavLM and MFA.Source: §5 · p. 4
- The paper introduces self-supervised WavLM as a front-end feature extractor for audio deepfake detection.Source: §5 · p. 4
- Jointly trained WavLM and MFA achieve competitive results on the ASVspoof 2019 and 2021 datasets.Source: §5 · p. 4