Source-linked AI summary
Automatic speaker verification spoofing and deepfake detection using wav2vec 2.0 and data augmentation
Hemlata Tak, Massimiliano Todisco, Xin Wang, Jee-weon Jung, Junichi Yamagishi, Nicholas Evans
TL;DR
Spoofing countermeasures lack reliable generalisation because representative training attacks are limited while real-world attack variation is broad. The paper uses a fine-tuned wav2vec 2.0 front-end with AASIST, self-attentive aggregation, and data augmentation, achieving large reported EER reductions across Logical Access and Deepfake tasks. The authors note that these comparisons use external pre-training data and a substantially more complex model.
Problem
Spoofing countermeasures lack generalisation because real-world attacks vary widely and representative spoofing training data is impractical to acquire.
Method
The paper replaces AASIST’s sinc-layer front-end with wav2vec 2.0, pre-trained on bona fide speech and fine-tuned with spoofing data, alongside aggregation and augmentation strategies.
Results
Up to a 90% relative EER reduction for Logical Access and up to an 88% reduction for domain-mismatched Deepfake detection are reported versus a sinc-layer baseline.
Takeaways & Limitations
Self-supervised front-ends, data augmentation, and self-attentive aggregation provide complementary improvements in spoofing detection generalisation.
Takeaways & Limitations
Comparisons with most literature are not fair because this work uses external pre-training data and a substantially more complex wav2vec 2.0 model.
Abstract
from arXiv · showhide
The performance of spoofing countermeasure systems depends fundamentally upon the use of sufficiently representative training data. With this usually being limited, current solutions typically lack generalisation to attacks encountered in the wild. Strategies to improve reliability in the face of uncontrolled, unpredictable attacks are hence needed. We report in this paper our efforts to use self-supervised learning in the form of a wav2vec 2.0 front-end with fine tuning. Despite initial base representations being learned using only bona fide data and no spoofed data, we obtain the lowest equal error rates reported in the literature for both the ASVspoof 2021 Logical Access and Deepfake databases. When combined with data augmentation,these results correspond to an improvement of almost 90% relative to our baseline system.
1. Introduction
Spoofing countermeasures struggle to generalise because real-world attacks vary widely while representative training data is impractical to collect. The paper explores self-supervised learning, fine-tuning, and complementary architectural and augmentation strategies to improve robustness.
- Real-world spoofing attacks span converted voice, synthetic speech, replay, algorithms, and recording devices, making representative training data impractical.
- 0.55% to 9.26%: the best ASVspoof 2021 LA baseline EER rises sharply from development to evaluation data.
- Six attack algorithms in ASVspoof 2019 LA illustrate why challenge training data may not support broad generalisation.
- Open-source speech synthesis and voice-conversion systems can generate additional attacks, but their limited number cannot fully represent wild conditions.
- The paper investigates self-supervised representations learned from diverse speech data, fine-tuning, data augmentation, and self-attentive aggregation.
2. Related work
Self-supervised speech models have been applied across speech tasks, including spoofing detection. However, existing spoofing studies do not isolate the specific contribution of self-supervised representations.
- Self-supervised speech models including CPC, wav2vec, HuBERT, wav2vec 2.0, and WavLM have shown promise across speech applications.
- HuBERT and wav2vec 2.0 have been used for recognition, mispronunciation, emotion, and spoofing detection tasks.
- Combining self-supervised representations with a Siamese network makes their specific benefit difficult to judge without alternative front-end comparisons.
3. AASIST baseline system
The baseline is AASIST, an end-to-end spectro-temporal graph attention network that transforms raw waveforms into spectral and temporal graphs before binary classification.
- AASIST processes raw waveforms with a sinc convolutional front-end initialized with 70 mel-scaled filters of kernel size 129.
- The residual encoder produces a feature map S ∈R^C×F×T, whose dimensions represent channels, spectral bins, and time frames.
- Max-pooling absolute feature values across temporal or spectral dimensions constructs temporal and spectral input graphs.
- Parallel graph modules model the spectral and temporal graphs using graph attention and graph-pooling layers.
- A heterogeneous stacking graph attention layer combines spectral and temporal graphs to model artefacts spanning both domains.
- The readout concatenates five nodes before a hidden fully connected layer produces bona fide-versus-spoofed predictions.
4. Self-supervised front-end
The paper replaces AASIST’s sinc-layer front-end with wav2vec 2.0, using self-supervised pre-training followed by joint fine-tuning for spoofing detection.
- Wav2vec 2.0 replaces the sinc-layer front-end and is connected to the AASIST countermeasure through the residual encoder.
- A CNN maps waveform samples x_1:L to latent features z_1:N, and a transformer maps them to output representations o_1:N.
- During pre-training, latent representations are quantised, masked, contextualised, and optimized with a contrastive loss against distractors.
- Pre-training uses only bona fide data, while fine-tuning uses in-domain bona fide and spoofed data for spoofing detection.
- Fine-tuning jointly optimizes the pre-trained XLS-R model and AASIST using the ASVspoof 2019 LA training partition.
- A self-attentive aggregation layer improves performance for both the sinc-layer and wav2vec 2.0 front-ends before AASIST classification.
5. Self-attention based aggregation layer
The self-attentive aggregation layer uses 2-D attention maps to emphasize discriminative spectral and temporal features before spoofing classification.
- Self-attention mechanism: The layer generates 2-D attention maps from representations S using convolution, activation, batch normalization, convolution, and softmax operations.This differs from conventional one-dimensional attention applied to a single domain.
- Feature aggregation: Attention weights are used to extract more relevant temporal and spectral representations through weighted summation.The normalized weight matrix is applied across time and frequency.
- Feature aggregation: The extracted temporal and spectral representations are passed to the AASIST countermeasure model for spoofing detection.The aggregation layer sits between the front-end representations and the downstream AASIST processing.
- Architecture configuration: The architecture configuration specifies dimensions as channels, frequency, and time, with batch normalization and SeLU applied beneath the dotted line to residual block outputs.These conventions are given in the model architecture table.
6. Experimental setup
Experiments train and validate on ASVspoof 2019 LA and evaluate on ASVspoof 2021 LA and domain-mismatched DF data using EER and min t-DCF, with data augmentation configurations matched to each scenario.
- Databases and metrics: Training and validation use the ASVspoof 2019 LA partitions, while evaluation uses the ASVspoof 2021 LA and domain-mismatched DF databases.The evaluation databases contain variability absent from the 2019 training and validation data.
- Databases and metrics: The 2021 LA data includes codec and transmission variability, whereas DF adds compression variability and sources beyond VCTK.The 2019 data used for training and validation contains neither type of variability.
- Databases and metrics: Evaluation uses Equal Error Rate and minimum Tandem Detection Cost Function, with EER emphasized for interpretability and min t-DCF retained as the challenge default.Both metrics are reported for the experimental evaluations.
- Data augmentation: Data augmentation is investigated as a complement to self-supervised learning because it can reduce overfitting and improve generalisation.The study focuses on whether the two approaches provide complementary benefits.
- Data augmentation: For LA, the augmentation configuration combines linear and nonlinear convolutive noise with impulsive signal-dependent additive noise.These strategies are intended to match convolutive and device-related noise in telephony applications.
- Data augmentation: For DF, augmentation works best with stationary signal-independent additive and randomly coloured noise strategies.The supplied passage introduces this DF-specific configuration but truncates its description.
- Reported results: Table 2 reports pooled EER and pooled min t-DCF for sinc-layer and wav2vec 2.0 front-ends, with or without self-attentive aggregation and data augmentation.Results are the best or average outcomes from three runs using different random seeds.
7. Results
Across ASVspoof 2021 Logical Access and DeepFake evaluations, wav2vec 2.0, self-attentive aggregation, and data augmentation improved spoofing-countermeasure performance, including the paper’s lowest reported EERs.
- 7.1. Front-end comparison: 11.47% to 6.15%: replacing the sinc-layer front-end with wav2vec 2.0 reduced baseline EER on ASVspoof 2021 LA by almost 46%.The wav2vec 2.0 EER remained high before adding the self-attentive aggregation layer and data augmentation.
- 7.2. Self-attentive aggregation layer: 4.48%: adding self-attentive aggregation to wav2vec 2.0 reduced LA EER from the corresponding baseline and produced almost a 50% relative improvement.The self-attentive layer also reduced the sinc-layer system’s EER to 8.73%.
- 7.4. DeepFake results: 2.85%: on ASVspoof 2021 DF, wav2vec 2.0 with both self-attentive aggregation and data augmentation reduced EER from 7.69%.The result was statistically significant and, according to the paper, the lowest EER reported for the ASVspoof 2021 DF database.
- 7.5. Simplified CM solution: 1.19% and 4.38%: a simplified back-end still achieved competitive pooled EERs on LA and DF using fine-tuned wav2vec 2.0 with database-optimised augmentation.The simplified system used a max-pooling layer, one graph layer, and one linear layer; the benefits remained complementary to data augmentation.
8. Conclusions and discussion
The wav2vec 2.0 front-end substantially improves spoofing detection generalisation, but comparisons are confounded by external pre-training data and practical deployment concerns remain.
- Conclusions and discussion: Up to 90% relative EER reduction was obtained for logical access spoofing, and up to 88% for domain-mismatched Deepfake detection.The Deepfake task used spoofed utterances generated with more than 100 attack algorithms.
- Conclusions and discussion: The reported results are not directly fair comparisons with most literature because the model used additional external pre-training data.Most literature results use fixed training data, whereas this model was pre-trained with external data.
- Conclusions and discussion: The wav2vec 2.0 model is substantially more complex than previous state-of-the-art solutions, leaving its practical applicability unresolved.Future work proposes analysing the front-end to reduce model size and complexity for resource-constrained scenarios.
- Conclusions and discussion: Future work should identify which characteristics of the self-supervised front-end provide the greatest benefit.This analysis is intended to guide scaling down the model for practical use.
A. Statistical analysis results
Figure 4 presents Holm-Bonferroni-corrected pairwise significance tests of EERs across LA and DF 2021 evaluation data.
- A. Statistical analysis results: The tests use EERs from the LA and DF 2021 evaluation datasets with Holm-Bonferroni correction at α = 0.05.The correction threshold is explicitly specified in the figure caption.
- A. Statistical analysis results: Gray squares indicate significant differences, whereas white squares indicate insignificant differences.The colors encode the significance outcome of each comparison.
- A. Statistical analysis results: Each black-square cell contains 3 × 3 pairwise tests between three training-evaluation runs of two models.Front-end systems follow the ordering used in Tables 2 and 3.
B. Breakdown EER’s (%) pooled over attacks on ASVspoof 2021 LA and DF databases
Tables 5 and 6 organize pooled EER breakdowns by codec condition for the ASVspoof 2021 LA and DF evaluation partitions.
- B. Breakdown EER’s (%) pooled over attacks on ASVspoof 2021 LA and DF databases: Table 5 reports EERs (%) for codec conditions C1–C7 on the ASVspoof 2021 LA evaluation partition.The results are the best average obtained from three runs with different random seeds.
- B. Breakdown EER’s (%) pooled over attacks on ASVspoof 2021 LA and DF databases: Table 6 reports EERs (%) for codec conditions DFC1–DFC8 on the ASVspoof 2021 DF evaluation partition.The caption specifies the codec-condition range and evaluation partition.