PaperlayerView exact revision on arXiv

Source-linked AI summary

Investigating self-supervised front ends for speech spoofing countermeasures

Xin Wang, Junichi Yamagishi

arXiv:2111.07725v3eess.AScs.SD

TL;DR

Reliable spoofing countermeasures are needed because advanced voice conversion and text-to-speech can attack speaker verification and fool humans. This study uses pre-trained self-supervised speech models as spoofing-countermeasure front ends and finds that fine-tuning a diverse-data front end improves performance across multiple ASVspoof test sets.

  • Problem

    Advanced voice conversion and text-to-speech can attack automatic speaker verification systems and fool humans, motivating reliable spoofing countermeasures.

  • Method

    The study evaluates pre-trained self-supervised speech models as spoofing-countermeasure front ends alongside different back-end architectures, front-end fine-tuning, and pre-training choices.

  • Results

    Fine-tuning a diverse-data pre-trained front end produced a countermeasure that matched a strong baseline on the 2019 LA test set and significantly reduced EER on the other test sets.

  • Takeaways & Limitations

    The results suggest that diverse speech pre-training and front-end fine-tuning are a potential direction for improving countermeasure generalizability.

  • Takeaways & Limitations

    Limited computing resources prevented exhaustive evaluation of all combinations of self-supervised models and back ends.

Abstract

from arXiv · show

Self-supervised speech model is a rapid progressing research topic, and many pre-trained models have been released and used in various down stream tasks. For speech anti-spoofing, most countermeasures (CMs) use signal processing algorithms to extract acoustic features for classification. In this study, we use pre-trained self-supervised speech models as the front end of spoofing CMs. We investigated different back end architectures to be combined with the self-supervised front end, the effectiveness of fine-tuning the front end, and the performance of using different pre-trained self-supervised models. Our findings showed that, when a good pre-trained front end was fine-tuned with either a shallow or a deep neural network-based back end on the ASVspoof 2019 logical access (LA) training set, the resulting CM not only achieved a low EER score on the 2019 LA test set but also significantly outperformed the baseline on the ASVspoof 2015, 2021 LA, and 2021 deepfake test sets. A sub-band analysis further demonstrated that the CM mainly used the information in a specific frequency band to discriminate the bona fide and spoofed trials across the test sets.

1. Introduction

The study motivates self-supervised front ends for spoofing countermeasures because conventional and supervised front ends can degrade under unknown attacks or domain mismatch. It investigates back-end architecture, front-end fine-tuning, and pre-trained model choice across multiple ASVspoof evaluations.

  • Closed-set spoofing countermeasures can degrade on unknown attacks or bona fide speech from mismatched domains.Supervised DNN front ends also require sufficient bona fide and spoofed speech, whose generation is laborious and technically demanding.
  • Self-supervised front ends use DNNs trained on speech data without spoofed trials, potentially providing features robust to unknown domains.The study notes that many pre-trained self-supervised models are available for off-the-shelf use, although training such models is costly.
  • The study investigates which back-end architecture, fine-tuning strategy, and pre-trained self-supervised model best support speech anti-spoofing.These questions are evaluated using experiments on ASVspoof training and test sets.
  • Experiments use the ASVspoof 2019 logical access training set and test sets from the 2015, 2019, and 2021 challenges.

2. Methods

The methods combine pre-trained Wav2vec 2.0 or HuBERT front ends with alternative back ends, while comparing fixed and jointly fine-tuned front ends. The study also compares several Fairseq models and examines the resulting configurations.

  • Wav2vec 2.0 and HuBERT are the self-supervised speech models considered as front ends.Wav2vec 2.0 uses a CNN to extract latent features and a Transformer to produce sequence outputs containing information from the full input.
  • The self-supervised model output is fed to a back end that produces a spoofing score for each input waveform.
  • Back end architecture: Three back ends are compared: LLGF uses LCNN, bidirectional LSTM layers, global average pooling, and an FC output layer; LGF removes LCNN, while GF also removes LSTM layers.
  • Fine-tuning strategy: The front end is either kept fixed or initialized from a pre-trained model and jointly fine-tuned with the rest of the countermeasure.
  • Pre-trained models: Several Fairseq pre-trained models are compared because their training data range from diverse speech corpora to limited single-corpus data.

3. Experiment

The experiments evaluate self-supervised-front-end countermeasures across multiple ASVspoof test sets, varying back ends, fine-tuning, and pre-trained models. Fine-tuning generally improves cross-dataset performance, while deeper back ends are more important when the front end remains fixed.

  • Experimental setup: The ASVspoof 2019 LA training set trained each countermeasure, with evaluation on 2019 LA, 2015, 2021 LA, and 2021 DF test sets.The 2019 LA test set represents a benign condition, whereas 2021 LA and DF include more adverse codec and domain conditions.
  • Back-end architectures: With a fixed W2V-XLSR front end, LLGF achieved lower EERs than LGF, and LGF outperformed GF in most statistically significant comparisons.Training curves and losses also favored the deeper LLGF back end.
  • Fine-tuning: When the front end was fine-tuned, even the simple GF back end achieved test-set results similar to those of deeper alternatives.Fine-tuned systems also converged more quickly, and LLGF–GF differences were not significant in most cases.
  • Fine-tuning: Fine-tuning W2V-XLSR improved EERs on known and partially known attacks and reduced EERs on unknown attacks except A11 with LLGF.The authors conclude that fine-tuning the self-supervised front end is worthy of trial.
  • Pre-trained models: Fixed W2V-Large2 and W2V-XLSR combined with LLGF performed similarly to or better than the LFCC baseline on all test sets.Both models were trained on speech from diverse corpora, unlike the other compared models.

4. Sub-band analysis

The sub-band analysis tested how spoofing countermeasures respond when selected frequency ranges are removed. Conventional and self-supervised front ends relied on different frequency information, with self-supervised systems mainly using 0.1–2.4 kHz.

  • Analysis method: Band-stop filtering was used to compare score distributions and infer which frequency bands each countermeasure uses.The analysis filtered test trials, scored them with trained countermeasures, and compared scores before and after filtering.
  • Experimental setup: The analysis evaluated five countermeasures on randomly selected subsets of the ASVspoof 2019 LA and 2021 DF test sets.The systems included a LFCC baseline and fixed or fine-tuned self-supervised front ends with different back ends.
  • Baseline behavior: 46% EER was observed on the ASVspoof 2019 LA test subset when the baseline’s 5.6–7.2 kHz band was removed.The baseline also degraded severely when filtering 7.2–8.0 kHz, indicating heavy reliance on high-frequency information.
  • Self-supervised front ends: The four self-supervised countermeasures were relatively insensitive to high-frequency filtering but changed notably when 0–0.8 or 0.8–2.4 kHz was removed.The pattern was consistent across both test subsets, with limited exceptions for fine-tuned W2V-XLSR on the 2019 LA subset.
  • Interpretation: Self-supervised countermeasures mainly relied on 0.1–2.4 kHz information to discriminate bona fide and spoofed trials across test sets.The authors suggest this band may contain useful, generalizable information, whereas high-frequency cues may be dataset-specific.

5. Conclusion

The study finds that pre-trained self-supervised speech models can serve as front ends for spoofing countermeasures, with fine-tuning producing stronger cross-dataset performance. Sub-band analysis indicates that these systems use information concentrated in 0.1–2.4 kHz.

  • Conclusion: A diverse-data self-supervised front end performed well when fixed and combined with a conventional LCNN-LSTM back end.The study focused on Wav2vec 2.0 and HuBERT-based models and compared several back-end configurations.
  • Conclusion: Fine-tuning the front end produced more notable improvement than keeping it fixed.The fine-tuned countermeasure performed decently on the 2019 LA test set and significantly outperformed the baseline on the 2015, 2021 LA, and 2021 DF test sets.
  • Conclusion: Self-supervised countermeasures relied on 0.1–2.4 kHz information to discriminate spoofed and bona fide trials across test sets.The sub-band analysis identified this frequency region as useful across the evaluated datasets.
  • Scope and caveat: The reported EERs cannot be directly compared with other studies because the pre-trained front end used more speech data.The authors present the findings as a potential direction for improving countermeasure generalizability.
  • Additional experiments: Additional strategies, including partial re-randomization and weighted sums of hidden features, were not consistently better than the strategies discussed.These additional results were reported in the appendix.

A. Statistical analysis results

Figure 4 reports Holm–Bonferroni-corrected pairwise significance tests of EERs on the ASVspoof 2019 LA evaluation.

  • Statistical analysis: The significance tests use EERs from ASVspoof 2019 LA and Holm–Bonferroni correction with α = 0.05.Dark gray indicates a significant difference; white indicates otherwise.
  • Statistical analysis: Each black-framed square contains 3 × 3 pairwise tests between three training-evaluation rounds of two models.The three rounds for each model follow the ordering used in Table 2.

B. Other experiment results

The appendix reports min tDCFs and decomposed EERs for models trained on the ASVspoof 2019 LA training set. The tables organize results across test sets and training-evaluation rounds.

  • Min tDCF: Table 4 reports min tDCFs on different test sets for models trained using the ASVspoof 2019 LA training set.The min tDCF values use the official ASVspoof 2021 code and differ slightly from the legacy ASVspoof 2019 metric.
  • Min tDCF: Table 4 preserves the ordering of the three training-evaluation rounds for each model from Table 2.Darker cell color indicates a higher EER value.
  • Decomposed EERs: Table 5 reports decomposed EERs in percentages across the test sets.EERs from the three training-evaluation rounds are averaged for each model.

B.3. Other experimental models

This section presents results from additional experimental models, without listing all experimental findings in the main text.

  • The section reports results from other experimental models while omitting the complete set of findings for concision.

B.3.1. Fine-tune on W2V-Large2

This section compares fine-tuned W2V-XLSR and W2V-Large2 using EERs across test sets. Their EERs are described as not substantially different.

  • W2V-XLSR and W2V-Large2 were compared under fine-tuning using EERs across multiple test sets.Both models were evaluated after training with the ASVspoof 2019 LA training set.
  • The EERs of W2V-XLSR and W2V-Large2 were not substantially different in the fine-tuned condition.
  • Both W2V-XLSR and W2V-Large2 were considered reasonably effective for anti-spoofing when fine-tuned.

B.3.2. Partially re-randomizing pre-trained SSL

The study evaluates partially re-randomizing pre-trained SSL weights by discarding and reinitializing the last three Transformer blocks. This reduced EER on ASVspoof 2019 LA but did not consistently improve other test sets.

  • The proposed strategy re-randomizes the last 3 Transformer blocks before fine-tuning a pre-trained SSL model.The strategy is motivated by the differing behavior of the final Transformer blocks.
  • EERs on the ASVspoof 2019 LA test set were reduced after re-randomizing the last three Transformer blocks.
  • Improvements were not consistent across other test sets, and LLGF with the strategy slightly increased ASVspoof 2021 LA evaluation EER to 8.21%.
  • Choosing how many layers to re-randomize requires analyzing hidden features from the Wav2vec 2.0 models.The authors note that this choice requires additional effort.

B.3.3. Use weighted sum of hidden features from Transformer blocks

The weighted-sum strategy combines hidden features from all Wav2vec 2.0 Transformer blocks using learned weights. Its effectiveness varied by pre-trained model and test set rather than producing consistent gains.

  • The method forms a1:N by taking a weighted sum of hidden features from all Wav2vec 2.0 Transformer blocks.The output z(i)1:N from each Transformer block is combined using its corresponding weight wi.
  • The weights wi are learned on the downstream task’s training set.
  • The weighted-sum strategy was tested with both fixed and fine-tuned Wav2vec 2.0 front ends.For fixed SSLs, wi was trained with the rest of the CM; for fine-tuned SSLs, wi was trained jointly with the whole CM.
  • Weighted sum improved EERs for W2V-XLSR on ASVspoof 2019 LA but not for W2V-Large2, and increased W2V-Large2 EERs on ASVspoof 2021 LA.
Loading 2111.07725v3…