PaperlayerView exact revision on arXiv

Source-linked AI summary

AI-Synthesized Voice Detection Using Neural Vocoder Artifacts

Chengzhe Sun, Shan Jia, Shuwei Hou, Siwei Lyu

arXiv:2304.13085v2cs.SDcs.MMeess.AS

TL;DR

Synthetic human-voice detection has received less attention than image and video detection, and audio's distinct characteristics make image-based approaches difficult to apply. The paper trains a multi-task RawNet2 detector with vocoder identification as a pretext task, and reports high overall classification performance with robustness to resampling and background noise.

  • Problem

    Synthetic human-voice detection has received less attention than image and video detection, and audio's different characteristics make image-based detection methods difficult to apply.

    Source: §1 · p. 1

  • Method

    A multi-task RawNet2 model shares its feature extractor between binary voice classification and vocoder identification, using the latter as a pretext task to focus on vocoder artifacts.

    Source: §1 · p. 2

  • Results

    The RawNet2 model achieved high classification performance overall across LibriSeVoc and two public DeepFake audio datasets, with robustness to resampling and background noise.

    Source: §1 · p. 2, §5 · p. 7

  • Takeaways & Limitations

    The method showed good detection robustness under resampling and background noise post-processing.

    Source: §1 · p. 2

    The conventional binary-classification approach assumes many synthetic voice samples are available and does not account for neural-vocoder statistical properties.

    Source: §3 · p. 4

Abstract

from arXiv · show

Advancements in AI-synthesized human voices have created a growing threat of impersonation and disinformation, making it crucial to develop methods to detect synthetic human voices. This study proposes a new approach to identifying synthetic human voices by detecting artifacts of vocoders in audio signals. Most DeepFake audio synthesis models use a neural vocoder, a neural network that generates waveforms from temporal-frequency representations like mel-spectrograms. By identifying neural vocoder processing in audio, we can determine if a sample is synthesized. To detect synthetic human voices, we introduce a multi-task learning framework for a binary-class RawNet2 model that shares the feature extractor with a vocoder identification module. By treating vocoder identification as a pretext task, we constrain the feature extractor to focus on vocoder artifacts and provide discriminative features for the final binary classifier. Our experiments show that the improved RawNet2 model based on vocoder identification achieves high classification performance on the binary task overall.

1. Introduction

Synthetic human-voice detection remains underdeveloped, while increasingly realistic synthetic voices have been used for impersonation and deception. This work targets neural-vocoder artifacts with a multi-task detector and introduces LibriSeVoc; experiments report effectiveness across three datasets.

1. Introduction

  • Synthetic human-voice detection has received less attention than image and video detection, and audio's different signal characteristics make image-based methods difficult to apply.The introduction also describes synthesized voices being used to impersonate people and deceive others.Source: §1 · p. 1
  • Neural vocoders synthesize waveforms from representations such as mel-spectrograms, and their artifacts are proposed as cues for detecting synthetic voices.The paper notes that neural vocoders are the final step in most AI-based audio synthesis models and that real audio is unlikely to be processed with them.Source: §1 · p. 1
  • LibriSeVoc uses self-vocoding samples from six neural vocoders to highlight vocoder signatures while controlling for other factors.The samples are generated from the same original audio signals.Source: §1 · pp. 1–2
  • Mel-spectrogram differences between an original voice and its self-vocoded version illustrate artifacts that can be subtle to visualize but captured by a trained classifier.Figure 1 places the original and self-vocoded spectrograms above their differences.Source: §1 · p. 2, Figure 1 · p. 2
  • The proposed RawNet2 achieved high classification performance overall, was evaluated on three datasets, and showed robustness to resampling and background noise.The experiments included LibriSeVoc and two public DeepFake audio datasets.Source: §1 · p. 2
  • A shared feature extractor connects binary real-versus-synthetic classification with vocoder identification, using the latter as a pretext task to learn discriminative vocoder-artifact features.The approach is presented as a way to address the insufficient number of existing real and synthetic voice samples by incorporating self-vocoding data.Source: §1 · p. 2

2. Related Works

The related work reviews deep-neural-network approaches to human voice synthesis, neural vocoders, and synthetic-voice detection. It highlights limited real-world detection performance as a motivation for further work.

2. Related Works

  • The section frames voice synthesis, neural-vocoder models, and AI-synthetic voice detection as the three areas relevant to the research.Source: §2 · p. 2

2.1. Human Voice Synthesis

  • Neural vocoders reconstruct waveforms from mel-spectrograms in both text-to-speech and voice-conversion systems, which use deep neural networks trained on large voice corpora.The review groups vocoders into autoregressive, diffusion, and GAN-based models.Source: §2.1 · p. 2, §2.2 · p. 3

2.2. Neural Vocoders

  • Autoregressive vocoders generate samples sequentially and are slower; diffusion vocoders are time-efficient but have inferior reconstruction quality and may produce noisier samples, while GAN-based vocoders combine generation speed and quality.Source: §2.2 · p. 3

2.3. AI-synthetic Human Voice Detection

  • Detection approaches include bi-spectral analysis and network-response features, while ASVspoof 2021 baselines include CQCC-GMM, LFCC-GMM, LFCC-LCNN, and RawNet2.The review also mentions spectro-temporal graph attention, unsupervised pretraining, biometric verification, and self-distillation approaches.Source: §2.3 · p. 3
  • An evaluation of 12 architectures on 37.9 hours of audio found related methods performed poorly on real-world data, with some degenerating to random guessing.Source: §2.3 · p. 3

3. Method

The method detects synthetic human voices by identifying vocoder artifacts, using a shared feature extractor for binary detection and vocoder identification. Vocoder identification acts as a pretext task to direct feature learning toward vocoder characteristics.

3. Method

  • Vocoder artifacts are treated as a key cue because real human voices typically lack them, apart from the study’s deliberately self-vocoded samples.Source: §3 · p. 3
  • RawNet2 operates directly on waveforms, reducing the risk of losing vocoder-artifact information compared with preprocessed mel-spectrogram or LFCC features.Source: §3 · p. 3
  • The binary-only approach assumes many synthetic voice samples and does not account for neural vocoders’ distinctive statistical properties.The passage notes that obtaining large numbers of synthetic samples is increasingly difficult as synthesis technology advances.Source: §3 · p. 4
  • Self-vocoding creates labeled synthetic voices by transforming real samples through a mel-spectrogram and reconstructing them with neural vocoders; LibriSeVoc supplies this vocoder-identification data.The multiclass loss is softmax, and λ controls the trade-off between binary and vocoder-identification losses.Source: §3 · p. 4
  • The framework jointly trains binary authentic-versus-fake classification and multiclass vocoder identification, sharing features so vocoder characteristics transfer to detection.The vocoder-identification loss directs the feature extractor toward vocoder-related artifacts.Source: §3 · p. 4, Figure 2 · p. 4

4. Experiments

The experiments evaluate the proposed detector on LibriSeVoc, WaveFake, and ASVspoof 2019, including intra- and cross-dataset tests and robustness to common post-processing operations.

4. Experiments

  • The experiments compare the method with state-of-the-art models across three datasets, using intra- and cross-dataset testing and post-processing robustness evaluation.Source: §4 · p. 4

4.1. Datasets

  • LibriSeVoc pairs each original mel-spectrogram with six vocoder reconstructions, isolating vocoder-specific artifacts across 34.92 hours of real and 208.74 hours of synthesized audio.The dataset is split 6:2:2 for training, development, and testing; evaluation also uses WaveFake and ASVspoof 2019.Source: §4.1 · p. 5

4.2. Implementation Details

  • Training uses RawNet2 with Adam at a learning rate of 0.0001 and batch size 32, with the loss weight λ set to 0.5; performance is reported as EER.Source: §4.2 · p. 5

4.3. Baseline methods

  • Comparisons include LFCC-LCNN, RawNet2, WavLM, and Wav2Vec2-XLS-R as baseline methods.Source: §4.3 · p. 6

4.4. Intra-dataset Evaluation

  • The proposed model achieves the lowest EERs on LibriSeVoc (0.13%) and WaveFake (0.19%), and performs best on ASVspoof 2019.Table 3 reports detection EER (%) for intra-dataset testing on three datasets.Source: Table 3 · p. 6, §4.4 · p. 6

4.5. Cross-dataset Evaluation

  • All methods degrade in cross-domain testing; the proposed method has the lowest EER on unseen MelGAN and Parallel WaveGAN samples but performs worse on other unseen vocoders.This limitation reflects vocoder artifacts that do not generalize beyond the training vocoder categories.Source: §4.5 · p. 6, Table 4 · p. 7

4.6. Robustness Evaluation

  • 0.13% detection EER on original LibriSeVoc rises to 2.73% on post-processed audio.Post-processing included resampling and added crowd noise at several SNRs; the authors report robustness to these common operations.Source: §4.6 · pp. 6–7, Figure 4 · p. 7

5. Conclusion

The paper detects synthetic voices by identifying neural-vocoder traces with a binary RawNet2 model sharing a feature extractor with vocoder identification. Experiments show high overall classification performance, while the authors identify broader data coverage and direct detection as future work.

  • The proposed binary-class RawNet2 shares its front-end feature extractor with vocoder identification to detect neural-vocoder traces.Source: §5 · p. 7
  • The authors report high overall classification performance for their synthetic-voice detection method.Source: §5 · p. 7
  • Future work will broaden LibriSeVoc's real-audio and environment coverage and explore direct detection using vocoder and other audio-signal cues.The authors describe the current approach as indirect.Source: §5 · p. 7

Social Impact Statement

Realistic, accessible AI-generated voices pose risks of impersonation and fraud. This work detects vocoder artifacts as a way to expose synthesized voices and mitigate those risks.

Loading 2304.13085v2…