PaperlayerView exact revision on arXiv

Source-linked AI summary

Detect All-Type Deepfake Audio: Wavelet Prompt Tuning for Enhanced Auditory Perception

Yuankun Xie, Ruibo Fu, Zhiyong Wang, Xiaopeng Wang, Songjun Cao, Long Ma, Haonan Cheng, Long Ye

arXiv:2504.06753v2cs.SDcs.AI

TL;DR

Existing countermeasures perform well on single-type audio deepfake detection, but evidence for detection across speech, sound, singing voice, and music is limited. This paper establishes an all-type benchmark and develops prompt- and wavelet-based SSL countermeasures. Co-trained WPT-XLSR-AASIST achieves an average EER of 3.58% across the all-type evaluation set.

  • Problem

    Cross-type detection across all audio types had not been explored, leaving uncertainty about whether single-type countermeasures generalize across speech, sound, singing voice, and music.

    Source: Introduction · pp. 1–2

  • Method

    The paper establishes an all-type ADD benchmark and develops PT-SSL and WPT-SSL, which adapt SSL features using learned prompts and wavelet-derived frequency prompts.

    Source: Introduction · p. 2

  • Results

    3.58% average EER: co-trained WPT-XLSR-AASIST achieved this result across the all-type ADD evaluation set.

    Source: Conclusion · p. 7

  • Takeaways & Limitations

    The benchmark evaluates current countermeasures across four audio types, and co-training with WPT prompts yields a reported all-type countermeasure.

    Source: Introduction · p. 2, Conclusion · p. 7

    The benchmark selection targets relatively clean audio without partially spoofed scenarios, limiting its stated scope to those conditions.

    Source: Dataset · p. 3

Abstract

from arXiv · show

The rapid advancement of audio generation technologies has escalated the risks of malicious deepfake audio across speech, sound, singing voice, and music, threatening multimedia security and trust. While existing countermeasures (CMs) perform well in single-type audio deepfake detection (ADD), their performance declines in cross-type scenarios. This paper is dedicated to studying the all-type ADD task. We are the first to comprehensively establish an all-type ADD benchmark to evaluate current CMs, incorporating cross-type deepfake detection across speech, sound, singing voice, and music. Then, we introduce the prompt tuning self-supervised learning (PT-SSL) training paradigm, which optimizes SSL front-end by learning specialized prompt tokens for ADD, requiring 458x fewer trainable parameters than fine-tuning (FT). Considering the auditory perception of different audio types, we propose the wavelet prompt tuning (WPT)-SSL method to capture type-invariant auditory deepfake information from the frequency domain without requiring additional training parameters, thereby enhancing performance over FT in the all-type ADD task. To achieve an universally CM, we utilize all types of deepfake audio for co-training. Experimental results demonstrate that WPT-XLSR-AASIST achieved the best performance, with an average EER of 3.58% across all evaluation sets.

Introduction

The paper addresses the lack of all-type cross-type audio deepfake detection evidence and introduces a benchmark and prompt-based SSL methods for this task. Its WPT-SSL approach captures frequency-domain information, and co-trained WPT-SSL-AASIST achieves the strongest reported all-type result.

Introduction

  • No prior work had explored cross-type detection across all audio types, and whether domain-invariant authenticity features exist remained uninvestigated.The identified types are speech, sound, singing voice, and music.Source: Introduction · p. 2
  • The paper establishes an all-type ADD benchmark covering cross-type detection among speech, sound, singing voice, and music.It evaluates handcrafted, raw-waveform, and SSL-based features with freezing and fine-tuning, using AASIST as the backend.Source: Introduction · p. 2
  • PT-SSL learns specialized prompt tokens for ADD while freezing the other transformer parameters.The paper presents prompt tuning as a parameter-efficient alternative that achieves performance close to fine-tuning.Source: Introduction · p. 2
  • WPT-SSL applies a discrete wavelet transform to some prompt tokens to obtain frequency-band tokens and enhance full-frequency perception.The paper reports that this approach can learn a type-invariant detection prompt in the HH frequency band.Source: Introduction · p. 2
  • WPT-SSL-AASIST outperformed fine-tuning under all ADD test conditions and achieved an average EER of 3.58%.The paper uses co-training across audio types to target a universal countermeasure.Source: Introduction · p. 2

Prompt Tuning Countermeasure

PT-SSL adapts a frozen SSL front end by learning prompt tokens, while WPT adds wavelet-derived prompts to improve frequency perception. Both feed transformer outputs to an AASIST backend.

PT-SSL-AASIST

  • PT-SSL prepends learnable prompt tokens at each transformer layer while keeping the other layer parameters frozen.The prompt tokens are learned for the ADD task; the paradigm is presented as rapidly adapting SSL features.Source: Prompt Tuning Countermeasure · p. 2, PT-SSL-AASIST · p. 2
  • For PT-SSL, an audio input is padded or chopped to fixed length L before passing through the frozen SSL feature extractor.For XLSR, the feature extractor comprises seven CNN layers before the transformer input E0.Source: PT-SSL-AASIST · p. 2
  • At each transformer layer, the prompt output is replaced by the prompt tokens used in the next layer’s computation.The prompt count p is preset, and l denotes the number of SSL layers.Source: PT-SSL-AASIST · p. 2
  • The final transformer output is sent to the AASIST backend, which captures spectro-temporal features and produces two-dimensional logits optimized with weighted cross-entropy.The described XLSR example uses 24 transformer layers.Source: PT-SSL-AASIST · p. 2

WPT-SSL-AASIST

  • WPT differs from PT in prompt initialization, adding wavelet-initial tokens to improve SSL embeddings’ frequency sensitivity for all-type ADD.The method uses Haar wavelets for the discrete wavelet transform.Source: WPT-SSL-AASIST · pp. 2–3
  • The Haar transform yields LL, LH, HL, and HH components, which are reshaped and concatenated into wavelet prompts.The components correspond to low frequency and high-frequency vertical, horizontal, and diagonal directions.Source: WPT-SSL-AASIST · p. 3
  • WPT concatenates the wavelet prompts with PT tokens at each layer, then sends the final transformer output to the AASIST backend.The figure contrasts PT-SSL-AASIST and WPT-SSL-AASIST, and Table 1 reports benchmark statistics for training, development, and evaluation sets.Source: WPT-SSL-AASIST · p. 3, Figure 2 · p. 3, Table 1 · p. 3

All-Type ADD Benchmark

The benchmark evaluates audio deepfake countermeasures across four audio types, using datasets, model baselines, and both single-type and all-type training protocols. Its dataset selection targets relatively clean conditions without partially spoofed scenarios.

All-Type ADD Benchmark

  • The section introduces the benchmark experimental setup, covering four audio-type datasets, countermeasures, training and testing protocols, and implementation.Source: All-Type ADD Benchmark · p. 3

Dataset

  • Dataset selection targets relatively clean audio without partially spoofed scenarios, to study cross-type ADD while removing other interferences such as noise.The authors state that reducing such interference benefits the study of cross-type ADD.Source: Dataset · p. 3
  • The benchmark covers speech, sound, singing voice, and music using Speech-19LA, Codecfake-A3, CtrSVDD, and FakeMusicCaps.The datasets use distinct splits or protocols, including held-out spoofing systems for Speech-19LA and singing voice, and a 7:1:2 split for Codecfake-A3.Source: Dataset · pp. 3–4

Baseline Countermeasure

  • Five baselines combine Spec-Resnet, AASIST, and three SSL-AASIST variants using MERT, WavLM, or XLSR front ends.Spec-Resnet represents spectrogram-based features; AASIST operates directly on waveforms, while the SSL variants pair their front ends with AASIST.Source: Baseline Countermeasure · p. 4
  • Frozen and fine-tuned SSL differ in whether SSL parameters are updated; PT and WPT are additional training paradigms.The experiments use frozen SSL for the Table 2 results.Source: Baseline Countermeasure · p. 4, Table 2 · p. 4

Training and Evaluation Protocol

  • The protocol trains on one audio type and tests on each type, then also evaluates models co-trained on all types against each evaluation set.The benchmark section describes these as single-type and all-type co-training experiments.Source: Training and Evaluation Protocol · p. 4

Implementation Details

  • Audio is resampled to 16,000 Hz and trimmed or padded to 64,600 samples; single-type training runs 50 epochs and co-training runs 20.For FT, the initial learning rate is 10^-6 and batch size 14; FR, PT, and WPT use 5×10^-4 and batch size 32.Source: Implementation Details · p. 4

Experiments

Co-training across speech, sound, singing voice, and music gives WPT-XLSR-AASIST the strongest reported all-type result. Across the experiments, prompt design and training paradigm affect detection performance, while WPT shows type-invariant representations and attention patterns.

Investigation for Single-Type Training

  • Single-type training transferred unevenly: XLSR-AASIST performed best across speech and singing, while sound- and music-trained models showed type-specific strengths.Frozen speech-trained XLSR-AASIST had 32.58% average EER and transferred to singing at 29.72%; fine-tuned models showed lower EERs on several in-domain tasks, but music-trained SSL features declined versus frozen features.Source: Investigation for Single-Type Training · p. 5, Table 3 · p. 5

Prompt Tuning Countermeasures

  • 10 prompt tokens yielded the best speech test-set EER (0.22%) and lowest average EER (30.79%); increasing token count raised parameter count while reducing effectiveness.The authors suggest excess prompts may make the 201 audio tokens sparse and hinder learning audio information.Source: Prompt Tuning Countermeasures · p. 5, Table 4 · p. 5
  • For speech-trained XLSR-AASIST, WPT achieved 0.15% ID speech EER and 27.55% average EER, outperforming FT, PT, and FR with 458 times fewer training parameters than FT.FR, PT, and WPT converged significantly faster than FT.Source: Table 7 · p. 6, Prompt Tuning Countermeasures · p. 6, Figure 4 · p. 6
  • The optimal PT-SSL-AASIST paradigm inserts prompts in each transformer layer and feeds both prompt and audio tokens from the final layer to AASIST.This outperformed Shallow-PT, After-PT, and Del-PT in the reported comparison.Source: Table 5 · p. 5, Prompt Tuning Countermeasures · p. 6

Co-trained Countermeasures

Prompt Tuning Countermeasures

  • Four wavelet prompt tokens gave the best overall result among the tested WPT settings, with 26.86% average EER; ten WPTs had a 0.11% speech EER but degraded on other types.The four tokens align with the LL, LH, HL, and HH frequency bands.Source: Prompt Tuning Countermeasures · p. 6, Table 6 · p. 6

Interpretability

  • In the t-SNE embeddings, FT separated audio types within genuine and fake regions, whereas WPT showed overlap among types while separating real from fake samples.The authors interpret this pattern as evidence that WPT maintains type invariance for all-type ADD.Source: Figure 5 · p. 6, Interpretability · p. 7
  • Attention patterns varied by audio type for FT but were more invariant for WPT, which focused on the fourth prompt token associated with high-frequency details.PT focused on its first prompt token, with attention changing across audio types.Source: Interpretability · p. 7, Figure 6 · p. 7

Related papers

Loading 2504.06753v2…