Source-linked AI summary
ADD 2022: the First Audio Deep Synthesis Detection Challenge
Jiangyan Yi, Ruibo Fu, Jianhua Tao, Shuai Nie, Haoxin Ma, Chenglong Wang, Tao Wang, Zhengkun Tian, Xiaohui Zhang, Ye Bai, Cunhang Fan, Shan Liang, Shiming Wang, Shuai Zhang, Xinrui Yan, Le Xu, Zhengqi Wen, Haizhou Li, Zheng Lian, Bin Liu
TL;DR
Audio deepfake detection needs evaluation beyond the challenging scenarios covered by earlier shared tasks. ADD 2022 introduces three tracks and associated datasets, metrics, and protocols, finding that a single model does not handle all fake scenarios well and that generalization remains open.
Problem
Earlier shared tasks, including ASVspoof 2021, did not cover many challenging real-life audio-deepfake scenarios.
Method
ADD 2022 evaluates low-quality, partially fake, and adversarial generation-and-detection scenarios using dedicated tracks, datasets, metrics, and protocols.
Results
The results show that it is difficult to use the same model for all fake scenarios and that detection generalization remains an open problem.
Takeaways & Limitations
Low-quality and unseen generated fake utterances can easily fool detection models, keeping generalization and evaluation metrics as future focuses.
Takeaways & Limitations
All tracks remain challenging, and the PF track has a high average EER despite a best result of 4.8%.
Abstract
from arXiv · showhide
Audio deepfake detection is an emerging topic, which was included in the ASVspoof 2021. However, the recent shared tasks have not covered many real-life and challenging scenarios. The first Audio Deep synthesis Detection challenge (ADD) was motivated to fill in the gap. The ADD 2022 includes three tracks: low-quality fake audio detection (LF), partially fake audio detection (PF) and audio fake game (FG). The LF track focuses on dealing with bona fide and fully fake utterances with various real-world noises etc. The PF track aims to distinguish the partially fake audio from the real. The FG track is a rivalry game, which includes two tasks: an audio generation task and an audio fake detection task. In this paper, we describe the datasets, evaluation metrics, and protocols. We also report major findings that reflect the recent advances in audio deepfake detection tasks.
1. INTRODUCTION
Deep learning has made generated speech increasingly realistic and difficult to distinguish from genuine audio, creating risks and motivating detection research. ADD 2022 was launched to address challenging real-life scenarios not covered by earlier shared tasks.
- Motivation: Deep learning enables realistic, human-like speech that is difficult for most people to distinguish from real audio.The misuse of this technology is described as a threat to society.
- Prior shared tasks: ASVspoof challenges advanced spoofed-speech detection mainly to protect automatic speaker verification systems from manipulation.ASVspoof 2021 included a speech deepfake task, but its audio was limited to compressed audio similar to the logical-access task.
- ADD 2022: ADD 2022 was launched to fill the gap left by limited coverage of challenging real-life fake-audio scenarios.The challenge was intended to foster research on detecting deepfake and manipulated audio.
- Paper scope: The paper describes ADD 2022 tracks, datasets, evaluation metrics, protocols, baseline models, and challenge results.The paper’s organization covers tracks, datasets, metrics, baselines, results, and conclusions.
2. TRACKS
ADD 2022 defines three tracks for distinct audio-deepfake scenarios: noisy fully fake audio, partially fake audio, and a generation-versus-detection rivalry game.
- Track overview: ADD 2022 includes low-quality fake audio detection, partially fake audio detection, and audio fake game tracks.The audio fake game contains separate generation and detection tasks.
- LF: The LF track tests bona fide and fully fake utterances under real-world noises and background-music effects.Fake audio is generated using diverse text-to-speech and voice-conversion algorithms.
- PF: The PF track distinguishes partially fake audio created by manipulating original bona fide utterances with real or synthesized audio.The task contrasts these manipulated utterances with real audio.
- FG-G: FG-G asks participants to generate intelligible, speaker-similar attack samples that can fool the FG-D detection model.Samples are generated according to given text and speaker identities.
- FG-D: FG-D detects fake audio, especially FG-G attacks, using two evaluation rounds containing unseen audio and later submitted generated speech.Track 3 frames generation and detection as a rivalry game to improve anti-attack ability from both sides.
3. DATASETS
ADD 2022 uses speaker-disjoint Mandarin speech datasets with shared training and development data, track-specific adaptation and test sets, and unseen evaluation audio.
- Dataset organization: The challenge datasets contain training, development, adaptation, and test sets, with shared training and development data but track-specific adaptation and test sets.Training, development, and adaptation sets include inputs and ground truth, while test sets do not.
- Dataset organization: No speakers overlap among training, development, adaptation, and test sets.This separation is part of the dataset protocol.
- Training and development sets: Training and development data use 40 male and 40 female speakers selected from AISHELL-3, partitioned into speaker-disjoint training and development sets.The datasets include genuine and fake utterances based on a high-fidelity multi-speaker Mandarin corpus.
- Adaptation data: Fake data are generated with mainstream speech-synthesis and voice-conversion systems, while FG-G participants are recommended to build multi-speaker systems based on AISHELL-3.The adaptation sets cover noisy fully fake audio, partially fake utterances, and organizer-generated fake audio for FG-D.
- Test data: Test sets contain unseen genuine and fake utterances, including noisy LF data, partially fake PF data, and distinct FG evaluation configurations.FG-D’s second-round test set adds generated speech submitted by FG-G participants to the first-round set.
4. EVALUATION METRICS
ADD 2022 evaluates its detection tracks with EER and its generation task with DSR, while Track 3.2 uses weighted EER across two evaluation rounds.
- Metric selection: EER evaluates Tracks 1, 2, and 3.2, while DSR evaluates the Track 3.1 generation task.EER measures distinguishing generated audio from real audio; DSR measures deception in the generation task.
- Equal error rate (EER): EER is the threshold θEER where false-alarm and miss rates are equal.Pfa(θ) decreases with θ, whereas Pmiss(θ) increases.
- Weighted EER: Track 3.2 ranks submissions separately for two evaluation rounds and uses weighted EER for the final ranking.The two rounds include distinct evaluation conditions, with their EERs combined through WEER.
- Deception success rate (DSR): DSR is based on wrong detections across all detection models, normalized by the evaluation-sample count and number of models.The generation task also evaluates intelligibility and similarity to prevent interference-sample submissions.
5. CHALLENGE RESULTS
ADD 2022 evaluated baseline and participant systems across multiple detection and generation settings, revealing substantial variation in performance and persistent difficulty across tracks.
- 120 teams from 15 countries requested the datasets, while participants submitted detection scores through CodaLab.
- Detection baselines: The challenge used six detection baselines, including GMM, LCNN, and RawNet2, with LFCC inputs for GMM and LCNN and raw waveforms for RawNet2.
- LF results: 31.7% was the average LF submission EER, while the best detection EER was 21.7%.Only 3 of 42 teams outperformed baseline S02; GMM achieved the lowest baseline EER, and direct training on training plus adaptation data improved all baselines.
- PF results: 37.9% was the average PF submission EER, while the best EER was 4.8%.The best baseline S01 was surpassed by 21 of 27 teams, GMM achieved the best baseline result, and direct training on training plus adaptation data worsened all baselines.
- FG results: 34.2% was the final average FG-D WEER, with a lowest final WEER of 10.1%; FG-G achieved a best DSR of 93.8% and average DSR of 56.1%.FG-D average EERs were 20.7% with a lowest EER of 8.3% in R1 and 43.1% with a lowest EER of 11.0% in R2.
- Analysis: All tracks remained challenging, especially LF, and adding generated fake samples from FG-G caused FG-D performance to degrade obviously.
6. CONCLUSIONS
The paper summarizes ADD 2022 as a challenge spanning three tracks and four tasks for difficult fake-audio scenarios. Its results indicate that robustness across scenarios and suitable evaluation metrics remain unresolved priorities.
- ADD 2022 addressed three challenging fake scenarios—LF, PF, and FG—across four tasks.
- The results show that one model is difficult to use across all fake scenarios, while detection generalisation remains an open problem.
- Low-quality and unseen generated fake utterances can easily fool detection models.
- Generalisation and evaluation metrics remain focuses for future evaluations.