Source-linked AI summary
ADD 2023: the Second Audio Deepfake Detection Challenge
Jiangyan Yi, Jianhua Tao, Ruibo Fu, Xinrui Yan, Chenglong Wang, Tao Wang, Chu Yuan Zhang, Xiaohui Zhang, Yan Zhao, Yong Ren, Le Xu, Junzuo Zhou, Hao Gu, Zhengqi Wen, Shan Liang, Zheng Lian, Shuai Nie, Haizhou Li
TL;DR
Audio deepfake detection must address realistic misuse and limitations of binary real/fake classification. This paper presents ADD 2023, a challenge introducing region localization and algorithm recognition, with datasets, metrics, protocols, and participant results showing that the new tasks remain difficult.
Problem
Realistic speech synthesis and voice conversion create misuse risks, while ADD 2022 focused mainly on binary real/fake classification and had limited fake-audio-game evaluation rounds.
Method
The paper describes ADD 2023’s challenge design, including new region-location and algorithm-recognition subchallenges, datasets, evaluation metrics, protocols, and baseline systems.
Results
The results indicate that fake region location and algorithm recognition remain challenging, although the best Track 3 team achieved an F1-score of 89.63%.
Takeaways & Limitations
ADD 2023 provides challenge data and evaluation results intended to promote further research on real-world audio deepfake detection and more advanced methods.
Takeaways & Limitations
ADD 2022’s techniques focused more on binary real/fake classification and included limited evaluation rounds for the fake audio game track.
Abstract
from arXiv · showhide
Audio deepfake detection is an emerging topic in the artificial intelligence community. The second Audio Deepfake Detection Challenge (ADD 2023) aims to spur researchers around the world to build new innovative technologies that can further accelerate and foster research on detecting and analyzing deepfake speech utterances. Different from previous challenges (e.g. ADD 2022), ADD 2023 focuses on surpassing the constraints of binary real/fake classification, and actually localizing the manipulated intervals in a partially fake speech as well as pinpointing the source responsible for generating any fake audio. Furthermore, ADD 2023 includes more rounds of evaluation for the fake audio game sub-challenge. The ADD 2023 challenge includes three subchallenges: audio fake game (FG), manipulation region location (RL) and deepfake algorithm recognition (AR). This paper describes the datasets, evaluation metrics, and protocols. Some findings are also reported in audio deepfake detection tasks.
1. Introduction
ADD 2023 responds to the expanding risks and limitations of audio deepfake detection by extending evaluation beyond binary real/fake classification. The challenge introduces localization and source-recognition goals and reports datasets, metrics, protocols, and participant results.
- Advances in speech synthesis and voice conversion produce realistic humanlike speech, creating misuse risks and motivating audio deepfake detection research.
- Earlier challenges, including ASVspoof 2021 and ADD 2022, helped foster research on audio deepfake detection.
- ADD 2022 focused mainly on binary real/fake classification and offered limited evaluation rounds for its audio fake game track.
- ADD 2023 targets manipulated-interval localization in partially fake speech and identification of the source algorithm generating fake audio.
- The paper describes datasets and evaluation metrics for the subchallenges and reports participant performance to examine current audio deepfake detection capabilities.
2. Subchallenges
ADD 2023 comprises audio fake game, manipulation region location, and deepfake algorithm recognition tasks, with new tasks, multi-round evaluation, and challenging datasets. Its datasets cover task-specific speech, partially fake audio, known algorithms, and unknown counterfeit audio.
- 2. Subchallenges: ADD 2023 includes audio fake game (FG), manipulation region location (RL), and deepfake algorithm recognition (AR), with RL and AR newly introduced.
- Track 1. Audio fake game (FG): FG includes separate generation and detection tracks, each evaluated over two rounds.
- Track 1. Audio fake game (FG): FG-G generates fake audio intended to fool FG-D, while FG-D detects fake utterances, especially samples generated in FG-G.
- Tracks 2–3: RL locates manipulated regions in partially fake audio, whereas AR recognizes deepfake-generation algorithms, including an unknown algorithm in evaluation.
- 2.1. Training and dev sets: Training and development data use task-specific subsets, including AISHELL-3 speech, real and fake utterances, partially fake audio, and seven Track 3 classes.
- 2.2. Test sets: The test sets are more challenging than the previous challenge and include unseen partially fake audio, added noise and format conversions, and an eighth unknown Track 3 class.
3. Evaluation metrics
ADD 2023 uses task-specific metrics for fake-audio generation and detection, manipulation-region location, and deepfake algorithm recognition. These metrics combine round-weighted error or deception measures with accuracy and F1-based measures tailored to each task.
- Track 1: Audio fake game: Track 1.1 evaluates generated-audio deception with DSR, while Track 1.2 evaluates fake detection using WEER.DSR measures deception success, and WEER is the weighted equal error rate for detection.
- Track 3: Algorithm recognition: Track 3 uses macro-average F1-score for open-set recognition of known and unknown deepfake algorithms.Although the formula iterates over known classes, false positives and false negatives include unknown-class samples.
- Track 2: Manipulation region location: Track 2 combines sentence accuracy with segment F1-score to evaluate both utterance-level classification and manipulated-region identification.Sentence accuracy uses TP, TN, FP, and FN, while segment metrics assess identification of fake areas.
4. Challenge results
ADD 2023 reports participant submissions, organizer baselines, and rankings across detection, fake-region location, and algorithm-recognition tasks. Fake-region location remains challenging, while the best algorithm-recognition result reaches 89.63% F1-score.
- 145 teams from 12 countries requested challenge data, submitted results, and received scores through CodaLab.
- Detection baselines: ADD 2023 provides six baseline systems spanning GMM-LFCC, LFCC-LCNN, wav2vec2-LCNN, pooled-output-free LCNN, and LFCC-ResNet approaches.The LFCC-ResNet baselines use thresholded softmax or OpenMax for known-versus-unknown algorithm recognition.
- Detection baselines: Track 2 combines sentence accuracy with segment F1-score to evaluate both genuine/fake discrimination and manipulated-area identification.
- Challenge results: 44.97% was the best two-round combined performance for Track 1.1, while Track 1.2’s best team achieved 12.45% WEER.Track 1.1 averaged 27.11% WDSR, and Track 1.2 averaged 49.94% WEER.
- Challenge results: 67.13% was the highest Track 2 score, and 89.63% was the best Track 3 F1-score, although average Track 3 performance remained low.Eleven teams exceeded baseline S04 in Track 2, and nine teams exceeded baselines S06 and S07 in Track 3.
5. Conclusions
ADD 2023 introduces new tasks and more difficult datasets to better simulate real-world audio deepfake detection. Results indicate that fake-region location and algorithm recognition remain challenging, particularly fake-region location.
- The challenge comprises four distinct subchallenges and introduces two new tasks with more difficult datasets.
- Fake-region location and algorithm recognition remain challenging, especially the fake-region location track.
- Future competitions will optimize datasets and competition rules to promote more advanced deepfake-audio research.