Source-linked AI summary
ASVspoof 2021: accelerating progress in spoofed and deepfake speech detection
Junichi Yamagishi, Xin Wang, Massimiliano Todisco, Md Sahidullah, Jose Patino, Andreas Nautsch, Xuechen Liu, Kong Aik Lee, Tomi Kinnunen, Nicholas Evans, Héctor Delgado
TL;DR
ASVspoof 2021 addresses the need for spoofed and deepfake speech detection that remains effective under realistic and unpredictable variation. The paper defines three challenge tasks, their databases and metrics, and reports results from baseline and participant systems. Despite the added difficulty and absence of matched training or development data, logical access achieved a minimum t-DCF of 0.2177 and an EER of 1.32%, while physical access remained difficult in real spaces and deepfake performance varied substantially between progress and evaluation phases.
Problem
ASVspoof 2021 targets the need for countermeasures that protect automatic speaker verification and detect spoofed speech under varied practical conditions, including a new deepfake scenario.
Method
The paper presents three challenge tasks, their evaluation databases, performance metrics, baseline systems, evaluation platform, and participant results.
Results
The lowest logical-access result was a min t-DCF of 0.2177 with an EER of 1.32%, while physical access and deepfake tasks showed greater difficulty or evaluation variability.
Takeaways & Limitations
The challenge results are encouraging despite increased variability and no newly matched training or development data, supporting continued progress in spoofed and deepfake speech detection.
Takeaways & Limitations
Further analysis remained ongoing because the challenge combined a new task with complex evaluation conditions and a tight evaluation and paper-submission schedule.
Abstract
from arXiv · showhide
ASVspoof 2021 is the forth edition in the series of bi-annual challenges which aim to promote the study of spoofing and the design of countermeasures to protect automatic speaker verification systems from manipulation. In addition to a continued focus upon logical and physical access tasks in which there are a number of advances compared to previous editions, ASVspoof 2021 introduces a new task involving deepfake speech detection. This paper describes all three tasks, the new databases for each of them, the evaluation metrics, four challenge baselines, the evaluation platform and a summary of challenge results. Despite the introduction of channel and compression variability which compound the difficulty, results for the logical access and deepfake tasks are close to those from previous ASVspoof editions. Results for the physical access task show the difficulty in detecting attacks in real, variable physical spaces. With ASVspoof 2021 being the first edition for which participants were not provided with any matched training or development data and with this reflecting real conditions in which the nature of spoofed and deepfake speech can never be predicated with confidence, the results are extremely encouraging and demonstrate the substantial progress made in the field in recent years.
1. Introduction
ASVspoof 2021 advances spoofing countermeasures through three tasks—logical access, physical access, and speech deepfake detection—under increasingly realistic conditions. The paper presents the tasks, databases, metrics, baselines, evaluation platform, and challenge results.
- ASVspoof 2021 comprises logical access, physical access, and speech deepfake detection tasks designed to advance countermeasures protecting automatic speaker verification systems.
- The logical access task detects synthetic and converted speech injected into telephony systems, extending prior editions with telephony encoding and transmission.
- The physical access task records replay attacks across real physical spaces, devices, reverberation, and additive noise rather than relying only on simulation.
- The new deepfake task targets spoofed speech generated from publicly available victim voice data for dissemination through social media, call centers, or similar applications.
- The paper overviews the task databases and metrics, describes baseline systems, and summarizes performance from baselines and challenge submissions.
2. Database
The ASVspoof 2021 databases introduce evaluation conditions designed to test generalization across telephony, physical-space, replay, codec, compression, and domain variability. Training and development data were not newly matched to the evaluation partitions, increasing the realism and difficulty of the benchmark.
- The 2021 database provides new evaluation partitions for logical access, physical access, and deepfake tasks without releasing new matched training or development data.
- Evaluation data across tasks contains nuisance variation but is distributed in a common 16-bit PCM, 16 kHz FLAC format.
- Logical access evaluation data includes telephony transmission through VoIP and PSTN systems, introducing encoding and transmission artefacts alongside spoofing effects.
- Physical access evaluation uses real rooms, varied microphones and replay devices, acoustic propagation, reverberation, and additive noise to model replay conditions.
- The deepfake evaluation data applies lossy codecs and draws from multiple sources, producing attacks generated with more than 100 different spoofing algorithms.
- Deepfake conditions vary codec type, variable bit rate, and undisclosed compression techniques, with some conditions remaining unrevealed during writing.
3. Performance measures
ASVspoof 2021 uses min t-DCF for LA and PA, reflecting tandem CM–ASV risk, and EER for DF, which excludes an ASV system.
- LA and PA: Min t-DCF evaluates the combined performance of a countermeasure and an unprotected ASV system for LA and PA.It models the countermeasure as a bona fide/spoof gate before ASV and reflects Bayes’ risk.
- LA and PA: The t-DCF parameters depend on predefined costs, priors, and ASV performance, with C2 increasing linearly with ASV spoof false alarm rate.The metric therefore penalizes countermeasure false alarms more when spoofing is more detrimental to ASV.
- LA and PA: The t-DCF parameters differ between LA and PA and between progress and evaluation partitions because the CM objective varies with ASV performance.This produces a metric that changes with the tandem system setting.
- LA and PA: ASVspoof 2021 retains C0, the ASV floor, which is the t-DCF obtained with an error-free CM and reflects ASV errors.The ASV operates at its target-nontarget EER threshold for each task and data partition.
- DF: EER measures a countermeasure’s discrimination between bona fide and spoof utterances for DF, without requiring cost or prior parameters.The DF task has no ASV system.
4. ASV system and CM baselines
The paper specifies a DNN-based x-vector/PLDA ASV system and four CM baselines, then reports their performance across ASVspoof 2021 scenarios and phases.
- ASV system: The ASV system combines a DNN-based x-vector speaker-embedding extractor with a PLDA backend.The extractor uses MFCC features and is trained with VoxCeleb2 data; the backend uses VoxCeleb1.
- ASV system: Spoofed-condition ASV EERs are universally higher than bona fide-condition EERs across tasks and progress and evaluation partitions.The comparison indicates greater ASV error under spoofing conditions.
- CM baselines: Four CM baselines comprise CQCC-GMM, LFCC-GMM, LFCC-LCNN, and RawNet2 systems.The first two use GMMs, the third uses an LCNN with LSTM layers and average pooling, and the fourth operates directly on raw waveforms.
- CM baselines: Figure 3 tracks system performance for LA, PA, and DF throughout the progress and evaluation phases.It presents performance evolution rather than a single fixed comparison.
- CM baselines: Baseline systems use ASVspoof 2019 training data and development data only, with DF systems using LA data and no data augmentation.This defines the baseline training setup for the 2021 evaluation.
5. Evaluation platform
ASVspoof 2021 used a web-based CodaLab platform with separate progress and evaluation phases that controlled submissions and exposed selected results.
- Platform: Participants submitted CM scores through the CodaLab website and received results through the challenge platform.The platform was new to the fourth edition.
- Progress phase: The six-week progress phase allowed up to three submissions per day and reported results on a progress subset.Participants could choose to submit their results to an anonymised leaderboard.
- Evaluation phase: The four-day evaluation phase allowed one submission, evaluated against trials not used during the progress phase.This separated progress feedback from final evaluation trials.
- Platform: Submissions increased steadily during the progress phase, with fewer submissions on weekends and more LA submissions than PA or DF submissions.Week 6 had twice as many submissions as week 3.
6. Challenge results
ASVspoof 2021 showed strong progress across logical access, physical access, and deepfake detection, while exposing task-specific generalization challenges. Participant systems improved substantially over baselines during progress, but deepfake performance degraded markedly in evaluation.
- Cross-task progress: 42% and 32% reductions in min t-DCF were achieved for the LA and PA tasks, respectively, during the progress phase.The corresponding metric was min t-DCF for both tasks.
- Logical access: 19 of 42 LA submissions outperformed baseline B03, with team T23 achieving a min t-DCF of 0.2177 and an EER of 1.32%.B03 had a min t-DCF of 0.3445.
- Physical access: The best PA submission improved from around 1.0 to around 0.7 during the first 10 days, then showed no further progress.It surpassed baseline B01 on the second day.
- Physical access: 13 of 23 PA submissions outperformed baseline B01, while team T07 achieved a min t-DCF of 0.68 and an EER around 24%.The task was described as more challenging than previous editions.
- Deepfake: 18 of 30 DF submissions beat the best baseline, but team T23’s evaluation EER was 15.64% after progress-phase EER fell from 11.6% to 0.10%.The evaluation results were substantially higher than those in the progress phase, indicating overfitting to the progress partition.
- Statistical significance: Statistical differences were significant among top LA and PA systems, whereas the top two DF systems did not differ significantly.Lower-ranked systems also showed similarities to higher-ranked systems in LA and PA.
7. Conclusion
ASVspoof 2021 comprised three detection tasks and introduced greater complexity through new tasks and more challenging data. Despite these conditions and the absence of matched training or development data, results were encouraging, while PA and DF exposed distinct difficulties.
- ASVspoof 2021 comprised logical access, physical access, and speech deepfake detection tasks.
- The challenge increased complexity through a new task, more practical data, and no new matched training or development data.
- 0.2177 minimum t-DCF and 1.32% EER were the lowest reported LA scores.
- PA remained difficult because replay attacks crossed multiple rooms with reverberation and additive noise in bona fide and spoofed data.
- DF evaluation performance was notably worse than progress performance, suggesting a high degree of overfitting.
- Further detailed evaluation analysis was planned for the ASVspoof 2021 workshop and subsequent work.