Source-linked AI summary
ASVspoof 2019: spoofing countermeasures for the detection of synthesized, converted and replayed speech
Andreas Nautsch, Xin Wang, Nicholas Evans, Tomi Kinnunen, Ville Vestman, Massimiliano Todisco, Héctor Delgado, Md Sahidullah, Junichi Yamagishi, Kong Aik Lee
TL;DR
ASVspoof 2019 examines how well spoofing countermeasures protect automatic speaker verification against synthesized, converted, and replayed speech. It evaluates top single and ensemble submissions across logical- and physical-access scenarios using tandem detection cost, finding substantial advances but weaker fusion and real-replay performance in important conditions.
Problem
Automatic speaker verification is vulnerable to spoofing, creating a need for countermeasures that reliably detect attacks across changing conditions.
Method
The paper analyzes challenge results from top single and fused countermeasure submissions across logical- and physical-access scenarios using the ASV-centric t-DCF metric.
Results
Countermeasure performance advanced substantially, especially for logical access, with the best primary system reaching a minimum t-DCF of 0.0692 versus an ASV floor of 0.0627.
Takeaways & Limitations
Fusion is particularly effective for logical access, whereas physical-access fusion shows less complementarity and real replay with additive noise remains substantially harder than simulated replay.
Takeaways & Limitations
Real replay data with additive noise show substantial performance gaps relative to simulated, noise-free data, and the paper reports limited understanding of spoofing artefacts and cues.
Abstract
from arXiv · showhide
The ASVspoof initiative was conceived to spearhead research in anti-spoofing for automatic speaker verification (ASV). This paper describes the third in a series of bi-annual challenges: ASVspoof 2019. With the challenge database and protocols being described elsewhere, the focus of this paper is on results and the top performing single and ensemble system submissions from 62 teams, all of which out-perform the two baseline systems, often by a substantial margin. Deeper analyses shows that performance is dominated by specific conditions involving either specific spoofing attacks or specific acoustic environments. While fusion is shown to be particularly effective for the logical access scenario involving speech synthesis and voice conversion attacks, participants largely struggled to apply fusion successfully for the physical access scenario involving simulated replay attacks. This is likely the result of a lack of system complementarity, while oracle fusion experiments show clear potential to improve performance. Furthermore, while results for simulated data are promising, experiments with real replay data show a substantial gap, most likely due to the presence of additive noise in the latter. This finding, among others, leads to a number of ideas for further research and directions for future editions of the ASVspoof challenge.
1 INTRODUCTION
ASVspoof 2019 addresses the vulnerability of automatic speaker verification to spoofing by evaluating countermeasures across synthesis, voice conversion, and replay attacks. The paper focuses on challenge results, system performance, generalisation, and future priorities.
- Motivation: ASV systems are vulnerable to spoofing attacks that can enable illegitimate access to protected resources, services, or devices.Countermeasures are auxiliary systems intended to detect and deflect such attacks.
- Challenge scope: ASVspoof 2019 was the first edition to include speech synthesis, voice conversion, and replay attacks in separate logical and physical access scenarios.The challenge also considered newer speech-generation technologies and introduced several advances over earlier editions.
- Evaluation: The 2019 edition shifted from equal error rate to the ASV-centric tandem detection cost function as its primary metric.The t-DCF reflects the impact of both spoofing and countermeasures on ASV performance.
- Evaluation: Strong countermeasure performance requires generalisation to spoofing attacks not seen in training or development data.The challenge therefore evaluates reliability beyond familiar attack conditions.
- Paper focus: The paper reports the top five single and fused systems for both challenge scenarios, analyses successful countermeasure techniques, and identifies priorities for ASVspoof 2021.The challenge included results from four spoofed-audio setups spanning logical access and simulated or real replay.
2 CHALLENGE OUTLINE
The challenge evaluates logical-access synthesis and conversion attacks alongside physical-access replay attacks under controlled and real conditions. It ranks participant countermeasures with normalized minimum t-DCF, a metric that combines CM and ASV operating costs.
- Scenarios: Logical access covers direct injection of synthetic or converted speech, whereas physical access presents replayed recordings through a fixed microphone.Physical-access data include simulated rooms and replay configurations, with real replay recordings collected at three sites.
- Scenarios: The logical-access database uses 17 diverse text-to-speech, voice-conversion, and hybrid systems with varied waveform-generation and acoustic-modeling methods.The attack technologies range from waveform concatenation to neural waveform modeling and from Gaussian mixtures to sequence-to-sequence networks.
- Rules: Participants submitted primary and single systems for both development and evaluation subsets, while ensemble classifiers were permitted only for primary systems.Systems had to process trial segments independently without using other trials for normalization or adaptation.
- Metrics: The primary metric is normalized minimum t-DCF, with a fixed ASV threshold and a variable countermeasure threshold.Its coefficients incorporate priors, detection costs, and ASV miss, false-alarm, and spoof false-alarm rates.
- Metrics: A normalized t-DCF of 0 denotes error-free ASV and countermeasure systems, while 1 denotes no improvement over an uninformative default countermeasure.The ASV floor represents an error-free countermeasure combined with an imperfect ASV system.
- Metrics: Performance is primarily pooled across attack conditions, with max min t-DCF additionally reporting the worst per-condition result in selected analyses.The minimum reflects oracle countermeasure calibration, while the maximum identifies the highest per-condition cost.
3 LOGICAL ACCESS SCENARIO
The logical-access results show substantial gains over baselines from both single systems and, especially, system fusion. The strongest systems combine diverse representations and neural classifiers, with fusion producing uneven benefits across teams.
- Submissions: 48 logical-access single-system submissions were made, and the paper discusses the top five single and primary systems.Figures and tables identify systems using anonymized team identifiers and compare them with the two baselines.
- Single systems: The best T45 single system improves on baseline B02 by 40% relative in t-DCF.T45 uses LFCC features with an LCNN rather than the B02 GMM-UBM classifier.
- Primary systems: The best T05 primary system improves on the best T45 single system by 56% relative, or 73% relative to B02.Its minimum t-DCF is 0.0692, close to the ASV floor of 0.0627.
- Primary systems: The top three primary systems combine at least five subsystems using diverse cepstral and spectral features and at least one DNN-type classifier.Examples include DFT, DCT, LFCC, CQCC, spectrogram, and raw-audio representations across CNN, ResNet, and related architectures.
- System combination: Fusion substantially improves T05 relative to its single system, while the corresponding gains for T45 and T24 are more modest.The contrast indicates that system-combination benefits vary across teams and component systems.
4 PHYSICAL ACCESS SCENARIO
The physical access scenario compares top single and primary replay-attack countermeasures, emphasizing their architectures and results against baselines. Top single systems rely on spectral features and DNN classifiers, while primary systems combine spectral and cepstral systems with limited additional benefit.
- Single systems: Top-5 single-system architectures include spectral frontends such as Mel-grams, CQT-grams, group-delay grams, and log-power CQT-grams with neural classifiers.The illustrated systems include modified ResNet and LCNN architectures.
- Primary systems: Top primary systems fuse multiple subsystems using varied frontends and DNN backends, combining scores by averaging or logistic regression.Examples use Mel, CQT, modified group-delay, LFCC, DCT, log-DFT, and CQCC features.
- Results: Table 3 reports PA results for top single and primary submissions, baselines, the ASV floor, and hidden real-replay data.PA worst cases are indexed by both attack identifier and environment identifier.
- Results: All top-5 single systems outperform both baselines, with the best T28 system achieving a 58% relative t-DCF improvement over baseline B01.The top systems are closely grouped in performance, and only one uses data augmentation.
- Results: Primary-system fusion improves only modestly over the best single system: T28 gains 2% relative, or 59% relative to B01.The best primary system reaches a min t-DCF of 0.1437 versus the ASV floor of 0.1354.
5 ANALYSIS
The analysis shows that countermeasure performance is often dominated by specific spoofing attacks or acoustic conditions. Fusion is more effective for logical access than physical access, although oracle fusion reveals additional potential for PA systems.
- 5.1 Generalisation to unseen attacks: Performance for LA varied substantially across varied attacks, while unknown-attack performance was similar to or better than known-attack performance.The worst-case varied attack was linked to a voice-conversion system combining a VAE acoustic model with direct waveform modification.
- 5.1 Generalisation to unseen attacks: As T60 reverberation increased in PA, min t-DCF increased even though countermeasure performance improved relative to the ASV floor.The degradation was therefore associated with tandem-system behavior rather than worsening CM performance alone.
- 5.2 Worst case scenario: Worst-case max min t-DCFs were substantially higher than pooled min t-DCFs for both scenarios.For LA, attack A17 produced a max min t-DCF over 6 times higher than the min t-DCF; PA performance was especially poor under small-room, high-reverberation, long-distance conditions and close-proximity AA or AC attacks.
- 5.3 Fusion performance: LA fusion was more beneficial than PA fusion: the best primary systems reached min t-DCFs of 0.0692 and 0.1437, respectively.The corresponding best single-system values were 0.1562 for LA and 0.1470 for PA.
- 5.3 Fusion performance: Oracle fusion substantially improved PA performance toward the ASV floor, whereas LA primary-system fusion improved only marginally.The PA result indicates scope for gains from more complementary subsystems; participant ensembles appeared insufficiently complementary.
- 5.4 Progress: LA progress kept pace with advances in TTS and VC, while apparent PA progress was confounded by differences in database noise conditions.The 2019 PA database lacked additive and convolutional channel noise present in the 2017 database, and real replay results later exposed a substantial gap.
6 RESULTS FOR REAL REPLAY RECORDINGS
The paper compares simulated and concealed real replay recordings to assess whether countermeasures trained and evaluated on simulation transfer to real data. Results differ substantially, with additive ambient noise identified as a likely contributor.
- Data: Real replay data comprised recordings from 26 speakers across three rooms and two talker-to-ASV distance categories.The collection contained 540 bona fide utterances and 2160 replay utterances.
- Results: Real replay results differed substantially from simulated-data results, with the best top-five-system real-data t-DCF equal to 0.3855 from baseline B01.The discrepancy suggests overfitting to simulated data or insufficient robustness to background noise.
7 FUTURE DIRECTIONS
The paper proposes future ASVspoof editions examine nuisance variation more directly, especially ambient and channel noise. It notes that controlled noise addition may be useful but that realistic noise simulation requires care because noise can affect speech production.
- Additive noise and channel variability: Future editions should study how ambient and channel noise affect countermeasure performance.The paper identifies controlled addition of nuisance variation as a possible approach, while cautioning that additive-noise simulation can influence speech production through effects such as the Lombard reflex.
- Additive noise and channel variability: Noise treatment in future challenge designs requires careful reflection because additive noise is more complex to simulate than telephony coding and compression effects.The paper contrasts readily simulated coding effects with the speech-production consequences of additive noise.
Quality of TTS/VC training data
The paper identifies the benign acoustic conditions used to train all ASVspoof 2019 TTS and VC systems as a scope boundary. It recommends more realistic training data in future editions.
- Quality of TTS/VC training data: All ASVspoof 2019 TTS and VC systems were trained with data recorded in benign acoustic conditions.The paper states that this setup is not representative of in-the-wild adversaries who may have only noisy training or adaptation data.
- Quality of TTS/VC training data: Future editions should consider TTS and VC attacks generated from more realistic data.The paper notes that such attacks may be less effective against ASV systems and more easily detectable.
- Diversified spoofing attacks: ASVspoof should diversify beyond its current view of spoofing by considering impersonation, non-speech, adversarial, localized, and blended attacks.The paper gives examples including replay in logical access and TTS/VC-generated speech replayed in physical access.
Joint CM+ASV score calibration
ASVspoof 2019’s tandem assessment creates three decision outcomes, making binary score calibration inadequate and motivating joint CM+ASV optimisation and calibration. Reproducibility of submitted systems is also identified as an unresolved concern.
- Joint CM+ASV score calibration: Three decision outcomes—target, non-target, and spoof—make existing binary score-calibration approaches unsuitable.The target and non-target outcomes are bona fide, while spoof is distinct.
- Joint CM+ASV score calibration: Future work could investigate joint CM+ASV system optimisation and calibration under the ASV-centric min t-DCF assessment.
- Joint CM+ASV score calibration: Anecdotal evidence indicates that some ASVspoof results are unreproducible, motivating system-description review or harmonised reporting.Suggested reporting includes details of the fusion scheme.
Explainability
Anti-spoofing results show promising detection potential, but the field lacks sufficient attention to explainability and understanding of the cues separating bona fide from spoofed speech. Security assessment also remains disconnected from the adopted t-DCF framework.
- Explainability: Anti-spoofing research has learned little about the artefacts or cues distinguishing bona fide from spoofed speech.Future work revealing these cues may help the community design better countermeasures.
- Explainability: The t-DCF metric does not meet Common Criteria security-assessment standards.Common Criteria uses category-based attack-potential points reflecting equipment, expertise, time, and system knowledge.
- Explainability: Bridging t-DCF and Common Criteria requires determining attack potentials for ASVspoof attacks.The paper identifies time, expertise, resources, and familiarity with the system as relevant questions.
Challenge model
ASVspoof has become a demanding, multi-institute challenge requiring substantial coordination and funding. The organisers therefore plan changes to its future event format and support model.
- Challenge model: ASVspoof 2019 required coordination among 6 organising institutes and 19 data contributors.The organisers anticipate needing additional industrial, institutional, or public funding.
- Challenge model: The initiative’s organisational demands motivate efforts to attract additional funding for future editions.
- Challenge model: Future ASVspoof editions will likely transition from an INTERSPEECH special session to a satellite workshop format.The organising team described 2019 as the last edition run as an INTERSPEECH special session.
8 CONCLUSIONS
ASVspoof 2019 expanded anti-spoofing evaluation to logical and physical access scenarios and adopted tandem detection costs, while reporting improved countermeasure performance. Gains were stronger for logical access, but real replay conditions and specific attacks or environments remain difficult, motivating broader and more interpretable future evaluations.
- 8 CONCLUSIONS: ASVspoof 2019 was the first edition to evaluate logical and physical access together and adopt t-DCF as its default metric.The paper focuses on challenge findings, trends, and top-performing systems for each scenario.
- 8 CONCLUSIONS: Countermeasure performance improved substantially, with larger gains for logical access than physical access.The edition also shifted toward deep architectures and ensemble systems.
- 8 CONCLUSIONS: Tandem systems incurred high detection costs under specific spoofing attacks or acoustic environments.Costs could stem from either countermeasures or automatic speaker verification systems.
- 8 CONCLUSIONS: Real replay data with additive noise showed substantial gaps relative to simulated, noise-free data.The paper identifies nuisance variation such as additive noise as a challenge for real-world performance.
- 8 CONCLUSIONS: Future editions will diversify spoofing and blended attacks while giving greater attention to optimisation, reproducibility, explainability, and standards.