Source-linked AI summary
Introducing the VoicePrivacy Initiative
Natalia Tomashenko, Brij Mohan Lal Srivastava, Xin Wang, Emmanuel Vincent, Andreas Nautsch, Junichi Yamagishi, Nicholas Evans, Jose Patino, Jean-François Bonastre, Paul-Gauthier Noé, Massimiliano Todisco
TL;DR
The paper addresses the lack of standardized definitions, attack models, datasets, and metrics for evaluating speech anonymization. It formulates the VoicePrivacy 2020 task, introduces evaluation resources and two baselines, and reports privacy gains alongside substantial utility costs. The results show increased speaker-verification EER but also higher ASR word error rates after anonymization.
Problem
Speech contains extensive personal information, but anonymization lacks formal definitions, common datasets, protocols, and meaningful evaluation metrics.
Method
The paper defines a speech-anonymization challenge with attack models, datasets, objective metrics, and two open-source baseline systems.
Results
Anonymization raises speaker-verification EER to 47–58% with original enrollment data, while increasing ASR WER by 19 to 67% relative.
Takeaways & Limitations
The challenge provides a common protocol for comparing anonymization systems while exposing a privacy–utility trade-off in the baselines.
Takeaways & Limitations
The evaluated attack models and downstream goals are not exhaustive, and additional experiments are needed for settings such as anonymized ASR training data.
Abstract
from arXiv · showhide
The VoicePrivacy initiative aims to promote the development of privacy preservation tools for speech technology by gathering a new community to define the tasks of interest and the evaluation methodology, and benchmarking solutions through a series of challenges. In this paper, we formulate the voice anonymization task selected for the VoicePrivacy 2020 Challenge and describe the datasets used for system development and evaluation. We also present the attack models and the associated objective and subjective evaluation metrics. We introduce two anonymization baselines and report objective evaluation results.
1. Introduction
Speech data exposes extensive personal information, including speaker identity, motivating privacy-preserving technologies. The VoicePrivacy 2020 Challenge addresses anonymization by defining the task, evaluation framework, and baseline systems.
- Speech can reveal demographic, geographic, health, emotional, political, religious, and identity information.
- Anonymization suppresses personally identifiable speech attributes while retaining other attributes.
- Existing anonymization methods include noise addition, speech transformation, voice conversion, speech synthesis, and adversarial learning.
- The field lacks formal definitions, attack models, common datasets, protocols, and meaningful metrics for comparing anonymization.
- The VoicePrivacy 2020 Challenge focuses on speech anonymization and provides a general reference alongside participant evaluation materials.
- The paper covers task formulation, attack models, datasets, metrics, two baseline systems, and objective evaluation results.
2. Anonymization task and attack models
VoicePrivacy frames anonymization as a privacy game: speakers publish speech that supports downstream goals while attackers use accessible data and prior knowledge to infer identity. The challenge specifies requirements, attacker access, and privacy and utility assessments.
- A privacy scenario is determined by the data, personal information, downstream goals, attacker-accessible data, and attacker prior knowledge.
- Speakers seek to hide identity while preserving other downstream goals, whereas attackers seek speaker identification from one or more utterances.
- Anonymized trial utterances should be waveform outputs that conceal identity, minimally distort other characteristics, and consistently map speakers to pseudo-speakers.
- Utility is assessed with ASR error and subjective intelligibility and naturalness, while some consistency and downstream ASR-training goals are deferred to post-evaluation.
- Attackers may access anonymized trials and original or anonymized enrollment utterances, but not the anonymization system itself.
- Objective and subjective speaker verifiability and linkability metrics represent different attacker models and enrollment conditions.
3. Datasets
The challenge uses public speech corpora across training, development, and evaluation, with trial and enrollment partitions designed around its attack models. The datasets also distinguish common and different trial subsets for VCTK development data.
- Training uses 2,800 hours of VoxCeleb-1,2 plus 600-hour subsets of LibriSpeech and LibriTTS.
- The development set contains LibriSpeech dev-clean and a VCTK-dev subset.
- Table 1 reports speaker and utterance counts for the training, development, and evaluation sets.
- Development corpora are split into trial and enrollment subsets according to the attack models.
- For LibriSpeech dev-clean, enrollment speakers are a subset of trial speakers; VCTK-dev uses the same speakers for both.
- The evaluation set comprises LibriSpeech test-clean and a VCTK-test subset.
4. Utility and privacy metrics
The evaluation framework combines privacy metrics for speaker verifiability and linkability with utility metrics for satisfying the task’s downstream requirements.
- Objective and subjective privacy metrics assess speaker verifiability and linkability under the defined attack models.
- Objective and subjective utility metrics assess whether the anonymized speech fulfills the requirements specified for the task.
4.1. Objective metrics
Objective evaluation uses ASVeval to measure speaker verifiability and ASReval to measure ASR decoding error. The ASV metrics are compared across original and anonymized trial/enrollment conditions, while ASR evaluation compares original and anonymized trial data.
- Objective speaker verifiability: ASVeval produces log-likelihood ratio scores for objective speaker-verifiability evaluation.It is trained on LibriSpeech trainclean-360 using Kaldi.
- Objective speaker verifiability: EER, Cllr, and Cmin_llr are computed from ASVeval speaker-verification scores.The EER is defined at the threshold where false-alarm and miss rates are equal; Cllr is based on PLDA scores and calibrated LLRs.
- Objective speaker verifiability: ASV metrics are compared for original trials with original enrollment, anonymized trials with original enrollment, and anonymized trials with anonymized enrollment.The number of target and impostor trials is reported in Table 2.
- ASR decoding error: ASReval outputs word error rate for decoding evaluation using a Kaldi LibriSpeech ASR system.The system uses a factorized TDNN-F acoustic model and a trigram language model.
4.2. Subjective metrics
Subjective evaluation covers speaker verifiability, speaker linkability, speech intelligibility, and speech naturalness. Listening tests use similarity ratings, spatial placement, and intelligibility or naturalness ratings to assess anonymized speech.
- Subjective speaker verifiability: Subjective speaker verifiability is assessed by rating the similarity between anonymized trial utterances and distinct original enrollment utterances.Listeners use a 1–10 scale from different speakers to the same speaker with highest confidence.
- Speaker linkability: Speaker linkability is assessed by placing anonymized utterances from different speakers in a space according to perceived speaker similarity.Each utterance is represented as a point, with distances expressing subjective speaker dissimilarity.
- Speech intelligibility: Speech intelligibility is rated for anonymized trial utterances and original enrollment utterances on a 1–10 scale.A rating of 1 means totally unintelligible and 10 means totally intelligible.
- Speech naturalness: Naturalness of anonymized speech is rated on a 1–10 scale from totally unnatural to totally natural.
5. Baseline software and results
The paper provides two open-source anonymization baselines and evaluates the primary baseline for speaker privacy and speech-recognition utility. The primary baseline substantially increases speaker-verification error but also increases ASR error, while the secondary baseline performs worse.
- Anonymization baselines: The primary baseline extracts x-vector, pitch, and bottleneck features, anonymizes each x-vector using distant pool vectors, and synthesizes speech from the anonymized speaker representation and original content-related features.The system uses x-vector anonymization followed by speech synthesis; the external pool is LibriTTS train-other-500.
- Anonymization baselines: The secondary baseline uses a simpler formant-shifting approach and is provided as additional inspiration.
- Objective evaluation results: Primary-baseline anonymization greatly increases EER across datasets, with original-enrollment EERs of 47 to 58%, near or above the 50% chance value.Anonymized enrollment produces lower EERs of 26 to 37%, suggesting that F0+BN features retain speaker information under that attack.
- Objective evaluation results: Primary-baseline anonymization increases ASR WER by 19 to 67% relative, although absolute WER remains below 7% on LibriSpeech and 16% on VCTK.
- Objective evaluation results: The secondary baseline produces inferior results, leaving substantial potential for challenge participants to improve over both baselines.
6. Conclusions
The VoicePrivacy 2020 Challenge defines a complete protocol for voice anonymization, including attack models, datasets, metrics, and two open-source baseline systems. The authors identify non-exhaustive downstream goals and attack models as an important boundary for future evaluation.
- Conclusions: The initiative formulates voice anonymization as a game between users and attackers and provides three possible attack models, datasets, evaluation metrics, and two open-source baseline systems.
- Conclusions: The listed downstream goals and attack models are not exhaustive, including anonymized data collection for ASR training and attacks that retrain ASV systems on anonymized data.