PaperlayerView exact revision on arXiv

Source-linked AI summary

The Third VoicePrivacy Challenge: Preserving Emotional Expressiveness and Linguistic Content in Voice Anonymization

Natalia Tomashenko, Xiaoxiao Miao, Pierre Champion, Sarina Meyer, Michele Panariello, Xin Wang, Nicholas Evans, Emmanuel Vincent, Junichi Yamagishi, Massimiliano Todisco

arXiv:2601.11846v1cs.CLcs.SDeess.AS

TL;DR

Voice anonymization must hide speaker identity without sacrificing linguistic and emotional utility, but evidence on this multi-objective trade-off remains challenging. The paper analyzes the 2024 VoicePrivacy Challenge through its task, attack model, metrics, baselines, and participant systems, finding that sophisticated architectures can achieve strong privacy while balancing utility. It also identifies limitations in current evaluation metrics and the difficulty of satisfying all requirements simultaneously.

  • Problem

    Speech contains sensitive identity-related information, creating a need to conceal speaker identity while preserving linguistic and emotional content for downstream tasks.

  • Method

    The paper presents the 2024 challenge framework, attack model, evaluation protocols, baseline systems, and analyses of participant submissions.

  • Results

    Only 6 submitted systems achieved EER ≥40%, while 4 met all challenge requirements; cascaded systems reached near-perfect anonymization but sacrificed emotion preservation.

  • Takeaways & Limitations

    Neural codec methods, disentanglement, self-supervised learning, and emotion-aware design improve the privacy-utility balance over simpler approaches.

  • Takeaways & Limitations

    Automatic utility metrics can diverge from perceptual quality, since lower WER does not necessarily indicate better quality or intelligibility.

Abstract

from arXiv · show

We present results and analyses from the third VoicePrivacy Challenge held in 2024, which focuses on advancing voice anonymization technologies. The task was to develop a voice anonymization system for speech data that conceals a speaker's voice identity while preserving linguistic content and emotional state. We provide a systematic overview of the challenge framework, including detailed descriptions of the anonymization task and datasets used for both system development and evaluation. We outline the attack model and objective evaluation metrics for assessing privacy protection (concealing speaker voice identity) and utility (content and emotional state preservation). We describe six baseline anonymization systems and summarize the innovative approaches developed by challenge participants. Finally, we provide key insights and observations to guide the design of future VoicePrivacy challenges and identify promising directions for voice anonymization research.

1. Introduction

The 2024 VoicePrivacy Challenge extends voice anonymization beyond speaker identity by requiring preservation of linguistic content and emotional state. It presents the challenge framework, participant systems, comparative analyses, and implications for future research.

  • Speech recordings contain sensitive information including speaker identity, health status, personality, and social background, motivating privacy-preserving speech technology.
  • VoicePrivacy develops anonymization methods that disguise speaker identity while preserving attributes needed for downstream speech tasks.
  • The 2024 edition newly required preservation of emotional state alongside linguistic content and other paralinguistic attributes.The requirement targets applications such as third-party speech analytics in call centers.
  • Preserving richer attributes while concealing identity increases task difficulty and encourages more balanced, progressively demanding anonymization systems.
  • The paper describes the challenge framework, evaluates 36 participant systems, compares them with earlier editions, and discusses privacy-evaluation limitations and future directions.

2. Challenge design

The challenge frames voice anonymization as a privacy-utility game: systems must replace speaker identity while preserving linguistic and emotional content under a semi-informed attacker. Evaluation uses public speech and emotional-speech resources with ASV, ASR, and SER measures.

  • 2.1. Voice anonymization task: The challenge asks systems to output speech waveforms that conceal utterance-level speaker identity while preserving linguistic and emotional content.
  • 2.2. Evaluation: Privacy is measured by ASV EER, while linguistic and emotional utility are measured by ASR WER and SER UAR.
  • 2.2. Attack model: An attacker uses enrollment utterances and the same anonymization system to adapt ASV and re-identify speakers in anonymized trial utterances.
  • 2.3. Training resources: Training resources are publicly available, with fixed development and evaluation data and flexible participant training resources subject to challenge constraints.
  • 2.3. Data: Development and evaluation use LibriSpeech for ASV and ASR and IEMOCAP emotional speech for SER evaluation.LibriSpeech provides read English speech, while IEMOCAP contains conversational emotional speech from ten actors.

3. Privacy and utility evaluation

The 2024 challenge evaluates anonymization by jointly measuring speaker de-identification, linguistic-content preservation, and emotional-state preservation across multiple privacy operating points.

  • Evaluation metrics: EER measures speaker de-identification, while WER and UAR measure linguistic-content and emotional-state preservation, respectively.These metrics rely on ASV, ASR, and SER systems.
  • Privacy evaluation: The privacy evaluator uses an ECAPA-TDNN ASV model trained on LibriSpeech-train-clean-360 and adapted to anonymized data for attacker evaluation.The attacker anonymizes enrollment and training data and retrains the ASV system on similarly anonymized input.
  • Privacy evaluation: Higher EER indicates better anonymization because it reflects increased difficulty distinguishing same-speaker from different-speaker trials.EER is defined where false-alarm and miss rates are equal.
  • Scope and limitation: The challenge’s privacy metric specifically measures speaker de-identification rather than complete protection of all personal or sensitive information.Names, locations, metadata, contextual details, and interaction patterns may remain identifying even with perfect speaker de-identification.
  • Utility evaluation: The ASR evaluator is trained on original LibriSpeech data and fixed during evaluation, with lower WER indicating better linguistic-content preservation.WER counts substitution, deletion, and insertion errors against the reference transcription.
  • Utility evaluation: UAR averages recall across four emotion classes and five folds, weighting classes equally so frequent emotions do not dominate evaluation.Higher UAR indicates better preservation of emotional state.
  • Privacy-utility trade-off: Systems are ranked separately by WER and UAR after meeting minimum EER targets of 10%, 20%, 30%, and 40%.This evaluates privacy-utility trade-offs at multiple operating points.

4. Baseline voice anonymization systems

Six baselines span signal processing, neural synthesis, neural audio codecs, and vector-quantized ASR-based pipelines, offering different privacy–utility trade-offs. B1 and B2 carry forward earlier approaches, while B3–B6 introduce newer synthesis and representation strategies.

  • Baseline overview: Six baseline systems include two established systems, B1 and B2, plus four new systems, B3–B6, with varying compromises between anonymization and utility.The new baselines were provided as development inspiration and were intended to offer enhanced anonymization performance.
  • B1: B1 extracts bottleneck, F0, and x-vector features before replacing speaker information and synthesizing anonymized speech at the utterance level.Its bottleneck features represent linguistic content, while F0 and x-vectors represent prosodic and speaker information.
  • B2: B2 uses training-free signal processing, shifting LPC-derived pole positions with the McAdams coefficient to alter speech characteristics.The method performs frame-by-frame LPC analysis and converts coefficients into z-plane pole positions corresponding to spectral peaks.
  • B3: B3 replaces the original speaker embedding with a GAN-generated pseudo-speaker embedding while retaining linguistic and general prosodic information for speech synthesis.The system extracts speaker embeddings, phonetic transcriptions, F0, energy, and phone duration before synthesis.
  • B4: B4 combines semantic tokens with randomly selected pseudo-speaker acoustic prompts, then uses a decoder-only Transformer and neural audio codec to synthesize anonymized waveforms.Semantic tokens encode spoken content, while acoustic tokens provide the voice style of another pseudo-speaker.
  • B5 and B6: B5 and B6 use vector-quantized ASR bottleneck features, F0, and target-speaker representations with HiFi-GAN synthesis to reduce speaker information in linguistic features.B5 uses wav2vec 2.0 with additional TDNN-F layers, whereas B6 uses a 12-layer TDNN-F acoustic model.

5. Submitted anonymization systems

The 2024 challenge drew broad participation and produced 36 submitted systems spanning disentangled voice conversion, kNN-VC, emotion-aware synthesis, pitch manipulation, and other approaches. Most systems explicitly separated linguistic, speaker, prosodic, and emotional representations before resynthesis.

  • Participation and submissions: More than 100 participants from 16 countries formed 40 teams, while 13 teams successfully submitted 36 anonymization systems.Teams submitted between one and six systems targeting different privacy categories and utility performance.
  • Dominant approaches: Most submitted systems use neural voice conversion with separate representations for linguistic content, speaker identity, prosody, and emotion.Speaker features are generally replaced with pseudo-speaker embeddings while other attributes are preserved or modified before synthesis.
  • kNN-VC systems: kNN-VC systems replace an input representation sequence with a similar sequence drawn from another randomly selected speaker, rather than using disentangled speaker embeddings.T8 and T19 use WavLM representations; T19-2 combines outputs from WavLM layers 6 and 12 to improve emotion-related representation.
  • Emotion preservation: Several systems directly target emotion preservation through emotion encoders, emotional similarity losses, or emotion-conditioned synthesis.Examples include T7 emotion encoders, T9’s emotional similarity term, T18’s emotion-conditioned FreeVC, and T25’s alternative emotion-recognition model.
  • Additional techniques: Other submissions modify established pipelines using pitch smoothing, additive Gaussian noise, vector quantization, mutual-information objectives, or disentangled discrete speech units.These techniques target speaker-information reduction while retaining content, prosody, or emotional characteristics.

5.2. Anonymization systems based on neural audio codecs

Neural audio codec systems anonymize speech by representing it with discrete codes that can separate speaker identity from linguistic content and prosody. Submitted systems extend this framework with disentanglement, distillation, pooled or random speaker identities, and prosody transformation.

  • Codec framework: Neural audio codecs transform speech into quantized codes that bottleneck speaker information while capturing linguistic content and prosodic features.Their unified codec framework separates and recombines speech components rather than relying primarily on explicit feature modification.
  • T10: T10 uses disentangled speaker, linguistic, and F0 representations with frame-level emotion distillation and teacher-based distillation for content and speaker information.Its anonymized speaker vector mixes an averaged speaker identity with a randomly generated identity using a controllable weight α.
  • T12: T12 replaces the original FACodec speaker embedding with an anonymized embedding while using pool averaging, cross-gender conversion, and Gaussian noise strategies.FACodec is used to disentangle prosody, content, and other acoustic detail.
  • T17: T17 adds a prosody anonymization module to FACodec that suppresses speaker-specific prosodic features and generates target-speaker prosody and timbre.The method’s dataset-wide speaker-embedding center may conflict with the requirement that trial utterances be processed independently.

5.3. Cascaded ASR+TTS anonymization systems

Cascaded ASR+TTS systems transcribe speech into intermediate text or phonetic representations and resynthesize anonymized speech. Newer variants add emotion, prosody, speaker-selection, or embedding-perturbation mechanisms beyond the traditional text-only cascade.

  • Cascade design: Cascaded systems pass ASR-derived word or phonetic sequences to TTS models that generate anonymized speech waveforms.The cascade uses transcription as an intermediate representation between input speech and resynthesized output.
  • Traditional cascades: Traditional ASR-to-TTS systems do not explicitly preserve prosodic or paralinguistic attributes when passing transcripts to synthesis.This distinguishes them from newer cascaded systems that add dedicated preservation mechanisms.
  • T8-1: T8-1 uses Whisper to derive English transcripts and VITS trained on LibriTTS to synthesize anonymized outputs.Its pipeline represents the conventional ASR-to-TTS cascade.
  • Advanced cascades: Advanced cascades incorporate mechanisms intended to preserve emotion, intonation, and speaking style alongside speech content.These systems extend the basic transcript-driven architecture with paralinguistic conditioning or preservation components.
  • Submitted variants: B3-based cascades use emotion embeddings, cross-gender speaker strategies, or pool-based embedding perturbation to modify speaker identity during synthesis.T30-2 additionally transforms F0 into latent features and combines them with emotion prototypes through cross attention and FiLM conditioning.

5.4. Hybrid anonymization systems

Hybrid anonymization systems combine complementary methods to balance privacy and utility, especially by randomly selecting between cascaded ASR-TTS and kNN-VC approaches for each utterance.

  • Hybrid systems combine multiple anonymization techniques to balance privacy and utility through dynamic method selection.
  • Randomized admixture systems select either cascaded ASR-TTS or kNN-VC for each utterance.
  • Cascaded ASR-TTS more strongly suppresses voice information, whereas kNN-VC better preserves linguistic and emotional content.
  • Several submissions also explored prosody prediction, HiFi-GAN synthesis, randomized F0 variation, and neural-codec speaker-identity mixing.

5.5. Trends

Challenge submissions explored diverse strategies for improving privacy while preserving linguistic content and emotional cues, including hybrid selection, speaker-embedding manipulation, emotion modeling, and content disentanglement.

  • Cascade-based ASR-TTS systems achieve approximately 48% EER while largely preserving linguistic content but compromising utility.
  • Admixture strategies randomly choose anonymization methods per utterance to adjust the privacy-utility trade-off.
  • Speaker embedding anonymization methods: Speaker-embedding systems balance privacy and utility through weighted random mixing, pool-based averaging, cross-gender selection, or single pseudo-speaker selection.
  • Emotion preservation approaches include pretrained emotion encoders, handcrafted acoustic features, GSTs, emotion distillation, and emotion-matched reference selection.
  • Content preservation uses VQ bottlenecks, pretrained SSL representations, and phonetic transcriptions to separate or retain linguistic information.

6. Results

VPC 2024 results reveal substantial privacy–utility trade-offs: systems that strongly conceal speaker identity do not consistently preserve linguistic content and emotion. Architecture choices, especially emotion-aware and hybrid designs, determine performance across operating points.

  • 6.1. Privacy achievement across categories: 28 submitted systems and 4 baselines reached EER ≥10%, while only 6 submitted systems reached EER >40%, with two non-compliant systems among them.The strongest compliant privacy category contained four submitted systems, including T8-1, which approached 50% EER.
  • 6.2.1. Neural codec-based systems: T10-2 achieved the best reported high-privacy balance with EER=40.4%, WER=3.2%, and UAR=60.8%.Its codec, serial disentanglement, and frame-level emotion distillation jointly supported linguistic and emotional utility at EER ≥40%.
  • 6.2.2. Cascaded ASR+TTS systems: T8-1 achieved the strongest privacy at EER=49.5% and preserved WER=3.75%, but its UAR fell to 30.6%.The transcription bottleneck preserves linguistic content while degrading paralinguistic attributes, illustrating a central limitation of text-based anonymization.
  • 6.2.3. Hybrid admixture systems: Hybrid T8 admixture systems maintained WER at approximately 3.2–3.3% while adjusting UAR with EER through per-utterance method selection.T8-5 reached EER=41.3% with UAR closer to pure VC systems at 47%, demonstrating flexible privacy–utility operating points.
  • 6.2.4. VC-based systems: Pure VC systems generally preserved utility well but typically remained below EER=10%, whereas T9 reached EER=34.5%, WER=2.4%, and UAR=61%.Emotion-aware VC systems improved emotion preservation but remained below neural codec performance in the highest privacy categories.
  • 6.3.2. Emotion preservation (UAR): The best and worst rule-compliant systems in the highest privacy category differed by more than 17% in UAR, underscoring the impact of emotion-aware architecture choices.Emotion-preserving systems ranked higher in UAR across privacy categories than systems without emotion-specific designs.
  • 6.3.2. Emotion preservation (UAR): Emotion-specific performance was uneven: sadness was the hardest class, while happiness-biased systems reached 80–94% happiness accuracy but only 29–45% sadness accuracy.T10-1 was comparatively balanced, preserving all emotion classes at 60–70% accuracy.
  • 6.4. Privacy-utility trade-off: Pareto analyses identified T9 and T10-1 as effective approaches for preserving both utility metrics while increasing privacy.T9 achieved ΔUAR = 14% and ΔWER = 28%, while T10-1 achieved ΔUAR = 9% and ΔWER = 39%.

7. Discussion and future perspectives

The discussion identifies important weaknesses in current privacy and utility evaluation, while proposing stronger attacks, broader metrics, more diverse datasets, and emerging anonymization approaches.

  • Evaluation reliability: EER-based privacy estimates can be overstated when attackers generalize poorly or use constrained data, features, models, and hyperparameters.External corpora, non-timbral features, and stronger models can produce stronger attacks.
  • Evaluation reliability: Stronger attacker models substantially reduce measured privacy and can change system rankings, demonstrating the need for reliable evaluation.In the Attacker Challenge, systems initially reaching EER≥40% fell to EER≥20% against stronger attacks.
  • Evaluation metrics: EER does not capture legal privacy concepts such as linkability, singling out, or inference, and can mask user-specific re-identification risks.Alternative linkability- and singling-out-based metrics indicate greater residual privacy risks than EER estimates suggest.
  • Evaluation metrics: WER may not align with subjective intelligibility and underrepresents naturalness, prosody, and acoustic quality.The discussion recommends multidimensional metrics combining objective and subjective utility with downstream-task performance.
  • Datasets and future directions: Current datasets may bias comparisons because privacy and content are evaluated on one corpus while emotion is evaluated on another, with limited language and accent coverage.Future datasets should include more ages, accents, languages, speech styles, emotional states, acoustic conditions, overlapping speech, and low-resource languages.
  • Emerging approaches: Recent research emphasizes disentanglement, multi-attribute control, speaker-embedding-free methods, multilingual generalization, and application-specific anonymization.These directions target control over identity, emotion, demographic attributes, and other speaker characteristics across broader scenarios.

8. Conclusions

The 2024 challenge attracted broad participation and advanced privacy-preserving speech systems that preserve linguistic and emotional content. Results also expose a difficult privacy–utility balance and the need for independently verified evaluation.

  • Challenge outcomes: 36 submissions from 13 teams across 16 countries more than doubled participation compared with the previous challenge edition.Participants came from academic and non-academic organizations and used diverse neural architectures.
  • Challenge outcomes: Only six submitted systems achieved EER ≥40%, and only four met all other challenge requirements, underscoring the difficulty of preserving utility alongside privacy.Cascaded ASR+TTS achieved near-perfect anonymization but sacrificed emotion preservation.
  • Technical findings: Neural codec systems using multi-stage disentanglement and frame-level emotion distillation achieved a better privacy–utility balance than cascaded ASR+TTS systems.Systems without explicit emotion-aware components generally underperformed systems using emotion distillation, encoder integration, or emotion-matched pseudo-speaker selection.
  • Technical findings: Five dominant trends include neural codecs, self-supervised or pretrained models, multi-stage attribute disentanglement, dynamic method selection, and emotion-aware design.Emotion-aware systems use encoders, distillation mechanisms, and emotion-conditioned synthesis.
  • Quantitative results: 49.5% EER was achieved by T8-1 versus 34% for baseline B5, while T10-1 reached 65% UAR versus 54% for B2 and T9 reduced WER to 2.37% versus 2.9% for B1.These are the reported best submitted-system comparisons for the individual metrics.
  • Evaluation reliability: Four systems initially reaching EER ≥40% later fell below 20% against stronger attacks, motivating independent and continuous two-stage evaluation.The conclusion recommends separating anonymization-system and attacker development teams.

A. Training resources

The challenge allowed a broad collection of speech datasets, pretrained models, and software resources for developing anonymization systems, alongside specified baseline resources.

  • Resource scope: Table 8 lists permitted training resources for anonymization systems, with stars marking multiple requests from different teams.The resource list spans datasets, models, and pretrained software.
  • Models and software: Permitted pretrained models included WavLM, wav2vec2 emotion models, ContentVec, w2v-BERT, ECAPA2, ECAPA-TDNN, NaturalSpeech 3, and NVIDIA HiFi-GAN.These models support representation learning, speaker processing, synthesis, and vocoding.
  • Datasets: Core speech datasets included CommonVoice, ESD, LibriSpeech, CREMA-D, RAVDESS, VCTK, EMO-DB, LJSpeech, Libri-light, LibriTTS, CMU-MOSEI, MUSAN, RIR, VGAF, and MSP-Podcast.These resources cover speech, emotion, noise, reverberation, and related modeling needs.
  • Models and software: Additional listed resources included Encodec, Resemblyzer, DISSC, and datasets such as VoxCeleb and CREMA-D-related speech resources.The list combines pretrained components with data for speaker, content, and acoustic modeling.

B. Results

Results are summarized using a normalized matrix across privacy and utility metrics, with systems ranked by their combined normalized performance.

  • Overall ranking: Figure 13 ranks systems using normalized ΔUAR, ΔWER, and ΔEER values relative to original-data results.The matrix combines the three normalized metrics into an overall score.
  • Normalization: The heatmap uses red-to-green colors to show poorer-to-excellent performance across the ranked systems.The combined score averages normalized performance over all three metrics.
  • Reported metrics: The results table reports EER and WER on LibriSpeech and UAR on IEMOCAP for development and test sets.These metrics cover privacy, linguistic content, and emotional utility.
Loading 2601.11846v1…