Source-linked AI summary
DP-VOXLET: Provable Speaker Anonymization for Disentangled Speech Representations
Ivoline Ngong, Jack D'Iorio, Hailey Schoppe, Christopher Liberatore, Nichole Schimanski, Taisa Kushner, Joseph P. Near
TL;DR
Existing speaker-anonymization systems can perform well empirically but lack provable privacy guarantees. This paper defines speaker differential privacy and implements a carefully designed speaker-representation perturbation mechanism in DP-VOXLET, achieving competitive anonymization utility while providing provable guarantees.
Problem
Existing speaker-anonymization systems achieve strong empirical performance but lack provable privacy guarantees.
Method
DP-VOXLET applies a randomized perturbation to disentangled speaker representations within a differentially private variational autoencoder framework.
Results
DP-VOXLET is competitive with existing anonymization systems on empirical benchmarks while providing provable privacy guarantees, with a highest EER of 41.2% in the reported challenge experiments.
Takeaways & Limitations
Speaker differential privacy provides theoretical bounds on adversarial speaker re-identification while maintaining high-quality output speech.
Takeaways & Limitations
The approach relies on disentanglement that may be imperfect; speaker information leaking into semantic contents can prevent perfect speaker privacy.
Abstract
from arXiv · showhide
Systems for speaker anonymization obfuscate the speaker of an utterance, while maintaining its original semantic contents and prosody. Recent solutions for speaker anonymization rely on learned representations that disentangle an utterance into semantic contents and speaker properties. To anonymize an utterance, these systems replace the speaker properties while leaving the semantic contents unchanged---an approach that can produce strong results on empirical measures of privacy. In this work, we introduce speaker differential privacy, a formal definition of speaker anonymization based on the framework of differential privacy, and a mechanism for speaker anonymization that provably satisfies the definition. In contrast to prior heuristic-based anonymization systems, our approach enables a provable lower bound on re-identification success rate (e.g. equal error rate) for any possible adversary. We implement our approach in a framework that is compatible with existing disentangled representations. Compared to the prior work on differential privacy for speaker anonymization, our approach achieves significantly higher utility.
1. Introduction
Speaker anonymization aims to hide speaker identity while preserving semantic content and prosody. Existing disentangled-representation systems perform well empirically, but this work introduces a differential-privacy framework and perturbation mechanism for provable privacy guarantees.
- Speaker anonymization seeks to obfuscate the speaker while preserving an utterance’s semantic contents and prosody.
- Disentangled representations separate semantic contents from speaker properties, enabling anonymization by replacing speaker information while leaving semantics unchanged.Recent systems may also disentangle emotional information and select replacement speakers from a predefined pool.
- Existing anonymization systems perform well on empirical benchmarks but do not provide provable privacy guarantees.
- This work introduces speaker differential privacy and DP-VOXLET, a general framework for applying it to existing disentangled speech representations.The framework defines speaker privacy formally and uses a perturbation mechanism designed to satisfy that definition.
- DP-VOXLET perturbs the disentangled speaker representation with carefully designed randomization rather than selecting a replacement speaker from a predefined pool.
- The approach targets the utility problem caused by perturbing speaker embeddings by using a differentially private variational autoencoder that remains in a well-conditioned embedding-space region.Empirical evaluation on the 2024 Voice Privacy Challenge benchmark is reported as competitive with state-of-the-art anonymization systems while providing provable guarantees.
2. Formal Speaker Anonymization
The paper formalizes speaker anonymization through speaker differential privacy and a Gaussian mechanism operating on disentangled speaker representations. Under the stated disentanglement assumption, the mechanism provides adversary-independent privacy guarantees and an EER lower bound controlled by the privacy parameter.
- 2.2. Privacy Definition: Speaker differential privacy limits an adversary’s ability to distinguish which of two possible speakers produced an anonymized utterance, using a trade-off function over rejection rules.The definition extends Gaussian differential privacy to utterances processed by encoder ϕ and decoder θ.
- 2.1. System Model, Assumptions, and Limitations: The system model assumes an encoder ϕ that separates an utterance into speaker attributes S and semantic contents C, with a decoder θ reconstructing the waveform.The reconstruction assumption is θ(ϕ(x)) = x to arbitrary accuracy.
- 2.1. System Model, Assumptions, and Limitations: Privacy requires that C contain no speaker-identity information, so utterances with the same content have identical content representations across speakers.This separation is necessary for the privacy guarantees.
- 2.1. System Model, Assumptions, and Limitations: The disentanglement assumption is unprovable in practice, and leakage of speaker information into C means the approach cannot ensure perfect speaker privacy.The authors report that empirical results suggest this leakage is minimal in practice.
- 2.2. Privacy Definition: The privacy parameter µ controls guarantee strength: µ = 0 represents perfect privacy, while small positive µ values provide strong privacy.Perfect privacy corresponds to an adversary treating every possible speaker as equally likely.
- 2.3. Privacy Mechanism: The Gaussian speaker mechanism clips speaker embeddings to bound L2 sensitivity and adds Gaussian noise whose level satisfies σ = U/µ.The perturbation generates a hypothetical speaker rather than selecting one from a predefined speaker pool.
- 2.3. Privacy Mechanism: The Gaussian speaker mechanism satisfies µ-speaker differential privacy and therefore yields a theoretical lower bound on any adversary’s speaker re-identification success.The theorem establishes the mechanism’s privacy guarantee, while the lower-bound result applies to adversaries performing distinguishing tasks.
- 2.4. Provable Adversarial Lower Bounds: The mechanism guarantees an EER lower bound that improves as µ approaches zero, reaching 50% at µ = 0 and about 35% at µ = 1.The bound is illustrated by the EER trade-off function computed from Definitions 1 and 2; when U = 1, µ = σ^-1.
3. The DP-VOXLET System
DP-VOXLET adds formal speaker anonymization to encoder–decoder voice conversion systems by perturbing speaker-identifying embeddings while preserving a valid embedding region. Its VAE-based design supports integration with existing systems and mitigates degradation from large perturbations.
- System architecture: DP-VOXLET integrates with encoder–decoder voice conversion systems that separate semantic information from speaker-identifying information.The framework applies the required perturbation to the speaker-identifying representation and includes wrappers for several PyTorch systems.
- System architecture: The implementation is a PyTorch library whose wrappers expose embedding extraction and voice generation for compatible systems.Current wrappers include OpenVoice, NaturalSpeech3, vec2wav2.0, and ControlVC.
- Speaker representation: A variational autoencoder learns valid speaker embeddings in a low-dimensional space that is robust to perturbations.The VAE is trained on speaker embeddings extracted from Common Voice using the target voice conversion system.
- Perturbation control: Large perturbations can degrade speech, so DP-VOXLET clamps each latent dimension to keep perturbed embeddings within the decoder’s understood region.The method enforces a bounded absolute value, or L∞-norm limit, on the latent representation.
4. Evaluation
The evaluation uses the 2024 Voice Privacy Challenge benchmark to measure privacy and utility. Increasing perturbation variance raises EER while keeping WER below 4%, and DP-VOXLET reaches competitive privacy against challenge systems and improves EER over related differential-privacy work.
- Benchmark: The benchmark evaluates privacy with speaker verification and utility with WER on anonymized utterances.The evaluation uses the semi-informed attacker setting and the librispeech-test dataset.
- Comparison to prior work: Less than 20% was the maximum EER reported for Shamsabadi et al., whereas DP-VOXLET achieved significantly improved EER with larger privacy budgets used by the prior work.The comparison uses librispeech but a different speaker verification model.
5. Conclusion
The paper introduces speaker differential privacy and implements it in DP-VOXLET to provide provable protection against speaker re-identification while maintaining high-quality output speech.
- Conclusion: Speaker differential privacy provides provable lower bounds on adversaries’ success in re-identifying speakers from anonymized utterances.The definition is implemented through a mechanism in DP-VOXLET, and empirical results suggest robust privacy with high-quality output speech.