Source-linked AI summary
Speaker Anonymization Using X-vector and Neural Waveform Models
Fuming Fang, Xin Wang, Junichi Yamagishi, Isao Echizen, Massimiliano Todisco, Nicholas Evans, Jean-Francois Bonastre
TL;DR
Shared speech can enable synthesis of utterances that preserve a person’s identity and threaten speaker verification security. This paper extracts linguistic and speaker features, replaces timbre with an x-vector-based pseudo-identity, and synthesizes anonymized speech. The method substantially increases verification-system EER while preserving speech quality with only slight degradation, though linguistic recognition remains weaker.
Problem
Shared speech can be reused to synthesize identity-revealing utterances that threaten automatic speaker verification and personal privacy.
Method
The method extracts PPG-based linguistic information and x-vector speaker identity, averages multiple external speaker x-vectors into a pseudo-identity, and synthesizes anonymized waveforms with neural acoustic and waveform models.
Results
The method greatly increases automatic speaker verification EER while causing only slight speech-quality degradation, but anonymized speech has higher WERs.
Takeaways & Limitations
X-vector-based synthesis can conceal speaker identity while retaining high-quality anonymized speech, supporting speaker anonymization for shared speech data.
Takeaways & Limitations
The framework depends heavily on ASR accuracy for PPG extraction, and imperfect factorization leaves linguistic-content errors, especially for sufficiently dissimilar averaged x-vectors.
Abstract
from arXiv · showhide
The social media revolution has produced a plethora of web services to which users can easily upload and share multimedia documents. Despite the popularity and convenience of such services, the sharing of such inherently personal data, including speech data, raises obvious security and privacy concerns. In particular, a user's speech data may be acquired and used with speech synthesis systems to produce high-quality speech utterances which reflect the same user's speaker identity. These utterances may then be used to attack speaker verification systems. One solution to mitigate these concerns involves the concealing of speaker identities before the sharing of speech data. For this purpose, we present a new approach to speaker anonymization. The idea is to extract linguistic and speaker identity features from an utterance and then to use these with neural acoustic and waveform models to synthesize anonymized speech. The original speaker identity, in the form of timbre, is suppressed and replaced with that of an anonymous pseudo identity. The approach exploits state-of-the-art x-vector speaker representations. These are used to derive anonymized pseudo speaker identities through the combination of multiple, random speaker x-vectors. Experimental results show that the proposed approach is effective in concealing speaker identities. It increases the equal error rate of a speaker verification system while maintaining high quality, anonymized speech.
1. Introduction
Speaker anonymization addresses the risk that shared speech can be exploited to recreate speaker identities and attack verification systems. The proposed method separates linguistic content from speaker identity, changes timbre through an anonymous pseudo-speaker, and synthesizes new speech while largely preserving quality.
- Motivation: Shared speech data can be used to synthesize utterances that reveal speaker identity and attack automatic speaker verification systems.The paper also notes that ASV can be used to search for information about people online.
- Motivation: Speaker anonymization modifies speech to sound like an anonymous speaker while maintaining linguistic content and speech quality.Speaker identity includes timbre, pitch, speaking rate, and speaking style; the paper focuses on modifying timbre.
- Approach: The method extracts linguistic information as a PPG and speaker identity as an x-vector, then combines the PPG with an anonymized pseudo-speaker identity to generate speech.The pseudo-speaker is formed by combining x-vectors from multiple arbitrary speakers using neural acoustic and waveform models.
- Findings: Experiments found that anonymized speech concealed speaker identity while remaining largely indistinguishable from the original speaker and causing only negligible quality degradation.The initial trial accepted a small loss of linguistic content to hide identity and maintain speech quality.
2. Related work
Related work distinguishes speaker anonymization from speech anonymization and separates physical perturbation from logical identity modification. Existing logical approaches use voice transformation, speaker-dependent synthesis, or ASV evasion, whereas this work combines multiple external speaker identities with one transformation function and evaluates identity, quality, and linguistic content.
- Conceptual distinction: Speaker anonymization suppresses speaker identity, whereas speech anonymization obscures linguistic content.The paper classifies speaker anonymization as physical or logical according to how the speech is manipulated.
- Conceptual distinction: Physical anonymization adds external sound to the waveform, while logical anonymization modifies speaker identity in the recorded speech signal.The proposed method belongs to the logical category.
- Prior approaches: Prior logical methods used voice transformation systems, convolutional transformation functions, or pools of pretrained transformation models.Some approaches averaged transformation results or let users select an approximate transformation from pretrained models.
- Distinction: The proposed method differs by training one transformation function and constructing an anonymized pseudo-speaker from multiple external speaker identities.This contrasts with approaches based on selecting among pretrained voice-transformation models.
- Evaluation scope: Earlier speaker-dependent synthesis and voice-conversion approaches did not consistently evaluate speaker identity, speech quality, and linguistic content together.The paper evaluates all three dimensions, unlike work focused only on degrading ASV performance.
3. Proposed speaker anonymization method
The system disentangles linguistic content, speaker identity, and prosodic information, then replaces the input x-vector with an anonymized identity before neural waveform synthesis. It uses x-vector selection strategies and acoustic and neural source-filter models to generate the anonymized speech.
- The method assumes speech content and speaker identity can be disentangled and that modifying identity features can anonymize the waveform.
- The system extracts an x-vector, PPG, and F0, anonymizes the x-vector using external speaker representations, and synthesizes speech from the anonymized x-vector, original PPG, and F0.
- 3.1. Feature extraction: The x-vector encodes speaker identity, while the PPG represents linguistic content as frame-level phoneme posterior probabilities.
- 3.1. Feature extraction: F0 remains unchanged because it carries context-related pitch information, although it may also encode speaker characteristics.
- 3.2. Anonymization: Two anonymization methods either average randomly selected pool x-vectors or average candidates whose cosine similarity to the original lies within [s −ϵ, s + ϵ].
- 3.3. Waveform generation: An autoregressive acoustic model generates Mel-spectrograms, and a neural source-filter model converts F0, the anonymized x-vector, and the spectrogram into waveform output.
- 3.3. Waveform generation: The x-vector extractor and ASR modules can be pretrained externally, but the acoustic and neural source-filter models require data from potential users.
4. Experimental setup
The evaluation measures anonymization, content preservation, and speech quality using ASV, ASR, and listener-based tests. It uses separately trained modules and evaluates American VCTK speakers under non-overlapping training, adaptation, enrollment, and test data conditions.
- The proposed system is evaluated with separate ASVeval and ASReval systems, with anonymization assessed by increased rejection of speech from enrolled speakers.
- Anonymization performance is measured with EER, content preservation with WER, and anonymized speech quality with listener-evaluated MOS.
- 4.2. Data recipes: The modules use TIMIT for PPG extraction, VoxCeleb for x-vector training and the anonymization pool, and VCTK for acoustic and NSF training.
- 4.2. Data recipes: The VCTK evaluation uses 11 randomly selected American target speakers and 19 non-target speakers, with American speakers selected to avoid large ASR word error rates.
- 4.2. Data recipes: ASVeval is trained on VoxCeleb, adapted with 2,580 VCTK utterances from 20 unused speakers, and evaluated without overlap among enrollment, adaptation, training, and test data.
- Subjective evaluation uses range selection with four dissimilarity levels—0.0, 0.2, 0.4, and 0.6—and up to 296 listeners rating randomly selected utterances.
5. Experimental results
The experiments show that x-vector-based anonymization substantially increases speaker-verification EER and can produce distinguishable pseudo-speakers, while speech quality remains comparatively stable. However, anonymization introduces higher WER, especially at larger speaker distances, revealing incomplete separation of speaker identity and linguistic content.
- Nearest-speaker anonymization: Anonymization substantially increased EER compared with the 2.04%–2.52% pre-anonymization range, with the highest rates at M = 300 using softmax PPGs.Softmax PPGs produced higher EERs than 6th-layer PPGs, consistent with containing less speaker identity information.
- Random-selection anonymization: Randomly selecting M = 10, 50, 100, or 200 speakers produced much higher EERs for anonymized speakers than for the original speaker.Each setting was repeated five times, and the reported result used the average EER.
- Random-selection anonymization: Random selection produced varied anonymized speakers, as shown by dissimilarity-score ranges computed from x-vectors across five repetitions.The dissimilarity score is defined as 1 −cos(x1, x2).
- Range-selection anonymization: As dissimilarity increased, EER rose greatly while MOS remained relatively stable, indicating anonymization with comparatively preserved speech quality.Figure 4 reports EER, WER, and MOS across different distances between anonymized and original speakers.
- Range-selection anonymization: Natural-speech WER was 9.49%, versus 10%–30% with 6th-layer PPGs and 25%–45% with softmax PPGs.The authors attribute higher WER partly to inaccurate phoneme posteriorgrams and report further degradation above dissimilarity 0.4.
- Subjective evaluation: Natural speech had MOS 4.05, whereas copy-synthesized and anonymized speech scored 2.31–3.25, with scores slightly decreasing as dissimilarity increased.Higher quality was obtained with 6th-sigmoid PPGs, while softmax PPGs produced more speaker-distinguishable speech.
- Subjective evaluation: Evaluator judgments increasingly classified anonymized speech as different from the original as dissimilarity increased.This supports the method’s ability to hide speaker identity in the subjective test.
6. Conclusion and future work
The proposed method anonymizes speech by combining x-vector speaker representations with PPG-based linguistic information and neural acoustic and waveform models. It raises ASV equal error rates with only slight quality degradation, but linguistic-content recovery remains imperfect and evaluation lacks a baseline.
- The system extracts linguistic and speaker information, then synthesizes speech using a PPG and an anonymized pseudo speaker formed by averaging multiple x-vectors.Neural acoustic and waveform models generate the anonymized waveform.
- Anonymized speech greatly increased the ASV equal error rate while causing only slight degradation in speech quality.
- Anonymized speech had higher word error rates, probably because phoneme-recognition errors affected PPG extraction.
- The method depends heavily on ASR accuracy for PPG extraction, motivating future unsupervised disentanglement of linguistic content and speaker identity.
- Because common databases, protocols, and metrics were unavailable, the paper did not include a baseline system and proposes standardizing future evaluation.