PaperlayerView exact revision on arXiv

Source-linked AI summary

Anonymization, Not Elimination: Utility-Preserved Speech Anonymization

Yunchong Xiao, Yuxiang Zhao, Ziyang Ma, Shuai Wang, Kai Yu, Jiachun Liao, Xie Chen

arXiv:2604.17000v1eess.AScs.AI

TL;DR

Large-scale speech data exposes both linguistic content and biometric speaker identity, while existing anonymization and evaluation practices can compromise or incompletely measure utility. The paper proposes SECA and F3-VA in a two-stage privacy framework, evaluates privacy across acoustic and content-based verification, and trains downstream models from scratch. The framework reports stronger privacy protection with minimal utility degradation relative to VoicePrivacy Challenge baselines.

  • Problem

    Large-scale speech datasets raise content- and voice-privacy risks, while direct testing with pretrained models provides only a partial view of anonymized-data utility.

  • Method

    The paper combines SECA for generative PII replacement with F3-VA for flow-matching voice anonymization and evaluates ASR, TTS, and SER models trained from scratch.

  • Results

    SECA + F3-VA consistently outperforms baselines in the normalized radar comparison while maintaining utility close to ground truth.

  • Takeaways & Limitations

    Training downstream models from scratch offers a more realistic assessment of anonymized speech as a useful training resource.

Abstract

from arXiv · show

The growing reliance on large-scale speech data has made privacy protection a critical concern. However, existing anonymization approaches often degrade data utility, for example by disrupting acoustic continuity or reducing vocal diversity, which compromises the value of speech data for downstream tasks such as Automatic Speech Recognition (ASR), Text-to-Speech (TTS), and Speech Emotion Recognition (SER). Current evaluation practices are also limited, as they mainly rely on direct testing of anonymized speech with pretrained models, providing only a partial view of utility. To address these issues, we propose a novel two-stage framework that protects both linguistic content and acoustic identity while maintaining usability. For content privacy, we employ a generative speech editing model to seamlessly replace personally identifiable information (PII), and for voice privacy, we introduce F3-VA, a flow-matching-based anonymization framework with a three-stage design that produces diverse and distinct anonymized speakers. To enable a more comprehensive assessment, we evaluate privacy using both acoustic- and content-based speaker verification metrics, and assess utility by training ASR, TTS, and SER models from scratch. Experimental results show that our framework achieves stronger privacy protection with minimal utility degradation compared to baselines from the VoicePrivacy Challenge, while the proposed evaluation protocol provides a more realistic reflection of the utility of anonymized speech under privacy protection.

I. INTRODUCTION

The paper frames speech privacy as protecting both what is said and who is speaking while retaining utility for downstream speech systems. It proposes a two-stage framework combining voice anonymization for interaction with content anonymization for storage and model training.

  • Framework: The proposed framework uses voice anonymization to obfuscate speaker identity while preserving conversational content, then content anonymization to modify semantics for storage and model training.The staged design addresses different privacy requirements across the data lifecycle.
  • Motivation: Speech data contains linguistic content and biometric voice markers, creating distinct content-privacy and voice-privacy risks.Content privacy concerns what is said, whereas voice privacy concerns who is speaking.
  • Motivation: Existing voice anonymization methods can rely on reference speakers or generative models, creating privacy leakage, limited diversity, or complex identity-selection requirements.The related approaches include external reference pools, embedding manipulation, VAEs, and GANs.
  • Contributions: The paper evaluates utility by training downstream ASR, TTS, and SER models from scratch rather than relying only on direct testing with pretrained models.This evaluation is intended to assess the value of anonymized speech as training data.
  • Contributions: F3-VA generates diverse anonymized speaker embeddings while explicitly controlling their deviation from the original speaker embedding.This is presented as the paper’s flow-matching-based voice anonymization contribution.
  • Contributions: SECA replaces personally identifiable information through generative speech editing while preserving acoustic integrity and fluency.The pipeline is designed to enhance traditional content-anonymization approaches.

II. RELATED WORK

Prior work protects speech privacy through feature manipulation, content rewriting, encryption, and direct speech editing, but each approach leaves utility, privacy, or evaluation limitations. The paper motivates jointly measuring acoustic and content privacy while assessing anonymized speech as a trainable data resource.

  • Voice anonymization: Voice anonymization commonly disentangles linguistic content, prosody, and speaker identity before manipulating the speaker representation.Semantic, pitch, and speaker-embedding features are typically used for these representations.
  • Voice anonymization: Reference-pool methods risk leaking pool-speaker privacy, while averaging or removing speaker representations can reduce diversity and realism.These trade-offs motivate direct generation of speaker identities.
  • Voice anonymization: VAE- and GAN-based identity generation can produce diverse or random speakers but often require additional selection or verification mechanisms.Those mechanisms increase overall system complexity.
  • Content anonymization: Content anonymization pipelines use ASR, NER, and localized redaction, or rewrite sensitive text with LLMs before speech resynthesis.Infilling strategies include silence insertion, noise replacement, and speech reversal.
  • Evaluation: Cryptographic approaches protect content without altering speech but primarily target secure storage or restricted access rather than flexible downstream use.The paper therefore motivates a broader utility assessment based on models trained from scratch.
  • Content anonymization: Text rewriting and resynthesis can discard original acoustic characteristics, speaking style, and fine-grained prosody, while editing methods may introduce boundary or prosody inconsistencies.SECA is proposed to modify PII-bearing segments while better preserving naturalness and prosodic consistency.
  • Evaluation: Speech anonymization evaluation should combine acoustic- and content-based privacy risks with training-from-scratch utility assessment.The VPC protocol measures speaker-verification degradation and direct downstream performance, while stronger attacker models include semi-informed adaptation.

III. PROPOSED METHODOLOGY

The methodology factorizes speech into speaker-irrelevant content and prosody features plus a speaker embedding, then reconstructs speech conditionally with flow matching. Quantization suppresses residual speaker identity, while anonymized speaker embeddings control the reconstructed voice.

  • Embedding-based Speech Reconstruction Backbone: The reconstruction backbone separates semantic content, prosody, and speaker identity before conditional synthesis with a DiT-based flow-matching model.The factorized design uses interpretable representations and supports training stability and generative diversity.
  • Embedding-based Speech Reconstruction Backbone: HuBERT-large semantic features are vector-quantized into cvq to create a speaker-irrelevant content representation.The VQ layer acts as an information bottleneck that suppresses residual speaker identity.
  • Embedding-based Speech Reconstruction Backbone: F0 contours are converted to semitones and centered by the utterance-level median to produce speaker-independent normalized pitch pnorm.Together with cvq, pnorm conditions the reconstruction backbone as a speaker-irrelevant feature.
  • Embedding-based Speech Reconstruction Backbone: The original speaker identity is represented by a 192-dimensional embedding extracted from the pretrained CAM++ model.This embedding is distinct from the speaker-irrelevant conditioning features.
  • Embedding-based Speech Reconstruction Backbone: During training, the backbone predicts a flow that transforms Gaussian noise into target mel-spectrograms using noisy mel inputs and aligned speaker-irrelevant features.The noisy input follows xt = (1 − t)x0 + tx1 across time t.

B. Flow-Matching Speaker Embedding Anonymizer

The speaker embedding anonymizer uses flow matching to transform Gaussian noise into anonymized speaker embeddings through a staged ODE-based process. It then conditions acoustic reconstruction on the anonymized embedding and speaker-irrelevant features.

  • Flow-matching formulation: The anonymizer takes a noisy speaker embedding and time variable as input to predict the flow along the probability path between Gaussian noise and speaker embeddings.The predicted flow specifies the instantaneous update direction at each time step.
  • Flow-matching formulation: Flow matching learns a reversible mapping between a standard Gaussian distribution and the distribution of real speaker embeddings.The model minimizes a conditional flow-matching loss that regresses the flow to the target path.
  • Three-stage anonymization: Given an original speaker embedding, the first ODE maps it backward into a Gaussian variable before anonymization.The original embedding is encoded from t = 1 to t = 0, yielding zorig ∼N(0, I).
  • Three-stage anonymization: Speaker obscuration combines the deterministically encoded variable with independently sampled Gaussian noise to control the retained original identity.The weight w controls the strength of the original speaker identity, while the normalization term preserves unit variance.
  • Three-stage anonymization: A second ODE maps the obscured variable back to the speaker embedding distribution, after which a backbone ODE generates the anonymized mel-spectrogram.The resulting mel-spectrogram is synthesized into an anonymized waveform by a vocoder.

C. Content Anonymization via Generative Editing

The content anonymization module detects PII, localizes its speech segments, selects matched replacement entities, and edits the corresponding audio generatively. This design targets continuity in the surrounding speech rather than replacing sensitive spans with silence or noise.

  • PII detection and localization: The pipeline detects PII in ASR transcriptions with Flair NER and uses forced alignment to localize the corresponding speech segments.The NER model identifies PER, LOC, ORG, and MISC entity types.
  • Pipeline: The four-stage content anonymization pipeline consists of ASR, NER, replacement matching from a PII pool, and generative speech editing.These stages connect transcription analysis to replacement audio generation.
  • Replacement selection: Replacement entities are selected from WikiAnn to match the original PII’s entity type and length.Matching is intended to preserve utterance-level semantics and rhythm while supporting timbre and prosody consistency.
  • Generative speech editing: F5-TTS updates the targeted segment using the original text, anonymized text, original speech, and the segment’s time boundaries.The method contrasts with replacing PII by silence or noise.

IV. EXPERIMENTAL SETUP

The experiments evaluate the framework across ASR, TTS, and SER using established open-source corpora, while constructing separate trial sets for acoustic- and content-based privacy assessment. The setup also specifies model configurations and optimization details for reproducibility.

  • Evaluation protocol: The experimental configuration includes datasets, baseline models, and a protocol covering both privacy and utility.These components define the evaluation setup for the proposed framework.
  • Datasets: Experiments use LibriSpeech, LibriTTS, and IEMOCAP for ASR, TTS, and SER, respectively.The benchmarks are selected to support reproducibility and comparability.
  • Datasets: Downstream training uses LibriSpeech-train-960, LibriTTS-train-554, and complete IEMOCAP with 5-fold cross-validation.The voice anonymization model is trained on LibriSpeech under the standard VPC setup.
  • Privacy evaluation: The corpora support both downstream utility evaluation and dedicated privacy trial sets under the standard ASV protocol.Separate trials are generated for acoustic-based ASV and content-based ASV, with dataset and trial summaries provided in Table I.
  • Model configuration: The backbone uses a DiT with hidden dimension 768, depth 14, and 12 attention heads, while ConvNeXtV2 blocks replace its feed-forward blocks.The VQ layer has dimensionality 1024 and codebook size 1024.

2) Speaker Embedding Anonymizer:

The section describes the speaker embedding anonymizer implementation, its VPC baselines, and the complementary privacy verification protocol. Privacy is measured through acoustic and linguistic speaker re-identification under explicitly constructed trials.

  • Anonymizer implementation: The speaker embedding anonymizer uses a U-Net for fixed-length embeddings, replacing DiT self-attention with lightweight MLP blocks.Its input and output embeddings are 192-dimensional, and timestep conditioning is projected to the same dimension.
  • Anonymizer implementation: The embedding anonymizer is trained separately for 5000 epochs with batch size 128 on 8 NVIDIA RTX 3090 GPUs.Training uses AdamW and OneCycleLR with the backbone’s optimization settings.
  • Baselines: F3-VA is compared with NAC and ASR-BN, the two anonymization baselines retained in the VPC 2024 attacker track.NAC uses pseudo-speaker acoustic tokens, whereas ASR-BN uses ASR bottleneck features and a target-speaker vector.
  • Efficiency: Table II compares anonymization systems by model size and real-time factor on a single NVIDIA RTX 3090 GPU.For F3-VA, the reported real-time factor uses 16 flow-matching inference steps.
  • Privacy evaluation: Privacy is assessed with acoustic- and content-based speaker verification using Equal Error Rate, where higher EER indicates stronger anonymization.A-EER measures acoustic identity leakage, while C-EER measures linguistic-content re-identification.
  • Trial construction: Each enrollment utterance produces two positive and two negative trials, with negative trials balanced across genders.Positive trials use same-speaker utterances, and negative trials use different-speaker utterances.
  • Trial construction: Acoustic trials use 5–15-second utterances, while content trials include only transcripts containing PII.The content restriction targets regions where content anonymization is actively applied.

B. Utility Evaluation

Utility is assessed through direct speech metrics and from-scratch training of ASR, TTS, and SER models, with normalized radar profiles enabling joint comparison across dimensions.

  • Downstream evaluation: ASR, TTS, and SER models are trained from scratch on anonymized data to evaluate downstream utility.ASR reports WER; TTS reports WER, SECS, and UTMOS; SER reports WA, UA, and F1.
  • Direct metrics: Direct evaluation additionally measures intelligibility with Whisper-large-v3 WER and perceptual quality with UTMOS.
  • Radar visualization: Metrics are normalized to [0, 1] using task-specific performance ranges, with 1 representing the best performance.The ranges cover WER, UTMOS, SECS, WA, A-EER, and C-EER.
  • Radar visualization: The radar chart compares anonymization systems across direct quality and downstream utility, where a larger enclosed area indicates stronger overall performance.The shaded area represents the overall utility-privacy profile.

A. Utility for Downstream Model Training

From-scratch downstream evaluation shows that F3-VA and SECA preserve utility near ground-truth performance, while baselines reveal weaknesses that direct metrics can miss.

  • Evaluation protocol: Five random-seed runs are averaged to evaluate utility and privacy through downstream task performance.
  • Voice anonymization: 2.46% WER on test-clean for F3-VA ASR is close to the 2.22% ground-truth result.
  • Content anonymization: 2.23% versus 2.22% ASR WER shows that SECA remains near ground-truth utility, while TTS SECS reaches 0.60.
  • Evaluation implications: Direct metrics alone can understate anonymization effects on learnability, whereas training models on anonymized data reveals hidden utility degradation.

C. Analysis of Content Anonymization

The SECA analysis isolates cascading ASR errors in content editing, while attacker studies show that stochastic anonymization is more robust to model-aware threats than deterministic guidance.

  • SECA ablation: Ground-truth-text and fully cascaded SECA configurations are compared to isolate editing quality from upstream ASR effects.
  • SECA ablation: 4.80% WER under fully cascaded SECA is primarily attributed to upstream ASR inaccuracies causing pronunciation deviations in edited segments.
  • SECA ablation: 27.03% versus 24.75% content-based EER shows an increase under the fully cascaded SECA pipeline.
  • Speaker-weight analysis: For Lazy-Informed attackers, EERla peaks at 52.08% when w = 0 and decreases as guidance magnitude increases.

F. Analysis against Speaker Embedding Anonymizer

F3-VA is compared with reference-pool speaker replacement and qualitatively examined through t-SNE, linking generated speaker diversity to downstream TTS utility and dispersed embeddings.

  • Ablation comparison: F3-VA with w = 0 generates anonymized speaker embeddings from random noise independently of the original speaker identity.
  • Ablation comparison: 0.565 versus 0.524 SECS and 2.12% versus 2.38% WER favor F3-VA over randomly selected ground-truth voices in downstream TTS.
  • Ablation comparison: The ablation attributes F3-VA’s stronger downstream TTS performance to generating a more diverse set of speaker embeddings than the original training data.
  • Embedding visualization: The t-SNE study uses 10 LibriSpeech speakers, 100 utterances each, and anonymization with w = −0.5.
  • Embedding visualization: Original embeddings form tight speaker clusters, whereas anonymized embeddings show no discernible clustering and spread quasi-uniformly.
  • Conclusion: The framework combines SECA and F3-VA and supports high-performing ASR, TTS, and SER models while protecting voice and elementary content privacy.
Loading 2604.17000v1…