PaperlayerView exact revision on arXiv

Source-linked AI summary

Voice Anonymization Made Simple: Training-Free Anonymization with Projected Classifier-Free Guidance

Xiang Shi, Han Zhu, Ming Li, Xiaoxiao Miao

arXiv:2610.08276v1eess.AS

TL;DR

Voice anonymization seeks to conceal speaker identity while preserving linguistic and paralinguistic information, but many methods require dedicated training. This paper changes inference-time conditioning and guidance in a pretrained flow-matching voice-conversion system, reporting improved privacy with competitive linguistic and emotion utility. Its evaluation is limited to English anonymization.

  • Problem

    Many existing voice-anonymization approaches require dedicated training, additional learned components, or retraining for a different voice-conversion model, prompting the question of whether inference changes alone suffice.

    Source: §I · p. 1

  • Method

    A training-free two-stage method regenerates source content under a randomly selected speaker context and uses source-aware projected guidance to suppress source-aligned generation information.

    Source: §I · pp. 1–2

  • Results

    On VoicePrivacy 2026 Track 1, the method improves speaker privacy while maintaining competitive linguistic and emotion utility compared with baselines.

    Source: §I · p. 2

  • Takeaways & Limitations

    Ablations indicate that random-target-conditioned AR construction and projected CFG together improve privacy while maintaining competitive linguistic and emotion utility.

    Source: §V · p. 6

    The study is limited to English anonymization.

    Source: §V · p. 6

Abstract

from arXiv · show

Voice anonymization aims to conceal speaker identity while preserving linguistic and paralinguistic information, yet many existing methods require dedicated training. We propose a training-free, two-stage anonymization method for pretrained flow-matching voice conversion systems. In the condition-construction stage, the source content representation is regenerated under a randomly selected speaker context to reduce residual source-speaker information while preserving linguistic content. In the speech-generation stage, the source speech is used to identify the generation direction associated with the original speaker, and the source-aligned component is suppressed while useful content guidance is retained. The proposed method modifies only inference-time conditioning and guidance, without updating any pretrained parameters. Experiments on VoicePrivacy 2026 Track 1 show that the two-stage method improves speaker privacy while maintaining competitive word error rate and emotion recognition performance.

I. INTRODUCTION

Voice anonymization seeks to prevent reliable source-speaker re-identification while preserving information needed downstream. This section motivates inference-only anonymization of pretrained flow-matching voice conversion and introduces a two-stage method with a favorable privacy–utility tradeoff.

I. INTRODUCTION

  • Voice anonymization aims to prevent reliable source-speaker re-identification while preserving linguistic content and prosodic attributes for downstream applications.Earlier approaches include signal processing, neural synthesis, diffusion, and flow matching; reported drawbacks include speech distortion, vulnerability to strong attackers, and loss of paralinguistic information,, –, –[31].Source: §I · p. 1
  • Many existing anonymization approaches require dedicated training, additional learned components, or retraining for a different voice-conversion model.The paper asks whether inference-procedure changes alone can convert a pretrained flow-matching voice-conversion model into an anonymizer.Source: §I · p. 1
  • The method regenerates source content under an external speaker context, then suppresses source-aligned generation directions while retaining useful content guidance.Both interventions operate at inference time without updating pretrained parameters.Source: §I · p. 1
  • The method is implemented on Seed-VC v2, whose condition-construction and speech-synthesis stages allow the interventions to be applied separately.Randomly selected target utterances guide source-content reconstruction; source-aware projected guidance steers generation away from the original speaker.Source: §I · p. 2
  • The two-stage method improves speaker privacy while maintaining competitive linguistic and emotion utility on VoicePrivacy 2026 Track 1.The reported comparison is against baselines.Source: §I · p. 2

II. PRELIMINARIES

Seed-VC v2 constructs conversion conditions from source and reference speech, then uses conditional flow matching to synthesize speech. Its conditions and guidance combine token/content and reference-speaker information.

A. Voice Conversion in Seed-VC v2

  • Seed-VC v2 uses source narrow-band tokens for linguistic content and target tokens as autoregressive context and prefix to generate the source-portion wide-band sequence.The generated sequence is converted to Mel-frame rate as µAR; the token prompt and generated content are concatenated after separate length regulation.Source: A. Voice Conversion in Seed-VC v2 · p. 2
  • The target utterance supplies a token prompt µprm, an acoustic prompt p, and a global style embedding s, with complementary roles in conversion.The token prompt represents what is spoken, the acoustic prompt how it sounds, the style embedding speaker and timbre, and µAR the next content to generate.Source: A. Voice Conversion in Seed-VC v2 · p. 2
  • Conditional flow matching generates the converted Mel-spectrogram from (µ, p, s) by integrating its velocity field from Gaussian noise with a fixed-step Euler solver.A pretrained BigVGAN v2 vocoder converts the resulting Mel-spectrogram into waveform audio.Source: A. Voice Conversion in Seed-VC v2 · p. 2
  • Standard voice conversion combines fully conditioned, token-conditioned, and unconditional predictions through multi-condition classifier-free guidance.The guidance equation scales joint guidance from all conditions and additional guidance from the acoustic prompt and style embedding separately.Source: A. Voice Conversion in Seed-VC v2 · p. 2

B. Built-in Anonymization in Seed-VC v2

Seed-VC v2’s built-in anonymization removes explicit target-timbre conditioning while retaining its two-stage generation pipeline. Its AR stage lacks target context, and speech generation uses token-only guidance.

B. Built-in Anonymization in Seed-VC v2

  • The built-in mode removes target context and prefix from AR generation, which receives the source narrow-band tokens.The absent context and prefix are denoted by ∅.Source: B. Built-in Anonymization in Seed-VC v2 · p. 2
  • Although the released wrapper still constructs a token-prompt prefix, the zero acoustic prompt and style embedding mean it is not an aligned in-context reference pair.Preliminary tests found little difference between source-derived and target-derived prefixes; the source utterance is used by default.Source: B. Built-in Anonymization in Seed-VC v2 · p. 2
  • Built-in speech generation uses token-conditioned and unconditional predictions, and the Euler solver and vocoder produce the anonymized waveform.The guidance scale γ is the intelligibility CFG scale.Source: B. Built-in Anonymization in Seed-VC v2 · p. 3

III. PROPOSED METHOD

The built-in mode leaves potential source-speaker information in both the token condition and generation guidance. The proposed method responds with random-target conditioning and source-aware projected guidance.

III. PROPOSED METHOD

  • Removing target-related AR inputs leaves the token condition largely determined by the source and may retain source-speaker information entangled with content.The built-in mode does not use the AR model to regenerate source content under an alternative speaker context.Source: §III · p. 3
  • Built-in guidance neither identifies source-related information in token guidance nor steers generation away from the source speaker.It reduces direct use of source acoustic conditions but leaves source-related information carried in the token guidance unaddressed.Source: §III · p. 3
  • The proposed method uses independently sampled speaker context to weaken residual source-speaker information and projected CFG to suppress source-aligned directions during generation.These are the Stage 1 and Stage 2 modifications, respectively.Source: §III · p. 3

A. Stage 1: Random-Target-Conditioned Token and Source-Acoustic Condition Construction

Stage 1 regenerates source-content tokens under an unrelated speaker context while keeping the prompt region source-aligned with its acoustic prompt. This reduces source-speaker information in the generated tokens, but cannot guarantee its complete removal.

A. Stage 1: Random-Target-Conditioned Token and Source-Acoustic Condition Construction

  • The AR model uses source narrow-band tokens for the linguistic sequence but a random target’s tokens as context and prefix to generate source-portion wide-band tokens.The resulting sequence is converted to the Mel-frame rate as µAR, preserving source linguistic content while using an unrelated speaker context for token realization.Source: A. Stage 1: Random-Target-Conditioned… · p. 3
  • The source-derived token-prompt condition remains aligned with the source acoustic prompt; substituting random-target prompt tokens can destabilize generation and degrade audio quality and WER.Preliminary tests found that prompt mismatch sometimes improved speaker separation, but also substantially degraded audio quality and increased WER.Source: A. Stage 1: Random-Target-Conditioned… · p. 4
  • The complete token condition concatenates the token-prompt and AR-token parts after each is regulated to its own Mel length.The token-prompt describes the prompt region, while the AR-token condition describes the source content to be generated.Source: A. Stage 1: Random-Target-Conditioned… · pp. 3–4
  • The source prompt and style embedding are retained for Stage 2 to expose the effect of source-acoustic conditioning, not as desired output attributes.The source-derived token prompt and acoustic prompt form an aligned prompt example, while the style embedding describes source timbre.Source: A. Stage 1: Random-Target-Conditioned… · p. 4
  • Because the token condition retains a source prompt and AR generation starts from source tokens, Stage 1 cannot guarantee removal of all source-speaker information.Source: A. Stage 1: Random-Target-Conditioned… · p. 4

Free Guidance for Conditional Flow-Matching Speech Generation

Stage 2 uses source acoustic conditioning to identify a direction for suppressing source-aligned guidance, while retaining token guidance for linguistic generation. The projection reference can be constructed at frame, utterance, or smoothed temporal granularity.

Free Guidance for Conditional Flow-Matching Speech Generation

  • The source prompt and style embedding identify the source-associated generation direction; the aligned component of token guidance is suppressed at each ODE step.The method does not assume that the source-conditioning direction is purely speaker-related or that token guidance is purely content-related.Source: Free Guidance for Conditional… · p. 4
  • Sampling starts from the token-conditioned field vc, using source prompt and style only as a reference for suppression rather than reconstruction.The remaining token guidance continues to support linguistic generation.Source: Free Guidance for Conditional… · p. 4
  • The projection reference is built from source-conditioning directions at three temporal granularities: Token uses the same frame, Global uses an utterance mean, and Smooth uses a local sliding-window mean.These references respectively preserve frame-level variation, remove local variation, or provide an intermediate smoothed reference; the Smooth window spans frames t−10 through t+10.Source: Free Guidance for Conditional… · p. 4
  • The guidance rule subtracts the source-aligned component from vc and adds token guidance, with λs controlling suppression and λc controlling token guidance.A shared Euler solver integrates the resulting field, and a pretrained vocoder renders the waveform.Source: Free Guidance for Conditional… · p. 4
  • Figure 2 examines λs and λc on the 30-utterance screening subset; higher dissimilarity means lower source-speaker similarity, while WER is corpus-level.Source: Figure 2 · p. 5

IV. EXPERIMENTS

Experiments first screen coefficient behavior and settings, then evaluate selected systems on the full VoicePrivacy Challenge 2026 Track 1 development and evaluation sets.

IV. EXPERIMENTS

  • The full-set experiments evaluate systems with selected scale pairs fixed and serve as ablations of random-target-conditioned µAR construction and projected guidance.Source: §IV · p. 5
  • A 30-utterance subset from VPC LibriSpeech development clean is used to check whether λs strengthens anonymization and λc strengthens content guidance while reducing WER.Source: §IV · p. 5
  • After checking coefficient roles, the same subset is used to select the best (λs, λc) pair for each random-target construction and projection-mode combination.Source: §IV · p. 5

A. Evaluation Protocol, Datasets, and Settings

The evaluation follows VoicePrivacy Challenge 2026 Track 1 (English), using its datasets, attack models, and ASR and SER evaluation models. A 30-speaker screening subset supports parameter selection, while the random-target pool draws from 500 LibriTTS speakers.

A. Evaluation Protocol, Datasets, and Settings

  • Privacy is measured by EER under lazy-informed and semi-informed attacks, content preservation by WER, and emotion preservation by UAR.Higher EER indicates stronger protection, lower WER is better, and higher UAR is better.Source: A. Evaluation Protocol, Datasets, and… · p. 5
  • The random-target pool contains one 3–6 s utterance from each of 500 LibriTTS train-other-500 speakers, sampled with replacement for each source utterance.Source: A. Evaluation Protocol, Datasets, and… · p. 5
  • The screening subset uses one long, clearly spoken utterance from each of 30 speakers; its selected settings empirically generalize well, though full-development-set tuning could improve them at substantially greater sweep cost.Source: A. Evaluation Protocol, Datasets, and… · p. 5
  • A fixed random seed and identical source–target assignments across projection modes and scale settings support controlled comparisons.The token-prompt part µprm remains source-derived to stay aligned with the source acoustic prompt.Source: A. Evaluation Protocol, Datasets, and… · p. 5

B. Results

On VoicePrivacy 2026 Track 1, the ablations support gains from both random-target condition construction and source-aligned guidance suppression. The selected Smooth-21 system exceeds official baselines on both EER attacks while maintaining comparable WER.

Verifying the Roles of the Two Guidance Scales

  • Increasing λs raises dissimilarity but excessive suppression increases WER, whereas increasing λc lowers WER while reducing dissimilarity.The results support λs as mainly controlling source-speaker suppression and λc as mainly controlling linguistic-content preservation.Source: B. Results · p. 5

Effect of Random-Target Conditioning

  • Alternative random seeds and source–target assignments changed results only slightly, suggesting unrelated target context matters more than the target’s exact identity for weakening source-speaker information in µAR.Source: B. Results · p. 5

Guidance Scale Grid Search on Libri dev

  • Scale pairs were selected by sweeping λs and λc from 0 to 3 in 0.1 steps, filtering for WER below 5%, then choosing stronger source-speaker dissimilarity; no later full-set tuning was performed.The 5% threshold is described as application-dependent.Source: B. Results · p. 6

Effect of Random-Target Conditioning

  • With scales fixed, random-target conditioning raises development Lazy-EER from 33.90% to 34.44% for Global, 34.20% to 34.79% for Smooth-21, and 33.48% to 33.80% for Token.All three projection modes achieve higher privacy with comparable utility when random-target conditioning is the only changed factor.Source: B. Results · p. 6, Table I · p. 6

Overall Comparison with Baselines

  • 29.39% Lazy-EER and 21.51% Semi-EER put selected Smooth-21 above built-in anonymization and the strongest official baseline B5 on both evaluation privacy measures, with comparable WER.Smooth-21 also achieves the highest development EERs.Source: B. Results · p. 6, Table II · p. 6

Effect of the Projection Method

  • Projection variants raise development Lazy-EER over built-in anonymization: Global reaches 33.90%, Smooth-21 34.20%, and Token 33.48%, versus 33.16% for built-in.WER remains close to built-in, and Smooth-21 has the strongest privacy result among the three projection granularities.Source: B. Results · p. 6, Table I · p. 6

V. CONCLUSION AND FUTURE WORK

The paper presents a training-free, two-stage method built on Seed-VC v2 and reports improved speaker privacy with competitive linguistic and emotion utility on VPC 2026 Track 1. The study is limited to English anonymization, with multilingual performance left for future work.

V. CONCLUSION AND FUTURE WORK

  • The method combines random-target-conditioned AR construction with source-conditioning-aware projected CFG to suppress source-speaker information across two stages.Source: §V · p. 6
  • On VPC 2026 Track 1, the two stages improve speaker privacy while maintaining competitive linguistic and emotion utility.Source: §V · p. 6
  • The study covers English anonymization; extending the method to VPC 2026 Track 2 and investigating multilingual performance are future work.Source: §V · p. 6
Loading 2610.08276v1…