Source-linked AI summary
Forget who you Forgot: Speaker Unlearning to Prevent Re-Identification in Zero-Shot Text-to-Speech
Hyoeun Kim, Yujun Lee, Kyuhong Shim
TL;DR
Zero-shot TTS can reproduce a speaker's voice from short reference recordings, while existing unlearning approaches may reduce similarity without sufficiently preventing gallery-based re-identification. GUARD uses a speaker gate and activation steering optimized with post-generation rewards to target population-level impostor similarity. On CosyVoice2, it reduces forget-speaker similarity and gallery re-identification while preserving speech quality and retain-speaker reproduction.
Problem
Existing speaker-unlearning approaches may reduce similarity without sufficiently suppressing gallery-based re-identification, while ZS-TTS voice reproduction raises risks of unauthorized cloning and impersonation.
Source: §1 · p. 1Method
GUARD combines a learned speaker gate with speaker-agnostic layer-wise activation steering optimized by group-relative post-generation rewards on a frozen TTS backbone.
Source: §1 · pp. 1–2Results
On CosyVoice2, forget-speaker SIM fell from 0.541 to 0.103 and top-1 re-identification accuracy in a 150-speaker gallery fell from 73.5% to 0.5%, without degradation in speech quality or retain-speaker reproduction.
Source: §1 · p. 2Takeaways & Limitations
Gallery-based re-identification provides an explicit evaluation of residual speaker identity alongside similarity reduction.
Source: §1 · pp. 1–2
Abstract
from arXiv · showhide
Recent zero-shot text-to-speech (ZS-TTS) systems can reproduce a speaker's voice with high fidelity from only a few seconds of reference speech, raising concerns over unauthorized voice cloning and impersonation. Speaker identity unlearning has recently emerged as an approach to selectively suppress this capability for speakers who opt out while preserving synthesis capability for other speakers. Although existing approaches reduce speaker similarity, preventing re-identification often faces severe degradation of speech quality. Motivated by this observation, we propose GUARD, a lightweight speaker identity unlearning framework that combines a learned speaker gate with speaker-agnostic activation steering on a frozen TTS backbone. The steering vectors are optimized using group-relative reward optimization to shift outputs from forget speakers toward population-level impostor similarity while preserving intelligibility and speech naturalness. On CosyVoice2, GUARD reduces forget-speaker similarity from 0.541 to 0.103 and re-identification accuracy in a 150-speaker gallery from 73.5% to 0.5%, while preserving retain-speaker reproduction. The results demonstrate that similarity reduction alone may not fully characterize successful speaker identity unlearning and highlight re-identification as a complementary criterion for its evaluation.
1. INTRODUCTION
Zero-shot TTS can reproduce unseen speakers from short recordings, creating risks of unauthorized cloning and impersonation; speaker unlearning seeks to honor post-deployment opt-outs while preserving synthesis for others. GUARD targets gallery-based re-identification by steering forget-speaker similarity toward the population-level impostor regime, with reported reductions in re-identification and no degradation in speech quality or retain-speaker reproduction.
1. INTRODUCTION
- Unlike anonymization and anti-cloning methods that protect individual speech samples, speaker identity unlearning addresses opt-out requests after deployment in a provider-controlled TTS setting.The introduction identifies unauthorized voice cloning, impersonation, voice phishing, and deepfake misuse as risks of ZS-TTS.Source: §1 · p. 1
- Existing speaker-unlearning methods reduce similarity but do not sufficiently suppress gallery-based re-identification, a risk that prior work has given limited attention.The stated goal is to suppress reliable re-identification while preserving linguistic content, speech naturalness, and retain-speaker reproduction.Source: §1 · p. 1
- GUARD combines GateNet with speaker-agnostic layer-wise activation steering and group-relative reward optimization on a frozen TTS backbone.The framework targets population-level impostor similarity, and its steering vectors support lightweight roster updates.Source: §1 · pp. 1–2, Figure 2 · p. 2
- On CosyVoice2, forget-speaker SIM fell from 0.541 to 0.103 and top-1 re-identification accuracy in a 150-speaker gallery fell from 73.5% to 0.5%, without degradation in speech quality or retain-speaker reproduction.The contribution summary also reports consistent improvements across other ZS-TTS backbones.Source: §1 · p. 2
2. RELATED WORK
Prior work has explored speaker unlearning, activation steering, and reward-based optimization for TTS. GUARD combines speaker-agnostic steering with post-generation rewards and evaluates residual identity through gallery-based re-identification.
2.1. Speaker Identity Unlearning for Zero-Shot TTS
- TGU and SGU fine-tune the full flow decoder using alternative speech targets, while TruS steers activations at inference; GUARD instead targets population-level impostor similarity and evaluates gallery-based re-identification.The related-work passage notes that stronger TruS steering can degrade naturalness and intelligibility, while weaker steering leaves substantial re-identification.Source: §2.1 · p. 2
2.2. Activation Steering for TTS
- Prior TTS activation-steering work controls emotion or accent, whereas GUARD optimizes speaker-agnostic layer-wise steering vectors for speaker identity unlearning using post-generation rewards.The cited studies use activation differences or trainable steering vectors for emotion control, and steering for accent modification while preserving speaker characteristics.Source: §2.2 · p. 2
2.3. Reward-Based Optimization for TTS
- Unlike prior reward-based TTS adaptations in weight space, GUARD optimizes layer-wise activation-steering vectors on a frozen backbone using anonymity, WER, and UTMOS rewards.The anonymity reward targets population-level impostor similarity; WER and UTMOS rewards address intelligibility and speech naturalness.Source: §2.3 · p. 2
3. METHOD
GUARD freezes the TTS backbone and steers decoder activations with layer-wise vectors weighted by a speaker-dependent gate. It optimizes the vectors to move forget-speaker similarity toward population-level impostor similarity while preserving linguistic content and speech naturalness.
3.1. Problem Formulation and Overview
- GUARD freezes the TTS backbone and adds speaker-agnostic, layer-wise steering vectors to flow-decoder activations, weighted by a coefficient for the input speaker.Each layer's trainable vector is shared across forget speakers.Source: §3.1 · pp. 2–3
3.2. GateNet for Selective Steering
- GateNet predicts a steering coefficient from the speaker embedding, while its hard threshold removes steering for inputs below the threshold.GateNet is trained separately and remains fixed during steering-vector optimization.Source: §3.2 · p. 3
3.3. Group-Relative Reward Optimization
- Group-relative optimization scores sampled steering-vector candidates using anonymity, intelligibility, and naturalness rewards, then updates the vectors while GateNet remains fixed.Rewards are normalized within each candidate group; at inference, GUARD uses the learned vectors without candidate sampling or reward computation.Source: §3.3 · p. 3
4. EXPERIMENTS
Experiments show that GUARD sharply reduces gallery-based re-identification while keeping speech quality and retain-speaker reproduction close to the original backbone. The analysis motivates targeting population-level impostor similarity rather than minimizing speaker similarity without limit.
4.1. Experimental Settings
- Evaluation uses held-out LibriTTS clean-460 utterances, 50 forget speakers and 995 retain speakers, with frozen CosyVoice2 as the primary backbone and F5-TTS and FireRedTTS for generalization.Each synthesis example uses a 3-second prompt; the forget-speaker evaluation utterances are also held out from steering-vector optimization.Source: §4.1 · p. 3
4.2. Results
- At α = 2.5, TruS reaches 2.0% id@150 but WER rises to 122.0% and UTMOS falls to 1.35; SGU and TGU remain at 56.5% and 56.0% id@150.SGU and TGU fine-tune the full flow decoder.Source: §4.2 · p. 4
- Forget-speaker SIM falls from 0.541 to 0.103 and id@150 from 73.5% to 0.5%, while WER, UTMOS, and retain-speaker metrics remain close to the original backbone.GUARD's detectability AUROC is 0.579, closest to 0.5 among evaluated methods.Source: §4.2 · p. 4
- Removing the impostor target lowers SIM from 0.103 to 0.056 without reducing re-identification and raises AUROC from 0.579 to 0.706; removing UTMOS or WER rewards degrades their respective measures.Without UTMOS reward, UTMOS falls to 1.25 and WER rises to 11.4%; without WER reward, WER rises to 6.7% and id@150 to 2.5%.Source: §4.2 · p. 4, Table 2 · p. 4
- A newly opted-out speaker's roster update takes GUARD 9.9 minutes, versus 6.3–7.8 hours for SGU and TGU, without TTS-backbone retraining.GUARD updates only its 24.8K-parameter GateNet while keeping steering vectors fixed.Source: §4.2 · p. 4, Table 3 · p. 4
- GUARD reduces id@150 from 95.5% to 6.0% on F5-TTS and from 91.5% to 18.0% on FireRedTTS while maintaining intelligibility and speech naturalness.Table 4 reports backbone generalization and defines the mean population-level impostor similarity as 0.129.Source: §4.2 · p. 4, Table 4 · p. 4
4.3. Analysis
- SIM above the population-level impostor regime can retain speaker-specific information that supports gallery re-identification, while reducing SIM below that regime does not further lower re-identification.Below-regime SIM can make protected outputs more distinguishable from genuine impostor recordings, motivating the population-level regime as a target.Source: §4.3 · p. 4
- GateNet selects whether steering is applied, while shared speaker-agnostic layer-wise vectors define the intervention; roster updates therefore require only GateNet updates.The vectors remain fixed as the forget roster changes, avoiding backbone retraining.Source: §4.3 · p. 4
5. CONCLUSION
The conclusion presents GUARD as a lightweight approach to speaker unlearning that reduces re-identification while preserving speech quality and retain-speaker reproduction. It identifies speaker-adaptive steering as future work.
5. CONCLUSION
- GUARD combines GateNet with speaker-agnostic layer-wise steering vectors optimized by group-relative reward optimization on a frozen ZS-TTS backbone.The framework drives forget-speaker SIM toward the population-level impostor regime while reducing gallery-based re-identification and preserving speech quality and retain-speaker reproduction.Source: §5 · p. 4
- For newly opted-out speakers, GUARD keeps the steering vectors fixed and requires only GateNet retraining.The conclusion proposes future work on speaker-adaptive vectors for individualized suppression while retaining lightweight updates.Source: §5 · p. 4
6. COMPLIANCE WITH ETHICAL STANDARDS
The study used existing public datasets and collected no new data; it required no ethical approval.