Silence-the-Mimic: Accelerating Imperceptible Perturbation Generation Against Voice Cloning

I. INTRODUCTION

• Recent VC and TTS models can clone a speaker’s voice from seconds of audio while generating arbitrary linguistic content, raising privacy, property, and reputation risks.Source: §I · p. 1
• Detection identifies synthetic speech after an attack, whereas protection perturbs speech before publication; existing protection methods often require sensitive quality-loss tuning and hundreds or thousands of iterations.Source: §I · p. 1
• STM applies psychoacoustic masking thresholds to frequency-domain perturbations and enforces perceptibility bounds, directing gradient updates solely toward the perturbations.The method uses STFT and COLA for stable inverse reconstruction.Source: §I · p. 1
• Experiments across state-of-the-art VC and TTS models report protection comparable to or better than leading white-box baselines while substantially accelerating perturbation generation.Source: §I · p. 1

II. RELATED WORK

• Audio watermarking embeds identifiers for later recovery, passive detection examines synthesis artifacts, and adversarial perturbation defenses target the cloning process through different generation and application choices.Source: §II · p. 2

III. METHODOLOGY

• STM uses a psychoacoustic hard-clamping strategy in the frequency domain instead of perceptual loss functions, controlling distortion while eliminating additional audio-refinement steps.Source: §III · p. 2

A. Problem Setup

• The threat model has a victim publishing speech, a third-party attacker using public samples to synthesize spoofed speech, and a verifier authenticating the victim.The attacker is not modeled as directly capturing the victim’s speech.Source: A. Problem Setup · p. 2
• The attacker’s synthesis model combines a speaker encoding of a target utterance with arbitrary linguistic content to generate speech mimicking the target’s vocal characteristics.Source: A. Problem Setup · p. 2
• Protection perturbs each utterance before publication so cloned speech is rejected by speaker verification while the protected audio remains perceptually unobtrusive and usable.The verifier accepts a trial when its similarity score exceeds a predefined threshold.Source: A. Problem Setup · p. 2
• The perceptual constraint uses H to quantify dissimilarity, with Hlimit setting maximum allowable distortion across feature channels such as time-frequency bins.Direct optimization of the stated formulation is often infeasible, so the method instead modifies the speaker identity perceived by the synthesizer.Source: A. Problem Setup · p. 2

B. Method Formulation

• STM converts speech to a spectrogram, estimates masking thresholds, and applies perceptibility constraints to generate protected speech.The constraint uses tunable headroom and floorroom tolerances for audible and inaudible time-frequency bins.Source: B. Method Formulation · pp. 2–3
• STM uses an STFT spectrogram with a Hann window and 50% frame overlap; the COLA condition supports iSTFT reconstruction without additional artifacts.The algorithm retains the magnitude component and leaves phase unchanged.Source: B. Method Formulation · pp. 2–3
• The fully masked setting leaves audible bins unchanged and keeps inaudible-bin perturbations strictly below the masking threshold.STM can alternatively relax the audible-bin constraint by a margin c > 0, trading a small reduction in perceptual quality for increased protection strength.Source: B. Method Formulation · p. 3
• Hard clamping returns excessive spectrogram values to the headroom and floorroom tolerances at the end of each training iteration.The clamp applies min and max element-wise.Source: B. Method Formulation · p. 3
• STM shifts the speaker-encoder output toward a target speaker embedding, rather than directly optimizing the verifier score.The target-based approach guides perturbations toward a valid speech-embedding subspace, which the paper reports yields higher audio quality than threshold-based approaches.Source: B. Method Formulation · p. 3
• A multi-step one-dimensional total-variation loss smooths perturbations across multiple temporal scales, reducing rapid oscillations.The loss considers multiple step sizes rather than only adjacent frames.Source: B. Method Formulation · p. 3

A. Dataset

• 500 utterances were selected from 100 CSTR VCTK speakers, with five utterances randomly chosen per speaker after excluding utterances without transcripts.The corpus is the CSTR VCTK Corpus.Source: A. Dataset · p. 4
• The longest remaining utterance from each speaker formed the embedding bank for target-based adversarial training.This yielded one embedding-bank utterance per speaker.Source: A. Dataset · p. 4
• All audio was downsampled to 16 kHz before training or inference with any model.Source: A. Dataset · p. 4

B. Target Models and Baselines

• The evaluation includes GPT-SoVITS, FreeVC, QuickVC, and TriAAN-VC, selected for state-of-the-art zero-shot or few-shot voice-cloning capability.Source: B. Target Models and Baselines · p. 4
• STM is compared with AntiFake and VoiceGuard, white-box adversarial-training defenses that generate time-domain perturbations and use loss-based audio-quality control.AntiFake integrates quality-control losses during perturbation generation; VoiceGuard refines audio afterward and selects its optimal weight by binary search.Source: B. Target Models and Baselines · p. 4

C. Evaluation Metrics

• Protection effectiveness is evaluated with TitaNet, ECAPA-TDNN, and Pyannote.audio speaker-verification models.Source: C. Evaluation Metrics · p. 4
• Audio quality is assessed with NISQA-predicted MOS and Noisiness, which capture perceived quality and degradation from additive noise, respectively.Source: C. Evaluation Metrics · p. 4
• PESQ measures perceptual speech quality, while STOI evaluates speech intelligibility.Source: C. Evaluation Metrics · p. 4

D. Training Details

• 512 FFT size and 256 hop size satisfy COLA; spectrograms are logarithmic and the psychoacoustic model is adopted from.Source: D. Training Details · p. 4
• Speaker encoders from GSV, FreeVC, QuickVC, and TriAAN-VC serve as enc_spk; the target embedding is selected as the fifth-largest distance from the original.Source: D. Training Details · p. 4
• 80 iterations use Adam at learning rate 1 with β1 = 0.9 and β2 = 0.999, combining L1 and cosine similarity losses; TV loss is added for FreeVC and QuickVC.Source: D. Training Details · pp. 4–5
• AntiFake uses 80, 500, or 1000 iterations, with 80 matching STM’s budget and 1000 its official setting; VoiceGuard uses 3000 training and 1500 refinement steps.Source: D. Training Details · p. 5
• ECAPA-TDNN, Pyannote, and TitaNet thresholds are 0.7318, 0.6466, and 0.7507, respectively, set using their VCTK Equal Error Rates.Source: D. Training Details · p. 5
• All methods run on one NVIDIA A100 80GB PCIe GPU; training time averages across the sampled dataset and excludes data I/O.Source: D. Training Details · p. 5

E. Main Results

• 1.59× faster than AntiFake on average (2.19 vs. 3.48 s/utterance) at 80 iterations, STM also records the lowest protection runtime across target models.The matched budget controls for optimization iterations.Source: E. Main Results · p. 5
• STM achieves the highest or second-highest rejection rates on nearly all SV systems while preserving superior perceptual quality versus baselines.The comparison covers SV rejection rates and audio-quality metrics.Source: Table I · p. 5, E. Main Results · p. 5

Robustness Against Purification Attacks

• Rejection rates stay above 90% under all tested purification operations, including stronger frequency-filtering and Griffin–Lim reconstruction attacks.The experiments apply four preprocessing operations to protected utterances before FreeVC inference.Source: F. Further Experiments · pp. 5–6, Table III · p. 6

Black-box Transferability

• On unseen ElevenLabs, STM achieves 71.50% SV rejection, 3.37 NISQA-predicted MOS, and 2.47 PESQ, versus AntiFake’s 81.80%, 2.30, and 1.22, respectively.STM retains substantial black-box protection while providing considerably better audio quality.Source: F. Further Experiments · p. 6

Effect of Quality Control Loss Weight in AntiFake

• For GSV, FreeVC, and QuickVC, AntiFake’s quality degrades when its protection approaches STM’s, while increasing its quality-control weight further reduces protection relative to STM.TriAAN-VC shows the opposite effect when adjusting the quality-control weight; the authors attribute this possibly to its variable-length speaker embeddings.Source: F. Further Experiments · p. 6

Effect of Targeted Embedding Shift

• Without directed embedding shift, audio quality substantially degrades for GSV, FreeVC, and QuickVC compared with STM’s targeted shift.The modified loss reverses the sign of each term and replaces the target speaker embedding.Source: F. Further Experiments · p. 6

Effect of Multi-Step TV Loss

• Without multi-step TV loss, perturbations vary abruptly and produce noticeable audio artifacts, reducing the practicality of protection.The corresponding evaluation is reported in Table VI.Source: F. Further Experiments · p. 6, Table VI · p. 6

• Experiments on state-of-the-art voice-cloning models show competitive protection with higher audio quality and substantially reduced processing time.Source: §V · p. 6
• The method perturbs audio in the frequency domain under psychoacoustic guidance to enforce strict perceptual limits.Source: §V · p. 6
• The method is presented as a fast and effective way to protect speech from voice-cloning attacks.Source: §V · p. 6