PaperlayerView exact revision on arXiv

Source-linked AI summary

TTSlow: Slow Down Text-to-Speech with Efficiency Robustness Evaluations

Xiaoxue Gao, Yiming Chen, Xianghu Yue, Yu Tsao, Nancy F. Chen

arXiv:2407.01927v1eess.ASeess.SP

TL;DR

TTS systems need stable, timely generation under minor input perturbations, but their efficiency robustness against adversarial attacks is underexplored. TTSlow addresses this gap with text and speaker-embedding attacks designed to encourage endless generation. Across two TTS models and three datasets, it slows generation and achieves high attack success rates.

Abstract

from arXiv · show

Text-to-speech (TTS) has been extensively studied for generating high-quality speech with textual inputs, playing a crucial role in various real-time applications. For real-world deployment, ensuring stable and timely generation in TTS models against minor input perturbations is of paramount importance. Therefore, evaluating the robustness of TTS models against such perturbations, commonly known as adversarial attacks, is highly desirable. In this paper, we propose TTSlow, a novel adversarial approach specifically tailored to slow down the speech generation process in TTS systems. To induce long TTS waiting time, we design novel efficiency-oriented adversarial loss to encourage endless generation process. TTSlow encompasses two attack strategies targeting both text inputs and speaker embedding. Specifically, we propose TTSlow-text, which utilizes a combination of homoglyphs-based and swap-based perturbations, along with TTSlow-spk, which employs a gradient optimization attack approach for speaker embedding. TTSlow serves as the first attack approach targeting a wide range of TTS models, including autoregressive and non-autoregressive TTS ones, thereby advancing exploration in audio security. Extensive experiments are conducted to evaluate the inference efficiency of TTS models, and in-depth analysis of generated speech intelligibility is performed using Gemini. The results demonstrate that TTSlow can effectively slow down two TTS models across three publicly available datasets. We are committed to releasing the source code upon acceptance, facilitating further research and benchmarking in this domain.

I. INTRODUCTION

TTSlow frames minor-input efficiency robustness as an underexplored security problem for real-time TTS and proposes adversarial attacks to expose unstable, excessively long generation. It combines text and speaker-embedding attacks and evaluates them across TTS settings and datasets.

I. INTRODUCTION

  • TTSlow targets the largely unexplored robustness of TTS inference efficiency against minor perturbations, which can delay security-critical speech generation.The paper motivates this problem through banking, home-security, and automotive applications.Source: §I · p. 1
  • TTSlow discovers nearly imperceptible malicious inputs that encourage endless speech generation and thereby reduce inference efficiency.Its design is based on the link between longer speech outputs, more inference steps, and longer generation time.Source: §I · p. 1
  • The approach includes TTSlow-text, using homoglyph and character-swap perturbations, and TTSlow-spk, using gradient optimization on speaker embeddings.The proposed objectives are designed for autoregressive and non-autoregressive TTS scenarios.Source: §I · pp. 1–2
  • The study systematically investigates inference-efficiency robustness in TTS and evaluates both attack strategies on two TTS models across three public datasets.The authors position this as the first study of this robustness property in TTS.Source: §I · p. 2

A. Text-to-speech Models

The paper distinguishes autoregressive and non-autoregressive TTS by how they determine speech length, then positions TTSlow as an efficiency-oriented adversarial attack addressing an identified robustness gap. It formulates the attack as constrained optimization under a perturbation budget.

A. Text-to-speech Models

  • Autoregressive TTS models stop using predicted stop tokens, whereas non-autoregressive models determine output length with duration predictors.SpeechT5 and VITS are selected as representative autoregressive and non-autoregressive backbones, respectively.Source: A. Text-to-speech Models · p. 2

B. Adversarial Attacks

  • Efficiency-oriented attacks target inference efficiency, but TTS robustness against adversarial security threats remains insufficiently explored.The paper presents TTSlow to study this gap for TTS systems.Source: B. Adversarial Attacks · p. 2

III. TTSLOW

  • TTSlow is introduced as an approach containing two attack strategies, TTSlow-spk and TTSlow-text, for evaluating TTS efficiency robustness.The formulation section defines the attack problem before presenting the two strategies.Source: §III · p. 2

A. Problem Formulation

  • TTSlow maximizes the victim TTS output sequence length while constraining the adversarial perturbation to remain within a maximum allowed threshold.Here, x is the benign input, f is the victim model, and len_f(·) is its output sequence length.Source: A. Problem Formulation · p. 2

B. Overview of TTSlow

TTSlow perturbs either the speaker embedding or text input to lengthen generated speech and reduce inference efficiency. The speaker-embedding attack uses projected gradient optimization under ℓ2 or ℓinf constraints.

B. Overview of TTSlow

  • A small perturbation increases synthesized speech from approximately 1.75 seconds to approximately 4.5 seconds, producing a longer generation process and reduced inference efficiency.TTSlow-spk targets multi-speaker models, while TTSlow-text applies to multi-speaker and single-speaker character-input models.Source: B. Overview of TTSlow · pp. 2–3

C. TTSlow-spk

  • TTSlow-spk keeps text fixed and adds an optimized perturbation to the speaker embedding before feeding it to the victim TTS model.The adversarial speaker embedding is updated through gradient optimization.Source: C. TTSlow-spk · p. 3, 1) Gradient Optimization · p. 3

1) Gradient Optimization:

  • The speaker attack iteratively updates perturbations using the TTSlow-spk loss to encourage longer, no-stop speech generation.The loss is optimized to slow speech generation while gradient-based updates modify the perturbation.Source: 1) Gradient Optimization · p. 3
  • The projection step maps the adversarial speaker embedding to the closest feasible point under an ℓp norm, with both ℓ2 and ℓinf considered.This projection enforces the permitted perturbation constraint.Source: 1) Gradient Optimization · p. 3

2) Attack Methodology:

TTSlow uses speaker-embedding and text-input attacks, with text attacks combining character swaps and homoglyph replacements under near-imperceptibility constraints. Its differentiable approximation ranks replacements by their effect on an objective encouraging non-stop speech generation.

2) Attack Methodology:

  • TTSlow-spk adds a constrained perturbation to the speaker embedding and iteratively optimizes it to reduce the TTSlow-spk loss.The optimization seeks near imperceptibility as measured by a distance norm.Source: 2) Attack Methodology · p. 3

D. TTSlow-text

  • TTSlow-text combines character swaps and homoglyph replacements while preserving text length and keeping perturbations nearly imperceptible.The swap strategy replaces a character with another vocabulary character, while the homoglyph strategy uses visually similar characters.Source: D. TTSlow-text · p. 3, 1) Attack Methodology · p. 4

2) Differentiable Objective Approximation:

  • Because text inputs are nondifferentiable, TTSlow-text uses a differentiable objective based on text embeddings to approximate adversarial replacement effects.The replacement increment measures the objective-gradient increase from changing token t to token ˆt.Source: 2) Differentiable Objective… · p. 4
  • The TTSlow-text objective encourages the TTS model to generate non-stop speech frames, thereby slowing speech synthesis.The method evaluates candidate character replacements using the resulting objective effect.Source: 2) Differentiable Objective… · p. 4

3) Perturbation Generation and Candidates Selection:

TTSlow generates constrained adversarial candidates and selects the fittest replacements, while defining objectives separately for autoregressive and non-autoregressive TTS length-control mechanisms.

3) Perturbation Generation and Candidates Selection:

  • TTSlow generates 100 adversarial candidates for each text strategy, filters invalid candidates, and retains the top three by fitness for the next iteration.Candidates must satisfy the specified perturbation constraints before selection.Source: 3) Perturbation Generation and… · p. 4

IV. TTSLOW OBJECTIVE FUNCTIONS

  • TTSlow defines distinct objectives for autoregressive models with end-of-speech tokens and non-autoregressive models with length predictors.SpeechT5 and VITS are used to explain the two attack settings.Source: §IV · p. 4

A. Autoregressive TTS Model

  • SpeechT5 processes text and speaker-embedding inputs through text encoders, speech-decoder modules, and an autoregressive speech decoder.The architecture overview frames separate attacks on text and speaker embedding.Source: A. Autoregressive TTS Model · p. 4

1) TTSlow-spk on Autoregressive TTS Model:

For autoregressive SpeechT5, TTSlow-spk attacks speaker embeddings to encourage non-stop tokens, while TTSlow-text modifies text to encourage the same behavior.

1) TTSlow-spk on Autoregressive TTS Model:

2) TTSlow-text on Autoregressive TTS Model:

  • TTSlow-text iteratively changes text through character-swap and homoglyph strategies while leaving the speaker embedding unchanged.The stated example changes “I HAVE A PUPPY” to “I HATE A PUPPY” through a character swap.Source: 2) TTSlow-text on Autoregressive TTS… · p. 5
  • Both autoregressive attack objectives seek perturbations that encourage non-stop speech and reduce inference efficiency.The approach optimizes either LTTSlow-text or LTTSlow-spk toward this outcome.Source: 2) TTSlow-text on Autoregressive TTS… · p. 5

B. Non-autoregressive TTS Model

For non-autoregressive VITS, TTSlow targets duration prediction: speaker-embedding and text attacks optimize perturbations that extend synthesized speech duration.

B. Non-autoregressive TTS Model

1) TTSlow-spk on Non-autoregressive TTS Model:

  • TTSlow-spk perturbs the VITS speaker embedding while keeping text unchanged, then feeds it into the decoder and stochastic duration predictor.The attack updates the embedding perturbation through gradient optimization.Source: 1) TTSlow-spk on Non-autoregressive… · p. 5
  • The non-autoregressive TTSlow-spk objective maximizes cumulative predicted duration, extending the synthesized waveform and promoting continuous speech.Here D denotes the duration predicted by the stochastic duration predictor.Source: 1) TTSlow-spk on Non-autoregressive… · p. 5

2) TTSlow-text on Non-autoregressive TTS Model:

  • TTSlow-text modifies text through character-swap and homoglyph strategies to amplify duration outputs while preserving the speaker embedding.Its objective is defined to generate longer-duration speech.Source: 2) TTSlow-text on Non-autoregressive… · p. 5
  • For VITS, both attack objectives optimize perturbations through duration maximization to reduce inference efficiency.The summary states this objective for both LTTSlow-text and LTTSlow-spk.Source: 2) TTSlow-text on Non-autoregressive… · pp. 5–6

V. EXPERIMENTS

The experiments evaluate TTSlow using three public TTS datasets and multiple victim models, attack settings, and baselines. The setup covers both speaker-oriented and text-oriented attacks across English speech and accents.

V. EXPERIMENTS

  • The experiment section introduces datasets and experimental setup before evaluating attack performance.The setup explicitly compares multiple attack approaches across the selected datasets.Source: §V · p. 6, A. Datasets · p. 6

A. Datasets

  • Three datasets—LibriSpeech, LJ-Speech, and English dialects—provide evaluation data spanning English speech and English accents.The study uses the first 100 utterances or sentences from each dataset to manage computational demands.Source: A. Datasets · p. 6

1) TTS Model Architecture and Attack Details:

  • SpeechT5-tts supplies the main architecture, using Transformer encoder-decoder blocks and a HiFi-GAN vocoder for waveform generation.The attack configuration uses 100 iterations, beam size 3, and target scalar y=0 for no stopping.Source: 1) TTS Model Architecture and Attack… · p. 6
  • VITS-VCTK serves as the speaker-attack victim, while MMS-TTS supplies the text-attack victim and extends VITS to 1,107 languages.Speaker attacks randomly select one of 109 VCTK speakers for the benign embedding input.Source: 1) TTS Model Architecture and Attack… · p. 6

2) Baselines:

The baselines use simple character perturbations or Gaussian speaker-embedding noise, while TTSlow is evaluated through speech length, attack success, and efficiency-related outcomes. Across models and datasets, the proposed attacks produce longer speech than corresponding baselines.

2) Baselines:

  • The text baseline swaps or replaces characters at a fixed 0.05 ratio, whereas the speaker baseline applies Gaussian perturbations without TTSlow’s objective or gradient optimization.These baselines isolate the contribution of the proposed optimization-oriented attack procedures.Source: 2) Baselines · p. 6

VI. RESULTS AND DISCUSSION

  • Attack success is defined using speech at least 20% longer than the original, with ASR computed as successful samples divided by dataset size.The evaluation also includes decoded transcriptions and intelligibility scores for adversarial samples.Source: §VI · p. 6

A. Vulnerability of the Victim TTS model

  • TTSlow-spk (l2) increases speech length by 313% relative to the original clean sample, demonstrating substantial victim-model vulnerability.The result is reported for the proposed attack in the victim-model vulnerability analysis.Source: A. Vulnerability of the Victim TTS model · p. 6

B. The Effectiveness of Adversarial Attacks

C. Attack Success Rate

TTSlow achieves higher attack success rates than its baselines, including 100% ASR for TTSlow-spk on LJ-Speech. Its attacks also increase GPU energy consumption and inference time relative to clean and baseline generation.

C. Attack Success Rate

  • The proposed TTSlow attacks achieve higher ASR values than their corresponding baselines, and TTSlow-spk reaches 100% ASR on LJ-Speech, indicating successful attacks on all samples.Source: C. Attack Success Rate · p. 7

D. Impact of Inference Efficiency

  • On LJ-Speech, Gaussian KDE plots compare inference energy for clean data, baselines, and TTSlow-spk and TTSlow-text attacks.The comparison is made for SpeechT5 using measurements from an NVIDIA A40 GPU.Source: D. Impact of Inference Efficiency · pp. 7–8
  • TTSlow significantly increases inference energy over both clean and baseline models, reducing the overall inference efficiency of TTS models.Clean and baseline models consume comparable GPU energy in the reported comparison.Source: D. Impact of Inference Efficiency · p. 8
  • TTSlow significantly increases inference time relative to clean and baseline models, indicating longer computation for the victim TTS system.The figures associate extended speech duration with higher computation and longer inference.Source: D. Impact of Inference Efficiency · p. 8

E. Adversarial Samples

The adversarial samples show that TTSlow can extend speech duration while remaining nearly imperceptible in some text perturbations. Gemini-based evaluation additionally reports reduced understandability and error patterns such as repetitions, incorrect words, silence, and noise.

E. Adversarial Samples

  • TTSlow-text produces speech samples approximately 61% longer than the text baseline and 94% longer than clean samples in the illustrated case study.The examples use seven homoglyph replacements and are described as nearly human-imperceptible in some cases.Source: E. Adversarial Samples · pp. 8–9

F. Evaluation with Gemini

  • Gemini assigns lower understandability scores to TTSlow outputs than to clean and baseline samples.The evaluation uses scores from 0 to 10, with 10 indicating perfect understanding.Source: F. Evaluation with Gemini · p. 9
  • Observed adversarial errors include word repetitions, incorrect word generation, long silence, and noisy speech.These patterns indicate that the attacks affect both inference efficiency and speech-content intelligibility.Source: F. Evaluation with Gemini · p. 9

VII. CONCLUSION

  • The conclusion reports that TTSlow decreases efficiency in both autoregressive and non-autoregressive TTS models and outperforms baselines across three public datasets.Future work is identified as accuracy-oriented attacks and advanced defenses for TTS adversarial attacks.Source: §VII · p. 9
Loading 2407.01927v1…