PaperlayerView exact revision on arXiv

Source-linked AI summary

Unlocking Fine-Grained and Within-Utterance Speaking Style Control in Prompt-Based Text-to-Speech Models

Jaehoon Kang, Yejin Lee, Yoonji Park, Kyuhong Shim

arXiv:2605.27376v1cs.CLcs.AI

TL;DR

Prompt-based TTS supports natural-language style control, but fine-grained attribute changes and time-varying transitions remain limited. The paper proposes training-free embedding interpolation and inference-time decoding interventions; its conclusion reports smoother transitions and preserved speaker identity alongside improved control.

  • Problem

    Prompt-based TTS often uses discrete attribute prompts and a fixed global style, limiting fine-grained control and within-utterance transitions.

    Source: §1 · p. 1

  • Method

    The paper uses training-free embedding interpolation for inter-utterance control and KV-cache swapping plus sliding-window masking for within-utterance transitions.

    Source: §1 · p. 2

  • Results

    Interpolation improves monotonicity and precise control without degrading speech quality, while cache swapping and window masking produce clearer, smoother transitions while largely preserving speaker identity.

    Source: §5 · p. 8

  • Takeaways & Limitations

    The approach provides fine-grained speaking-style control in prompt-based TTS across utterances and within an utterance.

    Source: §5 · p. 8

    Evaluation focuses on pitch, speed, and gender; intra-utterance window size trades transition strength against speaker similarity and speech quality.

    Source: Limitations · p. 9

Abstract

from arXiv · show

While prompt-based text-to-speech (TTS) models enable natural language-driven speaking style control, they often provide limited fine-grained control and apply a single global style across an utterance. This restricts practical use cases that require continuous style attribute interpolation across utterances and time-varying style transitions within a single utterance. In this paper, we propose novel techniques to achieve both capabilities in existing prompt-based TTS models. For inter-utterance style interpolation, we compute direction vectors between contrastive style prompts in the embedding space and perform simple interpolation, enabling smooth transitions between style characteristics. For intra-utterance style transition, we first identify a strong attention bias toward early tokens in autoregressive TTS decoders, causing the initial audio realization to dominate subsequent generation. To mitigate this effect, we introduce KV-cache swapping and sliding-window attention masking. Experiments demonstrate that our proposed inter-utterance interpolation achieves a 99-100% success rate in gender conversion, up to 36 Hz pitch variation, and up to 1.6 syllables-per-second speed change. Our intra-utterance transition maintains a speaker similarity of 0.81-0.91 and achieves perceptual smoothness scores of 3.48-4.48.

1 Introduction

Prompt-based TTS offers natural-language style control, but discrete prompts and a fixed global style limit fine-grained and time-varying changes. This work introduces training-free inference-time methods for continuous inter-utterance control and within-utterance transitions.

1 Introduction

  • Current prompt-based TTS offers coarse, discrete attribute control and a fixed global style, limiting fine-resolution and time-varying changes.The paper motivates both gradual attribute changes across utterances and transitions within one utterance.Source: §1 · pp. 1–2
  • Training-free inference-time methods provide continuous style control between utterances and style transitions within a single utterance.The approach combines embedding interpolation with mechanisms for intra-utterance transitions.Source: §1 · p. 2
  • Early generated speech can dominate later generation and weaken natural-language style guidance, while naïve prompt swaps or embedding interpolation fail to produce reliable within-utterance transitions.The paper terms this early-token dominance style self-referencing.Source: §1 · p. 2
  • KV-cache swapping and sliding-window self-attention masking reduce early-token dominance to unlock smooth within-utterance transitions without fine-tuning.Both inference-time mechanisms target the influence of early-generated tokens.Source: §1 · p. 2
  • The evaluation demonstrates control over pitch, speed, and gender attributes while maintaining speech quality.The introduction describes these as the evaluated attributes and quality criterion.Source: §1 · p. 2

2 Related Work

Prior work includes reference-audio and natural-language approaches to style control, alongside methods for fine-grained prosodic or emotional variation. The paper situates smooth, time-varying transitions as a remaining challenge for prompt-based TTS.

2.1 Style Controllable Text-to-Speech

  • Reference-based systems require matching audio, while natural-language prompts avoid that dependency but generally apply static style globally rather than time-varying control.Prompt-based methods can describe attributes such as gender, pitch, and speed, but often lack mechanisms for within-utterance modulation.Source: §2.1 · pp. 2–3
  • Figure 2 depicts embedding-direction interpolation as a smooth trajectory between male and female style representations.The visualization shows interpolated embeddings at α values from 0.4 to 1.6.Source: Figure 2 · p. 3

2.2 Fine-Grained Speaking Style Control

  • Prior fine-grained methods support word-level emotion, vocalizations, or arousal-valence control, while smooth intra-utterance style transitions remain an open challenge.The cited approaches include synthetic-data interleaving, multi-stage inference, and activation steering focused on distinct emotional attributes.Source: §2.2 · p. 3

3 Inter-Utterance Style Interpolation

This section develops training-free inter-utterance style interpolation by manipulating contrastive-prompt embeddings. It reports controlled gender, pitch, and speed outcomes across interpolation strengths.

3.1 Analysis: Style Vectors in Embedding Space

  • Embeddings for each attribute cluster together and remain separated from contrastive attribute clusters, suggesting linear interpolation can yield continuous transitions.Source: §3.1 · p. 3

3.3 Experimental Results

  • At α≥1.5, conversion reaches 98% Female→Male and 100% Male→Female; at α=2.0, F0 shifts reach 36.1 Hz decrease and 35.8 Hz increase.Source: Figure 3 · p. 4, §3.3 · pp. 4–5

3.2 Method

  • The method computes a direction between source and target prompt embeddings and applies a scaled version only at attribute-token positions.Source: §3.2 · p. 4

3.3 Experimental Results

  • At α=2.0, Quick→Slow reduces speaking rate by 1.4 SPS and Slow→Quick increases it by 1.6 SPS.Source: §3.3 · p. 5
  • At α=2.0, conversion directions score 1.62–2.00 for perceived style change while naturalness MOS remains above 3.99.Source: §3.3 · p. 5

4 Intra-utterance Style Transition

The section explains why mid-generation prompt changes fail and presents KV-cache swapping with sliding-window attention masking as a way to counter style self-referencing. Results show a trade-off: smaller windows strengthen transitions, while larger windows better preserve speaker similarity and smoothness.

4.1 Intuition: Why Naïve Approaches Fail

  • Cross-attention to style tokens changes during early generation but then becomes nearly fixed; early audio representations consequently dominate later autoregressive generation.The decoder’s “set-and-maintain” pattern is termed style self-referencing, making it resistant to new style information.Source: §4.1 · pp. 5–6, Figure 4 · p. 5

4.2 Method

  • At the transition, the method swaps in a target-style KV-cache from a parallel decoder and replaces the cross-attention style representation; the swapped region is n = ntext + k.The additional k buffer covers early audio tokens where style characteristics are actively encoded.Source: §4.2 · p. 6

4.3 Experimental Results

  • Window 256 yields +27.4 Hz for Low→High pitch and −2.29 SPS for Quick→Slow speed, versus +5.5 Hz and −1.34 SPS under Full; SIM is 0.81 versus 0.90–0.91.Table 3 reports results with KV-cache buffer size k = 48.Source: Table 3 · p. 7, §4.3 · p. 7
  • Window 256 reaches 88.0–96.2% transition detection versus 55.6–84.6% under Full; smoothness exceeds 3.48 across configurations and reaches 4.48 with larger windows.Table 3 defines Trans. as listener-perceived transitions and Smoothness on a 1–5 scale.Source: Table 3 · p. 7, §4.3 · pp. 7–8

4.2 Method

  • Sliding-window masking restricts attention to the replaced target-style initial region and recent tokens, combining new-style influence with local context for smoother progression.The mask addresses the source-style information retained in intermediate tokens that remain accessible under standard self-attention.Source: §4.2 · pp. 6–7

4.3 Experimental Results

  • Smaller windows forget intermediate source-style tokens faster, strengthening and abrupting transitions; larger windows retain more of them, yielding smoother but weaker changes.The trade-off follows from how much attention the decoder can give to intermediate tokens after the KV-cache swap.Source: §4.3 · p. 8

4.4 Analysis

  • Replacing only the style representation produces minimal style change, supporting the need to swap the KV-cache for effective intra-utterance transitions.Table 4 reports style-transition results without a KV-cache swap.Source: §4.4 · p. 8, Table 4 · p. 8
  • For High→Low pitch, swapping only text-region cache tokens reverses the change (+6.8 to +16.2 Hz), while including acoustic tokens with k ≥32 yields −5.3 to −12.4 Hz.The reported successful conversion holds across all tested window sizes when k ≥32.Source: §4.4 · p. 8, Figure 6 · p. 8

5 Conclusion

The conclusion presents training-free methods for fine-grained style control in prompt-based TTS, covering both continuous inter-utterance interpolation and intra-utterance transitions. It identifies style self-referencing as a barrier to mid-utterance changes and uses cache swapping and attention masking to mitigate it.

5 Conclusion

  • The paper presents training-free fine-grained control for prompt-based TTS through both inter-utterance interpolation and intra-utterance style transition.The approach targets continuous attribute control and within-utterance changes.Source: §5 · p. 8
  • Interpolating along style directions in the text encoder’s embedding space adjusts attribute intensity smoothly and improves monotonicity without degrading speech quality.This is the conclusion’s description of inter-utterance control.Source: §5 · p. 8
  • KV-cache swapping and sliding-window attention masking mitigate style self-referencing to produce clearer, smoother transitions while largely preserving speaker identity.The conclusion identifies self-referencing in autoregressive decoding as limiting mid-utterance prompt changes.Source: §5 · p. 8

Limitations

The paper’s evaluation and method have defined scope boundaries: three style attributes, a window-size trade-off, and validation only on prompt-based autoregressive TTS models.

Limitations

  • The study focuses on pitch, speed, and gender; emotion and intonation are identified as possible extensions.The conclusion frames these as attributes for future extension.Source: Limitations · p. 9
  • Smaller windows make transitions more pronounced but reduce speaker similarity, while larger windows better preserve speech quality but weaken the transition effect.The authors identify mitigating this trade-off as future work.Source: Limitations · p. 9
  • The methods have been validated only on natural-language prompt-based autoregressive TTS; applicability to non-autoregressive or diffusion models remains open.The paper notes those model types may lack KV-cache structures.Source: Limitations · p. 9

Ethics Statements

The ethics statement notes that continuous style interpolation could be misused to imitate or manipulate a specific speaker’s voice. It names fraud, disinformation, and non-consensual voice cloning as risks and points to detection and watermarking as parallel safeguards.

Ethics Statements

  • Continuous style interpolation could be exploited to imitate or manipulate a specific speaker’s voice.The authors frame this as a potential misuse, not the stated aim of the control methods.Source: Ethics Statements · p. 9
  • The stated potential harms include fraud, disinformation, and non-consensual voice cloning.These risks are connected to misuse of speaker imitation or manipulation.Source: Ethics Statements · p. 9
  • The authors call for synthetic-speech detection and audio watermarking to be developed in parallel to help address these risks.The passage presents these as mechanisms to prevent misuse.Source: Ethics Statements · p. 9

A Experimental Details

The experiments assess pitch, speaking rate, speaker identity, and gender conversion using objective measures.

A.1 Objective Evaluation Metrics

  • Pitch is measured as average fundamental frequency (F0) in Hz using PENN.PENN is identified as Pitch Estimator Neural Network.Source: §A.1 · p. 11
  • Speaking rate is measured in syllables per second (SPS), calculated as syllables divided by utterance duration.Source: §A.1 · p. 11
  • Speaker identity preservation is assessed by cosine similarity between audio-segment embeddings extracted with fine-tuned WavLM.Source: §A.1 · p. 11
  • Gender-conversion success rate is measured using a pretrained gender-classification model.Source: §A.1 · p. 11

A.2 Dataset and Model

The experiments use Parler-TTS-mini on LibriTTS-R test samples, with objective and participant evaluations tailored to each style-control task.

A.2 Dataset and Model

  • 400 LibriTTS-R test sentences were selected for inter-utterance interpolation, and 400 samples with text lengths of 50–70 tokens for intra-utterance variation.All samples were generated with Parler-TTS-mini.Source: §A.2 · p. 11

A.3 Inter-Utterance Style Interpolation

  • Inter-utterance objective evaluation varied interpolation strength across specified α values, measuring gender conversion at 0.5, 1.0, 1.5, and 2.0 and pitch and speed across −1.0 to 2.0.The pitch and speed set included −1.0, −0.5, 0.5, 1.0, 1.5, and 2.0; gender used the subset stated.Source: §A.3 · p. 11
  • Fifteen participants rated inter-utterance conversion direction on a −2 to +2 scale and audio quality on a 1-to-5 MOS scale.Participants evaluated samples at α values 0.5, 1.0, and 2.0.Source: §A.3 · p. 11

A.4 Intra-Utterance Style Transition

  • Intra-utterance objective evaluation compares pitch and speaking-rate differences between each sample’s first and last three seconds, alongside speaker similarity.Speaker similarity uses cosine similarity between speaker embeddings to assess identity preservation across the transition.Source: §A.4 · p. 11
  • Participants judged whether they detected the intended intra-utterance transition and rated its naturalness on a 1-to-5 scale.The detection rate is reported as Trans. (%) in Table 3.Source: §A.4 · p. 11

B Full Vector vs. Attribute-Only Vector Interpolation

Shifting all style-token positions performs comparably to shifting attribute tokens alone, while requiring extra tuning without clear benefit.

B Full Vector vs. Attribute-Only Vector Interpolation

  • The comparison shifts the direction vector either only at attribute-token positions or across all tokens, with a separate β factor for non-attribute tokens.Non-attribute β values ranged from −0.5 to 1.0 in increments of 0.25.Source: B Full Vector vs. Attribute-Only… · pp. 11–12
  • 99–100% gender-conversion success at α = 2.0, pitch changes up to −32.4 Hz and +35.4 Hz, and speed changes up to −1.5 and +1.7 SPS were reported for all-token shifting.These results were comparable to attribute-only interpolation across the three attributes.Source: Figure 7 · p. 12, B Full Vector vs. Attribute-Only… · p. 12
  • The full-vector approach achieved comparable success rates, acoustic changes, and speaker similarity, but added non-attribute-token tuning without clear benefits.The paper therefore adopts the simpler attribute-only strategy in its main experiments.Source: B Full Vector vs. Attribute-Only… · p. 12, Table 5 · p. 13

C Quantitative Analysis of Early-Token Attention Bias

Attention-weight variance analysis finds that style-token attention changes during early generation and stabilizes afterward; the appendix also documents prompts and the transition-generation procedure.

C Quantitative Analysis of Early-Token Attention Bias

  • At each audio position, the analysis computes variance across the attention-weight distribution over style tokens.The distribution is denoted a_t, and its mean attention weight at that position is also defined.Source: C Quantitative Analysis of Early-Token… · p. 13

D Style Prompt Examples

C Quantitative Analysis of Early-Token Attention Bias

  • Attention-weight variance fluctuates during initial generation and stabilizes afterward, supporting the finding that style-token attention is actively updated early and becomes fixed later.The analysis tracks variance across style tokens at each audio position and multiple decoder layers.Source: Figure 10 · p. 13, C Quantitative Analysis of Early-Token… · p. 13

E Algorithm for Intra-utterance Style Variation

  • The transition algorithm extracts a direction vector from source and target keywords and initializes separate decoders with source and target style embeddings.It applies the direction vector at the keyword position and precomputes the target KV-cache.Source: E Algorithm for Intra-utterance Style… · pp. 13–14

12: // Phase 2: Generate until transition with sliding window

  • Before the transition point, Decoder A generates audio tokens using a sliding-window attention mask.Source: §12 · p. 14

17: // Phase 3: Swap at the transition point

  • At the transition point, the algorithm replaces Decoder A’s key-value cache with Decoder B’s cache.Source: §17 · p. 14

20: // Phase 4: Continue generation with sliding window

  • After the transition, generation continues to the end with the sliding-window mask, then returns the generated audio sequence.Source: §20 · p. 14
Loading 2605.27376v1…