PaperlayerView exact revision on arXiv

Source-linked AI summary

Controlling Speaking Rate in Autoregressive TTS via Activation Steering

Francesco Verdini, Antonis Asonitis, Aref Farhadipour, Marzieh Razavi, Pierre-Edouard Honnet, Vijeta Avijeet, Juan Pablo Zuluaga Gomez

arXiv:2609.33810v1cs.SDcs.CLcs.LGeess.AS

TL;DR

Autoregressive TTS offers little post-training control over speaking rate. This paper learns a calibrated rate axis from synthetic time-stretched speech and clamps a decoder activation at inference. Across three systems, clamping holds a wider stable range than additive steering at extremes, and calibrated rates transfer to Seed-TTS-Eval; the paper also identifies scope limits including English-only evaluation and utterance-level control.

  • Problem

    Autoregressive TTS produces natural speech but offers little control over delivery once trained, including speaking rate.

    Source: §I · p. 1

  • Method

    The method learns a calibrated rate axis from synthetic same-utterance rate contrasts, selects a causally effective decoder layer, and clamps the activation projection at inference.

    Source: §I · p. 2, C. Steering by clamping, not adding · p. 3

  • Results

    Across three systems, additive steering breaks first at the extremes while clamping holds a wider stable range; calibrated intensities also move rate on unseen Seed-TTS-Eval voices.

    Source: B. The application rule decides at the… · p. 4, §VI · p. 6

  • Takeaways & Limitations

    Clamping is the usual safe choice at strong targets, but additive steering can suffice at moderate targets and is cleaner on CosyVoice2’s fast side.

    Source: §VI · p. 6

    The evaluation is English-only and uses utterance-level control, without demonstrating time-varying within-sentence control or streaming deployment.

    Source: §V · p. 6

Abstract

from arXiv · show

Autoregressive text-to-speech (TTS) systems synthesize natural speech but, once trained, offer little control over speaking rate. We show that speaking rate can be steered at inference time, without retraining, by clamping a single decoder block's activation along a discovered speed axis. A decoder-block analysis recovers the rate axis, a neutral operating point, and a per-step intensity scale; at inference, the activation's projection onto this axis is set to a fixed scalar. Learning this direction from synthetically time-stretched and time-compressed speech yields rate control that largely preserves speaker identity, generalizes across model architectures, and maintains high naturalness in objective and human evaluations. Unlike standard additive steering, which breaks at the slow extreme, clamping remains stable on all three systems tested; at moderate targets, the better rule depends on the model. Finally, we show that rate information is decodable across layers but causally steerable only within a mid-depth window, and demonstrate the effectiveness of our approach on the public Seed-TTS-Eval benchmark.

I. INTRODUCTION

Autoregressive TTS produces natural speech but offers little post-training control over delivery, including speaking rate. The paper proposes a calibrated, training-free steering recipe and evaluates rate control, side effects, and transfer across systems.

I. INTRODUCTION

  • Speaking-rate control is limited after training: prompts and reference voices set it only coarsely, while fine-tuning can be expensive or unavailable.The introduction asks whether activation steering can provide natural, reliable rate control in autoregressive TTS.Source: §I · p. 1
  • The recipe probes a same-text slow/neutral/fast corpus, locates a causally effective decoder layer, and clamps the activation projection to a requested rate.The neutral operating point and intensity are calibrated from the contrast; the intervention is selected by a causal sweep, not probe accuracy alone.Source: Figure 1 · p. 1, §I · p. 2
  • Without retraining, rate can move across a wide range while preserving intelligibility, naturalness, and voice identity; clamping remains stable at strong intensities where additive edits break.Additive edits compound over autoregressive generation and break at the slow extreme on all three systems, while the better rule at moderate targets depends on the model.Source: §I · p. 2
  • WSOLA time-stretching provides a synthetic rate contrast that changes tempo while preserving pitch and timbre, isolating rate without a separate disentanglement step.The method uses the same utterance at slow, neutral, and fast tempos.Source: Figure 1 · p. 1, §I · p. 2
  • Rate is decodable across most decoder layers but causally steerable only in a mid-depth window, so the intervention layer must be chosen with a causal sweep.The calibrated method transfers to the public Seed-TTS-Eval benchmark and is corroborated by a small human AB test.Source: §I · p. 2

II. METHOD

The method combines a calibrated rate axis, causal selection of the intervention layer, and inference-time clamping rather than additive steering.

II. METHOD

  • The recipe probes same-text rate contrasts, locates a causally effective layer with a sweep, and clamps the activation projection onto the calibrated axis at inference.Clamping sets the projection to a requested value rather than adding a fixed offset.Source: §II · p. 2

A. Setting

The setting is autoregressive TTS that generates discrete speech tokens from text and a reference voice; interventions rewrite only the current token’s hidden state.

A. Setting

  • The systems generate discrete speech tokens from a text prompt and reference voice, using acoustic-codec or semantic tokens.A forward hook rewrites the residual-stream output at the last sequence position; weights, prompts, and sampling parameters remain unchanged.Source: A. Setting · p. 2

B. Probing the attribute axis

A same-text slow/neutral/fast corpus provides the contrast for estimating a rate direction, neutral origin, and calibrated command scale.

B. Probing the attribute axis

  • Matched slow, neutral, and fast recordings are teacher-forced through the system, so class-mean activation differences primarily reflect rate rather than lexical content.The pole means define the signed direction, while the neutral mean sets the operating point so k=0 corresponds to neutral speech.Source: B. Probing the attribute axis · p. 2
  • The rate axis is the normalized difference between fast and slow class means, and its origin is the neutral mean.Changing the origin shifts the command’s meaning; a pole midpoint can reproduce the same clamp with an offset correction, while omitting centering creates a model-specific operating-point shift.Source: B. Probing the attribute axis · pp. 2–3
  • The intensity unit is calibrated from the labelled poles, expressing the target in class-gap units and setting the per-step scale used at inference.Held-out probes confirm monotonic ordering at the class-mean level, and the neutral mean is consistently not the pole midpoint.Source: B. Probing the attribute axis · pp. 2–3

C. Steering by clamping, not adding

The method replaces a fixed additive offset with a clamp that sets the activation’s projection on the rate axis to a target. Unlike an offset, the clamp adapts to the current state, preventing pushes from accumulating unchecked.

C. Steering by clamping, not adding

  • Additive steering applies a fixed coefficient at every generation step, while clamping sets the activation’s projection to the target s⋆= k∆.The additive rule is open-loop; clamping makes the correction depend on the current projection.Source: C. Steering by clamping, not adding · p. 3
  • The clamp correction shrinks to zero at the target, reverses after overshoot, and is bounded by the distance to the setpoint.Applying the clamp twice is equivalent to applying it once.Source: C. Steering by clamping, not adding · p. 3
  • The authors relate clamping to directional ablation’s fixed target of 0 and affine concept editing’s reset-and-offset operation, [20].The rate probe supplies graded, calibrated targets; norm rescaling is also available for either steering rule [28], [30].Source: C. Steering by clamping, not adding · p. 3

D. Where to intervene

The intervention site is chosen by a causal sweep rather than probe accuracy alone. The evaluated systems steer near half-depth, and Table I frames a comparison of real rate-binned and synthetic WSOLA rate axes on Qwen3-TTS.

D. Where to intervene

  • A causal sweep tests candidate layers with temporary directions and ±2∆ℓ steering, then selects the layer with the strongest rate span and recomputes the final direction there.Probe accuracy alone does not establish that editing a layer will change generated speech.Source: D. Where to intervene · p. 3

A. Models

  • Table I compares real rate-binned LibriTTS and synthetic WSOLA directions on Qwen3-TTS block 14 using per-layer class gap, naturalness, and speaker similarity.The caption identifies the comparison and metrics but does not provide cell values.Source: Table I · p. 3
  • The selected steering sites are Qwen3-TTS block 14 of 28, MOSS-TTS block 18 of 36, and CosyVoice2 block 14 of its 24-block language model.CosyVoice2 steering edits the LM residual stream, not the flow-matching acoustic model; all sites are near 50% depth.Source: A. Models · p. 3

B. Attribute corpora

The rate direction is learned from same-utterance synthetic tempo contrasts, while evaluation uses both fixed-intensity sweeps and target-based tests. The protocols measure rate movement alongside speech and speaker-quality effects.

B. Attribute corpora

  • The rate axis comes from 500 LibriTTS utterances converted into same-utterance slow, neutral, and fast triplets with pitch-preserving WSOLA time-stretching.The contrast is designed to isolate tempo from pitch, timbre, lexical content, and vocal affect without a separate disentanglement step.Source: B. Attribute corpora · p. 3
  • Compared with the synthetic direction, the real rate-binned axis has a per-layer gap 6× smaller and needs 3–6× the coefficient to reach a target.The instruction-conditioned alternative is often entangled with pitch and affect; the WSOLA axis could encode overlap-add artifacts, though the natural axis moves rate in the same direction.Source: B. Attribute corpora · p. 3

C. Protocol and metrics

  • The fixed-intensity protocol tests ±1, ±2, and ±3, escalating where tolerated, on a 17-word sentence in held-out voices and measures rate, runaway rate, WER, naturalness, and speaker similarity.It uses 50 voices per Qwen3-TTS cell and 20 per MOSS-TTS and CosyVoice2 cell.Source: C. Protocol and metrics · pp. 3–4
  • Figure 2 compares held-out slow-versus-fast probe AUC with the words-per-second span from clamp edits applied at individual blocks.The figure’s span indicates where intervention changes rate, while AUC measures whether rate is readable from activations.Source: Figure 2 · p. 4
  • The target-based protocol calibrates an intensity for ×1.25 speed-up or ×0.75 slow-down, then evaluates each model-rule pair on 20 sentences and five voices.It interpolates the first target crossing in log-ratio space and uses best-effort intensity at a plateau if the target is unreached.Source: C. Protocol and metrics · p. 4

IV. RESULTS

Rate is decodable across the decoder but causally steerable only in a mid-depth window. At the slow extreme, additive steering breaks across all three systems while clamping holds; on Qwen3-TTS, the temporal traces show additive drift or failure versus a sustained clamp rate.

B. The application rule decides at the extremes

  • At k = −2, additive steering breaks on all three systems while clamping the same projection preserves slow-down, intelligibility, naturalness, and speaker identity.The failure signature varies by system; Table II summarizes the slow extreme and its runaway, WER, and speaker-similarity measures.Source: Table II · p. 4, B. The application rule decides at the… · p. 4

A. Decodable everywhere, steerable in a window

  • Probe AUC stays near ceiling across depth on all three systems, but the rate span peaks mid-depth and falls near zero in shallow and late blocks.The gap between readable and writable rate means probe accuracy alone cannot locate an effective intervention site.Source: A. Decodable everywhere, steerable in… · p. 4

B. The application rule decides at the extremes

C. Quality at practical rate targets

At practical rate targets, the better steering rule depends on the model; on the English Seed-TTS-Eval benchmark, calibrated steering transfers to unseen voices and texts. CosyVoice2’s fast clamp remains an exception because it garbles.

C. Quality at practical rate targets

  • At ×1.25 speed-up and ×0.75 slow-down, both rules stay essentially at baseline for Qwen3-TTS and MOSS-TTS, without runaway steering.At these practical targets, Qwen3-TTS’s slow-direction non-termination disappears as an over-steering artifact.Source: C. Quality at practical rate targets · p. 5
  • CosyVoice2 garbles after steering, making additive the cleaner route even for a moderate fast clamp.Against WSOLA, the clamp has higher UTMOS in five of six cells, tying on MOSS-TTS fast, but higher CosyVoice2 WER: 1.7/17.6% vs. 1.1/0.7% (slow/fast).Source: C. Quality at practical rate targets · p. 5

D. Generalization to a public benchmark (Seed-TTS-Eval)

  • The benchmark uses 1088 released Common Voice utterances across 666 prompt voices, with each prompt voice cloning its own prompt and synthesizing its own target text.The study applies per-model intensities calibrated for ×1.25 and ×0.75 targets and uses Seed-TTS-Eval’s own evaluators.Source: D. Generalization to a public… · p. 5
  • ×0.68–0.77 for the requested ×0.75 and ×1.21–1.33 for the requested ×1.25 across a thousand unseen voices and texts, without re-tuning on Seed-TTS-Eval.Qwen3-TTS and MOSS-TTS keep WER close to baseline, while CosyVoice2’s fast clamp garbles and additive steering stays near baseline.Source: D. Generalization to a public… · p. 5

E. Human evaluation: competitive with a time-stretch baseline

The human evaluation compares clamp-steered audio with a matched-rate WSOLA time-stretch of the same unsteered utterance. Listeners slightly preferred the steered audio overall, and the clamp emits audio at the target rate directly rather than applying a post-hoc stretch.

E. Human evaluation: competitive with a time-stretch baseline

  • Listeners preferred clamp-steered audio in 48.2% of comparisons, heard no difference in 11.0%, and preferred time-stretch in 40.8%.The pooled preference was significant (two-sided exact sign test on decisive votes, p=0.027), though neither direction was individually significant.Source: E. Human evaluation: competitive with… · p. 5
  • The forced-choice comparison matched rate and wording by contrasting clamp-steered audio with a WSOLA stretch of the same unsteered utterance.The study collected 800 comparisons from 36 listeners.Source: E. Human evaluation: competitive with… · p. 5
  • Clamp steering emits audio at the target rate through a per-step decoder hook, whereas time-stretching requires a second pass over finished audio.The authors describe direct generation as a natural fit for streaming.Source: E. Human evaluation: competitive with… · p. 5

V. LIMITATIONS

The approach relies on a faithful synthetic transform to isolate speaking rate, and its evaluation leaves several broader settings untested. The conclusion reports stable control across three systems, with rule choice still model-dependent.

V. LIMITATIONS

  • A clean single-attribute rate axis depends on a faithful synthetic time-stretch that changes tempo while preserving pitch and timbre.How far the recipe extends to attributes such as pitch, timbre, or emotion remains open when no attribute-isolating transform is available.Source: §V · p. 6
  • The study evaluates utterance-level control, English only, and relies largely on objective proxies for intelligibility, quality, and speaker identity.It does not show time-varying control within a sentence or a streaming deployment; nonlinear rate geometry, axis stability, and per-utterance rate spread are also untested, and intensity is calibrated per model and rule.Source: §V · p. 6

VI. CONCLUSION

  • Across Qwen3-TTS, MOSS-TTS, and CosyVoice2, clamping holds a wider stable range than additive steering, but additive is cleaner for CosyVoice2’s fast direction.The same calibrated intensities move rate on a thousand unseen Seed-TTS-Eval voices, and listeners slightly prefer clamp output over matched-rate time-stretch.Source: §VI · p. 6

Related papers

Loading 2609.33810v1…