PaperlayerView exact revision on arXiv

Source-linked AI summary

EmphTTS: an emphasis-control TTS with reinforcement learning

Zirui Li, Rech Silas, Lauri Juvela, Tom Backstrom, Mikko Kurimo

arXiv:2609.27599v1eess.AS

TL;DR

Fixed global speaking-rate scaling is coarse for word-level emphasis, whose duration needs vary with the number and locations of emphasized words. EmphTTS optimizes a duration predictor with GRPO and an emphasis-specific reward. It achieves the best objective emphasis controllability and is significantly preferred over synthetic groundtruth and most baseline systems.

  • Problem

    Fixed global speaking-rate scaling is inherently coarse for word-level emphasis because the appropriate scale varies with the number and locations of emphasized words.

    Source: §2.1 · p. 2

  • Method

    EmphTTS post-trains the duration predictor with GRPO and an emphasis-specific reward, optimizing durations for emphasis while keeping the TTS model fixed.

    Source: §1 · p. 1, §2.2 · p. 2, §3 · p. 2

  • Results

    0.75 StressLM F1, versus Qwen3-TTS-VD’s 0.63, was EmphTTS’s best score; it was significantly preferred over synthetic groundtruth and most baseline systems.

    Source: §5 · p. 4, §6 · p. 4

  • Takeaways & Limitations

    The ablations support the importance of input-dependent duration optimization, as GRPO’s gains cannot be explained by simple global speaking-rate reduction.

    Source: §6 · p. 4

Abstract

from arXiv · show

Generating controllable and human-like emphasis remains an open challenge in text-to-speech, even when explicit emphasis control signals are provided in the text input, limiting the communicative accuracy of synthetic speech in real-world applications. Reinforcement learning has recently shown promise for post-training TTS systems to align with human preference, yet existing methods have not been applied to word-level prosodic control. We present EmphTTS, a non-autoregressive TTS system that applies Group Relative Policy Optimization (GRPO) to the duration predictor with an emphasis localization reward, enabling direct optimization for word-level emphasis. Evaluations show that EmphTTS achieves the best emphasis controllability and performs the best in emphasis objective evaluation. In subjective preference tests, EmphTTS is significantly preferred over synthetic groundtruth and most baselines. Ablation studies show that GRPO improves emphasis realization beyond supervised-finetuning-based duration modeling and simple speaking-rate adjustment, while alleviating the mismatch between the independently trained duration predictor and TTS model.

1. INTRODUCTION

Word-level emphasis carries meaning in speech, yet current TTS systems can fail to realize it even when it is explicitly marked. EmphTTS applies GRPO to a duration predictor to improve emphasis control.

1. INTRODUCTION

  • Word-level emphasis can distinguish meanings and support dialogue repair, cross-cultural communication, and language learning.It conveys contrast through amplitude, duration, and fundamental-frequency differences.Source: §1 · p. 1
  • Recent TTS systems remain limited in producing correct emphasis despite gains in intelligibility, human-likeness, and broad style control.The limitation applies whether emphasis is explicitly specified or inferred from context.Source: §1 · p. 1
  • Prior TTS reinforcement-learning work has not targeted word-level prosody, and GRPO is less direct for non-autoregressive diffusion TTS.The sampling process is not formulated as a discrete stochastic policy, motivating use of the probabilistic duration predictor as the policy.Source: §1 · p. 1
  • EmphTTS post-trains its duration predictor with GRPO while keeping the TTS model fixed, improving objective emphasis control beyond SFT and earning strong preference over most compared systems.The approach is intended to optimize duration decisions directly for emphasis realization.Source: §1 · p. 1

2. RELATED WORK

Non-autoregressive TTS uses duration estimates or predictors to determine speech length, while global speed scaling offers only coarse control of emphasis-related lengthening. Earlier TTS reinforcement-learning work focuses on preference alignment or intelligibility; EmphTTS adds an emphasis-specific reward.

2.1. Duration Modeling in TTS

  • A duration predictor in non-autoregressive TTS estimates generated speech length from text, unlike autoregressive TTS, which models duration implicitly through EOS prediction.Earlier non-autoregressive systems used phoneme-level duration modeling or a speech-length predictor; modern systems may estimate duration from prompt and target lengths.Source: §2.1 · p. 1
  • Global speed scaling is coarse for word-level emphasis because needed duration varies with the number and locations of emphasized words.Emphasis markers add only a few tokens, so the unscaled estimated duration can understate emphasis-related lengthening.Source: §2.1 · p. 2

2.2. Reinforcement Learning in TTS

  • TTS reinforcement learning includes preference-based methods that require preference datasets, as well as automatic-reward methods for speech objectives.The cited preference-based approaches align speech with human preferences, while diffusion loss-guided policy optimization targets naturalness.Source: §2.2 · p. 2
  • Prior GRPO work for TTS optimized intelligibility and speaker-level metrics, whereas EmphTTS introduces an emphasis-specific reward for fine-grained prosodic control.Transcription-centric rewards can collapse prosodic variation into monotone output; adding speaker-similarity rewards was reported to destabilize training.Source: §2.2 · p. 2

3. METHODOLOGY

EmphTTS combines an emphasis-fine-tuned TTS model with a duration predictor optimized by GRPO. It samples candidate durations, evaluates synthesized speech using intelligibility, speaker-similarity, and emphasis-localization measures, and compares several model variants.

3. METHODOLOGY

  • The system uses F5-TTS as its backbone and a DiTTo-TTS-style encoder-decoder duration predictor that estimates remaining speech length as a categorical distribution.The predictor receives text and prompt speech; at inference, target duration is sampled from its distribution at the end of the prompt.Source: §3 · p. 2
  • After separate SFT of the TTS model and duration predictor, GRPO optimizes the predictor while the fine-tuned TTS model remains frozen.The independent SFT models are directly combined before GRPO to address their mismatch in emphasis generation.Source: §3 · p. 2
  • For each input, the method samples candidate durations, synthesizes speech with the frozen TTS model, and evaluates WER, speaker similarity, and emphasis-localization balanced accuracy.The candidate rewards are normalized within each group before policy optimization.Source: §3 · p. 2
  • The GRPO objective uses a clipped surrogate and a reference predictor initialized from the SFT duration model.The method defines a policy-to-reference probability ratio for sampled durations.Source: §3 · pp. 2–3
  • The evaluation includes base systems, SFT ablations, a global-speed variant, and EmphTTS combining the fine-tuned TTS model with the GRPO-optimized predictor.The speed-factor variant tests whether slower speaking alone improves emphasis.Source: §3 · p. 3

4. EXPERIMENTS

The experiments compare EmphTTS with four open-source systems and internal ablations, using separate evaluations for general TTS capability, emphasis controllability, and listener judgments. The training and evaluation protocols specify their data sources, metrics, and sampling procedures.

4.1. Baselines

  • Four open-source baselines support prosody control through markup or voice-design instructions, with system-specific emphasis formatting selected for evaluation.The baselines are CosyVoice3, FishAudio-S2, Qwen3-TTS-VD, and VoxCPM2.Source: §4.1 · p. 3

4.2. Pre- & Post-training

  • Pre- and post-training use LibriTTS-R, Emilia, Expresso, and VCTK data, with emphasis labels represented by asterisks during SFT.The GRPO stage optimizes the SFT duration predictor while the SFT TTS model is inferred; its reward uses WER, speaker similarity, and balanced accuracy.Source: §4.2 · p. 3

4.3. Objective Evaluation

  • General TTS capability is evaluated on 1,088 Seed-TTS Eval English prompt-target pairs, while emphasis controllability is tested on 903 TinyStress-15k utterances.The TinyStress evaluation includes word-level emphasis F1 from StressLM alongside intelligibility and speaker-similarity measures.Source: §4.3 · p. 3

4.4. Subjective Evaluation

  • Subjective evaluation uses CMOS ratings against groundtruth and one-versus-rest EmphPref comparisons, retaining matched-WER utterances for emphasis judgments.Participants choose the sample with better emphasis or no preference; 25 valid submissions were collected per evaluation, and Table 1 reports objective and subjective results.Source: §4.4 · p. 3, Table 1

Baselines

The baseline comparison reports WER and speaker similarity for Base MTTS and MTTS-DP.

  • MTTS-DP has WER 6.48% and SIM 0.34, versus Base MTTS at WER 5.91% and SIM 0.33.Source: Baselines · p. 4

5. RESULTS

EmphTTS leads the reported objective emphasis-controllability comparison, while subjective preferences favor it over groundtruth and most baselines. Its intelligibility and speaker-similarity results vary across the reported datasets.

  • On Seed-TTS EN, baselines achieve WERs of 0.99%–1.84%; VoxCPM2 has the highest SIM at 0.753, and Qwen3-TTS-VD the best CMOS at 0.06 ± 0.17.The baselines substantially outperform the authors' smaller models in intelligibility and speaker similarity.Source: §5 · p. 4
  • On Seed-TTS EN, EmphTTS has WER 3.30%, versus 4.82% for MTTS-DP-SFT; its CMOS is −0.95 ± 0.21, alongside MTTS-DP-SFT's −0.94 ± 0.20.The authors report a 31% relative WER improvement over MTTS-DP-SFT and the best CMOS among their variants for both systems.Source: §5 · p. 4
  • On TinyStress-15k, EmphTTS has WER 3.25%, improving on MTTS-SFT (3.31%) and MTTS-DP-SFT (4.52%), but trailing the s = 0.9 variant (3.22%).Across the authors' models, WERs range from 3.22% to 4.52% and SIMs from 0.42 to 0.44; FishAudio-S2 has the lowest WER (0.25%) and VoxCPM2 the highest SIM (0.73).Source: §5 · p. 4
  • StressLM F1 is 0.75 for EmphTTS, above the strongest zero-shot baseline Qwen3-TTS-VD at 0.63 and the s = 0.9 variant at 0.71.The reported gain is 0.12 absolute F1 over the strongest baseline; the authors say it cannot be explained by global speaking-rate reduction alone.Source: §5 · p. 4
  • EmphTTS is significantly preferred over groundtruth (53.7% vs. 36.0%) and CosyVoice3 (56.0%), FishAudio-S2 (56.5%), and VoxCPM2 (54.2%).Its comparison with Qwen3-TTS-VD is near parity (45.7% vs. 44.8%) despite EmphTTS's substantially higher objective emphasis score.Source: §5 · p. 4

6. CONCLUSION

EmphTTS applies GRPO with an emphasis reward to duration prediction for word-level emphasis in non-autoregressive TTS. It achieves the best objective emphasis controllability and is significantly preferred over synthetic groundtruth and most baselines in subjective evaluation.

  • EmphTTS achieves the best objective emphasis controllability and is significantly preferred over synthetic groundtruth samples and most baseline systems.Source: §6 · p. 4
  • The system optimizes its duration predictor with GRPO using a balanced-accuracy emphasis reward to address mismatch with the TTS model.Source: §6 · p. 4
  • Ablations show that GRPO's gains cannot be explained by simple global speaking-rate reduction and support input-dependent duration optimization.Source: §6 · p. 4
  • Subjective emphasis preference can be influenced by overall speech quality, motivating joint use of objective and subjective measures.Source: §6 · p. 4
Loading 2609.27599v1…