Source-linked AI summary
E3-System: Expressive EmoData, Explicitly Controllable, Easy-to-Use TTS
Yiqiao Huang, Xiaolong Lin, Zhenpeng Zhan, Zhizheng Wu
TL;DR
Zero-shot TTS still struggles to connect natural-language style intent with faithful acoustic expression while preserving speaker identity. E3-System combines expressive data, intent-to-reference retrieval, and disentangled synthesis; it reports strong quality, speaker preservation, and expressive controllability.
Problem
Zero-shot TTS faces a gap between user style intent and acoustic realization, while text prompts lack acoustic detail and speech prompts entangle style with speaker identity.
Source: §1 · p. 1Method
E3-System combines EmoData, Style-RAG retrieval of acoustic style references from natural-language descriptions, and Dual-TTS with VC-augmented timbre–style disentanglement.
Source: §1 · p. 1Results
E3-System performs on par with strong state-of-the-art baselines in intelligibility and naturalness, slightly outperforming them in speaker similarity and outperforming them in speaker and style similarity.
Source: §3.3 · p. 4Takeaways & Limitations
The framework links natural-language style intent to acoustic references and transfers the retrieved style to arbitrary target speakers while preserving target timbre.
Source: §1 · p. 1
Abstract
from arXiv · showhide
Recent zero-shot Text-to-Speech (TTS) systems can clone unseen speakers, yet controllable expressiveness remains limited by a gap between semantic intent and acoustic realization. Text descriptions offer a natural control interface but provide weak acoustic grounding, whereas speech style prompts provide concrete prosody but entangle style with the prompt speaker's timbre. To bridge this gap, we propose E3-System, a unified intent-to-speech framework for expressive zero-shot TTS that couples three requirements: Expressive style modeling, Explicit timbre-style control, and Easy-to-use natural-language interaction. Given a target speaker and either a natural-language style description or a speech style prompt, E3-System generates expressive speech while preserving the target timbre. The key idea is to close the full control loop: building a large expressive style space, grounding user intent into usable acoustic style references, and synthesizing speech with disentangled timbre and style. To realize this loop, EmoData provides a thousand-hour multilingual emotional speech dataset, Style-RAG grounds text descriptions into high-quality acoustic style prompts from these data, and Dual-TTS transfers the grounded style to arbitrary target speakers through VC-augmented timbre-style disentanglement. Experiments show that E3-System achieves strong speech quality, speaker preservation, and explicit expressive controllability.
1. Introduction
E3-System unifies expressive style modeling, natural-language intent grounding, and disentangled synthesis to connect user intent with acoustic realization while preserving speaker identity. Its components address gaps in expressive data, flexible timbre–style control, and usable style prompting.
1. Introduction
- Large-scale TTS corpora are mostly neutral [7] [9], while emotional datasets are smaller [11] and newer resources remain limited in scale and multilingual coverage [14].The paper also identifies semantic–emotion mismatches in some emotional speech data.Source: §1 · p. 1
- Text descriptions are easy to use but lack acoustic grounding and fine prosodic detail, while speech prompts provide detail but require manual selection.Both approaches can yield unstable or costly control for non-expert users.Source: §1 · p. 1
- Style-RAG maps natural-language intents to high-quality style references, linking abstract intent with fine-grained acoustic control.The system is designed to minimize leakage while supporting explicit timbre–style control.Source: §1 · p. 1
- E3-System closes the control loop by combining an expressive style space, intent-based acoustic grounding, and timbre–style disentangled synthesis.EmoData supplies multilingual expressive speech, and Dual-TTS uses VC-augmented disentanglement for zero-shot timbre–style synthesis.Source: §1 · pp. 1–2
2. System Architecture
E3-System combines two-reference speech synthesis, expressive multilingual data, and retrieval-based style grounding. Dual-TTS separates target timbre from expressive style, EmoData supplies the style space, and Style-RAG retrieves acoustic references from it.
2. System Architecture
- E3-System uses a speaker prompt for target timbre and a style prompt for expressive control, then generates semantic tokens followed by acoustic speech.This dual-reference synthesis engine is the core of the system architecture.Source: §2.1 · p. 2, §2 · p. 2
2.1.1. Text-to-Semantic Generation
- The Text-to-Semantic stage encodes the speaker prompt globally and injects the style vector into reference semantic tokens; Dual-AR combines long-range context with lightweight refinement.The stage uses a 12-layer Contextual-AR and a 12-layer Fast-AR with 2× downsampling.Source: §2.1.1 · p. 2
2.1.2. Semantic-to-Acoustic Generation
- The Semantic-to-Acoustic stage predicts masked acoustic tokens from semantic and unmasked acoustic tokens, then refines generation from coarse patterns to local detail.Models are pretrained on compliant, enterprise-licensed speech data and fine-tuned on EmoData.Source: §2.1.2 · p. 2
2.1.3. Disentangle Training Paradigm
- VC-generated pseudo-targets address scarce paired ground truth, while different speaker, utterance, and style references are used to prevent timbre, content, and style leakage.Table 1 compares retrieved real speech with constructed pseudo-targets; the construction substantially improves SIM-Style while enabling arbitrary timbre–style combinations.Source: §2.1.3 · p. 2, Table 1 · p. 2
2.2. EmoData: Expressive Large-Scale Benchmark
- EmoData is a thousand-hour multilingual emotional-speech dataset with six emotions, supporting category-level and intensity/prosody-level control.Its automated pipeline filters for confident acoustic emotion and alignment between linguistic sentiment and acoustic prosody.Source: §2.2 · p. 2
2.3. Style-RAG: Intent-Based Acoustic Grounding
- Style-RAG retrieves validated style speech from a caption–waveform database using contrastive cross-modal similarity, grounding natural-language intent in acoustic references.The retrieved references can be used at inference without extra user effort.Source: §2.3 · p. 3
3. Experiments
E3-System outperforms state-of-the-art baselines in both speaker and style similarity for dual-prompt synthesis, while EmoData shows comparable perceptual quality to studio datasets. The experiments also assess the evaluation metrics, dataset utility, and single-prompt synthesis.
3.1. Evaluation Metrics
- The evaluation measures intelligibility, speaker and style similarity, naturalness, style-semantics alignment, and perceptual speaker and style matching.The objective metrics are WER, SIM-Spk, and SIM-Style; subjective metrics include SN-MOS, SS-MOS, and SM-MOS.Source: §3.1 · p. 3
3.2. Dataset Evaluation
- EmoData achieves perceptual quality comparable to studio datasets, and Table 3 reports high style-matching rates across emotion categories.The A/B preference test used 180 clips and 15 native speakers.Source: §3.2 · p. 3, Figure 4 · p. 4, Table 3 · p. 4
- Fine-tuning on raw data degrades intelligibility, whereas EmoData improves style similarity while preserving high intelligibility and timbre consistency.Table 4 presents the fine-tuning results.Source: §3.2 · p. 4, Table 4 · p. 4
3.3. Style-Timbre Disentangled Synthesis
- In single-prompt synthesis, E3-System performs on par with strong state-of-the-art baselines in intelligibility and naturalness and slightly outperforms them in speaker similarity.A downsampling factor of 2 yields a 1.8x speedup and slight quality gains over the single-AR baseline; larger factors degrade performance.Source: §3.3 · p. 4, Table 5 · p. 4
- E3-System outperforms state-of-the-art baselines in both speaker and style similarity for dual-prompt disentangled synthesis.The comparison evaluates arbitrary combinations of distinct speaker and style prompts.Source: §3.3 · p. 4, Table 6 · p. 4
- Replacing the disentangled strategy with standard adversarial training sharply reduces style similarity, while Style-RAG achieves comparable style similarity and speech quality to direct speech prompts.Direct speech prompts yield higher stylistic accuracy than RAG.Source: §3.3 · p. 4