Source-linked AI summary
EXPRESSO: A Benchmark and Analysis of Discrete Expressive Speech Resynthesis
Tu Anh Nguyen, Wei-Ning Hsu, Antony D'Avirro, Bowen Shi, Itai Gat, Maryam Fazel-Zarani, Tal Remez, Jade Copet, Gabriel Synnaeve, Michael Hassid, Felix Kreuk, Yossi Adi, Emmanuel Dupoux
TL;DR
Text-based speech synthesis and read-speech datasets leave expressive dimensions and spontaneous vocalizations poorly represented. The paper introduces EXPRESSO and evaluates discrete-unit expressive resynthesis. Its conclusion is that Encodec systems generally perform better for resynthesis, while HuBERT units offer greater voice and style controllability.
Problem
Text omits expressive dimensions and nonverbal vocalizations, and existing expressive datasets often rely on acted reading rather than spontaneous speech.
Source: §1 · p. 1Method
Results
Encodec systems generally perform better for resynthesis, while HuBERT units provide voice and style controllability through their disentanglement from speaker identity and partial disentanglement from expressivity.
Source: §6 · p. 4Takeaways & Limitations
Further work is needed to improve HuBERT-based expressive resynthesis while retaining its controllability.
Source: §6 · p. 4Style rendering differs across datasets for identical labels, limiting straightforward cross-dataset style comparisons.
Source: §5 · p. 4
Abstract
from arXiv · showhide
Recent work has shown that it is possible to resynthesize high-quality speech based, not on text, but on low bitrate discrete units that have been learned in a self-supervised fashion and can therefore capture expressive aspects of speech that are hard to transcribe (prosody, voice styles, non-verbal vocalization). The adoption of these methods is still limited by the fact that most speech synthesis datasets are read, severely limiting spontaneity and expressivity. Here, we introduce Expresso, a high-quality expressive speech dataset for textless speech synthesis that includes both read speech and improvised dialogues rendered in 26 spontaneous expressive styles. We illustrate the challenges and potentials of this dataset with an expressive resynthesis benchmark where the task is to encode the input in low-bitrate units and resynthesize it in a target voice while preserving content and style. We evaluate resynthesis quality with automatic metrics for different self-supervised discrete encoders, and explore tradeoffs between quality, bitrate and invariance to speaker and style. All the dataset, evaluation metrics and baseline models are open source
1. Introduction and related work
Text-to-speech systems rely on read speech and text, which omit many expressive dimensions. The paper introduces a dataset with improvised dialogue and benchmarks resynthesis using discrete speech units.
1. Introduction and related work
- Text omits expressive dimensions such as rhythm, intonation, emotion, emphasis, and nonverbal vocalizations, while reliance on read speech limits expressivity.The paper motivates textless approaches as a way to represent expressive speech that is difficult to transcribe.Source: §1 · p. 1
- Improvised dialogues prompted by situations and characters yield casual speech with spontaneous hesitations and laughter that SSL units can in principle capture.The dataset also reproduces expressive reading, where actors render requested expressions.Source: §1 · p. 1
- The benchmark encodes input audio into low-bitrate discrete units and resynthesizes the same content in a target voice while preserving expressive style.It evaluates content and style preservation across HuBERT units and Encodec, a high-bitrate baseline.Source: §1 · p. 1
2. The EXPRESSO dataset
EXPRESSO combines expressive reading, improvised dialogue, and singing recorded from four North American English speakers. Its materials and preparation support diverse expressive speech, with a boundary on sharing some song recordings.
2. The EXPRESSO dataset
- EXPRESSO contains 47 hours of expressive speech from four North American English speakers, spanning read speech, improvised dialogue, and singing.The dataset describes 26 styles selected for recognizability, application relevance, and range of vocalizations.Source: §2 · p. 1
2.1. Expressive reading
- Expressive reading applies seven styles to shared prompts, while unique scripts broaden linguistic diversity and include contrastive-emphasis and long-form material.The corpus contains roughly 21,000 words across 2,400 unique lines; long-form material includes a news article and narrative per speaker.Source: §2.1 · p. 2
2.2. Improvised dialogs section
- Improvised-dialogue prompts target styles or emotions through situations, and actors recorded in separate booths watched each other by video conference.A small number of interrupted dialogues were edited to remove the pause and maintain a more natural flow.Source: §2.2 · p. 2
2.3. Singing section
- Some recorded songs could not be shared because they were not in the public domain.The original singing recordings totaled 93 minutes.Source: §2.3 · p. 2
2.4. Data preparation
- For speech synthesis, long files are cut into 15-second segments, and train/dev/test splits provide roughly 60 seconds per speaker-style in dev and test.Each subset contains 1.5 hours of speech.Source: §2.4 · p. 2
3. Method
The method evaluates discrete units and resynthesized speech with measures of phonetic content, transcription fidelity, pitch, and expressivity. It tests content preservation across datasets and input-domain conditions.
3. Method
- The method evaluates unit bitrate, ABX discrimination, and phone-normalized mutual information, alongside resynthesis measures for content, pitch, and expressivity.ABX measures discrimination of minimally different triphones; PNMI measures mutual information between units and ground-truth phonemes normalized by phoneme entropy.Source: §3 · pp. 2–3
- Content preservation is measured by ASR word error rate on in-domain and out-of-domain inputs, with results reported across Expresso, VCTK, and Fisher.The evaluation distinguishes input and target speaker conditions; the table reports WER percentages and compares speaker-only with speaker-and-expression conditioning.Source: §3 · p. 3, Table 3 · p. 3
4. Models
The models compare HuBERT-based units quantized with k-means against Encodec units. HiFiGAN synthesizes HuBERT units with speaker or speaker-and-expression conditioning, while Encodec uses its own decoder.
4.1. Unit encoding
- Unit encoding compares HuBERT-LS960 and HuBERT-Mix1 with Encodec models using one or eight codebooks.HuBERT features are quantized with k-means trained on subsets of pretraining data or on EXPRESSO; Encodec codebooks have cardinality 1024.Source: §4.1 · p. 3
4.2. Vocoder
- HuBERT units are synthesized with HiFiGAN trained on LJ and VCTK, with or without EXPRESSO, and conditioned on speaker or speaker plus expression.Encodec units are decoded directly using the Encodec decoder.Source: §4.2 · p. 3
5. Results
The results compare discrete units on phonetic quality, content preservation, style, and pitch. HuBERT and Encodec show different tradeoffs, while expression conditioning and training-domain differences affect resynthesis outcomes.
5. Results
- HuBERT encoders trained on Mix1 generally outperform LS960 on phonetic metrics, while Encodec units perform poorly; the clustering-data advantage varies by metric.Clustering on EXPRESSO improves ABX-centroid and PNMI, but not ABX 1-hot, so the authors call for further study.Source: §5 · p. 3
- Swapping speakers costs 3% relative WER, while HuBERT-Mix1 improves over LS960 by about 10% in-domain and 19% out-of-domain.Vocoder training voices and conditioning did not show systematic results.Source: §5 · p. 3
- Style and pitch results are uniformly better with speaker-plus-expression conditioning than with speaker-only conditioning.Without expression conditioning, style classification still exceeds the 3.8% chance level, suggesting the units partly transmit style.Source: Table 4 · p. 4, Table 5 · p. 4, §5 · p. 4
- Encodec-8 style scores are on par with or better than the best HuBERT models, and its pitch preservation is six-fold better.Swapped speakers reduce style accuracy by around 10% relative and increase pitch error by 20% relative versus same-speaker results.Source: Table 4 · p. 4, Table 5 · p. 4, §5 · p. 4
- Encodec-8 yields twice the errors as the best HuBERT results on LS, but a much smaller drop on Fisher; Encodec-1 underperforms HuBERT-based resynthesis.The authors attribute Encodec-1’s weaker result to one 1024-entry codebook not fully capturing linguistic content.Source: §5 · pp. 3–4
- With VCTK and LJ voices trained only on read speech, style scores halve but remain well above chance, showing some zero-shot generalization.These scores also exceed those of EXPRESSO voices without expression conditioning.Source: §5 · p. 4
- On EmoV, style-classifier performance is lower than in-domain, and label mismatches suggest style rendering differs across datasets.For example, the classifier labeled original EmoV angry speech as “projected,” but resynthesized speech with angry conditioning as “angry.”Source: §5 · p. 4
6. Conclusion
The conclusion presents Expresso as a dataset for expressive discrete resynthesis and identifies a quality–controllability tradeoff between Encodec and HuBERT. It calls for better HuBERT resynthesis while retaining controllability.
6. Conclusion
- The paper presents Expresso and analyzes content and style preservation across several baseline discrete SSL models.Source: §6 · p. 4
- Encodec systems are generally better for resynthesis, while HuBERT units offer output-voice and style controllability through speaker and partial expressivity disentanglement.Source: §6 · p. 4
- Further work should improve HuBERT-based resynthesis quality while retaining controllability; discrete-pitch vocoder conditioning is suggested for pitch preservation.Source: §6 · p. 4