PaperlayerView exact revision on arXiv

Source-linked AI summary

Natural language guidance of high-fidelity text-to-speech with synthetic annotations

Dan Lyth, Simon King

arXiv:2402.01912v1cs.SDcs.CLeess.AS

TL;DR

Reference-audio conditioning limits natural-language-free control of speaker identity and style, while human-labeled speech descriptions are difficult to scale. The paper automatically labels a 45k-hour dataset and trains a speech language model; it reports high-fidelity, natural speech across varied attributes and recording conditions, with a narrow English audiobook-reading scope.

  • Problem

    Reference recordings limit speaker and style control, while speech-description datasets depend on human annotation that is challenging to scale.

    Source: §1 · p. 1

  • Method

    The authors automatically label a 45k-hour dataset across speech attributes and recording conditions, then train a speech language model conditioned on transcripts and natural-language descriptions.

    Source: §1 · p. 1, §3.3 · p. 3

  • Results

    The model controls a wide range of speech and channel attributes with high audio fidelity and naturalness, and its objective quality values exceed Audiobox’s and are often close to ground truth.

    Source: §4.1 · p. 4, §5 · p. 5

  • Takeaways & Limitations

    A single model can generate high-fidelity speech under natural-language control, including with amateur recordings and a comparatively small amount of clean data.

    Source: §5 · p. 5

    The work demonstrates efficacy only in the relatively narrow domain of English audiobook reading.

    Source: §5 · p. 5

Abstract

from arXiv · show

Text-to-speech models trained on large-scale datasets have demonstrated impressive in-context learning capabilities and naturalness. However, control of speaker identity and style in these models typically requires conditioning on reference speech recordings, limiting creative applications. Alternatively, natural language prompting of speaker identity and style has demonstrated promising results and provides an intuitive method of control. However, reliance on human-labeled descriptions prevents scaling to large datasets. Our work bridges the gap between these two approaches. We propose a scalable method for labeling various aspects of speaker identity, style, and recording conditions. We then apply this method to a 45k hour dataset, which we use to train a speech language model. Furthermore, we propose simple methods for increasing audio fidelity, significantly outperforming recent work despite relying entirely on found data. Our results demonstrate high-fidelity speech generation in a diverse range of accents, prosodic styles, channel conditions, and acoustic conditions, all accomplished with a single model and intuitive natural language conditioning. Audio samples can be heard at https://text-description-to-speech.com/.

1. Introduction

Large-scale speech models can adapt speaker and style from reference audio, while natural-language control is intuitive but difficult to scale because speech descriptions require costly annotation. This work uses automatic labeling and a 45k-hour dataset to train a model for natural-language control of speech attributes and recording conditions.

1. Introduction

  • Reference-speech conditioning limits practical use of large-scale TTS and can enable reproducing a speaker’s likeness from little enrollment data.The paper notes that this can be beneficial in some use cases but also has potential for harm.Source: §1 · p. 1
  • Natural-language descriptions offer intuitive control, but creating speech metadata from scratch and relying on human annotation impede scaling to large datasets.The paper contrasts this bottleneck with MLS, which contains over 10 million utterances across 50k hours.Source: §1 · p. 1
  • The authors automatically label a 45k-hour dataset across gender, accent, speaking rate, pitch, and recording conditions, then train a speech language model to control these attributes independently.They report combinations of speaker identity and style unseen in training data.Source: §1 · p. 1
  • With 1% high-fidelity audio in training and state-of-the-art audio codecs, the authors report extremely high-fidelity speech generation.The approach uses automatic labeling and found data rather than relying on human descriptions at scale.Source: §1 · p. 1

2. Related Work

Prior work controls speaker identity and style through reference recordings, latent-variable methods, or natural-language descriptions, but each approach has constraints. Missing descriptive speech metadata and human-annotation costs motivate automatic, large-scale labeling; concurrent systems broaden the comparison.

2.1. Control of speaker identity and style

  • Reference embeddings and global style tokens can control non-lexical speech variation, but tie users to existing recordings; latent-variable methods instead require disentanglement and analysis.The cited approaches include exemplar-derived embeddings [16] [17] and Gaussian-mixture or variational-autoencoder latent spaces [18].Source: §2.1 · p. 1
  • Natural-language control avoids those constraints, but speech lacks the descriptive metadata needed to support it at scale.The passage identifies missing natural-language descriptions as a key challenge for speech.Source: §2.1 · pp. 1–2
  • Earlier work combined statistical speech measures with synthetic emotion labels to create five axes of variation, each expressed through three-level keywords [11].Other studies used human-written descriptions on 44-hour and 6-hour datasets, while retaining fixed speaker IDs [12] [13].Source: §2.1 · p. 2
  • A later human-annotation approach addressed speaker identity but covered no accent or channel conditions and labeled only 16% of speakers in a much smaller dataset [14].The authors identify the scaling difficulty of human annotation as motivation for their work.Source: §2.1 · p. 2
  • The authors report outperforming Audiobox on naturalness, audio fidelity, and shared description attributes; PromptTTS2 [19] scales data but controls only four variables.The comparison is reported for concurrent work.Source: §2.1 · p. 2

3. Method

The method combines metadata from large English audiobook corpora, automatic attribute labeling, natural-language description generation, and a speech language model conditioned on transcripts and descriptions. It pairs broad but noisy training data with high-fidelity audio and labels for acoustic conditions.

3.1. Metadata collection

  • The study combines 45k hours of English MLS audio with 585 hours of enhanced LibriTTS-R, using transcripts and predictive-model gender labels supplied with both datasets.LibriTTS-R is included for its improved audio quality despite its smaller scale.Source: §3.1 · p. 2

3.1.1. Accent

  • An accent classifier labels the datasets; trained on resources covering 53 accents, it reaches 86% accuracy on a held-out test set before spot-checking.The classifier uses embeddings from a language-identification model and a linear classification layer.Source: §3.1.1 · p. 2

3.1.2. Recording quality

  • Recording quality is represented by estimated SNR and C50; adding enhanced LibriTTS-R motivates the hypothesis that the model can learn fidelity across accents and styles.The authors hypothesize this could support clean recordings for accents and styles represented only by low-fidelity training utterances; C50 indicates reverberation.Source: §3.1.2 · pp. 2–3

3.1.3. Pitch and speaking rate

  • Speaker-level mean pitch is labeled relative to gender, utterance-level pitch variation proxies monotony or animation, and speaking rate is phoneme count divided by utterance duration.Leading and trailing silences are removed before calculating speaking rate.Source: §3.1.3 · p. 3

3.2. Metadata preparation

  • Continuous pitch, rate, SNR, and C50 values are mapped into seven bins with descriptive keywords, while gender and accent labels are used directly.The SNR figure shows the MLS estimates and bin boundaries; a language model turns keyword sets into full descriptions to support varied user phrasing.Source: §3.2 · p. 3, Figure 2 · p. 3

3.3. Model

  • The model prepends transcripts and applies description conditioning through cross-attention, relying on text rather than audio or speaker embeddings; DAC supplies discrete audio tokens.The selected 44.1 kHz DAC representation runs at 86 Hz with nine codebooks, handled using a delay pattern.Source: §3.3 · p. 3

4. Experiments

Objective and listening tests assess description control and audio quality against Audiobox and ground truth. The model performs well on most tested attributes and outperforms Audiobox in reported fidelity and naturalness comparisons, with C50 control remaining weak.

4.1. Objective evaluation

  • Objective control tests use balanced, manually written descriptions with MLS and LibriTTS-R sentences, except accent tests, which use three named passages.The continuous-attribute results are correlations between description labels and synthesized labels, reported with 95% confidence intervals.Source: §4.1 · p. 3, Figure 3 · p. 4
  • Gender classification accuracy is 94% and accent accuracy is 68%; continuous attributes generally match descriptions except C50.The authors hypothesize that noisy labels and an imbalanced accent distribution contribute to the weaker accent accuracy; they report poor C50 control without identifying its cause.Source: §4.1 · p. 4, Figure 3 · p. 4
  • The model produces SQUIM values significantly higher than Audiobox and often close to ground truth, despite using only about 500 high-fidelity hours in the 45k-hour dataset.The evaluation uses reference-less estimates of PESQ, STOI, and SD-SRD.Source: Table 1 · p. 4, §4.1 · p. 4

4.2. Subjective evaluation

  • Listeners rated the model above Audiobox for relevance and naturalness; relevance also exceeded ground truth, which the authors link partly to label noise and audio artifacts.The authors note that Audiobox can control attributes such as age that their model cannot.Source: Table 2 · p. 4, §4.2 · p. 4
  • Two tests used 30 listeners: one rated description relevance, while the 1–5 naturalness-and-fidelity MOS test deliberately matched Audiobox’s conflated evaluation.The authors acknowledge that the MOS design combines judgments of speech naturalness and audio fidelity.Source: §4.2 · p. 4

5. Conclusion

The paper presents natural-language-guided, high-fidelity text-to-speech and reports control over a wide range of speech and channel attributes alongside naturalness. Its demonstrated efficacy is limited to English audiobook reading.

5. Conclusion

  • The method generates high-fidelity text-to-speech guided by natural-language descriptions.The authors describe it as highly effective.Source: §5 · p. 5
  • To the authors’ knowledge, the method combines control of a wide range of speech and channel attributes with high audio fidelity and overall naturalness.They report achieving this with amateur recordings and comparatively little clean data.Source: §5 · p. 5
  • The demonstrated efficacy is limited to audiobook reading in English.The authors plan to extend the work to more languages, speaking styles, vocal effort, and channel conditions.Source: §5 · p. 5
Loading 2402.01912v1…