Source-linked AI summary
Text-Free Prosody-Aware Generative Spoken Language Modeling
Eugene Kharitonov, Ann Lee, Adam Polyak, Yossi Adi, Jade Copet, Kushal Lakhotia, Tu-Anh Nguyen, Morgane Rivière, Abdelrahman Mohamed, Emmanuel Dupoux, Wei-Ning Hsu
TL;DR
Speech pre-training has mainly targeted discriminative tasks, while text-free generative modeling loses prosodic information needed for expressive, coherent speech. The paper introduces pGSLM, combining self-supervised units, explicit prosody streams, a multi-stream transformer, and adapted HiFi-GAN. Experiments show improved content and prosody modeling and prompted speech generation coherent with both content and prosody.
Problem
Speech pre-training has mainly been evaluated on classification, while text-free GSLM discards most prosodic information and therefore does not generate expressive, prosodically coherent speech.
Method
pGSLM jointly models self-supervised phonetic units with duration and normalized pitch using a multi-stream transformer, then converts outputs to waveforms with adapted HiFi-GAN.
Results
pGSLM improves phonetic content modeling and generates prompted speech coherent with both content and prosody.
Takeaways & Limitations
Explicit prosody modeling enables text-free speech generation that is aware of prompt content and prosody, while supporting natural, meaningful, and coherent output.
Takeaways & Limitations
Continuous prosody modeling can underfit multimodal distributions, producing near-zero pitch predictions and dull generated speech.
Abstract
from arXiv · showhide
Speech pre-training has primarily demonstrated efficacy on classification tasks, while its capability of generating novel speech, similar to how GPT-2 can generate coherent paragraphs, has barely been explored. Generative Spoken Language Modeling (GSLM) \cite{Lakhotia2021} is the only prior work addressing the generative aspects of speech pre-training, which replaces text with discovered phone-like units for language modeling and shows the ability to generate meaningful novel sentences. Unfortunately, despite eliminating the need of text, the units used in GSLM discard most of the prosodic information. Hence, GSLM fails to leverage prosody for better comprehension, and does not generate expressive speech. In this work, we present a prosody-aware generative spoken language model (pGSLM). It is composed of a multi-stream transformer language model (MS-TLM) of speech, represented as discovered unit and prosodic feature streams, and an adapted HiFi-GAN model converting MS-TLM outputs to waveforms. We devise a series of metrics for prosody modeling and generation, and re-use metrics from GSLM for content modeling. Experimental results show that the pGSLM can utilize prosody to improve both prosody and content modeling, and also generate natural, meaningful, and coherent speech given a spoken prompt. Audio samples can be found at https://speechbot.github.io/pgslm. Codes and models are available at https://github.com/pytorch/fairseq/tree/main/examples/textless_nlp/pgslm.
1 Introduction
The paper addresses the limited generative use of speech pre-training by extending text-free spoken language modeling with explicit prosody. Its pGSLM jointly models phonetic content and prosody to improve comprehension and generate coherent, expressive speech continuations.
- Speech pre-training has mainly been evaluated on discriminative tasks, while generative spoken language modeling remains comparatively unexplored.
- GSLM removes text dependence with discovered units but loses most prosodic information, limiting coherent tonal continuation.
- pGSLM jointly models self-supervised phonetic units with quantized fundamental frequency and duration to represent content and prosody without text supervision.
- pGSLM combines an autoregressive Multi-Stream Transformer Language Model with an adapted unit HiFi-GAN that converts predictions into waveforms.
- Joint prosody modeling improves phonetic content modeling, while pGSLM generates continuations coherent with prompts in both content and prosody.
- Proper choices of model and prosodic representation are crucial for natural, coherent, and expressive generated speech.
2 Related Work
The work extends prior prosody-aware speech modeling by removing text supervision and modeling prosody autoregressively within a generative speech system. It also evaluates prosody more extensively and separately from vocoder effects.
- Prosody has been used to improve speech comprehension tasks, including segmentation, dialogue act classification, parsing, pathology assessment, ASR, and language modeling.
- Unlike prior work that combines prosody with text, pGSLM uses self-supervised discovered units, making it applicable to written and unwritten languages without textual supervision.
- pGSLM models temporal dependencies between prosodic features, unlike TTS approaches that assume within-utterance conditional independence given text.
- The evaluation considers both teacher-forcing and sampling, whereas prior work generated prosody deterministically and did not model its multimodal nature.
- Prosody prediction is evaluated separately from synthesized-waveform prosody, avoiding conflation between the prediction module and vocoder.
3 Method
pGSLM represents speech as self-supervised phonetic units plus duration and normalized pitch, then predicts these streams with a delayed multi-stream transformer and synthesizes them with HiFi-GAN. The design addresses stream dependencies and prosodic multimodality through delayed prediction and quantization.
- 3 Method: pGSLM uses a multi-stream transformer language model and an adapted unit HiFi-GAN as its two main components.
- 3.1 Phonetic and Prosodic Representations: HuBERT-derived units provide the phonetic representation, with vocabulary size 100 and a frame rate of 50 units per second.
- 3.1 Phonetic and Prosodic Representations: Prosody is represented using unit duration and fundamental frequency, with segment-level pitch averaged over voiced frames or set to 0 when a segment is entirely unvoiced.
- 3.1 Phonetic and Prosodic Representations: Speaker-mean normalized log F0 reduces speaker information while preserving expressiveness by retaining variance.
- 3.2 Multi-Stream Transformer LM: The MS-TLM predicts unit, duration, and normalized pitch streams from segment representations, with each stream independently projected from the transformer output.
- 3.2.1 Delayed prosody prediction: A prosody delay factor addresses strong within-segment correlations by predicting the current unit alongside prosodic representations from a previous segment when ∆=1.
- 3.2.2 Quantizing prosodic representations: Continuous prosody modeling can underfit multimodal distributions, producing near-zero pitch predictions and dull generated speech.
- 3.2.2 Quantizing prosodic representations: Quantization represents duration with 32 integer bins and normalized log F0 with K=32 equal-probability bins.
4 Experimental Setup
The experiments evaluate prosody-aware spoken language models on English speech using multiple datasets, unit representations, model sizes, and stream-specific and full-speech continuation tasks.
- Data, model, and training: Models are trained on LibriSpeech and a 6K-hour Libri-Light subset, with HuBERT, CPC, and ground-truth phone representations evaluated.HuBERT and CPC use 100-unit vocabularies, while phone representations contain 41 frame-level units.
- Prosody and content evaluation: Evaluation measures meaningful content, consistent and diverse prosody, and combined speech generation in teacher-forcing and inference modes.The metrics assess individual streams and their combination.
- Prosody and content evaluation: Teacher-forcing evaluation uses unit-stream NLL and duration- and pitch-stream MAE on held-out data.Quantized pitch predictions are de-quantized to the means of their respective buckets.
- Prosody and content evaluation: Per-stream continuation provides a 3s prompt, samples the target stream autoregressively, and supplies ground-truth values for the other streams.Quantized prosody uses temperature sampling, while continuous prosody uses scale parameters selected to minimize Min-MAE.
- Prosody and content evaluation: Prosody continuation evaluates correctness with Min-MAE, consistency with Pearson correlation, and expressiveness with generated-value standard deviation.Min-MAE generates 20 samples per prompt, consistency considers sequences at least 6s long, and expressiveness compares variability with ground truth.
- Prosody and content evaluation: Full prompted speech completion samples all three streams from a 3s prompt until EOS or the reference length, then synthesizes output with adapted HiFi-GAN.Content is scored with maximum word-level continuation BLEU2, while human ratings assess sound quality, meaningfulness, and prosody on a 1–5 scale.
5 Results
Across teacher-forcing, continuation, sampling, and human evaluations, adding prosody input improves speech content and prosody modeling, while quantized prosody generally performs best on the introduced metrics.
- Prosodic Inputs Are Useful for Content and Prosody Modeling: Models with phonetic and prosodic inputs consistently outperform phonetic-only counterparts across HuBERT, CPC, and phone representations, feature types, and delay factors.The reported pattern is also significant relative to standard deviations from five random seeds for selected models.
- Prosodic Inputs Are Useful for Content and Prosody Modeling: Prosodic input lowers unit negative log-likelihood, including 1.522 vs. 1.336 for unit-only models, indicating improved speech content modeling.The same direction is reported for continuous-F0, quantized-F0, CPC-based, and phone-based models.
- Prosodic Inputs Are Useful for Speech Generation: With multiple sampled candidates, quantized prosodic streams improve prosody accuracy more than continuous streams, which benefit little or not at all for F0.The authors hypothesize that multinomial sampling covers multiple modes of the underlying prosody distribution.
- Prosodic Inputs Are Useful for Speech Generation: In continuation, prosodic input improves prosody accuracy and consistency and raises word-level BLEU relative to models without prosodic input.For continuous-prosody models, duration correlation rises from 0.176 to 0.344 and F0 correlation from 0.093 to 0.494.
- Prosodic Inputs Are Useful for Speech Generation: Quantized prosody achieves lower minimum-MAE errors and more diverse F0 values than continuous prosody, whereas continuous prosody has better correlation metrics.Quantized prosody reaches up to twice the F0 standard deviation of continuous models; word-level BLEU is similar, with the quantized model slightly ahead.
- Prosodic Inputs Are Useful for Speech Generation: Human evaluation favors the model with prosody input and quantized prosody on sound quality and meaningfulness, while matching continuous prosody on prosody ratings.Without prosody input, the quantized-prosody model performs significantly worse on all reported human-evaluation metrics.
- Prosodic Inputs Are Useful for Speech Generation: Overall, prosody input improves speech modeling, and architectures using quantized prosody generally perform better on the introduced metrics.The conclusion summarizes the teacher-forcing and continuation findings across the evaluated model variants.
6 Conclusion and Future Work
The work introduces pGSLM as a text-free model that explicitly represents content and prosody, with potential applications in conditional speech generation and related speech tasks.
- pGSLM models textual content and prosodic information explicitly without text supervision, using self-supervised units.
- Prosody improves content modeling and enables prompted speech generation that reflects both prompt content and prosody.
- The model could support conditional speech applications where prosody is essential, including conversational scenarios.
- The proposed metrics may evaluate diverse prosody in text-to-speech systems, while pGSLM may support speech classification and expressive synthesis.
A Analysis of Log F0 Distribution
The appendix compares speaker-level log F0 distributions across expressive and plain speech datasets, with each point representing a speaker in the dataset-level analyses.
- Figure A.1 compares log F0 distributions for each speaker–emotion combination in EmoV before and after speaker normalization.
- Figure A.2 plots each speaker’s log F0 mean and standard deviation for EmoV, Blizzard 2013, LJSpeech, and VCTK.
B HiFi-GAN Adaptation Analysis
The HiFi-GAN analysis evaluates quantized pitch representations using voicing and pitch-error metrics, while the appendix also provides a speaker-level linear F0 comparison.
- VDE measures the portion of frames with voicing-decision errors, while FFE counts frames with over-20% pitch deviation or voicing errors.
- Figure A.3 shows speaker linear F0 mean and standard deviation distributions on VCTK, with one point per speaker.
- The chosen quantizer achieves favorable VDE and comparable FFE without pre-training an F0 VQ-VAE quantizer.
C Example of Converting Frame-Level to Segment-Level Representations
The example converts frame-level unit and normalized log F0 sequences into segment-level representations by grouping consecutive units and averaging voiced-frame pitch.
- The six-frame example contains unit–normalized-log-F0 pairs, with zero pitch values marking two unvoiced frames.
- Each segment stores its unit, duration in frames, and average normalized log F0 over voiced frames, or zero when entirely unvoiced.
D Effects of F0 Representation on MS-TLM
The experiments compare pitch representations for speech resynthesis and content modeling. Mean-normalized and logarithmic pitch representations outperform their respective alternatives in content modeling.
- Content modeling: Mean-normalized pitch performs better than raw pitch, while log pitch performs better than linear pitch for content modeling.
- Speech resynthesis: Table 4 evaluates HiFi-GAN speech resynthesis using HuBERT units with different quantized pitch representations.The comparison varies pitch scale, normalization, and resolution on the Blizzard 2013 validation set.
E More Details of Human Evaluation
Human evaluation uses tailored instructions for MOS, MMOS, and PMOS. Raters evaluate generated continuations, with PMOS focusing specifically on prosodic coherence through intonation and rhythm.
- Evaluation instructions: The study provides separate Introduction and Task Instruction text for the three human evaluation metrics.These instructions are detailed in Table 6 and adapted for MOS, MMOS, and PMOS.
- PMOS: PMOS asks raters to judge subjective prosodic coherence in 2–8 second audio files, focusing on intonation and rhythm on a 1-to-5 scale.Raters are instructed to disregard sound clarity and meaning for this judgment.
- Evaluation instructions: The evaluation materials instruct raters to hear a prompt, beep, and generated continuation spoken by a different voice.