Source-linked AI summary
Generative Spoken Language Modeling from Raw Audio
Kushal Lakhotia, Evgeny Kharitonov, Wei-Ning Hsu, Yossi Adi, Adam Polyak, Benjamin Bolte, Tu-Anh Nguyen, Jade Copet, Alexei Baevski, Adelrahman Mohamed, Emmanuel Dupoux
TL;DR
Generative Spoken Language Modeling asks whether acoustic and linguistic structure can be learned from raw audio without text or labels, while evaluation remains difficult across speech’s acoustic and linguistic levels. The paper introduces unsupervised speech-to-pseudo-text pipelines and ASR-based metrics, finding performance above chance overall but varying substantially by linguistic level.
Problem
Learning language from raw audio without text or expert labels remains an open question, and speech generation requires evaluation across acoustic and linguistic levels.
Method
The paper builds unsupervised discrete speech encoder, language-model, and speech-decoder pipelines and evaluates them with ASR-based metrics validated against human judgments.
Results
Performance remained above chance overall, with 6.5% acoustic error, 31.3% lexical error, and syntax near chance at 42% error for the best models.
Takeaways & Limitations
ASR-based metrics provide practical model-comparison measures aligned with human judgments, while discrete-unit count affects speech resynthesis and language modeling differently.
Takeaways & Limitations
Further progress requires adapting language-modeling and text-to-speech architectures to speech’s continuous, multilayered variability beyond simply scaling training data.
Abstract
from arXiv · showhide
We introduce Generative Spoken Language Modeling, the task of learning the acoustic and linguistic characteristics of a language from raw audio (no text, no labels), and a set of metrics to automatically evaluate the learned representations at acoustic and linguistic levels for both encoding and generation. We set up baseline systems consisting of a discrete speech encoder (returning pseudo-text units), a generative language model (trained on pseudo-text), and a speech decoder (generating a waveform from pseudo-text) all trained without supervision and validate the proposed metrics with human evaluation. Across 3 speech encoders (CPC, wav2vec 2.0, HuBERT), we find that the number of discrete units (50, 100, or 200) matters in a task-dependent and encoder-dependent way, and that some combinations approach text-based systems.
1 Introduction
The paper targets textless NLP by learning spoken language from raw audio without text or expert labels, using discovered speech units to train language and speech-generation components. It focuses on evaluation across acoustic and language levels, and encoding and generation modes.
- Textless NLP could serve languages lacking large textual resources or standardized orthographies, which remain poorly supported despite being spoken by millions.
- Unsupervised speech representations motivate learning phonetic content directly from audio, although such representations are not necessarily phonemic.
- Automatically discovered discrete units encode speech as pseudo-text for training a unit language model and speech synthesizer without text.
- Evaluation is difficult because speech combines continuous, variable acoustic detail with higher-level language concepts, unlike text-based discrete representations.
- The paper proposes evaluation in 2x2 encoding-and-generation tasks spanning acoustic and language levels, including ASR-based metrics for generated intelligibility and meaningfulness.A pretrained ASR system converts generated waveforms back to text to support model-independent assessments.
2 Related work
Prior unsupervised speech representation learning uses reconstruction or prediction pretext tasks to capture features for downstream phone and semantic tasks. Related work also covers neural speech resynthesis, low-bitrate representations, voice conversion, and bitrate- and human-based evaluation, while this paper adds an ASR-based metric.
- Unsupervised speech representation learning: Unsupervised speech representation learning targets downstream phone discrimination and semantic prediction using large quantities of unlabeled speech.The literature constructs pretext tasks to exploit unlabeled speech.
- Unsupervised speech representation learning: Pretext tasks are broadly divided into reconstruction and prediction approaches.Reconstruction commonly uses autoencoding, while prediction approaches task a model with predicting speech-related information.
- Unsupervised speech representation learning: Reconstruction methods encode speech into a low-dimensional space and decode it back, with constraints including temporal smoothness, discreteness, and hierarchy.These constraints shape the encoded representation.
- Speech resynthesis: Neural vocoders have enabled natural-sounding speech and music generation, often conditioned on log mel-spectrograms.Related work also studies unsupervised low-bitrate speech representations across machine learning and speech communities.
- Speech resynthesis: Voice conversion combines ASR acoustic-model representations with identity and prosodic information; Zero-Resource evaluation uses bitrate and human evaluation, while this paper adds an ASR-based metric.The cited evaluation framework combines an automatic bitrate measure with human judgments.
3 Evaluation Methods
The evaluation combines automatic metrics for generated speech and encoded representations with human assessments. Generation is evaluated for intelligibility, meaningfulness, quality, and diversity, while representation metrics diagnose acoustic and linguistic properties.
- Generation metrics: Automatic generation metrics use ASR-derived text to assess intelligibility in speech resynthesis and meaningfulness in full speech generation.The evaluated pipelines are S2u→u2S for resynthesis and S2u→uLM→u2S for generation.
- Generation metrics: VERT aggregates bigram self-BLEU and auto-BLEU to measure generated-text diversity, including repetition within individual utterances.Auto-BLEU was introduced because self-BLEU could miss increasing within-sentence repetition at low sampling temperatures.
- Generation metrics: AUC summarizes the curve between perplexity and VERT reference boundaries, with lower values indicating systems closer to the oracle point.The boundaries are perplexity at oracle diversity and diversity at oracle perplexity.
- Encoding metrics: Encoding metrics provide unit-test-like diagnosis of learned representations across linguistic levels, drawing on Zero Resource challenge evaluations.Acoustic evaluation uses between-speaker ABX for phonetic-category separation, while language evaluation uses spot-the-word accuracy on matched word-pseudoword pairs.
- Human evaluation: Human evaluation measures intelligibility with MOS and CER and meaningfulness with subjective ratings, using samples rated on a 1-5 scale.The study evaluated 100 samples per method and enforced at least 15 raters per sample.
4 Proposed Systems
The proposed system comprises three unsupervised components: S2u, uLM, and u2S. It uses pretrained speech encoders discretized with k-means, a causal Transformer language model over pseudo-text units, and a Tacotron-2-based decoder adapted for arbitrary unit sequences.
- uLM: uLM is a causal Transformer trained on pseudo-text unit sequences from the clean 6k-hour LibriLight subset.The model uses the transformer_lm_big architecture with 12 layers, 16 attention heads, 1024-dimensional embeddings, FFN size 4096, dropout 0.1, sequences up to 3,072 units, and temperature sampling.
- uLM: Removing sequential repetitions of units is applied universally because preliminary experiments found it improves performance.The authors hypothesize that deduplication uses the Transformer's limited attention span more efficiently.
- u2S: u2S adapts Tacotron-2 to map pseudo-text units to log Mel spectrograms and appends an end-of-input token to support arbitrary sequences, including incomplete sentences.The EOI token signals the decoder to predict the end-of-output token when attending to it.
- u2S: Random chunks of aligned unit sequences and spectrograms are used with appended EOI tokens so the decoder cannot rely on trailing silence to predict output termination.This addresses the possibility that the decoder ignores EOI and correlates end-of-output prediction with a discrete silence token.
5 Results
Results show that unit count affects resynthesis and generation differently: more units consistently improve resynthesis, whereas generation depends on both unit count and system. Zero-shot performance is strongest acoustically, intermediate lexically, and near chance syntactically, with model- and bitrate-dependent patterns.
- Resynthesis: Resynthesis improves uniformly as the number of units increases across models, with HuBERT and CPC achieving the strongest overall results.The authors attribute the improvement to the speech-to-speech component benefiting from additional input detail.
- Generation: Generation is worst with 50 units, but 200 units is not consistently best, indicating effects from both unit count and system.The generation results average unconditional and conditional conditions across automatic and human evaluations.
- Zero-shot metrics: 6.5% error is achieved by the best model on ABX-across, compared with 31.3% error on spot-the-word and 42% error on sBLIMP.Performance remains above chance overall, but declines across acoustic, lexical, and syntactic levels.
- Zero-shot metrics: ABX score predicts lexical score with r = 0.85, while ABX and syntactic scores correlate with r = 0.71.Across models, CPC has the best ABX units and HuBERT the best language-model scores.
- Zero-shot metrics: More units improve all wav2vec metrics, whereas CPC and HuBERT show U-shaped patterns with best scores at intermediate unit sizes.The models with the highest bitrate do not always achieve the best results.
6 Discussion and Conclusion
The paper establishes Generative Spoken Language Modeling as an unsupervised bridge between speech and NLP, with automatic metrics validated against human judgments. Results support acoustic/phonetic quality but reveal language-level limitations and challenges for scaling, architecture, and low-resource languages.
- Contributions: Generative Spoken Language Modeling bridges speech and natural language processing through an unsupervised task related to Acoustic Unit Discovery, Spoken Language Modeling, Discrete Speech Resynthesis, and Text Generation.The authors provide metrics, baselines, and initial LibriLight results, and open-source the evaluation stack and best baseline models.
- Contributions: ASR-based PER and CER measure intelligibility, while text-generation metrics based on AUC for PPX/VERT measure higher-level linguistic meaningfulness.The two metric families correlated well with human judgment.
- Findings: Automatic metrics confirm representation and output quality at the acoustic/phonetic level but show that language-level improvements remain necessary.The authors expect larger training sets beyond 6k hours to improve performance.
- Findings: 200 units consistently optimize speech resynthesis, whereas language modeling does not necessarily favor the largest unit inventory.The resynthesis result reflects a bitrate/intelligibility trade-off, while more detailed acoustic units can be less suitable for language modeling.
- Challenges: Three challenges remain: determining the best encoder, adapting text-oriented TTS and language-model architectures to speech, and reducing evaluation dependence on textual and linguistic resources.The proposed metrics use external resources for evaluation and model selection, while training remains fully unsupervised; testing across high- and low-resource languages is a stated next step.
7 Appendix
The appendix validates ABX as a low-cost proxy for selecting models and unit sizes, and characterizes temperature effects on generated speech. It also defines a model-neutral continuation-temperature procedure for meaningfulness judgments.
- Metric validation: ABX correlates well with downstream human and automatic metrics, making it a useful proxy for preliminary model and unit size selection.It is less costly than generating TTS output and running human or ASR evaluations.
- Temperature effects: Four qualitative temperature zones range from repetitive outputs to complete sentences, unstructured bags of words, and relatively coherent, varied speech.The mid-temperature range is selected for the systems.
- Temperature selection: 10 continuations produced continuation temperatures varying between 0.5 and 0.9 across models and unit sizes.These temperatures were used for the main-paper MMOS results.
- Temperature selection: The continuation temperature temp@cont maximizes BLEU-2 computed between generated prompt continuations and an oracle continuation.This provides a model-neutral procedure for selecting a single temperature for human meaningfulness judgments.
S1 Implementation Details
This section provides information about model training.
- The section describes model training details.
S1.1 Speech Decoder
Speech decoder models were trained with distributed data parallelism for 500 epochs, selecting the checkpoint with the lowest validation loss. Chunking was used only for Unit-To-Speech, with its size increased progressively during training.
- Training setup: Speech decoder models were trained on 8 32-GB GPUs with a batch size of 32 per GPU for 500 epochs.Training used data distributed training.
- Model selection: Validation loss was computed every 5000 steps, and the model with the lowest loss was selected.
- Chunking: Chunking applied only to Unit-To-Speech, starting at size 50 and increasing by 5 per epoch.The chunk size was initialized to 50 and incremented by 5 per epoch.
S1.2 Training of ASR Models for evaluation · S1.2.1 Frozen ASR Model
The frozen ASR evaluation model is trained from scratch on LibriSpeech using a large wav2vec architecture with CTC loss. It does not use a pretrained model.
- S1.2.1 Frozen ASR Model: The frozen ASR model uses a LARGE wav2vec architecture.The architecture is trained from scratch rather than initialized from a pretrained model.
- S1.2.1 Frozen ASR Model: It is trained with Connectionist Temporal Classification (CTC) loss.The cited training objective is CTC loss from Graves et al. (2006).
- S1.2.1 Frozen ASR Model: The model is trained from scratch.The training procedure explicitly excludes use of the pretrained model.
- S1.2.1 Frozen ASR Model: Training uses the LibriSpeech 960hours dataset.The passage identifies LibriSpeech 960hours as the training dataset.
- S1.2.1 Frozen ASR Model: The ASR model is frozen for evaluation.The passage labels this system as the frozen ASR model used in the evaluation setup.
- S1.2.1 Frozen ASR Model: The training setup combines a large wav2vec model, CTC loss, scratch initialization, and LibriSpeech data.These components define the frozen ASR model’s reported evaluation configuration.
S1.2.2 Frozen Phoneme Recognition Model … S2.2 Generation Task
The supplementary sections describe supervised phoneme and fitted ASR evaluators, evaluation sets and metrics, and generation-task human and automatic evaluation across temperatures and unsupervised unit configurations.
- S1.2.2 Frozen Phoneme Recognition Model: The frozen phoneme recognizer uses BASE wav2vec with CTC training from scratch on LibriSpeech 960hours and gold phoneme transcriptions from g2p-en.It serves as a frozen evaluation model.
- S1.2.3 Fitted ASR Model: The fitted ASR model is a SMALL six-layer, four-head wav2vec Transformer with 256-dimensional embeddings and 1024-dimensional FFN, trained with CTC on synthesized speech.The reduced architecture is designed to speed training.
- S2 Supplementary Results: The supplementary results define separate metric sets for the resynthesized-speech and generation tasks.These metric sets are presented in Sections S2.1 and S2.2, respectively.
- S2.1 Speech Resynthesis Results on fitted ASR metrics: Speech synthesis models are trained on LJSpeech and evaluated on LibriSpeech dev_clean and a 1000-sample LJ Speech hold-out excluded from training and validation.The fitted ASR metrics are used for this evaluation.
- S2.2 Generation Task: Human judgements are usually higher at oracle PPX than at the other temperatures, while system-level result patterns remain globally comparable across settings.Correlation coefficients between human and automatic generation metrics are also reported.
- S2.2 Generation Task: Table S1 reports PER, CER without LM & Lexicon, and WER and CER with LM & Lexicon for three unsupervised models across quantized-unit counts, alongside a supervised topline.The table uses both fitted ASR and phoneme-recognition models.
- S2.2 Generation Task: Human generation evaluation reports MMOS for three unsupervised models and three unit sizes at VERT, PPX, and average temperatures.The temperature settings correspond to oracle VERT, oracle PPX, and their average.