Source-linked AI summary
Generative Spoken Dialogue Language Modeling
Tu Anh Nguyen, Eugene Kharitonov, Jade Copet, Yossi Adi, Wei-Ning Hsu, Ali Elkahky, Paden Tomasello, Robin Algayres, Benoit Sagot, Abdelrahman Mohamed, Emmanuel Dupoux
TL;DR
Spoken dialogue modeling must capture turn-taking and paralinguistic signals that text- and ASR-based systems omit or handle indirectly. dGSLM learns synchronized speech-unit streams from raw two-channel audio with a cross-attentive dual-tower transformer, and it reproduces naturalistic turn-taking and non-verbal vocalizations, though semantic coherence remains limited.
Problem
Text- and ASR-based dialogue systems do not adequately model natural spoken turn-taking, prosody, and non-verbal signals, while spoken-dialogue training data remain limited.
Method
dGSLM encodes each dialogue channel into discrete speech units and models the two streams with a dual-tower transformer using cross-attention and duration objectives.
Results
dGSLM reproduces synchronized turn-taking, pauses, overlaps, laughter, and backchannels, with a smaller human-evaluation gap on turn-taking than meaningfulness.
Takeaways & Limitations
The results provide a proof of principle that naturalistic conversational timing can emerge from textless raw-audio modeling.
Takeaways & Limitations
The model lacks semantically coherent speech, and spoken-dialogue modeling is constrained by the limited amount of publicly available speech data.
Abstract
from arXiv · showhide
We introduce dGSLM, the first "textless" model able to generate audio samples of naturalistic spoken dialogues. It uses recent work on unsupervised spoken unit discovery coupled with a dual-tower transformer architecture with cross-attention trained on 2000 hours of two-channel raw conversational audio (Fisher dataset) without any text or labels. We show that our model is able to generate speech, laughter and other paralinguistic signals in the two channels simultaneously and reproduces more naturalistic and fluid turn-taking compared to a text-based cascaded model.
1 Introduction
Natural spoken dialogue depends on coordinated turn-taking, including pauses, overlaps, backchannels, and laughter, but text- and ASR-based systems do not directly model these signals. dGSLM addresses this gap by learning dialogue from raw audio as synchronized streams of discrete speech units.
- Motivation: Human conversations coordinate speaker changes into fluent turns, while pauses and overlaps carry conversational information.Overlapping speech may include backchannels such as “hmm” or “yeah” and non-verbal vocalizations such as laughter.
- Motivation: Text-based dialogue systems largely impose turn-taking through alternating written turns, leaving spoken coordination outside the primary modeling problem.Spoken assistants commonly use wake words and silence thresholds, which can produce slow and unnatural conversations.
- Approach: dGSLM trains directly from raw conversational audio, bypassing text and ASR while learning discrete speech representations from spontaneous conversations.The representations capture verbal and non-verbal material and encode each interlocutor as a parallel stream of discrete tokens.
- Approach: The system uses a dual-tower transformer with cross-attention so two channel-specific autoregressive models can synchronize their generated units.The units are decoded into waveforms after autoregressive generation.
- Findings: dGSLM captures turn-taking and backchanneling more naturally than a text-based cascaded model, but remains weaker on deep syntactic and semantic dialogue content.The paper presents this as a proof of principle for modeling spontaneous conversational surface characteristics with language-modeling techniques.
2 Related work
Prior work provides discrete speech representations and substantial text-based dialogue-generation progress, but spoken dialogue modeling remains constrained by limited data and complex multimodal turn-taking cues. The paper positions its approach within efforts to model speech phenomena that text transcripts omit.
- Unsupervised Spoken Language Modeling: Self-supervised raw-audio models learn representations for downstream tasks, often using discrete units that can feed autoregressive speech-generation models.These units can represent prosody, intonation, and non-verbal vocalizations absent from written transcriptions.
- Dialogue Generation: Dialogue-generation research has advanced substantially through end-to-end neural and retrieval-augmented methods, especially for text-based language representation.The cited progress concerns general dialogue-generation systems rather than spoken turn coordination.
- Dialogue Generation: Publicly available spoken-dialogue data remain much smaller than text-dialogue resources, limiting the capacity of spoken dialogue models.The Fisher corpus contains roughly 12M words, compared with tens of billions of words for text-based dialogue systems.
- Turn-taking Modeling: Human turn-taking relies on prosodic, linguistic, and non-verbal cues, making it a challenging modeling problem.The paper uses established turn-taking event definitions to analyze model outputs.
3 Approach
The approach converts two-channel conversational audio into discrete units and models both speakers with a cross-attentive Dialogue Transformer Language Model. It combines edge-unit and delayed-duration objectives with textless speech generation and evaluates turn-taking from parallel audio outputs.
- System overview: The pipeline uses a HuBERT encoder, a Units-to-Units language model, and a HiFi-GAN-based Units-to-Speech decoder for conversational audio.HuBERT is trained on the conversation dataset, while waveform generation uses discrete units and speaker information.
- Dialogue Transformer Language Model: The Dialogue Transformer Language Model is a two-tower transformer with cross-attention and shared weights for jointly modeling two speaker channels.Cross-attention shares information between channels while shared weights avoid permutation-invariant training.
- Training objectives: The model is trained with edge-unit prediction and delayed duration prediction objectives.Edge-unit prediction targets changes between consecutive units, while delayed duration prediction uses a continuous duration output with an L1 loss.
- Generation: During generation, predicted edge units are repeated according to their rounded durations, and the overwritten units are fed back until the next edge unit.This inference procedure applies the first predicted edge unit and its duration across the corresponding number of time steps.
- Comparisons: The approach compares the cross-attentive two-tower model with a simpler single-tower dual-input model and a cascaded ASR, text-language-model, and TTS system.The single-tower baseline combines speaker channels at the embedding layer and separates them at the output layer.
- Turn-taking analysis: Parallel two-channel outputs support turn-taking analysis using VAD-derived IPUs, turns, pauses, gaps, overlaps, and backchannels.An IPU is a continuous speech stretch bounded by more than 200 ms of VAD silence on both sides.
4 Experimental Setup
The experiments train and compare discrete-unit dialogue models on two-channel Fisher audio, then evaluate training behavior and generated dialogue quality using automatic, event-based, semantic, and human-oriented measures.
- Data and models: The models use HuBERT-derived discrete units from 2,000 hours of two-channel Fisher audio, with 50 units per second.A 98/1/1 speaker-disjoint split is used, while vocal segments support HuBERT and HiFi-GAN training.
- Data and models: DLM uses two six-layer transformer towers with shared weights, cross-attention between channels, and up to 6,144 unit pairs per sample.The architecture models each channel autoregressively while enabling cross-channel communication.
- Baselines and training: Training comparisons vary cross-attention, edge-unit prediction, duration prediction, and delayed duration factors, alongside a single-tower MS-TLM baseline.The cascaded baseline trains a six-layer text language model on ASR-decoded, BPE-encoded turn-based conversations and synthesizes them with Google TTS.
- Evaluation: Evaluation covers per-channel unit and duration prediction, including NLL, prediction accuracy, MAE, and duration accuracy.A duration MAE of 1 corresponds to 20 ms of error.
- Evaluation: Generation is assessed with turn-taking statistics and consistency, natural-dialogue measures, semantic PPL and VERT, and human judgments.Conditional continuations use 117 30-second prompts, generate 90-second samples at temperature 1.0, and sample from the top 20 units.
- Evaluation: PPL@GT is estimated across temperatures from 0.3–2.0 by fitting an exponential relation between PPL and VERT.For conditional generation, conditional perplexity is computed from the prompt and generated sequence.
5 Results
The results show that cross-attention, edge-unit prediction, and duration modeling improve unit prediction, duration modeling, turn-taking statistics, natural dialogue events, and human-rated naturalness. However, semantic coherence remains weak, while cascaded systems produce rigid alternating turns.
- Content and Duration Modeling: Delayed duration prediction improves duration modeling, reducing d MAE from 1.47 to 1.23.
- Content and Duration Modeling: Adding edge-unit prediction improves edge-unit modeling, reducing u NLL from 2.95 to 2.49 and then to 2.26 with duration prediction.
- Turn-taking Event Statistics: DLM-1’s lack of cross-attention creates large gaps and overlaps despite reasonably good modeling scores.
- Turn-taking Event Statistics: DLM-3–5 produce more overlaps and shorter pauses and gaps, while all models except DLM-1 capture longer intra-turn pauses than between-turn gaps.
- Turn-taking Event Consistency: All models except DLM-1 and the cascaded model maintain good correlations between prompt and continuation event durations, indicating dialogue consistency.
- Natural Dialogue Event Statistics: Edge-unit prediction yields faster, less hesitant speech with more laughter and fewer filler words than models without it, while the cascaded model cannot produce laughter.
- Semantic Evaluation: The dialogue models fail to generate semantically coherent speech, whereas the cascaded model achieves better perplexity but remains below ground truth in conditional generation.
6 Conclusion and Future Work
dGSLM generates intelligible speech and non-verbal vocalizations from raw conversational audio while reproducing synchronized turn-taking patterns. Its lack of semantic coherence remains a central limitation, motivating larger linguistic units, text-based combinations, and pitch integration.
- Conclusion: dGSLM is presented as the first spoken dialogue generation model trained from raw audio, using 2k hours of telephone conversations.
- Conclusion: The model reproduces speech, laughter, backchannels, and synchronized distributions and durations of IPUs, gaps, pauses, and overlaps.
- Conclusion: The model reproduces longer inter-turn pauses than between-turn gaps, showing that pauses alone are not sufficient turn-change signals.
- Future Work: Although dGSLM lacks semantically coherent speech, the authors state that its turn-taking behavior emerges from relatively little data and could support more naturalistic dialogue systems.
- Future Work: The authors propose incorporating pitch, larger raw-audio linguistic units, or text-based models to improve semantic and conversational quality.
A Phonetic Quality of HuBERT Fisher
The experiment compares HuBERT Base models trained on Fisher and LibriSpeech using machine-ABX phonetic tests. Fisher training produces better Fisher-domain ABX discrimination, indicating a domain effect.
- Table A1 compares HuBERT Base models trained on 2000h of Fisher and 1000h of LibriSpeech.The evaluation uses machine-ABX phonetic tests on Fisher and LibriSpeech development data.
- The Fisher-trained model achieves better ABX discrimination on Fisher data than the LibriSpeech-trained model.The comparison supports a domain effect in training-set suitability.
- The reported domain effect favors Fisher as the training set for phonetic discrimination on Fisher conversations.
B Effects of Cross-Attention Layer
The experiment evaluates two-tower Transformer language models with different numbers of cross-attention layers. More layers improve scores, but four layers nearly match six with lower complexity.
- Table A2 evaluates NLL and Accuracy as functions of the number of cross-attention layers.The models are two-tower Transformers trained with next-step prediction.
- More cross-attention layers produce better NLL and Accuracy scores.
- 4 cross-attention layers achieve almost the same performance as 6 layers with less complexity.