PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Turn-Based Interfaces: Synchronous LLMs as Full-Duplex Dialogue Agents

Bandhav Veluri, Benjamin N Peloquin, Bokai Yu, Hongyu Gong, Shyamnath Gollakota

arXiv:2409.15594v1cs.CLcs.LGcs.SDeess.AS

TL;DR

Turn-based spoken-dialogue systems contrast with human interaction, where overlapping speech and backchannels are common. SyncLLM incorporates timing into LLM dialogue and trains with synthetic and real speech; it improves dialogue meaningfulness over dGSLM while maintaining turn-taking naturalness.

  • Problem

    Most spoken-dialogue models are turn-based, while human conversation includes overlapping speech and backchannels that require modeling synchrony.

    Source: §1 · p. 1

  • Method

    SyncLLM integrates periodic timing tokens into LLMs and trains on synthetic spoken dialogue generated from text data alongside real spoken dialogue.

    Source: §1 · p. 2

  • Results

    +2.2-point Mean Opinion Score (MOS) improvement in dialogue content Meaningfulness over dGSLM, while maintaining turn-taking Naturalness.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Simulated interactions between two agents show coherent dialogue with delayed input at Internet-scale latencies.

    Source: §1 · p. 3, §6 · p. 9

    The Llama-3 sequence-length limit constrains long-context modeling in dialogue.

    Source: §7 · p. 9

Abstract

from arXiv · show

Despite broad interest in modeling spoken dialogue agents, most approaches are inherently "half-duplex" -- restricted to turn-based interaction with responses requiring explicit prompting by the user or implicit tracking of interruption or silence events. Human dialogue, by contrast, is "full-duplex" allowing for rich synchronicity in the form of quick and dynamic turn-taking, overlapping speech, and backchanneling. Technically, the challenge of achieving full-duplex dialogue with LLMs lies in modeling synchrony as pre-trained LLMs do not have a sense of "time". To bridge this gap, we propose Synchronous LLMs for full-duplex spoken dialogue modeling. We design a novel mechanism to integrate time information into Llama3-8b so that they run synchronously with the real-world clock. We also introduce a training recipe that uses 212k hours of synthetic spoken dialogue data generated from text dialogue data to create a model that generates meaningful and natural spoken dialogue, with just 2k hours of real-world spoken dialogue data. Synchronous LLMs outperform state-of-the-art in dialogue meaningfulness while maintaining naturalness. Finally, we demonstrate the model's ability to participate in full-duplex dialogue by simulating interaction between two agents trained on different datasets, while considering Internet-scale latencies of up to 240 ms. Webpage: https://syncllm.cs.washington.edu/.

1 Introduction

Turn-based spoken dialogue systems rely on prompts or pauses, while human conversation includes overlapping speech and backchannels. SyncLLM adds real-world timing to LLM dialogue, uses synthetic data alongside limited real speech, and reports improved meaningfulness while maintaining naturalness.

1 Introduction

  • Full-duplex agents must model real-world timing, stream continuously, and handle delayed user input, while available spoken-dialogue datasets total only about 3k hours.Source: §1 · p. 2
  • A periodic synchronization token supplies a shared time frame, while predicting deduplicated token sequences avoids duplicate-token effects on semantic capability.Source: §1 · p. 2
  • The training recipe combines 212k hours of synthetic spoken dialogue generated from text dialogue with 2k hours of real-world spoken dialogue.Source: §1 · p. 2
  • +2.2-point Mean Opinion Score (MOS) improvement in dialogue content Meaningfulness over dGSLM, while maintaining turn-taking Naturalness.The result comes from a user study with n=32 using Llama3-8b.Source: §1 · p. 2
  • SyncLLM fine-tuned on Fisher training data generalizes to the out-of-distribution Candor test set while preserving dialogue meaningfulness and naturalness.Source: §1 · p. 3
  • Simulations between fine-tuned Llama3-8b agents demonstrate streaming full-duplex dialogue under Internet-scale latency and coherent interaction across different training datasets.Source: §1 · p. 3

2 Related work

Prior work spans multimodal speech-text models and spoken-dialogue tasks, but much spoken-dialogue modeling remains turn-based. The closest simultaneous-dialogue system, dGSLM, uses speech-only training and assumes immediate access to the other speaker’s hidden states.

2 Related work

  • Multimodal language models extend pretrained text LLMs with discrete speech tokens and aligned speech-text tasks such as ASR, TTS, S2T, and S2ST.Examples include VioLA (Wang et al., 2023), AudioPaLM (Rubenstein et al., 2023), VoxtLM (Maiti et al., 2023), and SUTLM (Chou et al., 2023).Source: §2 · p. 3
  • Prior spoken-dialogue work covers dialogue state tracking, turn-taking prediction, and response generation, while many LLM-based systems remain explicitly turn-taking.SpeechGPT, USDM, and Spectron (Nachmani et al., 2023) are described as limited to turn-taking settings.Source: §2 · p. 3
  • dGSLM (Lakhotia et al., 2021) models simultaneous dialogue but relies on speech-only training and assumes immediate access to the other speaker’s hidden states.The paper contrasts this with its use of language-model knowledge and its treatment of delayed responses.Source: §2 · p. 3

3 SyncLLM

SyncLLM generates interleaved speech chunks for both speakers on a wall-clock schedule, supporting synchronized dialogue. It uses deduplicated HuBERT tokens for modeling and interpolates them to reconstruct speech sequences.

3 SyncLLM

  • SyncLLM predicts fixed-duration speech chunks for both speakers in sequence, producing two streams synchronized with the real-world clock.The shared architecture can model backchannels, overlaps, and interruptions.Source: §3 · p. 3

3.1 Latency tolerant interaction

  • To handle unavailable current-time user audio, SyncLLM estimates that chunk, uses it to generate its next chunk, then replaces the estimate with actual user input.This supports interaction with Internet latencies up to 240 ms.Source: §3.1 · p. 4

3.2 Token sequence format

  • Dialogue is represented as parallel HuBERT-token streams for two speakers, interleaved with periodic speaker tags whose interval matches the model’s chunk size.HuBERT tokens are sampled at 25 Hz, or one token per 40 ms of audio.Source: §3.2 · p. 4
  • Deduplication removes repeated HuBERT tokens, which often reflect silence and carry about 50% lower semantic content per token than deduplicated sequences.Periodic speaker tags preserve coarse timing while the model predicts deduplicated sequences.Source: §3.2 · p. 4
  • Interpolation restores periodic HuBERT tokens for speech synthesis by repeating deduplicated tokens within each chunk, but may misestimate their original durations.The authors report the effect as imperceptible even at a 240 ms chunk size, with duration error bounded by chunk size.Source: §3.2 · pp. 4–5

4 Training

Training proceeds from synthetic speech generated from text dialogue to full-duplex-format synthetic data and then real spoken dialogue. The stages progressively teach text-speech alignment, within-utterance timing, and conversational turn-taking cues.

Stage 1: Turn-based spoken dialogue model with synthetic speech data

  • Stage 1 uses Bark TTS to create spoken versions of text-dialogue datasets and trains spoken-question-to-spoken-response alignment with Llama3-8b.The vocabulary is expanded with 501 HuBERT tokens and speaker tags.Source: §4 · p. 5
  • Stage 1 randomly varies the proportion of speech sentences in dialogue sequences, using a truncated normal distribution to mix text and speech during training.Training only on fully spoken sequences or increasing the speech proportion stepwise was unstable.Source: §4 · pp. 5–6, Figure 4 · p. 5

Stage 2: Full-duplex dialogue assuming no overlaps

  • Stage 2 converts text-dialogue turns into speech for one speaker and equal-duration silence for the other, teaching timing within utterances but not turn-taking cues.The resulting synthetic dialogue is tokenized in the full-duplex format.Source: §4 · p. 6
  • Before real-world dialogue training, a further synthetic-data stage uses shorter-turn text-dialogue datasets, equivalent to around 20,000 hours of synthetic spoken dialogue.The shorter turns better reflect the more frequent turn-taking of spoken dialogue.Source: §4 · p. 6

Stage 3: Modeling with real-world spoken dialogue data

  • The final training stage uses 2,000 hours of Fisher dialogue to teach timing and turn-taking cues, including pause distributions and backchanneling.Dialogue channels are tokenized separately and interleaved in the full-duplex format.Source: §4 · p. 6

5 Experiments

The experiments evaluate SyncLLM’s semantic quality, turn-taking naturalness, human-rated dialogue quality, and two-agent interaction. SyncLLM improves dialogue meaningfulness over dGSLM while maintaining naturalness, and its interaction results include coherent conversations across differently trained agents.

5 Experiments

  • The evaluation includes continuation and interaction settings, including Fisher-trained agents interacting with Fisher- or CANDOR-trained agents.SyncLLM-F denotes Fisher-trained continuation; SyncLLM-F-F and SyncLLM-F-C denote the two interaction pairings.Source: §5 · p. 6

5.1 Semantic evaluation

  • SyncLLM’s perplexity drop from ground truth is ~15, versus ~70 for dGSLM; Figure 6 compares Fisher and out-of-distribution Candor test splits.Candor is the out-of-distribution test set.Source: §5.1 · pp. 6–7, Figure 6 · p. 7

5.2 Naturalness evaluation

  • SyncLLM generations correlate better with ground-truth turn-taking event durations than dGSLM on both in-distribution Fisher and out-of-distribution Candor test sets.Table 2 reports Pearson correlations; chunk sizes appear in parentheses.Source: §5.2 · p. 7, Table 2 · p. 7

5.3 Human Evaluation

  • SyncLLM-based models significantly outperform dGSLM in dialogue-content Meaningfulness while nearly all models are at parity in turn-taking Naturalness.Both measures approach re-synthesized ground-truth values.Source: Table 3 · p. 8, §5.3 · p. 8
  • Thirty-two annotators rated 180 Fisher and CANDOR items for Naturalness and Meaningfulness using a five-point scale.Three unique raters scored each item; system scores average item medians, with item-level bootstrap confidence intervals.Source: §5.3 · pp. 7–8
  • Moving out of distribution, dGSLM drops -0.24 in M-MOS and -0.51 in N-MOS; SyncLLM-F drops -0.15 and -0.16, while SyncLLM-C drops -0.52 in M-MOS.SyncLLM-C shows no N-MOS decline in the reported comparison.Source: §5.3 · p. 8

5.4 Full-duplex interaction

  • In simulated LLM-LLM interaction, SyncLLM closely matches continuation-mode perplexity and performs better than dGSLM in continuation mode.Figure 7 compares ASR perplexity in continuation and interaction modes.Source: §5.4 · p. 8, Figure 7 · p. 8
  • Human ratings show LLM-LLM interactions outperform dGSLM in Meaningfulness but are slightly worse than single-model continuation.The interaction results also report coherent conversation when the user-side agent was trained on a different dataset.Source: §5.4 · p. 9, Table 4 · p. 9

6 Conclusion

The paper presents Synchronous LLMs as a post-training framework for converting autoregressive LLMs into full-duplex spoken dialogue agents. It reports improved dialogue meaningfulness while maintaining turn-taking naturalness, and robustness to delayed input in simulated agent interactions.

6 Conclusion

  • Synchronous LLMs is a post-training framework that converts an autoregressive LLM into a full-duplex spoken dialogue agent.Source: §6 · p. 9
  • Synchronous LLMs outperform state-of-the-art in dialogue meaningfulness while maintaining turn-taking naturalness.Source: §6 · p. 9
  • Simulated two-agent dialogue shows robustness to Internet-scale delayed input, even when agents lack immediate access to user-generated speech.Source: §6 · p. 9

7 Limitations and Risks

The limitations concern speech quality, dialogue expressivity, and context length; the paper also identifies inappropriate outputs and misuse as risks.

7 Limitations and Risks

  • The inherited Llama-3 sequence-length limit constrains long-context dialogue modeling and the use of higher-token-rate tokenizers such as EnCodec.The paper also notes that a more advanced speech generator could improve speech quality, and that expressivity and non-verbal sounds such as laughter remain unstudied.Source: §7 · p. 9
  • Inappropriate speech outputs may require toxicity mitigation, and bad actors could misuse the model for online scams.Speech watermarking is identified as one potential countermeasure to abuse.Source: §7 · p. 9

A.1 Hyperparameters

The three training stages use progressively smaller token batches and learning rates, with fewer warmup steps in later stages.

A.1 Hyperparameters

  • Stage 1 trains for 40k iterations with 1M-token batches, a 3×10−5 learning rate, and 500 warmup steps.The model uses Llama3-8b’s original sequence length of 8192 and 128 A100 GPUs.Source: §A.1 · p. 11
  • Stage 2 trains for 6000 iterations with 512k-token batches, a 2.2 × 10−5 learning rate, and 200 warmup steps.Source: §A.1 · p. 11
  • Stage 3 trains for 2000 iterations with 256k-token batches, a 1.5 × 10−5 learning rate, and 100 warmup steps.Source: §A.1 · p. 11

A.2 Benchmarking interleaving strategies

This appendix compares two text-speech interleaving strategies and gives human-rating instructions for naturalness and meaningfulness, including how evaluators compare prompt and continuation audio.

A.2 Benchmarking interleaving strategies

  • Sentence-level interleaving outperforms turn-level interleaving on all reported spoken-language-understanding benchmarks.Sentence-level selection randomly assigns each sentence a text or speech modality; turn-level selection assigns one modality to the whole turn. Table 5 reports zero-shot accuracy on WUGGY, BLIMP, Topic-StoryCloze, and StoryCloze.Source: §A.2 · p. 11, Table 5 · p. 12

B Naturalness-MOS Instructions

  • Naturalness ratings assess turn-taking independently of conversation content and voice clarity, comparing continuation audio with the preceding prompt.Evaluators are told that samples may be AI-generated, human recordings, or human recordings with AI-generated voices.Source: B Naturalness-MOS Instructions · p. 12

B.1 N-MOS & M-MOS

  • The N-MOS and M-MOS protocol provides separate human evaluations of turn-taking naturalness and dialogue-content meaningfulness.Naturalness ratings use a five-point scale from essentially non-human-like turn-taking to essentially indistinguishable from human-like turn-taking.Source: §B.1 · p. 12

B.1.1 Meaningfulness-MOS

  • Meaningfulness ratings judge whether the continuation is coherent and plausible given the prompt, even if its content is not perfectly grammatical.Evaluators listen to the prompt and continuation in sequence and rate the continuation’s meaningfulness in context.Source: §B.1.1 · p. 12

C Effect of latency on full-duplex interaction

In simulated two-model interaction, SyncLLM’s turn-taking performance remains close to continuation-setting performance at 160 and 200 ms, but drops at 240 ms. The paper also reports better prompt-to-generation turn-taking correlation than dGSLM.

C Effect of latency on full-duplex interaction

  • At 160 and 200 ms, interaction performance closely matches continuation-setting performance; it drops at 240 ms.The latency comparison uses Pearson correlation of turn-taking-event durations against ground-truth continuations on combined Fisher and Candor test sets.Source: C Effect of latency on full-duplex… · p. 13

D Turn-taking event correlation between prompt and generation

  • SyncLLM demonstrates better turn-taking-event correlation with prompts than dGSLM in the continuation setting.The comparison measures Pearson correlation between prompt and generated turn-taking events.Source: D Turn-taking event correlation… · p. 13
Loading 2409.15594v1…