PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Natural Conversational Behavior in Tandem Speech-to-Speech Models with Randomized Guidance

Manato Yaguchi, Yotaro Kubo, Hikaru Asano, So Kuroki

arXiv:2609.30773v1cs.CLeess.AS

TL;DR

Tandem speech-to-speech training needs intermediate backend guidance that ordinary conversation recordings do not contain, while generating that guidance per example with a simulator LLM adds preparation overhead. The paper derives guidance from target and randomly sampled corpus responses; synthetic-dialogue response quality is comparable to baselines, and real-conversation training improves turn-taking and naturalness over synthetic-data KAME while retaining a response-quality advantage over Moshi.

  • Problem

    Ordinary conversation recordings lack the intermediate backend guidance needed to train tandem speech-to-speech models, and per-example simulator-LLM generation adds data-preparation overhead when scaling to real conversations.

    Source: §1 · p. 1

  • Method

    Randomized intermediate guidance combines target-response guidance with randomly sampled potentially irrelevant responses from the corpus, avoiding per-example LLM simulation.

    Source: §1 · pp. 1–2

  • Results

    On synthetic dialogues, randomized guidance is competitive with LLM-generated and similarity-based baselines; on 3.8k hours of real conversations, it improves smooth turn-taking and audio-judge naturalness over synthetic-data KAME while retaining an MT-Bench score of 5.04 versus Moshi’s 1.96.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Randomized guidance provides a practical way to combine tandem models’ response-quality benefits with natural interaction learned from real conversations.

    Source: §4 · p. 4

Abstract

from arXiv · show

Tandem speech-to-speech architectures couple a responsive speech frontend with an asynchronous text backend. In KAME, a large language model (LLM) serves as the backend, supplying candidate responses as guidance to the speech frontend while the user is still speaking. Ordinary conversation recordings capture the eventual response but not the guidance the backend would supply during the user's utterance. Generating the missing guidance with a simulator LLM adds substantial data-preparation overhead when training on real conversations. We propose randomized intermediate guidance, which derives guidance directly from the conversation corpus rather than simulating backend LLM behavior. During training, target responses provide informative guidance, while randomly sampled responses provide potentially irrelevant updates during the utterance. This combination aims to teach the frontend to use backend information selectively. On synthetic dialogues, KAME trained with this recipe achieves response quality comparable to that of the LLM-generated and similarity-based baselines. Training on 3.8k hours of real conversations improves smooth turn-taking and audio-judge naturalness over synthetic-data KAME while retaining a response-quality advantage over Moshi. These results show that randomized guidance offers a practical route to combining the response-quality benefits of tandem models with natural conversational behavior learned from real speech.

1. INTRODUCTION

Real-time spoken dialogue must balance response quality with natural interaction, but existing approaches trade low-latency turn-taking against stronger language reasoning. The paper proposes corpus-derived randomized guidance to avoid per-example LLM simulation and evaluates it on synthetic and real conversations.

1. INTRODUCTION

  • Full-duplex S2S supports low-latency generation and flexible turn-taking, while cascaded ASR–LLM–TTS systems can disrupt conversational flow with sequential delays.Moshi’s knowledge and reasoning capabilities remain limited.Source: §1 · p. 1
  • Ordinary conversation transcripts lack the intermediate backend guidance needed to train tandem systems, and generating it per example becomes a bottleneck for real-corpus scaling.Synthetic dialogues improve response quality but do not directly capture spontaneous timing and vocal delivery.Source: §1 · p. 1
  • Randomized intermediate guidance combines the target response with randomly sampled responses as potentially irrelevant updates, avoiding per-example LLM simulation.The mixture is intended to teach the frontend to distinguish and exploit useful guidance.Source: §1 · pp. 1–2
  • On synthetic dialogues, randomized guidance is competitive with LLM-generated and similarity-based guidance; on 3.8k hours of real conversations, it improves turn-taking and naturalness over synthetic-data KAME while retaining an MT-Bench advantage over Moshi.The real-data model scores 5.04 on MT-Bench versus Moshi’s 1.96.Source: §1 · p. 2

2. RANDOMIZED INTERMEDIATE GUIDANCE

KAME’s training guidance stream is normally simulated by an LLM for every dialogue, even though ordinary dialogue data provide only utterances and target responses. Randomized intermediate guidance instead derives target and potentially irrelevant updates from the corpus, which can also provide audio targets and transcripts for real-conversation training.

2.1. LLM-Generated Guidance

  • KAME’s asynchronous LLM periodically supplies candidate responses from partial user inputs as a guidance stream, and simulator-generated traces are trained to converge to the target transcript.The simulator procedure must be repeated for each training dialogue.Source: §2.1 · p. 2

2.2. Randomized Intermediate Guidance

  • Randomized intermediate guidance combines target-response guidance with randomly sampled potentially irrelevant texts to train the frontend to distinguish useful updates without costly per-example simulation.The response pool is deduplicated, excludes responses from the current dialogue, and is filtered by length before uniform sampling without replacement.Source: §2.2 · p. 2

2.3. Application to Real Conversations

  • For real conversations, each retained response’s audio and aligned transcript are training targets; the transcript also supplies target guidance, while other corpus turns supply randomized updates.This constructs both targets and guidance directly from the corpus without per-example LLM generation.Source: §2.3 · p. 2

3. EXPERIMENTS

Randomized guidance is competitive with other guidance strategies on synthetic-dialogue response quality, and training on real conversations improves naturalness and smooth turn-taking over synthetic-data KAME. The real-data model retains a substantial MT-Bench advantage over Moshi, although interruption handling remains limited.

3.1. Response Quality with Randomized Guidance

  • The synthetic-dialogue comparison tests LLM-generated, Similarity-based, Target-only, and Randomized guidance strategies using the same pretrained Moshi checkpoint and inference-time backend.The experiment fine-tunes all parameters for one epoch and scores generated-response transcripts with a GPT-4 judge.Source: §3.1 · p. 3
  • Randomized guidance scores 6.09 on spoken MT-Bench, compared with 6.14 for Similarity-based, 5.54 for LLM-generated KAME, and 5.12 for Target-only.Table 1 evaluates response quality on a 30-question spoken MT-Bench subset.Source: Table 1 · p. 3, §3.1 · p. 3

3.2. Training on Real Conversations

  • Real-data KAME raises audio-judge naturalness from 4.05 to 4.79 and smooth turn-taking from 40.62% to 50.88% over synthetic-data KAME, while scoring 5.04 on MT-Bench versus Moshi’s 1.96.Pause handling remains similar, while interruption handling remains a limitation.Source: Table 2 · p. 4, §3.2 · p. 4

3.1. Response Quality with Randomized Guidance

  • The authors suggest randomized updates may act as negative sampling, encouraging reliance on guidance only when useful, since simulated intermediate guidance may not reflect inference-time update variability.This is presented as a possible explanation, not a demonstrated mechanism.Source: §3.1 · p. 4

3.2. Training on Real Conversations

  • The real-conversation experiment uses a 3.8k-hour English two-speaker corpus, about 2.8 times the audio duration of the 1.36k-hour synthetic corpus.Conversational behavior is measured on smooth turn-taking, interruption handling, and pause handling; the reported local acoustic/ASR-based rates are not directly comparable to published benchmark scores.Source: §3.2 · p. 4

4. CONCLUSION

The paper introduces randomized intermediate guidance, constructing training-time guidance from target responses and randomly sampled corpus responses rather than per-example LLM simulation. Its experiments report competitive synthetic-dialogue response quality and improved natural interaction from real conversational data while retaining an MT-Bench advantage over Moshi.

4. CONCLUSION

  • Randomized intermediate guidance derives missing training-time guidance from the corpus by combining target-response guidance with randomly sampled responses, avoiding per-example LLM simulation.On 3.8k hours of real conversations, it improves smooth turn-taking and audio-judge naturalness over synthetic-data KAME while retaining an MT-Bench advantage over Moshi.Source: §4 · p. 4

Related papers

Loading 2609.30773v1…