PaperlayerView exact revision on arXiv

Source-linked AI summary

Style-Talker: Finetuning Audio Language Model and Style-Based Text-to-Speech Model for Fast Spoken Dialogue Generation

Yinghao Aaron Li, Xilin Jiang, Jordan Darefsky, Ge Zhu, Nima Mesgarani

arXiv:2408.11849v1cs.CLcs.AIeess.AS

TL;DR

Style-Talker addresses limitations in speech-dialog systems that struggle to preserve emotional nuance and incur latency from ASR. It generates response text and speaking style from audio and dialogue context, outperforming baselines in naturalness, coherence, and processing speed, though it lacks explicit turn-taking mechanisms.

  • Problem

    ASR-LLM-TTS pipelines struggle to capture input emotional nuance, while prior style-based approaches use predefined categories requiring extensive labeling.

    Source: §1 · p. 2

  • Method

    Style-Talker combines an audio LLM that generates response text and speaking style from speech and preceding context with a style-based TTS model that synthesizes the response.

    Source: §1 · p. 2, §3.1 · pp. 4–5

  • Results

    Style-Talker demonstrates superior naturalness, coherence, and processing speed to cascade and end-to-end baselines; it is nearly twice as fast as the cascade and four times faster than SpeechGPT.

    Source: §5.1 · p. 9, §6 · p. 10

  • Takeaways & Limitations

    Generating 10-second audio from 10-second input in around 1.5 seconds makes Style-Talker feasible for real-time human-computer interaction.

    Source: §5.1 · p. 9

    The system does not include explicit turn-taking mechanisms, which future work will explore adding.

    Source: §6 · p. 10

Abstract

from arXiv · show

The rapid advancement of large language models (LLMs) has significantly propelled the development of text-based chatbots, demonstrating their capability to engage in coherent and contextually relevant dialogues. However, extending these advancements to enable end-to-end speech-to-speech conversation bots remains a formidable challenge, primarily due to the extensive dataset and computational resources required. The conventional approach of cascading automatic speech recognition (ASR), LLM, and text-to-speech (TTS) models in a pipeline, while effective, suffers from unnatural prosody because it lacks direct interactions between the input audio and its transcribed text and the output audio. These systems are also limited by their inherent latency from the ASR process for real-time applications. This paper introduces Style-Talker, an innovative framework that fine-tunes an audio LLM alongside a style-based TTS model for fast spoken dialog generation. Style-Talker takes user input audio and uses transcribed chat history and speech styles to generate both the speaking style and text for the response. Subsequently, the TTS model synthesizes the speech, which is then played back to the user. While the response speech is being played, the input speech undergoes ASR processing to extract the transcription and speaking style, serving as the context for the ensuing dialogue turn. This novel pipeline accelerates the traditional cascade ASR-LLM-TTS systems while integrating rich paralinguistic information from input speech. Our experimental results show that Style-Talker significantly outperforms the conventional cascade and speech-to-speech baselines in terms of both dialogue naturalness and coherence while being more than 50% faster.

1 Introduction

Spoken dialog systems have moved from multi-component pipelines toward streamlined ASR-LLM-TTS systems and end-to-end speech generation. Style-Talker integrates input audio with prior transcribed context and speaking style, reporting faster generation and more natural, coherent, emotionally congruent responses.

1 Introduction

  • The conventional spoken-dialog architecture combines speech recognition, language comprehension, dialog management, response generation, and speech synthesis.Deep-learning advances condensed this pipeline into ASR-LLM-TTS systems and motivated end-to-end speech-to-speech generation.Source: §1 · pp. 1–2
  • ASR-LLM-TTS systems struggle to capture input emotional nuance, while end-to-end speech LLMs face data, processing-speed, and computational-resource challenges.The paper also notes that some style-based systems rely on predefined categories and extensive labeling, hindering in-the-wild use.Source: §1 · p. 2
  • Style-Talker integrates input audio, prior transcribed speech context, and speaking style, and trains an audio LLM to generate response text and its associated style.It processes current input speech for transcription and style analysis during response playback, informing the next turn.Source: §1 · p. 2
  • Style-Talker is reported to be nearly twice as fast as the Whisper large-v2 cascade and more than four times faster than SpeechGPT.The paper also reports greater naturalness and coherence than both baselines, with responses emotionally congruent with dialog context.Source: §1 · p. 2

2 Related Works

Spoken-dialog research has pursued text-based systems and end-to-end speech-to-speech generation, each with distinct limitations. Style-Talker combines direct audio processing and learned speech styles with response text generation and style-based speech synthesis.

2 Related Works

  • The end-to-end approach bypasses intermediate text generation, whereas text-based systems generate a text response before TTS synthesis.The paper frames both approaches as having distinct challenges and limitations.Source: §2 · p. 2
  • The conventional cascade transcribes input speech, generates response text, and synthesizes response speech; Style-Talker merges the first two steps with an audio LLM.The response text and corresponding style are generated directly.Source: Figure 1 · p. 3

2.1 Text-Based Approaches and Paralinguistic Decoupling

  • Text-based dialog systems can decouple speech content from paralinguistic features, while transcription introduces latency.Style-Talker directly processes speech inputs, circumventing transcription delays, and adopts self-supervised speech styles learned for synthesis rather than predefined emotion categories.Source: §2.1 · p. 3

2.2 End-to-End Speech-to-Speech Generation

  • End-to-end speech-to-speech systems generate speech directly from speech input, but systems trained from scratch on speech corpora often lack semantic coherence.SpeechGPT and USDM integrate text-based LLMs with speech datasets, while challenges in human-like speech quality and multimodal training remain.Source: §2.2 · p. 3
  • Style-Talker fine-tunes an audio language model and a style-based TTS model to generate semantically and paralinguistically coherent responses.The paper presents this approach as distinct from end-to-end systems and reports improvements in dialog naturalness and processing speed.Source: §2.2 · p. 4

3 Methods

Style-Talker combines an audio LLM that generates response text and style with a style-based TTS model. It carries both transcribed speech and speaking style into dialogue context, while processing input transcription and style during response playback.

3.1 Style-Talker

  • The audio LLM generates response text and a style vector, which a style-based TTS model uses to synthesize speech reflecting the response content and style.During playback, ASR transcribes the incoming utterance and a style encoder extracts its style for the next turn.Source: §3.1 · pp. 4–5

3.1.1 StyleTTS 2

  • StyleTTS 2 is fine-tuned on diarized utterances to learn speaking styles self-supervised and encode them as fixed-length vectors, without explicit style annotations.Its style representation captures paralinguistic features beyond speech content, including speaker identity, prosody, lexical stress, formant transitions, and speaking rate.Source: §3.1 · p. 4, §3.1.1 · p. 5

3.1.2 Qwen-Audio

  • Qwen-Audio is fine-tuned to accept prior-turn text and style as context and generate response style through a projection head on a special <STY_OUT> token.The model predicts prosodic style, while a precomputed acoustic style supplies target-speaker timbre.Source: §3.1.2 · pp. 5–6

3.1.3 Conversation Context

  • Conversation context pairs each utterance’s transcription with its speaking-style vector, which ASR and the style encoder extract while the current response plays.The text-domain context can span up to two minutes, longer than speech-to-speech models such as SpeechGPT, whose context windows are described as limited.Source: §3.1.3 · p. 6

3.2 Training Objectives

  • Training jointly minimizes next-token text loss and style error, with the final objective defined as L = Ltext + λLstyle.The style error is the L1 distance between the ground-truth prosodic style and the style decoded from the <STY_OUT> representation.Source: §3.2 · p. 6

4 Experiments

The experiments evaluate Style-Talker on DailyTalk and PodcastFillers against cascade and speech-to-speech systems, using subjective and objective measures. Evaluation includes dialogue ratings, acoustic and semantic metrics, and real-time timing tests.

4.1 Datasets

  • DailyTalk contains studio-recorded, scripted two-speaker conversations, while PodcastFillers provides conversations from uncontrolled podcast recordings.PodcastFillers transcriptions were re-created with speaker diarization and filler words; data were split by podcast.Source: §4.1 · p. 7

4.2 Evaluations

  • The systems are compared with a traditional ASR-LLM-TTS cascade and SpeechGPT as an end-to-end baseline, but SpeechGPT is evaluated only on DailyTalk.The authors report that its limited context window prevented meaningful fine-tuning on PodcastFillers.Source: §4.2 · p. 7

4.2.1 Subjective Evaluations

  • Subjective evaluation uses MOS-N for response naturalness and MOS-C for coherence with the preceding conversation, rated by U.S. native English speakers on a 1–5 scale.Each test included 10–20 conversations with the previous three turns as context, and each response set was rated by 5–10 evaluators.Source: §4.2.1 · pp. 7–8

4.2.2 Objective Evaluations

  • Objective evaluation measures acoustic correlations, speaker similarity, text-generation metrics, WER, real-time factor, and system delay.Timing tests generate a 10-second response to a 10-second input; acoustic measures include pitch, energy, harmonics-to-noise ratio, and duration.Source: §4.2.2 · p. 8

5 Results

Style-Talker improves conversation naturalness and coherence over cascade and speech-to-speech baselines, and reports faster processing on noisy PodcastFillers recordings. Ablations indicate that prior-turn speaking styles matter for response quality, while the audio encoder extracts text and style cues without explicit input transcription and style.

5.1 Model Performance

  • Style-Talker significantly improves conversation naturalness and coherence over cascade and speech-to-speech baselines on DailyTalk.The reported comparison is based on subjective evaluation.Source: §5.1 · p. 8
  • Nearly twice the cascade system’s speed and four times SpeechGPT’s speed are reported on PodcastFillers, where SpeechGPT fails to produce meaningful speech.The dataset includes recordings in more realistic and noisy settings.Source: §5.1 · p. 9, Table 4 · p. 10
  • Style-Talker’s generated acoustic features have higher correlations with ground-truth responses than other baselines, while semantic metrics decline on PodcastFillers for all systems.Table 3 covers emotion-related acoustic features, speech duration, and speaker-embedding similarity; the passage does not give their values.Source: §5.1 · p. 9, Table 3 · p. 9

5.2 Ablation Study

  • Omitting previous-turn speaking styles adversely affects both semantic and acoustic response quality; replacing audio with transcription and style is less detrimental but requires slower ASR.The ablation compares averaged metrics across DailyTalk and PodcastFillers.Source: §5.2 · p. 9, Table 5 · p. 10
  • Adding the current input’s transcription and style does not improve performance, consistent with the finding that the audio encoder extracts those cues without explicit provision.The study describes the additional inputs as redundant.Source: §5.2 · pp. 9–10

6 Conclusions and Limitations

The conclusion reports stronger naturalness, coherence, and processing speed than traditional cascades and end-to-end baselines on challenging datasets. It also identifies response speed, downstream TTS quality on noisy data, and missing turn-taking mechanisms as limitations or areas for future work.

6 Conclusions and Limitations

  • Style-Talker integrates audio input with transcribed speech context and speaking style, and reports stronger naturalness, coherence, and processing speed than cascades and end-to-end baselines.The conclusion describes these results on challenging datasets.Source: §6 · p. 10
  • Response speed remains limited and may not always surpass spoken-dialog systems using real-time ASR.Generated speech quality also depends on downstream TTS performance, which may degrade on in-the-wild noisy datasets.Source: §6 · p. 10
  • Future work targets causal audio encoding, efficient audio summarization, stronger TTS for noisy real-world data, and explicit turn-taking strategies.The system currently has no explicit turn-taking mechanism.Source: §6 · p. 10

A Implementation Details

The prompt format places ongoing dialogue context before the incoming audio turn, whose transcription and style are not yet appended. It adds speaker-style tokens to the Qwen-Audio prompting format and specifies where input and output styles are represented.

A.1 Prompt Settings

  • The prompt uses a two-speaker conversation template with three rounds, placing the incoming audio in the last round before its transcription and style are appended.Earlier turns provide speaker, style, and text context, followed by an instruction to recognize the incoming speech and generate the next speaker’s style and text.Source: §A.1 · p. 14
  • The template adds speaker-style tokens to the Qwen-Audio prompting format.The paper uses <|extra_123|> as the input-style placeholder and <|extra_124|> as the output-style location identifier.Source: §A.1 · p. 14
  • Projected input styles replace embeddings at <|extra_123|> locations, while the response style is predicted from the final-layer representation of <|extra_124|>.Source: §A.1 · p. 14

A.2 Training and Inference Details

Training uses randomly cropped dialogue histories to predict the next turn, with context windows constrained by model and GPU memory. The paper also specifies shared fine-tuning settings and evaluation and baseline configurations.

A.2 Training and Inference Details

  • Models use randomly cropped dialogue as context to predict the next turn, with context truncated to 512 tokens for SpeechGPT and 1536 for Qwen models.The 1536-token limit is the maximum that fits in GPU memory; evaluation uses fixed randomly sampled crops and training uses newly sampled crops.Source: §A.2 · p. 14
  • Style-Talker and the Qwen baseline are fine-tuned with LoRA and shared AdamW training settings, including a 1e-4 peak learning rate and equivalent batch size of 8.Training uses bfloat16 precision on an Nvidia L40 GPU, for 20 epochs on DailyTalk or 100 on PodcastFilters.Source: §A.2 · p. 14
  • Evaluation fixes reference style from context and uses α = 0.3, β = 0 for DailyTalk and α = β = 0 for PodcastFilters.Sampling uses temperature 1 and top_k 0.8.Source: §A.2 · p. 14

A.3 Baseline Systems

  • The cascade uses Whisper large-v2 for ASR, fine-tuned Qwen-7B for language generation, and the same fine-tuned StyleTTS 2 as Style-Talker.Cascade and Style-Talker use the previous three turns as context; SpeechGPT uses one turn and its 16 kHz audio is used for fair DailyTalk comparisons.Source: §A.3 · p. 15

A.4 PodcastFiller Dataset

Podcast audio was prepared with speaker diarization and verbatim-focused transcription, followed by filtering of segments with multiple speaker indicators. Table 6 reports ablation-study semantic F-1 scores for four text-generation metrics, with proposed-model results copied from Table 2.

A.4 PodcastFiller Dataset

  • Diarization first divided long recordings into speaker segments, after which Whisper large-v2 transcribed them and segments with multiple speaker indicators were discarded.The filtering removed utterances that were not properly diarized.Source: §A.4 · p. 15
  • Whisper large-v2 was fine-tuned on a small verbatim-caption dataset to retain speaker turns, filler words, and other disfluencies that pretrained Whisper models tended to omit.The captions included speaker labels and disfluent speech.Source: §A.4 · p. 15
  • Table 6 reports ablation-study semantic F-1 scores (%) for BLEU, ROUGE-L, METEOR, and BERT Score, with proposed-model results copied from Table 2.Source: Table 6 · p. 15

B Detailed Ablation Study Results

The ablation evaluation omits WER because it varied little across configurations and added noise to aggregate scores, focusing instead on semantic and acoustic metrics. Table 7 presents acoustic results for proposed and ablated models.

B Detailed Ablation Study Results

  • WER was excluded from ablation semantic metrics because it showed little variation across configurations and added noise to aggregate scores.The authors instead focused on metrics reflecting semantic integrity and coherence.Source: B Detailed Ablation Study Results · p. 15
  • Semantic ablation results are presented in Table 6, while acoustic ablation results are presented in Table 7.Source: B Detailed Ablation Study Results · p. 16
  • Table 7 reports acoustic evaluation results for the proposed and ablated models, with proposed-model results copied from Table 3.Source: Table 7 · p. 16

C Subjective Evaluation Details

The subjective evaluation used MTurk filters for approval rate, location, and approved HIT count. Raters scored speech naturalness and conversational coherence, with DailyTalk speakers' non-native status noted for naturalness ratings.

C Subjective Evaluation Details

Loading 2408.11849v1…