PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards human-like spoken dialogue generation between AI agents from written dialogue

Kentaro Mitsui, Yukiya Hono, Kei Sawada

arXiv:2310.01088v1cs.CLcs.LGcs.SDeess.AS

TL;DR

Written dialogue lacks many features of human spoken conversation, including backchannels, laughter, and fluid turn-taking. CHATS generates speaker and listener speech from written dialogue while modeling these features. Evaluations report that it reproduces ground-truth distributions of backchannel frequency and turn-taking durations and produces more natural dialogue than a TTS baseline.

  • Problem

    Generating human-like spoken conversation from written dialogue is challenging because spoken dialogue includes backchannels, laughter, and smooth turn-taking.

    Source: §1 · p. 1

  • Method

    CHATS conditions a discrete-token dialogue system on speaker-side phonetic transcription to generate speaker speech alongside listener-side backchannels and laughter, with an explicit turn-timing mechanism.

    Source: §1 · p. 2

  • Results

    The proposed system reproduced ground-truth distributions of backchannel frequency and turn-taking event durations and produced more natural dialogue than the TTS baseline.

    Source: §5 · p. 9

  • Takeaways & Limitations

    The results support generating written-dialogue-based speech with backchannels and turn-taking while retaining coherent spoken content.

    Source: §1 · p. 2, §5 · p. 9

    The proposed system trails Ground Truth and Resynthesized speech in Dialogue Naturalness, while Resynthesized Sound Quality also trails Ground Truth.

    Source: §4.4 · p. 9

Abstract

from arXiv · show

The advent of large language models (LLMs) has made it possible to generate natural written dialogues between two agents. However, generating human-like spoken dialogues from these written dialogues remains challenging. Spoken dialogues have several unique characteristics: they frequently include backchannels and laughter, and the smoothness of turn-taking significantly influences the fluidity of conversation. This study proposes CHATS - CHatty Agents Text-to-Speech - a discrete token-based system designed to generate spoken dialogues based on written dialogues. Our system can generate speech for both the speaker side and the listener side simultaneously, using only the transcription from the speaker side, which eliminates the need for transcriptions of backchannels or laughter. Moreover, CHATS facilitates natural turn-taking; it determines the appropriate duration of silence after each utterance in the absence of overlap, and it initiates the generation of overlapping speech based on the phoneme sequence of the next utterance in case of overlap. Experimental evaluations indicate that CHATS outperforms the text-to-speech baseline, producing spoken dialogues that are more interactive and fluid while retaining clarity and intelligibility.

1 INTRODUCTION

Written dialogue can be transformed into human-like spoken conversation, but spoken dialogue’s backchannels, laughter, and fluid turn-taking are not fully captured in text. CHATS addresses this gap by generating speaker and listener speech from written dialogue while modeling these conversational features.

1 INTRODUCTION

  • The agent-to-agent setting has not been evaluated by most prior work on backchannels, laughter, and turn-taking, which has focused on human-to-agent conversation or isolated tasks.Spoken dialogue includes cues such as listener “uh-huh” responses and smooth transitions between speakers.Source: §1 · p. 1
  • Existing TTS-based systems are fully turn-based and neglect backchannels and turn-taking, while SoundStorm requires backchannel transcriptions and dGSLM cannot control utterance content via text.The paper also reports that meaningful dGSLM generation requires a vast dataset.Source: §1 · p. 2
  • CHATS generates spoken dialogue coherent with written input while adding backchannels, laughter, and smooth turn-taking.It generates speaker-side utterances from phonetic transcription and listener-side backchannels and laughter without their transcription.Source: §1 · p. 2
  • The study converts spontaneous-dialogue transcriptions into standard written dialogue by detecting and excluding backchannel transcriptions with rule-based and machine-learning methods.The source transcriptions also contain timestamps and laughter.Source: §1 · p. 2
  • The study explores conditioning dGSLM’s dual-tower Transformer with phonemes, TTS-task pretraining, and pitch representation.It analyzes effects on textual fidelity and, for pitch representation, prosody.Source: §1 · p. 2
  • CHATS predicts pause duration after utterances and handles cases where subsequent utterances overlap preceding ones.The mechanism is presented as echoing the rhythm and fluidity of human conversation.Source: §1 · p. 2

2 WRITTEN DIALOGUE PREPARATION VIA BACKCHANNEL EXCLUSION

Spoken dialogue transcriptions include listener responses and timing information that standard written dialogue usually lacks. The preparation method removes temporal metadata, merges closely spaced utterances from the same speaker, and excludes listener utterances to produce written dialogue made up of speaker utterances.

2 WRITTEN DIALOGUE PREPARATION VIA BACKCHANNEL EXCLUSION

  • Transcriptions contain listener utterances and temporal delineations absent from written dialogue, so they are converted to resemble written dialogue.Figure 1 compares spoken-dialogue transcription with written dialogue.Source: §2 · p. 2, Figure 1 · p. 3
  • The method omits temporal metadata, retains verbal content, and merges successive same-speaker utterances separated by less than 200 ms into inter-pausal units (IPUs).It then removes listener IPUs from the transcription.Source: §2 · p. 2
  • An IPU encompassing another speaker’s IPU is an s-IPU, the encompassed IPU is an l-IPU, and other IPUs are u-IPUs.A binary classifier trained on s-IPUs and l-IPUs is applied to u-IPUs; l-IPUs identified by either step are excluded.Source: §2 · p. 3
  • The resulting written dialogues consist exclusively of s-IPUs, and “utterance” denotes an s-IPU unless otherwise specified.Source: §2 · p. 3

3 CHATS

CHATS converts dialogue speech into discrete content and pitch units, predicts two-speaker speech with a unit language model, and reconstructs waveforms while modeling turn-taking.

3.1 SYSTEM ARCHITECTURE

  • The system has three modules: speech-to-unit extraction, a unit language model, and unit-to-speech waveform reconstruction.Source: §3.1 · p. 3

3.1.1 SPEECH-TO-UNIT (S2U) MODULE

  • The s2u module extracts discrete content units for spoken content and speaker-normalized log-F0 pitch units for tonal information.Source: §3.1.1 · p. 3

3.1.2 UNIT LANGUAGE MODEL (ULM)

  • The shared-tower uLM predicts content/pitch units and delayed durations from speaker IDs, current/next phonemes, and context; listener channels use LIS placeholders and generate without text transcription.Source: §3.1.2 · p. 4, Figure 2 · p. 4

3.1.3 UNIT-TO-SPEECH (U2S) MODULE

  • The u2s module reconstructs waveforms from content and pitch units and also uses a speaker embedding because those units retain minimal speaker information.Source: §3.1.3 · p. 5

3.2 TURN-TAKING MECHANISM (TTM)

  • For no overlap, the uLM predicts trailing silence; for overlap, it generates the next utterance’s onset concurrently, conditioned on its phonemes, and uses context units for continuation.Source: §3.2 · p. 5

3.3 DATA AUGMENTATION BY CONTEXT REDUCTION

  • For non-overlapping utterances, training varies context length from zero to 0.9C to address reduced context at dialogue starts; overlap cases are excluded.Source: §3.3 · p. 5

3.4 INFERENCE PROCEDURE

  • At inference, the system generates each utterance using its current and next phonemes plus previously generated context units, then synchronizes pitch and content units for waveform synthesis.Source: §3.4 · pp. 5–6

4 EXPERIMENTS

CHATS was evaluated for utterance fidelity, listener events, turn-taking, and human-rated dialogue quality. It outscored the baseline on all human-evaluation metrics, while analyses found backchannel and turn-taking patterns closer to Ground Truth than comparison systems.

4.1 SETUP

  • The evaluation used 74 hours of two-channel dialogue audio, and compared generated speech with recorded test dialogues using listener-event, turn-taking, and human ratings.The utterance-level test evaluated 4,896 utterances; dialogue-level evaluation included dGSLM, Baseline, and Proposed without TTM.Source: §4.1 · p. 6, §4.2 · p. 6, §4.3 · p. 7

4.2 UTTERANCE-LEVEL EVALUATION

  • In the TTS setting, Proposed’s PER was slightly worse than Baseline, but the degradation was minute given its additional listener-speech generation and turn-taking tasks.Pretraining and context units were effective, and data augmentation was crucial when the TTS setting provided no context.Source: Table 1 · p. 6, §4.2 · p. 7

4.3.1 LISTENER’S EVENT EVALUATION

  • Proposed’s backchannel proportions were closest to Ground Truth for both frequency and duration, while dGSLM produced too many and Baseline too few backchannels.Without the turn-taking mechanism, Proposed produced excessive backchannels.Source: §4.3.1 · p. 7, Table 2 · p. 7
  • Proposed achieved the lowest speaker-level backchannel-frequency MAE and a positive correlation with Ground Truth, supporting preservation of speaker-specific backchannel frequency.The table compares individual-speaker backchannel frequencies between reference and generated dialogues.Source: Table 3 · p. 7, §4.3.1 · p. 8

4.3.2 TURN-TAKING EVENT EVALUATION

  • Proposed’s turn-taking event-duration distributions resembled Ground Truth, whereas Baseline’s overlap distribution deviated because it cannot generate overlaps.Proposed without the turn-taking mechanism underestimated pause and gap durations.Source: §4.3.2 · p. 8, Figure 4 · p. 8
  • For individual speakers’ turn-taking event durations, Proposed consistently outperformed Baseline and Proposed without TTM and achieved results comparable to dGSLM, which used 30 seconds of recorded speech.Proposed used prompt tokens without recorded-speech prompts.Source: §4.3.2 · p. 8, Table 4 · p. 8

4.4 HUMAN EVALUATION

  • Proposed outscored dGSLM and Baseline across all human-evaluation metrics, with significantly higher Dialogue Naturalness than Baseline (p = 0.038).The evaluation covered Dialogue Naturalness, Meaningfulness, and Sound Quality.Source: §4.4 · p. 9, Table 5 · p. 9

5 CONCLUSION

The conclusion presents CHATS as a system for generating spoken dialogue from written dialogue, using speaker, text, and past speech conditioning alongside explicit turn-taking timing. Analyses found ground-truth-like backchannel and turn-taking patterns, and human evaluations found more natural dialogue than the TTS baseline.

5 CONCLUSION

  • CHATS generates spoken dialogues from written dialogues by conditioning its uLM on speaker, text, and past speech and explicitly handling turn-taking timing.The study reports reproducing ground-truth backchannel frequency and turn-taking event-duration distributions well.Source: §5 · p. 9
  • Human evaluations found CHATS produced more natural dialogue than the TTS baseline, and the authors report that backchannels and laughter can be generated without transcriptions.The authors identify larger datasets, improved naturalness and sound quality, raw-text conditioning, speaking-style consistency, and expressiveness as future work.Source: §5 · p. 9

A.1 DATASET AND PREPROCESSING

The dataset comprises 74 hours of two-channel Japanese dialogue recordings with transcriptions, split into training, validation, and test sets. Preprocessing normalized audio, applied voice activity detection and speech recognition, and manually corrected some files and transcripts.

A.1 DATASET AND PREPROCESSING

  • The collected dataset contains 538 dialogues from 32 pairs and 54 Japanese speakers, recorded as two-channel 96 kHz/24-bit audio.The dialogues were divided into 474/32/32 train/validation/test sets, with all speaker pairs represented in validation and test.Source: §A.1 · p. 13
  • The 1,076 channel audio files were downsampled to 16 kHz and 24 kHz for s2u and u2s, respectively, and normalized by average dBFS.Silero VAD and Whisper were used; start times, end times, and transcriptions were manually corrected for 645 files, and transcripts were converted to phonemes using Open JTalk.Source: §A.1 · p. 13

A.2 MODEL, TRAINING, AND INFERENCE

The system uses discrete content and pitch units, phonemes, and a dual-tower uLM to generate speech. Its pipeline combines pretraining and two-channel finetuning with unit-to-speech synthesis and nucleus-sampling inference.

A.2 MODEL, TRAINING, AND INFERENCE

  • The IPU classifier is a three-layer bidirectional LSTM and achieved 87.83% accuracy on an evaluation set of 2,604 speaker and 1,930 listener IPUs.The classifier was trained on 49,339 speaker and 27,794 listener IPUs.Source: §A.2 · p. 13
  • The s2u module extracts 500 content units from HuBERT representations and 32 pitch units, including one for unvoiced frames.HuBERT encodes 16 kHz speech at 50 Hz; WORLD extracts pitch every 20 ms.Source: §A.2 · p. 13
  • The uLM is a six-layer Transformer with four cross-attention layers; its shared-weight two-tower architecture is shown in Figure A.1.Its vocabulary includes content and pitch units, phonemes, speaker IDs, and special tokens; output is limited to content/pitch units, PAD, and EOS.Source: §A.2 · p. 13, Figure A.1 · p. 14
  • A single-channel uLM was pretrained on CSJ with next-utterance phonemes and context units omitted, then finetuned on 82,060 speaker IPUs with two channels.The single-channel pretraining is described as conventional TTS training; finetuning warm-started from its weights.Source: §A.2 · pp. 13–14
  • Nucleus sampling with p = 0.9 was used because top-20 sampling empirically produced speech misaligned with input phonemes.The authors suggest lower-probability units could correspond to pronunciations incongruent with the desired phoneme.Source: §A.2 · p. 14
  • The u2s module embeds content and pitch units into 128-dimensional vectors, sums them, then upsamples to produce a 24 kHz waveform.The model was trained on 130,050 utterances and decoded each channel and utterance individually to limit GPU memory use.Source: §A.2 · p. 14

B EFFECTS OF INTRODUCING PITCH UNITS

The pitch-unit ablation shows a trade-off: removing pitch units changes PER differently across systems, while pitch units yield F0 statistics closer to Ground Truth.

B EFFECTS OF INTRODUCING PITCH UNITS

  • Pitch units bring F0 statistics closer to Ground Truth, but their prediction worsens Resynthesized PER and improves PER for Baseline and Proposed systems when removed.The authors attribute the pronunciation difficulty to predicting pitch units, since pronunciation is mostly determined by content units.Source: B EFFECTS OF INTRODUCING PITCH UNITS · p. 15, Table B.1 · p. 15

C LAUGHTER EVALUATION

The proposed system’s generated laughter frequency and duration are closer to Ground Truth than Baseline and dGSLM, with or without a turn-taking mechanism.

C LAUGHTER EVALUATION

  • Laughter frequency and duration from the proposed system are closer to Ground Truth than Baseline and dGSLM, regardless of turn-taking mechanism.Baseline generated some laughter from laughter already present in written input; dGSLM could not use that information, underestimating laughter frequency.Source: C LAUGHTER EVALUATION · p. 15, Table C.1 · p. 15

D SPEAKER-SPECIFIC CHARACTERISTICS OF TURN-TAKING EVENTS

The speaker-specific plot summarizes median turn-taking-event duration across speakers; generated examples show backchannels, laughter, and some overlap, alongside synthesis and overlap-onset errors.

D SPEAKER-SPECIFIC CHARACTERISTICS OF TURN-TAKING EVENTS

  • Figure D.1 plots each speaker’s median turn-taking-event duration with a regression line and 95% confidence intervals.Each point represents a different speaker.Source: Figure D.1 · p. 16

E GENERATION CASE STUDIES

  • Generated examples closely follow the written dialogue, with listener backchannels and laughter and some utterances overlapping previous ones.The examples also show speaker-side laughter with an explicit LAU tag, but one speaker’s laughter is omitted and another utterance starts too quickly.Source: E GENERATION CASE STUDIES · p. 17, Figure E.1 · p. 17, Table E.1 · p. 17
Loading 2310.01088v1…