PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Human-like Multimodal Conversational Agent by Generating Engaging Speech

Taesoo Kim, Yongsik Jo, Hyunmin Song, Taehwan Kim

arXiv:2509.14627v1cs.HCcs.AIcs.CL

TL;DR

Because existing conversational datasets may be small or lack audio, this paper introduces the 31,000-utterance MultiSensory Conversation dataset and a multimodal model that generates text responses and voice descriptions. Experiments report that using visual and audio modalities supports contextually appropriate, engaging speech responses.

  • Problem

    Existing conversational datasets are often small or lack modalities such as audio, despite the complementary role of words, facial expressions, and speech tones in communication.

    Source: §1 · p. 1

  • Method

    The paper introduces a dataset of about 31,000 daily-conversation utterances and a multimodal model that takes images, audio, and text to generate a textual response and voice description.

    Source: §1 · p. 1, §3 · p. 3

  • Results

    The model effectively uses visual and auditory modalities to produce contextually appropriate speech responses, as supported by quantitative metrics and qualitative assessments.

    Source: §1 · p. 1, §4.2.1 · p. 3, §4.4 · p. 4

  • Takeaways & Limitations

    The authors conclude that their approach contributes to more natural, human-like conversation, and that the dataset may promote subsequent research.

    Source: §5 · p. 4

    Downloaded YouTube videos cannot be shared, so the authors release preprocessing code and require users to process their own legally obtained data.

    Source: G. Limitations · p. 7

Abstract

from arXiv · show

Human conversation involves language, speech, and visual cues, with each medium providing complementary information. For instance, speech conveys a vibe or tone not fully captured by text alone. While multimodal LLMs focus on generating text responses from diverse inputs, less attention has been paid to generating natural and engaging speech. We propose a human-like agent that generates speech responses based on conversation mood and responsive style information. To achieve this, we build a novel MultiSensory Conversation dataset focused on speech to enable agents to generate natural speech. We then propose a multimodal LLM-based model for generating text responses and voice descriptions, which are used to generate speech covering paralinguistic information. Experimental results demonstrate the effectiveness of utilizing both visual and audio modalities in conversation to generate engaging speech. The source code is available in https://github.com/kimtaesu24/MSenC

1. Introduction

Human communication relies on words, facial expressions, and speech tones, while adapting both what is said and how it is expressed. The paper addresses the gap between multimodal systems that generate text and the need for mood-responsive speech.

1. Introduction

  • The proposed conversational agent generates speech responses by adapting both what it says and how it expresses it to multimodal cues.Human responses adapt to words, facial expressions, and speech tones, including tone, emphasis, and delivery.Source: §1 · p. 1
  • Current multimodal systems can interpret diverse inputs but primarily generate text, while existing TTS is inadequate for human-like communication that reflects conversational mood.The paper also notes that some multimodal speech-generation approaches struggle with cross-modal consistency or lose acoustic details.Source: §1 · p. 1
  • The introduction presents text-only, text-and-audio, and text-audio-paralinguistic interaction as successive levels of conversational-agent capability.Source: Figure 1 · p. 1
  • MSenC is a curated dataset of about 31,000 utterances from daily-conversation YouTube videos, built to address existing datasets’ small size or missing modalities.The model uses BLIP-2 for cross-modal training and instruction-tunes an LLM to generate voice descriptions from the overall communication mood.Source: §1 · p. 1

2. MSenC Dataset

MSenC is designed for daily conversation with clear speech and rich visual and auditory context. Its construction includes dialogue and utterance segmentation, speaker assignment, and speech-description extraction.

2. MSenC Dataset

  • MSenC contains natural daily conversations with clear, high-quality English speech, no background music, and rich visual and auditory context.The dataset totals 21.5 hours, with an average utterance duration of 2.46 seconds.Source: §2 · p. 2

2.1.1. Dialogue Split

  • Dialogue segmentation manually filters video when speakers or settings change, helping isolate dialogue units for the dataset.The authors describe manually segmenting more than 36 hours of video and filtering out unnecessary parts.Source: §2.1.1 · p. 2

2.1.2. Utterance Split

  • Utterance segmentation uses timestamped Whisper-large-v3 ASR, scene detection for long audio, and speaker diarization for clips still exceeding 25 seconds.The diarization and ASR combination addresses diarization’s speaker-identification and over-fragmentation challenges.Source: §2.1.2 · p. 2

2.2.1. Speaker Assignment

  • Speaker assignment clusters WeSpeaker speech embeddings with HDBSCAN and cosine distance, accommodating a variable number of participants.Manual labels across 602 samples in 20 dialogues yielded 95.49% accuracy.Source: §2.2.1 · p. 2

2.2.2. Speech Description

  • Speech descriptions are extracted from annotations of gender, pitch, monotony, pace, and reverberation, then LLaMa-3 uses them to describe conversation mood.Gender recognition from raw speech achieved an F1 score of 99.93%.Source: §2.2.2 · p. 2

3. Method

The model processes visual, audio, and linguistic inputs and generates a textual response with a voice description. It uses multimodal features and instruction tuning to produce speech responses that incorporate paralinguistic cues.

3. Method

  • The model takes images, audio, and text for an utterance and generates a textual response together with a voice description.The generated description is used to specify how the response should be spoken.Source: §3 · p. 3

3.1. Multimodal Understanding

  • Video and audio are processed separately with Q-Formers, then projected into the LLM embedding space and concatenated with other modality features.Video is sampled at three frames per second, while the entire speech is used as audio input because sampling it would lose significant information.Source: §3.1 · p. 3

3.2. Speech Description Generation

  • Instruction tuning trains the model to generate a response and voice description from utterances combined with conversation history.Speaker instructions specify who delivers each utterance, allowing the model to respond or continue the previous utterance.Source: §3.2 · p. 3

3.3. Training Loss

  • The LLM backbone is fine-tuned with parameter-efficient fine-tuning to specialize it for generating paralinguistic descriptions in conversation.The training loss uses a target response sentence paired with its corresponding audio description.Source: §3.3 · p. 3

4. Experiment

Experiments evaluate multimodal inputs and generated speech on MSenC and MELD, including modality comparisons, human ratings, emotional continuity, and qualitative examples. The reported results favor the full audio-video-text input and the proposed system across the speech-evaluation criteria described.

4.1. Experimental Setup

  • Experiments use MSenC and MELD, with CLIP-VIT visual features, WavLM acoustic features, Mistral-7B, and Parler-TTS.The reported setup used a single NVIDIA A100 80G GPU; training took 30 hours.Source: §4.1 · p. 3

4.2.1. Modality Ablation

  • On MSenC, combining audio, video, and text yields the highest text-response performance; the same modality pattern is reported on MELD.METEOR and ROUGE are calculated on text responses, excluding voice descriptions.Source: §4.2.1 · pp. 3–4

4.3.1. User Study

  • The proposed system scores higher than StyleTTS2, HierSpeech++, and Parler-TTS on emotional, suitability and engagement, and conversational naturalness criteria.The human evaluation used five judges and 100 samples from the MSenC test set.Source: §4.3.1 · p. 4, Figure 4 · p. 4

4.3.2. Emotion Classification

  • The proposed model outperforms baseline models in maintaining consistent emotional expression across conversation on the MSenC test set.The evaluation classifies each utterance's emotion and compares it with the prior speech.Source: §4.3.2 · p. 4, Table 2 · p. 4

4.4. Qualitative Analysis

  • Qualitative examples contrast multimodal and text-only responses and compare generated speech descriptions with reference pace, pitch, and tone.The authors report that video gestures and audio tone convey urgency in the first dialogue example.Source: §4.4 · p. 4, Figure 5 · p. 4

5. Conclusion

The paper presents a multimodal dialogue model and a curated dataset for generating engaging speech from visual and audio inputs. The model outperforms baselines in experiments, while its single-voice output does not reproduce a speaker's exact voice from historical recordings.

  • The proposed dialogue model processes visual and audio inputs to generate engaging speech.The conclusion identifies the dataset as curated specifically for training such models.Source: §5 · p. 4
  • The model outperforms baselines in experiments, which the authors cite as evidence of its effectiveness.Source: §5 · p. 4
  • The model cannot reproduce a speaker's exact voice from historical recordings, but inference uses a consistent single voice.The authors state that this limitation does not affect inference.Source: §5 · p. 4

A. Implementation Details

Implementation uses Mistral-7B, a batch size of six, and Adam with a learning rate of 5e-5 and decay of 0.98. Video is sampled at three frames per second, while audio remains unsampled; training runs for 30 hours on one A100 GPU.

  • Training uses Mistral-7B, batch size 6, and Adam with a 5e-5 learning rate and 0.98 learning-rate decay.The model is trained for 10 epochs, selected using validation data.Source: A. Implementation Details · p. 6
  • Video frames are sampled at three per second for each utterance, while audio remains unsampled.Video and audio padding sizes are 50 and 800, respectively.Source: A. Implementation Details · p. 6
  • The maximum LLM input length is 800 tokens, covering about 10 multimodal histories, with older history truncated first.Training takes 30 hours on a single NVIDIA-A100 80G GPU.Source: A. Implementation Details · p. 6

B. MSenC Dataset Details

MSenC contains 1,120 dialogues and 31,409 utterances across 21.5 hours, with a reported male-to-female speaker rate of about 1:1.5. The section also describes speaker clustering and instruction tuning for generating voice descriptions.

B. MSenC Dataset Details

C. Instruction Tuning

  • Instruction tuning provides initial instructions, speaker ID information for each utterance, and a final instruction to generate voice descriptions.The illustrated text-only example represents inputs that actually integrate text, audio, and video.Source: C. Instruction Tuning · p. 6

D. LLM Fine-Tuning

The paper reports that parameter-efficient fine-tuning improves conversational performance over the base model. It evaluates performance using BLEU, METEOR, and ROUGE metrics.

D. LLM Fine-Tuning

E. Additional Qualitative Samples

The examples show video-informed text responses and speech descriptions with pace, pitch, and tone similar to the reference. Actual model inputs integrate text, audio, and video, although the illustration uses text for clarity.

E. Additional Qualitative Samples

  • Video-informed output text closely matches the reference context, unlike the text-based unimodal comparison.Source: E. Additional Qualitative Samples · pp. 6–7
  • Figure 8 illustrates instruction-tuning input, while actual inputs integrate text, audio, and video.The figure's text-only example is provided for easier understanding.Source: Figure 8 · p. 7
  • Table 5 reports LLM fine-tuning results on MSenC and MELD.Source: Table 5 · p. 7
  • The speech descriptions resemble the reference in pace, pitch, and tone, yielding engaging and contextually suitable responses.Source: E. Additional Qualitative Samples · p. 7

F. Details of Human Evaluation

The human evaluation used video-based histories of up to five entries and three response options. It assessed emotional expression, suitability and engagement, and conversational naturalness across 100 output samples.

F. Details of Human Evaluation

G. Limitations

The model cannot reproduce a speaker’s exact past voice, and YouTube video-sharing restrictions limit the release to preprocessing code and require users to process their own legally obtained data.

G. Limitations

  • The model cannot replicate a speaker’s exact past voice, but using one consistent voice does not hinder inference.Source: G. Limitations · p. 7
  • YouTube use carries potential copyright risks.Source: G. Limitations · p. 7
  • Downloaded videos cannot be shared, so only preprocessing code is released and users process their own legally obtained data.Source: G. Limitations · p. 7
Loading 2509.14627v1…