PaperlayerView exact revision on arXiv

Source-linked AI summary

S2Dialog: Multimodal Dialogue Retrieval with Semantic and Acoustic-Style Modeling

Xueqi Wang, Zhigang Wang, Runqing Zhang, Zhenqi Jia, Junfeng Zhao

arXiv:2608.14029v1cs.CL

TL;DR

Existing multimodal dialogue retrieval often remains utterance-level or unimodal, limiting modeling of whole-dialogue semantics and acoustic style. S2Dialog combines dedicated dialogue-level textual and acoustic retrievers with contrastive learning, outperforming representative baselines on DailyTalk with Recall@10 of 50.68% and Recall@50 of 83.56%.

  • Problem

    Existing retrieval methods are largely utterance-level or unimodal, lacking mechanisms to capture whole-dialogue semantic coherence and stylistic consistency across modalities.

  • Method

    S2Dialog uses dedicated dialogue-level textual and acoustic retrievers with textual-acoustic contrastive learning to align similar and separate unrelated dialogues.

  • Results

    Recall@10 of 50.68% and Recall@50 of 83.56%; S2Dialog consistently outperforms all baselines across every evaluation metric.

  • Takeaways & Limitations

    DailyTalk results support modeling complete multimodal dialogues for retrieving semantic and stylistic conversational references.

  • Takeaways & Limitations

    Evaluation is limited to DailyTalk, a relatively constrained conversational setting, and uses frozen Sentence-BERT and Wav2Vec2-IEMOCAP feature extractors.

Abstract

from arXiv · show

Multimodal dialogue retrieval aims to retrieve dialogues from multimodal dialogue banks that are similar to a target dialogue in terms of both textual semantics and acoustic conversational styles. Such dialogue-level retrieval is crucial for many dialogue-related tasks, including Emotion Recognition in Conversation, Spoken Dialogue Systems, and Conversational Speech Synthesis, where external dialogue examples can provide valuable semantic and stylistic references. However, existing retrieval methods are still largely limited to utterance-level or unimodal matching, and often fail to capture the global semantic coherence and stylistic consistency of an entire dialogue. To address this gap, we propose S2Dialog, a unified framework for dialogue-level semantic-style retrieval from multimodal dialogue banks. Specifically, S2Dialog consists of a Dialogue-level Textual Retriever and a Dialogue-level Acoustic Retriever, which encode the textual and acoustic modalities of a dialogue into dialogue-level representations, respectively. To further enhance multimodal retrieval, we introduce Dialogue-level Textual-Acoustic Contrastive Learning, which aligns semantically and stylistically similar dialogues while distinguishing unrelated ones. Extensive experiments on the multimodal dialogue dataset DailyTalk demonstrate that S2Dialog achieves outstanding retrieval performance.

1. Introduction

S2Dialog addresses limitations in dialogue retrieval by modeling complete multimodal dialogues for semantic-style matching. It combines dedicated textual and acoustic retrievers with contrastive learning and demonstrates effectiveness on DailyTalk.

  • Motivation: Existing dialogue retrieval methods are largely limited to utterance-level or unimodal matching and use aggregated representations not specifically optimized for retrieval.These approaches commonly derive dialogue-level representations by average-pooling utterance-level or unimodal features.
  • Contribution: S2Dialog is a dialogue-level semantic-style retrieval framework for multimodal dialogue banks.The framework is designed to address the limitations of existing retrieval approaches.
  • Method: Its Textual Retriever encodes an entire dialogue into a semantic representation, while its Acoustic Retriever captures full-dialogue conversational style.Both retrievers use dedicated dialogue-level encoders for their respective modalities.
  • Method: Dialogue-level Textual-Acoustic Contrastive Learning jointly models dialogue semantics and conversational styles for more discriminative retrieval.The method is intended to bridge textual and acoustic representations across complete multimodal dialogues.
  • Results: Extensive experiments on DailyTalk demonstrate S2Dialog’s effectiveness over representative retrieval baselines.The experiments evaluate the proposed multimodal dialogue retrieval framework.

2. Related Works

Prior dialogue retrieval methods construct dialogue-level representations by pooling or summarizing dialogue histories, whereas S2Dialog explicitly models complete conversational sequences with dedicated textual and acoustic retrievers. S2Dialog also extends cross-modal contrastive learning to align dialogue-level textual semantics and acoustic conversational styles in a unified retrieval space.

  • Dialogue Retrieval: Existing dialogue retrieval methods mainly build dialogue-level representations by pooling or summarizing dialogue histories, including attention-based pooling and historical-turn incorporation.UniRetriever dynamically aggregates query-relevant utterances through attention-based pooling, while HAConvDR incorporates relevant historical turns.
  • Dialogue Retrieval: S2Dialog uses dedicated textual and acoustic retrievers to explicitly model complete conversational sequences for dialogue-level semantic-style retrieval.This design differs from approaches that construct representations through pooling or summarization over dialogue histories.
  • Contrastive Learning: Contrastive learning improves representation discrimination by pulling positive pairs closer and pushing dissimilar samples apart in a unified embedding space.The paradigm has also been extended to cross-modal representation learning, aligning heterogeneous modalities within a shared latent space.
  • Contrastive Learning: S2Dialog introduces Dialogue-level Textual-Acoustic Contrastive Learning to bridge textual semantics and acoustic conversational styles across complete dialogues.The module optimizes dialogue-level textual and acoustic representations in a unified retrieval space, encouraging cross-modal alignment and improving discriminative capacity for semantic-style retrieval.

3. Methodology

S2Dialog combines dialogue-level textual and acoustic retrievers with textual-acoustic contrastive learning to represent semantic content and conversational style in a unified retrieval space. Its encoders aggregate utterance-level features over temporal dialogue context, while DTACL aligns similar dialogues and separates unrelated ones.

  • Framework overview: S2Dialog comprises the Dialogue-level Textual Retriever, Dialogue-level Acoustic Retriever, and Dialogue-level Textual-Acoustic Contrastive Learning module.DTR and DAR encode textual semantics and acoustic style, respectively; DTACL aligns similar dialogues and separates unrelated ones.
  • Dialogue-level Textual Retriever: DTR uses frozen Sentence-BERT embeddings, a GRU, and an MLP projection head to map variable-length dialogue text into fixed-dimensional semantic representations.Each utterance is represented in 512 dimensions, and the GRU’s final hidden state summarizes the textual discourse before projection.
  • Dialogue-level Acoustic Retriever: DAR uses frozen Wav2Vec2-IEMOCAP features, a GRU, and an MLP projection head to map acoustic sequences into fixed-dimensional dialogue-level style representations.Each speech segment is represented in 768 dimensions, while the GRU captures stylistic consistency and acoustic transitions across the dialogue.
  • Dialogue-level Textual-Acoustic Contrastive Learning: DTACL performs contrastive optimization over complete dialogues by jointly modeling textual and acoustic representations rather than relying on utterance-level or unimodal matching.Its positive and negative sets support textual and acoustic anchors using the target dialogue and similar or dissimilar MDB candidates.
  • Dialogue-level Textual-Acoustic Contrastive Learning: Minimizing the DTACL objective pulls semantically and stylistically similar dialogues together in the shared retrieval space while separating unrelated dialogues.This alignment produces more discriminative dialogue-level representations for semantic-style retrieval from multimodal dialogue banks.

4. Experiments

Experiments evaluate S2Dialog on DailyTalk using dialogue-level ground-truth construction, retrieval and representation-space metrics, representative baselines, and systematic ablations. The setup examines semantic-style retrieval across textual and acoustic modalities and the contribution of each retrieval and contrastive-learning component.

  • Dataset: DailyTalk contains 2,541 conversations and 23,773 audio clips totaling 20 hours, with dialogues averaging 9.356 turns and clips averaging 3.282 seconds.The corpus uses balanced male and female speech, 44.10 kHz sampling, 16-bit encoding, and an 8:1:1 train-validation-test split.
  • Ground-Truth Construction: Ground-truth Top-K and Bottom-K labels are constructed through automated cross-modal scoring followed by human refinement.Dialogue summaries are generated with BART-LARGE-CNN-SAMSUM2 and embedded using Sentence-BERT for dialogue-level textual similarity.
  • Implementation: S2Dialog uses a Sentence-BERT-based textual retriever and a Wav2Vec2-IEMOCAP-based acoustic retriever, each paired with a dialogue-level encoder.The textual encoder uses a 512-dimensional GRU and a 512-to-256-to-256 projection head with ReLU.
  • Evaluation Metrics: Recall@10, Recall@20, Recall@30, Recall@40, and Recall@50 measure consistency between retrieved results and ground-truth Top-K indices at different retrieval depths.TopT@K, BtmT@K, TopA@K, and BtmA@K additionally evaluate cosine similarities in textual and acoustic latent spaces for K∈{1, 25, 50}.
  • Baselines: Baselines span text pooling, multimodal pooling, text summarization, and multimodal summarization methods for dialogue-context modeling.Representative systems include UIMTH, UniRetriever, HAConvDR, MARS, CONVERSE, CORAL, and RADKA-CSS, covering textual, acoustic, semantic, and style-based retrieval strategies.
  • Ablation Studies: Ablations remove the textual retriever, acoustic retriever, textual contrastive loss, acoustic contrastive loss, or Bottom-K dialogues to test their individual contributions.The Bottom-K ablation specifically examines hard-negative mining, discriminative boundaries, and prevention of representation collapse.

5. Results and Discussion

S2Dialog consistently outperforms baseline and ablation variants in multimodal dialogue retrieval, while its textual/acoustic retrievers, contrastive objectives, and Bottom-K negatives improve representation discriminability. Similarity trends and modeling analyses further show sensitivity to semantic, acoustic-style, dialogue-order, and negative-sample structure.

  • Baseline comparison: S2Dialog outperforms all baselines, achieving Recall@10 of 50.68% and Recall@50 of 83.56%.Its dialogue-level textual and acoustic retrievers preserve global semantic coherence and conversational style, while contrastive learning strengthens cross-modal discrimination.
  • Ablation study: Removing the Textual Retriever lowers Recall@10 from 50.68% to 39.60%, while removing the Acoustic Retriever lowers it to 47.60%.The results identify textual semantics as central, with acoustic representations providing complementary style-related information.
  • Ablation study: Excluding Bottom-K samples causes the largest degradation, reducing Recall@10 from 50.68% to 21.12% and Recall@50 from 83.56% to 68.22%.Dissimilar dialogues provide negative supervision that helps separate irrelevant candidates from related ones.
  • Similarity trends: Full-model similarity decreases from 0.771 at TopT@1 to -0.103 at BtmT@1 and from 0.920 at TopA@1 to -0.610 at BtmA@1.The trends indicate discriminative semantic and acoustic-style spaces, whereas removing modality-specific retrievers or negative supervision weakens bottom-rank separation.
  • Dialogue modeling: Random shuffling reduces Recall@10 from 50.68% to 37.66%, while reversed modeling decreases Recall@10 and Recall@20 by 9.99 and 5.00 percentage points, respectively.These results show that retrieval benefits from the natural progression of textual semantics and acoustic dynamics across dialogue turns.
  • Case study: The case study uses a hotel-reservation target dialogue about room types and price confirmation, and its Top-1 and Bottom-1 examples validate semantic relationship capture.The study provides an intuitive scenario-similarity check for retrieved dialogues.

6. Conclusion and Future Work

S2Dialog introduces unified dialogue-level multimodal modeling through textual and acoustic retrievers jointly optimized with contrastive learning. Experiments on DailyTalk show that this combination enhances discrimination between heterogeneous dialogues.

  • Conclusion: S2Dialog shifts multimodal retrieval from utterance-level aggregation to holistic dialogue-level modeling.The framework represents entire dialogues rather than relying on conventional utterance-level aggregation.
  • Conclusion: The framework combines dialogue-level textual and acoustic retrievers with dialogue-level textual-acoustic contrastive learning.The two retrievers are jointly optimized to align the multimodal dialogue representations.
  • Conclusion: Experiments on the DailyTalk corpus show that this synergy enhances discriminative power between heterogeneous dialogues.The reported experimental validation attributes the improvement to the combination of dialogue-level retrieval and contrastive learning.

7. Limitations

S2Dialog is currently evaluated only on DailyTalk, a relatively constrained conversational setting. Future work will assess it on larger, more diverse multimodal dialogue corpora.

  • Evaluation scope: S2Dialog is currently evaluated only on the DailyTalk dataset, limiting validation to a relatively constrained conversational setting.DailyTalk provides high-quality multimodal dialogue data with aligned textual and acoustic information.
  • Future directions: Future work will extend evaluation to larger, more diverse multimodal dialogue corpora with more speakers and broader dialogue scenarios.
Loading 2608.14029v1…