PaperlayerView exact revision on arXiv

Source-linked AI summary

Response Ranking with Deep Matching Networks and External Knowledge in Information-seeking Conversation Systems

Liu Yang, Minghui Qiu, Chen Qu, Jiafeng Guo, Yongfeng Zhang, W. Bruce Croft, Jun Huang, Haiqing Chen

arXiv:1805.00188v3cs.IRcs.CL

TL;DR

Information-seeking conversation response ranking remains underexplored, particularly when models ignore knowledge beyond dialog utterances. The paper integrates external knowledge into deep matching networks using pseudo-relevance feedback and QA correspondence distillation, and reports superior response-selection performance across three datasets and multiple baselines.

  • Problem

    Response selection for information-seeking conversations is underexplored, and existing models often lack external knowledge beyond dialog utterances.

  • Method

    The paper integrates external knowledge into deep neural matching networks through pseudo-relevance feedback and QA correspondence knowledge distillation.

  • Results

    Across three information-seeking conversation datasets, the methods outperform various baselines, including state-of-the-art methods, on response selection.

  • Takeaways & Limitations

    The findings provide insights into using external knowledge with deep neural models for response selection in information-seeking conversations.

Abstract

from arXiv · show

Intelligent personal assistant systems with either text-based or voice-based conversational interfaces are becoming increasingly popular around the world. Retrieval-based conversation models have the advantages of returning fluent and informative responses. Most existing studies in this area are on open domain "chit-chat" conversations or task / transaction oriented conversations. More research is needed for information-seeking conversations. There is also a lack of modeling external knowledge beyond the dialog utterances among current conversational models. In this paper, we propose a learning framework on the top of deep neural matching networks that leverages external knowledge for response ranking in information-seeking conversation systems. We incorporate external knowledge into deep neural models with pseudo-relevance feedback and QA correspondence knowledge distillation. Extensive experiments with three information-seeking conversation data sets including both open benchmarks and commercial data show that, our methods outperform various baseline methods including several deep text matching models and the state-of-the-art method on response selection in multi-turn conversations. We also perform analysis over different response types, model variations and ranking examples. Our models and research findings provide new insights on how to utilize external knowledge with deep neural models for response selection and have implications for the design of the next generation of information-seeking conversation systems.

1 INTRODUCTION

The paper targets response selection in information-seeking conversations, where existing work underrepresents the setting and usually ignores external knowledge. It proposes deep matching methods that incorporate external evidence and evaluates them on benchmark and commercial datasets.

  • Motivation: The authors address the underexplored problem of response selection in information-seeking conversations, distinct from chit-chat and transaction-oriented dialogue.Information-seeking conversations aim to satisfy users’ information needs through interaction.
  • Motivation: Existing response-selection models often use only dialog utterances, although external collections may provide knowledge needed to distinguish responses across diverse topics.The paper identifies insufficient contextual signals as a particular challenge for information-seeking conversations.
  • Approach: The proposed framework integrates external knowledge into deep neural matching networks through pseudo-relevance feedback and QA correspondence knowledge distillation.Pseudo-relevance feedback retrieves external documents using a candidate response as a query, while correspondence knowledge captures match patterns from retrieved QA pairs.
  • Evaluation: The study evaluates response ranking on MSDialog, Ubuntu Dialog Corpus, and Alibaba’s commercial AliMe dataset.The datasets include both open benchmarks and commercial customer-service conversations.
  • Results: The methods outperform various deep text-matching baselines and the state-of-the-art method across multiple metrics for multi-turn response selection.The paper also analyzes response types, model variations, and ranking examples.
  • Contributions: MSDialog is introduced as a released benchmark for information-seeking technical-support conversations.It is constructed from Microsoft product support dialogs and released to the research community.

2 RELATED WORK

Prior work spans conversational search, neural conversational models, and neural ranking models, but this paper focuses on retrieval-based multi-turn response selection with external knowledge beyond dialog context.

  • Conversational Search: Conversational-search research includes systems designed for information-seeking interaction, but existing datasets may be too small for training neural models.The related work discusses Microsoft Information-Seeking Conversation data and its limited dialog volume.
  • Neural Conversational Models: Neural conversational research includes retrieval-based and generation-based methods, while this paper uses retrieval-based response ranking.The paper distinguishes its approach from prior multi-turn matching work by adding external knowledge beyond the dialog context.
  • Neural Conversational Models: The paper’s framework extends multi-turn matching by incorporating external QA correspondence signals alongside term- and sequence-level matching.The related-work comparison identifies external knowledge beyond dialog context as the main distinction from prior sequential matching networks.
  • Neural Ranking Models: Neural ranking models are categorized as representation-focused, interaction-focused, or hybrid models that combine matching signals.Interaction-focused models construct query-document interaction matrices for exact and semantic matching.

3.1 Problem Formulation

The task is to learn a ranking model that orders candidate responses for an information-seeking dialog using labeled conversations and a related external text collection.

  • Problem Definition: Each conversation consists of a dialog context, response candidates, and labels identifying the true response.The context includes the input utterance and preceding dialog turns.
  • Problem Definition: The model must generate a ranked list of candidate responses for any given dialog context.The ranking function is learned from the conversation dataset and external collection.
  • External Collection: External knowledge is represented by a collection related to the conversation topics, such as historical QA posts from Stack Overflow or AskUbuntu.The external collection may be any massive text corpus, with domain-specific QA dumps used in the paper.

3.2 Method Overview

The framework combines information retrieval, external knowledge extraction, and deep matching to score conversation response candidates. It has two implementations: DMN-PRF and DMN-KD.

  • Method Overview: The external collection statistics concern QA-pair retrieval and knowledge extraction, including questions with accepted answers.Table 3 defines “#QWithAcceptedA” as the number of questions with an accepted answer.
  • Method Overview: The framework retrieves QA pairs, extracts external knowledge, and models that knowledge with dialog utterances and response candidates to predict matching scores.The three modules are information retrieval, knowledge extraction, and deep matching.
  • Method Overview: The authors implement external-knowledge integration through pseudo-relevance feedback in DMN-PRF and QA correspondence knowledge distillation in DMN-KD.The two architectures are presented as distinct implementations of the framework.

3.3 Deep Matching Networks with Pseudo-Relevance Feedback

DMN-PRF expands candidate responses with terms learned from retrieved QA posts, then applies interaction-focused deep matching to rank responses. The network combines word-level and sequence-level similarities with CNN, BiGRU, and MLP components.

  • Pseudo-Relevance Feedback: DMN-PRF retrieves top QA posts with BM25 using each response candidate as a query, then uses them as external knowledge.The experiments use Stack Overflow for MSDialog and AskUbuntu for UDC; the number of retrieved posts is set to 10.
  • Pseudo-Relevance Feedback: The model computes a language model from retrieved QA posts, extracts frequent expansion terms, and appends them to each response candidate.The expansion uses W = 10 terms in the experiments.
  • Interaction Matching Matrix: The expanded responses and dialog utterances are represented as embedding sequences and matched with interaction-focused methods.The model builds a word pairwise similarity matrix M1 and a sequence hidden representation similarity matrix M2.
  • Interaction Matching Matrix: M1 captures word-level similarity, while BiGRU-based M2 captures local sequence structures such as phrases or text segments.The two matrices provide separate input channels for the convolutional neural network.
  • Convolution and Pooling Layers: CNN convolution and max-pooling operations transform the interaction matrices into high-level matching features for later aggregation.The resulting features are processed by a final BiGRU and MLP to generate the matching score.
  • Training: The model learns response rankings with pairwise hinge loss over positive and negative candidates and optimizes parameters using back-propagation with Adam.The loss includes an L2 regularization term and a hinge-loss margin.

Correspondence Knowledge Distillation

DMN-KD augments response-ranking networks with QA correspondence knowledge distilled from externally retrieved question–answer pairs. It encodes relationships between utterance and response terms in an additional interaction channel.

  • Correspondence Knowledge Distillation: DMN-KD adds an external QA correspondence channel, M3, to deep matching networks for conversation response ranking.M3 captures matching patterns between utterance and response terms in relevant retrieved QA pairs.
  • Correspondence Knowledge Distillation: The model retrieves relevant QA pairs using the response candidate and extracts correspondence information from their question–answer term co-occurrences.The retrieved pair set is used to construct QA term correspondence statistics for each candidate response and dialog utterance.
  • Correspondence Knowledge Distillation: QA pairs are used instead of isolated question or answer posts because the method requires question–answer term co-occurrence information.The paper explicitly motivates retaining the paired structure during extraction.
  • Correspondence Knowledge Distillation: M3 uses PPMI statistics to represent co-occurrence relationships between response words and dialog-utterance words in retrieved QA pairs.The matrix element m3,i,j is computed from word co-occurrence information derived from the external QA collection.
  • Correspondence Knowledge Distillation: The extracted knowledge captures patterns such as problem descriptions with solutions, symptoms with causes, and information requests with answers.These correspondence patterns are intended to help distinguish good response candidates from bad ones.

4 EXPERIMENTS

The experiments evaluate external-knowledge response-ranking methods against retrieval, deep matching, and multi-turn conversation baselines on three datasets. They use standard ranking metrics and tuned neural implementations.

  • Datasets: The evaluation covers UDC, MSDialog, and AliMe, combining an open technical-support benchmark with Microsoft and Alibaba customer-service conversations.The datasets represent both benchmark and commercial information-seeking conversation settings.
  • Data Construction: Negative responses for MSDialog are sampled from the top 1,000 BM25-retrieved agent responses, while the true agent response is positive.The MSDialog data is split chronologically into training, validation, and testing partitions.
  • Model Settings: DMN-PRF and DMN-KD are implemented with tuned hyperparameters, while DMN removes both external-knowledge mechanisms and BM25-PRF applies expansion within BM25.The ablated DMN provides a no-external-knowledge comparison.
  • Evaluation Metrics: The experiments measure MAP, Recall@1, Recall@2, and Recall@5; MAP equals MRR for UDC and MSDialog because each context has one positive response.AliMe contexts may contain multiple positive response candidates.

4.3 Evaluation Results

Across UDC, MSDialog, and AliMe, the proposed external-knowledge models generally outperform or match baseline response-ranking systems. Additional analysis examines response types and reports how the models compare on MSDialog.

  • Performance Comparison on UDC and MSDialog: DMN-PRF and DMN-KD outperform all baselines on UDC and MSDialog, with differences statistically significant at p < 0.05.The comparison includes traditional retrieval, deep text matching, and the state-of-the-art SMN model.
  • Performance Comparison on AliMe: On AliMe, DMN-PRF significantly outperforms competing baselines, while DMN-KD achieves comparable or better results than all baselines.Both external-knowledge models also outperform the degenerated DMN model.
  • Performance Comparison over Different Response Types: The MSDialog analysis evaluates model performance across annotated response types, including questions, answers, gratitude, and feedback.The response types are derived from 12 user-intent categories classified with TF-IDF features and a Random Forest model.

4.4 Model Ablation Analysis

Ablation results show that the interaction channels contribute to response selection, while context length has an intermediate optimum because short contexts omit information and long contexts introduce noise. A case study illustrates how external QA terms can promote a correct response.

  • 4.4 Model Ablation Analysis: Removing any interaction channel reduces performance: M1 and M2 matter for DMN-PRF, while M1, M2, and M3 matter for DMN-KD.The result supports the usefulness of word-level, sequence-level, and external QA correspondence matching.
  • 4.4 Model Ablation Analysis: Dot-product interaction is generally best, cosine similarity is next, and bilinear interaction performs worst, especially on MSDialog.The paper attributes bilinear degradation to its additional transformation matrix and greater overfitting risk on the smaller dataset.
  • 4.5 Impact of Conversation Context Length: Performance first increases and then decreases as conversation context length grows.Short contexts lack matching evidence, whereas overly long contexts can add topic-shift noise from older utterances.
  • Case Study: In a ranking example, DMN-KD and DMN-PRF place the correct response first, with retrieved terms such as “excel” helping despite its absence from the raw response.The example illustrates how external QA information can connect repeated context terms to the correct candidate.

5 CONCLUSIONS AND FUTURE WORK

The paper proposes deep matching networks that leverage external knowledge for response ranking in information-seeking conversations, with experiments showing improvements over baseline methods.

  • The proposed methods outperform various baselines, including state-of-the-art methods, on open benchmarks and commercial data.
  • External knowledge is incorporated through pseudo-relevance feedback and QA correspondence knowledge distillation.
  • The analysis examines different response types and model variations to provide insights into model applications.
Loading 1805.00188v3…