Source-linked AI summary
Sequential Matching Network: A New Architecture for Multi-turn Response Selection in Retrieval-based Chatbots
Yu Wu, Wei Wu, Chen Xing, Ming Zhou, Zhoujun Li
TL;DR
Multi-turn retrieval-based response selection must account for both relationships among ordered utterances and important contextual information. SMN matches responses with individual utterances at multiple granularities, accumulates their matching vectors chronologically, and significantly outperforms state-of-the-art methods on two datasets.
Problem
Multi-turn response selection must choose a response relevant to the current message and previous utterances while preserving important contextual information and utterance relationships.
Method
SMN matches each response with every context utterance at word and segment levels, distills matching vectors, and accumulates them in chronological order with a recurrent network.
Results
SMN significantly outperforms state-of-the-art methods, improving the best baseline by over 6% on Ubuntu R10@1, 3% on Douban R10@1, and 4% on Douban P@1.
Takeaways & Limitations
SMN provides a context-based matching model for multi-turn response selection, alongside a human-labeled dataset for research in this setting.
Takeaways & Limitations
SMN pays little attention to logical consistency, and candidate retrieval produced correct candidates for only 667 of 1000 test contexts.
Abstract
from arXiv · showhide
We study response selection for multi-turn conversation in retrieval-based chatbots. Existing work either concatenates utterances in context or matches a response with a highly abstract context vector finally, which may lose relationships among utterances or important contextual information. We propose a sequential matching network (SMN) to address both problems. SMN first matches a response with each utterance in the context on multiple levels of granularity, and distills important matching information from each pair as a vector with convolution and pooling operations. The vectors are then accumulated in a chronological order through a recurrent neural network (RNN) which models relationships among utterances. The final matching score is calculated with the hidden states of the RNN. An empirical study on two public data sets shows that SMN can significantly outperform state-of-the-art methods for response selection in multi-turn conversation.
1 Introduction
Multi-turn response selection must match a candidate response with the entire ordered conversation while preserving important information at multiple granularities. SMN addresses these challenges by matching each utterance-response pair, accumulating the resulting vectors chronologically, and outperforming established methods on two datasets.
- Problem: Multi-turn response selection chooses a response relevant to the current message and its previous-turn utterances.
- Challenges: The task requires identifying important words, phrases, and sentences while modeling relationships among context utterances.
- Limitations of Existing Work: Existing approaches either concatenate utterances and ignore their relationships or compress context into a vector that can lose important information.
- SMN: SMN matches each response with every context utterance, extracts word- and segment-level matching information, and accumulates matching vectors in temporal order.It uses similarity matrices and recurrent modeling to preserve pairwise information and utterance relationships.
- Results: Over 6% improvement on Ubuntu and 3% on Douban R10@1, plus 4% on Douban P@1, is reported over the best baseline.The paper also introduces the human-labeled Douban Conversation Corpus and reports empirical verification on public datasets.
2 Related Work
Related work includes retrieval-based and generation-based chatbot systems, with earlier retrieval research focused mainly on single-turn response selection. Later multi-turn methods concatenate context utterances or use multi-view modeling.
- Chatbot Methods: Data-driven chatbot research includes both retrieval-based and generation-based methods.
- Response Selection: Early retrieval-based studies focused on single-turn response selection, whereas later work addressed multi-turn conversation.
- Multi-turn Methods: Multi-turn methods included literal context concatenation, query reformulation with the input message, and multi-view modeling.
3 Sequential Matching Network
SMN performs response matching separately for each context utterance at word and segment levels, then accumulates the resulting matching vectors sequentially to model utterance relationships and compute a final score.
- Problem Formalization: Training learns the context-response matching function by minimizing cross entropy over labeled context-response pairs.A positive label denotes a proper response candidate and a negative label denotes an improper one.
- Model Overview: SMN decomposes context-response matching into utterance-response pairs and processes them sequentially with recurrent accumulation.The model uses three layers: pair matching, recurrent matching accumulation, and final score calculation.
- Model Advantages: SMN couples matching with utterance relationships so the response supervises information extraction from each utterance and utterance order contributes to the matching score.This design aims to preserve useful information from each utterance-response pair while modeling contextual order.
- Utterance-Response Matching: Word-level and segment-level similarity matrices provide two input channels whose important matching patterns are distilled by a CNN into a matching vector.The word matrix models word-level matching, while the segment matrix uses GRU-derived hidden vectors to model segment-level matching.
- Matching Accumulation: A GRU encodes matching vectors in chronological order, modeling dependencies among context utterances while filtering noise through reset and update gates.The recurrent layer uses temporal relationships to supervise how pairwise matching information is accumulated.
- Matching Prediction: The model supports three hidden-state aggregation variants—SMNlast, SMNstatic, and SMNdynamic—for empirical comparison.They use the last hidden state, a static linear combination, or an attention-based combination, respectively.
4 Response Candidate Retrieval
The retrieval procedure expands the current message with five tf-idf-selected keywords from previous utterances before retrieving response candidates.
- Response Candidate Retrieval: Candidate retrieval extracts the top 5 keywords from previous-turn utterances using tf-idf and appends them to the current message.The expanded message is then sent to the retrieval index to obtain response candidates.
5 Experiments
Experiments evaluate SMN on the Ubuntu and Douban corpora against multiple baseline families, then analyze its matching behavior, ablations, context-length effects, and limitations.
- Datasets: SMN is evaluated on Ubuntu and Douban, including a human-labeled open-domain test setting designed to simulate retrieval-based chatbot response selection.Ubuntu uses randomly sampled negatives, whereas Douban retrieves candidates and has human judgments.
- Baselines: The baseline suite includes basic neural and lexical models, Multi-View, DL2R, and advanced single-turn matching models applied to concatenated contexts.The comparison includes TF-IDF, RNN, CNN, LSTM, BiLSTM, MV-LSTM, Match-LSTM, Attentive-LSTM, and Multi-Channel.
- Evaluation Results: SMN outperforms baselines on all metrics across both data sets, with statistically significant improvements except R10@5 on Douban Corpus.The reported significance test uses a t-test with p-value ≤0.01.
- Evaluation Results: The results indicate that preserving utterance relationships and matching responses with utterances before accumulation is more effective than concatenating contexts or reformulating them heuristically.SMN significantly improves over Multi-View, while DL2R performs worse than SMN.
- Further Analysis: The model visualization shows that matching focuses on informative words in crucial utterances while GRU gates preserve their information through later turns.In the example, u1 contributes terms such as “unzip,” “rar,” and “files,” while u3 contributes little information.
- Further Analysis: Ablation results show that multi-channel 2D matching is especially important, while replacing GRU accumulation also reduces performance and using both matching channels performs best in most settings.Replacing 2D matching with an NTN causes a dramatic drop; replacing GRU with an MLP causes a smaller drop.
- Further Analysis: SMN remains better than baselines as contexts lengthen, with the performance gap increasing for longer contexts, indicating capture of long-range utterance dependencies.The context-length analysis is reported on MAP intervals from the Douban corpus.
- Limitations: Two limitations remain: SMN handles logical consistency imperfectly, and retrieval produces correct candidates for only 667 of 1,000 test contexts.The paper identifies both logical-consistency modeling and candidate retrieval as future-work areas.
6 Conclusion and Future Work
The paper presents a context-based model for multi-turn response selection and reports significant improvements over state-of-the-art methods on open datasets. It also publishes the first human-labeled multi-turn response-selection dataset and identifies logical consistency and candidate retrieval as future work.
- The paper presents a new context-based model for multi-turn response selection in retrieval-based chatbots.
- The model significantly outperforms state-of-the-art methods on open datasets.
- The authors publish the first human-labeled multi-turn response-selection dataset for the research community.
- Future work will address logical consistency in responses and improve candidate retrieval.