Source-linked AI summary
Neural Matching Models for Question Retrieval and Next Question Prediction in Conversation
Liu Yang, Hamed Zamani, Yongfeng Zhang, Jiafeng Guo, W. Bruce Croft
TL;DR
The paper addresses the limited study of neural models for question retrieval and introduces predicting a user’s next question from conversational context. It evaluates neural matching models on retrieval and conversation prediction, reporting strong performance on both tasks.
Problem
Neural matching models had been relatively unstudied for question retrieval, while predicting the next question from conversational context was introduced as a new task.
Source: §1 · pp. 1–2Method
The paper evaluates neural matching models that rank candidate sequences against a query, using a siamese LSTM-CNN architecture for question retrieval and conversational next-question prediction.
Source: §2.1 · p. 2Results
Takeaways & Limitations
The experiments demonstrate the potential of neural matching with representation learning for question retrieval and next-question prediction in conversations.
Source: §1 · p. 2Next-question prediction is a preliminary, more challenging task because conversational contexts are longer than candidate questions and their matching patterns can be more complex.
Source: §3.4 · p. 5
Abstract
from arXiv · showhide
The recent boom of AI has seen the emergence of many human-computer conversation systems such as Google Assistant, Microsoft Cortana, Amazon Echo and Apple Siri. We introduce and formalize the task of predicting questions in conversations, where the goal is to predict the new question that the user will ask, given the past conversational context. This task can be modeled as a "sequence matching" problem, where two sequences are given and the aim is to learn a model that maps any pair of sequences to a matching probability. Neural matching models, which adopt deep neural networks to learn sequence representations and matching scores, have attracted immense research interests of information retrieval and natural language processing communities. In this paper, we first study neural matching models for the question retrieval task that has been widely explored in the literature, whereas the effectiveness of neural models for this task is relatively unstudied. We further evaluate the neural matching models in the next question prediction task in conversations. We have used the publicly available Quora data and Ubuntu chat logs in our experiments. Our evaluations investigate the potential of neural matching models with representation learning for question retrieval and next question prediction in conversations. Experimental results show that neural matching models perform well for both tasks.
1 INTRODUCTION
The paper studies neural matching models for retrieving similar questions and predicting the next question in a conversation. It formalizes next-question prediction and evaluates these models on Quora question pairs and Ubuntu chat logs.
1 INTRODUCTION
- Neural models for question retrieval were relatively unstudied, despite similar-question retrieval being useful for question answering and avoiding duplicate questions in community Q&A.The paper studies neural networks for retrieving questions similar to a given question.Source: §1 · p. 1
- The paper introduces next-question prediction: predicting what question a user will ask from the preceding conversational context.The authors describe this as a new conversational response-ranking task.Source: §1 · p. 2
- The authors motivate conversational question prediction as a route to proactive recommendations and pre-selected answer sets for question answering systems.They also frame it as a way to better understand user intents during conversations.Source: §1 · p. 2
- The proposed siamese network uses LSTM representations followed by convolutional and max-pooling layers to score candidate questions for ranking.The architecture is applied to both question retrieval and next-question prediction.Source: §1 · p. 2
- Neural matching models outperform state-of-the-art non-neural approaches on question retrieval, and combining neural and basic retrieval scores provides further gains; the models also perform well at next-question prediction.The experiments use Quora data for retrieval and Ubuntu IRC logs for conversation prediction.Source: §1 · p. 2
2 NEURAL MATCHING MODEL
The model ranks candidate sequences against a query sequence, using LSTM-based contextual representations followed by convolution and pooling. It is trained with a pairwise ranking hinge loss.
2.1 Problem Definition and Model Overview
- For retrieval, the query is a question; for conversation prediction, it is prior questions and responses, with candidate next questions ranked against it.Each sequence pair is associated with a label.Source: §2.1 · p. 2
2.2 LSTM for Long Term Dependency Modeling
- The LSTM processes both sequences to model long-term dependencies, producing contextual representations; the model uses a unidirectional LSTM because Bi-LSTM did not improve performance and reduced training efficiency.Each hidden representation encodes its word with preceding context; the gates control information flow through the memory cell.Source: §2.2 · pp. 2–3
2.3 Convolutional and Max Pooling Layers
- Convolutional layers with different filter and pooling-window sizes learn sequence representations from the LSTM outputs for matching-score generation.The description specifies narrow convolution, ReLU, max pooling, and a final fully connected softmax layer.Source: §2.3 · p. 3
2.4 Loss Function and Training
- Training uses triples containing a query, positive candidate, and negative candidate in a pairwise ranking hinge loss, with regularization; parameters are optimized using Adam.The model’s matching score is used in the loss, which includes a margin term.Source: §2.4 · p. 3
3 EXPERIMENTS
Experiments evaluate neural matching on Quora question retrieval and Ubuntu next-question prediction using sampled candidate sets and a range of retrieval and neural ranking methods. Results report strong question-retrieval performance and improved conversation ranking, while identifying conversation prediction as the harder setting.
3.1 Datasets and Experimental Setting
- Quora positives came from similar question pairs with four BM25-retrieved negatives per query, while Ubuntu contexts merged prior question-response turns and used nine random negatives; both datasets used 8:1:1 splits.The Quora dataset contains 148,487 similar question pairs.Source: §3.1 · p. 3
3.2 Evaluation Metrics and Compared Methods
- Compared systems include lexical retrieval baselines, translation-based TRLM, averaged embeddings, a CNN-Match ablation, and LambdaMART combining baseline and neural scores.Source: §3.2 · p. 4
3.3 Experimental Results on Qestion Retrieval
- LSTM-CNN-Match outperformed basic retrieval, translation-model, and basic neural or word-embedding baselines; combining neural and basic retrieval scores yielded further gains.The reported comparisons support combining traditional IR and neural features in a learning-to-rank framework.Source: §3.2 · p. 4, Table 4 · p. 4, §3.3 · p. 5
- Removing the LSTM layer reduced retrieval performance, while an anime-query example shows LSTM-CNN-Match ranking the semantically similar “shows” question first where term-matching systems miss it.Source: §3.3 · p. 5, Table 5 · p. 5
3.4 Experimental Results on Predicting Qestions in Conversations
- On Ubuntu next-question prediction, the Combined Model led MRR and P@1, CNN-Match led P@5 and Recall@5, and neural matching improved ranking over traditional retrieval.LSTM-CNN-Match performed worse than CNN-Match on this task.Source: §3.4 · p. 5, Table 6 · p. 5
- Conversation prediction was more challenging because contexts are longer than candidate questions and matching patterns exceed semantic matching or paraphrase; attention, sequence-to-sequence, and memory networks are proposed directions.Source: §3.4 · p. 5
4 RELATED WORK
Prior work on question retrieval includes translation-based and category-based approaches, while neural conversation research has largely focused on response generation and ranking. Neural ranking models are described as representation-focused or interaction-focused, with some also combining exact and semantic matching.
4.1 Qestion Retrieval
- Question retrieval research includes translation-based models for lexical gaps and category-based methods that use question categories.Category-based approaches may not apply when new questions lack predefined categories, the setting this work addresses.Source: §4.1 · pp. 5–6
4.2 Neural Conversation Models
- Neural conversation models have addressed response generation and ranking, whereas this work focuses on finding questions from prior conversational context.Prior examples include encoder-decoder, context-sensitive, retrieval-based, and reinforcement-learning approaches.Source: §4.2 · p. 6
4.3 Neural Ranking Models
- Neural ranking models are grouped into representation-focused models that compare independent representations and interaction-focused models that capture query-document interactions.Some approaches aim to capture both exact term matching and semantic matching; weak supervision can also train models from retrieval-generated data.Source: §4.3 · p. 6
5 CONCLUSIONS AND FUTURE WORK
Neural matching models outperform baselines for question retrieval, with larger gains when combined with term-matching retrieval. For conversation question prediction, removing the LSTM performs best among the tested methods; the study is preliminary and points to more advanced models as future work.
5 CONCLUSIONS AND FUTURE WORK
- Neural matching models significantly outperform all baselines for question retrieval, and combining them with basic term-matching retrieval yields larger gains.For conversation question prediction, a simpler model without LSTM outperforms the other tested methods because LSTM layers cannot handle long question histories.Source: §5 · p. 6
- The study is preliminary; future work could explore context attention and external knowledge to improve candidate-question finding.The authors also identify developing more advanced models for matching conversational context with questions as an open direction.Source: §5 · p. 6