Source-linked AI summary
Deep Learning for Answer Sentence Selection
Lei Yu, Karl Moritz Hermann, Phil Blunsom, Stephen Pulman
TL;DR
Answer sentence selection asks which candidate sentences contain the answer to a question, an important component of open-domain question answering. The paper learns semantic question-answer matching with distributed sentence models and lightweight word-matching features, achieving state-of-the-art performance on TREC QA while avoiding extensive feature engineering and hand-coded resources.
Problem
Answer sentence selection must identify sentences containing answers to factual questions, while prior systems typically rely on extensive hand-crafted syntactic and semantic features and external resources.
Method
The paper learns semantic question-answer matching with bag-of-words and convolutional bigram sentence models, optionally combining the distributed signal with two simple word-matching features.
Results
The convolutional bigram model attains state-of-the-art performance on answer sentence selection, while distributional semantics boost MAP and MRR by approximately 10% over count-based models.
Takeaways & Limitations
The approach offers a simpler, more flexible alternative to feature-engineered systems and can be applied across languages using only a large corpus for initial word embeddings.
Takeaways & Limitations
Distributed representations are weak at handling cardinal numbers and proper nouns, so the model adds question-answer word-cooccurrence features to mitigate this issue.
Abstract
from arXiv · showhide
Answer sentence selection is the task of identifying sentences that contain the answer to a given question. This is an important problem in its own right as well as in the larger context of open domain question answering. We propose a novel approach to solving this task via means of distributed representations, and learn to match questions with answers by considering their semantic encoding. This contrasts prior work on this task, which typically relies on classifiers with large numbers of hand-crafted syntactic and semantic features and various external resources. Our approach does not require any feature engineering nor does it involve specialist linguistic data, making this model easily applicable to a wide range of domains and languages. Experimental results on a standard benchmark dataset from TREC demonstrate that---despite its simplicity---our model matches state of the art performance on the answer sentence selection task.
1 Introduction
Answer sentence selection identifies candidate sentences that answer factual questions and supports open-domain question answering as well as related applications. The paper replaces feature-heavy semantic matching with neural distributed sentence models and reports state-of-the-art performance on TREC QA.
- Problem: Answer sentence selection selects correct sentences answering factual questions from candidate sentences within open-domain question answering.It is also a standalone task with applications in knowledge base construction and information extraction.
- Prior limitations: Prior approaches combine syntactic matching with semantic features, but feature-based semantic models require extensive engineering and costly resources that are difficult to obtain for low-resource languages.Such models are also difficult to adapt across domains because feature extraction and resource development must be repeated.
- Task setting: The task differs from prior fixed-candidate question answering because candidate sentences are unseen during training and each question can have a different number of candidates.The model therefore addresses selection from variable candidate sets rather than a fixed set encountered during training.
- Approach: The paper learns semantic matching between questions and answers using bag-of-words and convolutional bigram sentence models over pretrained word embeddings.An enhanced model combines distributed matching with two word-matching features without external linguistic annotation.
- Results: The models match state-of-the-art results on a standard answer selection dataset created from the TREC QA track.The approach is presented as applicable to any language using a large corpus for initial word embeddings, without hand-coded resources beyond that corpus.
2 Related work
Related work spans distributed representations, sentence composition, answer sentence selection, and early neural approaches to question answering. Prior answer-selection systems largely emphasize syntactic or resource-intensive lexical-semantic matching, while neural work often uses different task settings.
- Compositional distributional semantics: Distributed word representations capture latent semantic information and word similarities, helping address sparsity in atomic representations.More complex NLP tasks require distributed representations of phrases and sentences, motivating composition models.
- Compositional distributional semantics: Composition methods build phrase- and sentence-level vectors from word representations using approaches including category theory, recursive autoencoders, and convolutional neural networks.These methods address the difficulty of directly learning distributional representations at the phrase level because of sparsity.
- Answer sentence selection: Answer sentence selection chooses a sentence containing the information needed to answer a question from candidates retrieved by an information extraction system.The task requires semantic and syntactic information to determine both the question's target and whether a candidate contains it.
- Answer sentence selection: Earlier answer-selection systems use syntactic transformations, dependency-tree edit features, or lexical-semantic resources such as WordNet within conventional classifiers.Yih et al.'s models combine WordNet with distributional representations to match semantic relations of aligned question-answer words.
- Applying neural networks to question answering: Early neural question-answering research includes knowledge-base models, joint question-answer projection networks, and recursive networks for quiz-bowl question answering.The quiz-bowl model selects from a relatively small fixed candidate set seen during training, unlike answer sentence selection with unseen candidates and variable candidate counts.
3 Model description
The paper models questions and candidate answers as vectors and learns their semantic relatedness, using bag-of-words and convolutional bigram sentence representations. An enhanced model adds word-overlap signals to address weaknesses of distributed representations.
- Answer sentence selection is treated as binary classification over question–candidate-answer–label triples.A label yij equals 1 for a correct answer and 0 otherwise.
- The model estimates answer correctness from question–answer relatedness in a shared vector space.A transformation matrix maps the answer representation toward a generated question, whose dot product with the given question is converted to a probability.
- The model parameters include the question–answer transformation, bias, and parameters introduced by sentence composition models.
- 3.1 Bag-of-words model: The bag-of-words model sums non-stop-word embeddings and normalises the resulting sentence vector by sentence length.
- 3.2 Bigram model: The CNN model addresses bag-of-words limitations by capturing word order, position-independent n-grams, internal syntactic structure, and long-range dependencies.Convolution and pooling provide sentence-level representations without relying on external parse trees.
- 3.2 Bigram model: The bigram model applies a nonlinear transformation to adjacent word vectors and uses average pooling to produce a full-sentence representation.The resulting sentence vector has the same dimensionality as the initial word embeddings.
4 Experiments
The models are evaluated on a TREC-derived answer sentence selection dataset using ranking metrics, with an enhanced system combining distributional scores and overlap features. The bigram model and added IDF-weighted counts improve results, and the best models outperform listed baselines and prior work on MAP while nearly matching the best MRR model.
- 4.1 TREC Answer Selection Dataset: The evaluation uses a TREC QA-derived dataset of factoid questions paired with candidate answer sentences, whose correctness was judged automatically or manually.
- 4.1 TREC Answer Selection Dataset: Candidate answers are ranked by relatedness to questions and evaluated with Mean Average Precision (MAP) and Mean Reciprocal Rank (MRR).MAP considers all correct-answer ranks, whereas MRR considers the rank of any correct answer.
- 4.2 Experimental setup: The enhanced classifier combines word co-occurrence count, IDF-weighted co-occurrence count, and the distributional model’s QA matching probability.This feature combination is motivated by distributed representations’ weakness with cardinal numbers and proper nouns.
- 4.3 Results: 10%–15%: adding IDF-weighted word-count features significantly improves performance for both unigram and bigram models.The bigram model also performs better than the unigram model.
- 4.3 Results: The best bigram-plus-count models outperform all baselines and prior work on MAP and come very close to Yih et al.’s best model on MRR.These results are reported despite the models’ lower complexity relative to Yih et al.’s models.
5 Discussion
The discussion shows that distributional semantics improve answer selection while reducing reliance on hand-engineered linguistic resources, but current models remain limited on numbers, proper nouns, and background knowledge.
- Results: Our best bigram + count models outperform all baselines and prior work on MAP and nearly match Yih et al. on MRR.Table 3 places these models alongside published results and identifies them as matching the current state of the art.
- Comparison with prior work: The approach is simpler and more portable than feature-based semantic models because it uses only word matching, IDF-weighted matching, and distributional representations.Unlike Yih et al.’s model, it does not depend on external linguistic resources beyond a large corpus for initial word embeddings and can be applied across languages.
- Limitations: Pure vector representations struggle with numbers and proper nouns, especially when out-of-vocabulary items are mapped to UNKNOWN.Approximately 5% of answer-selection-dataset words are absent from the pretrained embeddings, and surface-form matching is therefore important.
- Results: Adding distributional semantics boosts both MAP and MRR by approximately 10% over purely count-based and word-matching models.The combined model succeeds when semantic relations such as die–killed and performers–singing group distinguish candidates with equal word-match counts.
- Limitations: Some cases require understanding a sentence and background knowledge, such as recognizing the relation between a lead singer and the singer’s group.The Limp Bizkit example illustrates a limitation that distributional matching alone does not resolve.
6 Conclusion
The paper applies distributional sentence models to answer sentence selection by learning semantic matching between questions and answers and combining it with weighted co-occurrence counts. The bag-of-words model improves over the count-based baseline, while the convolutional bigram model improves further and reaches state-of-the-art performance.
- Method: The paper projects questions and answers into vectors, learns a semantic matching function, and combines it with a simple weighted QA co-occurrence counter.The approach uses distributional sentence models to represent question–answer pairs before combining semantic and count-based signals.
- Results: The bag-of-words sentence model significantly improves performance over the original count-based baseline.
- Results: The convolutional neural network sentence model over bigrams improves performance further and attains state of the art on answer sentence selection.The conclusion contrasts this model with the simpler bag-of-words variant.
- Takeaway: Compared with feature-engineered systems using external hand-coded semantic resources, the approach is much simpler and more flexible.