Source-linked AI summary
LSTM-based Deep Learning Models for Non-factoid Answer Selection
Ming Tan, Cicero dos Santos, Bing Xiang, Bowen Zhou
TL;DR
Answer selection must identify semantically relevant answers despite lexical mismatch and noisy candidate text. The paper proposes a feature-free biLSTM framework with CNN and question-conditioned attention extensions, evaluating it on InsuranceQA and TREC-QA. The proposed models outperform several strong baselines, while shared question-answer network parameters were observed to perform better and converge faster than separate parameters.
Problem
Answer selection is challenging because correct answers may not share lexical units with questions and may contain unrelated information.
Method
The paper builds question and answer embeddings with biLSTMs and cosine similarity, extending the framework with CNN representations and attention-based answer embeddings.
Results
The proposed models outperform non-DL, CNN-based DL, and other strong baselines on InsuranceQA and TREC-QA.
Takeaways & Limitations
Question-conditioned attention improves answer-selection performance, while combining CNN and attention yields no obvious improvement over attention alone.
Takeaways & Limitations
Preliminary experiments found shared question-answer network parameters performed better and converged faster than separate parameters, which had twice as many parameters.
Abstract
from arXiv · showhide
In this paper, we apply a general deep learning (DL) framework for the answer selection task, which does not depend on manually defined features or linguistic tools. The basic framework is to build the embeddings of questions and answers based on bidirectional long short-term memory (biLSTM) models, and measure their closeness by cosine similarity. We further extend this basic model in two directions. One direction is to define a more composite representation for questions and answers by combining convolutional neural network with the basic framework. The other direction is to utilize a simple but efficient attention mechanism in order to generate the answer representation according to the question context. Several variations of models are provided. The models are examined by two datasets, including TREC-QA and InsuranceQA. Experimental results demonstrate that the proposed models substantially outperform several strong baselines.
1 INTRODUCTION
Answer selection seeks the best candidate for a question, but correct answers may be semantically related without lexical overlap and may contain noise. The paper proposes a feature-free biLSTM framework, extended with CNN representations and question-conditioned attention, and reports improvements over strong baselines on InsuranceQA and TREC-QA.
- Task and challenge: Answer selection searches a candidate pool for the best answer to a question, which may have multiple ground-truth answers.At test time, candidate answers may differ from those observed during training.
- Task and challenge: Correct answers may lack shared lexical units with questions and may contain substantial unrelated information.
- Proposed approach: The proposed framework builds question and answer representations with biLSTMs, pooling, and a similarity metric without feature engineering, linguistic tools, or external resources.
- Proposed approach: CNN structures provide more composite question and answer representations, while attention generates answer embeddings according to question context.These are the paper’s two extensions to the basic framework.
- Evaluation: The models are evaluated on InsuranceQA and TREC-QA and outperform non-DL, CNN-based DL, and other strong baselines.
2 RELATED WORK
Earlier answer-selection methods used engineered features, linguistic tools, external resources, or deep-learning matching and ranking strategies. This work belongs to representation-and-similarity approaches, using biLSTMs and question-conditioned attention to address limitations of CNN-only and independently encoded models.
- Prior approaches: Earlier methods used semantic features, parse-tree matching, or dependency-tree edit sequences, often requiring additional resources and linguistic tooling.
- Prior approaches: Deep-learning approaches learned and matched question-answer representations, constructed joint feature vectors, or converted answer selection into classification or learning-to-rank.
- Positioning and novelty: The proposed framework belongs to similarity-based approaches and uses bidirectional LSTMs rather than only CNN architectures.
- Positioning and novelty: Unlike Feng et al. (2015), the models integrate CNN structures over biLSTM representations and generate answer embeddings according to question context.
3 APPROACH
The framework represents questions and answers with biLSTMs and matches them using cosine similarity, then extends representation learning with CNNs and question-conditioned attention.
- 3.1 BASIC MODEL: QA-LSTM: QA-LSTM independently encodes questions and answers with biLSTMs, then measures their distance using cosine similarity and trains with a hinge ranking loss.The loss contrasts a ground-truth answer with a randomly selected incorrect answer using a constant margin.
- 3.1 BASIC MODEL: QA-LSTM: Question and answer representations can use average pooling, max pooling, or concatenated final vectors from both LSTM directions.Dropout is applied to the resulting representations before cosine-similarity matching.
- 3.1 BASIC MODEL: QA-LSTM: Sharing network parameters between question and answer encoders performs significantly better and converges faster than using separate parameters.The shared architecture constrains corresponding question and answer vector elements to represent the same biLSTM outputs.
- 3.2 QA-LSTM/CNN: QA-LSTM/CNN applies convolutional filters to biLSTM outputs, using local windows and k-max pooling to produce more composite representations.The convolutional structure emphasizes selected parts of the sequence rather than weighting every token evenly; the experiments use k = 1.
- 3.3 ATTENTION-BASED QA-LSTM: The attention extension weights answer biLSTM outputs according to the question embedding before pooling, dynamically emphasizing question-relevant words.The weights are produced with a softmax and are described as question-dependent analogues of tf-idf weights.
4 INSURANCEQA EXPERIMENTS
InsuranceQA experiments compare QA-LSTM variants with pooling, CNN, and attention against non-DL and CNN baselines. Attention and combined architectures perform strongly, particularly for long answers, while alternative similarity choices can hurt accuracy.
- Experimental setup: InsuranceQA provides training, validation, and two test sets, with questions averaging 7 tokens and answers 94 tokens.Development and test questions use answer pools of 500 candidates.
- Baselines: The baselines include bag-of-word, a weighted dependency model, and CNN-based Architecture-II variants, including GESD.These represent non-DL and strong CNN-based comparison approaches.
- QA-LSTM variants: Max pooling substantially outperforms average pooling, while concatenating the final bidirectional vectors performs worst among basic QA-LSTM variants.The authors associate max pooling with retaining more local information.
- CNN extension: Using 4000 CNN filters gives the best validation accuracy among the CNN variants, with performance comparable to the best baseline.The CNN uses filter width m = 2; increasing the width to 3 or 4 did not improve performance.
- Attention extension: Attention improves Model (G) over max-pooled Model (C) by over 2% on validation and Test2, while Model (H) exceeds mean-pooled Model (B) by over 8% across datasets.Model (H) also improves over the best baseline by 3% on validation, 2.8% on Test1, and 1.2% on Test2, while using fewer parameters than Architecture-II.
- Additional analyses: Combining CNN and attention improves over Model (F) by 1% on all sets but shows no obvious improvement over Model (H), while GESD provides no accuracy gain.The authors hypothesize that CNN operations may cause effective attention to vanish.
- Answer-length analysis: For longer answers, the basic max-pooling Model (C) falls increasingly behind CNN and attention models as ground-truth answer length increases.This pattern appears in both Test1 and Test2, which are divided into 11 answer-length buckets.
5 TREC-QA EXPERIMENTS
The TREC-QA experiments evaluate the proposed models using established answer-selection splits and MAP/MRR metrics. The best proposed model achieves the strongest results for both metrics and outperforms the baselines.
- TREC-QA contains 1,162 training questions, 65 development questions, and 68 test questions after removing questions with only positive or negative answers.
- The experiments use Mean Average Precision (MAP) and Mean Reciprocal Rank (MRR), calculated with official evaluation scripts.
- The comparison includes prior work, with Wang and Nyberg (2015) identified as the strongest MAP baseline and Feng et al. (2015) as the strongest MRR baseline.
- Model B's attention improves Model A by 0.7% in MAP and approximately 2% in MRR, while CNN-based Model C improves both metrics further over Model A.
- Model E, which uses Model D's combined architecture with 500-dimensional LSTM hidden vectors, achieves the best results for both metrics and outperforms the baselines.
6 CONCLUSION
The paper concludes that its bidirectional-LSTM framework supports answer selection without feature engineering, linguistic tools, or external resources. CNN and attention extensions improve the representations, and experiments on TREC-QA and InsuranceQA outperform strong baselines.
- The framework uses bidirectional LSTMs and does not rely on feature engineering, linguistic tools, or external resources.
- The authors extend the basic framework with CNN-based composite representations and question-conditioned attention for answer embeddings.
- Experiments on TREC-QA and InsuranceQA show that the proposed models outperform a variety of strong baselines.
- Future work includes evaluating the approaches on Community QA answer quality prediction and recognizing textual entailment.
- The authors also plan to extend attention from word-level processing to phrasal or sentential levels.