Source-linked AI summary
Applying Deep Learning to Answer Selection: A Study and An Open Task
Minwei Feng, Bing Xiang, Michael R. Glass, Lidan Wang, Bowen Zhou
TL;DR
The paper addresses non-factoid question answering by matching questions with candidate answers through a general deep learning framework. It releases an insurance-domain corpus and open task, and reports superior performance over baselines, with further gains from architectural and similarity-metric techniques.
Problem
The paper studies answer selection for spoken question answering, where a question must be matched with the best answer candidate from a pool.
Method
The framework learns distributed representations of questions and answer candidates with CNN-based architectures and measures their matching degree using a similarity metric.
Results
The proposed method achieves significantly better accuracy than baseline methods, while architectural variants and novel GESD and AESD similarity metrics provide further improvements.
Takeaways & Limitations
The released insurance-domain corpus and benchmark enable fair comparison among methods, while the framework is adaptable to different languages or domains without linguistic tools.
Takeaways & Limitations
The technology targets applications such as online customer service and is not intended for question answering requiring reasoning.
Abstract
from arXiv · showhide
We apply a general deep learning framework to address the non-factoid question answering task. Our approach does not rely on any linguistic tools and can be applied to different languages or domains. Various architectures are presented and compared. We create and release a QA corpus and setup a new QA task in the insurance domain. Experimental results demonstrate superior performance compared to the baseline methods and various technologies give further improvements. For this highly challenging task, the top-1 accuracy can reach up to 65.3% on a test set, which indicates a great potential for practical use.
1. INTRODUCTION
The paper frames spoken question answering as matching questions with candidate answers, introduces an insurance-domain corpus and open benchmark, and defines practical candidate-pool constraints.
- 1. INTRODUCTION: The QA module is treated as an answer-matching and selection problem, choosing the candidate pair with the highest matching score.The framework requires a metric to measure each question–answer pair's matching degree.
- 1. INTRODUCTION: The general formulation assumes that every question has an answer candidate pool, which can be built with search engines or information-retrieval software.Examples include Google Search and Apache Lucene.
- 1. INTRODUCTION: The authors release an insurance-domain QA corpus and open task to support fair comparisons among methods.The corpus contains train, development, test1, and test2 parts and is presented as the first released insurance-domain QA task.
- 1. INTRODUCTION: The benchmark uses 500 answer candidates per question, including the ground-truth answer and randomly sampled negative answers.The full answer space contains 24 981 unique answers, but comparing every question with all candidates is described as impractical.
- 1. INTRODUCTION: The technology targets potential applications such as online customer service and is not intended for reasoning-dependent questions.The paper gives questions whose answers depend on temporal conditions as an example of the excluded reasoning requirement.
- 1. INTRODUCTION: The released corpus contains 24 981 answers and 2 386 749 words of answer text.Table 1 notes that multiple answers may correspond to a question, so answer counts can exceed question counts.
2. MODEL DESCRIPTION
The paper develops a CNN-based framework that represents questions and candidate answers as vectors, compares them with similarity metrics, and trains with a margin-based ranking loss. It explores multiple architectures, convolutional features, pooling, skip-bigrams, supervision, and alternative similarity metrics.
- The framework learns distributed vector representations for questions and answers, then measures their matching degree with a similarity metric.Candidate answers are selected by the highest matching score.
- Baseline Systems: The bag-of-words baseline sums idf-weighted word embeddings, while the IR baseline uses weighted term and proximity features.Both baselines calculate scores for question–candidate pairs and return the highest-scoring candidate.
- CNN-based System: CNN filters extract local features, and 1-MaxPooling retains the highest activation for each filter while discarding feature position.The convolution output can represent features such as n-grams; pooling selects the strongest match.
- Training and Loss Function: Training pairs each ground-truth answer with a sampled negative answer and updates parameters when their cosine-similarity gap falls below margin m.At test time, the candidate with the largest cosine similarity is selected from a pool of 500 answers.
- Architectures: Architecture I separately processes questions and answers through hidden, CNN, pooling, and tanh layers before comparing their vector representations.Architecture II shares corresponding hidden and CNN weights, while Architectures III and IV add post-CNN hidden layers, with IV sharing those layers.
- Architectures: Further variants increase filter counts, add skip-bigram convolutions, stack CNNQA layers, apply layer-wise supervision, and evaluate GESD and AESD similarity metrics.The skip-bigram operation allows pooling to adapt to bigram or skip-bigram features; GESD and AESD are reported as novel metrics with superior performance.
3. EXPERIMENTAL SETUP
The experiments use a Java deep-learning implementation with HOGWILD training, word2vec initialization, and multiple CNN architectures and similarity metrics.
- HOGWILD training lets threads update neural-network weights without locking, introducing randomness across runs.The implementation uses one training instance per thread and no locking.
- Architecture V uses two shared CNNQA layers, with the first layer’s multi-filter output forming a matrix for the next layer.Each row corresponds to one filter, and each column corresponds to an input bigram.
- Architecture VI uses two shared CNNQA layers and two cost functions.
- 100-dimensional word2vec embeddings initialize the model and remain trainable parameters for the QA task.Stochastic Gradient Descent is used, with L2 regularization added to the loss.
- Table 3 evaluates similarity functions with normalized question and answer vectors, including L1-, L2-, and inner-product-based metrics.Its bottom section evaluates the proposed metric with larger filter counts.
4. RESULTS AND DISCUSSIONS
Deep-learning models outperform the baselines, while shared layers, deeper CNNs, layer-wise supervision, and similarity-metric choices materially affect performance.
- The proposed method achieves significantly better accuracy than both word-embedding and traditional term-based baselines.
- Shared hidden or CNN layers outperform separate question and answer layers, reducing the parameter-learning difficulty.The authors attribute this to corresponding question and answer elements representing the same convolution result.
- Adding a hidden layer after the CNN degrades performance, indicating that the CNN already captures useful QA-matching features.
- More filters and two CNN layers improve accuracy by capturing more features and representing higher-level abstractions over wider input ranges.
- Layer-wise supervision alleviates the difficulty of effective learning in deep networks, improving results over an otherwise comparable two-layer CNN.
- Similarity metrics matter at equal model capacity, and cosine similarity is not the best choice for this task.The study categorizes metrics by L1 norm, L2 norm, and inner product, then proposes combinations of L2 and inner-product information.
5. RELATED WORK
The paper extends prior deep-learning NLP work through broader architecture experiments and additional investigations of similarity metrics, skip-bigram convolution, and layer-wise supervision.
- Unlike previous work, this paper conducts extensive experiments over various architectures.
- The study explores similarity metrics, skip-bigram-based convolution, and layer-wise supervision not presented in previous work.
6. CONCLUSIONS
The paper presents a language- and domain-adaptable deep-learning framework for answer selection, releases an insurance QA benchmark, and reports promising performance.
- The framework studies spoken QA as answer selection without relying on linguistic tools and can adapt to different languages or domains.
- The authors create and release an insurance-domain QA corpus and task to support fair comparisons among methods.
- 65.3% top-one accuracy is reached on the test corpus when selecting one answer from a pool of 500.
- The released dataset and benchmark target potential applications such as online customer service.