PaperlayerView exact revision on arXiv

Source-linked AI summary

aNMM: Ranking Short Answer Texts with Attention-Based Neural Matching Model

Liu Yang, Qingyao Ai, Jiafeng Guo, W. Bruce Croft

arXiv:1801.01641v2cs.IRcs.CL

TL;DR

Existing deep-learning approaches for question-answer matching often rely on additional features to perform well, while feature-engineered methods require manual linguistic resources. The paper proposes aNMM with value-shared weighting and question-term attention, and reports strong TREC QA results, including the best reported performance when combined with a simple additional feature.

  • Problem

    Feature-engineered QA ranking requires manual linguistic features and may require parsers or external knowledge, while existing neural models can miss arbitrary-position matching signals and question-term focus.

  • Method

    aNMM ranks short answer text using value-shared weights for matching signals and a question attention network that learns question-term importance.

  • Results

    aNMM trained with TRAIN-ALL beats previous state-of-the-art systems, including feature-engineering and deep-learning methods, on the summarized QA answer-ranking results.

  • Takeaways & Limitations

    The model achieves strong answer-ranking performance without manual feature engineering, expensive NLP parsing, or external knowledge bases beyond pre-trained word embeddings.

  • Takeaways & Limitations

    For training sets larger than TREC QA, aNMM-2 may perform better because it has more parameters, while the impact of hidden-layer count remains future work.

Abstract

from arXiv · show

As an alternative to question answering methods based on feature engineering, deep learning approaches such as convolutional neural networks (CNNs) and Long Short-Term Memory Models (LSTMs) have recently been proposed for semantic matching of questions and answers. To achieve good results, however, these models have been combined with additional features such as word overlap or BM25 scores. Without this combination, these models perform significantly worse than methods based on linguistic feature engineering. In this paper, we propose an attention based neural matching model for ranking short answer text. We adopt value-shared weighting scheme instead of position-shared weighting scheme for combining different matching signals and incorporate question term importance learning using question attention network. Using the popular benchmark TREC QA data, we show that the relatively simple aNMM model can significantly outperform other neural network models that have been used for the question answering task, and is competitive with models that are combined with additional features. When aNMM is combined with additional features, it outperforms all baselines.

1. INTRODUCTION

The introduction frames short-answer ranking as a question-answer matching problem constrained by the weaknesses of feature engineering and existing neural architectures. It proposes aNMM to model matching values and question-term importance, then evaluates it on TREC QA.

  • Feature-engineered QA ranking uses complex linguistic features but can require time-consuming, domain-dependent expertise, parsers, and external knowledge sources.
  • Existing CNN and LSTM matching models may miss detailed interactions because matching signals can occur at arbitrary positions and sequential models lack direct question-answer term interactions.
  • Question focus is also under-modeled: CNNs treat question terms equally, while LSTMs tend to emphasize terms nearer the sequence end.
  • The paper proposes aNMM with value-shared weights to model semantic-similarity value regularities rather than spatial regularities.
  • A question attention network uses a gating function to explicitly discriminate the importance of question terms.
  • On TREC QA, aNMM outperforms a linguistic-feature-engineering method without additional features and reaches state-of-the-art performance when combined with a simple additional feature.

2. RELATED WORK

The related work situates aNMM among neural text-matching, factoid QA, community-QA answer-ranking, and answer-retrieval methods. Prior work spans learned semantic representations, linguistic features, and parser-based structures.

  • Neural text-matching research includes models such as DSSM, CDSSM, ARC-I/ARC-II, DCNN, DeepMatch, MultiGranCNN, and MatchPyramid.
  • DSSM maps queries and documents into a shared semantic space, while ARC-I represents sentences separately before comparing them with a multilayer perceptron.
  • Factoid QA studies commonly use TREC QA data and formulate answer sentence selection through semantic matching, lexical-semantic models, or recursive neural networks.
  • Community-QA answer ranking has used similarity, translation, density, frequency, lexical-semantic, and discourse features.
  • Answer passage and sentence retrieval research uses syntactic and semantic structures from parsers and annotated passage-level datasets.

3. ATTENTION-BASED NEURAL MATCHING MODEL

This section introduces aNMM, an attention-based neural matching model for ranking answer texts, and identifies its architecture as the subject of the model presentation.

  • aNMM is an attention-based Neural Matching Model proposed for ranking answer texts.
  • The model is presented through an architecture shown in Figure 1, following definitions of its terminology.

3.1 Terminology

The terminology defines short answer text as the ranking object and represents question-answer semantic matching with a matrix and its row-wise vectors.

  • Short Answer Text denotes a short fact, answer sentence, or answer passage addressing a user's information need and serving as the ranking object.
  • In the experiments, the paper mainly ranks answer sentences containing correct answer facts in TREC QA data.
  • A QA Matching Matrix is an M by N matrix whose entries represent semantic similarity between question and answer terms using word-embedding cosine similarity.
  • A QA Matching Vector is one row of that matrix, containing the similarity between one question term and all terms in the answer.

3.2 Model Overview

aNMM ranks question–answer pairs by constructing matching signals, learning hierarchical abstractions with value-shared weights, and applying question attention to produce a final score.

  • aNMM constructs a QA matching matrix for each question–answer pair using pretrained word embeddings.
  • A deep neural network with value-shared weights in its first layer and fully connected layers thereafter learns hierarchical semantic-matching abstractions.
  • A question attention network learns question-term importance and produces the final ranking score.
  • The paper proposes and compares two neural matching architectures, beginning with the basic aNMM-1 architecture.

3.3 Value-shared Weighting

The value-shared weighting scheme addresses variable-length matching matrices by weighting matching signals according to their values rather than their positions.

  • Value-shared weighting also handles variable answer lengths without relying on CNN position-shared filters.Important question terms and semantically similar answer words may occur at arbitrary positions.
  • aNMM uses value-shared rather than position-shared weights because semantic matching depends on value regularities, not spatial regularities.In CNNs, a node’s weight depends on position; in aNMM, it depends on the matching-signal value.
  • The model groups matching signals into value ranges, learns their importance, and thereby combines different levels of matching evidence.With matching values in [−1, 1] and bin size 0.1, the scheme yields 21 bins.
  • After value-shared weighting produces a fixed-size hidden representation, fully connected layers learn higher-level representations.

3.4 Question Attention Network

The question attention network uses a softmax gate to weight matching signals by question-term importance, allowing aNMM to distinguish more and less important terms.

  • The question attention network uses a softmax gate to combine outputs from networks corresponding to individual question terms.
  • Query-term importance is represented by applying softmax to the dot product between each query-word embedding and a model parameter.The query-word embedding is normalized before the dot product is computed.
  • aNMM combines semantic matching signals weighted by question-attention outputs rather than treating all question terms identically.The softmax gates discriminate attention across different question terms.

3.5 Model Training

aNMM learns value-shared and attention-gating parameters with pairwise large-margin training on triples containing a preferred and a non-preferred answer.

  • The model learns value-shared weights and gating-function parameters as its two parameter sets.
  • Training constructs triples (q, a+, a−) in which a+ is matched with the question better than a−.
  • The hinge-style ranking loss is e(q, a+, a−; w, v) = max(0, 1 − S(q, a+) + S(q, a−)).S(q, a) denotes the predicted matching score for a question–answer pair.
  • Triples violating the desired margin are updated with backpropagation, while triples satisfying the margin are skipped.
  • The model uses stochastic gradient descent with mini-batches and an adaptive learning rate during optimization.The paper states that this learning-rate setting provides better convergence guarantees.

3.6 Extension to Deep Neural Networks with Multiple Sets of Value-shared Weights

aNMM-2 extends aNMM-1 by using multiple sets of value-shared weights, enabling multiple intermediate nodes and combined matching scores. Its parameters are learned through back propagation for ranking short answer texts.

  • 3.6 Extension to Deep Neural Networks with Multiple Sets of Value-shared Weights: aNMM-2 extends aNMM-1 with multiple sets of value-shared weights for each question–answer matching vector.aNMM-1 uses only one set, whereas aNMM-2 introduces multiple intermediate nodes in the first hidden layer.
  • 3.6 Extension to Deep Neural Networks with Multiple Sets of Value-shared Weights: A hidden layer learns multiple combined scores from the input layer using multiple weight vectors.These weight vectors form a two-dimensional matrix in the extended architecture.
  • 3.6 Extension to Deep Neural Networks with Multiple Sets of Value-shared Weights: The model combines hidden-layer outputs at the output layer using softmax gate functions from the question attention network.The hidden-layer outputs are summed after weighting by the gate-function outputs.
  • 3.6 Extension to Deep Neural Networks with Multiple Sets of Value-shared Weights: aNMM-2 uses parameters connecting the input layer, two hidden layers, and output layer.The three parameter sets are wkt, rt, and vp, respectively, and all are updated through back propagation.
  • 3.6 Extension to Deep Neural Networks with Multiple Sets of Value-shared Weights: After random initialization, back propagation updates the parameters until convergence, after which the learned model ranks short answer texts.The training process uses the same objective function referenced for the model.

4. EXPERIMENTS

Experiments evaluate aNMM on TREC QA using ranking metrics, learned matching weights, question-term importance, hyper-parameter sensitivity, and comparisons with neural and feature-engineered baselines. The results show strong answer-ranking performance, including improvements without additional features and leadership over prior systems when trained on TRAIN-ALL.

  • Data Set and Experiment Settings: The TREC QA benchmark contains factoid questions with single-sentence candidate answers, using TRAIN and larger TRAIN-ALL training sets.Candidate answers were automatically selected from document pools, with correctness identified through manual judgments or answer-sentence matching.
  • Data Set and Experiment Settings: MAP and MRR evaluate ranked answer sentences against ground truth, measuring performance across correct answers and the first correct answer respectively.The study computes both metrics with the official trec_eval scripts.
  • Value-shared Weight: Value-shared weights vary across match-signal ranges, while negative matching scores show little variation because most embedding-based similarity scores are positive.The analysis suggests removing negative-score bins to reduce model dimensionality and improve training efficiency.
  • Question Term Importance: Question attention learns meaningful term importance, identifying terms such as “khmer”, “rouge”, and “power” in a factoid question.Replacing learned attention outputs with IDF decreases answer-ranking performance, especially on TRAIN.
  • Learning without Combining Additional Features: 14.67% MAP improvement on TRAIN and 9.15% on TRAIN-ALL are achieved by aNMM-1 over Severyn et al.’s CNN without additional features.The passage also reports significant MRR improvements for aNMM-1 and aNMM-2 over prior deep learning methods.
  • Results Summary: 4.13% MAP and 3.83% MRR gains over Yih et al.’s feature-engineering baseline are achieved by aNMM-1 on TRAIN-ALL without additional features.The paper reports that TRAIN-ALL-trained aNMM beats previous state-of-the-art systems, while aNMM-1 is faster to train and aNMM-2 may suit larger datasets.

5. CONCLUSIONS AND FUTURE WORK

The paper concludes that aNMM improves short-answer ranking without additional features and reaches new state-of-the-art performance when combined with a simple additional feature.

  • aNMM uses value-shared weighting and question attention to rank short answer text.
  • On TREC QA tracks 8–13, aNMM outperforms a state-of-the-art linguistic feature-engineering method without additional features.
  • With a simple additional feature, aNMM achieves new state-of-the-art performance among existing methods.
  • Future Work: Future work will examine other deep learning architectures for answer ranking and extend evaluation to non-factoid question-answering datasets.
Loading 1801.01641v2…