Source-linked AI summary
Learning to Paraphrase for Question Answering
Li Dong, Jonathan Mallinson, Siva Reddy, Mirella Lapata
TL;DR
Natural-language QA must handle many surface forms for the same information need, but generated paraphrases can be inappropriate. The paper learns paraphrase usefulness jointly with QA from question-answer pairs, using a flexible framework across tasks. Experiments on three datasets show performance improvements across tasks, including state-of-the-art results on GraphQuestions and competitive results on two additional benchmarks.
Problem
QA systems must handle varied surface forms that express the same information need, while generated paraphrases may include inappropriate candidates.
Method
The framework generates candidate paraphrases, scores their QA usefulness with a neural model, and trains paraphrase scoring and QA jointly from question-answer pairs.
Results
Performance improves across tasks on three datasets, with state-of-the-art results on GraphQuestions and competitive performance on two additional benchmark datasets.
Takeaways & Limitations
The framework supports task-specific paraphrase learning and incorporating multiple paraphrase generators without tying the system to a particular generator or QA model.
Abstract
from arXiv · showhide
Question answering (QA) systems are sensitive to the many different ways natural language expresses the same information need. In this paper we turn to paraphrases as a means of capturing this knowledge and present a general framework which learns felicitous paraphrases for various QA tasks. Our method is trained end-to-end using question-answer pairs as a supervision signal. A question and its paraphrases serve as input to a neural scoring model which assigns higher weights to linguistic expressions most likely to yield correct answers. We evaluate our approach on QA over Freebase and answer sentence selection. Experimental results on three datasets show that our framework consistently improves performance, achieving competitive results despite the use of simple QA models.
1 Introduction
Question answering is challenged by varied surface forms for the same information need, so this paper learns task-useful paraphrases within an end-to-end QA framework. The approach improves performance across Freebase QA and answer sentence selection benchmarks.
- Motivation: Semantically equivalent questions can produce different answers unless a QA system recognizes their shared information need.The paper illustrates this with alternative questions about who founded Microsoft.
- Related work: Prior paraphrase-based QA methods generate rewrites using resources such as PPDB, Wiktionary, machine translation, and paraphrase corpora.Generated paraphrases often include inappropriate candidates, motivating methods that score their usefulness for the QA task.
- Framework: The proposed framework scores candidate paraphrases with a neural model and uses the learned weights alongside the original question in a QA model.Paraphrase scoring and QA are trained end-to-end from question-answer pairs, without requiring specific paraphrase or QA models.
- Framework: Its plug-and-play design learns paraphrases suited to different QA tasks and supports comparing paraphrase models across applications.The framework can incorporate several paraphrasing modules rather than depending on one generator.
- Results: Experiments on three datasets consistently improve performance, reaching state-of-the-art results on GraphQuestions and competitive results on two additional benchmarks.The evaluation covers QA over Freebase and text-based answer sentence selection, using simple QA models.
2 Problem Formulation
The framework estimates answers by combining the original question with a manageable set of generated paraphrases, whose quality is scored before QA prediction. It supports multiple paraphrase generators and a flexible QA model, with neural, rule-based, and PPDB-based generation described here.
- 2 Problem Formulation: The model estimates p(a|q) by summing contributions from the original question and its candidate paraphrases Hq.Because generating every paraphrase is intractable, the framework restricts Hq to a manageable size.
- 2 Problem Formulation: Candidate paraphrases are scored neurally, weighted by their likelihood of yielding correct answers, and combined with the original question in a QA model.The QA model predicts an answer distribution, and the two models’ results are fused.
- 2.1 Paraphrase Generation: The framework increases paraphrase coverage and diversity using PPDB lexical and phrasal rules, neural machine translation, and rules mined from related-question clusters.The framework is not inherently tied to these generators; other paraphrase generators could also be used.
- 2.1.1 PPDB-based Generation: PPDB generation replaces words and phrases using bidirectionally entailing lexical and phrasal rules from a large paraphrase database.The described PPDB resource contains over a billion paraphrase pairs in 24 languages.
- 2.1.2 NMT-based Generation: NMT generation translates a question into K-best German pivots and back-translates them with beam search conditioned on multiple pivots.The EN-DE and DE-EN models compute pivot and output-token probabilities, while multiple pivots reduce reliance on a single translation.
- 2.1.2 NMT-based Generation: Unlike PPDB’s mainly local lexical variation, NMT paraphrases operate on the surface level and can perform major rewrites using wider contextual information.NMT paraphrase rules are captured implicitly and cannot be easily extracted from a phrase table.
- 2.1.3 Rule-Based Generation: WikiAnswers rule mining extracts paraphrase templates from recurring question clusters, then matches input substrings to template pairs with exact or fuzzy matching.Templates require at most one wildcard, occurrence in at least ten clusters, and co-occurrence more than five times with the same arguments.
- 2.2 Paraphrase Scoring: The neural paraphrase scorer encodes questions and paraphrases with a shared bidirectional LSTM, computes s(q′|q), and normalizes scores over Hq ∪ {q}.The scoring function uses concatenation, element-wise multiplication, a parameter vector, and a bias; alternative dot-product and bilinear forms were not empirically better.
3 Experiments
Experiments evaluate PARA4QA across structured knowledge-base QA and answer sentence selection, using multiple paraphrase generators and comparisons against non-paraphrase, unsupervised-scoring, and separately trained baselines. PARA4QA consistently improves performance, with dataset- and question-type-specific effects from different paraphrase generators.
- Datasets: PARA4QA was evaluated on three datasets spanning structured knowledge-base QA and answer sentence selection.The datasets were WebQuestions, GraphQuestions, and WikiQA.
- Paraphrase generation: Candidate paraphrases came from PPDB, neural machine translation, and rules mined from WikiAnswers, with duplicate or trivial rewrites discarded.The NMT system used English↔German, while PPDB and WikiAnswers supplied alternative candidate-generation sources.
- Overall results: PARA4QA outperformed baselines lacking paraphrases, lacking paraphrase scoring, or lacking joint training, and achieved state-of-the-art results on GraphQuestions.The comparisons included SIMPLEGRAPH, AVGPARA, DATAAUGMENT, and SEPPARA.
- Ablations: Removing NMT paraphrases reduced performance most on GraphQuestions and WikiQA, whereas removing the rule-based generator hurt WebQuestions most.The authors relate the WebQuestions effect to the rule-based method’s higher coverage there than on GraphQuestions.
- Overall results: On WikiQA, PARA4QA outperformed related baselines, and adding word matching features produced state-of-the-art performance.WikiQA performance was assessed with MAP and MMR for ranking correct answers among candidate answer sentences.
- Question complexity: PARA4QA improved both simple and complex GraphQuestions, with a more pronounced impact on simple questions.Simple questions contain one relation; complex questions involve multiple relations or aggregation, and the authors attribute the difference partly to easier paraphrase generation and scoring for simpler questions.
4 Conclusions
The framework learns task-specific paraphrases jointly with QA models from question-answer pairs, without tying the approach to a particular paraphrase generator or QA system. Across three datasets, it improves performance across tasks and supports future exploration of paraphrase coverage, rewriting, and quality.
- Paraphrase scoring and QA models are trained end-to-end on question-answer pairs, learning paraphrases with a purpose.
- The framework is not tied to a specific paraphrase generator or QA system and can incorporate several paraphrasing modules.
- The framework can serve as a testbed for exploring paraphrase-generator coverage and rewriting capabilities.
- Three datasets show that the method improves performance across tasks.
- Improving paraphrase diversity and quality could further enhance QA performance, alongside more advanced scoring models and additional generators.