Source-linked AI summary
Transforming Question Answering Datasets Into Natural Language Inference Datasets
Dorottya Demszky, Kelvin Guu, Percy Liang
TL;DR
Existing NLI datasets omit some important reasoning phenomena, while large QA datasets offer abundant and diverse material. The paper learns QA2D models to convert question-answer pairs into declarative sentences and thereby construct NLI pairs. The resulting QA-NLI resource contains over 500,000 examples with broad inference diversity and generalizes across QA datasets despite primary training on SQuAD.
Problem
Existing NLI datasets cover many linguistic phenomena but lack important forms such as multi-sentence reasoning, motivating broader NLI resources.
Method
The paper learns QA2D sentence transformations that convert QA question-answer pairs into declarative sentences, which pair with passages as NLI examples.
Results
Over 500,000 QA-NLI examples are created, with broad inference phenomena, while the primarily SQuAD-trained model generalizes across diverse QA datasets.
Takeaways & Limitations
QA resources can be automatically converted into freely available NLI data that expands coverage of reasoning phenomena and may reduce some annotation artifacts.
Takeaways & Limitations
The rule-based QA2D system cannot handle several semantic idiosyncrasies, motivating a neural sequence generation model for improvement.
Abstract
from arXiv · showhide
Existing datasets for natural language inference (NLI) have propelled research on language understanding. We propose a new method for automatically deriving NLI datasets from the growing abundance of large-scale question answering datasets. Our approach hinges on learning a sentence transformation model which converts question-answer pairs into their declarative forms. Despite being primarily trained on a single QA dataset, we show that it can be successfully applied to a variety of other QA resources. Using this system, we automatically derive a new freely available dataset of over 500k NLI examples (QA-NLI), and show that it exhibits a wide range of inference phenomena rarely seen in previous NLI datasets.
1 Introduction
The paper addresses gaps in existing NLI datasets by automatically converting diverse QA resources into large-scale NLI data through QA2D sentence transformations. The resulting QA-NLI dataset contains over 500,000 examples and covers varied inference phenomena.
- Motivation: Existing NLI datasets capture many linguistic phenomena but lack some important forms of reasoning, including multi-sentence reasoning.This motivates augmenting and diversifying NLI resources.
- Approach: QA2D combines a question and answer into a declarative sentence, after which the passage and declarative sentence form an NLI premise-hypothesis pair.For example, “Who called Taylor?” with answer “Liz” becomes “Liz called Taylor.”
- Motivation: QA datasets are abundant and cover diverse reasoning strategies, making them suitable sources for expanding NLI datasets.Their organic task structure also makes QA data easy to crowdsource and available in established resources.
- Approach: The authors explore rule-based, crowdsourced, and neural approaches to produce declarative answer sentences.The approaches build on one another, with crowdsourced triples training the neural QA2D model.
- Results: 45–57% of automatically generated declaratives exactly match human answers, with BLEU scores of 73–83 across datasets.The neural model is primarily trained on SQuAD but generalizes across Wikipedia, newswire, and movie-plot QA resources.
- Results: QA-NLI contains over 500,000 freely available NLI examples and exhibits multi-sentence, meta-level, and presupposition-based inference phenomena.The authors quantify its diversity by reasoning type and required contextual scope and report evidence that it can reduce some annotation artifacts in SNLI and MultiNLI.
2 Approach
The approach transforms QA examples into NLI pairs by converting questions and answers into declarative sentences, then labeling those sentences according to entailment. It supports negative examples but produces only a two-way entailment/nonentailment distinction.
- QA-to-NLI conversion: A QA example consists of a passage, question, and answer span, which QA2D combines into a declarative answer sentence.If the answer is correct, the passage and declarative sentence form an entailed NLI pair.
- Negative examples: Incorrect answers or questions unanswerable from the passage produce negative, non-entailed NLI pairs.Incorrect options occur in multiple-choice datasets, while unanswerable questions occur in datasets such as SQuADRUn.
- Inference labels: Automated QA2D distinguishes entailment from nonentailment but cannot assign the three-way labels entailment, neutral, and contradiction.Incorrect answers and unanswerable questions do not reliably map to contradiction and neutral, respectively.
- Premise scope: The resulting premise may range from a short paragraph or sentence to an entire document, depending on the source QA dataset.Even document-scale premises remain valid NLI pairs under this construction.
3 Datasets
The paper transforms five structurally and topically diverse QA datasets into NLI resources, spanning domains, passage sizes, reasoning demands, and answer formats. It also notes that incorrect and unanswerable QA cases do not map straightforwardly onto NLI contradiction and neutral labels.
- The five source datasets are MovieQA, NewsQA, QAMR, RACE, and SQuAD, selected to maximize structural and topical diversity.
- Their domains include movie plots, newswire, Wikipedia, and English exams covering varied genres and topics.
- Incorrect answers and unanswerable questions cannot be assigned contradiction and neutral labels mechanically.
- Passages range from single sentences to multiple paragraphs, while answers may be spans within passages or free-response text.
- Question demands range from selecting arguments in one sentence to holistic reasoning about a passage.
4 QA2D: Rule-based
The rule-based QA2D system converts questions and answers into declarative sentences using syntactic structure, but semantic idiosyncrasies and parser errors limit its coverage. These limitations motivate a learned model.
- QA2D requires transforming question-answer pairs into declarative sentences, guided by wh-words, roots, auxiliaries, copulas, prepositions, and particles.
- The rule-based system depends on accurate part-of-speech tagging and dependency parsing to identify the relevant question and answer structure.
- About 10% of rule-based mistakes are attributed to tagging or parsing errors.
- Rules struggle with semantic choices such as whether bare organizational names require a definite article.
5 QA2D: crowdsourced
The authors collect human-authored declarative sentences through two crowdsourcing setups and use SQuAD as the main training source. Post-editing is more valid, but evaluation data is collected from scratch to avoid potential bias.
- Crowdsourcing uses either writing declarative sentences from scratch or editing rule-based outputs into well-formed sentences.
- Writing from scratch minimizes rule-based bias but is slower and more prone to typos than post-editing.
- 91% of from-scratch outputs were valid, compared with 97% of post-edited outputs.
- The authors exclusively use from-scratch collection for evaluation data because post-editing could bias the dataset.
- SQuAD supplies most data because of its size, quality, and syntactic diversity, yielding 85,713 filtered training QA pairs.
- The train split is 95% SQuAD, while dev is 81% SQuAD and test is evenly divided across the five datasets.
6 QA2D: Neural Sequence Model
The neural QA2D model learns to generate declarative sentences from question-answer pairs using an encoder-decoder architecture. Its evaluation compares this model with the rule-based system across diverse QA data and analyzes performance by question type and input length.
- The neural model learns p(D | Q, A) from collected question-answer-declarative sentence triples.
- It encodes questions and answers with shared-weight bidirectional three-layer LSTMs and generates outputs with a three-layer attention-based LSTM decoder.
- The model includes a copy mechanism and is trained with cross-entropy loss optimized by Adam.
7 QA2D: Results
The neural QA2D system consistently outperforms the rule-based system overall, while performance varies by question type, dataset, and input length. Human evaluation finds both systems reasonably complete, but less grammatical and natural than human outputs.
- Rule-based vs neural: NEURAL leads RULE-BASED by 2.6 BLEU points on average and 6.2% in exact match across datasets.
- Results by dataset: RULE-BASED performs 6-9 exact-match points below NEURAL on RACE, MovieQA, and SQuAD, where semantic answer modifications are often required.
- Results by question length: RULE-BASED is more robust beyond 30 tokens, while NEURAL performs better below 20 tokens, which comprise most examples.
- Results by question type: NEURAL performs best on who questions, whereas both models perform worst on which questions because these often require selecting and positioning words from the wh phrase.
- Human evaluation: Human outputs score 4.6 for grammar and 4.8 for completeness, compared with 3.8 grammar for RULE-BASED and 3.9 for NEURAL.Both automated systems are only 0.2-0.3 points below humans on completeness.
8 Analysis of NLI Datasets
The converted QA datasets contain varied reasoning scopes and types, and answer correctness strongly predicts inference judgments without reducing entailment to a binary distinction. Their word-class statistics also suggest fewer annotation artifacts than SNLI and MultiNLI.
- Reasoning phenomena: MovieQA, NewsQA, and SQuAD mostly require sentence-level reasoning, with remaining examples split between multi-sentence and argument-level reasoning at roughly 5:3.
- Reasoning phenomena: QAMR has two thirds argument-level examples and no multi-sentence reasoning, with many pairs arising from presuppositions.
- Reasoning phenomena: RACE predominantly requires multi-sentence reasoning and extensively uses world knowledge, meta-level reasoning, and human-psychology reasoning.
- Inference labels: More than 90% of examples from correct answers are more likely true than false, while about 80% from incorrect answers are not likely true.
- Inference labels: About half of examples in both correct- and incorrect-answer categories do not fit a strict entailment/contradiction dichotomy, supporting a non-binary notion of entailment.
- Annotation artifacts: Compared with SNLI and MultiNLI, MovieQA-derived non-entailments lack negation words and entailments lack positive or nonspecific words among their highest-PMI cues.
9 Related Work & Discussion
The paper situates automated QA-to-NLI conversion alongside related recasting and sentence-transformation tasks, while outlining future directions for stronger generation models and expanded NLI systems.
- QA to NLI: The paper is the first to automate conversion of QA datasets into NLI datasets, whereas earlier QA–NLI efforts used human experts for QA2D.It distinguishes its contribution from prior recasting work on semantic classification datasets.
- Text transformation tasks: QA2D produces declarative-to-declarative transformations, connecting QA-to-NLI with sentence simplification, paraphrase, and summarization.Declarative sentences can be chained under these operations to support more complex inferences.
- Text transformation tasks: Question generation resembles the reverse of QA2D, but emphasizes selecting interesting questions rather than robust sentence transformation.
- Neural sequence generation: General-purpose sequence-generation advances, especially syntactic or transducer-like architectures, could further improve QA2D data efficiency and performance.
- Future systems: Larger NLI training resources could support generative NLI, sentence decomposition, and sentence synthesis systems.