PaperlayerView exact revision on arXiv

Source-linked AI summary

Efficient and Robust Question Answering from Minimal Context over Documents

Sewon Min, Victor Zhong, Richard Socher, Caiming Xiong

arXiv:1805.08092v1cs.CL

TL;DR

Existing neural QA models struggle to scale to large document collections and are sensitive to adversarial inputs. This paper studies minimal answer-bearing context and introduces a dynamic sentence selector, achieving substantial speedups with comparable or better accuracy across several QA tasks.

  • Problem

    Neural QA models are difficult to scale to large corpora because they model complex document-question interactions and are sensitive to adversarial inputs.

  • Method

    The paper studies the minimal context needed for answers and uses a sentence selector that dynamically selects a question-dependent set of sentences for the QA model.

  • Results

    Across SQuAD, NewsQA, TriviaQA and SQuAD-Open, the system reduces training and inference times by up to 15× and 13×, respectively, with comparable or better accuracy and greater adversarial robustness.

  • Takeaways & Limitations

    Selecting minimal sentence context yields an efficient QA system that scales to large documents while preserving task performance and improving robustness to adversarial inputs.

Abstract

from arXiv · show

Neural models for question answering (QA) over documents have achieved significant performance improvements. Although effective, these models do not scale to large corpora due to their complex modeling of interactions between the document and the question. Moreover, recent work has shown that such models are sensitive to adversarial inputs. In this paper, we study the minimal context required to answer the question, and find that most questions in existing datasets can be answered with a small set of sentences. Inspired by this observation, we propose a simple sentence selector to select the minimal set of sentences to feed into the QA model. Our overall system achieves significant reductions in training (up to 15 times) and inference times (up to 13 times), with accuracy comparable to or better than the state-of-the-art on SQuAD, NewsQA, TriviaQA and SQuAD-Open. Furthermore, our experimental results and analyses show that our approach is more robust to adversarial inputs.

1 Introduction

Neural QA models are difficult to scale to long documents and vulnerable to adversarial inputs. The paper addresses both issues by selecting minimal sentence context before QA, finding that most SQuAD questions need only one sentence.

  • Full-document context modeling is inefficient for long or multiple documents and can cause models to focus on incorrect context under adversarial inputs.
  • 92% of answerable SQuAD questions can be answered using a single sentence.
  • The proposed sentence selector supplies the QA model with a question-dependent minimal set of sentences rather than a fixed number.
  • MINIMAL reduces training and inference times by up to 15× and 13×, respectively, while retaining accuracy comparable to or better than FULL across four QA datasets.
  • On SQuAD-Adversarial, MINIMAL outperforms the previous state-of-the-art model by up to 13% on the development set.

2 Task analyses

The analysis examines how much document context QA requires and finds that many questions can be answered from minimal context. These findings motivate a QA system that selects one or a few sentences rather than processing the full document.

  • Full-document context modeling is challenging and inefficient, motivating analysis of the minimal context needed to answer questions.
  • 98% of sampled SQuAD questions were answerable from the document, while 2% required information unavailable even in the full document.The latter cases included questions requiring background knowledge, such as Charles Dickens being an English Victorian author.
  • 85.1 F1 was achieved when DCN+ was trained and evaluated on an oracle sentence, compared with 83.1 F1 using the full document.
  • Accurately predicting the oracle sentence is identified as the basis for an efficient and robust QA system using only one or a few sentences.

3 Method

The method combines a sentence selector with a QA model, encoding document–question interactions and passing only high-scoring sentences to the reader. It uses variable sentence selection and three training techniques to improve selection accuracy.

  • Architecture: The overall architecture combines a sentence selector with a QA model that receives a reduced set of high-scoring sentences.Selection scores are computed in parallel before selected sentences are merged and passed to the QA model.
  • Sentence Selector: The sentence selector scores each sentence according to whether the question is answerable with that sentence.Its decoder uses sentence and question encodings to calculate bilinear similarities and produce answerability scores.
  • Architecture: The shared encoder computes sentence, question, and question-aware sentence embeddings before producing sentence and question encodings with an LSTM.The encoder is shared with S-Reader and uses document and question inputs.
  • Training: Three training techniques improve sentence selection: weight transfer, modified training labels, and score normalization.The encoder receives transferred weights, sentences yielding 0 F1 are treated as wrong, and scores are normalized across sentences from the same paragraph.
  • Sentence Selector: Thresholding sentence scores selects a question-dependent number of sentences rather than a fixed number.Changing the threshold dynamically controls the number of sentences selected during inference.

4 Experiments

Experiments evaluate sentence selection and QA across five datasets, showing that MINIMAL substantially improves efficiency while maintaining or improving accuracy and robustness to adversarial inputs.

  • Experimental setup: The experiments cover SQuAD, NewsQA, TriviaQA, SQuAD-Open, and SQuAD-Adversarial, using accuracy, MAP, F1, EM, and speed metrics.The study also measures selected-sentence counts and compares MINIMAL with FULL, oracle, Top k, and TF-IDF inputs.
  • SQuAD and NewsQA: Up to 5.6% MAP improvement comes from weight transfer, data modification, and score normalization, while Dyn selects more accurately with fewer sentences than Top k.On SQuAD, Dyn reaches 99.3 accuracy with 1.9 sentences per example, compared with Top 2 at 97.2 accuracy.
  • SQuAD and NewsQA: MINIMAL provides 6.7× training and 3.6× inference speedups on SQuAD, and 15.0× training and 6.9× inference speedups on NewsQA.Using S-Reader, MINIMAL remains comparable to FULL: 79.9 vs 79.8 F1 on SQuAD and 63.8 vs 63.2 F1 on NewsQA.
  • SQuAD and NewsQA: Dyn can select the minimum number of sentences needed for an answer, whereas Top k can select too few or too many and produce an incorrect answer.Some questions require combined context from multiple sentences rather than the oracle sentence alone.
  • TriviaQA and SQuAD-Open: On TriviaQA and SQuAD-Open, MINIMAL exceeds FULL on F1 and EM with inference speedup up to 13.8×, while Dyn outperforms TF-IDF selection.On the development-full set, Dyn reaches 59.5 F1 versus 51.9 F1 for TF-IDF with five sentences per question on average.

5 Related Work

Related work includes neural document QA, large-scale retrieval-based QA, and sentence-selection methods. The paper differs by targeting minimal variable-length context, model flexibility, and adversarial robustness.

  • Question Answering over Documents: Neural QA models commonly use coattention or bidirectional attention to represent codependent document-question context, but they remain sensitive to adversarial inputs.The cited prior work motivates reducing the context modeled by the QA system.
  • Question Answering over Documents: Large-scale QA approaches retrieve relevant documents or paragraphs, or use non-recurrent architectures to handle larger collections.These approaches address scale through retrieval or architectural simplification rather than the paper’s minimal sentence context.
  • Sentence selection: Sentence-selection research models sentence-question relevance and sometimes jointly selects sentences and answers questions.Raiman and Miller formulate extractive QA as iterative selection of sentences, answer starts, and answer ends.
  • Sentence selection: Compared with prior work, this paper selects a variable number of sentences per question, does not require end-to-end training, and evaluates robustness to adversarial inputs.The comparison also distinguishes its minimal-context objective from fixed-size sentence selection.

6 Conclusion

The paper concludes that minimal sentence selection yields an efficient and robust QA system for documents of varying sizes. Across five datasets, it preserves or improves accuracy while reducing computation and resisting adversarial inputs.

  • Conclusion: The proposed system selects a minimal set of sentences for QA, targeting scalability to large documents and robustness to adversarial inputs.This design follows the finding that most questions can be answered from a small sentence set.
  • Conclusion: Across five datasets, the method achieves up to 15× training and 13× inference speedups with accuracy comparable to or better than existing state-of-the-art.The conclusion presents efficiency and accuracy as joint outcomes across datasets with varying source-document sizes.
  • Conclusion: The experiments show that the approach is more robust to adversarial inputs.The conclusion treats adversarial robustness as a separate demonstrated outcome alongside efficiency and accuracy.

A Models Details

The system combines a sentence selector with a reader architecture and dynamically controls how many sentences are passed to the QA model. Its implementation uses standard neural training components and document-length truncation.

  • S-Reader: S-Reader separates encoding documents and questions from decoding answer-span start and end positions.The decoder scores answer positions using bilinear similarities between document and question encodings.
  • Model comparison: The architecture resembles DrQA’s Document Reader but uses different embedding procedures and hyperparameters.On SQuAD development data, S-Reader obtains 79.9 F1 versus 78.8 for DrQA’s Document Reader.
  • Training details: Models are implemented in PyTorch, trained with ADAM, and use tokenized documents with pretrained word and character embeddings.Documents are truncated to min(2000, max(1000, Lth)) words during training and evaluation.
  • Dynamic selection: Dyn orders sentences by selector scores and selects them dynamically according to a threshold.The threshold controls the number of selected sentences during inference, allowing an accuracy-speed trade-off.

B More Analyses

Additional analyses examine how many sentences answer questions, where FULL and MINIMAL fail, and how Dyn compares with alternative sentence-selection and QA systems.

  • Human analyses: 95% of sampled TriviaQA examples are answerable with one or two sentences, compared with 88% answerable using the full document.The analysis sampled 50 development examples from TriviaQA Wikipedia, whose documents average 488 sentences per question.
  • Error analyses: MINIMAL answers correctly more than 97% of the questions that FULL answers correctly.Figure 5 further breaks FULL and MINIMAL errors down by the sentence producing each model’s prediction.
  • Selection analysis: Dyn outperforms Top k in accuracy with a similar number of selected sentences on SQuAD and NewsQA.Figure 4 also shows how the number of selected sentences changes with the threshold.
  • Benchmark analyses: Tables 13 and 14 report TriviaQA Wikipedia and SQuAD-Open results, respectively, including comparisons with published systems.The TriviaQA comparison includes TF-IDF, Dyn, MEMEN, Mnemonic Reader, Reading Twice, and Neural Casecades; SQuAD-Open includes TF-IDF, Dyn, R3, and DrQA.
  • Error cases: On SQuAD, selector failures include choosing sentences containing “the Jin dynasty” instead of the oracle sentence.Other failures occur when the selector chooses the oracle sentence but the QA model predicts incorrectly or answers from another sentence.

C Full Results on TriviaQA and SQuAD-Open

On TriviaQA Wikipedia and SQuAD-Open, MINIMAL improves over FULL while also delivering substantial inference speedups and outperforming published state-of-the-art results.

  • Results: MINIMAL achieves higher F1 and EM than FULL, with inference speedup of up to 13.8×.This result is reported for TriviaQA Wikipedia and SQuAD-Open.
  • Results: MINIMAL outperforms the published state-of-the-art by 5.2 F1 on TriviaQA Wikipedia and 4.9 EM on SQuAD-Open.The comparisons are reported separately for the two datasets and metrics.

D Samples on SQuAD, TriviaQA and SQuAD-Adversarial

The paper provides indexed records for samples used in its human studies and analyses, covering the reported question sets.

  • Sample index: Table 15 lists the question IDs used for samples in the paper’s human studies and analyses.The table serves as an index for those sampled examples.
  • Sample index: The indexed samples support the paper’s reported human studies and analyses across the referenced datasets.The supplied passages identify the samples’ indexing role but do not provide individual sample outcomes.
  • Sample index: The sample index is presented separately from the paper’s experimental result tables.Table 15 contains question IDs, whereas the supplied passages identify no performance values for these records.
Loading 1805.08092v1…