PaperlayerView exact revision on arXiv

Source-linked AI summary

Compositional Questions Do Not Necessitate Multi-hop Reasoning

Sewon Min, Eric Wallace, Sameer Singh, Matt Gardner, Hannaneh Hajishirzi, Luke Zettlemoyer

arXiv:1906.02900v1cs.CLcs.AI

TL;DR

Multi-hop RC datasets may fail to require multi-hop reasoning when evidence contains redundant facts or distinctive entity types. The paper analyzes HOTPOTQA with a single-hop BERT model and a human evaluation, finding that single-hop reasoning solves much of the benchmark and that dataset evidence and distractor selection are central concerns.

  • Problem

    Compositional questions do not necessarily require multi-hop reasoning when the provided evidence makes answers identifiable from one paragraph.

  • Method

    The paper analyzes HOTPOTQA using single-paragraph BERT, human evaluation with one gold paragraph withheld, and detailed analysis of evidence and distractors.

  • Results

    67.08 F1 from single-paragraph BERT is comparable to state-of-the-art performance in HOTPOTQA's distractor setting, showing that many questions are answerable with one hop.

  • Takeaways & Limitations

    Multi-hop datasets should carefully control the evidence and distractors so that question answering genuinely requires multi-hop reasoning.

  • Takeaways & Limitations

    Selecting small distractor sets that reliably induce multi-hop reasoning remains an open challenge, while larger evidence collections introduce computational or retrieval challenges.

Abstract

from arXiv · show

Multi-hop reading comprehension (RC) questions are challenging because they require reading and reasoning over multiple paragraphs. We argue that it can be difficult to construct large multi-hop RC datasets. For example, even highly compositional questions can be answered with a single hop if they target specific entity types, or the facts needed to answer them are redundant. Our analysis is centered on HotpotQA, where we show that single-hop reasoning can solve much more of the dataset than previously thought. We introduce a single-hop BERT-based RC model that achieves 67 F1---comparable to state-of-the-art multi-hop models. We also design an evaluation setting where humans are not shown all of the necessary paragraphs for the intended multi-hop reasoning but can still answer over 80% of questions. Together with detailed error analysis, these results suggest there should be an increasing focus on the role of evidence in multi-hop reasoning and possibly even a shift towards information retrieval style evaluations with large and diverse evidence collections.

1 Introduction

The paper argues that compositional questions do not necessarily require multi-hop reasoning because evidence can make answers identifiable from a single paragraph. On HOTPOTQA, single-hop models and humans solve substantially more questions than the intended multi-hop design suggests.

  • Motivation: Multi-hop reasoning depends on both question composition and the evidence provided, so compositional questions may remain single-hop when entity types or redundant facts identify the answer.The paper illustrates this with a question whose intended two-step reasoning is bypassed because only one paragraph describes an animal.
  • Analysis: Redundant facts and fine-grained entity types often provide strong signals that let a model locate answers using one hop.In the example, the presence of only one animal among the paragraphs makes the answer immediately identifiable.
  • Open challenge: More carefully chosen distractors could induce multi-hop reasoning, but selecting such distractors remains difficult and current alternatives are insufficient.Adversarial distractors mitigate the issue, yet a retrained single-hop model recovers most of its original accuracy; very large evidence collections create computational or retrieval challenges.

2 Related Work

The paper situates its analysis within multi-hop QA datasets built from knowledge bases or crowd-written questions, focusing on HOTPOTQA. It also distinguishes its human analysis and distractor-selection investigation from related work.

  • Multi-hop QA: Multi-hop QA datasets were introduced to foster reasoning across multiple pieces of text and commonly contain compositional or complex questions.The paper reviews large-scale reading-comprehension datasets before describing multi-hop QA as an extension aimed at multi-text reasoning.
  • Dataset construction: Existing multi-hop QA datasets are constructed either from knowledge bases, such as WIKIHOP and COMPLEXWEBQUESTIONS, or through crowd workers, such as HOTPOTQA.The paper identifies these as two principal dataset-construction approaches.
  • HOTPOTQA: HOTPOTQA consists of multi-hop questions written to require reasoning over two paragraphs.The paper focuses on HOTPOTQA as its main empirical setting.
  • Relation to prior work: Related work reports similar findings on HOTPOTQA, while this paper adds human analysis and examines the difficulty of selecting distractor paragraphs with current retrieval methods.The distinction concerns both understanding why single-hop reasoning succeeds and evaluating distractor selection.

3 Single-paragraph QA

The section evaluates a single-paragraph BERT model that scores paragraphs independently and selects an answer, including in distractor and open-domain HOTPOTQA settings.

  • Model: The model scores each paragraph independently and selects the answer from the paragraph with the best score.For each example, it runs single-paragraph BERT on paragraphs in parallel and selects the answer with the smallest yempty score.
  • Evaluation Settings: HOTPOTQA evaluation includes distractor and open-domain settings.The distractor setting supplies ten paragraphs, whereas the open-domain setting considers all of Wikipedia and retrieves paragraphs using bigram TF-IDF similarity.
  • Distractor Setting: 67.08 F1 in the distractor setting is comparable to state-of-the-art performance.This setting combines two gold paragraphs with eight TF-IDF-selected distractors.
  • Open-domain Setting: The open-domain setting yields 38.06 F1 after retrieving and reranking paragraphs.The model retrieves 30 paragraphs and selects eight using predicted yempty scores.

4 Compositional Questions Are Not Always Multi-hop

Human and model analyses show that many compositional HotpotQA questions can be answered without both intended paragraphs, while a minority genuinely require multi-hop reasoning. Single-hop success is often enabled by weak distractors, redundant evidence, or distinctive entity types.

  • Human Analysis: 82.06 F1 human accuracy with one gold paragraph versus 87.37 F1 with all ten shows only marginal degradation when one hop is hidden.The study sampled 200 bridge questions and compared nine visible paragraphs with the standard ten.
  • Bridge Questions: 27% of bridge questions require multi-hop reasoning because distractors contain multiple plausible entities of the relevant type.For these questions, the answer cannot be determined from one paragraph alone.
  • Bridge Questions: 35% of bridge questions allow single-hop answers because weak distractors make entity-type matching sufficient.Such questions may become multi-hop in open-domain settings with many entities sharing the relevant type.
  • Entity-Type Matching: Removing most question information lowers single-paragraph BERT from 67.08 F1 to 52.13 F1, supporting a substantial role for entity-type cues.The reduced questions retain only the first five tokens beginning with the wh-word.
  • Bridge Questions: 26% of bridge questions are compositional but solvable using only part of the question because one condition is redundant.These questions remain single-hop regardless of the distractors.
  • Bridge Questions: 8% of bridge questions are non-compositional and single-hop because one sentence contains all information needed for the answer.
  • Comparison Questions: Single-paragraph BERT achieves near-chance accuracy on multi-hop comparison questions, indicating that most are not solvable by single-hop reasoning.Comparison questions are categorized as multi-hop, context-dependent, or single-hop using automatic rules; 2% cannot be categorized.

5 Can We Find Better Distractors?

Adversarial and entity-type-matched distractors make single-paragraph BERT less accurate, but retraining on the changed distractor distributions restores much of its performance. Thus, the proposed distractors mitigate rather than eliminate single-hop shortcuts.

  • Adversarial Distractors: Adversarial selection chooses eight distractors from 50 TF-IDF-retrieved paragraphs using the lowest yempty scores from single-paragraph BERT.The selected paragraphs are those the model believes contain the answer and overlap the original distractors by 9.82%.
  • Adversarial Distractors: 67.08 F1 falls to 46.84 F1 on adversarial distractors, but retraining raises performance to 60.10 F1.The model is evaluated on adversarial distractors after training either on standard or adversarial distractors.
  • Type Distractors: Entity-type filtering reduces the original model to 40.73 F1, while retraining on the filtered adversarial distribution increases performance to 58.42 F1.Filtering keeps candidate distractors whose entity types match those of the gold paragraphs.
  • Interpretation: Single-paragraph BERT struggles when distractor distributions change but recovers some original accuracy after retraining on the new distribution.

6 Conclusions

The paper argues that multi-hop reasoning requires carefully designed evidence, because compositional questions can remain single-hop with weak or redundant distractors. It examines retrieval and distractor-selection alternatives, finding persistent limitations.

  • Evidence and distractors: Adversarial distractor selection mitigates weak-distractor effects, but retraining single-hop models recovers most original accuracy.The result indicates that the selected distractors remain insufficient to enforce multi-hop reasoning.
  • Open-domain retrieval: TF-IDF often fails to retrieve gold paragraphs even with 500 candidates in open-domain retrieval.This retrieval failure limits single-paragraph BERT when the needed evidence is absent.
  • Evidence and distractors: 500 distractors still yield 53.12 F1 for single-paragraph BERT, showing that stronger distractor collection remains difficult.The paper proposes improved distractor-collection methods for future multi-hop datasets.

A Example Distractor Question

The example demonstrates how a compositional question can be answered in one hop when the evidence contains a uniquely identifiable animal paragraph. The gold paragraphs jointly connect the reserve, bonobos, and the animal’s former name.

  • Question and evidence: The question asks for the former name of the animal whose habitat Lomako Nature Reserve protected.The example identifies Paragraphs 1 and 5 as the two gold paragraphs.
  • Answer evidence: Paragraph 1 states that the bonobo was formerly called the pygmy chimpanzee.It also identifies the bonobo as Pan paniscus and distinguishes it from the common chimpanzee.
  • Question and evidence: Paragraph 5 states that the Lomako Forest Reserve was established to protect the habitat of Bonobo apes.This paragraph supplies the animal needed to answer the question.
  • Distractors: The distractor paragraphs describe other reserves, protected areas, islands, or locations rather than the target animal.This evidence set includes paragraphs about Guadeloupe, La Désirade, Saint-Barthélemy, and related sites.

B Full Model Details

Single-paragraph BERT processes each question–paragraph pair independently, predicts answer type and span information, and selects the answer from the paragraph with the strongest no-answer score.

  • Pipeline: Single-paragraph BERT retrieves and answers from one paragraph at a time rather than reasoning across paragraphs.For HOTPOTQA, the model creates a batch containing one question paired with each context paragraph.
  • Paragraph selection: The final answer is chosen from the paragraph with the lowest yempty score.The model independently scores each paragraph before selecting the final answer.
  • Input representation: The question and paragraph are concatenated with [SEP] and fed into BERT-BASE.The sequence is S = [q1, ..., qm, [SEP], p1, ..., pn].
  • Answer classification: Max-pooling and learned parameters generate four scalars indicating span, yes, no, or no answer.The four outputs are yspan, yyes, yno, and yempty.
  • Answer span: A candidate answer span is computed separately from the classifier and represented by its predicted start and end positions.The resulting span is formed from the sequence positions between the predicted boundaries.
  • Implementation: Sequences longer than 300 tokens are split into multiple examples during implementation.The model uses Adam with learning rate 5 × 10^-5 and lowercases inputs.

C Categorizing Comparison Questions

The paper categorizes comparison questions by extracting the compared entities and assigning an operation, then uses that operation to distinguish multi-hop, context-dependent multi-hop, and single-hop cases.

  • Operation identification: Ten question operations are identified to cover comparison questions, with entities extracted from the question and supporting facts.The entities are extracted using the spaCy NER tagger before selecting the suitable operation.
  • Question categories: Numerical operations such as greater, smaller, and ordering are classified as always multi-hop.The categorization includes Is greater, Is smaller, Which is greater, and Which is smaller.
  • Question categories: And, Or, Is equal, and Not equal are context-dependent multi-hop operations.For example, an And question can be answered immediately if one entity fails the required property.
  • Question categories: Which is true and Intersection are classified as single-hop operations.The paper states that these operations can be answered using one hop.
Loading 1906.02900v1…