Answering Complex Open-domain Questions Through Iterative Query Generation

1 Introduction

• One-step retrieve-and-read systems are limited on multi-hop questions when relevant context cannot be obtained in a single retrieval step.The introduction illustrates the problem with a missing entity that is not readily retrievable from the question alone.Source: §1 · p. 1
• Recent multi-hop QA datasets require evidence from multiple documents, but most existing work focuses on few-document settings with gold context provided.The paper motivates studying the harder open-domain setting, where evidence must be found in a large candidate pool.Source: §1 · pp. 1–2
• GOLDEN iteratively uses prior-hop retrieval results to generate natural-language queries and retrieve evidence, enabling interpretable open-domain multi-hop reasoning with off-the-shelf IR.The paper also presents a query-generation approach and a training procedure that uses minimal supervision to boost recall of gold supporting documents.Source: §1 · p. 2

2 Related Work

• Open-domain QA commonly retrieves passages using the question and then reads them, but one-step retrieve-and-read systems are ill-equipped when evidence is not retrievable from the question alone.Earlier systems used TF-IDF or improved the retrieval component with QA performance in mind (Nishida et al., 2018; Kratzwald and Feuerriegel, 2018; Nogueira et al., 2019).Source: §2 · p. 2
• QAngaroo and HOTPOTQA provide multi-hop QA tasks, but their few-document settings supply gold supporting facts alongside distractors.The paper focuses instead on HOTPOTQA's fullwiki setting, which it describes as truly open-domain and having more diverse questions.Source: §2 · p. 2
• Few-document multi-hop methods model entity graphs or score answer candidates, but can face scalability issues beyond a few dozen documents or candidates.Entity-centered graph expansion can also miss descriptive clues, while a distant-supervision neural retriever does not resolve the discoverability issue and lacks explainability.Source: §2 · pp. 2–3
• Iterative retrieval and question answering have also been proposed, but some approaches rely on fixed-schema logical forms or extra decomposition annotations.Talmor and Berant (2018) use questions based on logical forms, while Min et al. (2019b) require decomposition annotations and do not apply their approach to iterative retrieval.Source: §2 · p. 3

3.1 Problem Statement

• The task requires locating all gold supporting documents in a large corpus, where multi-hop evidence may not be retrievable from the original question alone.Unlike few-document QA, the open-domain setting requires the system to find the supporting documents itself.Source: §3.1 · p. 3

3.2 Model Overview

• GOLDEN Retriever iteratively uses gold-document information available during training to retrieve further supporting documents as reasoning unfolds.It generates natural-language queries from the original question and previously retrieved context, using text-based IR systems.Source: §3.2 · pp. 3–4, Figure 2 · p. 4

3.3 Query Generation

• A Document Reader is trained to select a context span as a search query, reducing the search space compared with generating free-form queries.At each step, the query is intended to retrieve the next supporting document.Source: §3.3 · p. 4

3.4 Deriving Supervision Signal for Query Generation

• Supervision uses lexical overlap between current context and intended supporting documents to form candidate queries, then selects an oracle query by retrieval recall.The overlap heuristics can capture entity names or descriptive text, and the resulting oracle queries provide supervised targets for the query generators.Source: §3.4 · p. 5
• 6.68% for d1 and 49.09% for d2: combined oracle-query R@5 improved recall over single-hop R@10 on HOTPOTQA dev.The comparison combines R@5 for both oracle queries, retrieving 10 documents across two queries, against R@10 for the single-hop query.Source: Figure 3 · p. 6, §3.4 · p. 6

3.5 Question Answering Component

• The final QA component processes paragraphs separately with shared encoder parameters and globally normalizes span scores, making representations paragraph-order-insensitive.It replaces the baseline model’s attention mechanisms with self-attention over concatenated question and context.Source: §3.5 · p. 6

4.1 Data and Setup

• The fullwiki evaluation used HOTPOTQA, with about 113k questions requiring two Wikipedia introduction paragraphs; retrieval was fixed at two steps with five documents per step.The authors left a stopping criterion for arbitrary reasoning steps to future work.Source: §4.1 · p. 7

4.2 End-to-end Question Answering

• GOLDEN located correct supporting facts better than CogQA and most top-ranking systems, while its QA performance was handicapped by not using pretrained contextualization models.The authors expected stronger QA models to boost performance and left their adoption to future work.Source: §4.2 · pp. 7–8
• 61.01% of dev questions had both gold documents retrieved by GOLDEN Retriever, versus 36.91% with Elasticsearch and 28.21% with the HOTPOTQA IR engine.The comparison uses the same GOLDEN QA component; Table 3 describes GOLDEN’s 10 paragraphs as five from each hop.Source: §4.2 · p. 8, Table 3 · p. 8
• Replacing G1 with its oracle only slightly improved end-to-end performance, while also replacing G2 produced a significant improvement.The authors attributed most performance loss to G2, which selects spans from a longer retrieval context.Source: §4.2 · p. 8, Table 4 · p. 8

4.3 Analysis of Query Generation

• Generated queries achieved 89.91% vs 87.85% retrieval performance for d1 and 61.01% vs 36.91% for d2, compared with oracle queries.On noncomparison questions, d2 recall increased from 27.88% to 53.23%.Source: §4.3 · p. 8
• Predicted queries were often small variations of oracle queries, but could be too short or select the wrong entity when multiple entities appeared in context.The authors describe the natural-language queries as making reasoning easier for humans to verify or intervene in.Source: §4.3 · pp. 8–9
• Span-selection query generation limits the kinds of queries and can yield undesired behavior, while the fixed number of query steps is a simplifying constraint.The authors cite ambiguous page names and oracle queries that use too much information from gold entities as examples.Source: §4.3 · p. 9

• GOLDEN Retriever improves recall of gold supporting facts through iterative reasoning and retrieval, giving the QA model better context for producing answers.The authors describe its performance as competitive with the state of the art.Source: §5 · p. 9

• The index includes Wikipedia page titles and introductory paragraphs from the HOTPOTQA-provided dump.Source: §A.1 · p. 12
• Titles use simple tokenization and lowercasing, while text uses a standard analyzer that also removes punctuation and stop words.Both fields include auxiliary bigram fields and asciifolding.Source: §A.1 · p. 12
• Search runs a multi-match BM25 query across fields and boosts title-related fields by 1.25.The default ranking score is the best field score.Source: §A.1 · p. 12

A.2 Reranking Search Results

• The heuristic retrieves at least 50 candidates and boosts scores when a document title exactly matches or is a substring of the query.This addresses cases where shared rare words cause naive full-text search to rank the desired page poorly.Source: §A.2 · p. 12
• Title-score boosting followed by reranking significantly improves these cases, placing the former president’s page first for “George W. Bush”.Table 7 reports recall percentages for Elasticsearch setups on the HotpotQA dev set using the original question.Source: §A.2 · p. 12, Table 7 · p. 12
• The title-match boost multiplies a document’s score by a heuristic constant between 1.05 and 1.5 before reranking.Source: §A.2 · p. 12

B Oracle Query Generation

• Overlap merging generalizes longest common subsequence and longest common substring by finding spans with high rates of overlapping tokens.The overlap rate is the number of overlapping tokens divided by the number of tokens considered in the span; stop words are ignored and remaining words lowercased.Source: B Oracle Query Generation · p. 12
• Candidate oracle queries combine cleaned questions, with or without punctuation, and cleaned document titles or paragraphs.Cleaning removes stop words and lowercases the remaining text.Source: B Oracle Query Generation · p. 12
• Candidate queries are tested in Elasticsearch, and queries that place the desired paragraph in the top five are further ranked using metrics such as query length.The resulting oracle queries are used to train the query generators.Source: B Oracle Query Generation · p. 13

C Training Details

• The query generators are trained on the training set to emulate the oracle queries.Source: §C.1 · p. 13
• The best query-generator model is chosen using F1 in span selection on the development set.Source: §C.1 · p. 13
• Hyperparameter experiments vary learning rate, training epochs, batch size, and the number of word embeddings to fine-tune.Final settings for both generators are reported in Table 8.Source: §C.1 · p. 13

C.2 Question Answering Model

• The QA component is trained on ten paragraphs from oracle queries, five per hop, with d1 and d2 replacing the lowest-ranked paragraph in each hop when absent.Source: §C.2 · p. 13
• The model follows Yang et al. (2018)’s baseline hyperparameters where possible, but uses Adam, a larger learning-rate patience, and gradient clipping at an ℓ2 norm of 5.Source: §C.2 · p. 13
• The QA model uses 10% dropout, hidden size 128, supporting-facts loss coefficient 10, and reads 10 paragraphs of up to 400 tokens each.Source: §C.2 · p. 13