PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-Step Reasoning Over Unstructured Text with Beam Dense Retrieval

Chen Zhao, Chenyan Xiong, Jordan Boyd-Graber, Hal Daumé III

arXiv:2104.05883v1cs.CL

TL;DR

Complex question answering requires combining multiple evidence pieces, while approaches that rely on linked documents are difficult to extend to datasets without that structure. BEAMDR builds evidence chains from unstructured text through dense-space beam search and query composition. On HOTPOTQA, it is competitive with state-of-the-art systems using Wikipedia links.

  • Problem

    Complex question answering requires combining multiple evidence pieces into a chain, while standard retrieval over unstructured text often retrieves only one passage and link-based approaches assume text corpora have structure.

    Source: §1 · p. 1

  • Method

    BEAMDR uses dense-space beam search to retrieve evidence chains, iteratively appending retrieval history to the query and training representations to distinguish correct from hard negative chains.

    Source: §1 · p. 1, §2.2 · p. 2

  • Results

    On HOTPOTQA, BEAMDR’s full retrieval is competitive with state-of-the-art cascade reranking systems using Wikipedia links, and query composition captures hidden semantic relationships between evidence.

    Source: §1 · p. 2

  • Takeaways & Limitations

    BEAMDR finds reasoning chains in dense space without semi-structured information and is competitive with more complex state-of-the-art systems.

    Source: §6 · p. 5

    Many BEAMDR errors could be avoided by simple term matching.

    Source: §4.3 · p. 5

Abstract

from arXiv · show

Complex question answering often requires finding a reasoning chain that consists of multiple evidence pieces. Current approaches incorporate the strengths of structured knowledge and unstructured text, assuming text corpora is semi-structured. Building on dense retrieval methods, we propose a new multi-step retrieval approach (BeamDR) that iteratively forms an evidence chain through beam search in dense representations. When evaluated on multi-hop question answering, BeamDR is competitive to state-of-the-art systems, without using any semi-structured information. Through query composition in dense space, BeamDR captures the implicit relationships between evidence in the reasoning chain. The code is available at https://github.com/ henryzhao5852/BeamDR.

1 Introduction

Complex question answering needs evidence chains, but standard text retrieval often stops after one passage and link-dependent approaches are difficult to extend to datasets without links. BEAMDR searches for chains in dense representations, and on HOTPOTQA is competitive with state-of-the-art systems that use Wikipedia links.

1 Introduction

  • Complex questions require combining multiple knowledge pieces into an evidence chain, while standard information retrieval over unstructured text typically retrieves only one passage.Graph and database sources can use their inherent structure to form chains.Source: §1 · p. 1
  • Approaches that combine Wikipedia text with hyperlinks are difficult to extend to real-world datasets that lack links between documents.The passage gives medical records as an example.Source: §1 · p. 1
  • BEAMDR uses dense-space beam search to find candidate evidence chains and iteratively composes queries by appending retrieval history.It also trains the representation to distinguish hard negative chains from correct chains, refreshing negatives with the model.Source: §1 · p. 1
  • On HOTPOTQA, BEAMDR’s full retrieval is competitive with state-of-the-art cascade reranking systems that use Wikipedia links.With standard reranking and answer-span extraction, the full-retrieval gain also carries over to finding answers.Source: §1 · pp. 1–2
  • Query composition in dense space captures hidden semantic relationships between evidence pieces in a reasoning chain.The paper reports this as a property of BEAMDR’s iterative retrieval.Source: §1 · p. 2

2 BEAMDR: Beam Dense Retriever

BEAMDR uses dense retrieval to build ordered evidence chains from unstructured text. It scores chains using query representations that incorporate retrieval history, searches with a beam, and trains embeddings to distinguish relevant from negative chains.

2.1 Preliminaries

  • Dense retrieval embeds queries and passages as vectors and scores their similarity; approximate nearest-neighbor search over maximum inner product enables efficient retrieval.The paper denotes this retrieval operation as MIPS.Source: §2.1 · p. 2

2.2 Finding Evidence Chains with BEAMDR

  • Finding evidence chains is difficult when passages share few words, and missing one piece can lead to errors in later steps.The task is to form an ordered chain of passages from a corpus; labeled evidence is available during training, but not testing.Source: §2.2 · p. 2
  • At each retrieval step, BEAMDR appends the retrieved chain history to the question, retrieves the next passage, and multiplies its normalized relevance score into the chain score.The chain score is the product of its individual evidence pieces’ normalized relevance scores.Source: §2.2 · p. 2
  • Because enumerating all chains is computationally impossible, BEAMDR maintains a beam of the k best candidate chains and returns the k highest-scoring chains after n steps.The beam is updated at each retrieval step.Source: §2.2 · p. 2
  • Training learns embeddings that distinguish relevant from negative chains, defining a negative chain as one containing at least one passage outside the gold evidence set.Because evidence pieces are unordered, the method uses heuristics to infer chain order.Source: §2.2 · p. 2

3 Experiments: Retrieval and Answering

On HOTPOTQA fullwiki, the experiments evaluate BEAMDR’s passage-chain retrieval and downstream answer extraction. Full retrieval is competitive with a hyperlink-based state-of-the-art system, while answer extraction outperforms GRR but is slightly below MDR.

3.1 Experimental Setup

  • Experiments use HOTPOTQA fullwiki to evaluate passage-chain retrieval and downstream answer finding, reporting retrieval and answer-span metrics.Retrieval uses a BERT-base encoder and evaluates ten chains of two passages each; answer extraction is reported with BERT base and large.Source: §3 · p. 3, §3.1 · p. 3

3.2 Passage Chain Retrieval Evaluation

  • On full retrieval, BEAMDR is competitive with state-of-the-art GRR and retrieves better than MDR; reranking BEAMDR outputs with BERT outperforms all baselines.The retrieval evaluation targets evidence chains drawn directly from the corpus.Source: §3.2 · p. 3
  • Passage Exact Match initially improves with larger beam sizes, but marginal gains diminish beyond a beam size of forty.Source: §3.2 · p. 3

3.3 Answer Extraction Evaluation

  • BEAMDR outperforms GRR on answer extraction but is slightly lower than MDR.The comparison uses the same implementation on reranked chains; Table 2 reports HOTPOTQA dev and test answer EM and F1.Source: §3.3 · pp. 3–4, Table 2 · p. 4
  • The T-SNE visualization depicts query and passage embeddings across retrieval steps as BEAMDR hops through learned representation space.Source: Figure 3 · p. 4

4 Exploring How we Hop

Section 4 examines how BEAMDR constructs multi-hop evidence chains through representation-space analyses, hop-wise comparisons, and manual error analysis.

4.1 Qualitative Analysis

  • After first-hop passages are appended, second-hop queries land closer to second-hop passages; Table 3 also reports little overlap between retrieved passages across hops.The authors interpret this as BEAMDR mimicking multi-step reasoning by hopping in learned representation space.Source: §4.1 · p. 4, Table 3 · p. 4

4.2 Hop Analysis

  • Only 21.9% of BEAMDR’s top-10 chains are link-connected; it slightly leads on first-hop passages, while GRR leads on second-hop retrieval before link filtering.BEAMDR wins after links filter the candidates.Source: Table 3 · p. 4, §4.2 · p. 4

4.3 Human Evaluation on Model Errors and Case Study

  • In a manual analysis of 100 bridge questions, BEAMDR and GRR each predicted 50 correctly.The analysis categorizes model errors.Source: §4.3 · p. 4, Table 4 · p. 4
  • BEAMDR’s largest relative strength is identifying question-entity passages, where TF-IDF can falter when context overlap is limited to the entity name.Some GRR errors also arise from using reverse links to find second-hop passages.Source: §4.3 · p. 4
  • Many BEAMDR errors could be avoided by simple term matching, as illustrated by a question whose wording overlaps with the answer passage.The example concerns a question about a screenwriter and a passage naming the film’s writers and stars.Source: §4.3 · p. 5

5 Related Work

Prior multi-step retrieval work searches knowledge graphs or uses Wikipedia-based passage retrieval and links. BeamDR instead retrieves chains through dense representations without relying on corpus semi-structure.

5 Related Work

  • Knowledge-graph question answering searches chains directly in graphs, but can falter when graph coverage is sparse; related systems also retrieve chains from Wikipedia abstracts.The cited Wikipedia-abstract systems include Nie et al. (2019), Zhao et al. (2020b), Asai et al. (2020), and Dhingra et al. (2020).Source: §5 · p. 5
  • Wikipedia-based systems typically retrieve first-hop passages or entity mentions, then select next-hop candidates through links and rerank them; Asai et al. (2020) use beam search with a link-based graph neural network.Dhingra et al. (2020) treat Wikipedia as a knowledge graph, while Nie et al. (2019) and Zhao et al. (2020b) directly retrieve passages.Source: §5 · p. 5
  • BeamDR retrieves evidence chains through dense representations without corpus semi-structure, while Qi et al. (2019, 2020) iteratively compose queries but use sparse retrieval.The paper describes the sparse and dense approaches as complementary.Source: §5 · p. 5

6 Conclusion

BEAMDR finds reasoning chains in dense space using beam search and globally refreshed negative chains. It is competitive with more complex state-of-the-art systems without semi-structured information, while extending its learned connections to broader tasks remains future work.

6 Conclusion

  • BEAMDR is competitive with more complex state-of-the-art systems without using semi-structured information.Source: §6 · p. 5
  • Beam search and globally refreshed negative chains enable BEAMDR to find reasoning chains in dense space.Source: §6 · p. 5
  • Future work should investigate using within-question connections to address ambiguity, entity mentions, cross-modal concepts, and related questions.Source: §6 · p. 5
Loading 2104.05883v1…