PaperlayerView exact revision on arXiv

Source-linked AI summary

Answering Open-Domain Multi-Answer Questions via a Recall-then-Verify Framework

Zhihong Shao, Minlie Huang

arXiv:2110.08544v2cs.CL

TL;DR

Open-domain questions can admit multiple valid answers, while rerank-then-read systems face evidence-selection and reading-budget problems. The paper proposes recall-then-verify, which separates reasoning for each answer, and reports state-of-the-art results on two multi-answer datasets.

  • Problem

    Open-domain questions can have multiple valid answers, while rerank-then-read must balance relevance and answer diversity, may filter useful evidence, and can create dependencies among answer predictions.

    Source: §1 · pp. 1–2

  • Method

    Recall-then-verify proposes answers from retrieved passages individually, aggregates relevant evidence for each candidate, and verifies each candidate with a large model.

    Source: §1 · p. 2

  • Results

    On two multi-answer datasets, the framework significantly outperforms rerank-then-read baselines and establishes new state-of-the-art records.

    Source: §7 · p. 9

  • Takeaways & Limitations

    The conclusion is that per-answer reasoning enables higher evidence usage and predictions mainly based on associated evidence, with less distraction from evidence for other gold answers.

    Source: §7 · p. 9

    The empirical analysis uses OPR as a representative reranker, which the authors note may still be far from perfect.

    Source: §4.4 · p. 4

Abstract

from arXiv · show

Open-domain questions are likely to be open-ended and ambiguous, leading to multiple valid answers. Existing approaches typically adopt the rerank-then-read framework, where a reader reads top-ranking evidence to predict answers. According to our empirical analysis, this framework faces three problems: first, to leverage a large reader under a memory constraint, the reranker should select only a few relevant passages to cover diverse answers, while balancing relevance and diversity is non-trivial; second, the small reading budget prevents the reader from accessing valuable retrieved evidence filtered out by the reranker; third, when using a generative reader to predict answers all at once based on all selected evidence, whether a valid answer will be predicted also pathologically depends on the evidence of some other valid answer(s). To address these issues, we propose to answer open-domain multi-answer questions with a recall-then-verify framework, which separates the reasoning process of each answer so that we can make better use of retrieved evidence while also leveraging large models under the same memory constraint. Our framework achieves state-of-the-art results on two multi-answer datasets, and predicts significantly more gold answers than a rerank-then-read system that uses an oracle reranker.

1 Introduction

Open-domain questions can be ambiguous and have multiple valid answers, exposing limits in rerank-then-read under fixed memory constraints. Recall-then-verify separates per-answer reasoning to use retrieved evidence while retaining large models, and reports new state-of-the-art records on two datasets.

1 Introduction

  • Ambiguous, open-ended questions can have multiple valid answers, and the answer set depends on both the question and relevant evidence.The passage reports that over 50% of a sampled set of Google search queries are ambiguous (Min et al., 2020b).Source: §1 · p. 1
  • Rerank-then-read must balance passage relevance and answer diversity, can filter out useful evidence, and can make one answer depend on evidence for other answers.These problems arise when a small reading budget is shared across answers and a generative reader predicts all answers in sequence.Source: §1 · pp. 1–2
  • Recall-then-verify predicts possible answers from each retrieved passage, aggregates evidence for each candidate, and verifies candidates with a large model.Separating each answer's reasoning is intended to avoid a shared reading-budget problem and make better use of retrieved evidence while retaining large models under the same memory constraint.Source: §1 · p. 2
  • The authors empirically analyze problems faced by rerank-then-read for open-domain multi-answer QA.Source: §1 · p. 2
  • The framework establishes new state-of-the-art records on two multi-answer QA datasets, with significantly more valid predictions.Source: §1 · p. 2

2 Related Work

Prior open-domain QA commonly assumes one answer or asks systems to predict one, despite reported ambiguity and multiple valid answers. This section situates AMBIGQA and distinguishes recall-then-verify from answer reranking.

2 Related Work

  • Earlier QA benchmarks generally assume one answer or require only one valid prediction, while AMBIGQA annotates as many valid answers as possible.Kwiatkowski et al. (2019) and Min et al. (2020b) report genuine ambiguity in open-domain questions.Source: §2 · p. 2
  • Rerank-then-read uses retrieval, passage reranking, and a reader, with the reranker especially important when memory limits a large reader to a few passages.The reranker must favor relevant passages while covering diverse answers; the authors report challenges with that framework for multi-answer QA.Source: §2 · pp. 2–3
  • Unlike answer rerankers that filter reader predictions or compare candidates, the answer recaller seeks high-recall possibilities and the verifier judges each candidate's evidence independently.The authors also note that another model could be added to filter false positives from the verifier.Source: §2 · p. 3

3 Task Formulation

The task asks a system to use corpus evidence to predict a set of valid answers for an open-ended question. Retrieval and reranking are evaluated for answer coverage, while QA performance is evaluated by F1.

3 Task Formulation

  • Given an open-ended question and a large text corpus, a system must use evidence to predict a set of valid answers.Source: §3 · p. 3
  • Questions and their corresponding answer sets are provided for training.Source: §3 · p. 3
  • MRECALL@k evaluates passage retrieval and reranking by whether the top-k passages cover at least k distinct answers, or all n answers when n is less than k.Source: §3 · p. 3
  • Question-answering performance is evaluated with F1 between gold answers and predicted answers.Source: §3 · p. 3

4 Rerank-then-Read Framework

The rerank-then-read pipeline retrieves passages, selects a small reranked set, and asks a generative reader to predict all answers from them. Analysis on AMBIGQA found that the reader predicted only 1,554 of 3,670 diverse gold answers covered by OPR and identified evidence-allocation and answer-dependency problems.

4.1 Passage Retrieval

  • Dense retrieval selects a set B of the most relevant passages from a much larger corpus for subsequent processing.The retriever measures relevance using the dot product of passage and question semantic vectors.Source: §4.1 · p. 3

4.2 Passage Reranker

  • JPR autoregressively selects passages to cover new answers while allowing repeated support when fewer than k answers are available.It jointly models passage selection using T5-base and tree decoding.Source: §4.2 · p. 3

4.3 Reader

  • The T5-3b reader encodes each selected passage with the question, then generates all answers in sequence.Its decoder attends to representations of all passages and separates answers with a [SEP] token.Source: §4.3 · pp. 3–4

4.4 Empirical Analysis

  • 1,554 of 3,670 diverse gold answers covered by OPR on AMBIGQA’s dev set were predicted by the reader.The analysis examines why relevant answers covered by the reranker may still be missed.Source: §4.4 · p. 4
  • Selected supporting-passage counts for predicted gold answers were widespread, making it difficult for the reranker to identify redundant or sufficient evidence in advance.Some redundant false-positive evidence could be replaced with passages covering other gold answers.Source: §4.4 · p. 4
  • Missed gold answers received 3.13 selected supporting passages on average versus 5.08 for predicted gold answers, despite more retrieved evidence being filtered out.OPR also used fewer retrieved supporting passages for missed answers.Source: §4.4 · p. 4
  • On 179 targeted questions, removing support for a predicted answer recovered a missed gold answer in 43.6%; 40.0% of successful cases had more relevant recovered-answer evidence.The analysis removed some supporting passages for a predicted answer without removing support for other gold answers.Source: §4.4 · p. 5

5 Recall-then-Verify Framework

Recall-then-verify proposes candidate answers from retrieved passages, gathers evidence separately for each candidate, and verifies them individually. This design separates answer reasoning while letting a large verifier use candidate-specific evidence under a memory constraint.

5.1 Overview

  • The framework recalls possible answers, aggregates retrieved evidence for each candidate, then verifies each candidate separately.Separating answer reasoning allows candidates to receive supporting passages individually and be judged mainly on their own evidence.Source: §5.1 · p. 5

5.2 Answer Recaller

  • The T5 recaller predicts answers from each retrieved positive passage and an “irrelevant” token for negatives, favoring high recall at low precision.A weak model can still achieve high recall from question and passage clues, but the candidates need verification against more evidence.Source: §5.2 · pp. 5–6

5.3 Evidence Aggregator

  • The evidence aggregator reranks passages for each question-candidate pair, prioritizing passages that cover the candidate and then semantic closeness.It selects the top-k passages for candidate verification.Source: §5.3 · p. 6

5.4 Answer Verifier

  • The verifier evaluates each candidate using its aggregated evidence and outputs candidates whose validity scores exceed threshold τ.The T5-3b verifier encodes passages independently and predicts “right” or “wrong.”Source: §5.4 · p. 6

6 Experiments

Experiments compare recall-then-verify with rerank-then-read baselines on two multi-answer datasets, then examine answer recalling, verification, errors, and inference efficiency. The framework outperforms the baselines on both datasets and is also evaluated on a lower-cost configuration.

6.1 Datasets–6.3 Implementation Details

  • The experiments use WEBQSP, repurposed for textual QA, and AMBIGQA, and compare against REFUEL and JPR.The system retrieves 100 passages and verifies candidates with k=10 passages; the verifier uses T5-3b and encodes up to 10 passages of at most 360 tokens under the stated memory constraint.Source: §6.1 · p. 6, §6.2 · p. 6, §6.3 · pp. 6–7

6.4 QA Results

  • The recall-then-verify framework outperforms state-of-the-art rerank-then-read baselines on both multi-answer datasets.The reported comparison is supported by Table 4; WEBQSP dev results are not directly comparable because the train/dev split differs.Source: §6.4 · p. 7, Table 4 · p. 7

6.5.1 Answer Recalling

  • On AMBIGQA, a T5-base recaller achieves high gold-answer coverage and competitive end-to-end performance, but the recaller alone still underperforms recall-then-verify.A small αneg reduces candidate counts without significantly lowering recall; increasing αneg improves precision but does not close the end-to-end gap.Source: §6.5.1 · p. 7, Table 5 · p. 7

6.5.2 Answer Verification

  • At k=10 and τ=0.5, the system predicts 31.7% more gold answers on all questions and 34.1% more on multiple-answer questions than OPR.Increasing k from 1 to 10 significantly boosts F1; τ controls the recall–precision balance. Under evidence-removal attacks, scores crossed the 0.5 threshold for 13.0% of missed candidates and 3.4% of predicted candidates.Source: Figure 4 · p. 7, §6.5.2 · p. 8, Figure 5 · p. 8

6.6 Error Analysis

  • Among 3,288 recalled gold answers, the verifier misses 1,242, while 80% of its “wrong” predictions are false negatives.For 76% of missed answers, the aggregator collects straightforward true-positive evidence; the authors conjecture that training to refute false negatives yields some unexpectedly low scores.Source: §6.6 · p. 8

6.7 Inference Efficiency

  • A faster T5-base recaller is less precise and produces more candidates, potentially increasing verifier workload unless αneg is raised.Inference cost can also be adjusted by tuning the number of verification passages k.Source: §6.7 · p. 9
  • With increased αneg and smaller k, the AMBIGQA dev configuration takes 1.88 seconds per question on a single V100 GPU and scores F1 50.7/38.2, versus Min et al. (2021)’s estimated 1.51 seconds and F1 48.5/37.6.Source: §6.7 · p. 9, Table 7 · p. 9

7 Conclusion

The paper analyzes rerank-then-read problems for open-domain multi-answer QA and proposes recall-then-verify, which reasons about each answer separately. On two multi-answer datasets, it reports new state-of-the-art results over rerank-then-read baselines.

7 Conclusion

  • Recall-then-verify separates answer reasoning, uses more retrieved evidence, and leverages large models under the same memory constraint; it significantly outperforms rerank-then-read baselines on two multi-answer datasets.The conclusion also describes predictions as more robust to distraction from evidence for other gold answers.Source: §7 · p. 9

A Implementation Details

The appendix describes the shared retrieval setup, reports retriever performance, and gives training and configuration details for the answer recaller and verifier.

A.1 Retriever

  • The retriever is fine-tuned on each multi-answer dataset using 100 retrieved passages per question, with answer-covering positives and passages covering no gold answer as hard negatives.The corpus is English Wikipedia from 2018-12-20, split into 100-word passages.Source: §A.1 · p. 12
  • The retriever underperforms DPR+ on MRECALL@5 and MRECALL@10, but its coverage of gold answers among 100 retrieved passages cannot be compared because DPR+ is unreleased.The reported retrieval results cover WEBQSP test and AMBIGQA dev.Source: §A.1 · p. 12, Table 8 · p. 12

A.2 Answer Recaller & Answer Verifier

  • The recaller is pretrained on NQ and fine-tuned on each dataset; αneg=0.1 shrinks candidate counts by nearly half without significant recall loss.The verifier is also pretrained on NQ and fine-tuned on the multi-answer datasets, with its threshold selected by dev-set F1.Source: §A.2 · p. 12

B.1 Single-Answer QA Result

The paper tests recall-then-verify on the single-answer NQ dataset by outputting the candidate with the highest validity score. It reports slightly better exact match than prior state-of-the-art rerank-then-read systems.

B.1 Single-Answer QA Result

  • On NQ, recall-then-verify slightly outperforms prior state-of-the-art rerank-then-read systems in exact match.For each question, the system outputs the candidate with the highest validity score.Source: §B.1 · p. 12, Table 9 · p. 12

B.2.1 Answer Recalling

On WEBQSP, weak recallers can cover answers but need verification, and T5-base trails T5-3b on test F1 (all) at αneg=0.1. Verification F1 varies little across τ, while lower τ predicts more gold answers.

B.2.1 Answer Recalling

  • A weak recaller can achieve high answer coverage, but stronger models improve precision and verification remains necessary; conservative negative filtering reduces verifier burden.The recaller alone has difficulty answering open-domain multi-answer questions.Source: §B.2.1 · pp. 12–13
  • 69.5/61.4 versus 72.1/63.3 recall at αneg=0.1 accompanied T5-base's lower F1 (all) than T5-3b on the test set.The comparison uses the same αneg=0.1; the authors conjecture T5-base may need a lower αneg to increase gold-answer coverage.Source: Table 10 · p. 13, §B.2.1 · p. 13
  • Table 10 reports WEBQSP recaller performance across models and αneg, with verifier-paired end-to-end QA results only in its final two rows.Source: Table 10 · p. 13

B.2.2 Answer Verification

  • F1 scores on WEBQSP are insensitive to a wide range of τ, while lowering τ helps predict more gold answers.Figure 6 reports verification performance at k=10; # Hit counts gold answers whose scores exceed a threshold.Source: Figure 6 · p. 13, §B.2.2 · p. 13

C Error Analysis

The error analysis concerns answer-verifier predictions on the AMBIGQA development set. Table 11 displays all annotated forms of gold answers, separated by semicolons.

C Error Analysis

Related papers

Loading 2110.08544v2…