Source-linked AI summary
Evidence Aggregation for Answer Re-Ranking in Open-Domain Question Answering
Shuohang Wang, Mo Yu, Jing Jiang, Wei Zhang, Xiaoxiao Guo, Shiyu Chang, Zhiguo Wang, Tim Klinger, Gerald Tesauro, Murray Campbell
TL;DR
Open-domain QA systems often extract answers from individual passages, although some questions require evidence combined across sources. This paper re-ranks baseline answer candidates with strength-based and coverage-based aggregation, achieving state-of-the-art results on three datasets. The approach advances performance while leaving harder reasoning and commonsense inference as future challenges.
Problem
Some open-domain questions require combining repeated or complementary evidence across multiple passages rather than extracting answers independently from single passages.
Method
The paper re-ranks baseline answer candidates using strength-based evidence aggregation and coverage-based aggregation across multiple passages.
Results
The models achieve state-of-the-art results on Quasar-T, SearchQA, and TriviaQA, with up to 8% F1 improvement on Quasar-T and SearchQA over previous best results.
Takeaways & Limitations
Aggregating strengthened and complementary evidence through re-ranking improves open-domain QA results across three public datasets.
Takeaways & Limitations
The methods address union or co-occurrence of passages but do not yet resolve harder open-domain QA requiring reasoning and commonsense inference.
Abstract
from arXiv · showhide
A popular recent approach to answering open-domain questions is to first search for question-related passages and then apply reading comprehension models to extract answers. Existing methods usually extract answers from single passages independently. But some questions require a combination of evidence from across different sources to answer correctly. In this paper, we propose two models which make use of multiple passages to generate their answers. Both use an answer-reranking approach which reorders the answer candidates generated by an existing state-of-the-art QA model. We propose two methods, namely, strength-based re-ranking and coverage-based re-ranking, to make use of the aggregated evidence from different passages to better determine the answer. Our models have achieved state-of-the-art results on three public open-domain QA datasets: Quasar-T, SearchQA and the open-domain version of TriviaQA, with about 8 percentage points of improvement over the former two datasets.
1 INTRODUCTION
The paper improves open-domain QA by re-ranking answer candidates using aggregated evidence from multiple passages, capturing both repeated and complementary support. It introduces strength-based and coverage-based re-rankers and reports state-of-the-art results on three datasets.
- Motivation: Open-domain QA can require aggregating evidence across passages because correct answers may be supported by repeated or complementary evidence.Repeated evidence can strengthen a candidate, while complementary passages can jointly cover multiple aspects of a question.
- Approach: The paper formulates multi-passage evidence aggregation as answer re-ranking over candidates generated by an existing QA system.The re-ranker uses global evidence features after candidate extraction rather than replacing the baseline candidate-generation stage.
- Approach: The strength-based re-ranker ranks candidates by how often their evidence occurs across passages and by aggregated candidate probabilities.It targets cases where multiple passages collaboratively reinforce the same answer.
- Approach: The coverage-based re-ranker concatenates contexts containing each answer and uses an attention-based match-LSTM to measure coverage of question aspects.This design targets candidates whose evidence is distributed across passages and jointly entails the answer.
- Results: The combined re-ranking approach achieves state-of-the-art results on Quasar-T, SearchQA, and TriviaQA, with up to 8% F1 improvement on Quasar-T and SearchQA.The reported gains are relative to previous best results.
2 METHOD
The method reranks answer candidates from a baseline open-domain QA system by aggregating evidence across passages. It combines strength-based signals with coverage-based matching of question aspects in concatenated evidence.
- Overall framework: The system retrieves passages, extracts top-K answer candidates with reading comprehension, then reranks those candidates using cross-passage evidence.The reranker reorders candidate spans so top-ranked answers are more likely to be correct.
- Strength-based reranking: Strength-based reranking favors answers supported repeatedly across passages, using occurrence counts or summed answer-span probabilities.Neither strength-based method requires training; they perform counting or probability aggregation over baseline predictions.
- Motivation: The coverage approach addresses cases where equal evidence counts cannot distinguish candidates whose passages differ in how completely they cover the question.This distinction targets complementary evidence distributed across multiple passages.
- Coverage-based reranking: Coverage-based reranking concatenates passages containing each answer into a pseudo passage and measures how well it matches the question.The model uses attention, comparison, and match-LSTM components to represent aspect-level and whole-question matching.
- Coverage-based reranking: The coverage model enhances question representations with answer information before matching each question aspect against the aggregated passage evidence.A further bidirectional LSTM aggregates aspect-level matching information, while max pooling produces an entire matching representation.
- Combining rerankers: The combined reranker softmax-normalizes outputs from two strength rankers and one coverage ranker, then selects the answer with the largest weighted score.The combination is intended to cover common cases handled by strength signals and harder cases requiring complementary evidence.
3 EXPERIMENTAL SETTINGS
Experiments use three public open-domain QA datasets with retrieved passages and compare reranking against established QA baselines. The study uses provided passages and baseline-generated candidate spans rather than retrieving additional passages.
- Datasets: Experiments evaluate Quasar-T, SearchQA, and open-domain TriviaQA using passages retrieved by search engines.The authors use the provided passages only and do not retrieve additional passages.
- Datasets: Quasar-T provides 100 sentence-level passages per question from ClueWeb09, while SearchQA provides about 50 Google snippets per question.Both datasets evaluate human performance with access to the same retrieved passages.
- Datasets: TriviaQA uses the open-domain setting with passages collected through Bing and sentence-level selection of the top 100 passages.The experiments use all information retrieved by the information-retrieval model.
- Baselines: Baselines include gated-attention, bidirectional-attention, answer-aggregation, and reinforced passage-selection reading-comprehension systems.R3 directly supports comparison because it generates the candidate answers used as the first stage of the proposed system.
- Candidate generation: A pretrained R3 model generates the top 50 candidate spans for training, development, and test data before further ranking.When the ground-truth answer is absent from training candidates, it is manually added to the candidate list.
4 RESULTS AND ANALYSIS
The experiments evaluate evidence-aggregation re-rankers across three open-domain QA datasets and several analyses. The coverage-based method is consistently strong, while candidate-list size and answer/question characteristics affect performance and remaining headroom.
- Overall results: TriviaQA evaluation uses filtered passages guaranteed to contain the correct answer, unlike the open-domain, passage-retrieval setting evaluated elsewhere.The leaderboard evaluation is passage-wise and therefore resembles closed-domain reading comprehension.
- Overall results: F1 56.0 and EM 50.9 on Wiki, plus F1 68.5 and EM 63.0 on Web, establish R3 as a competitive TriviaQA open-domain baseline.These results follow the leaderboard setting.
- Overall results: The full re-ranker significantly outperforms previous best performance, especially on Quasar-T and SearchQA, while coverage-based re-ranking performs consistently well across all three datasets.Coverage-based re-ranking is marginally below strength-based re-ranking on SearchQA.
- Performance analysis: Coverage-based re-ranking outperforms the baseline across answer lengths and question types, whereas strength-based counting is less stable across datasets.Strength-based probability remains close to baseline trends, likely because baseline probabilities dominate it.
- Performance analysis: The strength-based method performs significantly worse than coverage-based re-ranking on “why” questions, possibly because non-factoid answers yield fewer identical predicted spans.The two methods otherwise show similar trends across most question types.
- Re-ranking potential: A top-K baseline list can contain the correct answer even when top-1 accuracy is low, but re-ranking remains about 10% below the top-5 upper bound on both F1 and EM.This gap indicates further potential for re-ranking improvements.
- Candidate-list size: For coverage-based re-ranking, increasing K improves performance from low-coverage K=3, while top-5 and top-10 are comparable; the paper therefore reports K=5.The higher coverage of top-10 offsets the greater difficulty of ranking longer lists, without significant advantage over K=5.
- Candidate-list size: For strength-based counting, K=50 performs best, whereas performance drops at K=200 because more incorrect answers acquire high counts.The method reports K=50 for testing, and varying K has little effect on its computation speed.
5 RELATED WORK
The paper situates its approach within search-and-read open-domain QA and prior multi-step reading-comprehension methods. It introduces multi-passage evidence aggregation and re-ranking as a distinct focus from earlier single-passage or single-span approaches.
- Open Domain Question Answering: Open-domain QA systems retrieve passages and use reading-comprehension models to extract answers, but retrieval introduces noise without passage-level entailment annotations.Recent systems follow a search-and-read pipeline in which a document retrieval module supplies passages to RC models.
- Novelty: This work models relationships between questions and multiple passages, whereas earlier answer-re-ranking features were extracted from single passages.The authors characterize their work as the first to improve neural open-domain QA through multiple-passage evidence aggregation.
- Multi-Step Approaches for Reading Comprehension: The proposed re-ranking methods are introduced for neural open-domain QA and multi-passage reading comprehension.The two-step approach uses additional information or refines answer predictions beyond standard answer extraction models.
- Multi-Step Approaches for Reading Comprehension: Earlier multi-step systems differ by combining token-level representations, passing extractor probabilities directly, or using only one span from one passage.These design choices limit handling of redundant and complementary evidence, phrase answers, partial passages, or multiple evidence aggregation.
6 CONCLUSIONS
The paper reports that combining evidence from multiple retrieved passages improves open-domain QA. Its two re-rankers address consistent and complementary evidence, while harder reasoning and commonsense problems remain.
- Conclusions: Explicitly combining evidence from multiple retrieved passages improved open-domain QA, and the two re-rankers helped individually and more together.The authors report considerable advances over the state of the art on three open-domain QA datasets.
- Conclusions: The strength-based and coverage-based re-rankers target consistent and complementary evidence, respectively.The conclusion distinguishes the two re-rankers by the type of multi-passage evidence they model.
- Limitations and Future Work: The methods model union or co-occurrence of multiple passages, but harder QA problems still require reasoning and commonsense inference abilities.The authors identify these capabilities as directions for future work and possible extensions of their approach.