PaperlayerView exact revision on arXiv

Source-linked AI summary

Improving Passage Retrieval with Zero-Shot Question Generation

Devendra Singh Sachan, Mike Lewis, Mandar Joshi, Armen Aghajanyan, Wen-tau Yih, Joelle Pineau, Luke Zettlemoyer

arXiv:2204.07496v4cs.CLcs.IR

TL;DR

Open-domain question answering needs accurate passage retrieval, but standard retrieval typically postpones detailed token-level query–passage analysis. UPR addresses this with zero-shot question-generation likelihood scoring from a pre-trained language model, improving retrieval and open-domain QA results without task-specific re-ranking training.

  • Problem

    Text retrieval must identify useful passages before a reader can answer open-domain questions, while typical retrieval does not perform deep token-level document analysis.

  • Method

    UPR re-ranks passages using a pre-trained language model to score the likelihood of generating the input question conditioned on each passage, without task-specific training.

  • Results

    6%-18% points absolute: UPR improves top-20 retrieval accuracy across four QA datasets, and open-domain QA improves by up to 3 EM points on three benchmark datasets.

  • Takeaways & Limitations

    UPR can be applied on top of existing sparse or dense retrievers and further improves supervised retrievers and pre-trained readers.

  • Takeaways & Limitations

    Re-ranking large passage pools can have high latency because cross-attention complexity scales with question and passage tokens and PLM layers.

Abstract

from arXiv · show

We propose a simple and effective re-ranking method for improving passage retrieval in open question answering. The re-ranker re-scores retrieved passages with a zero-shot question generation model, which uses a pre-trained language model to compute the probability of the input question conditioned on a retrieved passage. This approach can be applied on top of any retrieval method (e.g. neural or keyword-based), does not require any domain- or task-specific training (and therefore is expected to generalize better to data distribution shifts), and provides rich cross-attention between query and passage (i.e. it must explain every token in the question). When evaluated on a number of open-domain retrieval datasets, our re-ranker improves strong unsupervised retrieval models by 6%-18% absolute and strong supervised models by up to 12% in terms of top-20 passage retrieval accuracy. We also obtain new state-of-the-art results on full open-domain question answering by simply adding the new re-ranker to existing models with no further changes.

1 Introduction

UPR adds a zero-shot re-ranker to passage retrieval, using a pre-trained language model for token-level question–passage analysis without task-specific data or tuning. Across QA datasets, this approach substantially improves retrieval and can outperform supervised dense retrievers.

  • Core contribution: UPR performs deep token-level query–passage analysis without task-specific data or tuning.It provides cross-attention between query and passage while avoiding task-specific training.
  • Core contribution: UPR re-scores retrieved passages by computing the likelihood of the input question conditioned on each passage.The method uses zero-shot question generation with a pre-trained language model.
  • Empirical results: UPR gains are consistent across retrievers and pre-trained language models, with instruction-tuned models such as T0 performing best as re-rankers.Ablation studies identify instruction tuning as beneficial for re-ranking.
  • Empirical results: UPR’s fully unsupervised retriever–re-ranker pipeline greatly outperforms supervised dense retrieval models such as DPR.The authors describe this as the first work to demonstrate this result to their knowledge.

2 Method

UPR re-ranks a retriever’s top-K passages by scoring how likely a pre-trained language model is to generate the question from each passage. It is zero-shot, retriever-agnostic, and uses cross-attention between question and passage tokens.

  • Retriever: UPR accepts the top-K passages returned by either sparse retrievers such as BM25 or dense retrievers such as DPR.The retriever supplies the candidate set that UPR reorders.
  • Unsupervised Passage Re-ranking: UPR reorders candidates using a relevance score derived from question-generation likelihood conditioned on each retrieved passage.The initial ordering is sorted by log p(q | z).
  • Unsupervised Passage Re-ranking: UPR incorporates cross-attention between question and passage tokens while requiring the model to explain every question token.This makes the re-ranker more expressive than dense retrieval alone.
  • Unsupervised Passage Re-ranking: The method assumes a uniform passage prior, reducing the ranking score to the question likelihood under each passage.The prior term is ignored because it is assumed uniform across retrieved passages.
  • Unsupervised Passage Re-ranking: A pre-trained language model estimates the average log-likelihood of question tokens conditioned on the passage, with no fine-tuning.The model is applied zero-shot after appending the instruction “Please write a question based on this passage.”

3 Experimental Setup

The experiments evaluate UPR across four open-domain QA datasets, diverse retrievers, and pretrained language models, using Wikipedia passages as evidence. They include robustness tests on keyword-centric and cross-domain collections.

  • Datasets: The evaluation uses SQuAD-Open, TriviaQA, Natural Questions, and WebQuestions for passage retrieval experiments.These are described as popular datasets used in prior passage-retrieval work.
  • Datasets: Evidence comes from a December 2018 English Wikipedia dump split into nonoverlapping 100-word passages, totaling over 21 million passages.
  • Robustness: Robustness is examined on keyword-centric datasets and questions from different domains, including Entity Questions, where dense retrievers have struggled relative to BM25.Entity Questions contains 22K short named-entity questions based on Wikipedia facts.
  • Robustness: The study also uses the BEIR Benchmark, which spans retrieval tasks and domains such as fact-checking, question answering, news, technical content, and Wikipedia.BEIR provides test queries, evidence documents, and relevance annotations across multiple datasets.
  • Retrievers: The experiments compare unsupervised retrievers BM25, MSS, and Contriever with supervised DPR and MSS-DPR.BM25 is keyword-based; MSS and Contriever are dense retrievers, while DPR and MSS-DPR are supervised dense retrievers.
  • Retrievers: BM25 ranks passages using keyword term frequency and inverse document frequency, while MSS predicts masked salient spans and Contriever learns through momentum contrastive training.

4 Experiments: Passage Retrieval

The experiments evaluate UPR for passage retrieval across datasets, retrievers, PLMs, candidate-pool sizes, and supervised-transfer settings. UPR consistently improves retrieval in many settings, while question generation and larger instruction-tuned PLMs are especially effective, with latency increasing as more candidates are reranked.

  • Experimental setup: UPR reranks top-1000 retrieved passages with T0-3B and evaluates conventional top-K retrieval accuracy.Top-K accuracy measures the fraction of questions whose top-K passages include a passage containing an answer-matching span.
  • Main results: 6%-18% absolute gains improve unsupervised models’ top-20 accuracy across four datasets, while supervised models gain up to 12%.Re-ranked Contriever also outperforms DPR by averages of 7% at top-20 and 4% at top-100 across all datasets.
  • Importance of question generation: Question generation p(q | z) substantially improves reranking over BM25 and MSS, whereas passage generation p(z | q) decreases retrieval accuracy versus baseline retrievers.The comparison uses the NQ development set with passages from BM25 and MSS and PLMs T0-3B and GPT-2.7B.
  • Pre-trained language models: T0 PLMs achieve the best top-K accuracy among compared rerankers, and scaling PLM size yields consistent improvements for T5-lm-adapt models.The comparison includes T5, T5-lm-adapt, T0, and GPT-neo models on NQ development passages from BM25 and MSS.
  • Candidate size and latency: Larger candidate pools improve top-20 accuracy but eventually plateau, while reranking latency increases linearly with the number of passages.The resulting accuracy-throughput tradeoff is evaluated on NQ development using BM25 passages and T0-3B.
  • Zero-shot supervised transfer: Supervised transfer with monoT5 strongly improves top-1 and top-5 accuracy over UPR, but UPR approaches or matches monoT5 when evaluating larger passage sets such as top-20 to top-100.The comparison reranks the top-1000 BM25 passages on NQ development, using monoT5 checkpoints fine-tuned on MS MARCO.
  • Main results: 8-20% absolute top-20 gains and 4-10% top-100 gains narrow the gap between BM25 and dense retrievers in open-domain question answering.The union of BM25 and Contriever outputs exceeds the current best results by 6% at top-20 and 3% at top-100.

5 Experiments: Question Answering

UPR is evaluated as a passage re-ranker for full open-domain QA by reranking retrieved passages before inference with pretrained FiD readers. Across retrievers and datasets, reranked passages improve exact-match performance without retraining the readers.

  • Experimental setup: The open-domain QA experiments evaluate SQuAD-Open, TriviaQA, and Natural Questions using exact match scores.The same evidence passages are used across the datasets.
  • Results: Reranked passages improve pretrained FiD performance for every retriever evaluated.The improvement is attributed to more accurate passages after reranking.
  • Results: FiD-large with reranked MSS-DPR passages achieves new state-of-the-art results, exceeding the pretrained FiD model by 1-3 EM points.This result is reported for the open-domain QA task in Table 7.
  • Implication: UPR provides performance gains through inference with reranked passages, without iterative retraining or expensive end-to-end training.The reader is trained beforehand and then used with the reranked passages during inference.
  • Caveat: Comparisons with prior work are difficult because the experiments use a larger, newer evidence set than the 2016 Wikipedia dump commonly used for SQuAD-Open.The authors retain a common experimental setup to reduce dataset-specific design choices.

6 Related Work

The paper relates UPR to generative pre-training, instruction tuning, query-likelihood document ranking, and standard retriever-reader architectures for open-domain QA. UPR differs from earlier ranking approaches by using pretrained language models without task-specific relevance training.

  • Relation to UPR: UPR builds on these lines of work by using off-the-shelf pretrained language models for passage reranking without finetuning.Its approach is presented alongside prior work on generative pre-training, query likelihood, and open-domain QA.
  • Generative Pre-training and Instruction Tuning: Generative pretrained transformers can follow natural-language instructions and perform tasks in zero-shot or few-shot settings.Instruction tuning further improves zero-shot performance by training across multiple tasks with task-specific instructions.
  • Document Ranking based on Query Likelihood: Query-likelihood ranking uses language models to compute relevance scores for queries, extending earlier count-based document models.Earlier count-based approaches can suffer from data sparsity, motivating later use of pretrained language models.
  • Open-Domain QA: Open-domain QA systems typically combine a retriever that identifies useful documents with a reader that generates a short answer.Retrievers commonly use dual encoders or multi-vector encoders for scalable search.

7 Conclusions and Future Work

The paper concludes that UPR is an unsupervised passage reranker that substantially improves retrieval and full open-domain QA when added to existing systems. It also identifies extensions to other retrieval tasks and task-specific instruction tuning as future directions.

  • Conclusion: UPR computes question-generation relevance scores with pretrained language models to perform unsupervised passage reranking.The method is framed as an approach for open-domain retrieval.
  • Conclusion: Across QA datasets, an unsupervised retriever-plus-UPR pipeline greatly outperforms strong supervised retriever models.UPR also improves supervised retrievers.
  • Conclusion: Using reranked passages with a pretrained reader achieves new state-of-the-art results on open-domain question answering.The paper reports this outcome without changes beyond inference with the reranked passages.
  • Future Work: Future work includes applying UPR to source-code retrieval and exploring instructions tailored to retrieval tasks such as similar-sentence search.The authors specifically mention prompt variations for the BEIR benchmark.

Limitations

UPR's main limitations concern reranking cost, dependence on the first-stage candidate pool, and sensitivity to the pretrained language model's training data.

  • Computational cost: Reranking a large passage pool can have high latency because cross-attention scales with question tokens, passage tokens, and PLM layers.The computational cost grows with the product of question and passage token counts and the number of model layers.
  • Retrieval dependence: UPR's maximum reranking performance is bounded by the first-stage retrieval results.For top-1000 candidates, top-100 reranking accuracy cannot exceed the top-1000 accuracy of the initial retriever.
  • Model dependence: UPR results may depend on the data used to train the PLM.The authors expect in-domain-trained PLMs to be more accurate for domain-specific retrieval or question-answering tasks than broad-coverage models.

Ethics Statement

The paper notes that deploying UPR with large language models carries fairness and safety risks. These risks include possible discrimination in retrieval and toxic or fabricated answers in downstream question answering.

  • The experiments use English Wikipedia as the evidence source.
  • Production deployment may produce re-ranked results that are not fair to all communities.The paper identifies potential discrimination and exclusion of marginalized groups as associated risks.
  • UPR may contribute to toxic or fake answers in question-answering applications.

A.4 BEIR Benchmark Results

On BEIR, UPR generally improves both BM25 and Contriever after re-ranking their top-1000 passages, though gains vary substantially by dataset and decline on several claim-based tasks.

  • BM25 improves on 12 of 15 BEIR datasets, while Contriever improves on 13 of 15 after re-ranking.The comparison uses NDCG@10 and Recall@100.
  • On average, NDCG@10 improves by 3-8% and Recall@100 by 5-6% after re-ranking.The performance gap between BM25 and Contriever also narrows.
  • The largest relative gains occur on datasets with information-seeking or question-form queries, while other datasets show moderate to little improvement.
  • Both retrievers lose performance on Fever and Climate-Fever, and BM25 also declines on Touche-2020.These datasets use statement-like claims, creating a different retrieval challenge for the re-ranker.

B.1 For all reported experimental results

The paper documents its mathematical setting, datasets, implementation resources, evaluation procedures, and reproducibility details. It also reports that repeated runs were infeasible for expensive open-domain QA experiments, so training used a shared seed.

  • The paper provides the mathematical setting and algorithm description in Section 2.
  • Table 8 compares instruction prompts for UPR on the NQ development set and reports better performance with simple instructions.The best results are highlighted in bold.
  • The authors provide source code, computing infrastructure details, model parameter locations, training configurations, and hyperparameter ranges.Experiments used V100 and A100 GPU infrastructure, with additional details reported across the paper and appendix.
  • Repeated open-domain QA runs were infeasible because experiments required tens of hours and substantial computing resources, so all training runs used seed 1234.The re-ranking experiments perform inference with open-source language models using a single prompt.
  • The datasets are open-source, in English, and use standard train, development, and test splits with statistics and preprocessing described in the paper.No new data were collected for this work.
  • Table 9 presents NQ examples comparing BM25 top-1 passages with UPR re-ranked passages from 1000 candidates.The table highlights answers and illustrates UPR's cross-attention-based passage ranking.
Loading 2204.07496v4…