PaperlayerView exact revision on arXiv

Source-linked AI summary

Zero-Shot Listwise Document Reranking with a Large Language Model

Xueguang Ma, Xinyu Zhang, Ronak Pradeep, Jimmy Lin

arXiv:2305.02156v1cs.IRcs.CL

TL;DR

Supervised ranking methods rely on large quantities of relevance labels, while zero-shot rerankers do not use task-specific training. LRL directly reorders candidate documents as a list, and the paper reports gains over pointwise rerankers on TREC, further improvements when used as a final stage, and effectiveness on selected MIRACL languages.

  • Problem

    Supervised ranking methods require large amounts of relevance judgments, and zero-shot rerankers lack the task-specific training used to learn relative relevance between documents.

    Source: §1 · p. 1

  • Method

    LRL uses an LLM to take a query and candidate-document list and directly generate a reordered list of document identifiers.

    Source: §1 · pp. 1–2

  • Results

    LRL is around six nDCG@10 points above UPR on average across TREC DL datasets, improves top-20 PRL results by five nDCG@10 and seven MRR@10 points on average, and outperforms the hybrid baseline on three MIRACL datasets.

    Source: §5 · pp. 4–5

  • Takeaways & Limitations

    The results support using LRL both to rerank first-stage retrieval results and as a final-stage reranker for pointwise results, including on selected non-English datasets.

    Source: §1 · p. 2, §5 · pp. 4–5

    Potential training–test data intermixing in GPT models means the paper’s use of “zero-shot” should be treated with caution.

    Source: §5 · p. 4

Abstract

from arXiv · show

Supervised ranking methods based on bi-encoder or cross-encoder architectures have shown success in multi-stage text ranking tasks, but they require large amounts of relevance judgments as training data. In this work, we propose Listwise Reranker with a Large Language Model (LRL), which achieves strong reranking effectiveness without using any task-specific training data. Different from the existing pointwise ranking methods, where documents are scored independently and ranked according to the scores, LRL directly generates a reordered list of document identifiers given the candidate documents. Experiments on three TREC web search datasets demonstrate that LRL not only outperforms zero-shot pointwise methods when reranking first-stage retrieval results, but can also act as a final-stage reranker to improve the top-ranked results of a pointwise method for improved efficiency. Additionally, we apply our approach to subsets of MIRACL, a recent multilingual retrieval dataset, with results showing its potential to generalize across different languages.

1 Introduction

Supervised ranking pipelines have succeeded but depend on large labeled datasets, leaving open how to rank well without human relevance judgments. LRL uses a large language model to reorder candidate documents directly, and the paper reports gains over zero-shot pointwise rerankers and on multilingual retrieval.

1 Introduction

  • LRL addresses the open question of building an effective text-ranking pipeline without human relevance judgments or task-specific training data.The paper motivates this gap by noting that supervised methods rely on large amounts of labeled data.Source: §1 · p. 1
  • LRL takes a query and candidate-document list and directly generates a reordered list of document identifiers.The authors hypothesize that considering multiple documents together can improve on pointwise reranking.Source: §1 · pp. 1–2
  • LRL outperforms UPR by around 6 points nDCG@10 on average across three TREC DL datasets and BM25 by an average 15 points nDCG@10 on a MIRACL subset.The authors note potential training–test data intermixing in GPT models, while arguing their pointwise-versus-listwise claim is orthogonal to dataset-pollution concerns and MIRACL is relatively new.Source: §1 · p. 2

2 Background and Related Work

This section situates zero-shot ranking among lexical and unsupervised dense retrieval methods, then describes UPR as a zero-shot reranking pipeline and contrasts direct LLM reranking with LLM-based synthetic-data approaches.

2 Background and Related Work

  • Supervised neural retrievers require large amounts of task-specific human-labeled query–document pairs, motivating this paper’s zero-shot setting.The paper defines zero-shot as having no task-specific supervised training.Source: §2 · p. 2
  • BM25 is a lexical retriever that estimates query–document relevance using statistics such as term frequency and inverse document frequency.Contriever is an unsupervised dense retriever trained with momentum contrastive learning on pseudo query–document pairs cropped from raw texts.Source: §2 · p. 2
  • UPR reranks BM25 or Contriever results using a pretrained language model’s average log-likelihood of query tokens conditioned on each document.The paper describes BM25 or Contriever followed by UPR as the state-of-the-art zero-shot multi-stage pipeline at the time of the work.Source: §2 · p. 2
  • InPars and Promptagator use LLM-generated synthetic queries to create training pairs, while HyDE generates hypothetical documents to augment queries for dense retrieval.This work instead uses LLMs directly as rerankers in a multi-stage ranking pipeline.Source: §2 · p. 2

3 Methods

LRL replaces independent document scoring with listwise generation, prompting an LLM to return an ordered list of candidate identifiers. The method also specifies a pointwise LLM baseline and a sliding-window procedure for longer candidate lists.

3.1 Listwise vs. Pointwise Rerankers

  • LRL takes a query and multiple documents together, then returns their identifiers in a relevance-ordered list rather than scoring each document independently.The authors describe this listwise sequence-to-sequence output as letting the model consider candidate documents simultaneously.Source: §3.1 · pp. 2–3

3.2 Prompt Design

  • The listwise prompt asks the LLM to sort candidate passages by query relevance and complete an ordered list of passage identifiers.For the pointwise baseline PRL, the model answers True or False and the probability of True becomes the relevance score.Source: §3.2 · p. 3

3.3 Progressive Reranking

  • For lists longer than the model’s capacity, progressive reranking starts with the last m documents and shifts a window toward the head by m/2 each round.The procedure aims to promote the m/2 most relevant documents toward the head, but cannot completely reorder the full candidate list.Source: §3.3 · p. 3

4 Experimental Setup

The experiments compare zero-shot ranking pipelines on three TREC Deep Learning tracks and three selected MIRACL languages. The setup uses GPT-3 variants and reports ranking metrics for the TREC and MIRACL comparisons.

4 Experimental Setup

  • The evaluation compares BM25, Contriever, UPR, and PRL with LRL on TREC DL19, DL20, and DL21.DL19 and DL20 use the 8.8-million-passage MS MARCO v1 corpus; DL21 uses the 138-million-passage MS MARCO v2 corpus.Source: §4 · p. 3
  • MIRACL experiments cover Chinese, Swahili, and Yoruba, selected from its 18-language ad hoc retrieval dataset.The paper says the three languages were chosen to examine non-English high- and low-resource settings, due to cost.Source: §4 · p. 3
  • TREC experiments use text-davinci-003, with LRL reranking 10 passages at a time and using progressive reranking for larger lists.MIRACL uses code-davinci-002, which accepts twice as many input tokens; the paper says GPT-3 tends to over-tokenize many non-English languages.Source: §4 · p. 3
  • Table 1 reports nDCG@10 and MRR@10 across the three TREC Deep Learning tracks and distinguishes unsupervised from MS MARCO-trained supervised pipelines.Its Source column identifies the previous ranking stage, while bold marks the best unsupervised and supervised results.Source: Table 1 · p. 4
  • Table 2 reports nDCG@10 on three MIRACL datasets, with LRL reranking the top 20 results from the hybrid system in row (3).The supervised rows include mDPR trained on MS MARCO, with one variant further trained on Mr. TYDI.Source: Table 2 · p. 4

5 Results

LRL improves on zero-shot pointwise rerankers across the TREC DL datasets, and an additional LRL stage further improves top-ranked results. It also outperforms the hybrid baseline on three MIRACL datasets, while the authors caution that possible GPT training-test data intermixing complicates the zero-shot designation.

5 Results

  • Possible overlap between GPT training data and test data means the paper's zero-shot designation should be treated cautiously.The authors note that this potential issue also affects other work.Source: §5 · p. 4
  • LRL averages around six nDCG@10 points above UPR and three above PRL across the three TREC DL datasets.The experiments use BM25 as the first-stage retriever because it is more effective on DL19 and DL20.Source: §5 · p. 4
  • Reranking the top 20 PRL results improves nDCG@10 and MRR@10 by five and seven points on average, respectively.Using LRL as an additional stage on the top 10 or 20 UPR or PRL results avoids progressive reranking over an overly long list.Source: §5 · p. 4
  • LRL outperforms some supervised dense retrievers, including DPR, but remains less effective than more sophisticated supervised pipelines such as TCT-ColBERT.The supervised comparison includes methods trained on the MS MARCO training set.Source: §5 · p. 4
  • On all three MIRACL datasets, LRL reranking of the top 20 hybrid-retrieved results outperforms the hybrid baseline.The first-stage hybrid combines BM25 and mContriever.Source: §5 · p. 5

6 Conclusion

The paper proposes LRL, a listwise zero-shot reranker using a large language model. It reports improvements over zero-shot pointwise rerankers on web search datasets and demonstrates effectiveness across languages.

6 Conclusion

  • LRL is a listwise zero-shot reranker that uses a large language model for reranking.Source: §6 · p. 5
  • The authors report improvements over state-of-the-art zero-shot pointwise rerankers on web search datasets.Source: §6 · p. 5
  • The paper demonstrates LRL's effectiveness across languages.Source: §6 · p. 5

Related papers

Loading 2305.02156v1…