PaperlayerView exact revision on arXiv

Source-linked AI summary

RankVicuna: Zero-Shot Listwise Document Reranking with Open-Source Large Language Models

Ronak Pradeep, Sahel Sharifymoghaddam, Jimmy Lin

arXiv:2309.15088v1cs.IRcs.CL

TL;DR

Zero-shot listwise LLM reranking has largely relied on proprietary models, raising reproducibility concerns. RankVicuna is an open-source 7B reranker whose effectiveness on TREC 2019 and 2020 is on par with GPT-3.5, though slightly behind GPT-4; its authors present it as a foundation for future reranking research.

  • Problem

    Prior zero-shot listwise LLM rerankers have relied on proprietary models, raising concerns about reproducibility and deterministic measurement.

    Source: §1 · p. 1

  • Method

    RankVicuna is an open-source LLM for zero-shot listwise reranking, trained with data augmentation that includes shuffled candidate orders.

    Source: §1 · p. 1, §3.2 · p. 4

  • Results

    On TREC 2019 and 2020, 7B RankVicuna achieves effectiveness on par with zero-shot GPT-3.5 reranking but remains slightly behind GPT-4.

    Source: §1 · p. 1

  • Takeaways & Limitations

    The authors share RankVicuna and its infrastructure as a foundation for further research on reranking with modern LLMs.

    Source: §7 · p. 8

    Data augmentation for robustness affects output quality under ideal conditions, creating an effectiveness–robustness tradeoff.

    Source: §1 · p. 2

Abstract

from arXiv · show

Researchers have successfully applied large language models (LLMs) such as ChatGPT to reranking in an information retrieval context, but to date, such work has mostly been built on proprietary models hidden behind opaque API endpoints. This approach yields experimental results that are not reproducible and non-deterministic, threatening the veracity of outcomes that build on such shaky foundations. To address this significant shortcoming, we present RankVicuna, the first fully open-source LLM capable of performing high-quality listwise reranking in a zero-shot setting. Experimental results on the TREC 2019 and 2020 Deep Learning Tracks show that we can achieve effectiveness comparable to zero-shot reranking with GPT-3.5 with a much smaller 7B parameter model, although our effectiveness remains slightly behind reranking with GPT-4. We hope our work provides the foundation for future research on reranking with modern LLMs. All the code necessary to reproduce our results is available at https://github.com/castorini/rank_llm.

1 Introduction

The paper addresses the reproducibility limits of zero-shot LLM reranking built on proprietary models by introducing RankVicuna, an open-source alternative. It reports competitive reranking and examines retrieval depth and data augmentation, including a robustness–effectiveness tradeoff.

1 Introduction

  • Prior zero-shot listwise LLM rerankers have relied on proprietary models, despite the value of open-source reranking infrastructure.The paper frames reproducibility and reliable, deterministic measurements as concerns with proprietary-model experiments.Source: §1 · p. 1
  • RankVicuna, a 7B open-source model, achieves effectiveness on par with zero-shot GPT-3.5 reranking on TREC 2019 and 2020, but remains slightly behind GPT-4.The authors also share the model checkpoints and code.Source: §1 · p. 1
  • Proprietary LLMs can rerank zero-shot, but their outputs may be non-deterministic or malformed, complicating their use in rigorous research.The paper contrasts these shortcomings with RankVicuna's deterministic, well-formed responses.Source: §1 · p. 1
  • With an effective first-stage retriever, reranking only the top 20 candidates in one pass yields improvement similar to reranking the top 100.RankVicuna consistently improves over the baseline retrieved results.Source: §1 · p. 1
  • Data augmentation improves robustness to shuffled candidates and variations in initial retrieval quality, but affects output quality under ideal conditions, creating an effectiveness–robustness tradeoff.The authors identify this tradeoff as a limitation of the augmentation techniques.Source: §1 · p. 2
  • The work aims to provide public models and infrastructure as a foundation for further reranking research.The authors hope this resource will stimulate exploration and innovation.Source: §1 · p. 2

2 Background and Related Work

This section situates reranking within multi-stage retrieval and traces the shift from trained transformer rerankers toward zero-shot LLM prompt-decoders. It distinguishes direct LLM reranking from work that uses LLMs indirectly to create training data or augment retrieval.

2 Background and Related Work

  • A retriever selects a small candidate set from a corpus, while a reranker further improves the resulting ranked list according to a retrieval metric.The section names nDCG and average precision as example metrics.Source: §2 · p. 2
  • Transformer reranking progressed from monoBERT cross-encoders to decoder-only and encoder–decoder models, which require many supervised query–passage pairs.MS MARCO is often used to supply such training data.Source: §2 · p. 2
  • Early transformer rerankers commonly estimated candidate relevance independently, while later pairwise and listwise losses with hard negatives improved high-precision reranking and alignment with first-stage retrieval.The section cites these later approaches as addressing limitations of pointwise ranking.Source: §2 · p. 2
  • RankVicuna targets zero-shot reranking without supervised task training and uses an LLM as a prompt-decoder; RankGPT3.5 serves as Vicuna's teacher because Vicuna cannot reorder document lists unaided.The paper calls this model class “prompt-decoders” and distinguishes it from BERT-style cross-encoders.Source: §2 · p. 2
  • Other LLM applications generate synthetic retrieval training data or augment queries, whereas RankVicuna uses an LLM directly as a reranker.The section also notes LLM-assisted approaches to retriever training and generative retrieval.Source: §2 · pp. 2–3

3 Methods

RankVicuna uses a Vicuna prompt for listwise reranking and trains the model on ranked lists generated by RankGPT3.5. Its training process includes malformed-output filtering and input-order augmentation; the untrained Vicuna baseline fails at reranking.

3.1 Prompt Design

  • The listwise task takes a query and candidate documents and returns reordered document identifiers to improve a retrieval metric such as nDCG.The paper motivates listwise ranking as enabling models to compare multiple documents simultaneously.Source: §3.1 · p. 3
  • The Vicuna prompt resembles RankGPT's but uses Vicuna's default system description and explicitly permits answers to depart from the identity ordering.Figure 1 presents the user input for both RankVicuna and the RankGPT replication.Source: §3.1 · p. 3, Figure 1 · p. 3

3.2 RankVicuna

  • RankVicuna is distilled from RankGPT3.5 orderings for 100K MS MARCO queries, each with 20 BM25 candidates, without human-labeled query–passage pairs.The authors therefore characterize the approach as zero-shot.Source: §3.2 · p. 3
  • Training excludes malformed teacher outputs, which account for about 12% of generations, and augments data with shuffled candidate orders while retaining original BM25 ordering.The authors apply this data augmentation in the paper's RankVicuna settings unless specified otherwise.Source: §3.2 · pp. 3–4
  • The 7B-parameter model is trained for two epochs with effective batch size 128 and learning rate 2 × 10^-5, taking roughly 80 hours on four RTX A6000 GPUs.The initial weights are Vicuna-7B-v1.5, instruction fine-tuned from LLaMA-v2.Source: §3.2 · p. 4
  • Out-of-the-box Vicuna fails at reranking, often returning the identity ordering or a malformed generation.RankVicuna is trained on RankGPT3.5-generated rankings rather than relying on the untrained model alone.Source: §3.2 · pp. 3–4

4 Experimental Setup

The experiments compare RankVicuna with unsupervised rankers and replicated proprietary prompt-decoders on TREC Deep Learning Track passage-retrieval test collections. They report nDCG@10 and MAP@100, with sliding-window reranking for top-100 lists and repeated runs for non-deterministic GPT models.

4 Experimental Setup

  • The comparisons include BM25, Contriever, replicated LRL with GPT-3.5, RankGPT with GPT-3.5 or GPT-4, and a Vicuna-based PRP-Sliding-10 variant.Vicuna is used for the PRP comparison to match the base LLM; the authors note that FLAN pretraining includes MS MARCO QA, which makes zero-shot results suspect.Source: §4 · p. 4
  • Evaluation uses TREC 2019 and 2020 Deep Learning Track passage-retrieval collections over the 8.8-million-passage MS MARCO v1 corpus, reporting nDCG@10 and MAP@100.The paper refers to the collections as DL19 and DL20.Source: §4 · p. 4
  • To rerank the top 100 candidates within context limits, the experiments use a sliding window of 20 documents with stride 10, matching RankGPT settings.Vicuna and GPT-3.5 have context size 4096; GPT-4 has context size 8192.Source: §4 · p. 4
  • GPT-3.5 and GPT-4 outputs are non-deterministic even at temperature zero, so results average six and three runs respectively, with 99% confidence intervals.GPT-4 runs are limited to three by the computation budget.Source: §4 · p. 4
  • The rerankers receive passages with [n] references replaced by (n), and inputs are preprocessed with ftfy's fix_text method.Source: §4 · p. 4

5 Results

Across DL19 and DL20, RankVicuna's effectiveness is comparable to RankGPT3.5, and its SPLADE++-based pipeline is on par with RankGPT4. The results also compare model size, prompt cost, and response formatting across rerankers.

5 Results

  • RankVicuna's 7B model matches 175B RankGPT3.5 overall, scoring higher on DL20 but slightly lower on DL19, and beats LRL on both tracks.The comparison is reported for reranking pipelines evaluated on DL19 and DL20.Source: §5 · pp. 4–5, Table 1 · p. 5
  • PRP-Sliding-10 (FLAN-T5-XXL) is comparable to RankVicuna in size and effectiveness, but requires about 40 prompts per passage versus RankVicuna's two.The prompt comparison uses a sliding window of 20 and stride of 10 for RankVicuna.Source: §5 · p. 5
  • RankGPT4 and PRP-Sliding-10 (FLAN-T5-UL2) outperform RankVicuna; the authors suggest model size and RankGPT3.5 teacher limits may contribute.The teacher-bound explanation is presented as a possibility, not a definitive cause.Source: §5 · p. 5
  • On SPLADE++ EnsembleDistil top-100 candidates, RankVicuna matches RankGPT4 effectiveness with a model more than two orders of magnitude smaller.This is the reported best system, using RankVicuna to rerank candidates from SPLADE++ EnsembleDistil.Source: §5 · pp. 4–5
  • RankVicuna's deterministic single run returned correctly formatted responses, whereas RankGPT3.5 and RankGPT4 occasionally returned malformed responses.RankGPT3.5 and RankGPT4 results were averaged across multiple runs.Source: §5 · pp. 5–6

6 Ablation Studies

These ablations show that reranking gains depend on first-stage candidates, while data augmentation trades some ideal-condition effectiveness for robustness. RankVicuna gains most of its progressive-reranking effectiveness early, with little additional boost from later passes.

6.1 First-Stage Candidate Generation

  • RankVicuna improves top-100 BM25 candidates by 30%–45% across metrics, versus 2%–4% for SPLADE++ ED candidates.Additional improvements from RankVicuna decrease as first-stage effectiveness increases.Source: §6.1 · p. 6
  • Reranking top 100 generally yields higher MAP@100, but with sufficiently effective first-stage retrieval, top-20 reranking can match top-100 nDCG@10.This parity is reported for DL19 rows 3–5 and DL20 rows 4–5.Source: §6.1 · p. 6

6.2 Data Augmentation

  • Without data augmentation, RankVicuna performs worse with BM25 but better with SPLADE++ ED, showing that augmentation can slightly reduce effectiveness depending on the first stage.The augmentation shuffles input documents and permutes teacher generations; Table 4 reports nDCG@10 and MAP@100 averages with 99% confidence intervals across six shuffled orderings.Source: §6.2 · pp. 6–7, Table 4 · p. 7

6.3 Effect of Progressive Reranking

  • RankVicuna outperforms PRPVicuna on DL19 and DL20; nDCG@10 rises from 0.5058 to 0.6837 on DL19 by pass two and from 0.4796 to about 0.6604 on DL20 by pass four.Later passes add little effectiveness for RankVicuna; a single pass takes about 30 seconds per query for both methods, while RankVicuna attends to 20 passages simultaneously and PRPVicuna handles two at a time.Source: Figure 2 · p. 7, §6.3 · pp. 7–8

6.2 Data Augmentation

  • On shuffled candidates, the no-augmentation model loses up to 34% effectiveness and has higher run-to-run variance, while the augmented model better retains effectiveness.The comparison uses shuffled candidate documents against each variant's ordinary ordering.Source: §6.2 · pp. 6–7

7 Conclusion

RankVicuna is an open-source, zero-shot listwise reranker, with effectiveness comparable to much larger models and stability compared with closed-source counterparts. The study also examines how first-stage retrieval and data augmentation relate to effectiveness and robustness.

7 Conclusion

  • RankVicuna is a zero-shot listwise reranking approach powered by the open-source Vicuna model.The paper reports effectiveness on par with much larger models and quantitatively demonstrates stability compared with closed-source counterparts.Source: §7 · p. 8
  • The study examines first-stage retrievers and identifies data augmentation as important for robustness to document shuffling and variations in initial retrieval quality.The conclusion frames fully open-source models as a foundation for further research on reranking.Source: §7 · p. 8
Loading 2309.15088v1…