PaperlayerView exact revision on arXiv

Source-linked AI summary

RankZephyr: Effective and Robust Zero-Shot Listwise Reranking is a Breeze!

Ronak Pradeep, Sahel Sharifymoghaddam, Jimmy Lin

arXiv:2312.02724v1cs.IR

TL;DR

Proprietary LLM rerankers create a reproducibility gap despite their effectiveness. RankZephyr addresses it with an open-source, zero-shot listwise reranker trained through strategic distillation choices, matching or sometimes surpassing RankGPT4 across evaluations, including uncontaminated NovelEval data.

  • Problem

    Proprietary rerankers’ non-reproducibility, non-determinism, and API opacity leave open-source models behind RankGPT4 in effectiveness.

  • Method

    RankZephyr is an open-source zero-shot listwise reranker trained through RankGPT4 distillation, ADA2 reorderings, variable windows, and shuffled inputs.

  • Results

    RankZephyr matches or occasionally surpasses RankGPT4 across evaluated datasets and outperforms proprietary counterparts on uncontaminated NovelEval data.

  • Takeaways & Limitations

    RankZephyr provides a reproducible, publicly available foundation for developing effective and efficient reranking models.

  • Takeaways & Limitations

    RankZephyr still underperforms its RankGPT4 teacher when reranking SPLADE++ ED outputs, showing it is not consistently better across settings.

Abstract

from arXiv · show

In information retrieval, proprietary large language models (LLMs) such as GPT-4 and open-source counterparts such as LLaMA and Vicuna have played a vital role in reranking. However, the gap between open-source and closed models persists, with reliance on proprietary, non-transparent models constraining reproducibility. Addressing this gap, we introduce RankZephyr, a state-of-the-art, open-source LLM for listwise zero-shot reranking. RankZephyr not only bridges the effectiveness gap with GPT-4 but in some cases surpasses the proprietary model. Our comprehensive evaluations across several datasets (TREC Deep Learning Tracks; NEWS and COVID from BEIR) showcase this ability. RankZephyr benefits from strategic training choices and is resilient against variations in initial document ordering and the number of documents reranked. Additionally, our model outperforms GPT-4 on the NovelEval test set, comprising queries and passages past its training period, which addresses concerns about data contamination. To foster further research in this rapidly evolving field, we provide all code necessary to reproduce our results at https://github.com/castorini/rank_llm.

1 Introduction

RankZephyr addresses the effectiveness and reproducibility gap between proprietary and open-source LLM rerankers through zero-shot listwise reranking. Evaluations examine training choices, reranking strategies, retrieval sources, ordering robustness, and performance on uncontaminated data.

  • Motivation: Proprietary LLM rerankers offer strong effectiveness but constrain reproducibility through non-deterministic outputs and API opacity.These limitations impede their utility in academic settings.
  • Approach: RankZephyr is a 7B-parameter open-source LLM designed to bridge or occasionally surpass RankGPT4’s zero-shot listwise reranking effectiveness.The model is publicly available and has deterministic behavior.
  • Reranking and retrieval: RankZephyr generally benefits from progressive reranking passes and improves over different first-stage retrieval models, while stronger candidate lists generally improve reranking quality.The study emphasizes that first-stage retrieval choices affect downstream reranking effectiveness.
  • Training: Strategic training choices, including RankGPT4 supervision, 5K ADA2 reorderings, variable window sizes, and shuffled inputs, improve effectiveness and robustness.These choices support reranking across different scenarios and candidate orderings.
  • Robustness: RankZephyr remains robust to shuffled initial document orderings, although initial ordering influences reranking effectiveness.This directly addresses sensitivity to candidate-list arrangement at inference time.
  • Evaluation: On NovelEval’s uncontaminated queries and passages, RankZephyr outperforms proprietary counterparts on the primary evaluation metric.The result evaluates performance beyond potential training-data contamination.

2 Background and Related Work

The paper situates RankZephyr within multi-stage information retrieval, contrasting pairwise and listwise reranking and building on prompt-decoder approaches. It addresses RankVicuna’s remaining effectiveness and variable-input limitations while studying candidate-generation effects.

  • Information retrieval: Information retrieval ranks a small set of highly relevant documents from a corpus, with relevance evaluated using metrics such as nDCG or average precision.Rerankers refine selections made by retrievers or preceding rerankers.
  • Pairwise reranking: Pairwise rerankers compare documents for a query and aggregate comparison scores into a single score for each document.This paradigm includes encoder-only and encoder–decoder rerankers such as duoBERT and duoT5.
  • Listwise reranking: Listwise reranking directly reorders document lists by relevance, and this study focuses on zero-shot settings without task-specific relevance pairs.The approach uses LLMs as rerankers in multi-stage pipelines.
  • Prompt-decoders: Prompt-decoders are LLM-based zero-shot rerankers introduced in contrast with BERT-style cross-encoders.RankVicuna distilled Rank-GPT3.5’s listwise effectiveness into a 7B-parameter Vicuna model.
  • Open-source reranking gap: RankVicuna still lagged behind RankGPT4 and was not trained for variable input-document counts, limiting generalization across candidate-list sizes.RankZephyr is presented as addressing both shortcomings.
  • Experimental scope: The experiments vary first-stage retrievers to examine their downstream effects rather than studying retrieval techniques themselves.The focus is on how candidate sources influence reranking.

3 RankZephyr

RankZephyr applies zero-shot listwise reranking by prompting an instruction-tuned LLM to reorder retrieved candidates, then trains it through teacher-generated reorderings. Its training combines varied candidate orderings, query selection, harder ADA2 candidates, variable window sizes, and progressive inference.

  • 3.1 Prompt Design: Given a query and candidate documents, RankZephyr reorders the candidates to optimize retrieval effectiveness.The listwise objective compares multiple documents jointly, with nDCG as an example target metric.
  • 3.2 Training Stage 1: RankZephyr distills teacher reorderings into a student model while preserving zero-shot operation because training uses no human-annotated relevance pairs.The initial phase uses RankGPT3.5-ranked top-20 BM25 passages from 100K MS MARCO v1 queries.
  • 3.2 Training Stage 1: Shuffled and original input orders expose the model to challenging reorderings while retaining an inference-like BM25 ordering.This augmentation changes ordering without requiring additional teacher-generated data.
  • 3.3 Training Stage 2: Distilling from GPT4: ADA2 candidate orderings create a harder training set while keeping the full pipeline zero-shot.The authors motivate this choice by analogy to hard negatives in cross-encoder training.
  • 3.3 Training Stage 2: Distilling from GPT4: Variable-length training on subsets of up to 20 passages addresses fixed-window limitations and supports reranking tasks with different input sizes.Subsets are randomly selected from the original input and ordered by the teacher.
  • 3.4 Inference: RankZephyrρ sequentially applies three sliding-window reranking passes, using each pass’s ranked list as the next pass’s input.The standard sliding-window setup uses a window size of 20 and stride 10 for top-100 reranking.

4 Experimental Framework

The experiments compare RankZephyr with sparse, dense, supervised, and proprietary rerankers across MS MARCO and BEIR collections. Evaluation uses standard top-100 candidate reranking and reports nDCG@10 and MAP@100, with progressive reranking treated as a separate setting.

  • Baselines: RankZephyr is evaluated against BM25, ADA2, SPLADE++ ED, RepLLaMA, RankGPT3.5, and RankGPT4.The baselines cover unsupervised and supervised sparse/dense rankers plus proprietary prompt-decoders.
  • Datasets: The evaluation spans TREC DL19–DL22, NEWS, and COVID, covering MS MARCO v1, MS MARCO v2, and BEIR collections.The datasets test both in-domain and broader task adaptability.
  • Evaluation Choices: The default RankZephyr model uses RankGPT3.5 followed by RankGPT4 training on 5K randomly sampled queries with ADA2 input orderings.RankZephyrρ uses the same model with three progressive reranking passes.
  • Evaluation Choices: Table 1 reports reranking pipelines over the top 100 results retrieved by the preceding first-stage model.Its nDCG@10 columns cover DL19, DL20, DL21, DL22, NEWS, and COVID.
  • Evaluation Choices: Effectiveness is measured with nDCG@10 and MAP@100, while nondeterministic RankGPT baselines are averaged over six or three runs.GPT3.5 and Zephyr use context size 4096; GPT4 uses 8192.

5 Results

RankZephyr achieves strong zero-shot reranking results across MS MARCO and out-of-domain collections, improving over open-source and, in selected settings, proprietary baselines. Progressive reranking generally delivers the strongest performance, but the model does not consistently exceed RankGPT4 on higher-quality candidate lists.

  • MS MARCO v1: RankZephyr establishes state-of-the-art MS MARCO v1 results and improves over pointwise RankLLaMA-13B despite using 7B parameters and no human labels.The comparison also holds when reranking RepLLaMA candidates.
  • MS MARCO v1: RankZephyr improves over RankGPT4 and PSC-aggregated RankGPT4, with relative gains as high as 15% on higher-quality RepLLaMA or SPLADE++ ED candidates.RankGPT4 is described as reportedly more than two orders of magnitude larger.
  • Progressive Reranking: Progressive reranking generally improves over the single-pass model and produces the best results, achieving state-of-the-art performance on DL19 and DL20.The comparison is between RankZephyrρ and its single-pass counterpart.
  • Out-of-Domain Evaluation: On MS MARCO v2, NEWS, and COVID, RankZephyr and RankZephyrρ remain effective on corpora and queries never exposed during training, substantially improving over SPLADE++ ED and RankVicuna.Progressive reranking also helps in these settings.
  • Limitations: On these out-of-domain datasets, RankZephyr still underperforms RankGPT4 when reranking SPLADE++ ED output.The authors identify this as evidence that more work is needed for consistent superiority over RankGPT4.
  • Limitations: The authors attribute part of the gap to RankZephyr’s 4096-token training context versus RankGPT4’s 8192-token context and lower Judged@10 scores by 2–4 points.The longer context may help RankGPT4 attend to longer biomedical abstracts and news articles.

6 Ablation Studies

The ablations examine how training choices, windowing, first-stage retrieval, candidate count, and input ordering affect RankZephyr’s reranking effectiveness and robustness. They also evaluate performance across out-of-domain and potentially uncontaminated data.

  • Teacher and Training Sources: RankZephyr’s full training setting uses 5K randomly chosen examples with RankGPT4 reorderings of ADA2 input orderings.Combining query-selection strategies with more queries produced no notable effectiveness improvement.
  • Variable Window Listwise Reranking: Variable window-size training generally improves effectiveness across first-stage retrievers and window configurations.RankZephyr trained only on window size 20 generalizes poorly to sliding windows of size 10.
  • First-Stage Retrieval Model: Better first-stage retrieval models produce better reranking effectiveness, while reranking stronger candidate lists yields diminishing returns.Top-100 BM25 reranking improves effectiveness by 45%–55% across metrics, compared with 7%–20% for SPLADE++ ED.
  • First-Stage Retrieval Model: RankZephyr outperforms RankLLaMA’s reranking of RepLLaMA candidates despite approximately 3% fewer judged documents in its top 10 ranks.The authors suggest listwise rerankers may elevate more unjudged passages than pointwise methods.
  • Data Augmentation: RankZephyr retains effectiveness with shuffled inputs, whereas RankGPT3.5 shows a marked decrease, indicating greater robustness to input-order perturbations.The authors connect this robustness to data augmentation during training.
  • Uncontaminated Data: NovelEval: NovelEval addresses possible benchmark contamination by using continuously updated queries and passages that have not been seen during training.NovelEval-2306 contains 21 questions from four domains and evaluates reranking of Google Search candidates.

7 Conclusions

The paper presents RankZephyr as an open-source model for zero-shot listwise reranking that matches or occasionally surpasses much larger proprietary models. Its experiments emphasize effectiveness, reproducibility, accessibility, robustness, and relevance to future retrieval-augmented generation systems.

  • RankZephyr is an open-source LLM optimized for zero-shot listwise reranking.
  • RankZephyr matches and occasionally surpasses the effectiveness of much larger proprietary models such as RankGPT4.
  • The work investigates progressive reranking, first-stage retrieval, instruction-finetuning choices, data augmentation, and performance on uncontaminated queries and passages.
  • RankZephyr supports future exploration of effective and robust reranking models as standalone components or parts of larger RAG pipelines.
Loading 2312.02724v1…