PaperlayerView exact revision on arXiv

Source-linked AI summary

Large Language Models are Effective Text Rankers with Pairwise Ranking Prompting

Zhen Qin, Rolf Jagerman, Kai Hui, Honglei Zhuang, Junru Wu, Le Yan, Jiaming Shen, Tianqi Liu, Jialu Liu, Donald Metzler, Xuanhui Wang, Michael Bendersky

arXiv:2306.17563v2cs.IRcs.CLcs.LG

TL;DR

LLM-based text ranking has struggled to outperform fine-tuned rankers because pointwise and listwise prompts are difficult for off-the-shelf models. The paper introduces Pairwise Ranking Prompting, which compares documents two at a time and achieves strong results with moderate-sized open-source LLMs. Across standard benchmarks, PRP competes with or outperforms larger and commercial LLM-based approaches while offering efficiency variants.

  • Problem

    Off-the-shelf LLM rankers have shown limited success outperforming fine-tuned baseline rankers on text-ranking benchmarks, while existing pointwise and listwise formulations are difficult for LLMs.

  • Method

    Pairwise Ranking Prompting compares a query with a pair of documents, reducing ranking complexity and supporting generation and scoring APIs through several efficiency variants.

  • Results

    Across TREC-DL and seven BEIR datasets, PRP achieves competitive or state-of-the-art performance, outperforming RankGPT by 4.2% and pointwise LLM approaches by over 10% in general.

  • Takeaways & Limitations

    Moderate-sized, open-source LLMs can deliver strong ranking performance with pairwise prompting, including competitive results from efficiency-oriented variants.

  • Takeaways & Limitations

    The study focuses on open-source LLMs and empirical ranking results, does not test GPT models directly, and notes potential data leakage for LLM-based methods.

Abstract

from arXiv · show

Ranking documents using Large Language Models (LLMs) by directly feeding the query and candidate documents into the prompt is an interesting and practical problem. However, researchers have found it difficult to outperform fine-tuned baseline rankers on benchmark datasets. We analyze pointwise and listwise ranking prompts used by existing methods and argue that off-the-shelf LLMs do not fully understand these challenging ranking formulations. In this paper, we propose to significantly reduce the burden on LLMs by using a new technique called Pairwise Ranking Prompting (PRP). Our results are the first in the literature to achieve state-of-the-art ranking performance on standard benchmarks using moderate-sized open-sourced LLMs. On TREC-DL 2019&2020, PRP based on the Flan-UL2 model with 20B parameters performs favorably with the previous best approach in the literature, which is based on the blackbox commercial GPT-4 that has 50x (estimated) model size, while outperforming other LLM-based solutions, such as InstructGPT which has 175B parameters, by over 10% for all ranking metrics. By using the same prompt template on seven BEIR tasks, PRP outperforms supervised baselines and outperforms the blackbox commercial ChatGPT solution by 4.2% and pointwise LLM-based solutions by more than 10% on average NDCG@10. Furthermore, we propose several variants of PRP to improve efficiency and show that it is possible to achieve competitive results even with linear complexity.

1 Introduction

Off-the-shelf LLMs have struggled with text ranking, partly because pointwise and listwise formulations impose difficult ranking requirements. Pairwise Ranking Prompting (PRP) reduces this burden and achieves strong benchmark performance with moderate-sized open-source models.

  • Off-the-shelf LLM rankers have generally underperformed well-trained baseline rankers, while GPT-4-based ranking remains costly and access-limited.
  • Pointwise and listwise prompts are difficult because they require calibrated predictions or coherent document permutations.
  • PRP compares a query with a pair of documents, reducing task complexity, addressing calibration, and supporting generation and scoring APIs.
  • PRP achieves state-of-the-art ranking performance on standard benchmarks with moderate-sized, open-source LLMs.
  • PRP variants include efficiency improvements, with competitive performance reported across ranking datasets.

2 Difficulties of ranking tasks for LLMs

Existing pointwise and listwise ranking approaches burden LLMs with calibrated relevance estimation or permutation generation, producing unreliable outputs. Pairwise comparisons offer a simpler alternative because LLMs can handle relative document comparisons more readily.

  • Existing LLM rankers have limited evidence of outperforming fine-tuned smaller rankers, motivating analysis of pointwise and listwise methods.
  • Pointwise approaches: Pointwise methods include relevance generation and query generation, which score documents through generated relevance outputs or query likelihoods.
  • Pointwise approaches: Pointwise relevance approaches require calibrated predictions for sorting, and generation APIs may not expose the required log probabilities.
  • Listwise approaches: Listwise approaches insert document lists into prompts but frequently produce missing, rejected, repeated, or inconsistent rankings.
  • Listwise approaches: Listwise methods rely on generation APIs, can fall back to initial rankings after failures, and become sensitive to input ordering.
  • Pairwise prompting: Pairwise prompting reduces the comparison problem to relative judgments, yielding local document orderings or ties from consistent prompt decisions.

3 Pairwise ranking prompting

Pairwise Ranking Prompting (PRP) uses LLM comparisons between document pairs as its basic ranking operation, supporting both generation and scoring APIs. The paper develops aggregation and sorting-based variants that trade off robustness, efficiency, and sensitivity to input order.

  • 3.1 Prompting design: PRP uses a query and two documents in each prompt, forming a basic pairwise ranking unit for all variants.The unit is denoted u(q, d1, d2).
  • 3.1 Prompting design: The scoring API is supported because the prompt has only two expected outputs, “Passage A” and “Passage B”; the main results use scoring mode.Generation mode can instead produce irrelevant outputs, although the paper reports few prediction failures.
  • 3.1 Prompting design: PRP queries each document pair in both orders to reduce sensitivity to prompt ordering, producing a local order or a tie when decisions conflict.The two queries are u(q, d1, d2) and u(q, d2, d1).
  • 3.2 All pair comparisons: PRP-Allpair enumerates all document pairs and aggregates their pairwise preferences into a score for each document.A consistent preference gives one point to the preferred document, while conflicting or irrelevant generation outputs give each document half a point; ties fall back to the initial ranking.
  • 3.2 All pair comparisons: PRP-Allpair supports parallel API calls and is highly insensitive to input ordering, but requires O(N^2) calls for N documents.Its ranking is effectively based on win ratio.
  • 3.3 Sorting-based: PRP-Sorting uses LLM pairwise preferences as the comparator in Heapsort, providing guaranteed O(N log N) computation complexity.The paper selects Heapsort and names this variant PRP-Sorting.
  • 3.3 Sorting-based: PRP-Sorting lowers computation complexity relative to PRP-Allpair and remains largely insensitive to input order, despite pairwise comparisons not being guaranteed transitive.The paper reports robust empirical performance and leaves theoretically guaranteed alternatives for future work.
  • 3.4 Sliding window: PRP-Sliding-K performs K linear-time sliding-window passes, exploiting the focus on Top-K metrics to reduce computation for large document sets.Each pass compares and swaps adjacent pairs from the bottom of an initial ranking, while the method may depend strongly on input order; PRP-Sliding-10 performs well empirically.

4 Experiments on TREC DL datasets

On TREC-DL2019 and TREC-DL2020, PRP is evaluated by reranking BM25’s top 100 passages with accessible open-source LLMs. PRP achieves highly competitive results, including best performance on TREC-DL2020 and strong efficiency-oriented variants.

  • Experimental setup: PRP reranks the top 100 BM25 passages for each TREC-DL query using open-source FLAN-T5 and FLAN-UL2 models.Evaluated models have 3B, 11B, and 20B parameters, with PRP variants including Allpair, Sorting, and Sliding-K.
  • Main results: FLAN-UL2 PRP variants achieve the best results on all metrics on TREC-DL2020.On TREC-DL2019, they rank second to GPT-4 on NDCG@5 and NDCG@10.
  • Main results: Over 10%: the best PRP methods outperform text-davinci-003 RankGPT on all ranking metrics.They remain competitive with supervised methods across the ranking metrics.
  • Main results: Smaller FLAN-T5-XL and FLAN-T5-XXL models also produce competitive results with PRP.The reported results indicate that pairwise comparisons generalize beyond the 20B-parameter model.
  • Efficiency: Sliding-window PRP variants generally provide robust ranking performance and produce some of the best metrics.These variants address efficiency concerns associated with pairwise ranking approaches.

5 Experiments on BEIR datasets

On seven BEIR datasets, the authors apply the same TREC prompt template while reranking BM25’s top 100 passages. PRP remains robust across domains and outperforms competing unsupervised LLM rankers while performing favorably against supervised methods.

  • Experimental setup: The BEIR evaluation covers Covid, Touche, DBPedia, SciFact, Signal, News, and Robust04 using NDCG@10.The reported collection metric is the average NDCG@10 across datasets.
  • Experimental setup: The same prompt template is used for all BEIR datasets and unsupervised LLM baselines.This avoids designing separate prompts that require prior dataset-specific knowledge.
  • Main results: PRP variants based on FLAN-UL2 with 20B parameters achieve the best overall results on the BEIR collection.PRP also produces the best ranking metrics among unsupervised LLM methods on all datasets.
  • Main results: 4.2%: PRP outperforms RankGPT, while exceeding pointwise LLM-based solutions by over 10% in general.PRP-Sliding-10 with FLAN-UL2 outperforms RankGPT on all seven datasets.
  • Main results: PRP-Sliding-10 with FLAN-UL2 slightly outperforms RankT5 on average and exceeds it on 5 of 7 datasets.The comparison is against the reported supervised state-of-the-art RankT5 reranker.

6 Ablation studies

The ablations examine PRP’s robustness to input ordering and API mode, as well as sliding-window choices. PRP is generally stable across scoring and generation APIs, while some sliding-window variants preserve robustness with lower comparison cost.

  • Robustness to input ordering: PRP-Allpair is robust to inverted initial document ordering, whereas PRP-Sliding-1 loses robustness on metrics other than NDCG@1.PRP-Sliding-10 is also quite robust because it focuses on top-k ranking metrics.
  • Scoring and generation modes: PRP requires scores for only two candidate outputs, enabling comparison between scoring and generation API modes.The scoring-mode experiments use probabilities from open-source LLMs.
  • Scoring and generation modes: PRP is extremely robust across scoring and generation APIs, including with smaller LLMs.The authors attribute this to few generation mistakes caused by the simplicity of PRP.
  • Scoring and generation modes: 0.02%: only about this fraction of predictions fails to follow the desired output format.The authors contrast this low rate with listwise approaches.
  • Sliding-window study: The sliding-window study varies the number of passes and compares backward with forward passes.These additional experiments are reported in Appendix A.

7 Discussion

The discussion emphasizes PRP’s accessibility, reproducibility, extensibility, and efficiency trade-offs. The approach avoids commercial API dependence and is easy to reproduce, but reducing inference calls and distilling rankers remain open directions.

  • Extendability: PRP is designed for both commercial black-box and open-source white-box LLMs through a simple generation-API formulation.The authors note that richer prompts and white-box score information could further improve performance.
  • Reproducibility: The same prompt template is used across all 9 datasets, supporting the claimed generality of pairwise ranking prompting.Standard aggregation methods make the experiments easy to reproduce.
  • Cost and efficiency: PRP uses open-source LLMs accessible to academic researchers, reducing the need to call commercial APIs.The paper frames this as alleviating monetary constraints rather than eliminating efficiency costs.
  • Cost and efficiency: Further reducing LLM calls remains an open research direction, including through active learning.The authors also identify distillation into servable models as a future direction for large-scale systems.
  • Data leakage: Using open-source LLMs leaves minimal label-leakage concerns compared with black-box commercial LLMs.The paper states that comparisons among pointwise and listwise approaches using the same LLMs are fair.

8 Related Work

The paper situates PRP among unsupervised LLM-based ranking methods and distinguishes it from approaches that use LLMs only as auxiliary tools or rely on different settings. It argues that pairwise prompting applies a general paradigm to text ranking while achieving competitive performance with accessible models.

  • Prior LLM ranking work includes pointwise and listwise approaches in the challenging unsupervised setting without demonstrations, fine-tuning, or additional-model training.
  • Several related methods use LLMs to generate training data, explanations, or hypothetical documents rather than directly exercising LLM retrieval or ranking capabilities.
  • Pairwise prompting has also been studied for preference modeling, generation evaluation, and recommender systems, but those applications differ from standard text ranking.
  • PRP’s stated novelty is adapting the general pairwise paradigm to text ranking and making it competitive with fine-tuned models and methods using giant blackbox LLMs.

9 Conclusion

The paper concludes that pairwise prompting enables competitive text ranking with moderate-sized, open-sourced LLMs. It also reports robust performance across nine datasets while addressing efficiency, input-order sensitivity, and API compatibility.

  • PRP delivers very competitive ranking performance with moderate-sized, open-sourced LLMs across nine datasets.
  • PRP reduces the burden of ranking tasks for LLMs compared with existing pointwise and listwise formulations.
  • The paper discusses efficiency mitigations and reports support for both generation and scoring APIs as well as insensitivity to input ordering.

10 Limitations

The paper identifies scope and evaluation boundaries, including its focus on open-sourced models and empirical ranking results. It also leaves theoretically grounded noisy-comparison methods and broader model evaluation for future work.

  • The study focuses on open-sourced LLMs and does not use GPT models, although it compares against results from other papers.
  • The evaluation is mainly empirical, while theoretically grounded methods for sorting from noisy comparisons remain outside the paper’s main analysis.
  • Testing PRP on GPT models is identified as a meaningful benchmarking effort for future work.

A More results on PRP-Sliding-K

Additional experiments examine PRP-Sliding-K behavior and broader evaluation choices. Results support more backward passes for top-K metrics, while forward passes are less effective and less efficient; the study also bounds its dataset and prompting scope.

  • A More results on PRP-Sliding-K: Strong NDCG@1 can be achieved with one backward pass, while additional passes improve other Top-K ranking metrics, producing BubbleSort-like behavior.
  • A More results on PRP-Sliding-K: Forward pass performs poorly because it mainly demotes results and is less efficient at bringing good results to the top.
  • The paper evaluates standard relevance-based ranking datasets, excludes NFCorpus because reported BM25 metrics do not match public consensus, and uses one prompt template across nine datasets.
  • PRP’s experimental setup uses open-sourced FLAN models without observed ranking supervision from the evaluated datasets or instruction fine-tuning on ranking tasks.
Loading 2306.17563v2…