PaperlayerView exact revision on arXiv

Source-linked AI summary

Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agents

Weiwei Sun, Lingyong Yan, Xinyu Ma, Shuaiqiang Wang, Pengjie Ren, Zhumin Chen, Dawei Yin, Zhaochun Ren

arXiv:2304.09542v3cs.CLcs.IR

TL;DR

The paper asks whether LLMs can directly rerank passages despite existing work emphasizing generation and differing ranking objectives. It introduces permutation generation, NovelEval, and permutation distillation to study effectiveness on known and unfamiliar information. GPT-4 surpasses supervised systems across nearly all datasets, while distillation yields a smaller model that outperforms monoT5 on BEIR.

  • Problem

    Existing IR work mainly uses LLMs for content generation, while direct passage ranking requires global relevance comparison and has been less studied.

  • Method

    The paper evaluates instructional permutation generation, introduces NovelEval for unfamiliar knowledge, and distills ChatGPT’s rankings into smaller specialized models.

  • Results

    GPT-4 surpasses supervised systems across nearly all datasets, and a 435M distilled student improves over monoT5 (3B) by 1.67 average nDCG on BEIR.

  • Takeaways & Limitations

    The findings support LLM-based passage reranking and indicate that permutation distillation can provide a more efficient specialized alternative.

  • Takeaways & Limitations

    The main analysis relies on proprietary ChatGPT and GPT-4, and reranking remains bounded by first-stage recall and sensitive to the initial passage order.

Abstract

from arXiv · show

Large Language Models (LLMs) have demonstrated remarkable zero-shot generalization across various language-related tasks, including search engines. However, existing work utilizes the generative ability of LLMs for Information Retrieval (IR) rather than direct passage ranking. The discrepancy between the pre-training objectives of LLMs and the ranking objective poses another challenge. In this paper, we first investigate generative LLMs such as ChatGPT and GPT-4 for relevance ranking in IR. Surprisingly, our experiments reveal that properly instructed LLMs can deliver competitive, even superior results to state-of-the-art supervised methods on popular IR benchmarks. Furthermore, to address concerns about data contamination of LLMs, we collect a new test set called NovelEval, based on the latest knowledge and aiming to verify the model's ability to rank unknown knowledge. Finally, to improve efficiency in real-world applications, we delve into the potential for distilling the ranking capabilities of ChatGPT into small specialized models using a permutation distillation scheme. Our evaluation results turn out that a distilled 440M model outperforms a 3B supervised model on the BEIR benchmark. The code to reproduce our results is available at www.github.com/sunnweiwei/RankGPT.

1 Introduction

This paper investigates whether instructed LLMs can directly rank passages for information retrieval and how their ranking ability can be transferred to smaller models. It proposes permutation generation, evaluates ChatGPT and GPT-4 on established and novel benchmarks, and studies permutation distillation.

  • Existing IR approaches mainly use LLMs to generate queries or passages rather than directly rank groups of passages for relevance.
  • Relevance re-ranking requires understanding user requirements, globally comparing passages, and ordering them by query relevance, creating a mismatch with common LLM generation objectives.
  • The paper asks how ChatGPT performs on passage re-ranking and how its ranking capabilities can be imitated by a smaller specialized model.
  • The proposed permutation generation approach instructs LLMs to output passage permutations directly, while sliding windows address context-length limitations.
  • Permutation distillation transfers ChatGPT’s rankings into a smaller specialized model using retrieved candidate passages and a RankNet-based objective.
  • GPT-4 surpasses supervised systems across nearly all datasets, while a 435M student improves over monoT5 (3B) by 1.67 average nDCG on BEIR.

2 Related Work

Prior work applies LLMs to information retrieval through generation, reranking, and capability distillation. The paper positions direct permutation generation as an alternative to methods dependent on output probabilities.

  • LLM-based retrieval includes text embeddings, differentiable search indexes, pseudo-document generation, and passage reranking.
  • Existing zero-shot reranking approaches use query generation or relevance generation, but their reliance on output log probabilities limits applicability to newer LLMs such as GPT-4.
  • Capability distillation has been used to transfer reasoning and generation abilities from large language models into smaller specialized models.

3 Passage Re-Ranking with LLMs

The paper uses LLMs to rerank retrieved passages by directly generating a relevance-ordered permutation. A back-to-first sliding-window procedure extends this approach beyond the model’s context limit.

  • Information retrieval uses multi-stage pipelines in which retrieval selects candidates and reranking produces a more precise ordering.
  • Previous LLM reranking methods have limited performance and depend heavily on model-output log probabilities.
  • Permutation generation asks an LLM to order identified passages by relevance to the query, producing a ranked list without an intermediate relevance score.
  • Because permutation generation can rank only a limited number of passages, the method applies sliding windows to rerank the retrieved set.
  • The sliding-window procedure reranks windows in back-to-first order, using window size w and step size s until all passages have been processed.

4 Specialization by Permutation Distillation

Permutation distillation transfers ChatGPT-generated passage orderings into smaller specialized ranking models. The method uses pairwise RankNet supervision and considers BERT-like and GPT-like architectures.

  • High deployment cost, latency, and unstable generation motivate distilling LLM reranking capabilities into smaller models.
  • The method directly uses model-generated permutations as distillation targets without consistency-checking or log-probability manipulation.
  • Training Objective: For each query, ChatGPT ranks 20 BM25-retrieved passages, producing a permutation that supervises the specialized model.
  • Training Objective: The specialized cross-encoder computes relevance scores for query-passage pairs, and RankNet optimizes their relative ordering.
  • The study evaluates BERT-like and GPT-like specialized model structures, including a DeBERTa-based cross-encoder and a LLaMA-based relevance generator.

5 Datasets

The evaluation uses three established retrieval benchmarks and the newly collected NovelEval test set. Together, these datasets cover diverse domains, multilingual low-resource retrieval, and questions designed to reduce benchmark contamination concerns.

  • The experiments use three benchmark datasets—TREC-DL, BEIR, and Mr.TyDi—and one newly collected test set, NovelEval.
  • TREC-DL uses the 2019 and 2020 competition test sets, containing 43 and 54 queries, respectively.
  • BEIR evaluation covers eight tasks spanning COVID-19, biomedical, argument, entity, claim-verification, tweet, news, and poorly performing-topic retrieval.
  • Mr.TyDi evaluates multilingual passage retrieval across ten low-resource languages, using the first 100 test samples from each language.
  • NovelEval-2306 contains 21 novel questions from four domains published after GPT-4's release, addressing possible prior knowledge of benchmark content.

6 Experimental Results of LLMs

GPT-4 with permutation generation outperforms supervised systems across major benchmarks and performs well on novel information, while results vary by language, instruction, and reranking configuration.

  • Results on Benchmarks: GPT-4 improves average nDCG@10 over monoT5 (3B) by 2.7 on TREC and 2.3 on BEIR.
  • Results on Benchmarks: 2.65 average nDCG improvement over mmarcoCE shows GPT-4 outperforming supervised systems across most Mr.TyDi languages.GPT-4 performs worse than mmarcoCE in Bengali, Telugu, and Thai.
  • Results on NovelEval: GPT-4 significantly outperforms monoT5 (3B) on NovelEval, while ChatGPT performs comparably to monoBERT on unfamiliar questions.NovelEval contains 21 novel questions and 420 passages that GPT-4 had not learned.
  • Compare Different Instructions: Permutation generation outperforms query and relevance generation, with GPT-4 exceeding ChatGPT at nDCG@10 despite comparable nDCG@1 performance.The comparison covers multiple instruction strategies and OpenAI API models.
  • Ablation Study on TREC: Performance is highly sensitive to initial passage order because BM25 provides a strong starting order for sliding-window reranking.The study compares BM25, random, and reversed-BM25 initial orders.
  • Ablation Study on TREC: More sliding-window passes may improve nDCG@10 but reduce top-passage performance, while reranking GPT-4 over ChatGPT’s top 30 improves accuracy at lower cost.In one configuration, nDCG@1 decreased by 3.88; the hybrid approach costs one-fifth of using GPT-4 alone.
  • Results of LLMs beyond ChatGPT: GPT-4 achieves the highest reranking performance among evaluated proprietary LLMs, whereas open-source models show a significant gap behind ChatGPT.The passage attributes the open-source gap possibly to the difficulty of generating permutations for 20 passages.

7 Experimental Results of Specialization

The paper distills ChatGPT’s passage-ranking behavior into specialized models and compares permutation distillation with supervised learning across model and data scales. Permutation-distilled models consistently outperform supervised counterparts, including a 435M DeBERTa model exceeding monoT5 on BEIR.

  • Scaling experiment: Figure 4 compares permutation-distilled and supervised DeBERTa models from 70M to 435M parameters and 500 to 10K training queries.The comparison covers TREC and BEIR performance across model sizes and training-data sizes.
  • Benchmark results: The specialized DeBERTa model achieves an average BEIR nDCG of 53.03, surpassing the 3B monoT5 state-of-the-art baseline.The paper attributes the result partly to more comprehensive ChatGPT relevance judgments than MS MARCO labels.
  • Benchmark results: Distilled specialized models surpass ChatGPT on both TREC and BEIR, which the paper associates with greater ranking stability.The paper reports that ChatGPT is unstable when generating passage permutations.
  • Scaling experiment: Permutation-distilled models consistently outperform supervised counterparts across all tested model and data sizes, especially on BEIR.All models use RankNet loss for a fair comparison.
  • Scaling experiment: Increasing model parameters improves ranking results more than increasing training data, while supervised-model performance varies unstably across scales.The paper suggests noisy MS MARCO labels may contribute to overfitting in supervised models.

8 Conclusion

The paper studies LLMs for passage re-ranking, introducing permutation generation, NovelEval, and permutation distillation. Experiments report capable ChatGPT and GPT-4 re-ranking and improved effectiveness and efficiency over supervised approaches.

  • Conclusion: The paper comprehensively studies passage re-ranking with large language models.It evaluates ChatGPT and GPT-4 across three benchmarks.
  • Conclusion: The proposed permutation-generation approach directly explores LLM capabilities for ranking passages.The method is presented as a central contribution of the study.
  • Conclusion: NovelEval is introduced to evaluate LLM ranking on unfamiliar knowledge.The test set is intended to further validate LLMs beyond potentially familiar benchmark content.
  • Conclusion: Permutation distillation demonstrates superior effectiveness and efficiency compared with existing supervised approaches.The conclusion presents distillation as the paper’s specialized-model approach.

Limitations

The study’s main analysis relies on proprietary ChatGPT and GPT-4, while its re-ranking results remain bounded by first-stage retrieval and sensitive to the initial passage order.

  • Model access: The primary analysis uses proprietary, non-open-source ChatGPT and GPT-4 models.Open-source models were also tested, but their results differed significantly from ChatGPT.
  • Task scope: Because the study focuses on re-ranking, its maximum ranking benefit depends on the recall of initial passage retrieval.The paper identifies first-stage retrieval as an upper-bound constraint on re-ranking performance.
  • Robustness: LLM re-ranking is highly sensitive to the initial passage order, which is usually determined by first-stage systems such as BM25.The paper calls for methods that improve robustness to this initial ordering.

Ethics Statement

The paper acknowledges ethical risks from using LLMs for ranking and documents its reliance on publicly accessed datasets and models. It also records implementation and evaluation details for the ranking methods and NovelEval.

  • Ethics and reproducibility: The authors state that the work uses publicly accessed datasets and models.This statement appears alongside their acknowledgment of the ACM Code of Ethics.
  • Risks and harms: LLMs may generate harmful, offensive, biased, or incorrect content, including racial, geographical, and gender biases in ranking results.The paper does not recommend LLM ranking for socially consequential tasks or critical decision-making.
  • Licensing: Distilled models inherit licensing constraints because they use ChatGPT, while distilled LLaMA models additionally carry LLaMA’s non-commercial license.These licensing conditions apply to the specialized models described in the paper.
  • Evaluation and implementation: The paper evaluates ranking methods using query-generation, relevance-generation, and direct permutation-generation instructions.The listed materials include prompts, model-specific instructions, and ranking-output formats.
  • Evaluation and implementation: Instructional query generation ranks passages using normalized log-probability scores for generating query tokens from passages.The score sums token-level log probabilities and is then used to rank passages.
  • Evaluation and implementation: Instructional relevance generation predicts Yes or No and normalizes the corresponding output probability into a relevance score from 0 to 2.The method relies on the probability of generating the selected answer.

C NovelEval-2306

NovelEval evaluates whether language models can rank newly collected, knowledge-intensive search passages, while the paper develops permutation-based ranking and distillation methods. The section also examines model stability, sliding-window settings, and API costs and latency.

  • NovelEval-2306: 21 questions from four domains form NovelEval, with 20 Google-retrieved passages per question manually labeled for relevance.Queries included recent hot topics, and annotators repeated the labeling process for consistency.
  • NovelEval-2306: All tested language models, including gpt-4-0314 and gpt-4-0613, achieved 0% question-answering accuracy on NovelEval.The result concerns answering the questions without retrieval augmentation.
  • Model stability: ChatGPT generated 14 duplicate passage identifiers across 97 TREC queries, while text-davinci-003 had the highest count of missing identifiers and GPT-4 had no duplicates.Missing passages were appended in their original order in the implementation.
  • Sliding window analysis: Window size 20 performed best for nDCG@10, whereas window size 40 performed best for nDCG@5.Larger windows may expand the ranking horizon but make long-context processing more difficult.
  • API cost and latency: The sliding-window permutation method requires 10 API calls to rerank 100 passages, averaging 11 seconds with gpt-3.5-turbo and 32 seconds with gpt-4.Latency depends on OpenAI service conditions, API versions, and network environments.

I Results of Specialized Models

Table 13 reports nDCG@10 results for specialized models on TREC and BEIR. It identifies both the best specialized system and the best overall system, with specialization based on MARCO or ChatGPT relevance judgments.

  • Results of Specialized Models: Table 13 compares specialized-model nDCG@10 results on TREC and BEIR, marking the best specialized and overall systems.Specialized models are fine-tuned on sampled queries using MARCO or ChatGPT relevance judgments.
Loading 2304.09542v3…