Source-linked AI summary
Beyond Yes and No: Improving Zero-Shot LLM Rankers via Scoring Fine-Grained Relevance Labels
Honglei Zhuang, Zhen Qin, Kai Hui, Junru Wu, Le Yan, Xuanhui Wang, Michael Bendersky
TL;DR
Binary prompts can struggle with partially relevant documents, while human relevance judgments often use multiple levels. This paper adds fine-grained labels or rating scales to pointwise LLM rankers and derives ranking scores from their likelihoods. Across eight BEIR datasets, the approach significantly improves ranking performance and appears to help LLMs distinguish subtle relevance differences.
Problem
Binary relevance prompts may provide inadequate choices for partially relevant documents, whose relevance can be difficult to classify accurately.
Method
The paper prompts pointwise LLM rankers with fine-grained relevance labels or rating scales and aggregates label likelihoods into ranking scores.
Results
Across eight BEIR datasets, fine-grained prompts significantly improve LLM ranking performance over binary labels, including datasets with binary ground-truth labels.
Takeaways & Limitations
Fine-grained labels may guide LLMs to distinguish subtle relevance differences, especially among documents ranked near the top.
Takeaways & Limitations
The method assumes access to generated-text likelihoods, which is unavailable through some proprietary LLM APIs, and the study evaluates ranking rather than relevance calibration.
Abstract
from arXiv · showhide
Zero-shot text rankers powered by recent LLMs achieve remarkable ranking performance by simply prompting. Existing prompts for pointwise LLM rankers mostly ask the model to choose from binary relevance labels like "Yes" and "No". However, the lack of intermediate relevance label options may cause the LLM to provide noisy or biased answers for documents that are partially relevant to the query. We propose to incorporate fine-grained relevance labels into the prompt for LLM rankers, enabling them to better differentiate among documents with different levels of relevance to the query and thus derive a more accurate ranking. We study two variants of the prompt template, coupled with different numbers of relevance levels. Our experiments on 8 BEIR data sets show that adding fine-grained relevance labels significantly improves the performance of LLM rankers.
1 Introduction
Pointwise zero-shot LLM rankers commonly use binary relevance labels, which may underspecify partially relevant documents. The paper introduces fine-grained labels as prompts and reports improved ranking across eight BEIR datasets.
- Binary relevance prompts can misclassify documents that do not directly answer a query but still contain helpful information.
- Human relevance judgments often use multiple levels, motivating analogous fine-grained options for zero-shot LLM rankers.
- Fine-grained labels such as “Highly Relevant,” “Somewhat Relevant,” and “Not Relevant” provide intermediate choices whose likelihoods produce ranking scores.
- Across eight BEIR datasets, adding intermediate labels significantly improves ranking, including datasets with binary ground-truth labels.
2 Related Work
Related work covers zero-shot LLM rankers that score documents pointwise or rank multiple documents pairwise and listwise. It also distinguishes LLM assessors, which use labels for evaluation rather than likelihood-based ranking.
- Zero-shot LLM ranking has shifted toward using general-purpose LLMs directly or indirectly instead of tuned learning-to-rank models.
- Pointwise methods score one document at a time, whereas pairwise and listwise methods process multiple documents but are typically applied iteratively to smaller sets.
- LLM assessors assign fine-grained relevance labels aligned with ground-truth judgments to create evaluation datasets, rather than using label likelihoods for ranking.
3 LLM Rankers
The paper extends pointwise relevance-generation rankers from binary prompts to graded labels or rating scales. It derives document scores from label likelihoods, including expected relevance and peak relevance likelihood.
- 3.1 Preliminaries: Binary relevance generation prompts each query-document pair for “Yes” or “No,” converts their likelihoods into a score, and sorts documents by score.
- 3.2 Prompts: Fine-grained relevance generation asks the LLM to choose among ordered labels such as “Not Relevant,” “Somewhat Relevant,” and “Highly Relevant.”
- 3.2 Prompts: Rating-scale prompting asks the LLM to assign relevance on a numeric scale, allowing arbitrarily fine-grained levels.
- 3.3 Ranking Scores: Expected relevance assigns numeric values to labels and aggregates their likelihoods into a document score, with the empirical default y_k = k.
- 3.3 Ranking Scores: Peak relevance likelihood ranks documents using the likelihood of the highest-relevance label and can reduce decoding cost by avoiding scores for every label.
4 Experiment Setup
The experiments rerank BM25-retrieved documents with FLAN PaLM2 S across eight BEIR datasets and compare binary, graded-label, rating-scale, and query-generation prompts. Performance is measured with NDCG@10.
- Data set: Experiments use Covid, Touche, DBPedia, SciFact, Signal, News, Robust04, and NFCorpus from BEIR, regardless of each dataset’s relevance granularity.
- Data set: BM25 retrieves the top 100 documents, which FLAN PaLM2 S then reranks using the evaluated prompting methods.
- Evaluation: Ranking performance is measured with NDCG@10.
- Compared methods: The comparison includes query generation, binary relevance generation, k-level relevance generation, and 0-to-k rating-scale relevance generation.
- Implementation: The default score for the proposed methods is expected relevance derived from label likelihoods.
5 Results
Fine-grained relevance labels improve zero-shot LLM ranking across BEIR datasets, while excessive label granularity offers no consistent benefit and can reduce performance. Likelihood-based scoring, especially peak relevance likelihood, performs strongly and helps distinguish highly ranked documents that binary prompts treat similarly.
- Overall performance: +2% average NDCG@10: RG-3L outperforms RG-2L and RG-YN across BEIR datasets.RG-S(0, 4) achieves a similar improvement, including on datasets with binary ground-truth labels.
- Number of relevance labels: RG-4L performs on par with RG-3L, showing that additional textual relevance levels do not always improve ranking.
- Number of relevance labels: RG-S(0, 4) through RG-S(0, 8) perform similarly, whereas larger scales such as RG-S(0, 9) and RG-S(0, 10) reduce performance.The performance trend remains consistent across datasets with different ground-truth label granularities.
- Ranking score derivation: Likelihood-PR achieves performance close to Likelihood-ER in most methods while using only the peak relevance label’s log-likelihood.Directly ranking generated labels or scores performs worse because it introduces ties between documents.
- Score distribution comparison: Fine-grained prompts let LLMs distinguish higher-scoring query-document pairs that RG-2L assigns nearly equal scores near 1.0.RG-3L and RG-S(0, 4) show strong score correlation across the range and achieve similar ranking performance on the Covid dataset.
6 Conclusion
The paper extends pointwise zero-shot LLM ranking beyond binary labels by scoring fine-grained relevance options and aggregating their likelihoods. The gains are not solely explained by more precise relevance estimation, and the approach may extend to recommendation settings.
- Fine-grained relevance labels replace binary choices, and aggregating their likelihood scores produces document rankings.
- The observed gains are not solely attributable to estimating relevance from more samples, because peak-label likelihood also performs strongly.
- The approach could extend beyond information retrieval to recommendation tasks where LLMs rate users’ likelihood of buying items.
7 Limitations
The study’s conclusions are bounded by access to generated-text likelihoods and by evaluating ranking performance without testing relevance calibration or LLM assessor quality.
- The methods assume access to predicted likelihoods for generated text, which many proprietary LLM APIs may not provide.
- The study evaluates ranking performance only, leaving whether the prompts improve LLM assessors and relevance calibration unresolved.Higher ranking performance does not always translate to higher relevance calibration performance.
A Alternative Relevance Labels
The study tests alternative fine-grained relevance-label wording and analyzes how label probabilities behave across ground-truth relevance levels. Three-level labels consistently outperform binary labels, while probability distributions shift systematically with relevance.
- Alternative textual labels: RG-3L consistently outperforms RG-2L across different textual relevance labels, indicating the finding generalizes beyond specific wording.RG-2L replaces “Not Relevant” with “Irrelevant”, while RG-3L replaces “Somewhat Relevant” with “Partially Relevant”.
- Score distributions: As ground-truth relevance increases, probabilities shift downward for lower relevance labels and upward for higher relevance labels in both rating-scale and RG-4L prompts.The analysis uses Covid query-document pairs with ground-truth relevance levels 0, 1, or 2.
- Score distributions: The probability distributions reveal that fine-grained prompts distinguish relevance levels through systematic label-probability shifts.These distributions provide insight into how expected-relevance and peak-likelihood scoring operate.
C Varying Assigned Relevance Values
The assigned value for an intermediate relevance label has limited effect when reduced, but equating it with the highest label sharply harms ranking performance. This supports preserving a distinction between partially and highly relevant documents.
- Assigned relevance values: Average NDCG@10 remains high as y1 decreases, even when the intermediate label receives the same value as the lowest label.Because NDCG@10 emphasizes top-ranked items, changing intermediate-label values may have limited effect on the top-ranked documents.
- Assigned relevance values: When y1 equals y2, average NDCG@10 drops significantly to approximately RG-2L performance.The experiment fixes y0 = 0 and y2 = 2 while varying the value assigned to “Somewhat Relevant”.
D Experiments on Other LLMs
Experiments with two additional LLMs show that scoring fine-grained relevance labels generally outperforms scoring binary labels, supporting the method's generalizability.
- D Experiments on Other LLMs: Fine-grained relevance scoring achieved better average performance than binary-label scoring on both additional LLMs.The evaluated models were FLAN PaLM2 XS and FLAN UL2.
- D Experiments on Other LLMs: The additional models were FLAN PaLM2 XS, a smaller FLAN PaLM2 S alternative, and the open-source 20B-parameter FLAN UL2.
- D Experiments on Other LLMs: The observed pattern was similar across the fine-grained variants RG-3L and RG-S(0, 4), compared with binary RG-2L.
E More Comparison Results
Additional comparisons show that fine-grained prompting narrows the gap with fine-tuned rankers, can outperform a single-stage listwise baseline, and remains robust to prompt enhancements and scoring choices. Expected relevance is generally less sensitive to rating-scale granularity than peak likelihood.
- Comparison with rankers: The proposed prompting method substantially shrinks the gap between zero-shot LLM rankers and RankT5, although it remains slightly below monoT5 and RankT5.The comparison uses fine-tuned MS MARCO rankers as reference systems.
- Comparison with rankers: The proposed methods perform slightly better than single-stage RankGPT, although the comparison may reflect differences between the LLMs used.The authors explicitly qualify the comparison because the underlying models differ.
- Scoring strategies: Expected-relevance scoring is less sensitive to k than peak-relevance-likelihood scoring.Figure 7 compares ranking-score derivation strategies for rating-scale methods.
- Prompt enhancements: Adding instructions improves both RG-2L and RG-3L, with RG-3L remaining 1.2% better; adding few-shot exemplars provides little further improvement.The authors suggest distribution discrepancy between TREC-DL and BEIR may explain the limited exemplar benefit.