Source-linked AI summary
Rank-R1: Enhancing Reasoning in LLM-based Document Rerankers via Reinforcement Learning
Shengyao Zhuang, Xueguang Ma, Bevan Koopman, Jimmy Lin, Guido Zuccon
TL;DR
Existing LLM rerankers either depend on instruction-following prompting or supervised fine-tuning, while zero-shot methods do not use annotated relevance data. Rank-R1 applies reinforcement learning with relevance-only rewards to encourage reasoning before ranking, matching supervised fine-tuning in-domain and outperforming zero-shot and supervised baselines out-of-domain, particularly with a 14B model.
Problem
Prompting-based rerankers rely on instruction following and, as zero-shot methods, do not leverage available human-annotated relevance data for improvement.
Method
Rank-R1 uses GRPO reinforcement learning to train an LLM reranker to reason over queries and candidate documents before selecting the most relevant document, using only correctness-based relevance rewards.
Results
Rank-R1 performs similarly to supervised fine-tuning in-domain and outperforms zero-shot prompting and supervised fine-tuning on out-of-domain BRIGHT, with the 14B model surpassing GPT-4-based reranking.
Takeaways & Limitations
Reasoning can improve generalization in reasoning-intensive document ranking and may make LLM-based reranking results more explainable.
Abstract
from arXiv · showhide
In this paper, we introduce Rank-R1, a novel LLM-based reranker that performs reasoning over both the user query and candidate documents before performing the ranking task. Existing document reranking methods based on large language models (LLMs) typically rely on prompting or fine-tuning LLMs to order or label candidate documents according to their relevance to a query. For Rank-R1, we use a reinforcement learning algorithm along with only a small set of relevance labels (without any reasoning supervision) to enhance the reasoning ability of LLM-based rerankers. Our hypothesis is that adding reasoning capabilities to the rerankers can improve their relevance assessement and ranking capabilities. Our experiments on the TREC DL and BRIGHT datasets show that Rank-R1 is highly effective, especially for complex queries. In particular, we find that Rank-R1 achieves effectiveness on in-domain datasets at par with that of supervised fine-tuning methods, but utilizing only 18\% of the training data used by the fine-tuning methods. We also find that the model largely outperforms zero-shot and supervised fine-tuning when applied to out-of-domain datasets featuring complex queries, especially when a 14B-size model is used. Finally, we qualitatively observe that Rank-R1's reasoning process improves the explainability of the ranking results, opening new opportunities for search engine results presentation and fruition.
1 Introduction
LLM rerankers either prompt models to reorder documents or fine-tune them on relevance judgments, but these approaches do not explicitly model reasoning. Rank-R1 applies reinforcement learning to generate reasoning before selecting the most relevant candidate, matching supervised fine-tuning in-domain and outperforming it out-of-domain.
- Zero-shot rerankers rely on instruction following and do not use available human-annotated relevance data for improvement.
- Fine-tuned rerankers use human relevance judgments but lack explicit reasoning because high-quality reasoning data is unavailable.
- Rank-R1 uses GRPO to generate reasoning steps before selecting the most relevant document, with rewards based only on correct identification.
- Rank-R1 matches supervised fine-tuning on in-domain TREC DL19 and DL20 datasets.
- On the out-of-domain BRIGHT dataset, Rank-R1 outperforms zero-shot prompting and supervised fine-tuning, with the 14B model surpassing GPT-4.
2 Method
Rank-R1 modifies Setwise reranking to require reasoning before selecting a candidate, then trains the reranker with GRPO using rule-based relevance rewards rather than reasoning annotations.
- Rank-R1 adapts DeepSeek-style reinforcement learning to enhance reasoning in an LLM-based Setwise reranker.
- Setwise prompting gives a query and candidate documents to the LLM, which selects the most relevant document during heap-based reranking.
- Rank-R1 adds a reasoning instruction that makes the LLM reason before predicting the most relevant candidate label.
- GRPO optimizes the reranker using generated reasoning processes and rewards tied to the selected relevant-document label.
- Training uses MSMARCO queries, one labeled relevant document, and 19 BM25-retrieved documents to construct each Rank-R1 prompt.
- A reward of one requires the prescribed reasoning-answer format and the correct ground-truth document label; otherwise, the reward is zero.
3 Experimental Settings
The experiments evaluate Rank-R1 in-domain on TREC DL19 and DL20 and out-of-domain on BRIGHT, comparing zero-shot, GRPO, supervised Setwise, and listwise rerankers across several LLM backbones.
- Datasets: TREC-DL19 and DL20 provide in-domain evaluation, while BRIGHT provides out-of-domain evaluation requiring complex query reasoning.
- Initial Retrieval: All methods rerank the top 100 documents retrieved by BM25 using the pyserini implementation.
- Rank-R1 Settings: Rank-R1 is evaluated zero-shot with its reasoning prompt or with GRPO training on MS MARCO using groups of eight generated answers.
- Comparison Methods: The comparisons include original Setwise zero-shot and supervised fine-tuning trained on approximately 400k MS MARCO datapoints.
- Comparison Methods: Additional baselines include the GPT-4-based RankGPT and the GPT-4-generated-data RankZephyr listwise reranker.
- Backbone LLMs: Setwise and Rank-R1 use instruction-tuned Qwen2.5 models ranging from 3B to 14B parameters, while comparison rerankers use Zephyr 7B or GPT-4.
4 Results
Rank-R1’s GRPO training improves in-domain ranking to supervised fine-tuning levels with less data and performs strongly on challenging out-of-domain BRIGHT queries. Its advantage is especially pronounced for larger models and reasoning-intensive ranking.
- In-domain effectiveness: Rank-R1’s reasoning process improves zero-shot Setwise ranking effectiveness, with particularly large gains for the 3B model.
- In-domain effectiveness: 18% of the MSMARCO training data was sufficient for GRPO training, compared with the full dataset used for SFT.
- Effect of quantity of training data: Rank-R1 requires significantly less data than Setwise SFT to reach the same early-training performance, but this data-efficiency advantage disappears after 5–7% of training data.
- Out-of-domain effectiveness: On BRIGHT, Rank-R1 generally outperforms or matches zero-shot and Setwise SFT rerankers, whereas RankZephyr fails to improve over BM25 on most datasets.
5 Analysis
During GRPO training, Rank-R1’s rewards increase while completion lengths remain broadly stable. Larger models generate longer responses and begin training with higher rewards, whereas smaller models improve faster.
- Reward score v.s. Response length: Rewards consistently increase during Rank-R1 training, with smaller models improving faster and larger models starting from higher initial rewards.
- Reward score v.s. Response length: Larger models tend to generate longer responses, but completion length does not noticeably increase as training proceeds.
- Reward score v.s. Response length: The stable completion length differs from DeepSeek-R1’s reported behavior, possibly because Rank-R1 starts from an instruction-tuned model and uses MSMARCO passage-ranking data.
- Reward score v.s. Response length: A GRPO-trained model focuses on the most relevant document, whereas the zero-shot model mainly describes document contents and predicts incorrectly.
6 Conclusion
Rank-R1 uses GRPO-based reinforcement learning to enhance reasoning in an LLM-based Setwise reranker without human-annotated reasoning data. The paper reports comparable in-domain effectiveness to supervised fine-tuning and stronger performance on reasoning-intensive ranking tasks, with potential explainability benefits.
- Rank-R1 uses GRPO-based reinforcement learning to enhance reasoning in an LLM-based Setwise reranker without human-annotated reasoning data.
- The example uses a Qwen2.5-3B-Instruct Rank-R1, whose GRPO-trained version gives the correct answer.
- Rank-R1 achieves effectiveness comparable to supervised fine-tuning on in-domain tasks.
- A 14B Rank-R1 achieves considerably higher effectiveness than the state-of-the-art non-reasoning Listwise reranker on reasoning-intensive ranking tasks.
- The reasoning capability may improve explainability and support new search-engine result presentation and fruition affordances.
A Prompt for Setwise reranker
The Setwise reranker prompt asks the assistant to identify the most relevant document from a candidate set and return its label in answer tags. The same prompt is used for zero-shot and supervised fine-tuning, while Rank-R1 adds reasoning instructions.
- The prompt asks the assistant to select the most relevant document from twenty candidates and return its label inside answer tags.
- The prompt uses placeholders for the query and candidate documents, with labels ranging from [1] to [20].
- Zero-shot and supervised-fine-tuned Setwise rerankers use this prompt, whereas Rank-R1 additionally includes reasoning instructions.
B Training hyper-parameters
The training setup uses the TRL library for GRPO and supervised fine-tuning, with LoRA adapters for both methods. Supervised fine-tuning computes loss only over the answer span.
- The TRL library is used for both GRPO and supervised fine-tuning training.
- Both training methods use LoRA adapters.
- Supervised fine-tuning computes cross-entropy loss only on the answer span and ignores prompt-token loss.