PaperlayerView exact revision on arXiv

Source-linked AI summary

Rank1: Test-Time Compute for Reranking in Information Retrieval

Orion Weller, Kathryn Ricci, Eugene Yang, Andrew Yates, Dawn Lawrie, Benjamin Van Durme

arXiv:2502.18418v2cs.IRcs.CLcs.LG

TL;DR

Information retrieval rerankers need to reason precisely over query–passage pairs while remaining useful beyond their training distribution. Rank1 distills 635,000 R1 reasoning traces from MS MARCO into smaller reranking models, which show strong reasoning and instruction-following performance, auditable outputs, and broader retrieval applicability. The approach remains slower than classification-head rerankers and can generate flawed or overly stringent judgments.

  • Problem

    Rerankers need additional reasoning over both queries and passages to support precise relevance decisions and auditable retrieval outputs.

  • Method

    Rank1 distills 635,000 R1 reasoning traces generated from MS MARCO query–passage examples into smaller reranking language models.

  • Results

    Rank1 shows state-of-the-art reasoning and instruction-following capabilities, including multilingual settings, despite English-only MS MARCO training from base models.

  • Takeaways & Limitations

    Test-time compute enables rerankers with auditable reasoning chains, user-prompt adaptability, and support for more complex information-retrieval tasks.

  • Takeaways & Limitations

    Rank1 requires more inference compute than non-test-time-compute rerankers and can produce hallucinated, biased, toxic, or overly stringent judgments.

Abstract

from arXiv · show

We introduce Rank1, the first reranking model trained to take advantage of test-time compute. Rank1 demonstrates the applicability within retrieval of using a reasoning language model (i.e. OpenAI's o1, Deepseek's R1, etc.) for distillation in order to rapidly improve the performance of a smaller model. We gather and open-source a dataset of more than 600,000 examples of R1 reasoning traces from queries and passages in MS MARCO. Models trained on this dataset show: (1) state-of-the-art performance on advanced reasoning and instruction following datasets; (2) work remarkably well out of distribution due to the ability to respond to user-input prompts; and (3) have explainable reasoning chains that can be given to users or RAG-based systems. Further, we demonstrate that quantized versions of these models retain strong performance while using less compute/memory. Overall, Rank1 shows that test-time compute allows for a fundamentally new type of explainable and performant reranker model for search.

1 Introduction

Rank1 applies test-time reasoning to reranking by distilling reasoning-language-model traces into smaller models that evaluate query–passage relevance. The resulting models support advanced reasoning, instruction following, auditable chains, and analysis of traditional IR benchmarks.

  • Motivation: Test-time compute lets rerankers reason over both queries and passages before scoring the initial top-k candidates.Reasoning solely over the query would lack passage context and function mainly as query expansion.
  • Approach: 635,000 R1 reasoning examples from MS MARCO were used to fine-tune Rank1 models.The dataset and trained models are open-source.
  • Results: Rank1 achieves state-of-the-art performance on advanced reasoning, complex negation, and multilingual instruction-following IR benchmarks.These capabilities emerge despite training only on English MS MARCO data from base, non-instruction-tuned models.
  • Results: 364% more unjudged documents than RankLLaMA-14B were surfaced on traditional IR benchmarks, suggesting these datasets may be model-saturated.The authors argue that newer benchmarks should emphasize reasoning, instruction following, and modern annotations.
  • Implications: Rank1 produces auditable reasoning chains for users or agentic RAG systems and adapts to user-given prompts.The paper presents these as benefits of applying test-time compute to information retrieval.

2 Model Training

Rank1 is trained by distilling R1 reasoning traces generated from diverse MS MARCO query–passage examples. Data filtering removes noisy or discordant examples, and base Qwen models of several sizes are fine-tuned with LoRA.

  • Data Preparation: R1 was prompted on MS MARCO using batch generation with temperature 0.3 and a 1000-token output limit.MS MARCO was selected for its topic diversity and common use in prior work.
  • Data Mix and Quality Filtering: The initial data mix balanced positives, sampled negatives, and hard negatives from Tevatron and mT5-13B.R1 labeled roughly 80% of the mT5 hard negatives as positive, indicating substantial label noise.
  • Data Preparation: 635,264 R1 generations contained 62.9% relevant and 37.1% non-relevant labels.Reasoning-chain lengths were approximately normally distributed across the data subsets.
  • Data Mix and Quality Filtering: 15% of R1 predictions disagreed with implied labels, so those instances were filtered out.The final mix favored MS MARCO positives and Tevatron negatives because broader balanced data performed worse.
  • Data Mix and Quality Filtering: Self-filtering removed another 10% of mostly false-negative hard-negative data, leaving 386,336 high-quality training examples.The final training set used official MS MARCO positives and negatives from the self-filtered set.
  • Model Training: Qwen 2.5 base models with 7B, 14B, and 32B parameters were fine-tuned rather than their instruction-tuned variants.Initial experiments found the base models performed better than instruction-tuned versions.
  • Model Training: LoRA training ran for up to two epochs or three days, with performance saturating after roughly 1.5 epochs.The implementation used LLaMA-Factory.

3 Experiments

RANK1 is evaluated across advanced reasoning, instruction-following, semantic-understanding, and traditional retrieval benchmarks, using pointwise reranking comparisons and strong baselines. Results show large gains on reasoning and multilingual instruction-following tasks, while traditional benchmarks have annotation limitations and extra test-time compute does not reliably help.

  • Evaluation setup: RANK1 reranks BM25-retrieved documents alongside pointwise and listwise baselines across reasoning, instruction-following, semantic-understanding, and traditional IR evaluations.Experiments use BM25 or mE5-base first-stage retrieval, with listwise models shown as non-comparable upper bounds because of inference-time and context advantages.
  • 3.2 Reasoning Capabilities: 18.7 vs 7.7 nDCG@10 on Stackoverflow shows RANK1-7B nearly doubling RankLLaMA-13B on BRIGHT.All rerankers score the top 100 BM25 documents selected using GPT-4o chain-of-thought retrieval, but rerankers do not receive that chain of thought at inference.
  • 3.2 Reasoning Capabilities: RANK1 achieves state-of-the-art BRIGHT performance despite using about 600k training examples versus RankLLaMA’s 7 million, with performance scaling to 32B parameters.The reported comparison uses the same MS MARCO training dataset, and the authors state that RANK1 is state of the art when reasoning is needed.
  • 3.4 Instruction-Following: 0.611 vs 0.517 multilingual-average nDCG@20 separates RANK1 from mT5-13B, while instruction-following reaches +10.1 versus +4.8 p-MRR.On the cross-lingual setup, the closest comparison is +7.3 versus +3.7 p-MRR; RANK1 uses only English reasoning data despite mT5-13B’s multilingual training.
  • 3.7 How much does the reasoning chain help?: 17.5 versus 27.5 average BRIGHT score shows that the non-reasoning model underperforms Rank1-7B, although it improves over RankLLaMA-7B’s 13.9.The authors attribute the expressive power needed for complex reranking tasks to the reasoning chains.

4 Model Releases

Rank1 is extended across alternative base models and quantized for lower-memory inference. Quantized versions retain strong reasoning and instruction-following performance while enabling larger models to run on a single 24GB GPU.

  • Alternate Base Models: Rank1 is trained with Llama 3.1 8B and Mistral Small 2501 24B to test whether the approach transfers beyond Qwen 2.5.The main Qwen models use 7B, 14B, and 32B base models; alternative versions also start from base rather than instruct variants.
  • Quantization: Quantized Rank1 models are substantially smaller and can all run on one 24GB GPU.The models are quantized with AutoAWQ, including the 32B version.
  • Quantization: Quantized Rank1 models retain strong reasoning and instruction-following performance despite a slight performance drop.They still significantly outperform the baselines on these capabilities while using one-third of the size.
  • Quantization: Quantization results compare int4 models with full-precision versions on SciFact, BRIGHT Biology, and Eng-Zho mFollowIR subsets.The reported table uses AutoAWQ int4 quantization and evaluates retained performance across three task subsets.

5 Related Work

Related work frames Rank1 within the shift from phrase matching toward instruction-aware and reasoning-intensive retrieval. Reasoning language models provide the distillation source for Rank1’s test-time-compute approach.

  • Retrieval Trends: Recent retrieval research increasingly targets reasoning, instruction following, and retrieval for RAG systems beyond simple semantic matching.The cited benchmarks reflect more complex information retrieval tasks.
  • Instruction-Following Retrieval: Instruction-based retrieval models are trained to understand user intent and adapt to new instructions.Examples include Instructor, TART, GritLM, FollowIR, Gecko, and Promptriever.
  • Reasoning Language Models: Reasoning language models improve performance on reasoning tasks by using models such as o1, Gemini Flash Thinking, and R1.R1 is described as the only model among these that exposes reasoning chains through APIs and has open weights.
  • Distillation: Rank1 distills reasoning language-model behavior through basic supervised fine-tuning rather than a more complex reinforcement-learning pipeline.The method uses the ability of smaller models to emulate stronger models from reasoning chains.
  • Reasoning Language Models: Open-source reasoning-model research also covers reproduction, calibration, confidence, and explainable agentic reasoning traces.The paper expects continued collaboration to improve reasoning models for retrieval.

6 Limitations and Future Work

Rank1’s limitations include overthinking, slower inference, and restricted training coverage. The paper proposes reinforcement learning, listwise reranking, and more curated multilingual or instruction-tuned data as future directions.

  • Overthinking: Rank1 can overthink, make mistakes, and reject passages that fail to contain a very specific phrase.On a TREC COVID query, it marked every passage non-relevant, but adapted better when prompted to assume the user lacked background knowledge.
  • Inference Speed: Test-time compute makes Rank1 slower than rerankers with only a classification head, although vLLM and quantization partly mitigate the cost.The paper states that additional compute remains unavoidable.
  • Future Work: The paper identifies future work in reinforcement-learning fine-tuning, listwise reasoning rerankers, and multilingual or instruction-tuned versions.These directions target final-answer alignment, multi-document reasoning, and broader training coverage.
  • Future Work: Rank1 was trained only on English and non-instruct data, leaving curated multilingual and instruction-tuned training as an open direction.The paper reports strong results despite this restricted training setup and expects curated datasets to improve performance.
  • Future Work: The authors expect test-time compute to support future retrieval applications because the training recipe is simple and effective.This is presented as a prospective conclusion rather than an established limitation result.

7 Conclusion

Rank1 is presented as the first reasoning reranker using test-time compute, trained by distilling more than 600,000 R1 reasoning traces. Despite English MS MARCO and base-model training, it achieves strong reasoning and instruction-following capabilities, including multilingual settings.

  • Contribution: Rank1 is the first reasoning reranker model designed to use test-time compute.The paper characterizes it as a new category of reranking model for more complex information retrieval tasks.
  • Method: Rank1 is trained by collecting more than 600,000 R1 reasoning traces and fine-tuning models on them.The training data comes from English MS MARCO queries and passages.
  • Results: Rank1 shows state-of-the-art reasoning and instruction-following capabilities, including in multilingual settings, despite using English MS MARCO and non-instruct base models.The conclusion highlights the result across the paper’s evaluated capability areas.
  • Takeaway: Rank1 enables more complex information retrieval tasks through reasoning-based reranking.This is the paper’s stated overall consequence of introducing the new reranker category.

8 Ethics Statement

RANK1’s generative reranking introduces risks associated with language-model outputs, including misinformation, toxicity, bias, and misuse for illicit information retrieval.

  • RANK1 can generate incorrect, hallucinated, toxic, or biased text, and these flaws may also occur in its R1-generated training dataset.
  • Improved ranking and instruction-following capabilities can increase the risk of retrieving information for illicit purposes.
  • The paper characterizes information retrieval as dual-use while encouraging improved retrieval capabilities to be used for positive purposes.

A Training and Hyperparameter Details

The authors train RANK1 models with LoRA using curated reasoning-chain data, standard hardware and training limits, and early stopping based on reasoning-oriented validation scores.

  • Training setup: Training uses up to two epochs or three days for the 32B model on 4x80GB H100 machines.
  • Training setup: LoRA fine-tuning uses rank 32, alpha 64, learning rate 1e-4, and effective batch size 128.
  • Inference setup: Models are evaluated on one H100 80GB GPU during inference, while baselines use default maximum-length and fp16 settings.
  • Data filtering: Manual inspection found that 12 of 20 sampled mT5 hard negatives were actually positives, making the training data noisier.
  • Data generation: The authors use a prompt template to generate R1 reasoning-chain data.

D Examples of Unjudged and Incorrect DL19 Labels

Re-annotation of DL19 labels found no correct-to-incorrect changes, while RANK1 identified many documents previously marked unjudged; several calibration attempts performed poorly.

  • Label analysis: Re-annotation covered 295 labels and found no changes from correct to incorrect, though some incorrect labels became correct.
  • Label analysis: Incorrect labels affected all models at roughly 6% of their top-10 documents, whereas unjudged documents were mostly found by RANK1.
  • Prompting: Data-specific prompts were used for BEIR and non-StackExchange BRIGHT subsets.
  • Failed approaches: Adding a ModernBERT calibration model on reasoning-chain outputs performed worse, although better curation might make it effective.
  • Failed approaches: Adding extra loss to the final next-token prediction produced sub-par performance because the final true/false token was easy to predict.

G Noise in BEIR benchmarks

The authors argue that BEIR contains substantial annotation and relevance noise, limiting its usefulness for distinguishing highly effective rerankers.

  • BEIR datasets contain issues that make them noisy and less accurate for judging highly effective systems.
  • SciFact: SciFact includes NEI queries without supporting evidence, effectively testing retrieval of the top BM25 document and adding substantial noise.
  • FiQA2018: FiQA2018 omits post details that sometimes clarify the query, making the best answer impossible to determine for some cases.
  • Supporting materials: Table 9 presents examples of incorrect and unjudged DL19 labels, while Table 10 lists dataset-specific prompts for BEIR and non-StackExchange BRIGHT subsets.
  • DBPedia: DBPedia’s partial-relevance labels can reward documents that do not mention the queried entity, contributing to evaluation noise.
  • Touche2020: Touche2020 has a cleaner version created by Thakur et al., but that version is not the standard BEIR evaluation set.
  • Partial relevance: Several datasets assign partial relevance to documents that are only tangentially related, making relevance difficult even for humans.
Loading 2502.18418v2…