LimRank: Less is More for Reasoning-Intensive Information Reranking

1 Introduction

• Reasoning-intensive relevance can require multi-step inference, contextual reasoning, and recognition of implicit query-document relationships beyond keyword matching or shallow semantic similarity.The introduction notes limitations of LLMs in these settings despite their effectiveness in general-purpose reranking.Source: §1 · p. 1
• Prior approaches typically use large-scale supervised fine-tuning, including nearly 700K reasoning traces for Rank1, which is expensive in compute and data.The authors hypothesize that carefully curated examples can steer latent LLM reasoning capabilities for reranking.Source: §1 · pp. 1–2
• LIMRANK-SYNTHESIZER generates 20K training examples, only 2.85% of Rank1's data, and the authors fine-tune Qwen2.5-7B to produce LIMRANK.The pipeline emphasizes domain diversity, real-world alignment, and difficulty diversity.Source: §1 · p. 2
• 28.0% nDCG@10 on BRIGHT and 1.2 p-MRR on FOLLOWIR make LIMRANK the best-performing model among those with 7B-level parameters.The paper also reports human-evaluation strengths in multi-hop reasoning, subtle instruction disambiguation, and context-sensitive reranking.Source: §1 · p. 2
• At 20K examples, LIMRANK's synthetic data variants outperform variants trained on datasets used by RANK1, Promptriever, and ReasonIR.The comparison uses synthetic IR datasets of the same size.Source: §1 · p. 2
• 30.3% accuracy on GPQA (vs. 28.3%) and 60.1% Recall@5 on LitSearch (vs. 60.8%) show competitive downstream performance as a plug-in reranker.The evaluations use GPQA for retrieval-augmented generation and LitSearch for scientific literature search.Source: §1 · p. 2

2 Related Works

• Reasoning-intensive information retrieval has practical relevance, and RANK1 and ReasonIR use large-scale reasoning-focused datasets to fine-tune rerankers and retrievers, respectively.The passage identifies RANK1 as a reranking approach and ReasonIR as a retrieval approach.Source: §2 · p. 2
• Other synthetic-data methods and reinforcement-learning approaches can improve reasoning capabilities, but are associated with high training, computational, data, or engineering overhead.The passage notes that some synthetic-data methods require large data volumes and that their training strategies add cost.Source: §2 · p. 2
• The paper positions its work as the first exploration of the “less is more” paradigm for reranking, drawing on evidence that small amounts of task-specific data can activate latent LLM reasoning abilities.The passage also notes that data-selection methods have been proposed and shown effective.Source: §2 · p. 2

3.1 Data Curation Guidelines

• The data-curation guidelines aim to maximize example informativeness and reasoning depth through diverse queries, which the authors believe can diversify reasoning chains and activate LLM potential.The pipeline is framed as minimizing example count while preserving informativeness and reasoning depth.Source: §3.1 · pp. 2–3, Table 1 · p. 3

3.2 Reasoning-intensive Data Generation

• The synthesis pipeline uses MS MARCO queries as seeds to create enhanced data, offering a straightforward alternative to methods that depend on large-scale aggregation from varied sources.The motivation notes that common IR datasets often have straightforward queries and limited complexity.Source: §3.2 · p. 3
• Persona-conditioned GPT-4o prompts create daily-life and expert-domain queries; chain-of-thought prompting generates positive passages and corresponding hard-negative passages.Daily-life queries target nuanced real-world situations, while expert-domain queries address domain knowledge and complex professional questions.Source: §3.2 · pp. 3–4

3.3 Data Filtering

• DeepSeek-R1 judges query-passage relevance, and the pipeline keeps pairs only when its judgment matches the previously generated relevance label, yielding 10,282 pairs.Pairs with mismatched judgments are filtered out.Source: §3.3 · p. 4

4.1 Main Results

• LIMRANK surpasses RANK1 on BRIGHT with 20K versus 6 million training examples, and the paper reports state-of-the-art results on the task.The improvement is described as modest; the authors say the smaller training set reduces training cost.Source: §4.1 · p. 4
• LIMRANK achieves state-of-the-art performance on FOLLOWIR and outperforms RANK1 while using substantially less training data.The authors attribute this possibly to the diversity and complexity of synthesized training data.Source: §4.1 · p. 4

4.2 Error Analysis

• LIMRANK has difficulty distinguishing direct from indirect relevance and is more likely to fail when only one or two golden passages exist.The authors suggest an adaptive reranker that distinguishes query intent and assigns relevance scores with greater contextual awareness as a future direction.Source: §4.2 · pp. 4–5

4.3 Experiments on Real-world Tasks

• 30.3% GPQA accuracy for LIMRANK versus 28.3% for RANK1, while LitSearch Recall@5 is 60.1% versus 60.8%, respectively.The authors describe LIMRANK's LitSearch result as comparable and its GPQA result as state of the art.Source: §4.3 · p. 5, Table 3 · p. 5

4.4 Analysis of Synthetic Training Data

• Daily-life queries boost reasoning and RAG performance, while long reasoning traces improve instruction-following and RAG performance.Table 4 compares configurations of the multi-level guideline, with LIMRANK denoting the full version.Source: §4.4 · p. 5, Table 4 · p. 5
• LIMRANK performs strongly despite using far less data, but performance plateaus when the training set becomes too limited.Source: §4.4 · p. 5
• With 20K examples, LIMRANK-trained rerankers consistently outperform variants trained on the RANK1, Promptriever, and ReasonIR synthetic datasets.Table 5 compares the different synthesis datasets.Source: §4.4 · p. 5, Table 5 · p. 5

• The paper proposes LIMRANK-SYNTHESIZER as a pipeline for generating high-quality information-retrieval training data.Source: §5 · p. 5
• Trained with fewer but higher-quality examples, LIMRANK achieves state-of-the-art results on reasoning and instruction-following benchmarks and strong downstream-task performance.The paper presents these results as empirical support for the “Less is More Hypothesis” in information retrieval and a cost-effective alternative to data-intensive approaches.Source: §5 · p. 5

Limitations

• The data selection process relies on human verification and basic filtering, which is resource-intensive and may not scale efficiently.The synthesized data is used only for pointwise rerankers; transfer to listwise and setwise rerankers remains unexplored.Source: Limitations · p. 6

A.1 Prompt for Query Expansion

• Personas are randomly sampled from PersonaHub, and prompts generate daily-life and expert-domain query expansions.Figures 2, 3, and 4 are used for persona integration, real-life expansion, and expert-domain expansion, respectively.Source: §A.1 · p. 7

A.2 Prompt for Passage Generation

• Passage generation prompts GPT-4 to solve queries with chain-of-thought, extract passage descriptions, create hard negatives, and generate passages.The hard-negative descriptions are generated from the query and positive-passage description.Source: §A.2 · pp. 7–8

A.3 Prompt for R1

• DeepSeek-R1 uses the same prompt as Rank1 to produce a reasoning chain and final relevance judgment.Source: §A.3 · p. 8

B.1 Main Experiments Settings

• LIMRANK uses 14,000 Rank1 MS MARCO examples and 6,000 LIMRANK-SYNTHESIZER examples, with positive and negative passages balanced.Source: §B.1 · p. 8
• BRIGHT evaluates reasoning retrieval across its subtasks, while FOLLOWIR evaluates instruction-following retrieval using BM25’s top 100 candidates.BRIGHT uses nDCG@10; FOLLOWIR reports top-five nDCG, MAP, and p-MRR.Source: §B.1 · p. 8
• LitSearch evaluates Recall@5 on specific questions, and GPQA tests RAG with BM25’s top 100 documents and a Qwen2.5-7B reader.The GPQA setup uses top-k three and the diamond subset.Source: §B.1 · pp. 8, 11
• Baselines include listwise RankerZephyr-7B and RankGPT4, setwise RankR1, and pointwise MonoT5, with BM25 retrieving initial candidates.Source: §B.1 · p. 11

B.2 Training and Hyperparameter Details

• Training follows Rank1 settings, using five epochs and LoRA fine-tuning with rank 32, alpha 64, learning rate 6e-5, and batch size 128.Experiments used 2×80GB H100 GPUs, and each training run took less than two hours.Source: §B.2 · p. 11

Training Dataset Size

• The data-scaling experiments use 2K, 10K, and 20K training examples, mixing Rank1 and LIMRANK-SYNTHESIZER samples at each scale.All models use the same training settings; results are referenced in Table 6.Source: §B.3 · p. 11

Dataset Design Components

• Daily-life queries improve reasoning-intensive tasks and RAG but may fail on instruction-following tasks; long traces benefit instruction following and RAG.The findings motivate query-adaptive reasoning-trace lengths in training data.Source: §B.3 · p. 11

Effectiveness of Our Dataset

• At the same dataset size, LIMRANK-trained rerankers consistently outperform variants trained on RANK1, Promptriever, or ReasonIR synthetic data.The comparison uses identical training settings and is presented as evidence of LIMRANK data’s effectiveness in a low-resource setting.Source: §B.3 · p. 11

Importance of Reasoning Traces

• Reasoning traces improve RANK1, while LIMRANK outperforms ReasonIR and RANK1 both with and without traces.The comparison uses the 20K-example setting.Source: §B.3 · p. 11

B.4 FollowIR Results

• Detailed FOLLOWIR results are provided in Table 7.Source: §B.4 · p. 11

B.5 Downstream Task Details

• LitSearch focuses on specificity questions, retrieving the top five most suitable documents under a setting that imposes a higher precision standard.Source: §B.5 · p. 11
• For GPQA diamond RAG, the pipeline reranks the top 20 documents from the filtered ReasonIR datastore and selects the top three for generation.Source: §B.5 · p. 11

C.1 LIMRANK-SYNTHESIZER Examples

• LIMRANK-SYNTHESIZER examples include generated queries in Table 8 and corresponding passages in Table 9.Source: §C.1 · pp. 11–12
• The result captions identify BRIGHT scaling results by nDCG@10 and FollowIR results by MAP@5, nDCG@5, and p-MRR on BM25 top-100 documents.Source: Table 6 · p. 12, Table 7 · p. 12

• In Figure 10, LIMRANK identifies indirect query–passage relevance that RANK1 marks irrelevant; the authors suggest indirect-relevance training examples may explain the difference.Source: Figure 10 · p. 12, §C.2 · p. 12
• Figures 11 and 12 show LIMRANK examples involving indirect and direct correlation, respectively, alongside fine-grained instruction variations.Source: §C.2 · p. 12, Figure 11 · p. 15, Figure 12 · p. 15