ReasonIR: Training Retrievers for Reasoning Tasks

1 Introduction

• Reasoning-intensive retrieval often needs broader supporting documents, whereas existing retrievers trained on short factual queries with straightforward document matches have struggled with it.Useful retrieved material can include background knowledge, tutorials, or demonstrations of similar reasoning methods.Source: §1 · p. 1
• REASONIR-SYNTHESIZER creates varied-length queries with synthesized documents and reasoning-intensive queries from real seed documents, plus superficially relevant but unhelpful hard negatives generated through a multi-turn approach.The synthetic data is more challenging and covers a broader range of query lengths than existing training datasets.Source: §1 · pp. 1–2
• 24.4 nDCG@10 with original queries, 29.9 with GPT4-rewritten queries, and 36.9 with an LLM reranker establish ReasonIR-8B’s BRIGHT state-of-the-art results.With GPT4-rewritten queries, it outperforms recent LLM reranker baselines while requiring over 200× less compute.Source: Figure 1 · p. 2, §1 · p. 2
• 6.39% and 22.58% relative gains over closed-book baselines on MMLU and GPQA, respectively, accompany stronger results than other retriever and search-engine baselines.These reasoning-intensive RAG tasks feed retrieved results to an LLM.Source: §1 · p. 2, Figure 1 · p. 2
• ReasonIR-8B consistently benefits from longer, more informative rewritten queries, while other retrievers plateau or worsen, making query length a test-time scaling dimension.It also outperforms other retrievers with an LLM reranker, supported by higher recall in its retrieved documents.Source: §1 · p. 2
• The method can incorporate newer LLMs for synthetic-data generation or as training base models, and the authors release its code, model, and data recipe.Source: §1 · p. 2

• A bi-encoder independently embeds each query and document, then scores them by cosine similarity.Source: §2 · p. 3
• RAG retrieves the top-k datastore documents, concatenates them with the query, and uses the augmented input to generate a response.Source: §2 · p. 3
• The objective is to improve RAG performance on reasoning-intensive queries by retrieving more relevant top-k documents, despite low lexical and semantic overlap.Source: §2 · p. 3
• Query rewriting can enrich a reasoning query using chain-of-thought and configurable length, while LLM reranking reorders top-K results to select k documents.The passage identifies rewritten-query length as a potential test-time scaling factor.Source: §2 · p. 3
• Retriever training uses contrastive learning to bring queries closer to positive documents than negatives, using in-batch and curated hard negatives; the temperature is 0.02.Hard negatives are irrelevant documents with high cosine similarity that the retriever may confuse as relevant, and their curation is crucial.Source: §2 · p. 3

• The pilot evaluates reasoning-intensive retrieval on BRIGHT, averaging nDCG@10 across its 12 subjects to measure top-10 ranking quality.Source: §3 · p. 3
• Public training queries average 20 tokens for NQ and 21 for MS MARCO, versus 194 tokens for BRIGHT, whose queries require reasoning to retrieve relevant documents.The authors identify this length and complexity gap as a mismatch between retrievers optimized for factual retrieval and reasoning-intensive tasks.Source: §3 · p. 3
• All retrievers improve on BRIGHT as rewritten REASON-QUERY length grows from 64 to 256 tokens, but dense retriever gains plateau beyond a point while BM25 continues to benefit.The authors note that GRIT-7B’s 256-token query training limit hinders long-query embedding, while Nomic’s very-long-context training differs from typical reasoning-query lengths of 64–2,048 tokens.Source: §3 · p. 4, Figure 2 · p. 4
• Query decomposition lowers BRIGHT performance from 12.1 to 10.5 with Nomic and from 20.4 to 17.3 with GRIT-7B, versus retrieving directly with the original query.The authors conclude that an information-rich long query is better than several decomposed short queries on BRIGHT.Source: §3 · p. 4

4 ReasonIR: Synthesizing Hard and Varied-length Retriever Training Data

• REASONIR-SYNTHESIZER combines public data, varied-length (VL) data, and hard query (HQ) data to train retrievers for reasoning-intensive tasks and longer effective query contexts.The section presents the pipeline and its data statistics in Figure 3.Source: §4 · p. 4

4.1 Public Data

• Public training data from real-world applications adapts autoregressive LLMs to retrieval embeddings, covering diverse topics and languages through short factual queries.Examples include MS MARCO, Natural Questions, and HotpotQA.Source: §4.1 · p. 4

4.2 Varied-length Synthetic Query and Positive Document Generation (VL )

• VL data uses LLM-generated queries of 300–2,000 natural words and corresponding positive documents to encourage retrieval from rich, lengthy query context.The positive document is generated alongside the query following a distillation approach.Source: §4.2 · p. 5

4.3 Reasoning-intensive Document-to-query Generation (HQ)

• HQ generation selects informative seed documents and prompts an LLM to brainstorm background knowledge, problem-solving patterns, and realistic scenarios before writing difficult, self-contained, diverse queries.Seed documents come from a diverse scientific knowledge pool and are filtered to remove those with FineWeb-Edu scores below 2; each seed document serves as the positive document.Source: §4.3 · pp. 5–6

4.4 Multi-turn Hard Negative Generation

• For VL and HQ, hard negatives are generated in a separate turn conditioned on the query and positive document, avoiding shortcomings of retriever-mined or jointly generated negatives.The authors identify poor retriever performance on reasoning queries, false negatives, and short, easy negatives as reasons for this design.Source: §4.4 · p. 6

4.5 Difficulty and Length Analysis

• HQ queries are shorter than VL queries but longer than public-data queries, and BM25 and GRIT-7B error rates show HQ is more challenging while public data is mostly solvable by semantic matching.The analysis samples 100,000 examples from each data type; BM25 errors indicate difficulty for lexical matching, while GRIT-7B errors indicate difficulty for semantic matching.Source: §4.5 · p. 6

4.6 ReasonIR-Rerank: A Simple but Effective Tie-Breaking LLM Reranking Method

• REASONIR-Rerank interpolates LLM reranking scores with base-retriever scores to break ties, achieving higher BRIGHT performance than existing reasoning-based reranker baselines.The authors attribute naive LLM rerankers’ poor performance mainly to too many tied reranking scores.Source: §4.6 · pp. 6–7

5.1 Reasoning-intensive Information Retrieval (IR) Performance

• 29.9 nDCG@10 on BRIGHT with REASON-QUERY and 24.4 on original queries make REASONIR-8B outperform existing retrievers and LLM rerankers.With the smaller LLAMA3.1-8B-INSTRUCT query rewriter, it achieves 28.0 nDCG@10 on BRIGHT.Source: §5.1 · p. 8
• ReasonIR-8B benefits from longer rewritten BRIGHT queries, while GRIT-7B and nomic-v1.5-text show diminishing or declining gains as query length increases.The authors attribute the difference to ReasonIR-8B better leveraging information in long rewritten queries.Source: §5.1 · p. 8
• Combining ReasonIR-8B with BM25 reaches 32.0 nDCG@10 on BRIGHT, and pairing it with QWENRERANK reaches 36.8 nDCG@10.QWENRERANK is a zero-shot QWEN2.5-32B-INSTRUCT reranker.Source: §5.1 · p. 8

5.2 Reasoning-intensive Retrieval-augmented Generation (RAG) Performance

• REASONIR-8B RAG improves closed-book MMLU by 3.9 and GPQA by 7.1 absolute points, outperforming GRIT-7B and you.com search on GPQA.The evaluation uses filtered MassiveDS datastores and the reader model for query rewriting; the search-engine baseline was run on GPQA only.Source: §5.2 · pp. 8–9
• Query rewriting improves MMLU by 0.8 for GRIT-7B and 0.6 for ReasonIR-8B, and raises search-engine GPQA performance from 33.8 to 36.4, but lowers dense-retriever GPQA performance.The authors hypothesize that the small reader model cannot write sufficiently good GPQA queries and leave developing a better rewriter to future work.Source: §5.2 · p. 9

5.3 Ablation Studies

• Public+VL+HQ training yields the best BRIGHT results—24.4 nDCG@10 on original queries and 29.9 on REASON-QUERY—while adding easy-query EQ to VL brings no improvement.Public+VL outperforms Public+HQ; the authors hypothesize that VL bridges the difficulty and length gap posed by HQ, and report synergy between varied-length and hard-query data.Source: §5.3 · pp. 9–10, Table 3 · p. 9

5.4 Computational Analysis

• ReasonIR-8B outperforms Rank1-32B with over 200× less test-time compute, while QWENRERANK uses less compute than Rank1-32B and is more performant.The compute estimates approximate encoding and decoding cost at 2N FLOPs per token, where N is the number of non-embedding LLM parameters.Source: §5.4 · p. 10

Retrieval and Reasoning

• Despite retrieval’s traditional weakness on reasoning tasks, recent reasoning models integrate retrieval, yet BGE-based systems still struggle and high-performing retrievers remain scarce as challenging benchmarks emerge.Examples include R1-Searcher, which relies on BGE.Source: §6 · p. 10

Test-time scaling for RAG

• Prior RAG scaling varies datastore size, context length, reranking, or iteration; this work studies a fifth axis: query-length scaling.Source: §6 · p. 10

Synthetic data generation

• Prior IR methods synthesize task-specific queries from documents, documents from queries, or tasks, queries, and documents jointly, but none prioritize reasoning-intensive retrieval.Source: §6 · p. 10

7 Discussion

• ReasonIR-8B improves reasoning-intensive IR and RAG; future directions include studying synthetic-data scaling, better seed-document selection, multilingual and multimodal versions, and multi-turn reasoning.The paper presents synthetic data generation for training reasoning-task retrievers.Source: §7 · p. 10

A.1 Prompt for Reasoning-intensive Document-to-query Data Generation (HQ)

• The HQ data-generation appendix documents its system and user prompts in Figures 4 and 5.Source: §A.1 · p. 16

A.2 Prompt for Multi-turn Hard-negative Generation

• The appendix shows the prompt used to generate multi-turn hard negatives in Figure 6.Source: §A.2 · p. 16

A.3 Prompt for Varied Length Data Generation (VL )

• Varied-length data generation brainstorms scenario instructions, generates a query and positive document, then uses the instruction-concatenated query to generate a hard negative.The prompts are shown in Figures 7, 8, and 6, respectively.Source: §A.3 · p. 16

A.4 Prompt for Reasoning Query Rewritting Reason-query; A.5 Prompts for Easy Query Generation (EQ)

• The appendix documents REASON-QUERY rewriting prompts and an easy-query generation baseline prompt, reusing the seed-document user prompt for the latter.The rewriting prompts were developed by BRIGHT (Su et al., 2024).Source: §A.4 · p. 16, §A.5 · p. 16

A.6 Prompts for Reranking; A.7 Prompts for Query Decomposition

• The appendix provides prompts for reranking top-100 candidates with a simplified Qwen-32b reranker and documents the prompt used by LangChain query decomposition.Reranking prompts vary by task; non-StackExchange tasks use data-specific prompts following Weller et al. (2025).Source: §A.6 · p. 16, §A.7 · p. 16

B Query Length Scaling

• For query-length scaling, the method adds a {MAX_TOKENS}-token constraint to Su et al. (2024)'s rewriting instruction and uses GPT4O-MINI.The maximum output-token setting is matched to the constraint; the prompt is in Appendix A.4.Source: B Query Length Scaling · p. 16

C Query Decomposition

• Prior work reports query decomposition as effective for multi-hop retrieval (Wang et al., 2024; Jin et al., 2025); this study evaluates it on reasoning-intensive queries using LangChain 3 (Zhou et al., 2022).Source: C Query Decomposition · p. 16
• 0.121 to 0.105 with Nomic and 0.204 to 0.173 with GRIT-7B: query decomposition lowers retrieval performance on BRIGHT versus an information-rich long query.Both methods used GPT4 for query rewriting and decomposition.Source: Table 4 · p. 23, C Query Decomposition · p. 23

D Details of Public Training Data

• The public training mix includes MS MARCO, Natural Questions, DUReader (He et al., 2017), FEVER (Thorne et al., 2018), and HotpotQA (Yang et al., 2018).Source: D Details of Public Training Data · p. 23
• It also includes MIRACL, Mr. Tydi (Zhang et al., 2021), QUORA, Squad (Rajpurkar et al., 2016), T2Ranking (Xie et al., 2023), and TriviaQA (Joshi et al., 2017).Source: D Details of Public Training Data · p. 23
• These public datasets provide a diverse base for adapting an autoregressive LLM into a bidirectional encoder for embedding tasks.Source: D Details of Public Training Data · p. 23

E Experimental Setup

• Section E supplements the paper with additional experimental-setup details.Source: E Experimental Setup · p. 23

E.1 Contrastive Training Setup

• Training adapts Llama3.1-8B to bidirectional attention and uses contrastive learning with hard and in-batch negatives for 1,000 steps at batch size 2,048.The setup uses a constant 2e-5 learning rate, 0.06 warm-up ratio, GradCache, and cross-device negatives.Source: §E.1 · p. 23

E.2 Reranker Setup

• Zero-shot Qwen2.5-32B-Instruct reranking scores documents from 0 to 5, normalizes scores to 0–1, and uses two strategies to address score ties.The paper reports that this approach performs well and contrasts it with traditional cross-encoder rerankers, which can degrade reasoning-intensive retrieval.Source: §E.2 · p. 23
• QwenRerank+BM25 uses the final score αSreranker + SBM25, setting α to 100 heuristically without parameter tuning so the reranker and BM25 scores are in the same range.Source: §E.2 · p. 23
• QwenRerank+retriever combines reranker and base-retriever scores with equal weights: 0.5 × Sreranker + 0.5 × Sretriever.Source: §E.2 · p. 24

E.3 Evaluation Setup

• BRIGHT retrieval is evaluated with nDCG@10; retrieval-augmented generation uses MMLU’s 56-subject macro average and GPQA, with datastores built from retrieved passage pools.The initial MMLU and GPQA pool merges and deduplicates the top 1,000 MassiveDS-1.4T4 passages retrieved using Contriever.Source: §E.3 · p. 24

F Additional Results on BRIGHT

• REASONIR-8B significantly increases the theoretical upper bound for BRIGHT reranking, while Oracle nDCG@10 results indicate substantial room for reranker improvement.Additional Table 5 results report BRIGHT Recall@100, which the paper describes as beneficial for post-retrieval reranking.Source: F Additional Results on BRIGHT · p. 24, Table 5 · p. 24

G Additional Reranker Results

• ReasonIR’s proposed LLM reranker is described as cheaper, training-free, and more performant than existing LLM reranker baselines, and performs better when paired with ReasonIR-8B.Source: G Additional Reranker Results · p. 24

G.1 Breaking Ties in QwenRerank

• Reranker results worsen when QwenRerank scores are used directly without the proposed tie-breaking strategies.Source: Table 7 · p. 25, §G.1 · p. 25

G.2 Rank1 vs QwenRerank

• QwenRerank directly scores query-document pairs without generating reasoning tokens, runs much faster, and outperforms Rank1, which generates traces before scoring (Weller et al. (2025)).Source: §G.2 · p. 25, Table 8 · p. 26

G.3 Comparison of Candidates from Different Retrieval Systems

• ReasonIR retrieves a better candidate pool for downstream reranking with the Qwen-32b reranker than the compared retrieval systems.Source: §G.3 · p. 25, Table 9 · p. 26

G.4 Reranker Generalization

• With top-100 candidates, Rank1-7B performs worse on ReasonIR-8B candidates than with BM25 or ReasonIR-8B alone, while Grit-7B reranking trails BM25 despite stronger standalone retrieval.The authors hypothesize that distribution shifts in candidates from different retrievers may degrade rerankers trained for particular retriever distributions.Source: §G.4 · p. 26, Table 10 · p. 27

G.5 Retrieval Overlap between BM25 and ReasonIR-8B

• 28.2% top-100 overlap between BM25 and ReasonIR-8B across 12 BRIGHT subtasks, versus gold-document overlap up to 53.5%, yields complementary pools that may explain hybrid gains.The differing top-100 documents also change the candidate basis for downstream rerankers.Source: §G.5 · p. 26, Table 11 · p. 27

H.1 Training with Reasoning Rewritten Queries (RQ)

• For RQ training, positives and negatives are reused from original queries under the assumption that rewriting does not change document relevance; better curation is left to future work.RQ queries are generated from the reasoning-intensive synthetic data, and the reuse is due to computational constraints.Source: §H.1 · p. 27
• 23.2 nDCG@10 makes VL best among individual types on original queries, while RQ leads reasoning queries at 28.5; HQ reaches 20.9 original-query performance versus 19.6 for public-only training.Every synthetic-data type improves retrieval when added to public data, but HQ does not improve reasoning-query performance and underperforms VL on both query types.Source: §H.1 · p. 27
• HQ’s greater length and difficulty may make it too challenging to learn directly, whereas VL’s more moderate difficulty helps the model adapt to query length.The authors hypothesize that query difficulty and length coverage affect mixture performance; HQ alone lacks enough length generalization for REASON-QUERY.Source: §H.1 · pp. 27–28
• 24.4 nDCG@10 on original queries and 29.9 on REASON-QUERY: HQ+VL performs best on both evaluations, even outperforming RQ+VL on reasoning queries.The authors attribute the mixture’s performance to coverage of query difficulty and length; adding all three types performs worse than HQ+VL, suggesting HQ matters more than RQ when mixed with VL.Source: §H.1 · pp. 27–28, Table 12 · p. 28

I Test-time Compute Comparison between Retriever and Reranker

• A retriever incurs about 2NLq FLOPS, while reranking k query-document pairs costs 2Nk(Lq + Ld + Lo), especially more when k or Lo is large.The passage gives k = 100 as typical and notes Rank1-7B outputs about 300 tokens before its similarity score.Source: I Test-time Compute Comparison between… · p. 29
• REASONIR-8B with GPT4 REASON-QUERY is more than 200× faster than Rank1-32B with a normal query and outperforms it on BRIGHT.The comparison is for the biology subtask of BRIGHT.Source: I Test-time Compute Comparison between… · p. 29, Table 15 · p. 30

J.2 Qualitative Examples for Traditional Retrieval and Reasoning-intensive Retrieval

• Traditional keyword matching can retrieve documents for simple queries, whereas reasoning is needed to retrieve relevant documents for reasoning-intensive queries.This contrast is illustrated in Figure 21.Source: Figure 21 · p. 30, §J.2 · p. 30

J Qualitative Examples

• REASONIR-SYNTHESIZER’s hard queries are qualitatively contrasted with easy queries from a simple baseline.The section presents examples to show the advantages of the synthesizer.Source: J Qualitative Examples · p. 30

J.1 Qualitative Examples for VL and HQ in Detail

• The detailed examples include a 167-token legal query whose hard negative shares a keyword but is irrelevant, and a 66-token sleep query whose positive explains circadian rhythm.The sleep example’s hard negative invokes night shifts and body temperature but does not explain the primary factor or natural oscillations asked about.Source: Table 16 · p. 31, Table 17 · p. 32

J.3 Qualitative Examples for Hard Query HQ Generation for Different Subjects

• Biology examples turn direct function questions into analytic problems about cone-cell concentration and color perception or fever-related hypothalamic set points.The examples contrast easy queries with hard queries that connect the documents’ concepts to broader scenarios.Source: Table 18 · p. 33, Table 19 · p. 33
• Coding and economics examples make queries more reasoning-intensive by adding string corruption or iteration constraints, and by requiring production-function or Phillips Curve analysis.The economics examples contrast formula recall and direct keyword reference with realistic scenario analysis.Source: Table 20 · p. 34, Table 21 · p. 35, Table 22 · p. 36