PaperlayerView exact revision on arXiv

Source-linked AI summary

RL-Index: Reinforcement Learning for Retrieval Index Reasoning

Yongjia Lei, Nedim Lipka, Zhisheng Qi, Utkarsh Sahu, Yuchen Zhuang, Wenqi Shi, Koustava Goswami, Franck Dernoncourt, Ryan A. Rossi, Yu Wang

arXiv:2606.16316v2cs.IRcs.AIcs.LG

TL;DR

Implicit logical relationships make relevant knowledge difficult to retrieve, while query-side reasoning adds online latency. RL-Index moves rationale generation and reinforcement-learning optimization to indexing, improving retrieval and question answering while shifting computation offline.

  • Problem

    Surface-based retrieval misses relevant knowledge when queries and documents are linked by implicit logical relationships rather than textual overlap.

  • Method

    RL-Index uses an LLM indexing agent to augment documents with rationales and optimizes generation with GRPO using retrieval similarity gain as reward.

  • Results

    RL-Index consistently improves retrieval and downstream QA across retrievers and generators; with BGE, nDCG@10 rises from 13.6 to 15.4 (+13.2%).

  • Takeaways & Limitations

    Offline rationale augmentation offers a plug-and-play retrieval strategy with a better inference latency–accuracy trade-off than query-side reasoning.

  • Takeaways & Limitations

    Training uses only positive documents, so the similarity-gain reward does not account for negative documents and is a weak proxy for retrieval ranking.

Abstract

from arXiv · show

Retrieving external knowledge is crucial for real-world tasks but remains difficult when queries and relevant knowledge are linked by implicit reasoning (e.g., shared theorems or coding logic). Existing methods rely mainly on query-side reasoning, leading to high online latency and underutilizing the reasoning semantics within the knowledge corpus. In this paper, we propose $\textbf{RL-Index}$, an indexing framework that formulates retrieval index reasoning as a reinforcement learning problem. Instead of performing reasoning at query time, RL-Index shifts reasoning to the indexing stage by augmenting documents with LLM-generated rationales that explicitly encode the latent query-knowledge relationship. To optimize the quality of these rationales, we employ Group Relative Policy Optimization (GRPO) and use retrieval similarity as a proxy reward signal, enabling direct optimization of indexing decisions for retrieval effectiveness. Extensive experiments on the BRIGHT benchmark demonstrate that RL-Index consistently improves both retrieval and downstream question-answering performance, while significantly reducing online inference latency. Moreover, the learned rationale augmentation generalizes across diverse retrievers and generators, highlighting its robustness as a plug-and-play indexing strategy across different retrieval systems.

1 Introduction

RL-Index addresses reasoning-intensive retrieval by moving query–knowledge reasoning from online queries to offline document augmentation. It optimizes an LLM indexing agent with GRPO and demonstrates improved retrieval and QA performance, lower inference latency, and transferability across retrieval systems.

  • Motivation: Traditional semantic and lexical retrieval methods miss relevance based on implicit logical relationships rather than textual overlap.Mathematical queries and relevant solutions may depend on the same theorem despite differing surface expressions.
  • Existing Limitations: Online query reasoning adds substantial query-time latency and underutilizes reasoning semantics in the knowledge corpus.Offline index reasoning instead enriches documents with query-oriented rationales in advance.
  • Method: RL-Index formulates retrieval index reasoning as an offline optimization problem that enriches documents with LLM-generated logical rationales.The indexing agent is optimized with Group Relative Policy Optimization using incremental document relevance as the reward.
  • Contributions: RL-Index is presented as the first framework to formulate document augmentation as reinforcement learning and align indexing-agent training with desired reasoning behavior through GRPO.The approach optimizes an open-source LLM with a simple reward design.
  • Results: Experiments on BRIGHT across multiple retrievers and LLMs show consistent retrieval and QA improvements, strong transferability, substantial efficiency gains, and interpretable rationale analysis.The framework improves retrieval of implicitly relevant documents while reducing inference latency.

2 Related Work

Reasoning-intensive retrieval targets queries whose relevant knowledge is connected by complex logical relationships rather than surface lexical or semantic matches. Query-rewriting methods address this challenge by adding explicit reasoning to queries during inference, often through multi-turn retrieval or rationales.

  • Reasoning-Intensive Knowledge Retrieval: Reasoning-intensive retrieval targets multi-hop connections and shared mathematical principles that surface matching cannot reliably uncover.Relevant documents may not match the query on the surface, and key evidence can be distributed across the corpus.
  • Reasoning-Intensive Knowledge Retrieval: Because relevant evidence is difficult to surface, first-stage retrieval often becomes the bottleneck for reasoning-intensive queries.The passage attributes this bottleneck to non-surface-matched documents and evidence distributed across the corpus.
  • Online Retrieval Reasoning by Query Rewriting: Online retrieval reasoning injects explicit reasoning into queries at inference time through multi-turn retrieval with interactive feedback or explicit rationales.These methods have achieved strong performance on reasoning-intensive benchmarks.

3 Framework

RL-Index shifts retrieval reasoning from online query processing to offline indexing by augmenting documents with traceable rationales that expose implicit links between document knowledge and potential user intents. A GRPO-trained indexer optimizes these augmentations using similarity-based rewards, and retrieval jointly scores original and rationale-augmented representations without additional LLM reasoning.

  • 3 Framework: RL-Index formulates retrieval enhancement as an offline indexing-time reasoning problem because raw documents often insufficiently express implicit evidence.The indexed corpus contains enriched document–rationale pairs.
  • 3 Framework: At query time, retrieval requires no additional LLM reasoning and jointly scores each original document with its rationale-augmented version.The retriever can use lexical, embedding-based, or LLM-derived embedding models, while the final score combines original and augmented-document scores.
  • 3 Framework: Each generated rationale combines thematic synthesis of core propositions with functional alignment explaining how those propositions satisfy potential user needs.Constraining explanations to extracted key points preserves traceability to verified document content.
  • 3 Framework: The rationale pairs support query–document similarity computation for GRPO reward training and retrieval during inference.The same offline-generated rationale artifacts serve both optimization and downstream retrieval.
  • 3 Framework: GRPO samples K rationale-augmented candidates per document and increases the policy probability of candidates with larger within-group relative advantages while constraining updates for stability.Clipping limits the importance ratio to [1 − ϵ, 1 + ϵ], while the KL term regularizes the policy.

4 Experiment

Experiments on BRIGHT show that RL-Index consistently improves retrieval and downstream QA across retrievers and generators. By moving reasoning offline into document augmentation, it also offers lower online latency than query-time reasoning while remaining transferable across retrieval systems.

  • Retrieval Performance: RL-Index consistently achieves the best nDCG@10 across BGE, SBERT, and Qwen retrievers under a unified setup.The evaluation uses Llama-3.2-3B-Instruct to generate rationales, with stable improvements across most encoder-only sub-tasks.
  • Retrieval Performance: Doc2Query does not consistently improve reasoning-intensive retrieval, whereas RL-Index produces more effective rationale enrichments than existing augmentation baselines.The paper attributes Doc2Query’s weakness to noise from directly appending synthetic queries to documents.
  • Transferability: RL-Index improves retrieval across cross-retriever transfer settings, with strongest results when training and deployment retrievers are aligned.The transferability study trains the augmentor using dense-retriever rewards and evaluates it with potentially different inference-time retrievers.
  • Efficiency: RL-Index improves retrieval effectiveness with a modest latency increase while remaining much faster than TongSearch’s online query rewriting.The framework shifts reasoning from query-time rewriting to offline document augmentation, eliminating online reasoning overhead.
  • Efficiency: 1,014 API tokens per document are required by SPIKE’s GPT-4o-based training-data construction process.This comprises 523 input tokens and 491 output tokens for each document; RL-Index is compared with SPIKE across training cost, augmentation overhead, and indexing footprint.

5 Rationale Generation Analysis

This section validates RL-Index through quantitative analysis of query–positive versus query–negative similarity margins and qualitative case studies of generated rationales. The analyses examine whether the reward design is sufficient and whether rationale augmentation captures underlying logical reasoning across natural-language and code domains.

  • Qualitative Analysis of Generated Rationale: The qualitative analysis uses natural-language and code case studies to assess whether RL-Index generates underlying logic rather than retriever-preferred wording.The section explicitly evaluates rationale content injected into documents and its relationship to retrieval improvement.
  • Quantitative Analysis of Retrieval Similarity: The quantitative analysis compares query similarity to positive and hard-negative documents before and after rationale augmentation using top-10 SBERT retrieval.For each query, the retrieved set contains the relevant document and hard negatives, and average similarities are computed separately for positive and negative document sets.
  • Qualitative Analysis of Generated Rationale: In the code case, RL-Index adds an intent-based explanation linking a request to stop at a specific distance with polygon-based stop logic.The original document contains correct Nav2 settings but uses low-level configuration text that does not match the user’s natural-language intent.

6 Conclusion

RL-Index frames reasoning-intensive document retrieval as an indexing-stage reinforcement-learning problem rather than relying on online query reasoning. It trains an LLM to expose latent document rationales and retrieves over both original and augmented documents.

  • 6 Conclusion: The framework shifts reasoning from online query processing to offline indexing for reasoning-intensive document retrieval.This perspective treats indexing as the site where latent document reasoning is made useful for retrieval.
  • 6 Conclusion: RL-Index trains an LLM as a document augmenter to make latent document rationales explicit.The method uses rationale generation to augment documents before retrieval.
  • 6 Conclusion: The method combines rationale generation with GRPO-based policy optimization and retrieves over original and augmented documents.These components jointly define the proposed reinforcement-learning-based indexing framework.

7 Limitations

RL-Index’s training reward uses only positive documents and positive-pair similarity gain, making it a simple but weak proxy for retrieval ranking because negative documents are not considered.

  • Limitations: RL-Index trains only on positive data, optimizing positive-pair similarity gain without considering negative documents.This design is described as simple but effective, yet insufficient for modeling retrieval ranking.
  • Limitations: The authors propose adding negative documents and negative similarity to the reward design in future work.

8 Dataset Details

BRIGHT contains 1,398 challenging real-world queries across diverse domains, organized into StackExchange, Coding, and Theorem-based collections. The datasets are additionally grouped by document type into Natural Language, Code, and Math to capture retrieval challenges from different content structures.

  • Dataset composition: BRIGHT contains 1,398 real-world queries spanning economics, psychology, robotics, mathematics, and software programming, designed to require deep comprehension and reasoning for retrieval.The benchmark comprises StackExchange, Coding, and Theorem-based collections.
  • Dataset composition: The StackExchange group includes Biology, Earth Science, Economics, Psychology, Robotics, Stack Overflow, and Sustainable Living.The corresponding abbreviations are Bio., Earth., Econ., Psy., Rob., Stack., and Sus.
  • Dataset composition: The Coding group comprises Leetcode and Pony, while the Theorem group comprises AoPS, TheoremQA-Question, and TheoremQA-Theorem.Their abbreviations are Leet., Pony, AoPS, TheoQ., and TheoT.
  • Document-type classification: For document-type analysis, the datasets are grouped into Natural Language, Code, and Math to capture retrieval challenges arising from different content structures.This classification follows Lee et al. (2025).
  • Document-type classification: Natural Language includes Biology, Earth Science, Economics, Psychology, and Sustainable Living; Code includes Leetcode, Pony, Robotics, and Stack Overflow; Math includes AoPS, TheoremQA-Question, and TheoremQA-Theorem.The document-type groupings use the abbreviations Bio., Earth., Econ., Psy., Sus., Leet., Pony, Rob., Stack., AoPS, TheoQ., and TheoT.

9 Experimental Outcomes

RL-Index improves retrieval while shifting reasoning from online queries to offline indexing, reducing latency and augmentation overhead. Its benefits remain robust across LLM augmentors, retrievers, and moderate combination-weight changes, with GRPO providing gains beyond prompted rationales alone.

  • LLM Transferability: RL-Index generalizes across LLM families, with Qwen-based rationale augmentation showing improvement trends similar to the LLaMA-based setting across most tasks.This indicates that the GRPO objective and rationale format are not tied to one underlying LLM augmentor.
  • Online Latency: 0 online reasoning latency is introduced by RL-Index because reasoning is shifted entirely to offline indexing, unlike TongSearch’s around 7.66s per-query reasoning.RL-Index also maintains substantially lower end-to-end online latency by avoiding query-time reasoning and rewriting.
  • Online Latency: RL-Index usually achieves lower query embedding latency than TongSearch by encoding shorter inputs, while maintaining substantially lower end-to-end online latency.The comparison covers online retrieval using different retrievers and datasets, with per-query latency reported in Tables 10 and 11.
  • Offline Cost: RL-Index generates fewer tokens than SPIKE during document augmentation, indicating lower augmentation-time overhead while maintaining stronger retrieval effectiveness.RL-Index training also uses query–document pairs without costly GPT-generated document augmentations, unlike SPIKE’s GPT-4o-based supervised training setup.
  • RL Optimization Ablation: RL optimization outperforms prompt-only rationale generation: the prompt-only baseline improves SBERT by +7.4% but changes BGE by -0.74% and Qwen by -26.2%.GRPO-optimized RL-Index achieves the highest average performance across all three retrievers, aligning rationale generation with retrieval preferences.
  • Sensitivity to α: 14.9 at α = 0 rises to 16.4 at α = 0.8, remains 16.3 at α = 1, and declines slightly to 16.2 at α = 1.2.Across all 12 domains, these results indicate robustness to moderate changes in the combination weight α.

10 Case Study

The case studies illustrate how RL-Index exposes implicit query–document relationships through Reasoning Traces in natural-language and code retrieval. In the natural-language example, rationale augmentation makes the application of stochastic dominance to the query more explicit than the original document does.

  • Cross-domain case studies: Case studies in natural language and code show how queries and documents are related through a Reasoning Trace.The natural-language example concerns choosing between lotteries under an exponential distribution, while the code example concerns stopping a robot when a dynamic obstacle is detected.
  • Natural-language case study: The natural-language case study asks how a consumer with VNM preferences chooses between uncertain lotteries with different distribution parameters.The query is framed under an exponential distribution and compares two lotteries.
  • Natural-language case study: The original natural-language document defines stochastic dominance but does not explain how to apply it to the query or bridge it to the gold answer.It also includes external links, making the connection between the query and answer difficult to establish.
  • Code case study: The code case study asks how to make a robot stop when it sees a dynamic obstacle n meters away, using the navigation stack as context.The source query asks for a demo or tutorial showing how to stop the robot from its path.
Loading 2606.16316v2…