PaperlayerView exact revision on arXiv

Source-linked AI summary

DeepRetrieval: Hacking Real Search Engines and Retrievers with Large Language Models via Reinforcement Learning

Pengcheng Jiang, Jiacheng Lin, Lang Cao, Runchu Tian, SeongKu Kang, Zifeng Wang, Jimeng Sun, Jiawei Han

arXiv:2503.00223v3cs.IR

TL;DR

Information retrieval needs better ways to bridge the gap between user needs and imperfect queries, while existing LLM augmentation methods often depend on costly supervised data. DeepRetrieval trains LLMs with reinforcement learning from retrieval outcomes, achieving strong results across retrieval tasks and literature-search benchmarks. Its reported scope includes broad cross-task gains, with limitations arising when retrievers have prior exposure to evaluation data and when decoding becomes costly.

  • Problem

    Existing LLM query-augmentation approaches commonly require expensive supervised learning or distillation data, while user queries often imperfectly express information needs.

  • Method

    DeepRetrieval uses reinforcement learning to train LLMs for query generation or rewriting directly from retrieval outcomes, with KL-regularized optimization.

  • Results

    DeepRetrieval outperforms prior methods across diverse retrieval tasks, doubles recall in literature search, and rivals leading models with 3B parameters.

  • Takeaways & Limitations

    The results support reinforcement learning as an effective paradigm for information retrieval across literature, evidence-seeking, classic retrieval, and SQL search.

  • Takeaways & Limitations

    Performance improvements can be limited when a retriever was trained on evaluation-dataset data, while no-think decoding made training and testing over 8× slower.

Abstract

from arXiv · show

Information retrieval systems are crucial for enabling effective access to large document collections. Recent approaches have leveraged Large Language Models (LLMs) to enhance retrieval performance through query augmentation, but often rely on expensive supervised learning or distillation techniques that require significant computational resources and hand-labeled data. We introduce DeepRetrieval, a reinforcement learning (RL) approach that trains LLMs for query generation through trial and error without supervised data (reference query). Using retrieval metrics as rewards, our system generates queries that maximize retrieval performance. DeepRetrieval outperforms leading methods on literature search with 65.07% (vs. previous SOTA 24.68%) recall for publication search and 63.18% (vs. previous SOTA 32.11%) recall for trial search using real-world search engines. DeepRetrieval also dominates in evidence-seeking retrieval, classic information retrieval and SQL database search. With only 3B parameters, it outperforms industry-leading models like GPT-4o and Claude-3.5-Sonnet on 11/13 datasets. These results demonstrate that our RL approach offers a more efficient and effective paradigm for information retrieval. Our data and code are available at: https://github.com/pat-jj/DeepRetrieval.

1 Introduction

DeepRetrieval addresses the semantic gap in information retrieval with reinforcement-learned query generation, avoiding supervised reference queries. It evaluates this approach across diverse retrieval settings and reports strong literature-search performance.

  • Traditional keyword and statistical methods struggle with semantic mismatches between users’ information needs and query formulations.
  • Existing LLM query-augmentation methods commonly rely on supervised learning or distillation, requiring expensive training data and computation.
  • DeepRetrieval trains LLMs through reinforcement learning, using retrieval metrics such as NDCG as rewards instead of supervision data.
  • 65.07% recall for publication search exceeds the previous SOTA of 24.68% using real-world search engines.
  • 63.18% recall for clinical trial search exceeds the previous SOTA of 32.11%.
  • The paper evaluates DeepRetrieval across literature, evidence-seeking, classic information retrieval, and SQL database-search tasks.

2 DeepRetrieval

DeepRetrieval formulates query augmentation as reinforcement learning: an LLM generates augmented queries, retrieval performance supplies the reward, and policy optimization updates the model. The framework combines task-specific retrieval objectives with output-format rewards and KL regularization.

  • DeepRetrieval trains LLMs for query generation or rewriting directly from retrieval outcomes rather than supervised query-augmentation pairs.
  • 2.1 Problem Formulation: The original user query is the RL state, the generated augmented query is the action, and retrieval performance is the reward.
  • The framework learns query-formulation strategies through trial and error without requiring supervised generation data.
  • 2.2.1 Reward Function Design: The reward combines task-specific retrieval performance with a format reward for adhering to the required output structure.
  • 2.2.1 Reward Function Design: Task-specific rewards use Recall@K, answer-span rank, NDCG@K, or SQL execution accuracy depending on the retrieval setting.
  • 2.2.1 Reward Function Design: KL regularization penalizes large deviations from the initial reference policy, stabilizing reinforcement-learning training.
  • Direct metric optimization focuses training on retrieval objectives rather than similarity to reference or ground-truth queries.

3 Experiments

DeepRetrieval is evaluated across literature, evidence-seeking, classic sparse and dense retrieval, and SQL database search, using task-specific retrieval metrics and real search engines. It delivers strong gains across these settings, while performance depends on training-data scale and retriever adaptation.

  • Literature Search and Evidence-Seeking Retrieval: DeepRetrieval achieves 65.07% and 63.18% Recall@3K on publication and trial search, respectively, surpassing prior literature-search performance.Publication search uses PubMed and trial search uses ClinicalTrials.gov search APIs.
  • Literature Search and Evidence-Seeking Retrieval: DeepRetrieval matches GPT-4o and Claude-3.5 on NQ and TriviaQA and outperforms them on SQuAD despite using 3B parameters.Evidence-seeking retrieval measures whether retrieved documents contain answer spans using H@N.
  • Classic Sparse and Dense Text Retrieval: DeepRetrieval-3B improves over all baselines on 7 of 8 sparse-retrieval datasets and leads on 6 of 8 dense-retrieval datasets.The comparison uses BM25 for sparse retrieval and multiple dense retrievers, with NDCG@10 as the metric.
  • Classic Sparse and Dense Text Retrieval: DeepRetrieval gains are smaller when BGE is already near-optimal through prior exposure to dataset training data, limiting query-rewriting headroom.On HotpotQA, FEVER, and MS MARCO, BGE-base-en-v1.5 often exceeds 0.7 NDCG@10 before query rewriting.
  • Classic Sparse and Dense Text Retrieval: BM25 with DeepRetrieval can match or exceed dense retrieval while completing retrieval 34× faster on a 5.42-million-document corpus.The reported runtimes are 352 seconds for BM25 and 12,232 seconds for dense retrieval under otherwise identical settings.
  • SQL Database Search: RL from scratch improves SQL execution accuracy by 18.25% on BIRD and 23.88% on Spider versus zero-shot Qwen2.5-Coder3B-Inst.It can also outperform supervised fine-tuning models trained with human-curated SQL and GPT-4o-distilled reasoning.

4 Discussions & Takeaways

DeepRetrieval’s retrieval gains reflect dataset-adaptive knowledge injection, direct optimization of search behavior, and reasoning-enabled exploration. These analyses also examine how query formulation choices affect retrieval outcomes.

  • Knowledge Injection: Figure 3 evaluates knowledge injection by comparing generated queries before and after an auxiliary LLM removes injected answer information.The postprocessing analysis isolates the contribution of prior-knowledge content in generated queries.
  • Knowledge Injection: Knowledge injection contributes differently across evidence-seeking datasets, with DeepRetrieval adapting its injection rate to each dataset.Its injection rates are 22.1% for NQ, 41.5% for TriviaQA, and 4.6% for SQuAD.
  • RL versus SFT: DeepRetrieval’s reinforcement learning outperforms supervised fine-tuning because it directly optimizes search-engine ranking behavior rather than imitating reference queries.The approach can discover query patterns involving precise Boolean expressions, operator usage, and term grouping.
  • Reasoning and Query Generation: Reasoning before query generation improves DeepRetrieval by encouraging broader exploration and more diverse, contextually enriched queries.This helps the model learn which query formulations yield more comprehensive and relevant retrieval results.

5 Conclusion

The paper situates query generation within established query-augmentation, reinforcement-learning, and retrieval research. It concludes that DeepRetrieval uses reinforcement learning without supervised reference queries, while noting a substantial runtime limitation for the no-reasoning setting.

  • Limitation: The no-reasoning Qwen setting was over 8× slower than Qwen and Llama, so its training was halted early.Testing at step 1000 took approximately 10.5 hours without reasoning versus 1 hour with reasoning-enabled models.
  • Conclusion: DeepRetrieval trains language models for query generation without supervised reference queries by optimizing retrieval metrics.The conclusion presents this as the paper’s central reinforcement-learning contribution.
  • Query Augmentation: Query augmentation refines an original query or enriches it with additional information to address incomplete information needs.The paper describes classical pseudo-relevance feedback, knowledge-base-driven augmentation, and text-to-SQL as related approaches.
  • LLM-based Query Augmentation: LLM-based query augmentation includes direct query rewriting and other methods that leverage language models to improve retrieval.The paper presents these methods as a later phase in query-augmentation research following classical approaches.
  • Reinforcement Learning: Reinforcement learning trains agents through interaction with an environment while maximizing rewards, and has become a major framework for LLM training.The related-work discussion contrasts RL with alternatives such as DPO and SimPO.

A.3 Difference between DeepRetrieval and Previous Work

DeepRetrieval differs from prior query-augmentation methods by removing supervision, using dynamic PPO trial-and-error, and directly rewarding task retrieval performance. Its policy iteratively generates augmented queries, receives retrieval-based rewards, and updates while PPO stabilizes optimization and encourages exploration.

  • No Supervision for Query Generation: DeepRetrieval eliminates supervised query-generation data, including labeled rewrites, warm-up supervision, and few-shot examples.The approach removes the need for costly human-annotated query rewrites or equivalent supervision.
  • RL with Trial & Error: DeepRetrieval uses PPO for dynamic trial-and-error rather than fine-tuning on static preference datasets.PPO enables query policies to explore and update from retrieval outcomes during training.
  • RL with Robust Reward Function: Retrieval metrics, including recall, NDCG, answer spans, and execution accuracy, are transformed directly into rewards across retrieval tasks.This reward design covers information retrieval, evidence seeking, and SQL generation.
  • Policy Optimization: For each original query q, the policy πθ generates an augmented query q′, receives retrieval-based rewards, and updates to improve future queries.The objective is to maximize expected reward while avoiding destructively large policy updates.
  • Policy Optimization: PPO combines clipped policy updates, value-function learning, and entropy regularization to stabilize training and encourage diverse query formulations.Clipping limits policy changes, the value function reduces variance, and entropy promotes exploration.
  • Training Process: Each training iteration has generation, preparation, and learning stages that update the actor and critic using retrieval-derived rewards.The actor generates queries, the critic computes values, and the reward model evaluates retrieval performance.

B.2 PPO Implementation

The implementation trains query-generation models with PPO and documents the software, hardware, datasets, and corpus construction used for evaluation. Experiments span literature, evidence-seeking, and classic retrieval settings with curated dataset splits and efficiency-oriented corpus subsets.

  • PPO Implementation: DeepRetrieval trains Qwen2.5-3B-Instruct as both actor and critic using PPO, with learning rates of 1e-6 and 1e-5 respectively.The implementation also uses KL coefficient 0.001, temperature 0.6, batch size 64, and PPO minibatch size 16.
  • Software: The implementation uses VERL, VLLM, PyTorch, Ray, and Flash Attention 2 for reinforcement learning, inference, deep learning, and distributed execution.These components support the software stack for training and inference.
  • Hardware: Three-billion-parameter tasks run on two NVIDIA A100 80GB GPUs, while BIRD and Spider runs with DeepRetrievalCoder-7B use four.The reported system also includes an AMD EPYC 7513 processor and 1.0 TB of RAM.
  • Literature Search Data: Literature-search datasets link systematic reviews to cited PubMed studies and filter ClinicalTrials.gov records by reported results and full-text availability.Publication and trial datasets are sourced from LEADS and constructed for literature-search evaluation.
  • Classic Retrieval Data: MS-MARCO subsets for Health, Science, and Technology are created by embedding queries, clustering them into 25 groups, and manually labeling 10 clusters.The resulting subsets preserve the original train-development mapping to prevent data leakage.
  • Classic Retrieval Data: The classic-retrieval corpus subsets include all ground-truth passages and randomly added noise passages to reach 800,000 passages from an 8,841,822-passage corpus.This construction improves dense-retrieval evaluation efficiency.

C.4 SQL Database Search

The SQL database-search evaluation uses Spider and BIRD, which test text-to-SQL generation under different database scales, domains, and complexity. The section also examines reasoning behavior, query-generation patterns, and the limited-data setting where DeepRetrieval does not consistently achieve the best results.

  • Datasets: Spider contains 10,181 natural-language questions paired with 5,693 complex SQL queries across 200 databases and 138 domains.Its databases contain multiple tables and provide a diverse text-to-SQL benchmark.
  • Datasets: BIRD contains 12,751 question-SQL pairs across 95 databases totaling 33.4 GB and spanning more than 37 professional domains.Unlike Spider, BIRD incorporates external knowledge and more intricate SQL queries.
  • Evaluation Setup: The evaluation uses non-overlapping database splits, with Spider tested on its test set and BIRD evaluated on its development set.This split design is intended to measure generalization across databases.
  • Evaluation Setup: Models receive only the full database schema, while comparison SFT models use ground-truth SQL and reasoning processes distilled from GPT-4o.The schema contains table metadata, including column names and data types.
  • Reasoning Behavior: Reasoning improves query generation, while reasoning length decreases over training rather than exhibiting the increasingly long chains reported for DeepSeek-R1.The paper associates concise reasoning with improving performance in this task.
  • Reasoning Behavior: Without reasoning, Qwen receives early rewards for verbose queries and later produces repetitive queries with redundant terms.Qwen and LLaMA with thinking capabilities instead produce structured queries that continue improving semantically.
  • Model Policies: Qwen-2.5-3B-Instruct and Llama-3-3B achieve comparable performance while generating substantially different query lengths.Qwen produces longer rewritten queries, whereas Llama tends to produce shorter ones.
  • Limitations: SciFact does not achieve its best performance with DeepRetrieval in sparse or dense settings, possibly because its training set contains only 800 examples.The paper suggests DeepRetrieval benefits more from a few thousand training examples.

D.3 SQL Database Search

SQL results show that task complexity, model capabilities, cold-start supervision, and reasoning affect reinforcement-learning outcomes. The section also defines a post-processing analysis for detecting knowledge injection and measuring its retrieval impact.

  • SQL Results: BIRD yields lower performance than Spider for the same models, and RL effectiveness is sensitive to model capability and task complexity.The paper attributes this sensitivity to the influence of early exploration on the subsequent training process.
  • Cold Start: 6.3% increase in execution accuracy occurs on BIRD with cold start for Qwen2.5-3B-Inst, from 41.40 to 44.00.On Spider, the corresponding increase is 2.2%, from 68.79 to 70.33.
  • Cold Start: DeepRetrieval3B-Coder gains only marginally on BIRD with cold start, from 49.02 to 50.52, and drops on Spider from 74.85 to 74.34.The results suggest stronger coding priors reduce reliance on additional cold-start supervision.
  • Reasoning Ablation: Removing reasoning can slightly improve DeepRetrieval3B-Base execution accuracy, from 68.79% to 70.24%, while reasoning generally improves performance.The paper links shorter reasoning chains with potential benefits on simpler tasks.
  • Knowledge Injection: Knowledge injection is defined as incorporating answer information into generated queries based on prior knowledge, potentially inflating retrieval performance.The analysis detects answer spans that cannot be derived from the original query without prior knowledge.
  • Knowledge Injection: The post-processing pipeline removes injected spans, evaluates original and cleaned queries, and compares their retrieval accuracy.It reports injection rate and the performance delta between original and cleaned queries.

E.3 Key Findings

DeepRetrieval’s performance gains vary in how much they depend on knowledge injection. Across datasets, the method adapts its augmentation strategy to dataset characteristics, while raw performance alone may obscure these differences.

  • 33.4% of Natural Questions queries received knowledge injection, yet performance remained relatively strong after injected content was removed.The passage attributes gains to both injection and other factors such as query reformulation.
  • 56.9% of TriviaQA queries received knowledge injection, and removing injected content caused significant performance drops.
  • 10.5% of DeepRetrieval’s SQuAD queries involved knowledge injection, which had minimal impact despite higher injection rates from some models.The passage suggests success relied more on understanding dataset distribution than prior knowledge.
  • DeepRetrieval adapts its augmentation strategy to dataset characteristics, explaining its consistent performance advantages across diverse retrieval tasks.
  • Raw performance metrics alone may not provide a complete picture of LLM-based query augmentation capabilities because knowledge injection affects datasets differently.

F Task Definitions, Metrics, and Retrieval Rewards

The paper defines retrieval tasks alongside evaluation metrics and task-specific rewards, with case studies spanning literature, evidence-seeking, sparse, and dense retrieval.

  • Table 8 organizes task definitions, evaluation metrics, and retrieval rewards for the different retrieval tasks.
  • The reward designs are task-specific rather than presented as a single universal retrieval objective.Table 8 is explicitly described as covering different retrieval tasks.
  • The reported case studies include literature searching on PubMed, evidence-seeking retrieval, classic sparse retrieval on FEVER, and classic dense retrieval on HotpotQA.

G.4 SQL Database Search

The SQL database search material is accompanied by a dedicated case-study table and a set of task-specific prompts covering search, retrieval, and knowledge-injection checks.

  • SQL Database Search: Table 13 presents a case study of SQL database search.
  • Prompt Design: The appendix includes prompts for literature search, evidence-seeking retrieval, classic sparse retrieval, and classic dense retrieval alongside the SQL-search materials.
  • Prompt Design: The SQL-search prompt is paired with a knowledge-injection check and cleaning stage.
  • Contributions: L. Cao implemented the SQL database search component and conducted the experiments on the BIRD and Spider datasets.
Loading 2503.00223v3…