PaperlayerView exact revision on arXiv

Source-linked AI summary

Revisiting Text Ranking in Deep Research

Chuan Meng, Litu Ou, Sean MacAvaney, Jeff Dalton

arXiv:2602.21456v2cs.IRcs.AIcs.CL

TL;DR

Deep research relies on ranking methods whose behavior is difficult to assess through opaque web-search APIs. The paper reproduces ranking practices across retrieval units, pipelines, and query formats, finding strong benefits from passage retrieval, suitable rankers, re-ranking, and Q2Q query translation.

  • Problem

    Black-box search APIs leave the performance of established text-ranking methods in deep research poorly understood.

  • Method

    The paper evaluates 2 open-source agents, 5 retrievers, and 3 re-rankers on BrowseComp-Plus across retrieval units, pipeline configurations, and query formats.

  • Results

    Text-ranking methods remain effective: BM25 with appropriate setup, passage units, re-ranking, and Q2Q improve reported deep-research ranking or answer outcomes.

  • Takeaways & Limitations

    Passage-level units, retrievers suited to web-search-style queries, re-ranking, and mitigation of training–inference query mismatch are supported components for deep research ranking.

  • Takeaways & Limitations

    The study uses only two LLM agents, and broader evaluation across model families and larger sizes remains future work.

Abstract

from arXiv · show

Deep research has emerged as an important task that aims to address hard queries that need extensive open-web exploration. To tackle it, most prior work equips large language model (LLM)-based agents with opaque web search APIs, enabling agents to iteratively issue search queries, retrieve external evidence, and reason over it. Despite search's essential role in deep research, black-box web search APIs leave the behaviour of established text ranking methods in deep research largely unclear. To fill this gap, we reproduce key findings and best practices for text ranking methods in deep research. We examine their effectiveness from three perspectives: (i) retrieval units (documents vs. passages), (ii) pipeline configurations (different retrievers, re-rankers, and re-ranking depths), and (iii) query characteristics (the mismatch between agent-issued queries and the training queries of text rankers). We perform experiments on BrowseComp-Plus, a deep research dataset with a fixed corpus, evaluating 2 open-source agents, 5 retrievers, and 3 re-rankers. We find that agent-issued queries typically follow web-search-style syntax (e.g., quoted exact matches), favouring lexical, learned sparse, and multi-vector retrievers; passage-level units are more efficient under limited context windows, and avoid the difficulties of document length normalisation in lexical retrieval; re-ranking is highly effective. We further propose a query-to-question (Q2Q) method that translates agent-issued queries into natural language questions, significantly reducing the query mismatch.

1 Introduction

The paper investigates how established text-ranking practices transfer to deep research, where black-box search APIs obscure ranking behavior. Across retrieval units, pipelines, and query formats, it evaluates methods and proposes Q2Q to address query mismatch.

  • Motivation: Black-box web search APIs leave the effectiveness of established text-ranking methods in deep research poorly understood.
  • Research gaps: The study examines document versus passage retrieval, re-ranking configurations, and mismatch between agent-issued and training queries.
  • Experiments: BrowseComp-Plus experiments cover 2 open-source agents, 5 retrievers, and 3 re-rankers.
  • Findings: Passage-level units support more search and reasoning iterations within context limits, while BM25 on passages reaches 0.572 accuracy in the reported retrieval settings.
  • Findings: Re-ranking improves ranking effectiveness and answer accuracy while reducing search calls, with deeper re-ranking producing larger gains.
  • Findings: Q2Q translates agent-issued web-search queries into natural-language questions and significantly improves neural retrieval and re-ranking.

2 Task definition

Deep research extends text ranking from single-shot retrieval to an iterative agent loop. At each step, the agent reasons, optionally searches, receives ranked results, and continues toward an answer.

  • Ad-hoc search: In ad-hoc search, a ranking method returns a ranked list of k documents from corpus C for user query q_u.
  • Deep research: In deep research, agent A takes q_u, performs repeated reasoning and search, and ultimately produces answer a.
  • Agent loop: The ranking method returns D_t, which is fed back to the agent to produce the next reasoning trace and continue the loop.
  • Assumption: Some agents may invoke search or reasoning consecutively, although the simplified definition assumes alternating steps.

3 Methodology

The methodology section organizes the study around research questions, experimental design, agents, ranking methods, dataset, evaluation protocol, and the proposed method.

  • The paper’s methodology covers research questions, experimental design, agents and ranking methods, the dataset, evaluation protocol, and the proposed method.

3.1 Research questions and experimental design

The experiments compare retrieval units, ranking pipelines, and query formats in order to test how text-ranking choices affect deep research performance.

  • RQ1: RQ1 compares retriever effectiveness on passage-level and document-level corpora.
  • RQ1: Document retrieval is evaluated with truncated documents, optional full-document reading, passages, and passages with optional source-document reading.
  • RQ2: RQ2 evaluates re-ranking across initial retrievers, re-ranker types, and re-ranking depths.
  • RQ3: RQ3 tests whether agent-issued queries and natural-language training-style questions produce different ranking performance.
  • RQ3: Q2Q translates agent-issued web-search queries into natural-language questions similar to those used in MS MARCO.

3.2 Experimental setup

The experiments evaluate representative agents, retrievers, re-rankers, and query reformulation methods on BrowseComp-Plus using document- and passage-level ranking units.

  • Agents: Two open-source LLM agents are evaluated, including gpt-oss-20b and GLM-4.7-Flash (30B).Both agents are trained to invoke web search; the study focuses on feasible model sizes rather than broader scaling.
  • Text ranking methods: The ranking setup covers lexical-based sparse, learned sparse, single-vector dense, and multi-vector dense retrievers.Representative methods include BM25, SPLADE-v3, RepLLaMA, Qwen3-Embed-8B, and ColBERTv2.
  • Text ranking methods: Three pointwise re-rankers represent different effectiveness–efficiency trade-offs: monoT5-3B, RankLLaMA-7B, and Rank1-7B.The selected methods range from an inexpensive non-reasoning model to an LLM-based and chain-of-thought-based re-ranker.
  • Dataset: BrowseComp-Plus contains 1,266 fact-seeking, reasoning-intensive queries with a document corpus and human-verified relevance judgments.On average, each query has 2.9 gold, 6.1 evidence, and 76.28 negative documents.
  • Corpus construction: Documents are segmented into canonical passages of at most 250 words, with passage-to-document mappings and document titles prepended when available.Passage retrieval is evaluated by mapping retrieved passages back to documents using the maximum retrieved passage score.
  • Evaluation: The evaluation reports search calls, recall, accuracy, and completion rate under the BrowseComp-Plus protocol.Accuracy uses an LLM judge, while completion rate measures whether an agent finishes before context-window or output-token limits.

4 Results and Discussions

Across the evaluated deep-research settings, passage retrieval, tuned BM25, re-ranking, and Q2Q reformulation improve agent effectiveness, while benefits depend on corpus units, context windows, and query format.

  • Retrievers on passage and document corpora: Passage retrieval improves answer accuracy over document retrieval without a full-document reader, especially for the shorter-context gpt-oss-20b agent.With SPLADE-v3, gpt-oss-20b reaches 0.516 accuracy on passages versus 0.476 on documents, an 8.4% relative improvement; it also makes more search calls and completes more often.
  • Retrievers on passage and document corpora: BM25 with gpt-oss-20b achieves the highest passage-corpus recall of 0.616 and answer accuracy of 0.572 across the retrieval settings.Agent-issued queries use keywords, phrases, and quotation marks for exact matching, making them well suited to lexical retrieval.
  • Retrievers on passage and document corpora: Single-vector dense retrievers consistently underperform the smaller SPLADE-v3 and ColBERTv2 retrievers across the evaluated settings.The paper relates this pattern to the stronger generalisation of learned-sparse and multi-vector retrieval across diverse query formats.
  • Retrievers on passage and document corpora: Enabling a full-document reader reduces search calls and recall but improves answer accuracy, with SPLADE-v3 accuracy rising from 0.476 to 0.529 for gpt-oss-20b.The reader compensates for information loss from document truncation; on passages it slightly reduces BM25 accuracy from 0.572 to 0.542.
  • BM25 hyperparameters: BM25’s document-corpus performance depends strongly on length normalisation, with k1 = 10 and b = 1 appearing to be a document-retrieval sweet spot.The default and optimal settings differ more on documents than passages, and larger b values generally improve performance.
  • Re-ranking in deep research: Re-ranking consistently improves recall and accuracy while typically reducing search calls, and BM25–monoT5 at depth 50 reaches 0.716 recall and 0.689 accuracy.Relative to no re-ranking, this configuration gains 16.23% recall and 20.45% accuracy while reducing search calls by 10.98%; deeper re-ranking generally improves effectiveness.
  • Training–inference query mismatch: Q2Q (Q+R) reformulation improves neural retrieval and re-ranking by reducing mismatch between agent-issued web-search queries and natural-language training queries.With SPLADE-v3, Q2Q (Q+R) yields relative gains of 7.34% recall and 7.95% accuracy, whereas Q2Q (Q) provides little to no improvement.

5 Related Work

Prior deep-research work spans neural and lexical text ranking, multi-hop web exploration, and increasingly transparent evaluations. The supplied examples define how query reformulation and reasoning-based re-ranking are examined in this setting.

  • Text ranking: Text-ranking research includes lexical, single-vector dense, multi-vector dense, learned sparse, and cross-encoder re-ranking methods.The passage identifies BM25 as a long-standing unsupervised lexical retriever and describes neural-ranker development from pre-trained language models and labeled data.
  • Deep research: Deep research extends multi-hop question answering through iterative web-scale search and evidence synthesis across the open web.Most existing approaches equip LLM agents with live web search APIs.
  • Reasoning-based re-ranking: Rank1 can incorrectly treat independent keywords as a coherent semantic unit when reasoning over an irrelevant retrieved passage.The example describes a search containing separate terms such as “radiation” and “protein,” followed by an incorrect prediction.
  • Query reformulation: Q2Q translates agent-issued search queries into natural-language questions similar to MS MARCO training queries.The comparison evaluates raw agent queries against natural-language questions, while Q+R additionally incorporates the agent’s recent reasoning trace.
  • Transparent evaluation: BrowseComp-Plus enables white-box analysis by pairing a fixed document corpus with human-verified relevance judgments.This resource addresses the limited transparency of live web-search APIs and supports systematic evaluation of ranking methods.

6 Conclusions & Future Work

The paper finds that established IR ranking methods remain effective for deep research, while query-format mismatch limits neural rankers. It also identifies passage-level retrieval and broader model and configuration evaluations as important directions.

  • Conclusions: IR text-ranking methods remain highly effective in deep research across the evaluated settings.The experiments covered 2 open-source agents, 5 retrievers, and 3 re-rankers on BrowseComp-Plus.
  • Conclusions: BM25 with appropriate setup outperforms neural rankers in most cases, while learned sparse and multi-vector dense retrievers generalise better than single-vector dense retrievers.gpt-oss-20b with BM25 on the passage corpus achieves the highest answer accuracy across the study’s retrieval settings.
  • Conclusions: Re-ranking improves recall and answer accuracy while reducing search calls, with deeper re-ranking depths further amplifying these gains.This supports re-ranking as an effective component of deep-research ranking pipelines.
  • Conclusions: Web-search-style agent queries induce distribution drift for neural rankers, whereas Q2Q significantly mitigates this drift and improves neural ranking performance.The mismatch also limits Rank1, which often misinterprets such queries and shows no clear advantage over non-reasoning methods.
  • Conclusions: Passage-level units benefit agents with limited context windows and reduce sensitivity to document-length normalisation in lexical retrieval.The paper identifies this as an additional finding beyond reproducing established ranking results.
  • Future Work: Future work should evaluate more LLM agents and model sizes, additional rankers and configurations, and conversational deep-research scenarios.The authors identify these as directions for assessing generalisability and extending the task setting.
Loading 2602.21456v2…