Source-linked AI summary
Rethinking Reasoning-Intensive Retrieval: Evaluating and Advancing Retrievers in Agentic Search Systems
Yilun Zhao, Jinbiao Wei, Tingyu Song, Siyue Zhang, Chen Zhao, Arman Cohan
TL;DR
Reasoning-intensive retrieval lacks evaluation and training aligned with complementary evidence gathering in agentic search. The paper introduces BRIGHT-PRO and RTriever-Synth, and finds that aspect-aware and agentic evaluation reveal hidden retrieval behavior while RTriever-4B improves over its base model.
Problem
Existing benchmarks use narrow gold sets and isolated evaluation, while synthetic corpora often optimize single-passage relevance rather than complementary evidence portfolios for agentic search.
Method
The paper builds BRIGHT-PRO with expert-annotated multi-aspect evidence and static and agentic protocols, then trains RTriever-4B on aspect-decomposed RTriever-Synth.
Results
Aspect-aware and agentic evaluation expose retrieval behaviors hidden by standard metrics, while RTriever-4B substantially improves over its Qwen3-Embedding-4B base.
Takeaways & Limitations
Retriever evaluation and training should target full evidence portfolios rather than individual passages.
Takeaways & Limitations
BRIGHT-PRO is limited to seven expert domains and a relatively small set of costly expert annotations, while the reported training uses one-positive-one-negative triplets.
Abstract
from arXiv · showhide
Reasoning-intensive retrieval aims to surface evidence that supports downstream reasoning rather than merely matching topical similarity. This capability is increasingly important for agentic search systems, where retrievers must provide complementary evidence across iterative search and synthesis. However, existing work remains limited on both evaluation and training: benchmarks such as BRIGHT provide narrow gold sets and evaluate retrievers in isolation, while synthetic training corpora often optimize single-passage relevance rather than evidence portfolio construction. We introduce BRIGHT-Pro, an expert-annotated benchmark that expands each query with multi-aspect gold evidence and evaluates retrievers under both static and agentic search protocols. We further construct RTriever-Synth, an aspect-decomposed synthetic corpus that generates complementary positives and positive-conditioned hard negatives, and use it to LoRA fine-tune RTriever-4B from Qwen3-Embedding-4B. Experiments across lexical, general-purpose, and reasoning-intensive retrievers show that aspect-aware and agentic evaluation expose behaviors hidden by standard metrics, while RTriever-4B substantially improves over its base model.
1 Introduction
Reasoning-intensive retrieval must support multi-step evidence gathering, yet existing benchmarks and training corpora emphasize isolated passage relevance. The paper introduces aspect-aware evaluation and complementary-evidence training, with experiments showing improved retriever and agentic-search assessment.
- Traditional retrieval succeeds for factoid and single-hop queries but struggles when complex questions require multi-step reasoning and diverse evidence.
- Existing evaluation uses narrow gold passage sets and isolated retriever testing, while synthetic training often rewards single-positive ranking instead of complementary evidence portfolios.
- BRIGHT-PRO expands queries with expert-annotated multi-aspect gold evidence and evaluates retrievers in both static and iterative agentic workflows.The agentic protocol measures reasoning completeness, iteration efficiency, and final response quality.
- RTriever-Synth decomposes reference-answer reasoning into aspects, generates complementary positives, and creates positive-conditioned hard negatives that omit required aspects.
- The study shows that aspect-aware and agentic protocols expose retrieval behaviors hidden by standard rankings, while RTriever-4B improves substantially over its base model.
2 Related Work
Related work frames reasoning-intensive retrieval as retrieval of evidence useful for multi-step reasoning, alongside agentic systems that iteratively search, read, and synthesize. BRIGHT-PRO extends these directions by evaluating retrievers within such workflows.
- Reasoning-Intensive Information Retrieval: BRIGHT targets queries requiring multi-step reasoning to identify useful evidence rather than merely superficially relevant passages.
- Reasoning-Intensive Information Retrieval: Recent reasoning-retrieval work trains retrievers with synthetic queries and hard negatives to encourage reasoning-aware evidence selection.
- Agentic Search System and Evaluation: Agentic search systems combine LLM planning with iterative search, reading, and synthesis for complex queries, motivating evaluation beyond isolated retrieval.
3 BRIGHT-PRO Benchmark
BRIGHT-PRO builds an expert-annotated benchmark around reasoning aspects, weighted evidence, and reviewed positive documents. Its construction iteratively refines aspect coverage and verifies annotation reliability.
- BRIGHT-PRO uses the StackExchange subset of BRIGHT because it best represents open-domain natural-language reasoning.
- Reasoning Aspect Annotation: Each query is decomposed into reasoning aspects representing coherent subproblems or perspectives, because complete reasoning requires complementary evidence.
- Reasoning Aspect Annotation: Aspect importance is weighted from annotator Likert scores so evaluation emphasizes contributions to correctness rather than interchangeable evidence accumulation.
- Positive Document Collection: Original positives are re-audited, new documents are accepted only when they provide explicit, precise, and credible aspect evidence, and segments become independent positives.
- Aspect-Guided Positive Doc Collection: Annotators iteratively split, merge, or clarify aspects as document collection reveals missing assumptions, dependencies, or overlap.
- Data Quality Control: Weighted Cohen’s κ = 0.742 indicates stable importance ratings across independent reviewers during quality control.
4 BRIGHT-PRO Evaluation Protocol
The evaluation separates static retrieval quality from retriever performance inside a controlled agentic workflow. Static metrics reward aspect novelty, while agentic protocols measure evidence coverage, answer quality, and search efficiency.
- Static Evaluation: Static evaluation uses aspect-annotated gold passages and α-nDCG@k with novelty penalty α = 0.5 to reward complementary aspect coverage.
- Agentic Pipeline and Response Evaluation: Each retriever is placed in the same LLM agent with one search tool, identical prompts and interfaces, matched retrieval depth, and fixed stopping budgets.
- Agentic Pipeline and Response Evaluation: GPT-5 judges answers against references built from annotated aspects, assigning aspect coverage and overall quality scores aggregated into reasoning completeness and answer quality.
- Fixed-Round Protocol: Fixed-round evaluation uses exactly R ∈ {1, 2, 3} rounds with five passages per round, isolating retriever quality under matched interaction budgets.
- Adaptive-Round Protocol: Adaptive-round evaluation lets the agent stop when evidence is sufficient, measuring retrieval rounds alongside completeness, quality, and AER.AER uses overall quality, search-round count, and γ = 0.05.
5 RTriever: Learning to Retrieve Complementary Evidence
RTriever-Synth trains retrievers to select complementary evidence by decomposing analytical queries into reasoning aspects and pairing aspect-specific positives with conditioned hard negatives. RTriever-4B is obtained by LoRA fine-tuning Qwen3-Embedding-4B on this synthetic corpus.
- Synthetic corpus construction: The pipeline realizes each reasoning aspect as a complementary positive passage rather than synthesizing positives directly from the query.This design targets evidence coverage across distinct subproblems within the same analytical query.
- Synthetic corpus construction: RTriever-Synth rewrites MS MARCO seeds into realistic deep-research queries before generating training evidence.The pipeline uses MS MARCO queries as semantic seeds and PersonaHub personas as format examples for more natural query surfaces.
- Synthetic corpus construction: Each query receives a self-contained reference answer that is decomposed into two to three non-overlapping reasoning aspects.Each aspect is tied to an explicit rationale identifying the corresponding part of the reasoning.
- Synthetic corpus construction: Positive-conditioned hard negatives share topical or lexical cues with the query but omit the information needed by the positive passage.The negative generator conditions on the query and positive-passage summaries and produces a rationale for each failure.
- Training: 140K complete query-positive-negative bundles are generated after filtering, and RTriever-4B is LoRA fine-tuned on sampled one-positive-one-negative training pairs.The model starts from Qwen3-Embedding-4B and is trained on a randomly sampled subset of RTriever-Synth.
6 Experiment
BRIGHT-PRO evaluates twelve public retrievers and RTriever-4B through static, fixed-round, and adaptive-round protocols. Aspect-aware and agentic results separate retriever behaviors that standard static rankings do not reliably predict.
- Experiment setup: BRIGHT-PRO benchmarks twelve publicly available retrievers plus RTriever-4B across classical lexical, general-purpose, and reasoning-intensive categories.Baselines are evaluated using their native tokenizer, embedding interface, pooling strategy, and validated context length.
- Static retrieval: Static evaluation reports α-nDCG@25 across BRIGHT-PRO subsets and NDCG@10 for BRIGHT, with models ranked by overall performance.The static table highlights the best and second-best models for each subset.
- Static retrieval: Aspect-aware static evaluation places five reasoning-intensive retrievers, including RTriever-4B, 4–14 points above every general-purpose embedder.RTriever-4B beats every 7–8B general-purpose retriever despite having 4B parameters.
- Fixed-Round Agentic Retrieval: In fixed-round retrieval, BGE-Reasoner-8B leads with Round-3 α-nDCG@15 = 63.0 and Overall = 4.31, while RTriever-4B ranks third on Overall at 4.25.DIVER-4B ranks ahead of DIVER-4B-1020 on Overall despite trailing it in static retrieval.
- Fixed-Round Agentic Retrieval: BM25 rises from static performance of 40.3 to Round-3 α-nDCG@15 = 51.5 when LLM follow-ups supply concrete keywords.This agentic behavior is not visible in static evaluation.
- Adaptive-Round Agentic Retrieval: Adaptive evaluation ranks BGE-Reasoner-8B first at AER = 3.65 and RTriever-4B second across the two agents, while penalizing longer search.GTE-7B has Overall = 4.51 but AER = 3.44 because it requires 6.67 rounds.
- Qualitative analysis: Trace analysis identifies early evidence efficiency, evidence deprivation, repetition bias, aspect tunnel vision, and hypothesis hopping as recurring agentic behaviors.An Antarctic ice-sheet run covers all four reasoning aspects in three rounds, whereas other runs remain speculative or fail to expand evidence coverage.
7 Conclusion
The paper introduces multi-aspect benchmark supervision, agentic retriever-in-the-loop evaluation, and aspect-decomposed synthetic training. Its experiments show that evidence-portfolio evaluation reveals behaviors missed by single-passage metrics and that RTriever-4B improves over its base model.
- Contributions: BRIGHT-PRO adds multi-aspect gold sets and agentic retriever-in-the-loop protocols, while RTriever-Synth trains complementary evidence selection.The two resources address evaluation and training for reasoning-intensive retrieval.
- Findings: Aspect-aware evaluation exposes retrieval behaviors hidden by single-passage metrics, and static retrieval quality does not always predict agentic utility.The conclusion frames these findings around evaluating complete evidence portfolios rather than individual passages.
- Findings: RTriever-4B substantially improves over its Qwen3-Embedding-4B base after training on complementary positives and positive-conditioned negatives.The conclusion identifies these evidence structures as useful supervision for reasoning-intensive retrieval.
Limitations and Future Work
The benchmark is limited in domain coverage and scale, while the training study uses a simple one-positive-one-negative objective. The authors identify broader annotation and richer training objectives as future directions.
- Scope and data: BRIGHT-PRO uses the StackExchange subset of BRIGHT, covering seven expert domains that may not represent the full diversity of reasoning-intensive retrieval.The authors propose adding broader expert domains to improve representativeness.
- Scope and data: BRIGHT-PRO contains a limited number of examples because human expert annotation is costly.Semi-automated or hybrid annotation pipelines are proposed to scale data creation while maintaining quality and reliability.
- Training scope: RTriever-4B is fine-tuned on a subset of one-positive-one-negative triplets, leaving multi-positive objectives and aspect-aware sampling unexplored.The authors also identify negative curricula and the effects of synthetic evidence structure as future research directions.
B Data Quality Control Protocol
The paper defines aspect-aware retrieval metrics and evaluates retrieval within controlled agentic protocols. Data and answer-quality procedures use reviewed reasoning aspects, fixed prompts, shared settings, and a fixed query sample.
- Review: Reviewers verify that reasoning aspects are comprehensive, balanced, conceptually sound, and collectively capture the complete reasoning process.
- Review: Positive documents are re-examined to ensure explicit, credible, directly relevant evidence for their assigned aspects, with redundant or loosely related content removed.
- Metrics: Aspect-aware metrics map gold passages to unique aspects and measure aspect coverage, diversity, weighted recall, standard recall, and NDCG.
- Answer evaluation: GPT-5 generates citation-grounded reference answers from annotated aspects and positive passages, while an LLM judge scores aspect coverage and holistic answer quality.
- Agentic protocol: Both agent backends share prompts, tool schema, top-k, truncation, and interaction limits, while evaluation uses the same fixed sample of 175 queries for every retriever.
G Data Examples
The appendix includes domain-specific BRIGHT-PRO examples spanning biology, earth science, economics, psychology, robotics, Stack Overflow, and sustainable living.
- Examples: BRIGHT-PRO includes a Biology example.
- Examples: BRIGHT-PRO includes an Earth Science example.
- Examples: BRIGHT-PRO includes an Economics example.
- Examples: BRIGHT-PRO includes a Psychology example.
- Examples: Additional examples cover Robotics, Stack Overflow, and Sustainable Living.
H Experiment Results
The experiments assess retrievers with static and agentic measures, then illustrate how adaptive search can achieve efficient coverage, continue after success, or tunnel into one aspect.
- Evaluation results: The appendix reports A-Recall@25, Recall@25, and NDCG@25 across domains, while agentic results use cumulative α-nDCG cutoffs through round 3.
- Early-round efficiency: Three search rounds suffice to cover all four reasoning aspects for the Antarctic ice-sheet question.
- Failure cases: The Gazebo plugin case reaches thirteen rounds without retrieving gold evidence and produces speculation that the plugin no longer exists.
- Failure cases: Repeated off-target retrievals dominate twelve adaptive rounds for the interfamilial-hybridization question.
- Aspect tunnel vision: All seven climate-data search rounds drill into one aspect while never querying the user’s data-source question.
- Hypothesis hopping: The interruption query runs six rounds, versus a task median of five, although the first gold passage appears in round 1 and the last new gold evidence in round 6.