PaperlayerView exact revision on arXiv

Source-linked AI summary

Retriever-Free Retrieval: Navigating Documents with Decision Models

Chao Wan, Paul Jünger, Beixi Du, Bodong Liu, Navaneetha Krishnan, Kilian Q. Weinberger

arXiv:2610.35868v1cs.AI

TL;DR

Retrieval often relies on lexical matching or fixed embedding spaces, so a generator cannot recover evidence its retriever misses. DecisionRAG navigates corpus hierarchies using shared-context, non-generative decisions; it outperforms dense retrieval with an 8B reranker on two benchmarks and retrieves across four benchmarks in 0.7–1.8 seconds per query.

  • Problem

    Conventional lexical or fixed-embedding retrievers can miss evidence that even a powerful generator cannot recover.

    Source: §1 · p. 1

  • Method

    DecisionRAG navigates corpus hierarchies by jointly evaluating candidates in a shared context at each level, without generating tokens or using a first-stage retriever.

    Source: §1 · pp. 1–2

  • Results

    13.8 R@1 points on MultiDoc2Dial and 17.4 Hit@1 points on Natural Questions: DecisionRAG outperforms dense retrieval with an 8B reranker; retrieval takes 0.7–1.8 seconds per query across four benchmarks.

    Source: §1 · p. 2, §5.1 · p. 7

  • Takeaways & Limitations

    Under a balanced hierarchy, the number of sequential decisions grows only logarithmically with corpus size, and retrieval can inherit improvements in decision models.

    Source: §8 · p. 10

    Retrieval quality depends on hierarchy structure and node descriptions, and irreversible pruning can discard relevant passages when their ancestors are dropped.

    Source: §7 · p. 10

Abstract

from arXiv · show

While language models have become dramatically more capable, retrieval still largely relies on lexical matching or similarity in fixed embedding spaces. This creates a capability asymmetry: what the retriever fails to find, even the strongest generator cannot recover. We propose DecisionRAG, which brings the capabilities of pretrained language models to retrieval using decision models that jointly evaluate candidate branches in a single forward pass, without generating any tokens, and show that judging candidates in a shared context makes their relevance estimates comparable. Like a person browsing a library, DecisionRAG navigates the corpus's inherent structure from broad topics down to individual passages, deciding at each level which branches are worth exploring. Without task-specific training or a first-stage retriever, DecisionRAG outperforms dense retrieval with an 8B reranker by 13.8 R@1 points on MultiDoc2Dial and 17.4 Hit@1 points on Natural Questions. Across four benchmarks, including a corpus of 2.68 million Wikipedia paragraphs, retrieval takes just 0.7-1.8 seconds per query.

1. Introduction

DecisionRAG uses pretrained language models to navigate corpus hierarchies with non-generative decisions, addressing the gap between increasingly capable generators and conventional retrievers. It does not require a first-stage retriever or task-specific training, and improves on dense retrieval with an 8B reranker on two reported benchmarks.

1. Introduction

  • Retrieval-augmented generation fetches external evidence at inference time to provide access to information absent from training and ground responses in verifiable sources.The cited background frames RAG as a core component of modern language-model pipelines.Source: §1 · p. 1
  • Conventional retrievers use term-frequency heuristics or fixed embedding similarity, creating a capability asymmetry because generators cannot recover evidence the retriever misses.Rerankers only reorder first-stage results, while token-generating agentic navigation can be slow and costly.Source: §1 · p. 1
  • Decision models answer multiple questions in parallel against a shared context, making candidate relevance judgments comparable without sampling tokens.This makes them a fit for applying pretrained language-model capability to retrieval choices.Source: §1 · p. 1
  • DecisionRAG navigates corpus structure from broad categories to passages, jointly evaluating candidates at each level in one forward pass without generating tokens.It requires no first-stage retriever, precomputed index, or task-specific training at query time.Source: §1 · p. 2
  • 13.8 R@1 points on MultiDoc2Dial and 17.4 Hit@1 points on Natural Questions: DecisionRAG outperforms dense retrieval with an 8B reranker without task-specific training.Across four benchmarks, including 2.68 million Wikipedia paragraphs, retrieval takes 0.7–1.8 seconds per query.Source: §1 · p. 2

2. Related Work

Prior retrieval approaches use lexical or embedding matching, reranking, agentic search, or hierarchical representations, with limits including candidate-pool dependence and sequential token-generation latency. DecisionRAG combines hierarchical retrieval with non-generative relevance judgments.

2. Related Work

  • Rerankers refine rankings but remain limited by candidate recall from the initial retrieval stage; cross-encoders score pairs, while LLM rerankers generate or iteratively select rankings.The cited approaches include permutation generation and selection from candidate subsets.Source: §2 · p. 2
  • Agentic and LLM-guided hierarchical methods navigate iteratively, but generating tokens sequentially at each step can incur substantial latency across multiple rounds.Related approaches include MemWalker, PageIndex, and LATTICE.Source: §2 · p. 2
  • Other related work includes trained tree navigation, dense hierarchical retrieval, recursively summarized hierarchies, and methods emphasizing document-level context.Examples named are DocQN, RAPTOR, and dense hierarchical retrieval.Source: §2 · p. 2
  • DecisionRAG combines hierarchical retrieval with non-generative relevance judgments, avoiding both a first-stage candidate pool and token generation during branch evaluation.It uses existing document structure where available and constructs document groups otherwise.Source: §2 · p. 2
  • BM25 and dense retrieval match query terms or embeddings, while DecisionRAG jointly judges hierarchy candidates, keeps top-k documents and then top-r passages, and never expands pruned documents.The figure caption marks its BM25 term frequencies, IDF values, scores, and rankings as illustrative.Source: Figure 2 · p. 3

3. Method

DecisionRAG models retrieval as passage finding in a hierarchical corpus and traverses that structure using shared-context decision judgments. Its procedure narrows the search through successive levels and can optionally refine the final ranking.

3.1. Problem Formulation

  • The corpus is modeled as a tree with passages at its leaves, increasingly broad groups at internal nodes, and a short description for each node.Source: §3.1 · p. 3

3.2. Relevance Judgments with Decision Models

  • Against a shared context, parallel binary judgments score candidates separately, while multiple-choice judgments evaluate options as competing alternatives.Source: §3.2 · p. 3

3.3. Hierarchical Retrieval

  • DecisionRAG narrows retrieval level by level, retaining promising branches while pruning candidates whose descendants are never evaluated.Source: §3.3 · p. 4
  • The procedure screens documents with binary judgments, scores passages within retained documents, and can optionally rerank top passages with multiple-choice judgments.Source: §3.3 · pp. 4–5
  • Because pruning is irreversible, retrieval budgets favor recall and trade coverage against cost.Source: §3.3 · p. 5

3.4. Hierarchy Construction and Scaling

  • Higher-order hierarchies can use existing corpus structure or offline document clustering, and need not be uniform.Source: §3.4 · p. 5
  • Although DecisionRAG needs no first-stage retriever, dense-retriever passages can be added to recover evidence discarded during pruning.Source: §3.4 · p. 5
  • Latency depends on hierarchy depth rather than corpus size, but the analytical bound concerns sequential depth, not total compute.Source: §3.4 · p. 5

4. Experimental Setup

DecisionRAG is evaluated on four retrieval benchmarks, using dataset-specific hierarchies and configurations. The setup compares conventional retrieval and reranking baselines using retrieval, answer-quality, cost, and latency measures.

4.1. Datasets and Corpus Construction

  • The four benchmarks cover goal-oriented dialogue, scenario-based questions, conversational technical-document retrieval, and open-domain search.Natural Questions uses its DAPR version; the benchmark set also includes MultiDoc2Dial, ConditionalQA, and MTRAG.Source: §4.1 · p. 5, Table 1 · p. 6
  • Natural Questions pages are clustered into 1,000 groups; the other three datasets use hierarchies built from existing document structure and metadata.Each Natural Questions cluster receives a one-sentence summary generated from its page titles.Source: §4.1 · p. 6

4.2. Retrieval Configurations

  • Conversational benchmarks use the last six dialogue turns and a notepad; final ranking scores the top 20 passages, or top five on Natural Questions.MultiDoc2Dial retains the top ten pages, while ConditionalQA retains three; MTRAG and Natural Questions use their own staged page and heading or cluster budgets.Source: §4.2 · p. 6
  • Navigation paths vary by dataset: help center→page→passage, page→paragraph, page→heading→passage, or cluster→page→paragraph.The paths correspond respectively to MultiDoc2Dial, ConditionalQA, MTRAG, and Natural Questions.Source: §4.2 · p. 6

4.3. Baselines

  • Baselines include first-stage lexical and dense retrieval, rerankers, and published reference results; the published results are not rerun under this evaluation setup.Rerankers are applied to dense-retrieved candidates; the baseline set also includes GPT-5 reranking.Source: §4.3 · p. 6

4.4. Evaluation Metrics and Protocol

  • The evaluation reports conventional retrieval metrics, downstream answer quality, paired-bootstrap 95% intervals, per-query cost, and online latency.A shared Qwen3.5-27B generator answers from each system’s top five passages; MTRAG answer acceptance is LLM-judged.Source: §4.4 · pp. 6–7

5. Results

Without task-specific training or a conventional first-stage retriever, DecisionRAG improves retrieval over dense retrieval with an 8B reranker on MultiDoc2Dial and Natural Questions. The results also cover downstream answers, task-trained judges, and measured latency and cost.

5.1. Retrieval Effectiveness

  • DecisionRAG, without task-specific training or a conventional retrieval index, substantially outperforms dense retrieval with an 8B reranker, improving MultiDoc2Dial R@1 by 13.8 percentage points and Natural Questions Hit@1 by 17.4 points.Source: §5.1 · p. 7
  • Against GPT-5 reranking, DecisionRAG has comparable or higher rank-one accuracy on MultiDoc2Dial and Natural Questions, while GPT-5 leads ConditionalQA Hit@1, 73.4 versus 69.7.DecisionRAG also records R@5 34.2 versus 32.5 for the best baseline on MTRAG; adding dense candidates improves R@10 on MultiDoc2Dial, ConditionalQA, and Natural Questions.Source: §5.1 · p. 7

5.2. Retrieval-Augmented Generation

  • On MTRAG, judged answer correctness rises from 39.4% with Qwen3-Reranker to 44.1%; adding dense candidates raises it to 47.3%, versus 34.0% for GPT-5 reranking.The dense-candidate configuration also achieves 54.8% exact match on Natural Questions and 47.6% on ConditionalQA.Source: §5.2 · p. 7, Table 3 · p. 7

5.3. Learning a Retrieval Judge

  • 60.7 R@1 on MultiDoc2Dial: the task-trained Qwen3-8B decision model exceeds the strongest published task-trained retriever by 5.7 points.The framework can use task-specific adaptation beyond the pretrained Jev model.Source: §5.3 · p. 8, Table 4 · p. 8

5.4. Latency and Cost

  • 0.68–1.76 seconds per query: DecisionRAG is faster than dense retrieval with Qwen3-Reranker-8B and GPT-5 reranking under the measured serving configurations.API costs range from 0.22¢ per query on MultiDoc2Dial to 2.33¢ on Natural Questions; the measured latency is 1.73 seconds on Natural Questions and 1.76 seconds on MTRAG.Source: §5.4 · p. 8, Table 5 · p. 8

6. Understanding DecisionRAG

The analyses examine how shared-context judgments, ranking choices, and navigation budgets affect DecisionRAG’s retrieval quality and efficiency. They also compare decision-based navigation with generation and reranking.

6.1. Scoring Candidates in a Shared Context

  • 53.7% Hit@1 with 1,000 candidates: shared-context judgments outperform independent scoring, and multiple-choice ranking of the binary top 255 retains much of its advantage; scoring selected-page passages together raises MultiDoc2Dial R@1 from 36.7% to 44.4%.Figure 5 tracks Hit@1 as the candidate pool grows; the MultiDoc2Dial gain is +7.7 points (95% CI: [3.3, 12.2]).Source: §6.1 · p. 8, Figure 5 · p. 8

6.2. Request Size

  • 60.6% Hit@1: multiple-choice ranking of all 200 candidates is 6.8 points above binary judgments of all 200, while binary scoring reaches 54.3% with 50 candidates per request.Figure 6 varies request size for a fixed 200-candidate pool; larger requests add little beyond about 50 candidates for binary scoring.Source: §6.2 · pp. 8–9, Figure 6 · p. 9

6.3. Decisions versus Generation

  • Token-by-token generation brings no consistent accuracy gain: GPT-5 leads DecisionRAG by 4.1 Hit@1 points on ConditionalQA but trails by 6.0 on Natural Questions, while being 34 to 145 times slower and 21 to 25 times more expensive.The comparison keeps DecisionRAG’s hierarchy and budgets fixed.Source: §6.3 · p. 9, Table 6 · p. 9

6.4. Does Multiple-Choice Ranking Improve Retrieval?

  • Multiple-choice ranking improves MultiDoc2Dial R@1 from 50.1% to 51.3% and ConditionalQA Hit@1 from 71.9% to 76.5%; its additional request takes 0.12–0.15 seconds and adds 4–33% API cost.Table 7 compares configurations with and without multiple-choice ranking while holding the candidate pool and preceding binary judgments fixed.Source: §6.4 · p. 9, Table 7 · p. 9

6.5. Navigation versus Reranking

  • Navigation beats reranking dense candidates by 7.7 Hit@1 points on ConditionalQA and 7.0 on Natural Questions; it ranks the relevant MultiDoc2Dial passage first 70.1% versus 61.9%.The authors suggest that reading whole pages in order rather than isolated passages may explain the difference.Source: §6.5 · pp. 9–10, Table 8 · p. 10

6.6. Navigation Budget and Hierarchy

  • Keeping 15% of Natural Questions’ clusters matches flat-scoring Recall@10, 86.2% versus 84.5%, at a fifth of the cost and a third of the latency; at 1% recall falls to 63.7%.Figure 7 compares cluster budgets; increasing MultiDoc2Dial’s retained pages from three to ten raises R@10 from 81.0% to 83.6%, but R@1 only from 46.9% to 47.4%, with 83% higher API cost and 28% higher median latency.Source: §6.6 · p. 10, Figure 7 · p. 10, Table 9 · p. 10

7. Limitations

DecisionRAG depends on a corpus hierarchy whose structure and node descriptions let the decision model identify relevant branches.

7. Limitations

  • Poorly organized documents or uninformative descriptions can hurt retrieval; irreversible pruning can also discard relevant passages when their ancestor branches are dropped.Larger exploration budgets and complementary dense retrieval can mitigate missed passages.Source: §7 · p. 10

8. Conclusion

DecisionRAG retrieves by navigating corpus hierarchies with non-generative decisions, jointly evaluating candidates in a shared context. Under a balanced hierarchy, its sequential decisions grow logarithmically with corpus size.

8. Conclusion

  • Under a balanced hierarchy, retrieval requires a number of sequential decisions that grows only logarithmically with corpus size.Candidates are evaluated jointly in a shared context, and retrieval progressively narrows the search space.Source: §8 · p. 10

A. Dataset and Corpus Details

The evaluation spans four differently structured corpora and describes how candidates are represented and routed through each corpus. Configuration choices are made on design sets separate from evaluation sets, without updating Jev’s parameters.

A.1. MultiDoc2Dial

  • MultiDoc2Dial uses a help-center–page–passage hierarchy over 4,110 passages, evaluated on 4,201 turns from 661 validation dialogues.Its test split provides no usable evaluation examples.Source: §A.1 · p. 14

A.2. ConditionalQA

  • ConditionalQA searches 58,483 paragraphs across 652 UK government guides, evaluating 285 development questions, 271 with annotated evidence.The relevant guide is not assumed known; evidence-labeled questions average about four paragraphs.Source: §A.2 · p. 14

A.3. MTRAG–IBM Cloud

  • MTRAG’s IBM Cloud subset contains 8,578 pages and 72,442 passages, with pages grouped into 172 products; evaluation covers 205 turns, 188 labeled.Its expanded relevance judgments add LLM-judged relevant passages to the original gold, reducing dependence on ELSER’s retrieval sessions.Source: §A.3 · p. 14

A.4. Natural Questions

  • Natural Questions uses 2,682,017 Wikipedia paragraphs, with evaluation on a random sample of 500 questions and pages clustered into 1,000 groups.The gold paragraphs for each question lie on one page; group descriptions are generated from page titles.Source: §A.4 · p. 14

A.5. Design Sets and Configuration Selection

  • Configuration selection uses design sets disjoint from evaluation, including synthetic and MTRAG-UN conversations because MTRAG has no training split.Selection does not update Jev’s parameters.Source: §A.5 · p. 15

B.1. Candidate Representations and Questions

  • At each stage, the judge receives the query, numbered candidate descriptions, and one yes/no question per candidate; conversational queries include six recent turns.Candidates appear in document order, grouped under parent pages where applicable, and each stage receives a new context without earlier-stage probabilities.Source: §B.1 · p. 15

B.2. MultiDoc2Dial and the Conversational Notepad

  • MultiDoc2Dial keeps the highest-scoring help center, conditionally retains alternatives, then scores pages and passages; a retrieval-built notepad tracks recently retrieved pages.The notepad uses the system’s own decisions, not gold evidence, and prior passages are neither marked nor excluded.Source: §B.2 · p. 15

B.3–B.6. Corpus routing and final ranking

  • ConditionalQA scores all guides and keeps three; MTRAG rescales an initial page shortlist in a shared context and adds a heading stage for pages with over 60 passages.For Natural Questions, cluster descriptions and page titles route the search before paragraphs are scored; paragraph probabilities are multiplied by their page’s. Final multiple-choice ranking uses the top 20 candidates, or top five on Natural Questions.Source: §B.3 · p. 15, §B.4 · p. 15, §B.5 · pp. 15–16

C. Full Benchmark Results

The full benchmark results compare DecisionRAG against retrieval and reranking systems, while the supplementary analyses report MTRAG comparisons and a shared-context ablation. The supplied passages do not provide the full table values across benchmarks.

C. Full Benchmark Results

  • DecisionRAG’s MTRAG R@5 exceeds released ELSER by 7.6 points, last-turn SPLADE-v3 by 6.8, and six-turn Qwen3-Reranker by 5.8.The reported 95% intervals are [2.2, 13.1], [1.9, 12.3], and [1.9, 9.6], respectively; it is also ahead of last-turn Qwen3-Reranker and GPT-5 reranking.Source: Table 12 · p. 17, C. Full Benchmark Results · p. 17

C.1. Retrieval-augmented generation

  • The retrieval-augmented generation setup uses Qwen3.5-27B to answer from passages ordered by each system’s rank.Natural Questions answers are limited to five words; MTRAG replies indicate when passages lack the answer, and a model judges answer correctness.Source: §C.1 · pp. 17–18

D.1. Shared context and conversational memory

  • Adding a notepad raises paired MultiDoc2Dial R@1 by 4.0 points, while shared scoring with notepads exceeds separate scoring by 7.7 points.Differences resample whole dialogues; reported intervals are [0.7, 8.0] and [3.3, 12.2].Source: §D.1 · p. 18, Table 13 · p. 18

D.2. Remaining retrieval errors

The remaining analyses locate retrieval errors, test relevance and duplicate-handling choices, and report latency measurements. Their results include sensitivity to larger page budgets and a faster MTRAG page-scoring setup.

D.2. Remaining retrieval errors

  • Even when navigation reaches the relevant page, passage ranking remains a substantial source of error: on ConditionalQA, the first result is from another page for 9.6% of evidence-labeled questions.With the gold page given, MultiDoc2Dial passage scoring reaches 62.7 R@1 and 94.3 R@5; ConditionalQA also ranks another paragraph from the gold page first for 13.7%.Source: §D.2 · p. 18

D.3. Dense candidates and multiple-choice ranking

  • With larger page budgets, added dense candidates and multiple-choice ranking yield at most +0.6 MultiDoc2Dial R@1 and +4.2 ConditionalQA Hit@1.Every MultiDoc2Dial interval contains zero; comparisons use design sets and their primary metrics.Source: §D.3 · p. 18, Table 14 · p. 19

E. Evaluation Sensitivity — identical passages

  • Text matching raises MultiDoc2Dial R@1 to 55.8 for DecisionRAG and 57.0 with dense candidates, versus 40.3 for dense retrieval with Qwen3-Reranker.The ordering of systems is unchanged; duplicate passages are grouped by normalized text so duplicate copies cannot create multiple hits.Source: §E.1 · p. 19

E.2. ConditionalQA under DAPR relevance judgments

  • Under DAPR relevance judgments, DecisionRAG exceeds dense retrieval with Qwen3-Reranker by +5.3 nDCG@10 points and GPT-5 by +4.0.DecisionRAG’s DAPR R@10 exceeds GPT-5 by +8.2; with dense candidates, nDCG@10 is 71.2.Source: §E.2 · p. 19

F. Latency and Cost

  • Using MTRAG page-pass requests capped at 10k tokens reduces median latency from 3.09 to 1.76 seconds without changing R@5 or nDCG@5 detectably.On 188 held-out turns, paired differences are −0.7 and −0.2 points, with intervals containing zero; reported system latency ratios are not controlled speedups because hardware and samples differ.Source: F. Latency and Cost · pp. 19–20, Table 15 · p. 20
  • DecisionRAG encodes no passages, while online measurements exclude offline indexing and, for dense-candidate rows, exclude dense encoding and search.Only Natural Questions requires an offline hierarchy-building pass; its page titles and first sentences are embedded for clustering and cluster descriptions are generated.Source: F. Latency and Cost · pp. 19–20

G. Open-Weight Judge Training

Open-weight DecisionRAG judges score candidate branches as yes/no options in a shared context, and retrieval-data training produces checkpoints with different evaluation coverage. The LoRA checkpoint is evaluated on both datasets, while MultiDoc2Dial-only checkpoints achieve higher MultiDoc2Dial recall.

G. Open-Weight Judge Training

  • Each candidate is posed as a yes/no question after shared context, and a scoring head estimates P(yes) from question-option similarities.Candidate questions cannot attend to one another; the head normalizes option scores with a softmax.Source: G. Open-Weight Judge Training · p. 20
  • General models train on public multiple-choice and classification data, using gold options and dataset distractors or sampled class labels, without MultiDoc2Dial or ConditionalQA data.The mixture includes reading comprehension, science and commonsense QA, inference, classification, dialogue, tool-use, and web-navigation decisions.Source: G. Open-Weight Judge Training · p. 21
  • Retrieval examples are generated from navigation stages: MultiDoc2Dial uses routing, page, and passage stages, while ConditionalQA uses page and paragraph candidates shaped to match inference inputs.ConditionalQA page examples include all 652 pages; paragraph examples cover paragraphs from five candidate pages.Source: G. Open-Weight Judge Training · p. 21
  • The reported 8B trained model uses LoRA and a fully trained head, trained for one epoch on a sample of MultiDoc2Dial dialogues and all ConditionalQA training examples.The model continues from the general 8B checkpoint; overlong passage-stage records are dropped rather than truncated in the described retrieval-data runs.Source: G. Open-Weight Judge Training · p. 21
  • Only the LoRA checkpoint is evaluated on both datasets; MultiDoc2Dial-only checkpoints achieve higher MultiDoc2Dial recall.Table 17 reports open-weight judge results.Source: G. Open-Weight Judge Training · p. 21, Table 17 · p. 21

Related papers

Loading 2610.35868v1…