PaperlayerView exact revision on arXiv

Source-linked AI summary

Scan or Index? large Evidence Coverage, Answer Quality, and Cost in Long-Document RAG

Juan Terven, Diana-Margarita Córdova-Esparza, Julio-Alejandro Romero-González, Edgar Chavez-Urbiola, Juan Irving Vasquez, Julian Andres Ramirez-Bautista, Silvia L. Chaparro-Cárdenas, Gendry Alfonso-Francia

arXiv:2610.28957v1cs.CL

TL;DR

Retrieval for long documents must balance access to useful evidence against repeated processing, while evidence coverage and answer quality may diverge. This paper compares 11 retrieval configurations across research articles and textbooks, tracing evidence, latency, cost, and generated-answer scores. On books, Hybrid + Jev leads in evidence coverage, while Hybrid + Decisions leads in Answer Accuracy among retrievers.

  • Problem

    For questions about long documents, it remains unclear whether broader passage access preserves more useful evidence or improves answers enough to justify repeated work.

    Source: §1 · p. 1

  • Method

    The study compares 11 retrieval configurations on QASPER articles and complete textbooks, tracing retrieved evidence, latency, API cost, and answers from a fixed generator.

    Source: §1 · pp. 1–2, §2.1 · p. 3

  • Results

    94.1% evidence coverage for Hybrid + Jev was the highest observed on books, while Hybrid + Decisions had the highest observed retriever Answer Accuracy (0.892).

    Source: §5 · p. 9

  • Takeaways & Limitations

    Full scans can lose evidence during reduction, and retrieval choice calls for considering evidence retention, answer quality, and repeated-use cost together.

    Source: §5 · p. 9

    The evaluation concerns retrieval within a known document under five-passage and 2,500-token limits, with eight textbooks and automated answer scores.

    Source: §2.3 · p. 5, §4 · p. 9

Abstract

from arXiv · show

Scanning every passage in a long document seems thorough, but does it preserve more useful evidence or produce better answers? We compare 11 retrieval configurations spanning lexical and dense search, hybrid reranking, and full-document scanning with Jev, OpenAI Decisions, and Cloudflare Clef. Our evaluation uses 396 QASPER questions and 80 questions from eight complete textbooks, with common passages, a five-passage limit, and a fixed answer generator. On books, Hybrid + Jev achieves the highest observed evidence coverage (94.1%), compared with 88.7% for Jev full scan and 81.3\% for OpenAI dense retrieval. Full scans examine every passage initially, yet later reduction steps can discard relevant evidence. Higher evidence coverage also does not consistently produce better answers: Hybrid + Decisions achieves the highest observed book Answer Accuracy among retrieval methods (0.892), despite recovering less reference evidence. Full scanning incurs substantially higher latency on books under the tested deployments, while reusable dense retrieval has lower projected API cost than paid decision-based methods as queries accumulate. Most QASPER articles already fit within the hybrid candidate pool, limiting the benefit of scanning. These results highlight evidence retention, answer quality, and reuse cost as complementary criteria for choosing a long-document retrieval strategy.

1 Introduction

This paper compares how retrieval methods retain evidence from a known document and whether broader access is worth the repeated work. Its contribution is to connect evidence retention, answer quality, reduction losses, and reuse cost while distinguishing source-evidence coverage from answer correctness.

1 Introduction

  • Dense retrieval reuses passage representations across queries, while full-document Jev scanning processes passages in windows and can lose evidence during later reduction.The central question is whether broader access improves final evidence enough to justify repeated work.Source: §1 · p. 1
  • On books, Hybrid + Jev retained the most annotated evidence on average while requiring much less query time than Jev full scanning in the measured deployment.The study treats evidence coverage and answer correctness as distinct outcomes.Source: §1 · p. 2

1.1 Related work

  • Prior retrieval work supplies lexical, dense, token-level, and hybrid ranking methods, while model-based reranking compares passages within candidate sets.The paper focuses on how annotated evidence survives repeated selection across a document.Source: §1.1 · p. 2
  • Long-context approaches vary in how they present documents, but this paper's full scan selects original passages and supplies only finalists to the generator.It does not provide the generator with the entire document.Source: §1.1 · p. 2
  • Retrieval and answer evaluation measure different properties; source-span coverage can miss alternative support, and automated answer scores do not verify citation correctness.The study also separates human-referenced articles from textbooks evaluated with model-generated questions and interprets scores within model-based evaluation limits.Source: §1.1 · p. 2

2 Experimental design

The experiments follow evidence from each specified document through retrieval and answer generation, holding passages and the answer generator common across methods. They compare configurations on articles and complete textbooks, tracking evidence, latency, API cost, and answer quality.

2.1 Data and retrieval setup

  • The evaluation uses 396 QASPER questions and 80 questions across eight textbooks, with questions and references fixed before testing.Textbook references were primarily model-generated and checked against sources for textual alignment, not independently human-validated.Source: §2.1 · p. 3
  • Articles contain 2–70 passages, while books contain 662–2,245; methods return up to five passages, and hybrid rerankers use a shared pool of up to 50 candidates.The 11 configurations include Jev, Decisions, and Clef selecting from that pool or scanning all passages before reduction; all use common passages.Source: §2.1 · p. 3, Table 1 · p. 3, Figure 1 · p. 4

2.2 Evidence, latency, and API cost

  • Evidence coverage measures recovered annotated support, with equal-weight evidence units, overlapping tokens counted once, and complete recovery requiring one full alternative evidence set.Coverage applies to answerable questions; failed assigned retrievals receive zero, and reduction traces follow evidence through selection stages.Source: §2.2 · p. 4
  • Latency is measured over completed executions under specified deployments, while API-cost projections combine per-document preparation cost with per-query charges.Query-cost averages include unanswerable queries and recorded failed-attempt charges; local computing resources are unpriced.Source: §2.2 · pp. 4–5

2.3 Answer evaluation

  • Answers are generated with a fixed model and 2,500-token context, then scored for faithfulness and reference agreement using automated judgments.The evaluation remains automated without independent human assessment, and neither metric establishes citation-label correctness.Source: §2.3 · p. 5

3 Results

Across the 11 methods, the clearest differences appear on books: Hybrid + Jev leads observed evidence coverage, while Hybrid + Decisions leads observed Answer Accuracy among retrievers. Full scans do not consistently improve book results and take substantially longer.

3 Results

  • Hybrid + Jev has the highest observed book evidence coverage, while Hybrid + Decisions has the highest observed book Answer Accuracy among retrievers.Full scanning does not consistently improve either result on books.Source: §3 · p. 5

3.1 Evidence coverage and reduction

  • On QASPER, Jev, Decisions, and Clef exceed 96% evidence coverage in both configurations, versus 90.4% for OpenAI dense retrieval.The shared hybrid candidate pool contains the complete article in 395 of 396 questions, so the corpus offers little separation between preselection and full scanning.Source: §3.1 · p. 5, Table 2 · p. 6
  • 94.1% coverage for Hybrid + Jev exceeds 88.7% for Jev full scan, 76.6% for Decisions full scan, and 91.6% for Clef full scan on books.Jev and Decisions fail on four and eight book scans, respectively, while Clef completes all 80.Source: §3.1 · p. 5, Table 2 · p. 6
  • Jev full-scan coverage falls from 100% initially to 93.2% in the final five passages, with most loss occurring during intermediate reductions.Clef retains 98.8% before its final comparison but ends at 91.6%; Decisions ends at 85.2% across 72 completed scans. Figure 2 traces completed scans, whereas Table 2 includes failed attempts.Source: §3.1 · p. 6, Figure 2 · p. 6

3.2 Latency and API cost

  • On books, median hybrid reranking takes 1.72 s with Jev, 4.51 s with Decisions, and 54.69 s with Clef, versus 25.10 s, 63.36 s, and 23.3 min for full scans.Hybrid times exclude candidate generation, while full-scan times include selection and reduction; these measurements describe the tested deployments.Source: §3.2 · p. 6, Table 3 · p. 7
  • At 1,000 queries per book, projected API cost is US$0.01351 for OpenAI dense, at least US$2.52 for Hybrid + Jev, and at least US$64.43 for Jev full scan.Under the measured book-query mix, dense retrieval becomes cheaper than Hybrid + Jev by the sixth query; unresolved charges could move the crossover earlier.Source: §3.2 · p. 7, Figure 3 · p. 7, Table 4 · p. 8

3.3 Answer quality

  • Hybrid + Decisions scores 0.892 in book Answer Accuracy, versus 0.864 for Hybrid + Jev, although their difference is not clear: 0.028 [−0.005, 0.062].The no-evidence control scores 0.808 on books, and context packing reduces Jev full-scan coverage from 92.0% in retrieved passages to 90.4% in supplied context.Source: §3.3 · pp. 7, 9, Figure 4 · p. 8, Table 5 · p. 8
  • On QASPER, full-scan Answer Accuracy differences from dense retrieval are −0.011 for Jev and 0.021 for Clef, with intervals spanning zero.On 40 unanswerable QASPER questions, Jev, Decisions, and Clef configurations abstain in only 11–14 cases, versus 17 for OpenAI dense and all 40 for the reference-context control.Source: §3.3 · p. 9

4 Limitations

The findings are limited to retrieval within a known document under specified passage and context limits, and the book evaluation covers eight textbooks. Timing and cost comparisons also reflect particular deployments and exclude some operating costs.

4 Limitations

  • The evaluation covers eight textbooks and largely model-generated references and automated answer scores, while most QASPER articles fit within the hybrid pool.Coverage includes failed retrievals, but answer scores use only completed evaluations.Source: §4 · p. 9
  • Latency comparisons reflect different hardware, precision, concurrency, and hosted-service conditions, and API charges exclude local computing such as Clef’s GPU use.Hosted model behavior can vary between calls, so the reported charges do not represent total operating cost.Source: §4 · p. 9

5 Conclusion

Across 11 retrieval configurations, the book evidence-coverage leader differed from the answer-accuracy leader. Full scanning reached every passage initially but could lose evidence during reduction, so evidence retention, answer quality, and repeated-use cost are distinct considerations.

5 Conclusion

  • 94.1% was the highest observed book evidence coverage, achieved by Hybrid + Jev; Hybrid + Decisions had the highest observed retriever Answer Accuracy at 0.892.Both Clef variants reached 98.8% evidence coverage on QASPER, and greater evidence recovery did not consistently improve answer quality.Source: §5 · p. 9
  • Full scanning initially reaches every passage but can lose evidence during reduction, while Hybrid + Jev had the highest observed book coverage and answer scores.For 1,000 queries per document, dense retrieval had much lower projected spending than paid decision APIs, including its initial embedding cost.Source: §5 · p. 9
  • The paper frames evidence retention, answer quality, and cost over repeated use as complementary criteria for choosing a retrieval method.Full scanning took substantially longer on books in the tested deployment.Source: §5 · p. 9

A Corpus and reproducibility

The corpus combines QASPER research articles and complete textbooks, with fixed test subsets and common retrieval constraints. Reproducibility checks document the book sample, evaluation controls, and implementation details.

A Corpus and reproducibility

  • The eight test books contain 4.46 million tokens and 10,508 passages, with ten questions per book split across factual, explanatory, and multi-section types.Development uses two separate books, and the test subsets are fixed across methods.Source: A Corpus and reproducibility · p. 12, Table 6 · p. 12
  • The evaluation used fixed test subsets, 11 retrieval configurations, and two answer controls across the textbook sample.Table 6 distinguishes timing repetitions from answer evaluation, and all 13 answer arms share the answer questions.Source: A Corpus and reproducibility · p. 12, Table 6 · p. 12
  • QASPER contributes 396 questions from 264 articles; evidence coverage uses 320 answerable questions from 230 articles, while costs and execution counts include all 396.The exact oracle reaches 100% reference coverage within five passages for all 320 answerable QASPER questions and all 80 book questions; fitting the passages into answer context remains separate.Source: A Corpus and reproducibility · p. 12
  • The implementation records model checkpoint prefixes and specifies Clef’s attention and convolution settings, while full environments and execution records are documented in the repository.Clef used SDPA attention and the PyTorch CUDA convolution fallback, without quantization or CPU parameter offloading.Source: A Corpus and reproducibility · p. 12
  • Metric checks met expected ranges for all Faithfulness and Answer Accuracy groups, while a separate Factual Correctness diagnostic failed one condition and was not used for test scoring.A separate review of 100 book questions and 148 supporting spans found no confirmed reference error, and no questions or spans were changed.Source: A Corpus and reproducibility · p. 12

B Retrieval diagnostics and resource use

The diagnostics trace failures and evidence loss through retrieval stages, while the resource analysis separates preparation from per-query costs. The projected crossover depends on repeated queries and confirmed API charges.

B Retrieval diagnostics and resource use

  • Across 5,236 primary retrieval assignments, 5,222 completed and 14 failed; failed retrievals remain in evidence evaluation with zero coverage.Failed or invalid timing observations are excluded from latency statistics; Table 7 distinguishes primary failures from repeated-run timing exclusions.Source: B Retrieval diagnostics and resource use · p. 12, Table 7 · p. 13
  • 98.75% of book reference evidence was already in the shared hybrid pool of up to 50 candidates before reranking, but reduction losses differed by method.Jev and Decisions lost most evidence during intermediate reductions, while Clef lost most when selecting the final five; these traces include completed scans only.Source: B Retrieval diagnostics and resource use · p. 13, Table 8 · p. 13
  • Leaving out one book at a time gives Jev full scan a 5.8–8.9-point advantage over OpenAI dense, but its 0.2–4.3-point range over Hybrid + BGE contrasts with a full-sample interval that includes zero.The omission ranges describe book-subset variation and are distinct from paired bootstrap intervals.Source: B Retrieval diagnostics and resource use · p. 13
  • Preparation is a one-time document cost for shared local indexes, whereas OpenAI preparation covers document embedding calls.Local preparation is not multiplied by the number of methods reusing those indexes.Source: B Retrieval diagnostics and resource use · p. 13, Table 9 · p. 13
  • Dense retrieval becomes cheaper than the Hybrid + Jev lower bound by the sixth query per book under the projection.The computed crossover is 5.2063 queries, and unknown Jev charges could move it earlier; API charges omit local computing and some unknown consumption is a lower bound.Source: B Retrieval diagnostics and resource use · pp. 13–14, Table 10 · p. 14

C Answer comparisons and evaluation outcomes

The section reports paired book answer comparisons and accounts for answer-generation and evaluation outcomes, with scores limited to applicable completed evaluations.

C Answer comparisons and evaluation outcomes

  • Table 11 reports paired Answer Accuracy differences (first method minus second), with 95% intervals and jointly scored-question counts, averaged within eight books.The intervals are descriptive and unadjusted for multiple comparisons; paired effects cannot be reconstructed by subtracting conditional means.Source: Table 11 · p. 14, C Answer comparisons and evaluation… · p. 14
  • 0.069 [0.050, 0.090] and 0.028 [0.010, 0.050] were the paired book Faithfulness gains for Jev full scan over OpenAI dense and Hybrid + BGE, respectively.These comparisons cover answers with supplied evidence that did not abstain, not every assigned answer.Source: C Answer comparisons and evaluation… · p. 14
  • 2,586 of 2,600 assignments produced usable answers; ten missing answers followed retrieval failures and four followed generation errors.Table 12 reports completed-answer counts with abstentions in parentheses, while missing answers remain separate.Source: C Answer comparisons and evaluation… · p. 14, Table 12 · p. 15
  • 4,233 of 4,244 requested judge evaluations were scored; failed and inapplicable metrics received no imputed score.The 928 inapplicable answer–metric combinations and missing or failed outputs remain outside numerical scores.Source: C Answer comparisons and evaluation… · pp. 14–15, Table 13 · p. 15
Loading 2610.28957v1…