PaperlayerView exact revision on arXiv

Source-linked AI summary

Re-ranking and Late Interaction Drive Retrieval Quality: A Controlled Comparison of RAG Strategies for Scientific Question Answering

Bhagyesh Rathi, Eshan Chawla, William B. Andreopoulos

arXiv:2609.38473v1cs.IRcs.AIcs.LG

TL;DR

The paper addresses limited evidence about which retrieval strategies perform best on large-scale scientific corpora. It compares six RAG pipelines in a shared evaluation using synthetic scientific questions and direct retrieval metrics; ColBERT leads on answer quality and gold-paper retrieval, while reranking is strongest among single-vector pipelines.

  • Problem

    Evidence comparing retrieval strategies under controlled conditions on large-scale scientific corpora remains limited, despite the need to understand their trade-offs.

    Source: §2.6 · p. 2

  • Method

    The study benchmarks six retrieval pipelines on a shared corpus and evaluation setup, using synthetic problem- and method-focused questions and answer-judge and gold-paper retrieval metrics.

    Source: §3.1 · p. 3, §3.3 · p. 4, §3.4 · p. 4

  • Results

    ColBERT RAG leads on both query types, with overall judge score 3.94/5 and gold-paper Hit@3 of 92.8–94.8%; Rephrased & Reranked RAG is the strongest single-vector pipeline.

    Source: §5 · p. 7, §4.7 · p. 7

  • Takeaways & Limitations

    The comparison finds late-interaction retrieval strongest overall and Rephrased & Reranked strongest among single-vector pipelines, while Classic RAG remains a robust, cost-effective baseline.

    Source: §7 · p. 8

    A single LLM judge does not see retrieved papers, so faithfulness is assessed from answer text rather than checked against evidence.

    Source: §3.4 · p. 4, §6 · p. 8

Abstract

from arXiv · show

Retrieval-Augmented Generation (RAG) is now the standard way to ground Large Language Models (LLMs) in external knowledge, yet the design space of retrieval pipelines is large and the trade-offs between variants are not well understood, especially on domain-specific corpora at realistic scale. In this work, we present a controlled comparison of six retrieval strategies for scientific question answering: (i) classic top-k dense retrieval, (ii) LLM-based query rephrasing, (iii) query rephrasing followed by LLM-based reranking, (iv) multi-query fusion via Reciprocal Rank Fusion (RRF), (v) an agentic tool-call pipeline in which the generator decides for itself whether to retrieve, and (vi) late-interaction retrieval with ColBERTv2. All six pipelines share the same generator (Meta-Llama/Llama-3.1-8B-Instruct), prompt, and evaluation protocol; the five single-vector pipelines additionally share SPECTER2 embeddings and a Chroma vector store; and all six retrieve from the full corpus of 463,971 arXiv papers dated 2024-2025. To support reproducible, large-scale evaluation, we also release a synthetic question dataset of 19,484 problem-statement and methodology questions generated by Llama-3.1-8B-Instruct from a random sample of 10,000 papers across academic domains (query generation succeeded for 9,742 of them), and every strategy is evaluated on this same query set. We describe the architecture and implementation of each pipeline, release the code and the synthetic question dataset, and evaluate each strategy with an LLM-as-a-judge protocol along multiple quality dimensions, together with direct gold-paper retrieval metrics. The result is an open testbed for studying the cost and quality trade-offs of RAG design choices on a research-literature corpus, and a basis for future work on faithfulness, retrieval robustness, and agentic retrieval.

1 Introduction and Motivation

The paper addresses limited guidance on retrieval strategies for large-scale scientific RAG by comparing six pipelines under a shared evaluation framework. It also contributes a synthetic scientific-question dataset and evaluates retrieval and answer quality.

1 Introduction and Motivation

  • Practitioners lack clear guidance on retrieval strategies for realistic, large-scale scientific corpora, while prior studies often isolate components or use small benchmarks.Source: §1 · p. 1
  • The study compares six retrieval strategies under a unified framework, sharing a corpus, generator, and evaluation setup, with five single-vector pipelines also sharing embeddings and a vector database.The strategies are classic dense retrieval, query rephrasing, rephrasing with reranking, Reciprocal Rank Fusion, agentic retrieval, and ColBERTv2 late interaction.Source: §1 · p. 1
  • The paper contributes a unified evaluation framework for scientific literature and a controlled comparison of six retrieval pipelines.Source: §1 · p. 1
  • The authors release a synthetic scientific question-answering dataset generated from arXiv papers across multiple academic domains.Source: §1 · pp. 1–2
  • The evaluation combines LLM-as-a-judge assessment of generated answers with retrieval-oriented metrics.Source: §1 · p. 2

2 Related Work

Related work establishes RAG and scientific dense retrieval as foundations, then reviews query reformulation, reranking, rank fusion, agentic retrieval, and evaluation. The paper positions its study as a controlled comparison of these strategies on large-scale scientific corpora.

2.1 Retrieval-Augmented Generation

  • RAG combines language models with external retrieval; REALM [8], DPR, and SPECTER2 [25] illustrate retrieval-model and scientific-embedding approaches.Source: §2.1 · p. 2

2.2 Query Reformulation and Expansion

  • Query rewriting reformulates short or corpus-misaligned queries; HyDE [6], RQ-RAG [2], and DMQR-RAG [13] extend query reformulation or multi-query retrieval.Source: §2.2 · p. 2

2.3 Reranking Methods

  • Reranking re-evaluates retrieved candidates with more expensive models, with BERT cross-encoders [16] and sequence-to-sequence rankers [17] among prior approaches.Source: §2.3 · p. 2

2.4 Rank Fusion Techniques

  • Reciprocal Rank Fusion combines ranked outputs from multiple retrieval systems using reciprocal-rank scores.Source: §2.4 · p. 2

2.5 Agentic and Tool-Augmented Retrieval

  • Agentic retrieval lets models decide when and how to retrieve, with ReAct [19], Toolformer [18], Self-RAG [1], and MA-RAG [15] representing related approaches.Source: §2.5 · p. 2

2.6 Evaluation of RAG Systems

  • The paper identifies limited controlled comparisons of retrieval pipelines on large scientific corpora and benchmarks dense, rewriting, reranking, fusion, agentic, and late-interaction approaches together.Source: §2.6 · pp. 2–3

3 Methods

The study builds a synthetic scientific-question benchmark from a random sample of recent arXiv papers. Each paper contributes problem- and method-focused queries, with the originating paper serving as the retrieval target.

3.1 Dataset, setup, synthetic query generation

  • The benchmark samples 10,000 papers from 463,971 arXiv records dated 2024–2025 across academic domains, using reservoir sampling with seed 42.The retained records are selected by last-updated date, and some gold papers have pre-2024 arXiv identifiers.Source: §3.1 · p. 3
  • Llama-3.1-8B-Instruct generates two retrieval-oriented queries per sampled paper from its abstract: one about the problem and one about the method.Source: §3.1 · p. 3
  • Generation succeeded for 9,742 papers, yielding 19,484 query–gold-paper pairs used to evaluate every strategy; 258 papers were dropped after generation or parsing failures or absent abstracts.Source: §3.1 · p. 3
  • Retrieval aims to find each query’s source paper and semantically related corpus documents, supporting evaluation of lexical and semantic retrieval.Source: §3.1 · p. 3

Embeddings and vector store

The methods use a persistent Chroma collection for single-vector retrieval and a separate PLAID index for ColBERT. Across strategies, the study standardizes answer generation and scoring, then reports both conditional and unconditional quality measures alongside gold-paper retrieval metrics.

Embeddings and vector store

  • Documents combine title, abstract, and category labels in a persistent Chroma collection, with SPECTER2 encoding and cosine-distance search retrieving three documents by default.Submitter, categories, and date are retained as metadata.Source: Embeddings and vector store · p. 3

3.2 Models and serving

  • The generator is Llama-3.1-8B-Instruct and the judge is Qwen2.5-32B-Instruct; experiments run on one NVIDIA DGX Spark.The judge differs from the generator’s model family.Source: §3.2 · pp. 3–4

3.3 Retrieval strategies

  • All six strategies pass three retrieved documents to the generator, while ColBERT uses token-level late interaction and the other five retrieve through Chroma.The first five share a grounding prompt template; Tool Call RAG uses a similar formulation within a tool-calling turn.Source: §3.3 · p. 4

3.4 LLM-as-a-judge evaluation

  • The judge first determines whether an answer attempts the question, assigning zero to every metric for non-answers, then scores six dimensions from 1–5 for attempted answers.The judge sees the question and answer but not retrieved papers, and the metric names are not formally defined in its prompt.Source: §3.4 · p. 4

3.5 Aggregation

  • Results separate problem and method queries, report answer rates and conditional metric means, and add unconditional means that retain refusals as zeros.The paired query-level comparisons also include Hit@1, Hit@3, and MRR@3 for the known gold source paper.Source: §3.5 · pp. 4–5

4 Results

Across the six strategies, ColBERT leads on overall judge score, gold-paper retrieval, and answer rate; rephrasing with reranking is the strongest single-vector approach. Fusion and Tool Call trail Classic, while query rephrasing alone has mixed retrieval results.

4.1 Overall comparison

  • ColBERT achieves the highest overall judge score on both query types (3.94/5), ahead of Rephrased & Reranked (3.77) and Classic (3.60).The ranking is unchanged when refusals remain as zeros: ColBERT scores 3.94 and Rephrased & Reranked scores 3.75.Source: §4.1 · p. 5

4.2 Answer rate

  • ColBERT answers 99.9% of queries for both types, while Tool Call has the lowest answer rates: 98.3% for problem and 96.9% for method queries.Tool Call skipped retrieval on 57 queries; in 30 cases it answered from parametric knowledge without retrieved evidence, which the judge accepted.Source: §4.2 · p. 5, Table 2 · p. 5

4.3 Per-metric scores

  • ColBERT leads on accuracy, completeness, faithfulness, and relevance for both query types, while completeness is the lowest-scoring metric for every strategy (2.71–3.17).Rephrased & Reranked is the strongest Chroma-based strategy on accuracy, faithfulness, relevance, and overall; Tool Call edges it on completeness and clarity.Source: §4.3 · p. 5

4.4 Retrieval effectiveness

  • ColBERT retrieves the gold paper in the top three for 92.8% of problem queries and 94.8% of method queries, roughly double Rephrased & Reranked’s 47.5% and 52.8%.For problem queries, Query Rephrased has lower Hit@3 than Classic (39.7% vs. 46.3%); gold-hit queries score 0.36–1.18 points higher than misses for Chroma-based strategies.Source: §4.4 · pp. 5–6, Table 5 · p. 6

4.6 Per-strategy figures

  • The strategy charts show per-metric averages, answer rate and overall quality by query type, gold-paper retrieval effectiveness, and paired per-query effects versus Classic.The paired-effect figure includes 95% confidence intervals, with positive values favoring the named strategy.Source: §4.6 · p. 6, Figure 1 · p. 6, Figure 2 · p. 6

4.5 Paired comparison against the Classic baseline

  • ColBERT (+0.27 problem, +0.47 method; pooled +0.37, 95% CI [+0.36, +0.38]) and Rephrased & Reranked (+0.13, +0.23; pooled +0.18, 95% CI [+0.17, +0.19]) significantly outperform Classic on both query types.Fusion and Tool Call are significantly worse than Classic on both query types; Query Rephrased is worse on problem queries but slightly better on method queries.Source: §4.5 · p. 6, Table 6 · p. 7

4.7 Summary of findings

  • The findings identify ColBERT as the top pipeline (3.94/5 overall; Hit@3 above 92% for both query types) and Rephrased & Reranked as the strongest single-vector pipeline (3.77/5; pooled paired delta +0.18 vs. Classic).Fusion and Tool Call fall below Classic; method questions are harder for every strategy, with problem-minus-method gaps from 0.04 for ColBERT to 0.22 for Classic.Source: §4.7 · pp. 6–7

5 Discussion

ColBERT achieved the strongest retrieval and answer-quality results, while re-ranking led among single-vector strategies. Across strategies, answer quality was similar when the gold paper was retrieved, linking most between-strategy differences to retrieval frequency.

5 Discussion

  • Among single-vector strategies, Rephrased & Reranked RAG achieved the highest overall score (3.77/5); multi-query fusion and the agentic tool-call pipeline fell below the classic single-shot baseline (3.60/5).The authors suggest simpler architectures may offer cost-efficiency without significantly compromising performance.Source: §5 · p. 7
  • ColBERT retrieved the gold paper in the top 3 for 92.8–94.8% of queries and scored 3.94/5 overall (+0.37 vs. Classic).It used the same full 463,971-paper corpus as the Chroma-based strategies.Source: §5 · p. 7
  • When the gold paper was retrieved, all strategies produced similar overall scores (≈3.8–4.0), so most between-strategy differences arose from retrieval frequency.The authors attribute nearly all of those differences to how often the right evidence reaches the generator.Source: §5 · p. 7

6 Limitations

The study’s findings are bounded by its judge, synthetic-query construction, scoring choices, model setup, and corpus date filter.

6 Limitations

  • A single LLM judge did not see retrieved papers, so faithfulness was judged from answer text and scores remain an imperfect proxy for human judgment.The judge was Qwen2.5-32B, from a different model family and larger than the generator.Source: §6 · p. 8
  • Gold relevance came from synthetic queries generated from paper abstracts, which may simplify research relevance.Source: §6 · p. 8
  • Headline metric aggregation excluded zero-valued scores, requiring joint interpretation with answer rates and unconditional means.The authors also report unconditional means and paired per-query effects to mitigate this issue.Source: §6 · p. 8
  • ColBERT differs in both encoder and late-interaction scoring, so its advantage cannot be attributed to late interaction alone.Source: §6 · p. 8
  • All pipelines used one 4-bit-quantized 8B generator at temperature 0, limiting generalizability to larger or more creative generation settings.Source: §6 · p. 8
  • The same model generated the synthetic queries and answered them, potentially biasing retrieval difficulty relative to independently authored queries.Source: §6 · p. 8
  • The corpus filter used metadata last-updated dates; about a fifth of gold papers were first submitted before 2024.The corpus should therefore be read as papers updated in 2024–2025, not strictly published then.Source: §6 · p. 8

7 Conclusion

The paper presents a unified evaluation of six retrieval pipelines for scientific question answering. ColBERT achieved the highest gold-paper retrieval rates and answer quality, while re-ranking led among single-vector pipelines and Classic remained robust and cost-effective.

7 Conclusion

  • ColBERT achieved the highest gold-paper retrieval rates and answer quality among the six evaluated pipelines.Source: §7 · p. 8
  • Among single-vector pipelines, Rephrased & Reranked RAG achieved the highest answer quality.Source: §7 · p. 8
  • The Classic baseline provided robust, cost-effective performance.Source: §7 · p. 8

Related papers

Loading 2609.38473v1…