PaperlayerView exact revision on arXiv

Source-linked AI summary

Less LLM, More Documents: Searching for Improved RAG

Jingjie Ning, Yibo Kong, Yunfan Long, Jamie Callan

arXiv:2510.02657v3cs.IRcs.CL

TL;DR

Prior work has not established how retrieval-corpus scale and LLM size interact in modular RAG, despite the overhead of scaling generators. This paper studies that trade-off under controlled corpus and model variation. Across open-domain QA benchmarks, larger corpora often let smaller generators approach or exceed larger ones, primarily through greater answer-bearing evidence coverage.

  • Problem

    How corpus size and LLM size interact in modular RAG remains insufficiently examined, while larger generators typically carry greater inference and deployment overhead.

    Source: §1 · p. 1, §2 · p. 2

  • Method

    The study pairs multiple corpus scales and generator sizes in a controlled design, holding retrieval, prompting, and decoding settings fixed.

    Source: §3.3 · p. 3

  • Results

    Across open-domain QA settings, corpus scaling often partially offsets model downsizing, with smaller or mid-sized generators on larger corpora frequently approaching or exceeding larger models under the same evidence budget.

    Source: §6 · p. 12

  • Takeaways & Limitations

    Retrieval-side scaling is a design option when additional corpus is available and retrieval costs fit deployment constraints; mid-sized generators tend to convert added evidence coverage into end-task gains more effectively.

    Source: §6 · p. 12

    The evaluation is limited to open-domain QA with a large web corpus that may overlap with modern LLM pretraining data, so extrapolation to largely unseen domain or enterprise corpora warrants caution.

    Source: §4.1 · p. 4

Abstract

from arXiv · show

Retrieval-Augmented Generation (RAG) couples document retrieval with large language models (LLMs). While scaling generators often improves accuracy, it also increases inference and deployment overhead. We study an orthogonal axis: enlarging the retriever's corpus, and how it trades off with generator scale. Across multiple open-domain QA benchmarks, corpus scaling consistently strengthens RAG and can in many cases match the gains of moving to a larger model tier, though with diminishing returns at larger scales. Small- and mid-sized generators paired with larger corpora often rival much larger models with smaller corpora; mid-sized models tend to gain the most, while tiny and very large models benefit less. Our analysis suggests that these improvements arise primarily from increased coverage of answer-bearing passages, while utilization efficiency remains largely unchanged. Overall, our results characterize a corpus-generator trade-off in RAG and provide empirical guidance on how corpus scale and model capacity interact in this setting.

1 Introduction

RAG combines retrieved evidence with LLMs, but generator scaling brings higher inference and deployment overhead while the interaction between corpus and generator scale remains unclear. This work varies corpus size with a fixed retrieval pipeline and evidence budget, finding that larger corpora can partly offset smaller generators.

1 Introduction

  • Prior work often improves RAG by scaling generators, which typically raises inference and deployment overhead; how retriever-side scaling interacts with generator scale is less understood.RAG supplies LLMs with retrieved evidence for open-domain QA.Source: §1 · p. 1
  • A 1.7B model with a 4× larger corpus outperforms a 4B model, and a 4B model with a 2× larger corpus consistently outperforms an 8B model.The study evaluates this trade-off with Qwen3 models and ClueWeb22 across NQ, TriviaQA, and WebQ.Source: §1 · pp. 1–2
  • The study characterizes corpus–generator trade-offs, including when corpus scaling can match generator-scaling gains, which model sizes benefit most, and how evidence coverage and context utilization relate to gains.The analyses focus on answer-bearing evidence coverage and context utilization.Source: §1 · p. 2

2 Related Work

Prior work has improved language models through tuning and scaling, and retrieval-augmented systems through larger memories and corpora. The direct interaction between corpus growth and LLM size in modular RAG remains insufficiently examined.

2 Related Work

  • Scaling LLMs generally improves accuracy across tasks [3] [14], while very large models such as 540B-parameter PaLM incur computational costs that limit practicality.Instruction tuning [34] and prompt engineering are also used to improve alignment with queries.Source: §2 · p. 2
  • Integrated retrieval-augmented language models such as RETRO [2] and Atlas show benefits from larger inference-time datastores, but require retrieval integration through pretraining-time vector memories and co-training.This paper instead studies modular RAG with independently scalable corpora and a fixed evidence budget.Source: §2 · p. 2
  • Modular retrieval progressed from DPR to efficiency-oriented ANCE and Contriever, enabling broader corpora such as LoTTE and BEIR, but corpus growth itself remained insufficiently studied.Source: §2 · p. 2
  • Research on RAG has not jointly examined retrieval-corpus expansion and LLM size enough to explain their trade-off in overall system performance.Earlier analyses often isolate single variables and remain fragmented or descriptive.Source: §2 · p. 2

3 Methodology

The methodology varies corpus scale and generator size in a controlled design to examine when retrieval-corpus expansion can compensate for smaller models and how benefits differ by model capacity.

3 Methodology

  • The framework asks whether enlarging the corpus lets smaller generators match or surpass larger ones, and how corpus-expansion benefits vary with model capacity.Source: §3 · p. 3

3.1 Retriever: Corpus Scaling

  • Corpus scaling is simulated by randomly partitioning a fixed corpus into approximately equal, disjoint shards and activating more shards to increase corpus scale.The canonical corpus scale uses a prefix of the shards, with corpus size proportional to the number activated.Source: §3.1 · p. 3
  • For a query, the retriever searches the active corpus, retrieves documents, segments and reranks them, then passes the top-m chunks to the generator.Source: §3.1 · p. 3

3.2 Generator

  • The generator family consists of models indexed by parameter size from the same architecture family, each receiving a fixed template containing the query and retrieved chunks.Source: §3.2 · p. 3

3.3 Trade-off Formalization

  • The full-factorial design pairs every corpus scale with every generator while holding retrieval, prompting, and decoding settings fixed, so only corpus scale and model size vary.Evaluation scores Pm(n, x) are defined for F1 and EM.Source: §3.3 · p. 3
  • Corpus-as-compensation is measured by the smallest corpus scale at which a smaller generator matches the one-shard baseline of a larger generator.The study reports this threshold and efficiency curves across corpus scales and model sizes, using F1 and EM.Source: §3.3 · p. 3

4 Experiment

The experiments evaluate controlled corpus–generator comparisons on three open-domain QA benchmarks using EM and token-level F1. Retrieval and generator implementations are specified to support comparisons across corpus and model scales.

4 Experiment

  • The experiment section frames the work as controlled comparisons across corpus scales to evaluate the research questions.Source: §4 · p. 4

4.1 Benchmarks

  • The study evaluates NQ, TriviaQA, and WebQ, but cautions that ClueWeb22-A may overlap with modern LLM pretraining data, limiting extrapolation to domain-gap or enterprise RAG with largely unseen corpora.Source: §4.1 · p. 4
  • The benchmarks comprise 1,769 NQ queries, 1,000 TriviaQA questions sampled from the rc.web test split, and 2,032 WebQ queries.Source: §4.1 · p. 4

4.2 Evaluation Metrics

  • The experiments report Exact Match and token-level F1 using official evaluation scripts, with EM based on exact normalized-answer matches and F1 on token overlap.Source: §4.2 · p. 4

4.3 Retriever: Implementation Details

  • The evaluation uses a 30% ClueWeb22-A subset of about 264M English documents, divided into 12 balanced shards of about 22M documents, with fixed retrieval and reranking settings across scales.Retrieval uses MiniCPM-Embedding-Light and DiskANN, then passes the top eight chunks to the generator; a Qwen3 embedding encoder check showed consistent corpus-scaling trends.Source: §4.3 · p. 4

4.4 Generator: Implementation Details

  • The generator experiments use Qwen3 base models from 0.6B to 14B parameters, sharing a tokenizer and architecture family.The authors caution that model scales may differ in minor training-recipe details, so size comparisons are empirical trends within the Qwen3 series.Source: §4.4 · p. 5

5 Results & Analysis

Across NQ, TriviaQA, and WebQ, corpus growth can let smaller generators catch up to larger tiers, though tiny models need more shards than medium-to-large models. Retrieval gains are strongest early and primarily track increased answer-bearing evidence coverage, while context utilization remains stable.

5.1 The Effect of Corpus Scaling as Compensation

  • On NQ, n⋆(0.6B→1.7B)=6, versus n⋆(4B→8B)=2 and n⋆(8B→14B)=2.Table 1 marks the first scale where a smaller model catches the next model’s n=1 baseline.Source: Table 1 · p. 5, Compensation Effect · p. 6
  • On TriviaQA, tiny-model thresholds are n⋆(0.6B→1.7B)=10 and n⋆(1.7B→4B)=7; medium-to-large pairs typically need 2 shards, and WebQ at most 3.The NQ and TriviaQA thresholds for 4B→8B and 8B→14B are both 2.Source: Table 2 · p. 6, Compensation Effect · p. 6
  • A single-shard retrieval costs ∼7.168 MFLOPs/query, versus ∼1.736 TFLOPs/query for Qwen3-0.6B generation, excluding one-time pretraining and index-construction costs.The authors’ representative serving-time proxy therefore finds arithmetic compute dominated by generation in this setup.Source: Serving-Time Compute Proxy · pp. 6–7
  • Reversing shard order modestly lowers absolute QA performance, while the relative additional corpus needed for a smaller model to catch up remains largely stable.Subsequent results use forward shard order for consistency.Source: Figure 3 · p. 7, Corpus Quality vs. Quantity · p. 7
  • Larger corpora increase the chance that retrieved chunks contain gold answer strings; coverage often rises with scale and is substantially higher on TriviaQA than NQ or WebQ.The Gold Answer Coverage Rate measures whether at least one of the top-8 retrieved chunks contains a gold answer string; larger n can also bring direct answer terms into context.Source: Why Does Corpus Scaling Improve RAG? · p. 8

5.2 Differential Effects Across LLM Sizes

6 Conclusion

Controlled evaluations characterize a corpus–generator trade-off: larger corpora can partly offset downsizing, chiefly by increasing answer-bearing evidence coverage. Benefits vary by model size, and gains diminish at larger corpus scales.

6 Conclusion

  • Across datasets, larger corpora often let smaller generators approach or exceed larger models under the same evidence budget, sometimes closing one to two model-size tiers.In several settings, corpus scaling yields gains comparable to moving to a larger generator tier.Source: §6 · p. 12
  • Gains primarily track greater coverage of answer-bearing evidence while utilization remains roughly stable, and marginal improvements become indistinguishable from zero beyond roughly a 5× corpus increase.The same qualitative scaling trends appear with an auxiliary retriever built using a different encoder.Source: §6 · p. 12
  • Mid-sized generators tend to translate improved evidence coverage into larger end-task gains than very small models, which need steep corpus expansions, or very large models, which gain less at the margin.The analysis mainly uses the Qwen3 family, with extensions to other model and retriever families left for future work.Source: §6 · p. 12
  • When retrieval costs are acceptable and more corpus is available, retrieval-side scaling is a design option; gold-answer coverage and Utilization Ratio offer diagnostics for expected gains.For corpus-limited settings, the paper suggests coverage-first domain curation and retrieval-quality improvements rather than adding low-yield documents.Source: §6 · p. 12
  • The study’s evidence is centered on Qwen3 because the authors lacked another open-source LLM series with homogeneous, wide-ranging variants.They identify additional model and retriever families, domain-specific corpora, and siloed corpora as future extensions.Source: §6 · p. 12

Appendix

The appendix captions identify trade-off tables for TriviaQA and WebQuestions and context-benefited-rate figures for TriviaQA and WebQ.

Appendix

  • Table 3 is labeled as reporting n⋆ for TriviaQA generator-size transitions.The caption does not state table values or identify a winning configuration.Source: Table 3 · p. 13
  • Table 4 is labeled as reporting n⋆ for WebQuestions generator-size transitions.The caption does not state table values or identify a winning configuration.Source: Table 4 · p. 13
  • Figures 10 and 11 are labeled as Context-Benefited Rate plots for TriviaQA and WebQ, respectively.Their captions specify benchmark scope but not the plotted outcomes.Source: Figure 10 · p. 13, Figure 11 · p. 13

Related papers

Loading 2510.02657v3…