PaperlayerView exact revision on arXiv

Source-linked AI summary

How Does Generative Retrieval Scale to Millions of Passages?

Ronak Pradeep, Kai Hui, Jai Gupta, Adam D. Lelkes, Honglei Zhuang, Jimmy Lin, Donald Metzler, Vinh Q. Tran

arXiv:2305.11841v1cs.IRcs.CL

TL;DR

Generative-retrieval techniques had largely been evaluated only on corpora around 100k documents, leaving their behavior at millions of passages unclear. This study scales evaluations to the full 8.8M-passage MS MARCO task and finds synthetic-query-to-identifier training strongest, while larger parameter counts alone do not ensure better retrieval.

  • Problem

    Prior generative-retrieval techniques were evaluated on corpora around 100k documents, leaving open how well they work on larger corpora and which components remain useful at scale.

    Source: §1 · p. 1

  • Method

    The paper empirically evaluates generative-retrieval techniques across corpus scales, including the full 8.8M-passage MS MARCO task and models up to 11B parameters.

    Source: §1 · p. 2, §9 · p. 13

  • Results

    Synthetic-query-to-Naive-ID T5-XL achieves 26.7 MRR@10 on 8.8M passages, while the same setting scaled to T5-XXL (11B) performs worse.

    Source: §9 · p. 13

  • Takeaways & Limitations

    Synthetic queries as document representations were the only approach that remained effective as corpus size grew, while compute-cost comparisons favored naive methods over more sophisticated ones on full MS MARCO.

    Source: §9 · p. 13

    The study does not exhaustively evaluate techniques or model-scaling scenarios, and it does not saturate Atomic ID scaling because of extreme parameter requirements.

    Source: §7 · p. 12

Abstract

from arXiv · show

Popularized by the Differentiable Search Index, the emerging paradigm of generative retrieval re-frames the classic information retrieval problem into a sequence-to-sequence modeling task, forgoing external indices and encoding an entire document corpus within a single Transformer. Although many different approaches have been proposed to improve the effectiveness of generative retrieval, they have only been evaluated on document corpora on the order of 100k in size. We conduct the first empirical study of generative retrieval techniques across various corpus scales, ultimately scaling up to the entire MS MARCO passage ranking task with a corpus of 8.8M passages and evaluating model sizes up to 11B parameters. We uncover several findings about scaling generative retrieval to millions of passages; notably, the central importance of using synthetic queries as document representations during indexing, the ineffectiveness of existing proposed architecture modifications when accounting for compute cost, and the limits of naively scaling model parameters with respect to retrieval performance. While we find that generative retrieval is competitive with state-of-the-art dual encoders on small corpora, scaling to millions of passages remains an important and unsolved challenge. We believe these findings will be valuable for the community to clarify the current state of generative retrieval, highlight the unique challenges, and inspire new research directions.

1 Introduction

Prior generative-retrieval studies focused on corpora around 100k documents, leaving open which techniques work at larger scales. This study evaluates those techniques up to 8.8M passages and finds that synthetic-query-to-identifier training is strongest, while scaling parameters alone does not solve large-corpus retrieval.

1 Introduction

  • Existing generative-retrieval techniques had been evaluated on corpora around 100k documents, leaving their effectiveness on larger corpora and the components vital at scale open.The study evaluates popular approaches first on Natural Questions and TriviaQA, then scales MS MARCO from 100k to 1M and 8.8M passages and models up to 11B parameters.Source: §1 · pp. 1–2
  • Synthetic queries mapped directly to document identifiers are the most effective strategy as corpus size grows, while other modeling strategies are largely unnecessary.The paper identifies synthetic-query generation as the most critical component of generative retrieval at scale.Source: §1 · p. 2
  • Architecture modifications that implicitly increase parameters improve results under the same T5 initialization, but those gains vanish when the naive approach is scaled to similar parameter sizes.The paper emphasizes that parameter count alone does not capture compute cost or the trade-offs in comparing models.Source: §1 · p. 2
  • For the best sequential IDs, effectiveness peaks at T5-XL (3B) and is slightly worse at T5-XXL (11B) under fixed experimental settings.The authors describe this as counter to the common conception that generative retrieval is limited by model capacity.Source: §1 · p. 2
  • 26.7 MRR@10 is the result for synthetic-query-to-Naive-ID T5-XL on 8.8M passages; on 100k passages, T5-Base reaches 82.4 MRR@10 versus GTR-Base at 83.2 MRR@10.The study reports that this is the strongest effectiveness among the techniques considered on the full MS MARCO passage ranking task.Source: §1 · p. 2

2 Related Work

Prior work spans sparse and dense first-stage retrieval, synthetic-query generation, reranking, and generative retrieval. Yet generative retrieval had not been trained or evaluated on corpora larger than roughly 100k documents.

Retrieval and reranking

  • Before generative retrieval, BM25 and dual encoders supported first-stage retrieval, while Transformer-based sequence-to-sequence models such as monoT5 reranked retrieved documents [Nogueira et al., 2020].The cited work also developed generation-based reranking for effectiveness and efficiency [Pradeep et al., 2021b; Zhuang et al., 2022a; Hui et al., 2022].Source: §2 · pp. 2–3

Query generation and model designs

  • Synthetic-query generation was used to improve retrieval across corpora and task setups [Nogueira et al., 2019c; Nogueira et al., 2019a; Ma et al., 2022].Generative-retrieval studies also explored identifier designs and model changes, including NCI.Source: §2 · p. 3

Generative retrieval

  • Generative retrieval replaces the retrieval pipeline with a sequence-to-sequence model that maps queries to document identifiers; DSI encodes corpus information in model parameters.DSI was shown to outperform a dual-encoder baseline on Natural Questions.Source: §2 · p. 3

Open scale question

  • Earlier generative-retrieval work had not trained or evaluated systems on corpora larger than O(100k) documents.The authors position their scaling study as relevant to applications beyond traditional information retrieval, including recommender systems and vision [Rajput et al., 2023; Zhang et al., 2023].Source: §2 · p. 3

3 Methods

The paper studies generative retrieval as a sequence-to-sequence system that encodes a corpus through document-to-identifier mappings and retrieves identifiers from queries. It varies document representations, identifier formats, and decoding or training components.

3.1 Background

  • DSI trains a sequence-to-sequence model to generate document identifiers from document content or relevant queries, then generates ranked identifiers at inference.The model parameters encode the mapping between document content and identifiers.Source: §3.1 · p. 3

3.2 Inputs and Targets

  • Document representations are indexing-task inputs and document identifiers are targets; long documents are costly to encode, while naive identifiers may lack semantic meaning.The study varies both design choices.Source: §3.2 · p. 4

3.2.1 Document Representations

  • FirstP uses a document’s first 64 tokens, while DaQ uses ten randomly selected 64-token chunks; for short MS MARCO passages, the two are essentially the same with sufficient context.Source: §3.2.1 · p. 4

3.2.2 Synthetic Query Generation

  • Synthetic queries address differences between indexing and retrieval: queries are short and specific, while documents are longer and only positive examples have retrieval queries.The proposed approach generates queries for documents and trains the retrieval model to predict their identifiers from those queries [Zhuang et al., 2022b; Wang et al., 2022].Source: §3.2.2 · p. 4

3.2.3 Document Identifiers

  • The study considers Atomic IDs as single tokens, Naive IDs as textual strings, and Semantic IDs as hierarchical clusters, alongside 2D Semantic IDs.Atomic IDs add a vocabulary token per document and therefore increase parameter count by corpus size times embedding dimension; 2D Semantic IDs encode cluster position.Source: §3.2.3 · pp. 4–5

3.3 Model Variants

  • PAWA uses position-specific output projections for 2D Semantic IDs, while constrained decoding uses a trie to restrict generation to valid identifiers.Consistency loss compares output distributions from two dropout-masked forward passes, but the authors excluded it from the final setup after observing training instability and NaN losses.Source: §3.3 · pp. 5–6

4 Experimental Setting

The experiments compare generative-retrieval configurations across Natural Questions, TriviaQA, and progressively larger MS MARCO passage corpora. They evaluate retrieval effectiveness under a shared set of metrics and describe model variants and training settings.

4 Experimental Setting

  • Experiments cover English retrieval tasks and assess generative-retrieval behavior across varying corpus scales.Source: §4 · p. 6

4.1 Corpus and Training Data

  • MS MARCO experiments use 100k-, 1M-, and 8.8M-passage variants; in the full corpus, around 6% of passages have a corresponding training query.Natural Questions and TriviaQA have corpora of 109K and 74K documents, respectively; 58% of TriviaQA documents have a relevant training question.Source: §4.1 · p. 6

4.2 Synthetic Query Generation

  • MS MARCO variants use docT5query to generate 40 questions per passage, while Natural Questions and TriviaQA reuse 20 and 15 generated questions per document, respectively.The study also trains an in-domain question-generation model for Natural Questions and TriviaQA.Source: §4.2 · pp. 6–7

4.3 Evaluation Dataset and Metrics

  • Evaluation uses Recall@1 for Natural Questions, Recall@5 for TriviaQA, and MRR@10 for MS MARCO passage-ranking variants.MS MARCO uses 6,980 validation examples; Natural Questions and TriviaQA use development sets of 7,830 and 7,993 questions.Source: §4.3 · p. 7

4.4 Model Variants

  • Model variants test labeled versus synthetic queries, FirstP versus DaQ representations, identifier versions, and stacked architecture additions; scaling experiments compare larger Naive and Semantic ID models with T5-Base plus Atomic IDs.The comparison targets similar parameter counts, since T5-Base with Atomic IDs totals 7B parameters for 8.8M document identifiers.Source: §4.4 · p. 7

4.5 Implementation Details

  • Models are initialized with the pre-trained T5-Base model; MS MARCO variants use a maximum input length of 128, and sequential-ID inference uses beam search with 40 beams.MSMarco1M and MSMarcoFULL models train to convergence or at most 9M steps; T5-XXL uses 128 TPUv4 chips.Source: §4.5 · p. 7

5 Experimental Results

The experiments ablate retrieval and document-representation choices on small corpora, then test which approaches persist as MS MARCO grows to 8.8M passages and models scale to 11B parameters. Synthetic queries are central to performance at larger corpus sizes, while proposed architecture modifications and further naive parameter scaling have limits.

5.1 Ablations over Small Corpora

  • The strongest small-corpus technique combination sets a new state of the art on NCI’s NQ variant without architecture changes or learned docids.Table 2 reports small-scale Natural Questions and TriviaQA results in Recall@1 and Recall@5, respectively.Source: §5.1 · p. 8, Table 2 · p. 8
  • Document representation dominates small-corpus performance: D2Q is essential for strong retrieval and yields a 7pt+ gain over other proposed techniques.Dataset training queries alone perform worst due to low document coverage; FirstP improves on them, and DaQ improves on FirstP. At this scale, the best result uses a mixture of FirstP, DaQ, labeled queries, and synthetic queries, with question-answering-domain synthetic queries outperforming an MS MARCO-trained generator.Source: §5.1 · p. 8
  • At small scale, Naive and Semantic IDs perform about on par across configurations, while Atomic IDs are consistently best; proposed modeling techniques do not reliably improve on synthetic queries alone.On NQ100k, Atomic IDs add 80M parameters to a 220M-parameter T5-Base model, a 36% increase.Source: §5.1 · p. 8

5.2 Scaling Corpus Size

  • On MS MARCO, synthetic queries improve over the original DSI formulation by 2–3x, and synthetic-query-to-docid alone is the most effective, straightforward strategy.This contrasts with NQ and TriviaQA, where FirstP and DaQ provided substantial value; the authors suggest the difference may relate to Wikipedia documents' informative openings.Source: §5.2 · pp. 8–9
  • 80.3, 55.8, and 24.2 are the best D2Q-only Atomic ID results as MS MARCO grows from 100k to 1M to 8.8M passages.Vanilla Semantic IDs underperform Naive IDs at full scale; constrained decoding provides only marginal value and is generally not worth its added complexity.Source: §5.2 · p. 9, Table 3 · p. 9

5.3 Scaling Model Size

  • 26.7 MRR@10 is the best MSMarcoFULL result, from 2.8B-parameter Naive IDs, versus 24.2 MRR@10 for T5-Base Atomic IDs with 7B parameters.Scaling Naive IDs to 11B lowers performance to 24.3 MRR@10 from T5-XL's 26.7; performance generally improves with parameter count, but PAWA and 2D Semantic IDs underperform naive scaling by 4 points at around 700M parameters.Source: §5.3 · pp. 9–10, Table 4 · p. 9

6 Discussion

The discussion identifies synthetic-query representation as central to effectiveness on larger corpora, while emphasizing that model scaling involves quality and compute trade-offs. The authors do not yet identify a best approach for competitive large-scale retrieval.

6.1 Why are synthetic queries effective?

  • Synthetic queries are the most critical component as corpus size grows; mapping them directly to document identifiers is the most effective strategy considered.The discussion attributes their effectiveness partly to closing document-coverage and query-distribution gaps, with query diversity also affecting performance.Source: §6.1 · p. 10
  • 92.9% query coverage at 100k falls to 51.6% at 1M and 5.8% at FULL; adding synthetic queries improves MSMarco100k by 3.3x and MSMarco1M by 3.9x.For NQ100k, replacing labeled queries with synthetic queries hurts, while for TriviaQA it helps; the reported coverage gains are 1.6% and 42.3%, respectively.Source: §6.1 · p. 11
  • Higher Jaccard similarity between synthetic and evaluation queries correlates with higher MRR@10, and more query exposure typically improves effectiveness across similarity buckets.Even at the lowest similarity, retrieval remains strong, so query distribution is not the sole determinant of performance.Source: §6.1 · p. 11
  • 82.4 MRR@10 with all 100 synthetic queries exceeds 80.3 with 40 and closes the gap with GTR-Base at 83.2 MRR@10 on MSMarco100k.RankT5-based selection hurt performance, while using all 100 samples performed best.Source: §6.1 · p. 11

6.2 Which model scaling approach is best?

  • 14 days gives T5-XL Naive IDs their optimal result, while after 7 days they match T5-Base Atomic IDs at 24.5 MRR@10.Both systems were trained on the same hardware and hyperparameter settings.Source: §6.2 · p. 12
  • Atomic IDs compute an 8.8M-document output projection once, while sequential IDs require O(d·k) decoding steps to rank k documents.The experiments used k = 40; scaling Naive IDs to compete also makes individual decoding steps more expensive.Source: §6.2 · p. 12
  • The authors cannot yet identify a best approach because generative retrieval has not achieved competitive MS MARCO passage-ranking results; at 100k, Atomic IDs are the highest-quality efficient option.With a fixed parameter budget, Naive IDs perform best on MSMarcoFULL and best in quality overall.Source: §6.2 · p. 12

7 Limitations

The study covers selected generative-retrieval techniques and scaling configurations rather than an exhaustive set. It focuses on effectiveness for large corpora, leaving other scales and practical questions outside its scope.

  • The technique and scaling evaluations are non-exhaustive, Atomic IDs are not fully saturated, and smaller-corpus model scaling is outside the study's scope.The authors also leave investigation of the maximum corpus size for state-of-the-art performance to future work.Source: §7 · p. 12

8 Future Directions

The authors note that longstanding problems remain, including achieving state-of-the-art results on large corpora and updating models with new documents. They identify further questions about model scaling and compute trade-offs.

  • Achieving state-of-the-art results on large corpora and updating models with new documents remain open problems.Source: §8 · p. 13
  • The authors ask how large language models and parameter scaling can benefit generative retrieval on large corpora, since the MS MARCO experiments do not show the expected benefit.They also ask whether default T5 parameterizations are optimal for memorization-heavy retrieval tasks.Source: §8 · p. 13
  • The authors call for scaling recipes and laws that maximize retrieval performance, and architectures that interpolate between Atomic-ID and sequential-ID compute trade-offs.Source: §8 · p. 13

9 Conclusion

The study evaluates generative retrieval on the full 8.8M-passage MS MARCO task. Synthetic queries remain effective as corpus size grows, but compute-aware comparisons and parameter scaling expose unresolved challenges.

  • The paper provides the first empirical study of generative retrieval on the full MS MARCO passage-ranking task, covering 8.8M passages.Source: §9 · p. 13
  • 26.7 MRR@10 is the strongest MS MARCO result, from synthetic queries to Naive IDs with T5-XL (3B); the same setting performs worse at XXL (11B).Source: §9 · p. 13
  • Synthetic queries are the only document representation strategy that remained effective and necessary as the MS MARCO passage corpus scaled.Source: §9 · p. 13
  • With parameter count fixed, naive methods outperform more sophisticated ones on full MS MARCO when compute cost is considered.Source: §9 · p. 13
Loading 2305.11841v1…