PaperlayerView exact revision on arXiv

Source-linked AI summary

Scalable and Effective Generative Information Retrieval

Hansi Zeng, Chen Luo, Bowen Jin, Sheikh Muhammad Sarwar, Tianxin Wei, Hamed Zamani

arXiv:2311.09134v1cs.IR

TL;DR

Generative retrieval has shown empirical success mainly on small, often artificial collections, leaving its large-scale effectiveness in question. RIPOR combines prefix-oriented ranking optimization with relevance-based DocID construction using residual quantization. It outperforms generative retrieval baselines on the reported benchmarks and is comparable to dense retrieval models.

  • Problem

    Prior generative retrieval models had demonstrated success mainly on small, often artificially constructed collections, prompting skepticism about their real-world impact.

    Source: §1 · p. 1

  • Method

    RIPOR optimizes ranking at DocID prefixes and constructs initial DocIDs by residual-quantizing task-trained relevance-based document representations.

    Source: §1 · p. 2, §3.2 · p. 4

  • Results

    RIPOR outperforms all generative retrieval baselines; versus LTRGR, it improves MRR@10 by 30.5% on MSMARCO Dev and achieves comparable results to dense retrieval models.

    Source: §4.2.1 · p. 6

  • Takeaways & Limitations

    The experiments demonstrate the feasibility of effective generative retrieval at scale, a milestone toward real-world applications.

    Source: §1 · p. 2

    The HNSW index might slightly impact performance compared with dense retrieval models using a flat index.

    Source: §4.2.2 · p. 6

Abstract

from arXiv · show

Recent research has shown that transformer networks can be used as differentiable search indexes by representing each document as a sequences of document ID tokens. These generative retrieval models cast the retrieval problem to a document ID generation problem for each given query. Despite their elegant design, existing generative retrieval models only perform well on artificially-constructed and small-scale collections. This has led to serious skepticism in the research community on their real-world impact. This paper represents an important milestone in generative retrieval research by showing, for the first time, that generative retrieval models can be trained to perform effectively on large-scale standard retrieval benchmarks. For doing so, we propose RIPOR- an optimization framework for generative retrieval that can be adopted by any encoder-decoder architecture. RIPOR is designed based on two often-overlooked fundamental design considerations in generative retrieval. First, given the sequential decoding nature of document ID generation, assigning accurate relevance scores to documents based on the whole document ID sequence is not sufficient. To address this issue, RIPOR introduces a novel prefix-oriented ranking optimization algorithm. Second, initial document IDs should be constructed based on relevance associations between queries and documents, instead of the syntactic and semantic information in the documents. RIPOR addresses this issue using a relevance-based document ID construction approach that quantizes relevance-based representations learned for documents. Evaluation on MSMARCO and TREC Deep Learning Track reveals that RIPOR surpasses state-of-the-art generative retrieval models by a large margin (e.g., 30.5% MRR improvements on MS MARCO Dev Set), and perform better on par with popular dense retrieval models.

1 INTRODUCTION

The introduction argues that generative retrieval’s promise has not translated to large-scale effectiveness, in part because prior approaches overlook prefix-level ranking and relevance-aware DocID construction. RIPOR addresses both design issues and demonstrates strong results on large-scale benchmarks.

1 INTRODUCTION

  • Prior generative retrieval models had succeeded mainly on small, often artificially constructed collections, prompting skepticism about their real-world impact.On MSMARCO, BM25 achieved 300% higher MRR than DSI [41], a gap reduced to 76% after query-generation augmentation [53].Source: §1 · p. 1
  • RIPOR’s prefix-oriented ranking optimization trains every relevant DocID prefix to score above non-relevant prefixes, progressively refining predictions while retaining prefix emphasis.It uses a margin-based pairwise loss and multi-objective progressive learning.Source: §1 · p. 2
  • RIPOR constructs initial DocIDs from task-trained relevance-based document representations, decomposed into vectors with residual quantization, rather than relying on document syntax or semantics.The introduction identifies fixed, inappropriately constructed DocIDs as a bottleneck for later optimization.Source: §1 · p. 2
  • RIPOR beats the best generative retrieval model by 30.5% in MRR@10 on MSMARCO and outperforms popular dense retrieval models in most settings.The evaluated collection contains 8.8 million passages.Source: §1 · p. 2

2 INTRODUCTION TO GENERATIVE IR

Generative retrieval represents documents with DocIDs and generates those identifiers autoregressively from queries. This section explains how the model scores and decodes DocIDs to produce a ranked document list.

2 INTRODUCTION TO GENERATIVE IR

  • Each document has a unique DocID, represented as a sequence of tokens; the model generates DocID tokens autoregressively for a query.DSI constructs identifiers using root-to-leaf paths in a hierarchical k-means tree over document embeddings.Source: §2 · p. 2
  • Existing models score query–DocID pairs using log conditional probabilities, while this paper adopts conditional logits for lower computation cost and better alignment with its margin-based pairwise loss.The conditional-logit approach computes dot products between DocID token embeddings and query-conditioned hidden vectors.Source: §2 · pp. 2–3
  • Constrained beam search generates the top K valid DocIDs according to the scoring function, then maps them back to documents to form a ranked list..Source: §2 · p. 3

3 METHODOLOGY

RIPOR combines relevance-based document ID construction with ranking optimization at each decoding prefix, then trains the model through initialization, seq2seq pre-training, and rank-oriented fine-tuning.

3 METHODOLOGY

  • RIPOR views the generative model as a dense encoder, constructs document IDs with residual quantization, and then applies seq2seq pre-training and rank-oriented fine-tuning.Its three phases are DocID initialization, seq2seq pre-training, and rank-oriented fine-tuning.Source: §3 · p. 3, §3.2 · p. 4, §3.3 · p. 5

3.1 Prefix-Oriented Ranking Optimization

  • Full-sequence ranking alone may not ensure relevant documents survive beam search, so RIPOR optimizes pairwise rankings between positive and negative document ID prefixes at each decoding step.The loss reweights the golden margin by prefix position, emphasizing earlier decoding steps.Source: §3.1 · p. 3
  • Progressive training starts with the shortest prefix and extends to the full ID sequence, while multi-objective learning retains ranking knowledge from earlier prefixes.Prefix-oriented optimization is performed at lengths 4, 8, 16, and 32 for efficiency.Source: §3.1 · pp. 3–4

3.2 Relevance-Based DocID Construction

  • RIPOR initializes document IDs from relevance-based representations, aiming to give similar documents similar, hierarchically structured IDs that reflect query-document relevance.Residual quantization is selected for its recursive hierarchy and its shorter IDs relative to product quantization at strong performance, saving inference time and memory.Source: §3.2 · p. 4

3.3.1 DocID Initialization.

  • For DocID initialization, the model is trained as a dense encoder using MarginMSE, first with BM25-sampled documents and then with documents retrieved by the trained model.The BM25 stage samples the top 100 documents per query.Source: §3.3.1 · p. 5

3.3.2 Seq2seq Pre-training.

  • Seq2seq pre-training uses document-associated pseudo queries as input proxies for predicting the corresponding document IDs, with cross-entropy loss.The pseudo queries are generated for each document using doc2query.Source: §3.3.2 · p. 5

3.3.3 Rank-oriented Fine-tuning.

  • Rank-oriented fine-tuning first trains on retrieved negatives with full-sequence ranking, then adds beam-search negatives and progressive prefix optimization, followed by self-negative fine-tuning.The beam-search stage combines its negatives with those from nearest-neighbor search and applies the multi-objective prefix loss.Source: §3.3.3 · p. 5

4.1 Experiments Settings

The experiments use MSMARCO and TREC Deep Learning Track datasets, report dataset-specific retrieval metrics, and compare RIPOR with generative, sparse, and dense retrieval baselines.

4.1.1 Dataset.

  • Evaluation covers MSMARCO with 8.8 million passages and 532,000 training queries, plus TREC Deep Learning Track 2019 and 2020 passage retrieval sets.The evaluation sets contain 7,000 MSMARCO Dev queries and 43 and 54 TREC queries, respectively; annotations are incomplete.Source: §4.1.1 · p. 5

4.1.2 Implementation Details.

  • The experiments use T5-base, 32-token document IDs with a vocabulary of 256, and doc2query-generated pseudo queries for seq2seq pre-training.Experiments were run on eight 40GB A100 GPUs.Source: §4.1.2 · p. 5

4.1.3 Baselines.

  • Generative retrieval baselines include DSI, DSI-QG, NCI-QG, SEAL, MINDER, and LTRGR; other paradigms include BM25 and dense retrieval models.Dense baselines include DPR, ANCE, MarginMSE, and TAS-B; the table caption specifies HNSW indexing for dense retrieval models.Source: §4.1.3 · pp. 5–6, Table 1 · p. 6

4.2 Experiment Results

RIPOR outperforms generative retrieval baselines on the reported benchmarks, with gains over LTRGR and performance comparable to dense retrieval models. Ablations report contributions from prefix optimization, multi-objective learning, self-negative fine-tuning, and seq2seq pre-training, while also showing substantial drops for alternative document representations and quantization.

4.2.1 Main Results.

  • RIPOR outperforms all generative retrieval baselines; versus LTRGR, it improves MRR@10 by 30.5% on MSMARCO Dev and NDCG@10 by 94% on TREC-20 test.RIPOR also achieves improvements over ANCE on MSMARCO Dev and TREC’DL 19 and 20.Source: §4.2.1 · p. 6

4.2.2 Ablation Studies.

  • HNSW indexing might slightly affect performance relative to dense retrieval models using flat indexes.Source: §4.2.2 · p. 6
  • 19% degradation in MRR@10 follows removal of prefix-oriented ranking optimization, which protects relevant DocID prefixes from early beam-search pruning.Without the optimization, relevant prefixes are not explicitly ensured to score above irrelevant prefixes.Source: Table 2 · p. 6, §4.2.2 · p. 7
  • 5% further improvement in MRR@10 comes from multi-objective learning within prefix optimization, which mitigates forgetting in later progressive-training stages.The method adds only minimal loss computation and no training computational overhead.Source: Table 2 · p. 6, §4.2.2 · p. 7
  • Self-negative fine-tuning improves MRR@10 by 2.5% and Recall@10 by 3.5%.The hard negatives help rank relevant DocIDs above high-scoring hard negatives.Source: Table 2 · p. 6, §4.2.2 · p. 7
  • Seq2seq pre-training improves MRR@10 by 4%, with the model encoding document information across the corpus.The passage relates this to the indexing phase in dense retrieval models.Source: Table 2 · p. 6, §4.2.2 · p. 7
  • Using sentence-T5 hidden representations for documents instead of treating the generative model as a dense encoder produces a 73% drop in MRR@10.The authors attribute the decline to sentence-T5 not being optimized for query-dependent document similarity.Source: Table 2 · p. 6, §4.2.2 · p. 7
  • Replacing RQ with PQ decreases MRR@10 by 197%; the authors suggest PQ may not capture the hierarchical document structure important to beam search.The passage characterizes PQ as unsuitable for generative retrieval in these experiments.Source: Table 2 · p. 6, §4.2.2 · pp. 7–8

4.3 Analysis and Discussion

The analysis finds that document-ID length and vocabulary size relate to retrieval effectiveness, and that longer prefixes improve both document clustering and retrieval performance. Ablations indicate that prefix-oriented optimization and relevance-based initialization matter, while product quantization shows limited gains at longer prefixes.

4.3 Analysis and Discussion

  • At L=16, increasing vocabulary size from 512 to 1024 improves MRR@10 by 5.5%; longer DocIDs also improve MRR@10 and Recall@10 when extra parameters are held constant.The analysis varies DocID length L and vocabulary size V while examining their effects on model effectiveness.Source: §4.3.1 · p. 8

4.3.2 The quality of document approximated representation.

  • For prefix lengths L≥8, documents relevant to the same query cluster together; clustering quality improves as L increases.The authors attribute the improvement possibly to reduced distortion between approximated and original document representations at larger L.Source: §4.3.2 · p. 8

4.3.3 The influence of prefix-length.

  • Longer prefixes consistently improve retrieval, with larger gains at L≤8; at equal prefix length, larger vocabularies tend to perform better.These trends are reported for different RIPOR DocID combinations in Figure 4.Source: §4.3.3 · p. 8
  • Removing prefix-oriented optimization or relevance-based DocID initialization reduces performance; initialization’s removal leads to a larger drop, while PQ gains barely increase for L≥8.The authors suggest that product quantization may not capture hierarchical nuances among documents, limiting benefits from longer prefixes.Source: §4.3.3 · p. 8

5 RELATED WORK

The related-work section places generative retrieval alongside sparse, re-ranking, and dense neural retrieval approaches. It contrasts their representations and retrieval mechanisms, while noting that re-ranking models are limited to document re-ranking by efficiency constraints.

5 RELATED WORK

  • Neural sparse retrieval adapts BERT to re-weight subwords and uses L1 or FLOP regularizers to preserve sparsity for inverted-index search [13].The section frames this approach as inspired by TF-IDF and BM25.Source: §5 · pp. 8–9
  • Re-ranking models score concatenated query-document inputs, but efficiency constraints limit them to document re-ranking [30] [52].Source: §5 · p. 9
  • Dense retrieval uses bi-encoders to embed queries and documents, then approximate nearest-neighbor search for efficient retrieval [16] [17] [19] [20] [33] [46] [49] [50].Source: §5 · p. 9
  • Generative retrieval maps documents to DocIDs and uses constrained beam search to retrieve likely IDs; learned item DocIDs have also been used for sequential recommendation [3] [41] [35].The section notes that DocIDs may capture semantic relationships between items and serve as priors for downstream tasks.Source: §5 · p. 9

6 CONCLUSIONS AND FUTURE WORK

RIPOR targets generative retrieval on large-scale datasets with prefix-oriented ranking optimization and residual-quantization DocID construction. The authors report that its constructed IDs capture relevance-based document similarity and identify efficiency and other knowledge-intensive tasks as future directions.

6 CONCLUSIONS AND FUTURE WORK

  • RIPOR combines prefix-oriented ranking optimization with residual quantization for DocID construction, aiming to improve generative retrieval on large-scale datasets.Source: §6 · p. 9
  • The authors report that RIPOR’s DocID construction captures relevance-based document similarity, improving information-retrieval effectiveness.Source: §6 · p. 9
  • Future work will further optimize efficiency and integrate RIPOR into knowledge-intensive tasks such as open-domain QA and fact verification.Source: §6 · p. 9

A PERFORMANCE ON A SMALLER-SCALE SAMPLED DATASET

This appendix reports results on an artificially constructed 1M-passage MSMARCO subset and cautions against using it for further work. The subset has much higher training-query document coverage than the full 8.8M-passage collection.

A PERFORMANCE ON A SMALLER-SCALE SAMPLED DATASET

  • The authors do not recommend further work on MSMARCO-1M: 51.6% of its documents are covered by training queries, versus 5.8% in the real MSMARCO-8.8M collection.Source: A PERFORMANCE ON A SMALLER-SCALE… · p. 11
  • The 1M subset begins with 522K passages relevant to the 532K training queries and 7K Dev queries, then samples additional passages.Source: A PERFORMANCE ON A SMALLER-SCALE… · p. 11
  • All six RIPOR entries in Table 4 carry * marks for significant improvement over all generative retrieval baselines on the 1M subset.The table covers MSMARCO Dev and TREC Deep Learning Track results.Source: Table 4 · p. 12
Loading 2311.09134v1…