Generative Dense Retrieval: Memory Can Be a Burden

1 Introduction

• NCI performs well early but poorly late when decoding identifiers, which the authors attribute to weak memory for fine-grained document features.The comparison is with AR2 and is presented as evidence of GR’s fine-grained memory problem.Source: §1 · p. 1, Table 1 · p. 2
• Scaling training and candidate corpora from 334K to 1M reduced NCI by 11.0 R@100 versus AR2’s 2.8; NCI’s 1M-to-334K setting lost 5.7 R@100.Table 2 compares NCI and AR2 on the NQ validation set across training- and candidate-corpus settings.Source: §1 · p. 2, Table 2 · p. 2
• GDR applies memorization to query-to-cluster matching, then dense representations to fine-grained matching from clusters to documents.A shared query encoder supports both stages; cluster construction and cluster-adaptive negative sampling further support the approach.Source: §1 · p. 2
• Adding new documents to GDR requires cluster insertion and dense-representation extraction, without rebuilding document identifiers or retraining the model.By contrast, the described GR update process requires updating the cluster tree and retraining to rememorize documents.Source: §1 · p. 2
• GDR reports higher recall scores than advanced SR, DR, and GR methods, alongside significantly improved scalability.The proposed coarse-to-fine process combines memorizing and matching mechanisms.Source: §1 · p. 2

2.1 Dense Retrieval

• Dense retrieval uses query and document encoders to rank documents by representation similarity, avoiding corpus memorization but offering limited query-document interaction.GDR combines DR’s fine-grained matching with memorizing-based deep interaction at the coarse stage.Source: §2.1 · p. 3

2.2 Generative Retrieval

• Generative retrieval generates document identifiers, with prior methods varying identifier construction or training; these approaches require memorizing the whole corpus.Examples include titles (Cao et al., 2021), hierarchical identifiers (Tay et al., 2022), n-grams (Bevilacqua et al., 2022), and query-based methods (Wang et al., 2022; Mehta et al., 2022; Sun et al., 2023).Source: §2.2 · p. 3

3.1 Order of Applying Two Mechanisms

• GDR autoregressively maps queries to cluster identifiers, then uses document representations to match and rank documents within recalled clusters.The inter-cluster score captures coarse-grained matching; intra-cluster similarity and the inter-cluster score contribute to the final ranking.Source: §3.1 · pp. 3–4

3.2 Construction of Memory-friendly CIDs

• 0.636 vs 0.516 Opre for their CIDs versus the previous study; adaptive c keeps cluster counts in the same order of magnitude across corpus sizes.The strategy aims to give documents relevant to the same query similar CID prefixes while setting cluster count according to model memory rather than corpus size.Source: §3.2 · pp. 4–5

3.3 Cluster-adaptive Negative Sampling

• Cluster-adaptive sampling treats nonrelevant documents in the positive document’s cluster as intra-cluster negatives and in-batch negatives as inter-cluster negatives.The strategy is intended to supply more intra-cluster discriminative training signals.Source: §3.3 · p. 5

3.4 Training and Inference

• Training combines original and augmented queries while freezing precomputed document representations; inference beam-searches cluster identifiers, matches documents within clusters, then reorders them by overall score.Each document receives five DocT5Query pseudo-queries and five sampled groups of 40 consecutive terms; the paper says fewer augmented queries save training expenses.Source: §3.4 · p. 5

4.1 Experimental settings

• The NQ experiments use 334K, 1M, 2M, and 4M candidate-corpus settings, with R@k and Acc@k evaluated at k=20 and 100.The settings are formed from NQ’s relevant passages and progressively augmented with passages from the 21M corpus.Source: §4.1 · p. 6

4.2 Main Results

• GDR’s R@100 descent rate averages 3.50% as corpora scale, close to SR (3.29%) and DR (3.19%), while GR exceeds 15.25%.SR and DR descent rates remain below 4.06% across the three scaling directions.Source: Figure 2 · p. 6, §4.2 · p. 7
• GDR ranks first on R@k with an average 3.0 improvement and second on Acc@k across the reported comparisons.The reported rankings are GDR > BM25 > AR2 > NCI on R@k, and AR2 > GDR > BM25 > NCI on Acc@k.Source: §4.2 · p. 6, Table 3 · p. 7

4.3 Ablation studies on Model training

• On NQ334K, using the finetuned encoder for cluster identifiers improves both NCI and GDR over BERT-generated identifiers; gains are larger for NCI (10.1 vs 8.4 R@100).The authors relate the larger NCI gain to its heavier memory burden.Source: §4.3 · p. 7
• For GDR, cluster-adaptive negative sampling outperforms the widely used BM25 strategy by 1.1 on R@100, providing more intra-cluster discriminative training signals.Source: §4.3 · p. 7

4.4 Analysis

• After adding Set B, GDR’s R@100 drops 1.9 on Set A and reaches 66.2 on Set B, versus NCI’s 18.3 drop and 27.7 on Set B.GDR can encode added documents using dense representations and nearest-cluster search, without having memorized those documents.Source: Table 6 · p. 8, §4.4 · p. 8
• Combining Sinter and Sintra performs best at β=1; relying only on Sintra lowers R@100 by 1.7, while using only Sinter gives identical within-cluster rankings.The authors conclude that both coarse- and fine-grained matching signals matter.Source: §4.4 · pp. 7–8
• GDR’s retrieval efficiency falls between DR and GR: it generates clusters autoregressively, then uses pre-indexed retrieval within clusters.The authors leave efficiency improvements for GR and GDR to future work.Source: Table 7 · p. 8, §4.4 · p. 8

• GDR divides retrieval into inter-cluster matching by autoregressively generating cluster identifiers and intra-cluster matching by dense-representation similarity.It also proposes cluster-identifier construction and cluster-adaptive negative sampling strategies.Source: §5 · p. 8
• The coarse-to-fine design focuses limited memory on query–cluster interaction while using fine-grained document features for intra-cluster mapping.The authors say this retains the strengths of memorizing while introducing memorizing-free matching.Source: §5 · p. 8
• On NQ, GDR achieves state-of-the-art R@k performance and better scalability.Source: §5 · p. 8

• GDR inference speed needs further improvement before use in real-time retrieval services.Source: Limitations · p. 9
• GDR’s Acc@k performance falls short of AR2, which the authors suppose may reflect query-encoder capacity being used for inter-cluster matching.Source: Limitations · p. 9
• Generalization to larger-scale corpora has not been tested because NQ4M training cost 70 hours on 8 NVIDIA A100 GPUs.Source: Limitations · p. 9

• All datasets used in the study were publicly available.Source: Ethics Statement · p. 9
• No annotators were employed for data collection.Source: Ethics Statement · p. 9
• The authors state that the datasets contained no harmful content and were used consistently with their intended research purpose.They also report citing the datasets and relevant works.Source: Ethics Statement · p. 9

• For each identifier position, the method checks whether the predicted token keeps the sequence within a prefix of a relevant document identifier.Source: §A.1 · p. 11
• The error rate is conditioned on the prefix through position i-1 belonging to a relevant identifier’s prefix.Source: §A.1 · p. 11
• A prediction counts as an error when adding the token at position i leaves no prefix of any relevant document identifier.Source: §A.1 · p. 11

A.2 Magnitude of CIDs

• The proposed strategy controls total cluster counts across datasets of different scales, helping guarantee GDR’s memorizing volume.The passage describes counting CIDs for datasets with different scales of candidate documents.Source: §A.2 · p. 11
• Table 8 reports approximately 30,000 clusters, above the expected count of 5,000 because the constructed cluster tree is unbalanced.Source: §A.2 · p. 11
• Preliminary studies found that setting Exp(|CID|) to 5,000 in Algorithm 1 offers a favorable efficiency–performance budget.Source: §A.2 · p. 11

A.3 Experiments on TriviaQA

• Experiments on TriviaQA549K verified GDR’s good generalization.The results compare GDR with other methods; Table 9 reports experimental results on TriviaQA549K.Source: §A.3 · p. 11, Table 9 · p. 11
• TriviaQA549K was constructed following the procedure used to construct NQ334K.Source: §A.3 · p. 11