From Matching to Generation: A Survey on Generative Information Retrieval

• Traditional sparse and dense retrieval improve accuracy and efficiency but depend on large-scale indexes, lack end-to-end optimization, and require users to summarize ranked documents.Sparse methods include Boolean Retrieval, BM25, SPLADE [65], and Uni-COIL [163], while dense methods include DPR [117] and ANCE.Source: §1 · p. 2
• GenIR has two primary forms: Generative Document Retrieval generates valid ranked DocIDs from model parameters, while Reliable Response Generation directly produces reliable, user-centric answers.These approaches represent distinct information-delivery forms and have not yet been technically integrated.Source: §1 · pp. 2–3, Figure 1 · p. 2

Generative Document Retrieval

• Generative document retrieval assigns document identifiers and learns query-to-DocID mappings, using constrained beam search to replace large-scale indexes with end-to-end parametric retrieval.Research covers model training and structure [6] [153], identifier design [18] [265] [288], continual learning [80] [124] [192], and related adaptations.Source: §1 · p. 3

Reliable Response Generation

• Reliable response generation offers customized, detailed answers but can be irrelevant [85], factually contradictory [90] [104], outdated [291], or toxic [93] [263].Research addresses internal knowledge [37] [56] [119] [132] [243] [285], external knowledge [5] [113] [151] [204] [245], citations [129] [142] [156] [204] [314], and personal assistance [327].Source: §1 · p. 3

• The survey is presented as the first comprehensive review to categorize GenIR techniques, assess metrics and benchmarks, and discuss challenges, solutions, and future research directions.Its coverage includes generative retrieval, reliable response generation, evaluation, challenges, and prospects.Source: §1 · pp. 3, 5, Figure 2 · p. 4

2.1 Traditional Information Retrieval

• Traditional retrieval progresses from inverted-index sparse methods such as TF-IDF and BM25 to dense Transformer encoders that capture underlying semantics.Sparse methods efficiently match terms but struggle with synonyms, specialized terminology, and context; dense methods represent queries and documents as vectors and rank them by similarity.Source: §2.1 · pp. 5–6

2.2 Generative Retrieval

• Generative retrieval uses pretrained models such as T5 and BART to generate relevant document identifiers directly, avoiding large-scale document indices.Each document receives a unique DocID, prefix constraints ensure valid identifiers, and constrained beam search ranks candidates by their generation probabilities.Source: §2.2 · pp. 6–7
• Generative retrieval models learn query–DocID associations with cross-entropy loss and produce relevance-ordered documents without traditional indexing structures.The section presents this as the simplest generative retrieval method and previews later advances in architectures, training strategies, and DocID design.Source: §2.2 · p. 7

2.3 Large Language Models

• Large language models include encoder-decoder architectures such as T5 and BART and decoder-only architectures such as GPT, enabling direct generation of information users seek [55] [173] [374].This generative capability supports complex and personalized information needs, including cases where users cannot easily formulate precise queries.Source: §2.3 · pp. 7–8

2.4 Augmented Language Models

• Because LLMs hallucinate on complex, long-tail, and real-time tasks [90] [359], retrieval and tool augmentation ground responses in external knowledge and extend functionality.Citation-generating methods further improve verifiability and trustworthiness by exposing source documents behind generated content [143] [204] [256].Source: §2.4 · p. 8

3 GENERATIVE DOCUMENT RETRIEVAL: FROM SIMILARITY MATCHING TO GENERATING DOCUMENT IDENTIFIERS

• GR methods increasingly address model training, architectures, document identifiers, incremental learning, task adaptation, and generative recommendation.The field evolved from GENRE [18] and model-based retrieval proposals to methods including DSI, DynamicRetriever, SEAL [13], and NCI.Source: §3 · p. 8

3.1.1 Model Training.

• GR trains query-to-DocID mappings, augmenting seq2seq training with sampled document pieces, generated pseudo queries, ranking objectives, and distillation or reinforcement-learning signals.DSI separates indexing and retrieval training; DynamicRetriever samples passages, terms, and N-grams; NCI and DSI-QG [375] generate pseudo queries; GenRRL, DGR [161], and ListGR optimize ranking.Source: §3.1.1 · pp. 9–10

3.1.2 Model Structure.

• GR model structures combine semantic decoding, external memory, hierarchical clustering, dense retrieval, multi-stage generation, and self-retrieval to improve retrieval precision, recall, scalability, or memory efficiency.PAWA, NP-Decoding, and PAG modify decoding; MEVI [346] and GDR [342] combine generative and dense retrieval; TOME, DiffusionRet, and Self-Retrieval [275] use multiple generative stages or unified models.Source: §3.1.2 · pp. 10–11

3.2 Design of Document Identifiers

• Document identifiers range from static or learnable numeric codes to interpretable text identifiers, with semantic clustering, quantization, reconstruction, contrastive learning, and relevance-aware optimization shaping their design.Numeric identifiers use atomic, hierarchical k-means, PQ, or RQ [189] codes, while learnable methods include GenRet [265], Tied-Atomic [206], LMIndexer [112], ASI, and RIPOR [344]. Text identifiers include titles, URLs, substrings, term sets, and learnable N-grams.Source: §3.2.1 · pp. 11–12, §3.2 · p. 11

3.2.2 Text-based Identifiers.

• Text-based identifiers exploit language-model capabilities and interpretability through titles, URLs, substrings, term sets, and learnable lexical representations, but title identifiers require uniqueness within the corpus.GENRE [18] uses Wikipedia titles and achieves superior retrieval across 11 KILT datasets, whereas duplicated or meaningless web titles motivate combining URLs and titles in Ultron. SEAL [13], MINDER, AutoTSG, NOVO, and GLEN [135] provide more flexible alternatives.Source: §3.2.2 · pp. 12–13

3.3 Incremental Learning on Dynamic Corpora

• Incremental GR adapts to continuously changing corpora through SAM-based optimization and rehearsal, constrained representation learning, incremental product quantization, and task-specific adapters that limit forgetting.DSI++ [192] rehearses pseudo queries; IncDSI [124] protects both new-document retrieval and existing performance; CLEVER [25] updates only selected centroids; CorpusBrain++ [80] trains adapters while fixing a shared backbone.Source: §3.3 · pp. 13–14

3.4.1 Separate Training.

• Downstream adaptation applies GR to fact verification, KILT, multi-hop retrieval, code search, conversational QA, and other tasks through separate training, task-specific identifiers, remote supervision, and reinforcement learning.GERE [27] generates titles for claim-related documents; CorpusBrain [28] and UGR [26] support multiple retrieval granularities; DearDR [283], Re3val [257], GMR [131], CodeDSI [203], and GCoQA [158] target additional task settings.Source: §3.4 · p. 14, §3.4.1 · p. 14

3.4.2 Joint Training.

• Jointly trained GR systems unify retrieval and generation with shared encoders, task-specific decoders, multi-task objectives, ranking-guided decoding, and auxiliary DocID-understanding tasks.UniGen jointly supports retrieval and question answering, while CorpusLM integrates GR, closed-book generation, and retrieval-augmented generation.Source: §3.4.2 · p. 15

3.4.4 Generative Recommender Systems.

• LC-Rec [360] supplements semantic identifiers with sequential prediction, explicit index-language alignment, and recommendation-oriented implicit alignment to integrate semantic and collaborative information.The learned identifiers help large language models adapt to sequence recommendation tasks.Source: §3.4.4 · p. 16
• IDGenRec [273] generates concise, semantically rich, platform-agnostic item identifiers from metadata and alternately trains the identifier generator and recommendation model.Its diversified ID generation algorithm targets improved sequential recommendation performance and accuracy.Source: §3.4.4 · p. 16
• SEATER [254] builds equal-length identifiers with a balanced k-ary tree and recursively clustered item-text vectors, preserving semantic consistency across index levels.A Transformer encoder-decoder model uses contrastive and multi-task learning to enhance level-wise index semantics.Source: §3.4.4 · p. 16
• ColaRec [309] derives generative item identifiers from a pretrained recommender, represents users through aggregated item content, and aligns semantic and collaborative spaces.It combines item-indexing generation loss with contrastive loss in an end-to-end recommendation framework.Source: §3.4.4 · p. 16

• Large language models can generate direct, personalized responses that reduce users’ effort in collecting and integrating information.Source: §4 · p. 16
• Faithful GenIR systems must address hallucination, prolonged inference time, and high operational costs.Source: §4 · p. 16
• The section develops four strategies: internal model optimization, external knowledge enhancement, increased accountability, and personalized information assistants.Source: §4 · p. 16

4.1 Internal Knowledge Memorization

• Internal knowledge memorization combines structural enhancements, training strategies, and inference techniques to improve user-centric, reliable responses.The surveyed strategies include scaling and integrating models, optimizing data and training, engineering prompts, and refining decoding.Source: §4.1 · p. 16, §4.1.1 · p. 17

4.1.1 Model Structure.

• Scaling model parameters and training data improves retained knowledge, downstream performance, and instruction following, while MoE increases effective parameters without proportionally increasing inference costs [116] [34] [227] [58] [61] [106] [136].LLM-Blender combines outputs from multiple language models but incurs high deployment costs.Source: §4.1.1 · p. 17

4.1.2 Training and Inference.

• Training reliability improves through cleaner data and factuality-oriented objectives, while prompting and decoding methods guide reasoning, verification, diversity, and faithfulness [79] [132] [264] [56] [211] [313] [306] [332] [49] [268] [339] [133] [134] [292].Examples include textbook synthesis, forum-dialogue fine-tuning, deduplication, contrastive learning, factuality-aware preference optimization, self-verification, and faithfulness-aware decoding.Source: §4.1.2 · pp. 17–18

4.1.3 Knowledge Updating.

• Knowledge updating addresses evolving information through incremental learning that preserves prior knowledge and knowledge editing that corrects or adds facts [301, 303, 321, 351, 191].Incremental pre-training and fine-tuning use continual corpora, prompts, regularization, replay, feedback, or orthogonal subspaces, whereas editing uses added parameters, locate-then-edit methods, or meta-learning [267, 100, 39, 119, 323, 236, 201, 305, 94, 83, 193, 194, 154, 182, 17, 199, 270].Source: §4.1.3 · pp. 18–20

4.2 External Knowledge Augmentation

• External knowledge augmentation addresses hallucination, in-domain knowledge gaps, and knowledge-updating challenges through retrieval augmentation and tool augmentation.Retrieval grounds generation in external documents, whereas tools provide timely information and broaden supported tasks such as translation and image generation.Source: §4.2 · p. 20, §4.2.1 · p. 20, §4.2.2 · p. 22

4.2.1 Retrieval Augmentation

• RAG methods combine a retriever and generator and comprise sequential, branching, conditional, and loop frameworks for increasingly adaptive query processing.Sequential pipelines retrieve before generation and may rewrite queries or compress documents; branching processes multiple pipelines, conditional methods decide whether retrieval is needed, and loop methods iteratively coordinate retrieval and generation.Source: §4.2.1 · pp. 21–22

4.2.2 Tool Augmentation

• Tool augmentation overcomes retrieval’s dependence on existing corpora by acquiring timely data and expanding responses through search engines, knowledge graphs, APIs, and model-based tools.These tools support time-sensitive search, structured reasoning, real-world data access, autonomous API invocation, multimodal tasks, and delegation to specialized models.Source: §4.2.2 · pp. 22–23

4.3 Generating Response with Citation

• Citation-based response generation improves GenIR reliability by making each piece of knowledge traceable to its source, thereby enhancing user trust and adoption [88].Source: §4.3 · p. 23

4.3.1 Direct Generating Response with Citation

• Direct generation uses intrinsic model memory or generative retrieval to produce cited responses without conventional document searching, with prompting, iterative critique, planning, and DocID generation improving attribution.According-to prompting [314] guides citation from pre-training data; IFL [129] iteratively improves citation accuracy, correctness, and fluency; and 1-PAGER [97] generates N-gram DocIDs with constrained decoding, matching retrieval-then-read accuracy and surpassing closed-book QA in evidence attribution.Source: §4.3.1 · p. 24

4.3.2 Retrieval-based Response with Citation

• Retrieval-based citation generation retrieves relevant documents and either incorporates citations during response generation or verifies and updates evidence iteratively with retrieval, NLI, planning, memory, and reinforcement learning.WebGPT [204], LaMDA [282], and WebBrain [223] train on cited web or Wikipedia data; SearChain [326], LLatrieval [156], AGREE [335], VTG [261], and HGOT [60] add reasoning, verification, adaptation, memory, or hierarchical planning mechanisms.Source: §4.3.2 · pp. 24–25
• Citation after generation first produces a response, then retrieves supporting evidence and edits or annotates claims, while localized attribution methods reduce manual fact-checking and preserve or improve the original output.RARR [70], PURR [24], CEG [150], and Attribute First, then Generate [256] respectively use post-editing, unsupervised denoising, NLI-supported citations, or source-first generation.Source: §4.3.2 · p. 25

4.4 Personal Information Assistant

• Personalized information assistants aim to understand users’ personalities and preferences to generate responses that better meet their information needs.The section covers personalized dialogue and domain-specific personalization, with its overview presented in Figure 6.Source: §4.4 · p. 25

4.4.1 Personalized Dialogue System.

• Personalized prompt methods use interaction histories, ratings, profiles, long-term interests, or prompt rewriting to generate more tailored responses and recommendations.Examples include ChatGPT in-context learning, LaMP profile retrieval, BookGPT’s querying and verification, PerSE’s review-based preference inference, and prompt rewriting for frozen LLMs.Source: §4.4.1 · pp. 25–26
• Personalized fine-tuning learns user profiles and preferences through large persona or conversation datasets, specialized architectures, memory integration, reward modeling, and preference-conditioned model combinations.Methods include Persona-Chat with 5 million personas, over 700 million Reddit conversations, implicit profile learning, generative segmentation memory, bidirectional memory–knowledge selection, three-stage reward learning, and Personalized Soups.Source: §4.4.1 · p. 26
• Retrieval-enhanced, parameter-efficient, and multimodal approaches further personalize assistants by enriching profiles, aligning responses with character settings, storing behavioral patterns, or converting behavior into natural-language preferences.LAPDOG, SAFARI, OPPU, and PMG illustrate retrieval, planning, personalized PEFT, and multimodal preference extraction.Source: §4.4.1 · p. 26

4.4.2 Domain-specific Personalization.

• Domain-specific personalization supports healthcare applications including physiological-data processing, diagnosis assistance, traditional Chinese medicine, conversational health agents, multidisciplinary reasoning, mental-health monitoring, psychological consultation, and medication planning.Representative systems include Zhongjing, openCHA, MedAgents, Mental-LLM, Psy-LLM, and PharmacyGPT.Source: §4.4.2 · pp. 26–27
• Academic, educational, and other applications personalize reviews, writing, exercises, recipes, headlines, emails, tweets, and household-robot assistance from users’ documents, progress, histories, examples, or preferences.Examples include RevGAN, Pearl, adaptive exercise generation, EduChat, LaMP, and TidyBot.Source: §4.4.2 · p. 27

• The evaluation covers metrics, benchmarks, and performance analysis for generative information retrieval methods.Source: §5 · p. 27

5.1.1 Metrics.

• Evaluation uses Recall, R-Precision, MRR, MAP, and nDCG to measure coverage, cutoff precision, early relevant retrieval, aggregate accuracy, and relevance ordering.nDCG additionally accounts for both document relevance and positions in the ranked list.Source: §5.1.1 · pp. 27–28

5.1.2 Benchmarks.

• Benchmarks span MS MARCO, NQ, KILT, TREC Deep Learning, DynamicIR, and ExcluIR, covering web search, question answering, knowledge-intensive, dynamic-corpus, and exclusionary retrieval.DynamicIR finds GR more adaptable to evolving knowledge and more efficient in memory, indexing time, and FLOPs than dense retrieval.Source: §5.1.2 · p. 28

5.1.3 Analysis.

• Analyses find that DSI remembers pseudo-query-to-DocID mappings but poorly distinguishes relevant documents from random ones, limiting retrieval effectiveness.Generative and dense retrieval share relevance-computation patterns, including query-document dot products and term matching in alignment matrices.Source: §5.1.3 · pp. 28–29
• On 8.8M MS MARCO passages, synthetic pseudo-query augmentation is the only effective large-corpus technique; synthetic-query training with Naive DocIDs and T5-XL reaches MRR@10 26.7, whereas T5-XXL declines.OOD studies also show query-variation sensitivity across retrievers, while pretraining improves GR adaptability to new query types and tasks.Source: §5.1.3 · p. 29

5.1.4 Experiments.

• On MS MARCO and NQ, GR remains competitive: TSGen reaches Recall@1 of 0.384 and 0.708, surpassing DPR and SPLADEv2 on NQ.On MS MARCO, TSGen and DGR also exceed DPR, while TSGen is comparable to SPLADEv2.Source: §5.1.4 · p. 30, Table 3 · p. 30
• GR methods generally outperform traditional sparse and dense retrieval approaches in most tasks, with GENRE achieving the highest scores in FEVER (0.847), AY2 (0.928), WnWi (0.877), and WnCw (0.706).Source: §5.1.4 · p. 31
• Title- and substring-based DocIDs perform strongly on KILT, with GENRE reaching FEVER 0.847 and SEAL leading QA categories including NQ 0.658, HoPo 0.565, TQA 0.715, and WoW 0.527.Title identifiers benefit from document-entity alignment and pretrained knowledge, while substring identifiers capture meaningful document fragments for precise question answering.Source: §5.1.4 · pp. 31–32
• Numeric Semantic DocIDs deteriorate on KILT as corpus size grows: BART scores near zero, including FEVER 0.003 and AY2 0.001, because training pairs cover few documents.The models tend to memorize DocIDs without generalizing across the broader corpus, consistent with scaling difficulties reported for DSI on 8.8 million passages.Source: §5.1.4 · p. 32

5.2.1 Metrics.

• Generated-response evaluation addresses accuracy, fluency, relevance, and related qualities through rule-based, model-based, and human-evaluation metrics, including exact matching, n-gram overlap, and predictive ability.Exact Match requires complete word-level identity with the reference answer, while BLEU measures output–reference n-gram overlap. Perplexity reflects average predictive ability for a text corpus, with lower values indicating stronger predictive ability.Source: §5.2.1 · pp. 32–33
• Human evaluation assesses accuracy, relevance, fluency, and safety more reliably for complex real-world tasks, but is costly, time-consuming, subjective, and difficult to standardize.These limitations reduce evaluator consistency and result comparability, while some tasks require professional evaluators.Source: §5.2.1 · pp. 33–34

5.2.2 Benchmarks and Analysis.

• General benchmarks test broad knowledge and reasoning through diverse tasks, including MMLU [84], BIG-bench [259], LLM-Eval [166], and Chinese-focused C-Eval [92].C-Eval [92] covers 52 academic fields and includes the harder C-Eval Hard subset; LLM-Eval [166] also shows evaluation outcomes depend on the selected LLM and decoding strategy.Source: §5.2.2 · p. 34
• Benchmarks reveal capability gaps: ToolBench [226] covers over 16,000 APIs; TruthfulQA reports 58% model versus 94% human truthfulness; ALCE [71] finds incomplete citation support; RealTime QA [118] finds outdated answers when retrieval is insufficient.Source: §5.2.2 · pp. 34–35
• Safety and ethics benchmarks assess multilingual safety, toxicity, bias, value alignment, and broader trustworthiness, showing continuing risks and supporting more accurate, reliable, safe, and timely GenIR systems.SafetyBench [358] compares 25 LLMs across 11,435 questions, while TrustGPT [93] identifies ethical issues even in advanced models.Source: §5.2.2 · pp. 35–36
• TrustLLM found trustworthiness positively correlated with functional effectiveness; proprietary models usually outperformed open-source models, although Llama2 achieved comparable high performance.It evaluated 16 mainstream LLMs on 30 datasets across truthfulness, safety, fairness, robustness, privacy, and machine ethics.Source: §5.2.2 · pp. 35–36

6.1.1 Scalability Issues.

• GR trails dense retrieval on million-document web corpora but outperforms it below 300K documents, highlighting unresolved scalability limits.Challenges include insufficient <query, DocID> training data, rote memorization of unseen identifiers, uncertain model structures, and catastrophic forgetting during incremental updates.Source: §6.1.1 · p. 36, §6.1.2 · p. 37

6.1.3 Document Identifier.

• High-quality DocIDs are critical: Wikipedia titles support GR on 5.9 million documents, whereas inaccurate titles contribute to decline on 8.8 million MS MARCO passages.Future work includes constructing lightweight semantic identifiers, balancing textual interpretability against numeric efficiency, and reducing the latency of FM-Index-based substring methods.Source: §6.1.3 · p. 37

6.1.4 Efficiency Concerns.

• Constrained beam search can reach several hundred milliseconds when returning 100 or more documents, motivating concise identifiers of 16 tokens or fewer and faster decoding.Multimodal GR also needs better image representations, end-to-end training, and extension beyond text and images to audio and video.Source: §6.1.4 · pp. 37–38, §6.1.5 · p. 38

6.2.1 Improving Accuracy and Factuality.

• Reliable response generation requires stronger models and clearer memory mechanisms, while improving retrieval augmentation, autonomous retrieval timing, long-context dialogue use, and tool cost-effectiveness.Potential directions include alternative architectures, better training data, embedding-based augmentation, adaptive retrieval, and evaluating whether tool benefits justify training or inference costs.Source: §6.2.1 · p. 38

6.2.2 Real-time Properties of GenIR Systems.

• Because pretrained knowledge is fixed, GenIR needs real-time knowledge acquisition and continual updating that avoids forgetting existing knowledge.Continual learning and knowledge editing are proposed, but bias from large unfiltered datasets remains unresolved and requires mechanism-level understanding and mitigation.Source: §6.2.2 · p. 39, §6.2.3 · p. 39

6.2.3 Bias and Fairness; 6.2.4 Privacy and Security.

• GenIR must address plagiarism, unsafe disclosure of memorized private information, and persistent biases through citation-aware generation, fairness methods, privacy defenses, and generated-content detection.These risks arise from language models reproducing training data and propagating stereotypes or private information, while legal and technical safeguards continue to develop.Source: §6.2.4 · p. 39, §6.2.3 · p. 39

6.3.1 Unified Framework for Retrieval and Generation.

• A unified GenIR model could combine document-level retrieval with reliable response generation, but existing UniGen and CorpusLM systems generalize poorly beyond limited tasks or Wikipedia.The survey therefore proposes a large search model that autonomously generates DocIDs and responses, deciding when retrieval is needed.Source: §6.3 · p. 39, §6.3.1 · pp. 39–40

6.3.2 Towards End-to-End Framework for Various IR Tasks.

• End-to-end GenIR remains challenging because GR and downstream generation have not been seamlessly integrated; future systems should align knowledge with DocIDs and generate referenced answers.Such systems could connect document sources to answers while supporting additional retrieval tasks, multilingual retrieval, and multimodal retrieval.Source: §6.3.2 · p. 40

Generative Document Retrieval

• The survey systematically covers generative document retrieval, including training, document identifiers, incremental learning, downstream adaptability, multimodal retrieval, and generative recommendation systems.It also organizes evaluation methods and benchmarks for generative document retrieval.Source: §7 · p. 40

Reliable Response Generation

• For reliable response generation, the survey reviews internal knowledge memorization, external knowledge enhancement, citation-grounded responses, and personal information assistance.The survey also organizes evaluation methods and benchmarks for response generation.Source: §7 · p. 40

Challenges and Future Directions

• The survey identifies GenIR challenges involving scalability, dynamic corpora, document representation, efficiency, accuracy, real-time capabilities, bias, fairness, privacy, and security.It proposes potential solutions and future research directions for these challenges.Source: §7 · p. 40

Unified Framework

• The survey envisions a unified framework combining retrieval and generation tasks, potentially extending to an end-to-end system for diverse information retrieval tasks.This vision is intended to promote further development of GenIR.Source: §7 · p. 40

A.1 Evaluation Metrics for Generative Document Retrieval

• Recall@k measures the proportion of relevant documents retrieved within the top k results for each query.It uses the number retrieved within the cutoff relative to the query’s total relevant documents.Source: §A.1 · p. 41

A DETAILS FOR EVALUATION

• The evaluation framework spans complementary views of retrieval quality, from coverage and precision to first-hit ranking, all-relevant ranking, and position-discounted graded relevance.These views correspond to Recall@k, R-Precision, MRR, MAP, and nDCG, respectively.Source: §A.1 · p. 41

A.1 Evaluation Metrics for Generative Document Retrieval

• R-Precision measures precision at rank R, where R equals the number of relevant documents for the query.The metric counts relevant documents retrieved within the top R positions.Source: §A.1 · p. 41
• Mean Reciprocal Rank (MRR) averages the reciprocal rank of the first relevant document returned across queries.For each query, rank_q denotes the position of its first relevant result.Source: §A.1 · p. 41
• Mean Average Precision (MAP) averages precision across queries while considering the exact positions of all relevant documents.Its calculation uses P@k and an indicator identifying whether the document at position k is relevant.Source: §A.1 · p. 41
• Normalized Discounted Cumulative Gain (nDCG) evaluates both document relevance and positions in the returned result list.It uses graded relevance, discounted cumulative gain, and the ideal discounted cumulative gain at cutoff k.Source: §A.1 · p. 41

MS MARCO

• MS MARCO evaluates generative retrieval at document and passage granularity using approximately 3.2 million documents and 8.8 million passages from real Bing queries.Its diverse question types and document genres are intended to approximate complex web-search scenarios.Source: §A.2 · pp. 41–42

Natural Questions (NQ)

• Natural Questions evaluates retrieval for real-world information-seeking queries over approximately 3.2 million Wikipedia documents, emphasizing precise answer retrieval.Answers are extracted from web pages in Google search results.Source: §A.2 · p. 42

KILT

• KILT combines five knowledge-intensive task categories—fact checking, entity linking, slot filling, open-domain QA, and dialogue—over approximately 5.9 million Wikipedia pages.It evaluates retrieval and knowledge integration across complex language tasks and multiple domains.Source: §A.2 · p. 42

TREC Deep Learning Track 2019 & 2020

• The TREC Deep Learning Tracks 2019 and 2020 assess deep-learning retrieval through document-ranking and passage-ranking tasks.These campaigns focus on improving retrieval efficiency and effectiveness.Source: §A.2 · p. 42