PaperlayerView exact revision on arXiv

Source-linked AI summary

From Matching to Generation: A Survey on Generative Information Retrieval

Xiaoxi Li, Jiajie Jin, Yujia Zhou, Yuyao Zhang, Peitian Zhang, Yutao Zhu, Zhicheng Dou

arXiv:2404.14851v4cs.IRcs.AIcs.CL

TL;DR

Traditional information retrieval relies on matching and indexed document rankings, while generative approaches aim to generate document identifiers or direct responses. This survey systematically synthesizes GenIR methods, evaluations, challenges, and future directions. It reports competitive generative-retrieval performance in several benchmarks while highlighting unresolved efficiency, reliability, and freshness issues.

  • Problem

    Traditional retrieval depends on large-scale indices and document rankings, while generated responses can be irrelevant, contradictory, outdated, or toxic.

  • Method

    The survey organizes GenIR into generative document retrieval and reliable response generation, reviewing methods, identifiers, knowledge enhancement, evaluation, challenges, and prospects.

  • Results

    GR methods are competitive with sparse and dense baselines on MS MARCO and NQ, while numeric DocID methods can perform near zero across KILT tasks.

  • Takeaways & Limitations

    GenIR is presented as a shift from matching toward generation through document-identifier retrieval and direct, user-centric responses.

  • Takeaways & Limitations

    Reliable response generation still requires better external-knowledge integration, retrieval timing, dialogue-history use, and updating without forgetting existing knowledge.

Abstract

from arXiv · show

Information Retrieval (IR) systems are crucial tools for users to access information, which have long been dominated by traditional methods relying on similarity matching. With the advancement of pre-trained language models, generative information retrieval (GenIR) emerges as a novel paradigm, attracting increasing attention. Based on the form of information provided to users, current research in GenIR can be categorized into two aspects: \textbf{(1) Generative Document Retrieval} (GR) leverages the generative model's parameters for memorizing documents, enabling retrieval by directly generating relevant document identifiers without explicit indexing. \textbf{(2) Reliable Response Generation} employs language models to directly generate information users seek, breaking the limitations of traditional IR in terms of document granularity and relevance matching while offering flexibility, efficiency, and creativity to meet practical needs. This paper aims to systematically review the latest research progress in GenIR. We will summarize the advancements in GR regarding model training and structure, document identifier, incremental learning, etc., as well as progress in reliable response generation in aspects of internal knowledge memorization, external knowledge augmentation, etc. We also review the evaluation, challenges and future developments in GenIR systems. This review aims to offer a comprehensive reference for researchers, encouraging further development in the GenIR field. Github Repository: https://github.com/RUC-NLPIR/GenIR-Survey

1 INTRODUCTION

GenIR shifts information retrieval from similarity-based document matching toward generation. The survey organizes GenIR around generative document retrieval and reliable response generation, and reviews their methods, evaluation, challenges, and prospects.

  • Traditional Information Retrieval: Traditional sparse and dense retrieval improve efficiency or semantic precision but depend on large-scale indices and return document rankings that users must summarize.Sparse methods use word-level matching, while dense methods encode queries and documents to capture semantics.
  • Generative Information Retrieval: Generative document retrieval uses language-model parameters to generate relevant document identifiers instead of relying on traditional indexed matching.DocID generation is constrained by a prefix tree to produce valid identifiers.
  • Reliable Response Generation: Reliable response generation directly produces user-centric answers, but language models may generate irrelevant, contradictory, outdated, or toxic content.These reliability problems motivate strategies for improving internal knowledge, external knowledge use, and response grounding.
  • Survey Scope: The survey systematically categorizes GenIR research into generative document retrieval and reliable response generation while covering evaluation, challenges, and future directions.Its analysis includes model training and structure, document identifiers, knowledge enhancement, benchmarks, and limitations.

2 BACKGROUND AND PRELIMINARIES

The background contrasts sparse and dense similarity retrieval with generative retrieval, which produces document identifiers directly. It also introduces language-model augmentation as a way to provide more current and grounded information.

  • Sparse Retrieval: Sparse retrieval uses inverted indexes and term-based relevance measures, but matching can struggle with synonyms, specialized terms, and context.TF-IDF and BM25 are representative sparse methods.
  • Dense Retrieval: Dense retrieval encodes queries and documents as vectors, then ranks documents by vector similarity.The document and query encoders may be shared or task-specific, and cosine similarity can measure relevance.
  • Generative Retrieval: Generative retrieval models directly generate query-relevant DocIDs, enabling end-to-end retrieval without large-scale pre-built document indices.Documents receive unique identifiers through a mapping from documents to DocIDs.
  • Generative Retrieval: Prefix-constrained decoding restricts each generated token to prefixes of valid DocIDs, while beam search ranks candidate identifiers by generation probability.The resulting DocIDs are mapped back to the documents considered most relevant to the query.
  • Augmented Language Models: Retrieval augmentation supplies external knowledge during generation to improve factual grounding and reduce hallucinations, especially for long-tail or real-time information.Tool augmentation is another strategy for extending language-model capabilities through specialized tools or APIs.

3 GENERATIVE DOCUMENT RETRIEVAL: FROM SIMILARITY MATCHING TO GENERATING DOCUMENT IDENTIFIERS

Generative document retrieval shifts IR from matching queries against indexed documents to generating relevant document identifiers with a model’s parametric memory. The survey organizes progress around training, model structure, identifiers, and related retrieval capabilities.

  • Research scope: Research on generative retrieval spans model training, architectures, document identifiers, incremental learning, task adaptation, multimodal retrieval, and generative recommendation.The survey presents these topics as major directions in the field’s development.
  • Model training: Standard training maps queries to relevant DocIDs using sequence-to-sequence objectives, while DSI separates document indexing from query-DocID retrieval fine-tuning.DSI’s indexing stage associates document tokens with pre-built DocIDs, and its retrieval stage uses labeled query-DocID pairs.
  • Model training: Pseudo queries generated from documents, sampled passages, terms, and n-grams augment training to improve memorization of document identifiers.DynamicRetriever samples document pieces, while NCI generates pseudo queries or directly uses documents as queries.
  • Model training: Generative retrieval models improve ranking through multi-task distillation, ranking losses, and reinforcement learning with relevance feedback.These methods target indexing and ranking quality without necessarily changing the model structure.
  • Model structure: Specialized architectures and decoding methods combine generative and dense retrieval, multiple models, or external representations to improve efficiency, recall, scalability, or retrieval precision.MEVI and GDR organize documents into clusters, while TOME and DiffusionRet use auxiliary generation stages.
  • Document identifiers: DocID designs include numeric and text-based identifiers, with semantic structure or unique titles helping models represent and retrieve documents.Titles work well when unique, but duplicated or meaningless web titles motivate combining URLs and titles.

4 RELIABLE RESPONSE GENERATION: DIRECT INFORMATION ACCESSING WITH GENERATIVE LANGUAGE MODELS

Reliable response generation uses language models to provide personalized answers directly aligned with users’ information needs. The survey frames this approach around reducing user effort while addressing hallucination, latency, and operational-cost challenges.

  • Motivation: Generative language models can provide personalized, user-centric answers directly aligned with users’ informational needs.Direct response generation reduces the time users spend collecting and integrating information.
  • Challenges: Reliable response generation remains challenged by hallucination, prolonged inference time, and high operational costs.The survey organizes mitigation strategies around internal optimization, external knowledge, accountability, and personalization.

4.1 Internal Knowledge Memorization

Internal knowledge memorization improves reliable response generation through model scaling and integration, training-data and training-method optimization, prompting, decoding, and knowledge updating. The survey also covers incremental learning and targeted knowledge editing for evolving information.

  • Scope: Internal model optimization covers structural enhancements, training strategies, and inference techniques for building reliable responses.The survey’s framework includes model scaling, model structure, training, inference, incremental learning, and knowledge editing.
  • Model Structure: Scaling model parameters and training data increases knowledge capacity, downstream-task performance, and instruction-following ability.The survey reports this pattern across GPT and LLaMA model families.
  • Model Structure: Mixture-of-Experts selectively activates parameter subsets, increasing effective parameters without proportionally increasing inference costs.LLM-Blender instead combines outputs from multiple language models but incurs high deployment costs.
  • Training: Training-data optimization addresses noise, misinformation, and incompleteness that can contribute to hallucinations and other output problems.The survey describes synthetic textbooks and curated dialogue data as examples of data-focused improvements.
  • Training: Training-method optimization uses techniques such as contrastive learning to identify correct knowledge and reduce incorrect-token generation.MixCL uses an external knowledge base, while CaliNet assesses erroneous knowledge learned by the model.
  • Inference: Prompt engineering and decoding strategies improve response reliability by guiding reasoning, sampling, factual-token selection, and hidden-layer interventions.The survey discusses CoT, nucleus-based methods, DoLa, and ITI as representative inference approaches.
  • Knowledge Updating: Knowledge updating addresses evolving information through incremental learning that retains prior knowledge and knowledge editing that modifies targeted model knowledge.Incremental learning includes continued pre-training and fine-tuning, while editing methods locate and update knowledge-related parameters.

4.2 External Knowledge Augmentation

External knowledge augmentation improves response reliability by grounding generation in retrieved documents or timely information from tools. The survey categorizes retrieval-augmented generation by retriever–generator interaction and describes tool use for rapidly changing data.

  • Overview: External knowledge augmentation addresses hallucinations, in-domain knowledge gaps, and knowledge-updating challenges through retrieval and tool augmentation.The two approaches differ in whether knowledge comes from document corpora or invoked tools.
  • Retrieval-Augmented Generation: Retrieval-Augmented Generation retrieves relevant documents and integrates them into the generator’s input to ground responses in existing reliable knowledge.A typical RAG system contains a retriever and a generator.
  • Sequential RAG: Sequential RAG retrieves information first and then uses it for response generation, with later methods adapting retrievers, rewriting queries, or compressing documents.These stages can use frozen generators, generator feedback, or document compressors to improve fit and efficiency.
  • Branching RAG: Branching RAG processes a query through multiple pipelines and merges their outputs for finer-grained query or retrieval handling.Examples include tree-structured decomposition, independently processed documents, and combining generated with retrieved documents.
  • Conditional RAG: Conditional RAG decides whether retrieval is needed because retrieved knowledge can conflict with the generator’s own knowledge.Methods use classifiers, confidence scores, or response-consistency tests to route queries.
  • Loop RAG: Loop RAG repeatedly interacts between retrieval and generation, supporting complex and diverse queries through multi-turn processing.ITER-RETGEN, IR-COT, and FLARE vary how retrieval is interleaved with generation and confidence assessment.
  • Tool Augmentation: Tool augmentation supports timely acquisition of rapidly changing information, including finance and news data.Search engines are one tool category and can be combined with query decomposition or model-controlled reasoning.

4.3 Generating Response with Citation

Reliable response generation adds citations to generated answers, either from model-intrinsic knowledge or through retrieval-based evidence integration. Research explores citation generation during response production and citation addition after generation.

  • Motivation: Citation-enhanced response generation aims to provide user-centric answers while making their information sources clear.Citations are described as improving trust and supporting adoption.
  • Direct Generating Response with Citation: Intrinsic-memory methods generate citations without relying on a retrieval module.Prompting can guide models to cite pre-training information, while iterative critique improves citation accuracy, correctness, and fluency.
  • Direct Generating Response with Citation: Generative retrieval can produce cited responses by directly generating the corresponding document identifiers.This connects response generation with the document-retrieval approach described for GenIR.
  • Retrieval-Enhanced Citation: 1-PAGER combines evidence retrieval and answer generation by constrained decoding of N-gram DocIDs, matching retrieval-then-read accuracy and surpassing closed-book QA attribution.The process partitions the corpus, selects documents, and generates responses step by step.
  • Retrieval-Enhanced Citation: Retrieval-integrated systems either generate citations within responses or add citations after responses are produced.Within-generation methods include retrieval verification and iterative support checking; post-generation methods locate evidence and revise attribution.
  • Retrieval-Enhanced Citation: Training and optimization methods improve citation-inclusive generation through sentence-level rewards, preference learning, and source-aware association of DocIDs with knowledge.These approaches target more accurate attribution and greater verifiability.

4.4 Personal Information Assistant

Personal information assistants use user histories, profiles, preferences, and behavioral patterns to generate more personalized responses. Research mainly follows personalized prompting or fine-tuning, with retrieval-enhanced and multimodal extensions across several domains.

  • Overview: Personalized information assistants aim to understand users’ personalities and preferences to better meet their information needs.The survey covers personalized dialogue and domain-specific personalization.
  • Personalization Methods: Personalization research mainly uses prompt design or model fine-tuning.These approaches incorporate interaction histories, profiles, personas, conversations, or preference data.
  • Personalized Prompt: Personalized prompting uses interaction histories, ratings, user profiles, reviews, and prompt rewriting to guide language-model outputs.Examples include in-context learning, retrieved personalized history, and rewriting prompts for text-to-image generation.
  • Personalized Fine-tuning: Personalized fine-tuning trains dialogue and response models with persona, conversation, or domain-specific preference data.Reported approaches include Persona-Chat, large Reddit conversation data, reward-model learning, and dynamically combined preference policies.
  • Personalized Extensions: Retrieval-enhanced and parameter-efficient methods incorporate story information or user-specific behavioral patterns to improve response consistency and personalization.Multimodal personalization can transform user behavior into natural language for preference extraction.
  • Applications: Applications span healthcare, academia, education, recipes, news headlines, and other personalized generation tasks.Examples include medical assistance, academic writing, adaptive exercises, recipe generation, and headline personalization.

5 EVALUATION

The evaluation section introduces metrics and benchmarks for assessing generative information retrieval methods and discussing their performance.

  • Evaluation: Evaluation of GenIR methods uses metrics and benchmarks to analyze system performance.The section is presented as covering evaluation measures and performance discussions.

5.1 Evaluation for Generative Document Retrieval

Generative document retrieval is evaluated using accuracy, ranking, efficiency, and relevance metrics across static, dynamic, out-of-distribution, and knowledge-intensive benchmarks. Results show strong performance for several DocID designs, while numeric semantic identifiers and some learnable approaches remain weaker or inconsistent.

  • Metrics and Benchmarks: Recall, R-Precision, MRR, MAP, and nDCG evaluate complementary aspects of generative retrieval effectiveness.They respectively address retrieval coverage, cutoff precision, first-relevant rank, average precision, and relevance ordering.
  • Analysis: DynamicIR reports that GR adapts better to evolving knowledge and uses less memory, indexing time, and FLOPs than dense retrieval systems.The comparison concerns dynamically updated corpora and temporally informed data.
  • Analysis: DSI strongly recalls specific DocIDs from pseudo queries but struggles to distinguish relevant documents from random ones.The analysis separates mapping memorization from relevance discrimination.
  • Analysis: On large MS MARCO collections, synthetic-query augmentation was the only effective method examined, with T5-XL reaching MRR@10 26.7 while T5-XXL declined.The result indicates that scaling parameters alone did not improve this setup.
  • Analysis: GR models lose performance under query variations but show varying adaptability to new query types and tasks, with pre-training improving flexibility.The findings motivate stronger out-of-distribution robustness for changing information sources.
  • Experiments: On MS MARCO and NQ, GR methods are competitive with sparse and dense baselines; TSGen reaches Recall@1 0.708 on NQ, exceeding SPLADEv2 0.624 and DPR 0.502.On MS MARCO, TSGen and DGR also exceed DPR in Recall@1.
  • Experiments: Term-set and multi-view DocIDs perform strongly, with LTRGR achieving Recall@10 0.759 on MS MARCO.Multi-view methods combine document views with ranking and distillation training.
  • Experiments: KILT results favor title and substring DocIDs, while numeric semantic DocIDs degrade sharply as corpus size increases.GENRE leads several KILT categories, SEAL performs strongly on QA tasks, and BART’s FEVER score with semantic IDs is 0.003.

5.2 Evaluation for Response Generation

Response-generation evaluation combines rule-based, model-based, and human metrics to assess accuracy, fluency, relevance, factuality, safety, and predictive ability. Benchmarks further evaluate language understanding, factual accuracy, reliability, and timely information, while human evaluation remains costly and subjective.

  • Evaluation Metrics: Response-generation quality is evaluated with rule-based, model-based, and human metrics covering accuracy, fluency, relevance, and related dimensions.The survey organizes evaluation into these three categories.
  • Rule-based Metrics: Exact Match measures the ratio of outputs that completely match reference answers at the word or character level.It is commonly used for precise question-answering tasks such as NQ, TriviaQA, and SQuAD.
  • Rule-based Metrics: BLEU and ROUGE evaluate generated text through lexical overlap with reference texts, with ROUGE emphasizing recall of reference information.BLEU uses n-gram overlap, while ROUGE includes n-gram and longest-common-subsequence variants.
  • Model-based Metrics: Perplexity is the exponentiated average negative log-likelihood of a text sequence, and lower values indicate stronger predictive ability.The probability term represents predicting each word from its preceding words.
  • Model-based Metrics: Model-based metrics capture semantic relationships using neural representations or generation probabilities rather than relying only on lexical overlap.Examples include BERTScore, BLEURT, BARTScore, and GPTScore.
  • Human Evaluation Metrics: Human evaluation assesses criteria such as accuracy, relevance, fluency, and safety but is expensive, time-consuming, subjective, and difficult to compare consistently.These constraints limit its widespread application and result comparability.
  • Benchmarks and Analysis: Benchmarks assess language understanding, factual accuracy, reliability, and timeliness, with trustworthiness generally positively correlated with functional effectiveness.Proprietary models typically outperform open-source models in trustworthiness, although Llama2 shows comparable high performance.

6 CHALLENGES AND PROSPECTS

The survey identifies scalability, document representation, efficiency, knowledge freshness, bias, and integration as central challenges for generative information retrieval. It outlines research directions spanning improved training and identifiers, real-time updating, fairness, and unified retrieval-generation systems.

  • 6.1 Generative Document Retrieval: GR accuracy falls below dense retrieval on million-document web-search corpora, although it outperforms dense retrieval on collections smaller than 300K.The survey links this scaling gap to training data, training methods, model structure, incremental learning, document identifiers, and inference efficiency.
  • 6.1 Generative Document Retrieval: Wikipedia titles support optimistic GR performance, whereas nonunique or poorly representative titles in 8.8 million MS MARCO passages substantially reduce performance.The survey calls for lightweight DocIDs that accurately represent documents in general corpora.
  • 6.2 Reliable Response Generation: Reliable response generation requires stronger knowledge memorization and better external-knowledge use, including retrieval decisions, retrieval timing, embeddings, and long conversational history.The survey presents retrieval-augmented generation as effective but identifies several unresolved design questions.
  • 6.2 Reliable Response Generation: Outdated model knowledge can reduce reliability, motivating continual learning and knowledge editing, while biases from large unfiltered datasets remain unresolved.The survey highlights real-time knowledge acquisition and bias mechanisms as areas requiring further investigation.
  • 6.3 Unified Framework: Integrating generative document retrieval with reliable response generation could combine document-level retrieval with direct response generation.The proposed large search model would autonomously generate DocIDs and reliable responses, including deciding when to retrieve knowledge during generation.

7 CONCLUSION

The survey reviews generative information retrieval through generative document retrieval and reliable response generation, covering their research progress, evaluation, challenges, and future directions.

  • The survey organizes GenIR into generative document retrieval and reliable response generation.
  • For generative document retrieval, it reviews model training, identifier design, incremental learning, downstream adaptation, multimodal retrieval, and generative recommendation systems.
  • For reliable response generation, it covers internal knowledge memorization, external knowledge enhancement, cited responses, and personal information assistance.
  • The survey organizes evaluation resources and discusses challenges involving scalability, dynamic corpora, document representation, efficiency, accuracy, real-time capability, bias, privacy, and security.

A.1 Evaluation Metrics for Generative Document Retrieval

The survey describes standard metrics for evaluating generative document retrieval, covering recall, precision at relevant-document cutoffs, reciprocal rank, average precision, and position-aware ranking quality.

  • Recall: Recall@k measures the proportion of relevant documents retrieved within the top k results.
  • R-Precision: R-Precision measures precision at rank R, where R equals the number of relevant documents for the query.
  • Mean Reciprocal Rank (MRR): Mean Reciprocal Rank averages the reciprocal rank of the first relevant document returned across queries.
  • Mean Average Precision (MAP): Mean Average Precision averages precision across queries while considering the positions of all relevant documents.
  • Normalized Discounted Cumulative Gain (nDCG): nDCG accounts for both graded document relevance and the positions of documents in the result list.

A.2 Benchmarks for Generative Document Retrieval

The survey presents benchmarks spanning web retrieval, question answering, knowledge-intensive tasks, and deep-learning document and passage ranking.

  • MS MARCO: MS MARCO evaluates retrieval and question answering through document and passage ranking benchmarks derived from Bing search queries.Its document-ranking benchmark contains approximately 3.2 million documents, while passage ranking contains around 8.8 million passages.
  • MS MARCO: MS MARCO uses diverse question types and document genres to approximate complex web-search scenarios for evaluating generative retrieval systems.
  • Natural Questions (NQ): Natural Questions evaluates generative retrieval on real-world information-seeking queries using approximately 3.2 million Wikipedia-page documents and their web-derived answers.
  • KILT: KILT combines five categories of knowledge-intensive tasks, including fact checking, entity linking, and slot filling, using Wikipedia as its primary corpus.Its corpus contains approximately 5.9 million wiki pages and supports cross-domain knowledge integration tasks.
  • TREC Deep Learning Track 2019 & 2020: The TREC Deep Learning Tracks 2019 and 2020 evaluate deep-learning retrieval through document-ranking and passage-ranking tasks.
Loading 2404.14851v4…