PaperlayerView exact revision on arXiv

Source-linked AI summary

Dense Text Retrieval based on Pretrained Language Models: A Survey

Wayne Xin Zhao, Jing Liu, Ruiyang Ren, Ji-Rong Wen

arXiv:2211.14876v1cs.IR

TL;DR

Text retrieval needs representations and relevance matching that handle natural-language queries, including cases where lexical matching is insufficient. This survey reviews PLM-based dense retrieval through architecture, training, indexing, and integration, finding strong performance on complex queries while identifying annotation and scalability boundaries.

  • Problem

    Text retrieval requires effective semantic representations and relevance matching, especially for complex queries that lexical matching may not resolve.

  • Method

    The survey systematically reviews PLM-based dense retrieval through architecture, training, indexing, and integration, supported by practical resources and implementations.

  • Results

    PLM-based dense retrieval has dominated top results for document and passage retrieval in the 2021 Deep Learning Track and is more effective on complicated queries than sparse retrieval.

  • Takeaways & Limitations

    The survey provides a comprehensive, practical reference for designing dense retrieval systems and identifies representation, training, indexing, and pipeline issues.

  • Takeaways & Limitations

    Dense retrieval datasets often contain incomplete relevance annotations, which can create false negatives and harm retrieval performance.

Abstract

from arXiv · show

Text retrieval is a long-standing research topic on information seeking, where a system is required to return relevant information resources to user's queries in natural language. From classic retrieval methods to learning-based ranking functions, the underlying retrieval models have been continually evolved with the ever-lasting technical innovation. To design effective retrieval models, a key point lies in how to learn the text representation and model the relevance matching. The recent success of pretrained language models (PLMs) sheds light on developing more capable text retrieval approaches by leveraging the excellent modeling capacity of PLMs. With powerful PLMs, we can effectively learn the representations of queries and texts in the latent representation space, and further construct the semantic matching function between the dense vectors for relevance modeling. Such a retrieval approach is referred to as dense retrieval, since it employs dense vectors (a.k.a., embeddings) to represent the texts. Considering the rapid progress on dense retrieval, in this survey, we systematically review the recent advances on PLM-based dense retrieval. Different from previous surveys on dense retrieval, we take a new perspective to organize the related work by four major aspects, including architecture, training, indexing and integration, and summarize the mainstream techniques for each aspect. We thoroughly survey the literature, and include 300+ related reference papers on dense retrieval. To support our survey, we create a website for providing useful resources, and release a code repertory and toolkit for implementing dense retrieval models. This survey aims to provide a comprehensive, practical reference focused on the major progress for dense text retrieval.

1 INTRODUCTION

Text retrieval evolved from sparse lexical and feature-based methods toward neural dense representations, while PLM-based dense retrieval became a major focus. This survey organizes that progress around architecture, training, indexing, and integration, with practical resources for researchers and engineers.

  • Text retrieval supports information seeking by returning relevant documents or passages for natural-language queries across applications such as question answering and Web search.
  • Sparse vector methods, probabilistic relevance models, language modeling, and learning-to-rank shaped earlier retrieval systems but relied on lexical representations, heuristics, or hand-crafted features.
  • Neural retrieval learns query and document embeddings from labeled data and measures relevance through semantic similarity in a latent representation space.
  • PLMs encode semantic knowledge from large-scale text and can be fine-tuned for downstream retrieval tasks.
  • The survey focuses on PLM-based dense retrieval and organizes related studies by architecture, training, indexing, and integration.
  • It emphasizes practical techniques, emerging topics, a resource website, and released implementations for dense retrieval models.

2 OVERVIEW

Dense retrieval represents queries and texts with neural encoders and matches their dense vectors, forming a first-stage component within a multi-stage retrieval pipeline. The survey covers task settings, training assumptions, architectures, indexing, and pipeline integration.

  • 2.1 Task Setting and Terminology: The survey unifies document, passage, and sentence retrieval as finding relevant texts from a large collection in response to a natural-language query.
  • 2.1 Task Setting and Terminology: A complete retrieval system uses multiple stages: first-stage retrievers reduce the candidate space, while later rerankers rank candidate texts.
  • 2.2 Formulation for Dense Retrieval: Dense retrieval maps queries and texts to dense vectors and computes relevance with a similarity function such as inner product.
  • 2.2 Formulation for Dense Retrieval: Dense retriever training assumes relevance judgements, typically binary positive-text labels, while negative texts are obtained through sampling or other strategies.
  • 2.3 Key Aspects: The survey examines architecture, training, indexing, and integration as four aspects of building capable first-stage dense retrieval systems.

3 DATASETS, EVALUATION AND RESOURCES

This section surveys datasets, evaluation metrics, and software resources for dense retrieval, while noting annotation and language-coverage limitations. It distinguishes metrics for first-stage retrieval from those used for ranking.

  • 3.1 Datasets: Dense retrieval datasets provide relevance judgments for training and evaluation, with information retrieval, question answering, and other task categories represented.Wikipedia and Web are major sources; MS MARCO and Natural Questions are widely used for evaluating dense retrieval models.
  • 3.1 Datasets: Most datasets provide few relevance judgments per query, creating incomplete annotations and potential false negatives that can harm retrieval performance.Most MS MARCO queries, for example, contain only one labeled positive.
  • 3.1 Datasets: Most released datasets are English, while DuReader-retrieval adds a large-scale Chinese resource with 90K queries and over 8M passages.The Chinese dataset is designed to reduce false negatives in development and testing sets.
  • 3.2 Evaluation Metrics: MRR averages the reciprocal rank of the first retrieved positive text, while NDCG incorporates the positions and graded relevance of retrieved texts.Precision@k and MAP summarize precision among retrieved positions and across positive-text positions, respectively.
  • 3.2 Evaluation Metrics: Recall@k and Accuracy@k commonly evaluate first-stage retrieval, whereas MRR, NDCG, and MAP are more common for ranking.Top-k Accuracy measures whether retrieved texts contain an answer, while traditional Recall@k measures retrieved relevant texts among all relevant texts.
  • 3.3 Code Library Resource: Open-source resources support dense-retrieval implementation and reproducibility, including Tevatron, Pyserini, and RocketQA-based models and toolkits.Tevatron supports processing, training, encoding, and retrieval; Pyserini supports sparse and dense retrieval plus evaluation scripts.

4 ARCHITECTURE

This section introduces Transformer and PLM foundations, then reviews PLM-enhanced sparse retrieval approaches. It emphasizes term weighting and expansion while distinguishing these methods from dense semantic retrieval.

  • 4.1.1 Transformer and Pretrained Language Models: Transformers use self-attention, allowing each token to attend to all input positions and enabling efficient parallelization on GPUs or TPUs.This architecture makes very large neural networks more flexible and easier to train.
  • 4.1.1 Transformer and Pretrained Language Models: PLMs are pretrained on large-scale general text with self-supervised losses and then fine-tuned for downstream tasks.This pretraining-then-fine-tuning paradigm builds on Transformer architectures and variants.
  • 4.1.2 PLMs for Sparse Retrieval: PLM-based sparse retrieval work mainly improves lexical retrieval through contextualized term weighting or query and document expansion.These approaches address retrieval problems without replacing the underlying sparse, lexical-matching framework.
  • 4.1.2 PLMs for Sparse Retrieval: DeepCT regresses BERT token representations into context-specific term weights, while HDCT extends this process to long documents by aggregating passage-level weights.HDCT uses titles, Web inlinks, and pseudo-relevance feedback as data resources.
  • 4.1.2 PLMs for Sparse Retrieval: docTTTTTquery enriches documents with predicted relevant queries, whereas SPLADE methods project query and text terms into vocabulary-sized weighted vectors.SPLADE applies sparsity regularization so the expanded representations can use inverted indexes.
  • 4.1.2 PLMs for Sparse Retrieval: PLM-enhanced sparse methods can reuse traditional inverted indexes by adding payloads such as contextualized embeddings, but they remain based on lexical matching.COIL additionally incorporates semantic matching through [CLS] embeddings to alleviate term mismatching.

4.2 Neural Architecture for Dense Retrieval

Dense retrieval uses cross-encoders and bi-encoders to model query–text semantics, trading cross-encoder interaction capacity against bi-encoder efficiency and indexability.

  • Cross-Encoder Architecture: Cross-encoders concatenate a query and text and let a PLM model token-level interactions across the entire sequence.The combined sequence uses [CLS] and [SEP] symbols, with the [CLS] representation commonly used for matching.
  • Bi-Encoder Architecture: Bi-encoders independently encode queries and texts, then compute relevance from similarity between their dense embeddings.Cosine similarity and inner product are example scoring functions.
  • Bi-Encoder Architecture: Single-representation bi-encoders are simple, while multi-representation variants improve fine-grained interaction through multiple contextual embeddings.Multi-representation models store contextualized representations during training and indexing, enabling richer query-time matching.
  • Bi-Encoder Architecture: Multi-representation bi-encoders can improve retrieval effectiveness but impose substantially higher maintenance costs, including increased index size.ColBERTer reduces these costs through dimension reduction, unique whole-word representations, and contextualized stopword removal.
  • Comparison between Cross- and Bi-Encoder: Cross-encoders generally achieve stronger fine-grained matching than bi-encoders, including multi-representation variants reported on MS MARCO.Multi-representation bi-encoders outperform single-representation bi-encoders but remain below cross-encoders.
  • Comparison between Cross- and Bi-Encoder: Bi-encoders support flexible architectures and approximate nearest-neighbor search, making them suitable for efficient large-scale candidate recall.Practical systems commonly use bi-encoders for first-stage retrieval and cross-encoders for reranking or reading.

4.3 Sparse Retrieval v.s. Dense Retrieval

Sparse and dense retrieval capture complementary relevance signals: dense models help with semantic mismatch, while sparse models retain advantages for exact matching and zero-shot settings.

  • Strengths and Weaknesses: Dense retrievers often outperform classic sparse models on question-answering benchmarks requiring semantic matching despite few overlapping query and answer terms.DPR retrieves text connecting “bad guy” with “villain,” illustrating semantic handling of term mismatch.
  • Strengths and Weaknesses: Dense retrievers can perform worse than sparse models in zero-shot target-domain retrieval and on rare exact-match entities or compositions.BEIR experiments report limited zero-shot capacity, while DPR fails on “Thoros of Myr” where BM25 succeeds.
  • Strengths and Weaknesses: Dense retrieval’s compressed latent representations may lose salient lexical information, whereas sparse retrieval relies on lexical matching.This mechanism helps explain the different behavior on semantic versus exact-match queries.
  • Theoretical Analysis: Increasing embedding size can help dense retrievers mimic bag-of-words performance as document length increases, while larger indexes can hurt dense retrieval more sharply.The reported explanation is that false-positive probability rises with index size, especially when dimensionality decreases.
  • Theoretical Analysis: Existing traditional IR axioms only partially explain or may not suit PLM-based retrieval, whose behavior differs across ranking architectures and can react to non-relevant content.Some dense models nevertheless show awareness of term importance in text representations.
  • Integration: Hybrid retrieval combines sparse and dense signals, but conventional hybrids require maintaining both inverted and dense-vector indexes.Densified lexical representations aim to support end-to-end lexical-semantic retrieval with lower deployment complexity.
  • Integration: Lexicon-aware dense retrieval enhances lexical matching through lexicon-augmented contrastive learning and rank-consistent regularization.LED is reported to outperform multiple competitive baselines.

5 TRAINING

The survey frames training PLM-based dense retrievers as central to retrieval quality, focusing on candidate-space scale, limited relevance judgments, and pretraining discrepancy.

  • Training: Training is presented as key to achieving good retrieval performance in PLM-based dense retrievers.The discussion focuses on bi-encoders used for first-stage retrieval.
  • Training: The survey organizes training challenges around large-scale candidate spaces, limited relevance judgments, and pretraining discrepancy.It introduces these issues before discussing corresponding solutions.

5.1 Formulation and Training Issues

PLM-based dense retrievers are trained by optimizing query–text relevance, with sampled negatives and additional constraints addressing large candidate spaces, limited labels, and pretraining mismatch.

  • 5.1.1 Loss Function: Dense retrieval training maximizes the likelihood of relevant texts given queries, typically using a similarity function over query and text embeddings.Exact likelihood over the collection is costly, so training usually samples a smaller negative set.
  • 5.1.1 Loss Function: Negative sampling increases efficiency by contrasting each positive text with a selected set of negative texts, resembling InfoNCE-style contrastive optimization.The objective raises positive-text likelihood while lowering sampled-negative likelihood.
  • 5.1.1 Loss Function: Binary cross-entropy predicts query–text relevance from a vector combination of dense representations, with labels allowing either positive or negative texts.The match vector can use operations such as concatenation, and relevance probability is modeled with a sigmoid.
  • 5.1.1 Loss Function: Similarity functions include inner product, cosine similarity, and Euclidean distance, and studies compare their effects on retrieval performance.The supplied passage introduces empirical comparison of similarity choices but does not report its outcome.
  • 5.1.2 Incorporating Optimization Constraints: Jointly optimizing query-oriented and text-oriented objectives has been reported to improve retrieval performance through symmetric optimization.Text-oriented optimization adds negative query likelihood alongside query-oriented likelihood.
  • 5.1.2 Incorporating Optimization Constraints: Text-text similarity constraints separate positive passages from semantically similar negatives rather than optimizing only query–text similarity.The constraint incorporates text-text similarity into the normalization term.
  • 5.1.3 Major Training Issues: Large candidate collections create a train–test candidate-space shift because computation limits restrict training to a small number of sampled negatives.Sampled-negative quality significantly affects retrieval performance.
  • 5.1.3 Major Training Issues: Limited relevance judgements create likely false negatives, while retrieval-unspecialized PLM pretraining and generic [CLS] representations can produce suboptimal retrieval.The survey identifies negative selection, data augmentation, and retrieval-oriented pretraining as responses to these bottlenecks.

5.2 Negative Selection

Negative selection is central to dense-retriever training because candidate collections are large and random or in-batch negatives may not adequately represent difficult retrieval distinctions.

  • Negative selection: Sampled negatives are required for efficient negative log-likelihood training, making high-quality negative selection important for retrieval performance.The section reviews random, in-batch, cross-batch, and hard-negative strategies.
  • Random and batch negatives: In-batch negatives reuse positives from other queries, while cross-batch negatives reuse examples across GPUs to increase negatives efficiently.Gradient caching extends the cross-batch idea to single-GPU training at greater computational cost.
  • Hard negatives: Hard negatives are irrelevant texts with high semantic similarity to the query, and they help models discriminate relevant from irrelevant texts.In-batch and cross-batch methods do not guarantee hard negatives.
  • Static hard negatives: Static hard negatives are selected by a fixed retriever, commonly by sampling top results from sparse or dense retrievers such as BM25.The selection target is texts that are difficult for the dense retriever to discriminate.
  • Dynamic hard negatives: Dynamic hard negatives update the negative selector during training, with ANCE periodically refreshing indexed embeddings and ADORE adapting query-side retrieval.These methods address the iterative nature of dense-retriever optimization.
  • Denoised hard negatives: Noisy hard negatives can harm retrieval because they are more likely to be false negatives, making negative quality especially consequential.Dense retrievers are reported to be sensitive to the quality of sampled negatives.
  • Effect of negative sampling: In-batch sampling cannot generate sufficiently informative negatives because batches represent a much smaller distribution than the full collection.Negative informativeness is linked to training convergence through gradient norms.
  • Effect of negative sampling: Random negative sampling minimizes total pairwise errors, whereas hard-negative sampling minimizes top pairwise errors and better targets top-rank optimization.The two strategies therefore optimize different retrieval objectives.

5.3 Data Augmentation

Data augmentation expands scarce relevance supervision through auxiliary datasets, knowledge distillation, and pseudo-label generation for dense retrievers.

  • Data augmentation: Data augmentation addresses limited relevance judgements by adding labeled datasets or generating pseudo relevance labels through knowledge distillation.These approaches target the mismatch between available supervision and large PLM capacity.
  • Auxiliary labeled datasets: Multi-dataset training can produce comparable performance to task-specific models and better performance in few-shot settings.DPR variants share query and passage encoders either separately or jointly across tasks.
  • Auxiliary data sources: Combining text, tables, lists, and knowledge bases by flattening structured data into text is useful across five question-answering datasets.The supplied passage reports usefulness overall but does not provide a numerical result.
  • Zero-shot retrieval: In zero-shot retrieval, auxiliary datasets can alleviate data scarcity, but final performance is highly affected by which auxiliary datasets are used.This makes auxiliary-data choice an important scope condition.
  • Knowledge distillation: Knowledge distillation transfers knowledge from a more capable teacher to a less capable bi-encoder student when human relevance judgements are limited.Cross-encoders can serve as teachers because they model fine-grained query–text interactions.
  • Distillation for the student network: Teacher predictions can train students with hard labels from thresholded scores or soft labels that approximate teacher relevance scores.Hard-label distillation filters low-confidence predictions, while soft-label methods use score-based losses.
  • Distillation functions: KL-divergence loss outperformed MSE and max-margin loss for question-answering distillation, while Margin-MSE was also reported as more effective than alternatives.The reported comparisons concern different empirical studies and tasks or settings.
  • Advanced distillation methods: Large teacher–student capacity gaps can reduce distillation effectiveness, motivating progressive distillation with gradually enhanced teachers.The survey describes progressive distillation as an alternative to direct distillation with strong teachers.

5.4 Pretraining for Dense Retrieval Models

Retrieval-oriented pretraining adapts PLMs to dense retrieval by improving task alignment, generating synthetic supervision, enriching context, and strengthening whole-text representations.

  • Motivation: Generic PLM representations can be suboptimal downstream because pretraining lacks task-specific optimization, motivating retrieval-related pretraining strategies.These strategies can also alleviate relevance-data scarcity.
  • Self-supervised pretraining: Self-supervised retrieval tasks mimic matching using signals from original text, including sentence context, article structure, hyperlinks, recurring spans, and representative words.Examples include ICT, BFS, WLP, RSR, and ROP.
  • Generation-augmented pretraining: Synthetic question–text pretraining generates pseudo pairs through pipeline or end-to-end approaches, including answer extraction, question generation, and filtering.End-to-end methods condition question-and-answer generation on input text and may add selection steps.
  • Generation-augmented pretraining: Generation-augmented pretraining improves zero-shot retrieval capacity when target-domain training datasets are unavailable.The supplied evidence reports usefulness across several studies without specifying a single metric.
  • Retriever-enriched pretraining: Retriever-enriched pretraining augments masked-language modeling with relevant contexts retrieved from a large background corpus.REALM trains the context retriever through the usefulness of retrieved passages for masked-word prediction.
  • Retriever-enriched pretraining: REALM experiments indicate that suitable training and supervision strategies are important for improving open-domain question answering.The reported tuning used larger batches, more retrieved documents, exact vector search, and human evidence annotations.
  • Representation-enhanced pretraining: The original [CLS] embedding is not explicitly designed to represent whole-text meaning, motivating representation-enhanced pretraining for dense retrieval.This is identified as a limitation of typical bi-encoder representation practice.
  • Representation-enhanced pretraining: Autoencoder pretraining compresses text into [CLS] and reconstructs the original text, encouraging the embedding to capture more text information.A paired decoder performs reconstruction from the [CLS] representation.

5.5 Empirical Performance Analysis with RocketQA

RocketQA experiments on MS MARCO passage retrieval isolate the effects of several dense-retriever optimization techniques. Cross-batch negatives, denoised hard negatives, data augmentation, passage-centric pretraining, and dynamic listwise distillation each improve performance under the reported comparisons.

  • The experiments use MS MARCO passage retrieval and compare RocketQA variants with BM25 and DPR baselines.The study releases implementation and reproduction code for the reported experiments.
  • Cross-batch negatives improve absolute MRR@10 by 0.9 percentage points compared with DPRERNIE.Using more negatives during cross-batch training, typically through a larger batch size, is reported as important.
  • Denoised hard negatives improve absolute MRR@10 by 3.1 percentage points over the corresponding RocketQA variant.The authors contrast this with rapidly decreasing performance when non-denoised hard negatives are used.
  • Data augmentation adds 0.6 percentage point in absolute MRR@10 when combined with cross-batch training and denoised hard negatives.Performance also improves as the amount of pseudo-labeled data increases.
  • Passage-centric pretraining improves absolute MRR@10 by 0.7 percentage point in the reported variant comparison.The technique characterizes a more comprehensive semantic relation among queries, positive texts, and negative texts.
  • Dynamic listwise distillation achieves the best performance among the RocketQA variants.It jointly trains retriever and reranker using soft-label distillation.

6 INDEXING FOR DENSE RETRIEVAL

Dense retrieval indexing is formulated as nearest neighbor search over embedding vectors rather than lexical matching. The survey covers index structures and product quantization that trade search efficiency, memory, and retrieval quality, while noting limits of fully in-memory solutions at extreme scale.

  • Dense retrieval searches for candidate text embeddings nearest to a query embedding under a similarity or distance function.The formal objective selects the candidate maximizing the similarity score.
  • Approximate nearest neighbor search trades search efficiency against retrieval quality, with ANN-Benchmarks and Big-ANN-Benchmarks providing public evaluations at million and billion scales.These benchmarks maintain performance rankings under different settings.
  • Search efficiency can improve by reducing similarity computations through hashing-, clustering-, or graph-based index structures.Hashing searches selected buckets, clustering searches likely clusters, and graph methods navigate nearest-neighbor connections.
  • Product quantization compresses real-valued text embeddings and reduces the time and memory overhead of similarity computation.It can also be combined with clustering-based or graph-based indexes.
  • Product quantization splits each embedding into subvectors and stores nearest-centroid assignments as a compact vector of PQ indices.When each centroid set has fewer than 256 centroids, each index can use one byte, and representations can be pre-computed before search.
  • The survey mainly discusses in-memory ANNS, but extremely large collections such as 100 billion items require hybrid memory-disk multi-level indexes.DiskANN is cited as an example of this approach.

7 INTEGRATION WITH RERANKING

Dense retrieval systems commonly combine a first-stage retriever with one or more rerankers in a cascade. The survey compares separate, adaptive, and joint training approaches for coordinating these stages and their relevance signals.

  • First-stage retrieval efficiently recalls candidates from a large corpus, while reranking applies more capable models to a smaller candidate set.Practical pipelines may contain multiple reranking stages or multiple first-stage retrievers.
  • PLM-based Rerankers and Multi-stage Ranking: PLM-based cross-encoders such as monoBERT estimate query-text relevance by jointly encoding the concatenated pair.Cross-encoders substantially improve over traditional ranking methods in the cited discussion.
  • PLM-based Rerankers and Multi-stage Ranking: Multi-stage ranking can combine BM25, monoBERT, and duoBERT as successive rankers that refine candidate ordering.The approach is described as a processing pipeline with different ranking stages.
  • Pipeline Training: Separate retriever and reranker optimization may miss cascading information correlations and produce suboptimal pipeline ranking.The reranker is not adjusted to the retriever’s result distribution when the stages are trained independently.
  • Pipeline Training: Adaptive training lets retriever and reranker components respond to each other’s retrieval results or intermediate representations.Examples include sampling retriever top results as reranker negatives and alternating updates with pseudo-relevance labels.
  • Pipeline Training: RocketQAv2 jointly optimizes bi-encoder and cross-encoder relevance distributions using dynamic listwise distillation.The mechanism distills the reranker to the retriever within a unified listwise learning approach.

8 ADVANCED TOPICS

The survey covers advanced dense-retrieval topics including zero-shot generalization, query-variation robustness, model-based retrieval, and retrieval-augmented language models. It reviews evaluation findings, enhancement strategies, and the benefits and current scale limitations of model-based retrieval.

  • 8.1 Zero-shot Dense Retrieval: Zero-shot retrieval is difficult because dense retrievers depend heavily on large-scale relevance judgements that are expensive to acquire for new domains or tasks.The survey therefore examines zero-shot capabilities and out-of-distribution performance as important research questions.
  • 8.1 Zero-shot Dense Retrieval: BEIR evaluates zero-shot dense retrieval across nine tasks and 18 diverse datasets, exposing weaknesses of many in-domain bi-encoder retrievers.Cross-attentional reranking and late-interaction models show better zero-shot performance, while BM25 can outperform dense retrievers on simple entity-centric questions.
  • 8.1 Zero-shot Dense Retrieval: Synthetic query-text generation, sparse-dense fusion, score fusion, and distillation from sparse retrievers are proposed to improve zero-shot retrieval.These methods respectively augment target-domain training data or strengthen exact-term matching, which empirical studies identify as useful for zero-shot performance.
  • 8.2 Robustness to Query Variations: Dense retrievers are more sensitive than traditional lexical methods to query typos and alternative formulations, motivating augmentation and local ranking alignment.Local ranking alignment encourages original queries and their variations to produce similar distributions over in-batch passages and queries.
  • 8.3 Model based Retrieval: Model-based retrieval predicts relevant document identifiers directly with a parametric model instead of searching an index, offering a unified, end-to-end, index-light paradigm.Its main practical boundary is scale: most studies use MS MARCO subsets or task-specific datasets, while full MS MARCO remains challenging.
  • 8.4 Retrieval-Augmented Language Model: Scaling retrieval corpora to trillions of tokens improves the capacity of retrieval-augmented language models, while joint retriever-and-encoder optimization provides an alternative.The cited studies compare frozen pretrained retrievers with REALM’s jointly optimized knowledge retriever and encoder.

9 APPLICATIONS

The survey describes dense retrieval applications across retrieval settings, NLP tasks, and industry practice. These applications include temporal, structured, multilingual, cross-lingual, domain-specific, cross-modal, question-answering, entity-linking, and knowledge-grounded dialogue scenarios.

  • 9.1 Information Retrieval Tasks: Temporal retrieval research introduces SituatedQA, which requires answers conditioned on tempo-spatial contexts; existing dense retrievers cannot fully solve this setting.The dataset targets temporal-dependent questions that are common in real application scenarios.
  • 9.1 Information Retrieval Tasks: Dense retrieval has been extended to tables, multilingual collections, cross-lingual QA, mathematical information retrieval, code retrieval, and biomedical retrieval.Domain adaptation may require continual pretraining, handling specialized formats, and addressing limited training data.
  • 9.1 Information Retrieval Tasks: Cross-modal retrieval maps or fuses representations across modalities such as text and images or video, with vision-language pretraining improving shared representation learning.The survey identifies cross-modal fusion or mapping as the key design issue.
  • 9.2 Natural Language Processing Tasks: Open-domain QA commonly retrieves documents or passages first and then derives answers with a reader, while dense methods such as DPR and ORQA outperform BM25 on multiple QA datasets.RAG and Fusion-in-Decoder use generative sequence-to-sequence models and show better open-domain QA performance than extractive readers.
  • 9.2 Natural Language Processing Tasks: Entity linking uses PLMs for candidate recall or reranking so contextual mentions can be linked efficiently within large knowledge bases.The surveyed systems include BERT-based rankers and the BLINK two-stage approach.
  • 9.2 Natural Language Processing Tasks: Retrieval-augmented dialogue addresses factual incorrectness and knowledge hallucination by retrieving external knowledge from dialogue contexts before generation.The retrieved knowledge is used to enhance dialogue generation in end-to-end trained systems.

10 CONCLUSION

The survey organizes PLM-based dense retrieval around architecture, training, indexing, and pipeline integration, while also covering advanced topics and applications. It provides a broad literature synthesis and supporting resources, but large-scale indexing and model-based retrieval remain practical challenges.

  • 10 CONCLUSION: The survey reviews recent PLM-based dense-retrieval progress across four major aspects and includes more than 300 related reference papers.It also provides a website, paper collection, code repository, and toolkit for dense-retrieval research and implementation.
  • 10 CONCLUSION: Bi-encoders provide efficient retrieval but weaker semantic interaction, whereas cross-encoders are more effective but less efficient and are commonly used for reranking.Multi-representation techniques are used to enhance bi-encoder capacity.
  • 10 CONCLUSION: Negative selection, data augmentation, and specialized pretraining address large candidate spaces, limited relevance judgements, and pretraining discrepancy in PLM-based retrieval.Using more or higher-quality negatives typically improves retrieval, and distillation can strengthen bi-encoders.
  • 10 CONCLUSION: Approximate Nearest Neighbor Search enables efficient search over large dense-vector indexes, which are otherwise extremely slow to query.Dense-vector indexing remains a central systems challenge for practical retrieval.
  • 10 CONCLUSION: Jointly optimizing retrieval and reranking is preferred because optimizing the complete multi-stage pipeline is more complicated than optimizing an individual component.The conventional pipeline progressively reduces the search space and refines the highest-ranked texts.
  • 10 CONCLUSION: Academic solutions for instantly maintaining embedding indexes remain lacking, despite commercial software supporting immediate data addition, editing, and deletion.This marks a practical boundary for dense-retrieval infrastructure.
Loading 2211.14876v1…