PaperlayerView exact revision on arXiv

Source-linked AI summary

Semantic Models for the First-stage Retrieval: A Comprehensive Review

Jiafeng Guo, Yinqiong Cai, Yixing Fan, Fei Sun, Ruqing Zhang, Xueqi Cheng

arXiv:2103.04831v4cs.IR

TL;DR

First-stage retrieval needs semantic models that efficiently achieve high recall, because term-based methods face vocabulary mismatch and may miss relevant documents. This survey unifies classical, early semantic, and neural retrieval methods, emphasizing sparse, dense, and hybrid paradigms, and identifies challenges and future directions; its synthesis notes stronger effectiveness when neural methods are combined with term-based retrieval.

  • Problem

    First-stage retrieval requires efficient high-recall models to return potentially relevant documents, while semantic models must address vocabulary mismatch in retrieval.

  • Method

    The survey organizes first-stage retrieval under a unified framework connecting classical, early semantic, and neural methods, with emphasis on sparse, dense, and hybrid paradigms.

  • Results

    The survey reports that neural semantic retrieval methods are often more effective when combined with term-based retrieval methods rather than replacing them.

  • Takeaways & Limitations

    The survey identifies unresolved challenges and promising directions intended to motivate further research on first-stage semantic retrieval.

  • Takeaways & Limitations

    Jointly learning retrieval representations and indexes for information retrieval remains at an early stage, limiting the maturity of advanced indexing schemes.

Abstract

from arXiv · show

Multi-stage ranking pipelines have been a practical solution in modern search systems, where the first-stage retrieval is to return a subset of candidate documents, and latter stages attempt to re-rank those candidates. Unlike re-ranking stages going through quick technique shifts during past decades, the first-stage retrieval has long been dominated by classical term-based models. Unfortunately, these models suffer from the vocabulary mismatch problem, which may block re-ranking stages from relevant documents at the very beginning. Therefore, it has been a long-term desire to build semantic models for the first-stage retrieval that can achieve high recall efficiently. Recently, we have witnessed an explosive growth of research interests on the first-stage semantic retrieval models. We believe it is the right time to survey current status, learn from existing methods, and gain some insights for future development. In this paper, we describe the current landscape of the first-stage retrieval models under a unified framework to clarify the connection between classical term-based retrieval methods, early semantic retrieval methods and neural semantic retrieval methods. Moreover, we identify some open challenges and envision some future directions, with the hope of inspiring more researches on these important yet less investigated topics.

1 INTRODUCTION

Modern search uses multi-stage pipelines in which efficient first-stage retrieval must achieve high recall, but classical term-based models remain vulnerable to vocabulary mismatch. This survey organizes traditional, early semantic, and neural first-stage retrieval methods under a unified framework.

  • 1 INTRODUCTION: The first stage retrieves candidate documents efficiently, while later stages apply more complex models to prune and improve the ranked list.This division balances search efficiency and effectiveness in large-scale retrieval.
  • 1 INTRODUCTION: First-stage retrieval seeks high recall by returning as many potentially relevant documents as possible within a short time.Its goal differs from later re-ranking stages, which process only a small candidate set.
  • 1 INTRODUCTION: Classical term-based models are efficient and provide reasonable recall, but their discrete representations suffer from vocabulary mismatch and limited semantic coverage.These models use bag-of-words representations, inverted indexes, and methods such as BM25.
  • 1 INTRODUCTION: Research on first-stage semantic retrieval expanded rapidly with word embeddings after 2013 and deep learning after 2016.These methods pursue denser or improved document representations to alleviate limitations of symbolic term matching.
  • 1 INTRODUCTION: The survey reviews early and neural semantic retrieval methods, clarifies their connection with classical retrieval, and identifies open challenges and future directions.It emphasizes recent neural methods and organizes them as sparse, dense, and hybrid paradigms.

2 MAJOR APPLICATIONS OF SEMANTIC RETRIEVAL MODELS

First-stage semantic retrieval supports several large-scale applications, including ad-hoc retrieval, open-domain question answering, and community-based question answering. These applications require candidate recall before later ranking or answer extraction, while the survey excludes some other scenarios because of page limitations.

  • 2 MAJOR APPLICATIONS OF SEMANTIC RETRIEVAL MODELS: The survey identifies ad-hoc retrieval, open-domain question answering, and community-based question answering as three major applications.All require a first-stage retrieval component over a document or question-answer collection.
  • 2 MAJOR APPLICATIONS OF SEMANTIC RETRIEVAL MODELS: Ad-hoc retrieval returns documents ranked by relevance to a short user query, whose length difference from documents creates vocabulary mismatch.Early evaluations commonly used TREC collections such as AP, WSJ, and Robust.
  • 2 MAJOR APPLICATIONS OF SEMANTIC RETRIEVAL MODELS: Open-domain question answering retrieves a small set of documents from a large collection before extracting an answer span.Its pipeline therefore depends on document retrieval as an initial stage.
  • 2 MAJOR APPLICATIONS OF SEMANTIC RETRIEVAL MODELS: Community-based question answering retrieves existing answers or duplicate questions and then uses the associated answer to respond.Both approaches first recall candidates from the full collection before producing the final result.
  • 2 MAJOR APPLICATIONS OF SEMANTIC RETRIEVAL MODELS: The survey notes other semantic-retrieval scenarios, including entity linking, e-commerce search, and sponsored search, but does not discuss their works because of page limitations.It suggests that text-retrieval techniques may generalize to these tasks.

3 BACKGROUND

First-stage retrieval ranks documents across very large corpora, so models must combine effective relevance scoring with efficient offline indexing and online serving. Classical term-based methods achieve speed through sparse bag-of-words representations and inverted indexes, but their shallow lexical matching leaves semantic gaps.

  • Problem Formalization: First-stage retrieval recalls potentially relevant documents from corpora ranging from millions to billions, making efficiency a crucial concern.Unlike re-ranking, it ranks across the full collection rather than a small candidate set.
  • Problem Formalization: A unified retrieval model maps queries and documents to representations and applies a scoring function to rank corpus documents by predicted relevance.The document representation must support offline precomputation, while query processing must remain efficient online.
  • Indexing Methods: Indexing techniques, including inverted and dense vector indexes, are essential for rapid retrieval during online serving.The choice of index must match the representation paradigm used by the retrieval model.
  • Indexing Methods: Dense semantic representations motivate approximate nearest neighbor search because inverted indexes rely on sparse document-term matrices.Brute-force similarity search becomes impractical at collection scale, whereas ANN trades slight precision loss for much faster retrieval.
  • Classical Term-based Methods: Classical term-based models use bag-of-words representations with manually defined features, enabling efficient retrieval through inverted indexes.Vector space, probabilistic, and language-model approaches differ in scoring foundations but retain the bag-of-words assumption.
  • Classical Term-based Methods: Shallow lexical modeling makes classical methods efficient but leaves vocabulary mismatch and incomplete semantic understanding as clear drawbacks.These limitations motivate more sophisticated semantic models for first-stage retrieval.

4 EARLY METHODS FOR SEMANTIC RETRIEVAL

Early semantic retrieval methods enrich lexical representations with phrases, query or document expansion, term dependencies, topics, and translation relationships. These approaches address particular semantic gaps, but their benefits can be inconsistent, costly, or limited by shallow representations and mismatched training signals.

  • 4.1 Query Expansion: Query expansion adds relevant terms to the original query, using global resources or local feedback to broaden retrieval.Pseudo-relevance feedback is effective in many settings but can cause query drift and inconsistent improvement.
  • 4.1 Query Expansion: Query expansion does not always improve retrieval consistently, and pseudo-relevance feedback is particularly prone to query drift.Neural embeddings and deep language models were later used to enhance expansion methods.
  • 4.2 Document Expansion: Document expansion enriches every corpus document before indexing, supplementing inverted-index posting lists with related terms or concepts.Unlike query expansion, changing the method requires re-indexing the corpus, making rapid experimentation costly.
  • 4.3 Term Dependency Models: Term dependency models incorporate ordered or contiguous term relationships so phrase-level relevance can complement independent term matching.Phrase scores may be combined with term-level scores, while n-gram extensions and Markov Random Fields model increasingly structured dependencies.
  • 4.3 Term Dependency Models: Term dependency models increase complexity, yet early gains were limited before Markov Random Fields reported a clear improvement over term-based baselines.Their contextual modeling nevertheless motivated neural retrieval methods that capture richer interactions.
  • 4.4 Topic Models: Topic models represent queries and documents in latent topic spaces, but topic-only retrieval often yields small gains or poor performance over term-based baselines.Combining topic representations with term-based methods is generally needed, partly because topic vectors lose detailed term-level matching signals.

5 NEURAL METHODS FOR SEMANTIC RETRIEVAL

Neural semantic retrieval uses neural networks to learn representations and scoring functions that capture deep semantics and complex interactions end to end. The survey organizes these methods into sparse, dense, and hybrid architectures.

  • Neural Retrieval: Neural networks provide expressive representations that convert discrete text into dense vectors capturing semantic and syntactic features.This capability drove neural approaches beyond the earlier focus on re-ranking toward first-stage semantic retrieval.
  • Neural Retrieval: Neural semantic retrieval learns representation functions and scoring functions end to end from data.The survey categorizes these architectures as sparse, dense, and hybrid retrieval methods.

5.1 Sparse Retrieval Methods

Sparse retrieval methods preserve efficient term-based indexing while incorporating neural models to improve term weighting, document expansion, or learned sparse representations. These approaches aim to add semantic information without abandoning sparse-vector retrieval.

  • Sparse Retrieval Methods: Sparse retrieval represents queries and documents with sparse vectors, activating only a small number of dimensions.The survey distinguishes neural weighting schemes from directly learned sparse representations.
  • Neural Weighting Schemes: Neural weighting schemes predict semantic term importance or augment documents with additional terms before classical indexing.Examples include replacing heuristic term weights and expanding documents with neural sequence-to-sequence models.
  • Neural Weighting Schemes: Contextual models such as DeepCT map BERT representations to context-aware term weights that replace the original TF field in inverted indexes.This extends neural term weighting from static embeddings to contextualized representations.
  • Neural Weighting Schemes: Document expansion methods use neural sequence-to-sequence models to add elite terms, thereby promoting them in the inverted index.The doc2query approach is presented as a neural implementation of document expansion.
  • Sparse Representation Learning: Sparse representation learning directly builds latent sparse query and document vectors whose dimensions need not have clear concepts.This differs from symbolic term weighting while retaining sparse representations for indexing.
  • Sparse Representation Learning: LSH-based approximations make interaction-focused neural models more applicable to first-stage retrieval by using fast histogram calculations and cached term vectors.The approach was evaluated with DRMM, KNRM, and Conv-KNRM.

5.2 Dense Retrieval Methods

Dense retrieval replaces sparse term representations with independently encoded query and document embeddings, then scores them through similarity functions. The survey covers term-level and document-level representations, contextual architectures, multi-vector designs, and the efficiency constraints shaping first-stage retrieval.

  • Dense Retrieval Methods: Dense retrieval uses twin encoders to independently produce query and document embeddings, which a matching function scores and approximate nearest-neighbor search indexes.This dual-encoder design supports offline document encoding and online retrieval.
  • Dense Retrieval Methods: Dense retrieval models divide into term-level methods, which aggregate fine-grained matching signals, and document-level methods, which use one or more coarse global representations.The distinction follows the form of the learned document representation.
  • Term-level Representation Learning: DC-BERT achieves 10x speedup over original BERT document retrieval while retaining about 98% of QA performance on SQuAD and Natural Questions.It pre-encodes document term representations offline and uses configurable high-layer Transformer interaction.
  • Document-level Representation Learning: Document-level approaches commonly compute relevance from query and document embeddings using dot product or cosine similarity.Early methods aggregated word embeddings, but such aggregation loses contextual and word-order information.
  • Document-level Representation Learning: DPR learns dense text-block embeddings with a BERT-based dual encoder for retrieval.The survey presents DPR as a representative more powerful representation-learning model for first-stage retrieval.
  • Document-level Representation Learning: Knowledge distillation can transfer ColBERT’s expressive MaxSim scoring into a simple dot product, enabling single-step approximate nearest-neighbor search.This approach retains a document-level representation architecture while simplifying retrieval.
  • Document-level Representation Learning: Early dual-encoder neural models often failed to beat BM25 on academic benchmarks, especially when trained on whole documents rather than titles.These limitations motivated models designed specifically for first-stage retrieval.

5.3 Hybrid Retrieval Methods

Hybrid retrieval combines sparse exact-matching signals with dense semantic representations to balance fidelity and generalization in first-stage retrieval. The survey describes hybrid architectures that merge sparse and dense scores, with empirical benefits over purely lexical or neural alternatives.

  • 5.3 Hybrid Retrieval Methods: Hybrid retrieval combines sparse representations that preserve exact matching with dense embeddings that encode semantic information.Sparse methods offer discriminative hard matching, while dense methods provide soft matching but sacrifice detailed low-level features.
  • 5.3 Hybrid Retrieval Methods: Hybrid models use multiple representation functions to produce sparse and dense query-document representations, then merge their scores into a final matching score.This architecture is presented as a way to combine the merits of sparse and dense retrieval.
  • 5.3 Hybrid Retrieval Methods: Word-embedding models alone can underperform traditional language models, whereas combining embeddings with term-based methods improves retrieval effectiveness.This pattern is reported across monolingual, bilingual, and non-telescoping retrieval settings.
  • 5.3 Hybrid Retrieval Methods: BOW-CNN combines bag-of-words and convolutional representations, outperforming TF-IDF and showing greater robustness than pure CNN retrieval for long texts.Its final score combines separate BOW and CNN similarity components.
  • 5.3 Hybrid Retrieval Methods: CLEAR retrieval without re-ranking is reported as almost as accurate as a BERT re-ranking pipeline, while other hybrid approaches combine BERT with BM25.These results support the complementary use of semantic and lexical retrieval models.

5.4 Model Learning

Neural semantic retrieval models learn representations and scoring functions from data, with training objectives and negative sampling central to model quality. The survey reviews common losses and sampling strategies, emphasizing that hard-negative mining remains an open practical challenge.

  • 5.4.1 Loss Functions: Training objectives seek to satisfy document preference relationships so models produce effective ranked lists for queries.The survey discusses sampled cross entropy and hinge loss as commonly used objectives.
  • 5.4.1 Loss Functions: Loss functions use relevant documents and irrelevant-document sets, making negative-example construction a crucial input to retrieval-model learning.Negative examples are often not explicitly labeled because training data may provide only positive examples.
  • 5.4.2 Negative Sampling Strategies: Negative sampling includes random, static hard, and dynamic hard strategies, with hard negatives drawn from retrieved or model-predicted documents.Random and hard negatives are commonly used together rather than treated as mutually exclusive.
  • 5.4.2 Negative Sampling Strategies: Periodically refreshing the index and retrieving top-ranked documents is presented as a practical compromise for hard-negative mining.This strategy helped a BERT-based siamese architecture exceed term-based methods and concurrent neural models in reported retrieval tasks.
  • 5.4 Model Learning: Neural semantic retrieval models learn representation and scoring functions with deep learning, organized into sparse, dense, and hybrid paradigms.Document representations are often precomputed and stored in retrieval structures to support fast search.
  • 5.4.2 Negative Sampling Strategies: How to mine negative documents efficiently and effectively remains an open problem because negative sampling strongly affects retrieval-model quality.The survey identifies negative sampling as decisive for learning high-quality retrieval models.

6 CHALLENGES AND FUTURE DIRECTIONS

The survey identifies open challenges in pretraining, supervision, evaluation, and indexing for first-stage semantic retrieval. It emphasizes task-aware learning, better treatment of biased and scarce labels, contextualized efficiency evaluation, and tighter integration between representation learning and indexing.

  • 6 CHALLENGES AND FUTURE DIRECTIONS: Pretraining for first-stage retrieval remains limited, and retrieval-oriented objectives should account for high recall and task-dependent characteristics.A dual-encoder pretraining approach did not outperform BM25 on SQuAD, illustrating the need for retrieval-specific design.
  • 6 CHALLENGES AND FUTURE DIRECTIONS: Pooling-based benchmark construction creates bias because experts judge only a subset of documents while unlabeled documents are often treated as irrelevant.The survey discusses debiased objectives and hard-negative mining as responses to this problem.
  • 6 CHALLENGES AND FUTURE DIRECTIONS: Supervised IR data is scarce and prone to long-tail and sparsity issues, motivating weakly supervised or unsupervised learning strategies.Contrastive learning is cited as one promising direction.
  • 6 CHALLENGES AND FUTURE DIRECTIONS: Efficiency metrics should be reported together with effectiveness measures at corresponding cutoffs, rather than interpreted in isolation.This recommendation reflects the distinct efficiency and effectiveness priorities of retrieval and re-ranking stages.
  • 6 CHALLENGES AND FUTURE DIRECTIONS: Dense retrieval commonly separates representation learning from index construction, leaving the index unable to use supervised information and potentially reducing compatibility.Joint learning of encoders and indexes remains at an early stage for information retrieval.
  • 6 CHALLENGES AND FUTURE DIRECTIONS: Advanced approximate-nearest-neighbor algorithms are needed to improve the balance between retrieval efficiency and precision at large scale.ANN search improves efficiency by sacrificing part of brute-force precision.

7 CONCLUSION

The survey synthesizes semantic models for first-stage retrieval, connecting early and neural approaches while organizing neural methods into sparse, dense, and hybrid paradigms. It also reviews learning issues and identifies challenges and future directions for the field.

  • 7 CONCLUSION: The survey summarizes current semantic-retrieval research, analyzes existing methodologies, and develops insights for future work.Its scope includes early semantic models, neural methods, learning topics, and open challenges.
  • 7 CONCLUSION: Neural semantic retrieval is reviewed through sparse, dense, and hybrid paradigms alongside loss functions and negative sampling strategies.The survey connects these methods with earlier semantic and classical term-based retrieval approaches.
  • 7 CONCLUSION: The survey aims to help researchers learn from past successes and failures and motivate new ideas for first-stage retrieval.It frames semantic retrieval as part of the broader intersection of deep learning and information retrieval.
Loading 2103.04831v4…