PaperlayerView exact revision on arXiv

Source-linked AI summary

Pretrained Transformers for Text Ranking: BERT and Beyond

Jimmy Lin, Rodrigo Nogueira, Andrew Yates

arXiv:2010.06467v3cs.IRcs.CL

TL;DR

Text ranking with transformers has advanced rapidly, but applying these models still raises challenges for long texts and out-of-distribution queries and documents. This survey synthesizes transformer-based reranking and dense retrieval techniques, finding that ranking foundations are sturdy while generalizability and robustness remain open research areas.

  • Problem

    Applying transformers to text ranking is constrained by maximum input lengths, excessive memory use, inference latency, and out-of-distribution generalization.

  • Method

    The survey synthesizes transformer applications to text ranking, including pretrained models, self-supervised learning, reranking architectures, and dense representations.

  • Results

    Transformer-based ranking foundations are sturdy, with substantial, robust, and widely replicated effectiveness improvements from designs such as monoBERT.

  • Takeaways & Limitations

    Transformer architectures and pretraining have significantly advanced the state of the art in text ranking, while model application techniques are increasingly well understood.

  • Takeaways & Limitations

    Dense retrieval techniques trained on MS MARCO data are less effective than BM25 overall when applied zero-shot across diverse domains in BEIR.

Abstract

from arXiv · show

The goal of text ranking is to generate an ordered list of texts retrieved from a corpus in response to a query. Although the most common formulation of text ranking is search, instances of the task can also be found in many natural language processing applications. This survey provides an overview of text ranking with neural network architectures known as transformers, of which BERT is the best-known example. The combination of transformers and self-supervised pretraining has been responsible for a paradigm shift in natural language processing (NLP), information retrieval (IR), and beyond. In this survey, we provide a synthesis of existing work as a single point of entry for practitioners who wish to gain a better understanding of how to apply transformers to text ranking problems and researchers who wish to pursue work in this area. We cover a wide range of modern techniques, grouped into two high-level categories: transformer models that perform reranking in multi-stage architectures and dense retrieval techniques that perform ranking directly. There are two themes that pervade our survey: techniques for handling long documents, beyond typical sentence-by-sentence processing in NLP, and techniques for addressing the tradeoff between effectiveness (i.e., result quality) and efficiency (e.g., query latency, model and index size). Although transformer architectures and pretraining techniques are recent innovations, many aspects of how they are applied to text ranking are relatively well understood and represent mature techniques. However, there remain many open research questions, and thus in addition to laying out the foundations of pretrained transformers for text ranking, this survey also attempts to prognosticate where the field is heading.

1 Introduction

Text ranking produces ordered texts for a query, most commonly in search. The survey examines how transformer models, especially BERT, reshaped ranking quality and practice across NLP and IR.

  • Text ranking orders texts from a corpus by estimated relevance to a user's query and information need.
  • Search commonly presents ranked results with titles, metadata, and query-focused snippets.
  • Transformers such as BERT produced a paradigm shift in NLP, IR, and broader human language technologies.
  • BERT improved text-ranking quality over prior methods across tasks, domains, and problem formulations, with robust replicated evidence.
  • Transformer applications extend beyond search, including machine translation and other language-processing tasks.

1.1 Text Ranking Problems

Text ranking spans search and many downstream NLP applications, but traditional exact-match methods miss semantic matches. The section traces the field's progression to deep neural ranking and BERT's rapid effectiveness gains.

  • Text ranking supports search, question answering, fact verification, entity linking, and downstream information-processing modules.
  • Maron and Kuhns framed information retrieval as producing an ordered list of documents most likely to satisfy user information needs.
  • Exact-match techniques cannot retrieve documents when query and document terms do not match, motivating semantic matching approaches.
  • BERT-based models achieved substantially higher effectiveness than pre-BERT models in TREC 2019 across submissions from different teams.
  • Pre-BERT interaction-based models were generally more effective but slower than representation-based models, which support fast nearest-neighbor search over precomputed vectors.
  • Within less than a week on MS MARCO passage ranking, BERT improved effectiveness by around eight absolute points, approximately a 30% relative gain.
  • The BERT passage-ranking results were rapidly replicated, and BERT's dominance remained evident on later leaderboards.
  • The survey intentionally focuses on core text ranking and omits detailed treatment of question answering, summarization, and recommendation.

2 Setting the Stage

Text ranking evaluates ordered texts against information needs, but both relevance judgments and test collections introduce representational and coverage limitations. Reliable comparisons therefore depend on careful evaluation resources, while transformer ranking also faces practical data and domain constraints.

  • Information needs and relevance: Text ranking orders corpus texts by estimated relevance to a query representing a user information need.The survey treats search as the most common formulation, while noting that text ranking appears in broader information-access applications.
  • Information needs and relevance: Narratives may describe an information need more faithfully yet harm exact-match ranking by adding terms that distract the model.The passage notes that outcomes vary by domain and topic set.
  • Evaluation: 60% overlap is a commonly cited level of assessor agreement, with reported values ranging from around 30% to greater than 70%.Agreement depends on study design, information needs, and the agreement metric.
  • Information needs and relevance: Relevance judgments are imperfect because assessors interpret external topic representations and may judge similar texts inconsistently.Topic statements can omit details or contain inconsistencies, while assessor interpretations can vary across texts.
  • Evaluation: System comparisons are generally reliable with well-constructed test collections even though absolute metric values require cautious interpretation.The survey distinguishes comparative reliability from the inability to quantify absolute effectiveness precisely.
  • Evaluation: Test collections are costly to create but can remain reusable for years, while older collections may underrepresent innovations unlike the techniques used to build their judgment pools.The survey cites TREC as an organizational structure supporting large-scale evaluation and notes that Robust04 remains widely used.

3 Multi-Stage Architectures for Reranking

This section presents relevance classification as a basic BERT ranking design, then situates it within reranking pipelines and extensions that address effectiveness, efficiency, and long-document limitations.

  • Relevance classification ranks texts by estimating each text’s probability of belonging to the relevant class.This directly realizes the Probability Ranking Principle by sorting texts in decreasing estimated relevance probability.
  • The survey begins its BERT ranking discussion with monoBERT, based on the relevance-classification design introduced by Nogueira and Cho.It then examines contrastive and ablation experiments, model variants, and how BERT works.
  • Applying pretrained transformers requires fine-tuning a pretrained model with labeled data from the target task, with task-specific techniques affecting the process.
  • BERT’s inability to handle long input sequences creates difficulty for ranking full-length documents such as news articles, scientific papers, and web pages.The survey presents multiple approaches for overcoming this limitation.
  • Multi-stage ranking architectures extend retrieve-and-rerank systems with reranker pipelines, while later work explores models that improve speed, effectiveness, or their tradeoff.The survey also covers knowledge distillation, compact student models, redesigned transformer architectures, and pretrained sequence-to-sequence adaptations.
  • Binary relevance is an oversimplification, and ordinal relevance modeling improves representation while still leaving information need satisfaction multifaceted.

3.1 A High-Level Overview of BERT

BERT is an encoder-only transformer that converts token sequences into contextual embeddings and is pretrained with self-supervision before task-specific fine-tuning. Its masked language modeling objective and bidirectional context distinguish its representations, while its input format supports classification and ranking tasks.

  • BERT generates contextual embeddings for input sequences, unlike static representations such as word2vec and GloVe.A multilingual variant, mBERT, can process more than 100 languages, although this survey focuses on English BERT.
  • BERT represents each input token with token, segment, and position information, then outputs a corresponding contextual embedding sequence.The [CLS] representation is typically used for classification, while [CLS] and [SEP] structure BERT inputs; ranking inputs pair query and candidate text sequences.
  • BERT is an encoder-only transformer, contrasting with GPT’s decoder-only architecture and the full sequence-to-sequence transformer design.
  • Self-supervised pretraining removes dependence on externally labeled data during pretraining and provides task-independent starting points for fine-tuning.Because texts provide their own training signals, substantially more data can be used, and modest labeled datasets can support effective fine-tuning.
  • BERT’s masked language model objective randomly masks input tokens and trains the model to predict them with cross entropy loss.Unlike autoregressive modeling, BERT’s bidirectional modeling uses context from both directions and is associated with higher GLUE effectiveness.
  • BERT tokenizes inputs into subwords, commonly with WordPiece, to reduce vocabulary size.WordPiece splits words such as “scrolling” into “scroll” and “##ing,” although subword boundaries need not correspond to linguistic units.

3.2 Simple Relevance Classification: monoBERT

monoBERT can rank effectively despite pointwise training and depends on sufficient data, suitable candidate depth, and carefully structured inputs. Its performance reflects both first-stage retrieval quality and transformer relevance signals.

  • Training objective: Pointwise cross-entropy training does not directly optimize ranking metrics such as MAP, a phenomenon called metric divergence.In practice, more accurate relevance classification generally improves ranking metrics despite this mismatch.
  • Training objective: Even without explicitly addressing these deficiencies, monoBERT works well as a ranking model in practice.
  • Candidate generation: Reranking quality depends on first-stage retrieval: improved BM25 results produce more effective reranked results, while reranking does not change recall.
  • Training data: monoBERT is data hungry: with 20K training instances, it barely improves upon BM25, and with few examples it can perform worse.Effectiveness improves as monoBERT is fine-tuned with more data.
  • Candidate depth: Increasing the candidate count from 1000 to 10000 raises MRR@10 from 0.372 to 0.377, while 50000 yields the same value.Latency increases linearly with the number of processed candidates, so deployment should use the largest practical k.
  • Input representation: Removing position embeddings drops MRR@10 by nearly six points, whereas removing segment embeddings causes only a small loss.These ablations indicate that token positions are important and separator tokens can help distinguish query and document tokens.
  • Input representation: The organization of the input template is important, including the choice of delimiters separating query and document text.
  • Model variants: BERT-like replacements can be swapped into ranking models, but each variant requires empirical validation because NLP improvements may not transfer to information access.

3.3 From Passage to Document Ranking

Passage-to-document ranking methods adapt transformers to long texts by selecting or aggregating sentence and passage representations, while balancing effectiveness against input-length and efficiency constraints. Experiments show strong cross-domain transfer, benefits from staged fine-tuning, and unresolved differences in how traditional retrieval scores contribute.

  • Handling Long Documents: BERT’s quadratic time and space complexity makes directly processing long documents impractical, motivating passage- and sentence-level strategies.The limitation arises from all-to-all attention at every encoder layer, so additional hardware is not a practical solution.
  • Document Ranking with Sentences: Birch transfers relevance knowledge from tweet data to news articles, with significant gains over BM25 + RM3 on Robust04 and MAP gains on Core17 and Core18.The transfer occurs despite substantial differences between tweet and newswire domains.
  • Document Ranking with Sentences: Birch performs best when MS MARCO pre-fine-tuning is followed by tweet-data fine-tuning, outperforming either dataset alone.The top-ranked sentence is generally a strong relevance indicator; adding more sentences yields at most minor gains and can reduce effectiveness.
  • Document Ranking with Sentences: BERT captures semantically relevant sentence–query matches even without exact query terms, providing qualitative evidence for semantic relevance modeling.For “international art crime,” a relevant sentence about stolen Renaissance paintings contained no query terms but was identified as most relevant.
  • Effectiveness: Birch-based models achieved effectiveness that surpassed pre-BERT neural models and at least matched the best non-neural techniques known at publication.Later models such as PARADE and monoT5 subsequently exceeded these results.
  • Experimental Interpretation: Comparisons across Birch–Passage conditions indicate that improvements cannot be cleanly attributed to individual architectural changes because the experimental designs differ.Birch–Passage varies in performance relative to the original Birch model across BERTBase and ELECTRA Base settings, query types, and interpolation conditions.

3.4 From Single-Stage to Multi-Stage Rerankers

Transformer reranking extends retrieve-and-rerank systems from single-stage models to multi-stage pipelines that offer more effectiveness–efficiency operating points. These approaches improve flexibility but remain bounded by input-length and recall assumptions.

  • Pairwise Reranking: Pairwise aggregation combines candidate-level pairwise scores into one score per document before producing the final ranked list.The SUM, BINARY, MIN, and MAX methods differ in how each candidate is compared with the others.
  • Multi-Stage Architectures: Two-stage pipelines expand the design space by jointly tuning the initial retrieval depth k0 and second-stage depth k1.This enables operating points unavailable to a single monoBERT reranking stage.
  • Effectiveness–Efficiency Tradeoffs: Beyond a few hundred BERT inferences, reallocating computation to a pairwise second stage can outperform simply increasing single-stage reranking depth.The comparison is framed in terms of effectiveness under a fixed computational budget.
  • Multi-Stage Architectures: Multi-stage ranking generalizes retrieve-and-rerank by applying multiple rerankers after initial candidate retrieval.The survey contrasts monoBERT with pipelines such as monoBERT plus duoBERT and cascade transformers.
  • Listwise Reranking: Listwise rerankers estimate candidate relevance jointly, but feeding multiple texts into each inference pass creates long-sequence challenges.Candidate-text length limitations make efficient listwise inference technically difficult.
  • Cascade Transformers: Cascade transformers can reduce inference cost substantially with modest effectiveness decreases, including about half on ASNQ and 37% on GPD.The reported reductions depend on the α setting and dataset.

3.5 Beyond BERT

Beyond standard BERT rerankers, researchers explore distillation, ranking-specific transformer architectures, and sequence-to-sequence models. These alternatives target efficiency, data efficiency, or new relevance-modeling strategies, but their benefits involve tradeoffs and unresolved questions.

  • Knowledge Distillation: Knowledge distillation transfers a larger teacher’s behavior to a smaller student to reduce latency or parameters while retaining task effectiveness.The survey treats distillation as a general-purpose approach for controlling neural ranking tradeoffs.
  • Knowledge Distillation: A six-layer distilled model can slightly exceed its teacher while running twice as fast, whereas a four-layer model achieves a 9× speedup with a small effectiveness decrease.The tradeoff depends on the student model size and distillation procedure.
  • Knowledge Distillation: Distilled models consistently outperform directly trained smaller models, and ranker distillation can remove about a quarter of PARADE’s parameters while reducing latency by about a third.The reported result indicates efficiency gains without significant effectiveness harm in the cited experiments.
  • TK, TKL, and CK: Transformer Kernel models redesign transformer architectures specifically for ranking and are less effective but more efficient than monoBERT.TK is more effective than earlier pre-BERT interaction models and improves on them across latency budgets.
  • Open Questions: It remains unclear whether TK, TKL, and CK can benefit from BERT-style self-supervised pretraining, leaving the roles of pretraining and architecture unresolved.The survey identifies this as an open comparison between representation quality and relevance-focused architecture.
  • monoT5: monoT5-base surpasses monoBERT-large while using 220M versus 340M parameters and running approximately twice as fast at inference.Scaling to monoT5-3B adds 1.6 effectiveness points but makes the model approximately 14 times larger and 10 times slower than monoT5-base.
  • monoT5: With only 1K positive query–passage instances, monoT5 exceeds BM25, while monoBERT requires around 10K positive instances to beat BM25.The survey interprets this pattern as evidence that monoT5 extracts more value from limited training data.

3.6 Concluding Thoughts

Transformer ranking has developed from monoBERT reranking into a diverse design space spanning input-length solutions, ranking-specific architectures, larger pretrained models, and generative approaches. Direct dense ranking and additional future directions remain open areas beyond the surveyed reranking frontier.

  • Reranking Progression: The reranking progression moved from monoBERT relevance classification toward methods addressing input-length limits, BERT variants, and increasingly sophisticated architectures.The survey places these developments within a broader evolution of transformer ranking.
  • Design Diversity: TK, TKL, and CK rethink transformer architectures from the ground up, while other approaches emphasize more pretraining, larger models, or generative ranking.These branches illustrate the diversity of current transformer-ranking designs.
  • Future Directions: Direct ranking with learned dense representations is an emerging area, while further techniques for enriching query and document representations remain under development.The survey identifies both topics as directions beyond the immediate reranking discussion.

4 Refining Query and Document Representations

This section examines transformer-based query and document expansion methods for mitigating vocabulary mismatch and improving first-stage retrieval. The survey finds that gains depend on the expansion method and must be weighed against downstream effectiveness and computational cost.

  • Motivation: Vocabulary mismatch can hide relevant documents when they omit query terms, motivating richer query and document representations.Increasing first-stage candidate depth can recover such documents, but only deeper in the ranked list.
  • Document expansion: Document expansion can replace first-stage retrieval with an expanded index, improving effectiveness or preserving effectiveness at lower reranking latency.doc2query remains keyword-based and is about 50× faster than monoBERT, although its effectiveness is substantially lower.
  • Query expansion: Query expansion methods can feed augmented queries to transformer rerankers, but existing methods may produce unnatural language that monoBERT handles poorly.The survey reports this straightforward combination as ineffective.
  • Query expansion: CEQE improves over static-embedding expansion, while its gains over BM25 + RM3 are smaller and require costly BERT inference over top-k candidates.Among CEQE variants, max pooling is consistently most effective, but fine-tuning the reranker does not necessarily improve expansion-term selection.
  • Effectiveness and efficiency: Contextual expansion methods can improve retrieval, but their end-to-end value remains uncertain when computational costs and downstream reranking effects are included.The survey notes that no lightweight yet effective transformer method for pseudo-relevance feedback had been established.
  • Combined expansion: doc2query–T5 and query expansion provide additive improvements, with doc2query–T5 plus BM25 + RM3 achieving the highest effectiveness.Predicted query terms both copy source terms and add new terms; excluding stopwords, 69% are copied and 31% are new.

5 Learned Dense Representations for Ranking

This section introduces learned dense representations as a direct alternative to sparse first-stage retrieval and surveys transformer-based approaches for ranking. Dense retrieval aims to encode queries and documents as vectors whose similarity reflects relevance, while addressing efficiency and interaction tradeoffs.

  • Overview: Dense retrieval moves text ranking from sparse exact-match signals toward continuous representations intended to capture semantic matches.Ranking is performed directly on vector representations generated by neural encoders.
  • Task formulation: The survey connects dense retrieval with sentence similarity, paraphrase, and entailment because these tasks all involve ranking texts according to relationships between their meanings.This framing brings relevance and semantic-similarity research closer together.
  • Task formulation: Unlike multi-stage architectures, dense retrieval learns query and document transformations that maximize similarity for relevant texts and minimize it for non-relevant texts.The transformations map token sequences to fixed-width vectors.
  • Efficiency: Dense retrieval supports direct top-k ranking through nearest neighbor search when the comparison function uses simple operations such as inner products.Efficient scalable implementations are available in open-source libraries.
  • Motivation: Multi-stage ranking is difficult to train end-to-end because sparse retrieval and dense reranking stages are optimized separately.First-stage retrieval is typically optimized for recall, while later models use dense representations.
  • Transformer encoders: Fine-tuned BERT can model relationships such as entailment, paraphrase, and query–document relevance when trained with task-appropriate annotated data.The survey cites strong empirical evidence that BERT excels across these relationship-estimation tasks.

5.2 Nearest Neighbor Search

Nearest-neighbor dense retrieval offers scalable vector ranking, but the surveyed evidence shows a persistent effectiveness–efficiency tradeoff. Bi-encoders are generally less effective than cross-encoder rerankers, while training improvements and hybrid methods can narrow practical gaps.

  • Core tradeoff: Bi-encoders are less effective than comparable cross-encoders but substantially more efficient because ranking reduces to simple vector operations.This efficiency makes dense retrieval attractive despite weaker interaction modeling.
  • Training: Bi-encoder effectiveness depends strongly on negative-example selection, motivating DPR’s in-batch and BM25 negatives and ANCE’s approximate-nearest-neighbor negatives.ANCE appears more effective than the DPR training scheme and its ablation conditions.
  • Evaluation: Replication studies reproduced DPR’s dense-retrieval results closely but reported a substantive contrary finding about the value of hybrid techniques.The replicated scores were generally within a tenth of a point of the originals.
  • Evaluation: Across passage and document ranking, dense bi-encoders remain less effective than reranking sparse results with cross-encoders.The comparisons also show that MaxP is more effective than FirstP.
  • Efficiency: ANCE’s synchronous ANN index refresh supplies informative negatives but adds substantial training cost, taking approximately 10 hours for the MS MARCO document collection on a multi-GPU server.The reported gains therefore come with slower training rather than a free efficiency improvement.

5.5 Enhanced Transformer Bi-encoders for Ranking

Enhanced bi-encoders introduce richer interactions or multiple representations to improve dense retrieval beyond single-vector comparisons. These designs can approach cross-encoder effectiveness, but they incur costs in latency, index size, or retrieval complexity.

  • Design space: Simple bi-encoders are fast but constrained by nearest-neighbor search, which limits the complexity of their query–document comparison function.Enhancing the comparison function creates a direct effectiveness–efficiency design problem.
  • Hybrid architectures: PreTTR masks query–document attention in lower BERT layers so corpus representations can be precomputed, while upper layers retain a relevance-scoring role.This creates a hybrid between bi-encoder and cross-encoder designs.
  • Multi-vector representations: Poly-encoders were more effective than bi-encoders and more efficient than cross-encoders in fixed-candidate latency evaluations.They were not evaluated end-to-end with nearest-neighbor search, limiting the scope of that efficiency comparison.
  • Multi-vector representations: Multi-representation models such as ME-BERT outperform single-representation DE-BERT, but require greater query latency and larger indexes.Dense–sparse hybrids further improve effectiveness over dense retrieval alone in the reported comparisons.
  • Late interaction: ColBERT uses per-token representations and MaxSim to preserve rich token-level interactions while remaining compatible with efficient nearest-neighbor retrieval.Its approach replaces inverted-index first-stage retrieval with nearest-neighbor search followed by lightweight MaxSim reranking.
  • Late interaction: ColBERT approaches cross-encoder effectiveness at a fraction of monoBERT query latency, although it remains slower than pre-BERT neural models and doc2query.The survey characterizes it as a compelling point in the effectiveness–efficiency tradeoff space.
  • Index cost: Storing ColBERT’s per-token vectors can require substantial space: an illustrative MS MARCO passage index would need approximately 225 GB.The estimate assumes 8.8M passages, 50 tokens per passage, 128 dimensions, and 4 bytes per dimension.

5.6 Knowledge Distillation for Transformer Bi-encoders

Knowledge distillation transfers ranking knowledge from more effective but slower models into efficient bi-encoders. The surveyed evidence indicates that this approach can improve bi-encoder effectiveness, while training efficiency and model-size tradeoffs remain central considerations.

  • Motivation: Distillation is attractive for reranking because smaller models reduce memory requirements and inference latency across many candidate texts.For dense retrieval, its value also includes transferring the behavior of more powerful cross-encoders into bi-encoders.
  • Distillation framework: A three-step procedure fine-tunes a teacher, caches teacher scores for training triples, and distills those scores into a DistilBERT bi-encoder.The procedure can also support ensembles of multiple teacher models.
  • Distillation framework: Margin-MSE trains the student to match the teacher’s score margin between relevant and non-relevant texts for a query.For a triple (q, d+, d−), the loss compares student and teacher score differences using mean squared error.
  • Empirical results: Distillation improved DistilBERT bi-encoder effectiveness on the MS MARCO passage-ranking development set compared with the same model trained without distillation.The reported condition used Margin-MSE, a DistilBERT student, and an ensemble of three cross-encoders.
  • Empirical results: TCT-ColBERT used an efficient ColBERT teacher with in-batch negatives and achieved bi-encoder effectiveness on par with ANCE.Hard-negative mining added further improvements, while hybridization with BM25 or doc2query–T5 provided additional benefits.
  • Training efficiency: Randomly sampled training batches often contain low-information examples, motivating training procedures that assemble more informative samples for distillation.TAS-B used a teacher ensemble combining a cross-encoder and ColBERT with a DistilBERT student.
  • Distillation framework: Distillation transfers knowledge from slower, more effective cross-encoders or ColBERT models into bi-encoder students.The student is trained from teacher-derived scores rather than directly optimizing only the original relevance labels.

5.7 Concluding Thoughts

Dense retrieval with learned transformer representations is a significant development, but it remains complementary to sparse retrieval and constrained by generalization, index size, and document-length challenges. The survey identifies substantial unexplored design space alongside these practical limitations.

  • Open challenges: Dense retrieval is an active research area, but its techniques still face unresolved challenges in generalization, efficiency, and long-document processing.The section frames these issues as continuing challenges despite progress in learned dense representations.
  • Generalization: Zero-shot evaluations show that dense retrieval can be ineffective when applied to texts from different domains or query types.BEIR organized diverse datasets and provided evidence for this out-of-distribution weakness.
  • Efficiency: A minimal Lucene index for 8.8M MS MARCO passages occupies 661 MB, whereas a comparable 768-dimensional Faiss HNSW index occupies 42 GB.The comparison highlights index size as an important efficiency dimension for dense retrieval.
  • Long documents: Most dense retrieval models process short segments rather than full documents, commonly encoding document segments independently.The survey notes that this approach does not generally integrate evidence from multiple document parts.
  • Long documents: Dense retrieval techniques for full-length documents generally lack methods that integrate evidence from multiple document parts while remaining compatible with nearest-neighbor search infrastructure.The survey identifies only a possible exception not designed for ad hoc retrieval.
  • Future exploration: The design space remains largely unexplored, including alternative multi-vector representations and broader applications of topic-aware sampling.The survey emphasizes that these techniques can be mixed and matched in combinations not yet examined.
  • Hybrid retrieval: Dense–sparse hybrids provide complementary relevance signals and appear more effective than either representation alone.Dense retrieval also motivates approximate nearest-neighbor methods such as HNSW because sparse inverted indexes are not designed for dense-vector search.
  • Hybrid retrieval: Sparse retrieval and inverted indexes are unlikely to be completely supplanted by dense retrieval techniques.The survey notes production deployments of dense retrieval while retaining a role for sparse methods.

6 Future Directions and Conclusions

The survey concludes that transformer-based ranking has produced substantial, replicated advances while leaving important questions about architecture, multi-stage integration, generalization, and interactive access. Future systems are likely to combine sparse and dense signals rather than rely on dense retrieval alone.

  • Conclusions: Transformer-based ranking has produced substantial, robust, and widely replicated effectiveness improvements across many tasks.The survey presents these advances as notable given that BERT debuted only around three years earlier.
  • Multi-stage ranking: The best dense retrieval techniques remain less effective than the best reranking architectures because retrieved outputs can be reranked to improve effectiveness.Dense retrieval can replace sparse first-stage retrieval, while reranking remains a later stage in the architecture.
  • Multi-stage ranking: Dense representations can serve both first-stage retrieval and reranking, enabling multi-stage systems that combine sparse and dense representations.The survey cites several reranking examples using dense representations.
  • Open research questions: Distribution mismatches between training representations and representations encountered in multi-stage systems remain an open research problem.The issue arises when models trained on isolated text distributions are inserted into multi-stage architectures.
  • Open research questions: Out-of-distribution performance remains a concern, motivating domain adaptation, transfer learning, and zero-shot or few-shot approaches.The passage states that trained models may not perform well on out-of-distribution data.
  • Limited-data evaluation: Hybrid ensembles, fusion methods, and synthetic in-domain data generation produced strong TREC-COVID results, including top-scoring runs.The reported systems combined multiple retrieval signals, document representations, or generated training pairs.
  • Limited-data evaluation: A T5-3B document-expansion and monoT5/duoT5 reranking pipeline achieved the best automatic runs in TREC-COVID rounds 4 and 5 in a zero-shot setting.The models were trained only on MS MARCO passage data.
  • Limited-data evaluation: TREC-COVID results showed that transformer-based automatic runs could outperform non-transformer runs by large margins, while transformer feedback runs often barely exceeded non-transformer competitors.Simple relevance feedback remained competitive in later rounds.

Version History

The survey’s later versions reorganized and expanded coverage of query and document representations and learned dense retrieval, while undergoing substantial copy editing and adding recent references.

  • Substantive revisions: Version 0.95 introduced a new section on refining query and document representations, including query expansion techniques.It also rewrote the learned dense representations section to incorporate new developments in dense retrieval.
  • Substantive revisions: Version 0.95 reduced the emphasis on domain-specific applications as a standalone subsection and interwove that material throughout the survey.The revision also reorganized related content across the document.
  • Editorial history: Major changes were marked with a custom LaTeX command, but unmarked text was not necessarily unchanged because the entire survey underwent copy editing.The marking convention identified only changes considered substantially different by the authors.
  • Editorial history: The final preproduction version contained no major content changes from version 0.95, adding only a few recent references and further copy editing.The major-change annotations remained largely unchanged from version 0.95.
Loading 2010.06467v3…