PaperlayerView exact revision on arXiv

Source-linked AI summary

Utilizing BERT for Information Retrieval: Survey, Applications, Resources, and Challenges

Jiajia Wang, Jimmy X. Huang, Xinhui Tu, Junmei Wang, Angela J. Huang, Md Tahmid Rahman Laskar, Amran Bhuiyan

arXiv:2403.00784v1cs.IRcs.AIcs.CL

TL;DR

BERT-based information-retrieval research is surveyed against challenges that include semantic understanding and the cost of manual feature engineering. The survey organizes approaches and resources; its reviewed state-of-the-art methods achieve around 50% MRR@10, leaving room for improvement.

  • Problem

    IR models face challenges in accurately understanding semantic information in queries and documents, while manual feature engineering requires domain expertise and incurs significant cost.

    Source: §1 · p. 2

  • Method

    The survey organizes BERT-based ad-hoc IR approaches into categories and presents challenges associated with their use.

    Source: §1 · p. 4, §3 · p. 8

  • Results

    Around 50% MRR@10 was the accuracy of state-of-the-art methods across different datasets.

    Source: §5 · p. 27

  • Takeaways & Limitations

    The reported accuracy indicates considerable room for improvement.

    Source: §5 · p. 27

    LLMs require significant computational resources for training and inference, and cannot answer time-sensitive questions beyond their pretraining timeline.

    Source: §1 · p. 3

Abstract

from arXiv · show

Recent years have witnessed a substantial increase in the use of deep learning to solve various natural language processing (NLP) problems. Early deep learning models were constrained by their sequential or unidirectional nature, such that they struggled to capture the contextual relationships across text inputs. The introduction of bidirectional encoder representations from transformers (BERT) leads to a robust encoder for the transformer model that can understand the broader context and deliver state-of-the-art performance across various NLP tasks. This has inspired researchers and practitioners to apply BERT to practical problems, such as information retrieval (IR). A survey that focuses on a comprehensive analysis of prevalent approaches that apply pretrained transformer encoders like BERT to IR can thus be useful for academia and the industry. In light of this, we revisit a variety of BERT-based methods in this survey, cover a wide range of techniques of IR, and group them into six high-level categories: (i) handling long documents, (ii) integrating semantic information, (iii) balancing effectiveness and efficiency, (iv) predicting the weights of terms, (v) query expansion, and (vi) document expansion. We also provide links to resources, including datasets and toolkits, for BERT-based IR systems. A key highlight of our survey is the comparison between BERT's encoder-based models and the latest generative Large Language Models (LLMs), such as ChatGPT, which rely on decoders. Despite the popularity of LLMs, we find that for specific tasks, finely tuned BERT encoders still outperform, and at a lower deployment cost. Finally, we summarize the comprehensive outcomes of the survey and suggest directions for future research in the area.

1 INTRODUCTION AND MOTIVATION

The survey brings BERT-based information-retrieval research into a coherent resource, focusing on pretrained transformer encoders and reviewing challenges in applying them to IR.

1 INTRODUCTION AND MOTIVATION

  • The survey targets IR approaches using pretrained transformer encoders such as BERT and reviews their challenges.It aims to provide researchers and industry professionals with a comprehensive overview.Source: §1 · p. 3
  • It logically organizes updated BERT-for-ad-hoc-IR literature to help readers understand models from multiple perspectives and discover research opportunities.The survey’s structure is intended to support step-by-step understanding.Source: §1 · p. 4
  • The survey analyzes BERT-based models’ advantages and disadvantages and compares state-of-the-art approaches across categories.It organizes ad-hoc IR approaches into categories, including long-document handling, semantic integration, efficiency, term-weight prediction, and query and document expansion.Source: §1 · p. 4
  • It presents challenges to using BERT for IR, examines IR toolkits, and summarizes survey outcomes with future research directions.The survey also provides resources, including datasets and BERT-based ranking models.Source: §1 · p. 4

2 BACKGROUND

This section frames relevance ranking as central to information retrieval and traces retrieval methods from traditional and neural models to pretrained encoders and their extensions. It also surveys extensions to pretrained language models, contrasts BERT-based approaches with unified text-to-text and decoder-based models, and describes retrieval-augmented generation as a way to bring external information into LLM responses.

2.1 Traditional Retrieval Models

  • Traditional retrieval includes Boolean, vector-space, language, and probability models; Boolean retrieval lacks ranking, while vector-space methods cannot resolve polysemy or synonymy.The first three categories treat queries and documents as word sets and ignore token positions.Source: §2.1 · p. 4
  • BM25 weights search terms using intra-document and query term frequencies, while language models estimate document relevance probabilistically.Source: §2.1 · pp. 4–5

2.2 Neural Ranking Models

  • Neural ranking models learn query–document relevance representations; representation-focused models encode texts separately, whereas interaction-focused models compare them to capture more contextual information.Traditional exact-match scoring can limit positional and semantic information and cause vocabulary mismatch.Source: §2.2 · p. 5

2.3 Pretrained Language Models

  • Pretrained encoders such as BERT support parallel sequence processing and task-specific fine-tuning, unlike earlier models that retain fixed word-vector representations.BERT uses a bidirectional transformer encoder and auto-encoding pretraining with masked language modeling and next-sentence prediction, unlike GPT’s decoder-based autoregressive approach.Source: §2.3 · pp. 5–6

2.4 Improvements to and Extensions of Pretrained Language Models

  • XLNet predicts tokens in random order with a permutation language model and two-stream self-attention to address BERT’s pretraining–fine-tuning mismatch from [MASK].It is an autoregressive model based on Transformer-XL and factorizes the joint probability of predicted tokens using the product rule.Source: §2.4 · p. 6
  • RoBERTa [141] achieves state-of-the-art performance on multiple tasks while retaining BERT’s architecture and changing pretraining, including dynamic rather than static masking.Static masking is applied once during data processing; dynamic masking changes the masked pattern whenever a sequence is fed to the model.Source: §2.4 · p. 6
  • As pretrained models grow, memory constraints, longer preparation, and unexpectedly poor performance motivate ALBERT [151]’s word-vector decomposition and cross-layer parameter sharing.Source: §2.4 · p. 6
  • ALBERT replaces next-sentence prediction with sentence-order prediction to assess sentence coherence, while ELECTRA combines BERT with a generator–discriminator structure.Source: §2.4 · p. 7
  • Some pretrained-model extensions outperform BERT with fewer parameters and less data while achieving results comparable to the state-of-the-art model.Source: §2.4 · p. 7
  • MacBERT masks similar words to reduce pretraining–fine-tuning mismatch; biomedical and clinical BERT pretraining has achieved state-of-the-art performance across biomedical tasks [43].ALBERT replaces BERT’s next-sentence prediction task with sentence-order prediction.Source: §2.4 · p. 7
  • TK [105] [106] independently encodes queries and documents with up to three transformer layers before kernel pooling, while TKL uses local self-attention over document-term windows.These designs adapt KNRM to balance retrieval effectiveness and efficiency.Source: §2.4 · p. 7
  • T5 [25] converts NLP tasks into text-to-text tasks, allowing one model, objective, training process, and decoding process to serve different tasks.The passage notes that T5 uses more hyperparameters and training data than BERT.Source: §2.4 · p. 7
  • Decoder-based LLMs can address tasks without task-specific fine-tuning and generate human-like responses, but their inference costs and pretraining-time knowledge limit real-world use.The passage describes these models as pretrained on self-supervised data and aligned using techniques such as reinforcement learning based on human feedback.Source: §2.4 · p. 7
  • Retrieval-augmented generation [89] addresses LLM knowledge limits by retrieving relevant external documents for a query and adding them to context; BERT embeddings support vector-similarity retrieval.Source: §2.4 · p. 7
  • The survey calls RoBERTa [141], ALBERT [151], ELECTRA, MacBERT, TK [105] [106], and T5 [25] BERT-based variants because they modify the original BERT model [44] with adjustable objectives.Source: §2.4 · p. 8

3 BERT FOR AD-HOC IR AND CHALLENGES

This section surveys BERT-based methods for ad-hoc information retrieval and their challenges, covering long-document handling, semantic integration, ranking and efficiency, contextual term weighting, and query and document expansion.

3.1 Preliminary Overview of BERT

  • BERT learns context from both left and right sides, but its input limit of 512 tokens constrains its ability to process long documents.The survey describes BERT as a transformer encoder pretrained to predict masked tokens from their surrounding context.Source: §3.1 · pp. 8–9, §3.2 · p. 9

3.2 Utilizing BERT for Handling Long Documents

  • Long-document approaches use aggregation-guided models or block selection; aggregation may lose important information and increase time, memory, and energy costs.Aggregation-guided methods combine scores from smaller segments, while block selection forms a shorter document for BERT processing.Source: §3.2 · p. 9
  • KeyBLD selects relevant blocks from long documents and aggregates a few into a shorter BERT input; experiments report that it outperforms relevant state-of-the-art approaches.Unlike aggregation-guided methods, KeyBLD uses local query–block preranking to select key blocks.Source: §3.2 · p. 10

3.3 Utilizing BERT for Integrating Semantic Information

  • Aggregation methods include sentence-level score, passage-level score, and passage-level representation aggregation; sentence-level approaches combine top-k BERT scores with traditional document scores [132].The survey groups these approaches as methods for integrating semantic information while handling long documents.Source: §3.3 · pp. 10–11
  • On TREC Microblog, BERT outperformed previously developed neural ranking models and demonstrated suitability for passage ranking [132].The studies also evaluated Robust04, where documents were split into sentences and top-three sentence scores were aggregated with the original document score.Source: §3.3 · p. 11
  • Birch (MS →MB) achieved the best results, outperforming Birch fine-tuned on MS MARCO after sequential fine-tuning on MS MARCO and TREC Microblog.The reported comparison suggests that fine-tuning across datasets can better fit the target dataset.Source: §3.3 · p. 11
  • Long-document retrieval can aggregate passage scores [108] or passage-level signals [23], selecting first or maximum scores, or aggregating passage representations through CNNs or a transformer.The strategy may pose time, memory, and energy-consumption problems, and irrelevant query passages may introduce noise to the final representation.Source: §3.3 · pp. 11–12
  • BERT-like models represent documents better than state-of-the-art non-BERT approaches for ad-hoc document ranking.The survey cites multiple studies supporting this comparison.Source: §3.3 · p. 12
  • On Robust04 and WebTrack 2012–14, CEDR models using fine-tuned BERT representations outperformed the compared neural ranking models [112].CEDR combines BERT embeddings with neural rankers; incorporating the [CLS] vector was effective in dense calculations.Source: §3.3 · p. 13
  • Interaction-focused BERT models were dataset-dependent: BERTLast-Int led on MS MARCO, whereas BERTMult-Int led on ClueWeb09-B for NDCG@20 and ERR@20 [89].BERTMult-Int had the worst MRR@10 among the three models on MS MARCO.Source: §3.3 · p. 13
  • Multi-stage ranking produced more accurate lists than single-stage ranking by modeling relationships between documents, while duoBERT incurred higher computational cost [100].The framework reranks an initial candidate list with monoBERT, then compares document pairs with duoBERT.Source: §3.3 · p. 14
  • Fine-tuning BERT on Bing user clicks improved TREC Web Tracks accuracy by 16% over using masked-language modeling alone.Source: §3.3 · p. 14
  • Weakly supervised BERT studies include ReInfoSelect [65], which uses policy gradients to select useful a-d pairs, and weakly supervised passage ranking via BERT [93].Source: §3.3 · p. 15
  • On MS MARCO, Anserini’s BM25 candidate list improved 12% over Microsoft’s BM25 implementation, affecting the final retrieval comparison between monoBERT and Nogueira et al.’s BERT models [99].The survey notes that monoBERT uses BERT as a binary relevance classifier.Source: Table 2 · p. 16, §3.3 · p. 16
  • On the development set, monoBERT + duoBERT_SUM achieved the best performance with TCP, and monoBERT + duoBERT_BINARY and monoBERT + duoBERT_SUM outperformed monoBERT + duoBERT_MIN.BERTBase outperformed BERTLast-Int, BERTMulti-Int, and BERTTerm-Trans. On TREC CAR, monoBERT + duoBERT_BINARY and monoBERT + duoBERT_SUM delivered the best performance, while monoBERT + duoBERT_MAX performed worst, about 3% lower than BERTLarge.Source: §3.3 · pp. 16–17
  • On Robust04, CEDR-KNRM had the best NDCG@20 among CEDR variants, while PARADE-CNN and PARADE-Transformer outperformed them, supporting hierarchical aggregation for document ranking.CEDR-PACRR performed slightly worse than CEDR-DRMM.Source: §3.3 · p. 17

3.4 BERT for Balancing Effectiveness and Efficiency

  • Because BERTBase has 110 million parameters and BERTLarge has 340 million, the survey groups efficiency strategies into dual encoders that reduce query-document interaction and distillation that reduces model parameters.Dual encoders map queries and documents to separate low-dimensional embeddings; the section also covers effective training strategies.Source: §3.4 · p. 17
  • ColBERT outperformed BERT (Rep) by independently encoding queries and documents, then aggregating maximum token-pair similarities through late interaction, reducing complexity from n^2 to n.Documents can be indexed offline, reducing time cost.Source: §3.4 · pp. 17–18, Figure 8 · p. 18
  • Efficiency methods use distillation, adaptive inference, parameter sharing, and cached representations to reduce model size, computation, or inference time.TinyBERT transfers BERT knowledge through transformer and task-specific distillation; DistilBERT [125] is a lightweight general-purpose pretrained model, and TwinBERT precomputes document representations for offline caching.Source: §3.4 · pp. 18–19
  • Pretrained BERT models outperform sparse retrieval models such as BM25, ColBERT leads dual-encoder models, and TSA-Balanced surpasses TSA and ANCE.The survey attributes the training-strategy gains to careful negative-sample selection, which helps distinguish positive from negative instances.Source: Table 3 · p. 20, §3.4 · p. 20
  • On MS MARCO and TREC DL, ANCE outperforms random negative sampling, while TSA-Balanced outperforms both TSA and random sampling.TSA clusters queries with K-means; TSA-Balanced samples negatives within a batch using the maximum margin.Source: §3.4 · p. 20

3.5 BERT-based Predictions of Term Weights

  • HDCT represents each passage token with BERT embeddings and uses a linear layer to assign real-valued weights; experiments found significantly better performance than term frequency.Source: §3.5 · p. 21
  • DeepImpact [9] produces independent term weights and uses DocT5Query expansion terms, while SparTerm [137] predicts vocabulary-wide passage importance for sparse representations.SparTerm combines an importance predictor with a gating controller; its logarithmic function smooths the importance distribution and limits term dominance.Source: §3.5 · pp. 21–22
  • SPLADE outperformed the poorest-performing SparTerm and BM25-based EPIC, while DeepImpact followed by a ColBERT reranker was more effective.The comparison highlights sparse representations and independent term-based weighting.Source: §3.5 · p. 22

3.6 BERT-based Query Expansion

  • On Robust04, ClassicQEConcepts performed worst, while FilteredGoogleQuestions performed better on “Query Title” and “Query description”.Padaki et al. compared query-expansion methods for BERT reranking, including structure and concept expansions.Source: §3.6 · pp. 22–23
  • BERT-QE-LLL (BERT-Large during three phases) recorded gains of 2.5%, 2.5%, and 3.3% on P@20, NDCG@20, and MAP@1k versus BERTLarge, respectively, at 30% higher computational cost.Source: §3.6 · p. 23
  • In Robust04, variations of CEQE improved P@20 by 10.5% and MAP@1000 by 19.9%.CEQE uses contextualized embeddings for the query and terms in top-ranking feedback documents, selecting expansion terms closest to the query embeddings by a similarity measure.Source: §3.6 · pp. 23–24
  • ColBERT-PRF [134] introduced pseudo-relevance feedback for dense retrieval, using representative feedback embeddings to expand queries and significantly outperforming on TREC 2019 and 2020.It uses K-Means clustering to extract representative feedback embeddings and incorporate them into the query representation.Source: §3.6 · p. 24
  • ANCE-PRF [35] improved query representations without changing the document index, and its improvement margin over ANCE exceeded that obtained with strong dense-retrieval models such as ColBERT.It learns query embeddings directly from pseudo-relevance labels using the query and top-retrieved documents.Source: §3.6 · p. 24

3.7 BERT-based Document Expansion

  • Using Doc2query, BM25 yielded an improvement of more than 15% in MAP and MRR@10 on both the TREC-CAR and MS MARCO datasets.Source: §3.7 · p. 24
  • The document-expansion retriever outperformed relevant state-of-the-art approaches in both passage ranking and document ranking.It generates passage query sets with a pointer–generator model, indexes them, and reranks retrieved candidates with BERT.Source: §3.7 · p. 24

4 RESOURCES FOR BERT-BASED RANKING MODELS

The section surveys commonly used ad-hoc retrieval datasets and provides resources and toolkits for developing and comparing BERT-based ranking models.

4 RESOURCES FOR BERT-BASED RANKING MODELS

  • The section lists MatchZoo, Anserini, Pyserini, Birch, PyTerrier, Capreolus, OpenNIR, and OpenMatch as resources for BERT-based IR.It also describes ad-hoc retrieval datasets used in these applications.Source: §4 · p. 25

4.1 Datasets

  • MS MARCO supplies passage-ranking data, while TREC CAR contains 29M English Wikipedia documents and five folds of questions.MS MARCO has 8.8M passages; its training set contains 500K query–relevant-document pairs, and its evaluation set has approximately 6,800 queries.Source: §4.1 · p. 25
  • Robust04, ClueWeb09-B, and ClueWeb12-B13 provide test collections ranging from roughly 500K newswire documents to 52M webpages.Robust04 has 250 topics; ClueWeb09-B has 200 queries, and ClueWeb12-B13 has 100 queries.Source: §4.1 · p. 25
  • TREC Microblog collections pair tweet corpora with topics from the 2011–2014 tracks, supporting social-network and real-time search.The 2011 collection contains approximately 16 million tweets and 50 queries; the 2013 collection has approximately 240M tweets.Source: §4.1 · pp. 25–26

4.2 Resources for BERT-based Ranking Model

  • MatchZoo provides modules for preprocessing, model building, training, and evaluation across text-matching tasks.It supports TensorFlow, CNTK, and Theano, and runs on CPUs and GPUs.Source: §4.2 · p. 26
  • Pyserini supports reproducible sparse and dense retrieval, while Birch reranks Anserini’s top-k candidates with BERT.PyTerrier provides dataset-download and experiment interfaces for comparisons on the same queries and metrics.Source: §4.2 · p. 26
  • Capreolus, OpenNIR, and OpenMatch support model comparison or ranking workflows, including document retrieval, reranking, and transfer learning.The section notes that Capreolus, OpenNIR, and OpenMatch are suitable for fair comparisons through training-time hyperparameter tuning.Source: §4.2 · p. 26

5 CONCLUSIONS AND FUTURE WORK

The conclusion synthesizes BERT-based IR methods, resources, and comparisons with decoder-based LLMs. It reports MRR@10 near 50% across evaluated datasets and outlines research directions.

5 CONCLUSIONS AND FUTURE WORK

  • The review organizes BERT methods around long-document handling, semantic integration, efficiency, term-weight prediction, and query and document expansion.It also analyzes the advantages and disadvantages of approaches and provides resources including MatchZoo, Anserini, Pyserini, Birch, PyTerrier, Capreolus, OpenNIR, and OpenMatch.Source: §5 · p. 27
  • The survey emphasizes encoder-based BERT models’ benefits over decoder-based LLMs such as ChatGPT, which require significant computational power.The supplied conclusion does not specify a task-level performance advantage or quantify deployment cost.Source: §5 · p. 27
  • MRR@10 for evaluated state-of-the-art methods was around 50% across different datasets, leaving considerable room for improvement.The survey compared published benchmark results because models used inconsistent datasets and evaluation metrics.Source: §5 · p. 27
  • The conclusion identifies combining contextualized BERT embeddings with neural ranking models through multi-task optimization as a future direction [1].It also proposes reinforcement learning to transform weakly supervised learning into unsupervised learning and reduce training-data cost [36] [72].Source: §5 · p. 27
  • Future directions include applying generative LLMs across retrieval and ranking stages [130] [153] and to fields such as biomedical and chemical IR.The conclusion also points to query and document expansion [67] and multimodal conversational search.Source: §5 · p. 27
Loading 2403.00784v1…