PaperlayerView exact revision on arXiv

Source-linked AI summary

RepBERT: Contextualized Text Embeddings for First-Stage Retrieval

Jingtao Zhan, Jiaxin Mao, Yiqun Liu, Min Zhang, Shaoping Ma

arXiv:2006.15498v2cs.IR

TL;DR

Exact-term-match approaches dominate first-stage retrieval, while the paper identifies difficulty retrieving documents at the semantic level. RepBERT represents queries and documents with fixed-length contextualized embeddings and inner-product scores, achieving state-of-the-art initial retrieval on MS MARCO Passage Ranking with efficiency comparable to bag-of-words methods.

  • Problem

    Bag-of-words first-stage methods rely on exact term matches and can hardly retrieve documents at the semantic level.

    Source: §1 · p. 1

  • Method

    RepBERT encodes documents offline and queries online as fixed-length embeddings, using their inner products as relevance scores.

    Source: §1 · p. 1

  • Results

    RepBERT achieves state-of-the-art initial retrieval on MS MARCO Passage Ranking, substantially outperforming other first-stage techniques on MRR@10 and approaching docTTTTTquery’s Recall@1000.

    Source: §4.3.2 · pp. 3–4, §5 · p. 7

  • Takeaways & Limitations

    RepBERT’s efficiency is comparable to bag-of-words models, supporting the possibility of replacing them for initial retrieval.

    Source: §4.3.2 · p. 4

    The authors attribute the mismatch between recall and reranking performance to a possible training–testing distribution mismatch when retrievers differ.

    Source: §4.4.2 · p. 5, §4.4.3 · p. 5

Abstract

from arXiv · show

Although exact term match between queries and documents is the dominant method to perform first-stage retrieval, we propose a different approach, called RepBERT, to represent documents and queries with fixed-length contextualized embeddings. The inner products of query and document embeddings are regarded as relevance scores. On MS MARCO Passage Ranking task, RepBERT achieves state-of-the-art results among all initial retrieval techniques. And its efficiency is comparable to bag-of-words methods.

1 Introduction

RepBERT addresses the difficulty of retrieving semantically relevant documents with exact-term-match first-stage methods. It uses fixed-length text embeddings and inner-product relevance scores, and evaluates on MS MARCO Passage Ranking.

1 Introduction

  • RepBERT applies deep neural models directly to first-stage retrieval to address exact-match methods’ difficulty retrieving documents at the semantic level.The introduction notes that recent deep-language-model improvements to bag-of-words methods still rely on exact term match signals.Source: §1 · p. 1
  • Documents are encoded offline, while online query embeddings are scored against document embeddings by inner product and retrieved as a Maximum Inner Product Search problem.The paper notes that MIPS algorithms can have sub-linear computational complexity.Source: §1 · p. 1
  • RepBERT uses BERT to represent queries and documents and is evaluated on the MS MARCO Passage Ranking benchmark.The name reflects its categorization as a representation-focused model in information retrieval.Source: §1 · p. 1

2 Related Work

The related-work discussion contrasts neural retrieval’s effectiveness in Open QA with the continued dominance of bag-of-words first-stage retrieval. Existing deep-language-model enhancements still modify bag-of-words approaches rather than replacing exact-match retrieval.

2 Related Work

  • Bag-of-words models remain dominant for first-stage information retrieval despite neural retrievers’ effectiveness in Open QA tasks [11] [12].The paper notes that almost all public methods on the MS MARCO Passage Ranking leaderboard use bag-of-words initial retrieval.Source: §2 · p. 1
  • Prior work found that fixed-length text embeddings risk losing details and that term interactions matter for superior ranking.The authors suggest powerful language models such as BERT [4] may address this issue.Source: §2 · p. 2
  • DeepCT [1] replaces BM25 term-frequency weights with BERT-computed weights, while doc2query [2] and docTTTTTquery expand documents with predicted queries.docTTTTTquery uses T5 [5] as its expansion model.Source: §2 · p. 2

3 RepBERT

RepBERT encodes queries and documents with BERT-derived contextualized embeddings and scores them by inner product. Training uses a margin loss over relevant and irrelevant documents, with in-batch documents supplying negatives.

3.1 Model Architectures

  • BERT tokenizes each input with boundary tokens, produces contextualized token vectors, and averages them into a text embedding using shared query and document encoder weights.The input begins with [CLS] and ends with [SEP].Source: §3.1 · p. 2
  • Inner products between query and document embeddings serve as relevance scores, a design chosen mainly for efficiency.Source: §3.1 · p. 2

3.2 Training

  • Training seeks larger embedding inner products for relevant than irrelevant query-document pairs and uses MultiLabelMarginLoss.In-batch negatives use documents from other pairs in the mini-batch; typically each query has one positive and B −1 negatives.Source: §3.2 · p. 2

4.1 Dataset

The experiments use MS MARCO Passage Ranking, an English ad-hoc retrieval benchmark with millions of passages and separate training, development, and held-out evaluation queries.

4.1 Dataset

  • MS MARCO contains approximately 8.8 million passages, 0.5 million training queries, and 6.9 thousand development queries.Its held-out evaluation set has about 6.8 thousand queries, with results provided by the organizers.Source: §4.1 · p. 2
  • The benchmark is an English dataset for ad-hoc retrieval, and the paper calls its basic retrieval units “documents.”Source: §4.1 · pp. 2–3
  • BERT segment embeddings numbered 0 and 1 are assigned to query tokens and document tokens, respectively.Source: §4.1 · p. 2
  • Table 1 covers first-stage and two-stage retrieval model performance on the MS MARCO Passage Ranking dataset.Source: Table 1 · p. 3

4.2 Baselines

The paper compares RepBERT with four public first-stage retrieval techniques and two two-stage systems. The baselines include BM25-based approaches and neural rerankers.

4.2 Baselines

  • Four public MS MARCO first-stage baselines are BM25(Anserini), doc2query [2], DeepCT [1], and docTTTTTquery.The last three use deep language models to improve BM25.Source: §4.2 · p. 3
  • BiLSTM + Co-Attention + self attention is the strongest listed non-ensemble, non-BERT two-stage method, using BM25 for retrieval and deep attention networks for reranking.Its associated paper is.Source: §4.2 · p. 3
  • Nogueira et al. [20] provide another two-stage baseline, pairing BM25 retrieval with BERT Large reranking.Source: §4.2 · p. 3

4.3 First-Stage Retrieval

On MS MARCO, RepBERT substantially outperforms other first-stage retrieval techniques on MRR@10 and achieves high recall. Its reported efficiency is comparable to bag-of-words models.

4.3 First-Stage Retrieval

  • The section compares RepBERT with other retrieval techniques on first-stage retrieval performance.Source: §4.3 · p. 3

4.3.1 Settings

  • The experiment fine-tunes BERT base for 350k steps on 18.2M query-document pairs and reports no dev-set improvement from another 100k steps.Training used MS MARCO Train Triples, one Titan XP GPU, and a maximum document length of 256 tokens.Source: §4.3.1 · p. 3
  • For reported latency, the 26GB of document embeddings required three Titan XP and two GeForce GTX 1080ti GPUs to retrieve the top 1,000 documents per query.The paper says efficiency could improve with more advanced GPUs or TPUs.Source: §4.3.1 · p. 3

4.3.2 Discussion

  • RepBERT substantially outperforms other first-stage retrieval techniques on MRR@10 and beats the best non-ensemble, non-BERT two-stage method.The supplied passages do not give the MRR@10 values.Source: §4.3.2 · p. 3
  • RepBERT’s recall is reported as near docTTTTTquery, which uses the powerful T5 [5] language model, and it significantly outperforms other baselines.The authors suggest that more advanced language models could further improve RepBERT.Source: §4.3.2 · pp. 3–4
  • RepBERT’s efficiency is comparable to bag-of-words models, although its current retrieval implementation does not use optimized MIPS algorithms [6, 7, 8].The authors plan to investigate those algorithms in future work.Source: §4.3.2 · p. 4

4.4 Rerank based on RepBERT

The section examines RepBERT as a first-stage retriever for reranking, comparing reranking outcomes across retrieval depths and investigating how differences from BM25 candidates may relate to performance.

4.4.1 Settings

  • The experiments rerank documents from different first-stage retrievers with a public BERT Large model fine-tuned on MS MARCO, evaluating performance at different depths.The authors also compare recall at different depths because they consider recall important for reranking.Source: §4.4.1 · p. 4, §4.4 · p. 4

4.4.2 Discussion

  • RepBERT has the best recall at small depths and ties docTTTTTquery at large depths, but the authors’ expectation of best reranking at every depth is qualified by later results.The expectation is based on Figure 1a’s recall findings.Source: §4.4.2 · p. 5
  • According to Table 2 and Figure 1b, RepBERT achieves the best results at small depths; at depth 50, docTTTTTquery is best, though RepBERT significantly outperforms other baselines, and performance gaps shrink at depths 500 or 1000.Source: §4.4.2 · p. 5

4.4.3 Mismatch

  • The authors speculate that BM25-based reranker training can mismatch testing on RepBERT candidates, contributing to worse reranking despite strong recall at large depths.Their explanation is that training uses BM25 top-1,000 negatives, whereas testing may use documents from a different retriever.Source: §4.4.3 · p. 5, §4.4.2 · p. 5
  • The section measures consistency as the average share of a retriever’s top-n documents also present in BM25’s official top-1,000, and reports combination-table scopes for accuracy and recall.The combination tables describe accuracy and recall for different technique combinations.Source: §4.4.3 · p. 5, Table 3 · p. 6, Table 4 · p. 6
  • RepBERT’s consistency is significantly lower than other methods, especially at large depths; many retrieved documents are absent from BM25’s official candidates.The mismatch is consistent with RepBERT using semantic matching while prior methods retain exact-match signals.Source: §4.4.3 · pp. 5–6, Figure 2 · p. 5

4.5 Combination of Semantic Match and Exact Match

The section combines RepBERT’s semantic-match retrieval with exact-match techniques by merging their ranked lists, then reports that these combinations improve retrieval performance. RepBERT paired with docTTTTTquery is identified as the paper’s best first-stage retriever.

4.5.1 Method

  • The combination method alternates documents from both top-1000 lists, removes duplicates found at lower positions, and truncates the result to 1000 documents.The lists are defined as each technique’s top-n retrieved documents for a query.Source: §4.5.1 · p. 6
  • Combination order matters: the tables report performance relative to the first technique, and fa + fb differs from fb + fa in MRR@10.The tables also report retrieval accuracy and recall.Source: §4.5.1 · p. 6
  • Recall@2000 is 0.980 for RepBERT+docTTTTTquery without truncation, versus 0.967 after truncation.The authors note that other combination methods may achieve better performance.Source: §4.5.1 · p. 6

4.5.2 Discussion

  • BM25 and its improved versions achieve their best ranking accuracy and recall when combined with RepBERT.The authors attribute this to RepBERT complementing the baselines’ missing semantic matching ability.Source: §4.5.2 · p. 7
  • RepBERT+docTTTTTquery is the paper’s best first-stage retriever; exact-match combinations improve RepBERT’s recall and may improve its ranking accuracy.The authors conclude that exact-match signals are also helpful for semantic-matching retrievers.Source: §4.5.2 · p. 7

5 Conclusion

RepBERT uses contextualized embeddings for first-stage retrieval and achieves state-of-the-art initial retrieval performance on MS MARCO Passage Ranking.

5 Conclusion

  • RepBERT uses contextualized embeddings for first-stage retrieval and achieves state-of-the-art initial retrieval performance on MS MARCO Passage Ranking.The paper highlights representation-focused neural models as a possible replacement for bag-of-words first-stage retrieval and plans to test generalization on other datasets and retrieval of long text.Source: §5 · p. 7
Loading 2006.15498v2…