PaperlayerView exact revision on arXiv

Source-linked AI summary

SimLM: Pre-training with Representation Bottleneck for Dense Passage Retrieval

Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, Furu Wei

arXiv:2207.02578v2cs.IR

TL;DR

Retrieval pretraining does not consistently translate general-purpose language-model improvements into retrieval gains. SimLM pretrains a passage representation bottleneck with replaced language modeling, and the authors report substantial gains across retrieval settings, with state-of-the-art results under a stronger training setup.

  • Problem

    General-purpose pretraining improvements verified on NLU benchmarks do not consistently yield retrieval gains, motivating pretraining tailored to dense retrieval.

    Source: §1 · p. 1

  • Method

    SimLM uses a deep encoder and shallow decoder around a [CLS] representation bottleneck, trained to predict original tokens from replaced text.

    Source: §1 · p. 1, §3.1 · p. 3

  • Results

    SimLM achieves substantial gains over competitive methods with BM25 hard negatives and state-of-the-art performance when combined with mined negatives and cross-encoder re-ranker distillation.

    Source: §1 · p. 2

  • Takeaways & Limitations

    As initialization for dense-retriever training, SimLM achieves competitive results on several large-scale passage-retrieval datasets.

    Source: §6 · p. 9

    SimLM is not a zero-shot dense retriever: labeled-data fine-tuning is needed for a high-quality model, and pretraining requires extra computation.

    Source: Limitations · p. 9

Abstract

from arXiv · show

In this paper, we propose SimLM (Similarity matching with Language Model pre-training), a simple yet effective pre-training method for dense passage retrieval. It employs a simple bottleneck architecture that learns to compress the passage information into a dense vector through self-supervised pre-training. We use a replaced language modeling objective, which is inspired by ELECTRA, to improve the sample efficiency and reduce the mismatch of the input distribution between pre-training and fine-tuning. SimLM only requires access to unlabeled corpus, and is more broadly applicable when there are no labeled data or queries. We conduct experiments on several large-scale passage retrieval datasets, and show substantial improvements over strong baselines under various settings. Remarkably, SimLM even outperforms multi-vector approaches such as ColBERTv2 which incurs significantly more storage cost. Our code and model check points are available at https://github.com/microsoft/unilm/tree/master/simlm .

1 Introduction

General-purpose pretraining does not consistently improve retrieval, motivating SimLM’s bottleneck-based pretraining for dense retrievers. The authors report substantial gains over competitive methods with BM25 hard negatives and state-of-the-art performance when adding mined negatives and cross-encoder distillation.

1 Introduction

  • Improved pretraining techniques validated on GLUE do not consistently improve retrieval, a mismatch also noted by Lu et al. (2021).Table 1 reports MS-MARCO MRR@10 and GLUE results and describes their performance trends as inconsistent.Source: §1 · p. 1, Table 1 · p. 1
  • SimLM uses a deep encoder and shallow decoder linked by a [CLS] bottleneck to predict original tokens from randomly replaced text, encouraging passage information compression.The decoder’s limited capacity requires it to rely on the bottleneck; the objective works with plain texts and does not require generated pseudo-queries as GPL does (Wang et al., 2022).Source: §1 · pp. 1–2
  • Unlike Condenser and coCondenser (Gao and Callan, 2021; Gao and Callan, 2022), SimLM omits encoder-decoder skip connections and uses an ELECTRA-like objective with gradients at all positions and no [MASK] inputs.The authors state that this design increases sample efficiency and decreases input-distribution mismatch between pretraining and fine-tuning.Source: §1 · p. 2
  • SimLM reports substantial gains over competitive methods with BM25 hard negatives and state-of-the-art performance with mined hard negatives and cross-encoder-based re-ranker distillation.Experiments cover MS-MARCO passage ranking, TREC Deep Learning Track datasets, and Natural Questions.Source: §1 · p. 2

2 Related Work

Related work contrasts dense retrieval with lexical retrieval and surveys approaches to lexical matching and retrieval-specific pretraining. SimLM is positioned among methods that aim to make a single bottleneck representation retain the information needed for matching.

2 Related Work

  • Dense retrieval maps queries and passages to low-dimensional vectors for semantic matching, while BM25 can suffer from lexical mismatch; ANN methods such as HNSW support efficient inference.The field has shifted toward neural dense retrieval, though BM25 remains efficient and interpretable.Source: §2 · p. 2
  • To address missed exact matches, prior work adds BM25 as a teacher, uses token-level MaxSim in ColBERT (Khattab and Zaharia, 2020), or incorporates lexical signals in COIL (Gao et al., 2021).The paper presents its pretraining method as compatible with existing approaches.Source: §2 · p. 2
  • Retrieval-specific pretraining includes Wikipedia-structure tasks ICT, BFS, and WLP (Chang et al., 2020), alongside representative-word prediction, contrastive objectives, domain-matched pretraining, and neighboring-text pairs.The cited approaches include Ma et al. (2021, 2022), Izacard et al. (2021), Oguz et al. (2022), and Neelakantan et al. (2022).Source: §2 · p. 2
  • Condenser, coCondenser, SEED, DiffCSE, and RetroMAE pursue a [CLS] representation that retains important text information; SimLM removes encoder-decoder skip connections to further constrain that bottleneck.The passage identifies RetroMAE as concurrent work combining a bottleneck architecture with masked auto-encoding.Source: §2 · pp. 2–3

3 SimLM

SimLM pre-training uses an encoder bottleneck and replaced language modeling to train passage representations. Its encoder initializes biencoder retrievers in a staged fine-tuning pipeline.

3.1 Pre-training

  • The model creates two corrupted views using potentially different replacement rates, and every encoder-replaced token is also replaced in the decoder view to increase task difficulty.The encoder and decoder predict the original tokens at all positions using replaced language modeling; their losses are summed for the final pre-training loss.Source: §3.1 · p. 3
  • Pre-training uses a passage collection without queries or human-labeled data; after training, the decoder is discarded and only the encoder is retained for fine-tuning.The paper describes directly pre-training on the target corpus when it is available before queries.Source: §3.1 · p. 3
  • The encoder compresses passage semantics into a [CLS] bottleneck vector that a shallow decoder uses alongside its input for pre-training.Because the decoder has limited capacity, it relies on the bottleneck, requiring the encoder to pass semantic information through it.Source: §3.1 · p. 3

3.2 Fine-tuning

  • Fine-tuning trains Retriever1 with in-batch and BM25 negatives, mines hard negatives for Retriever2, then uses a cross-encoder to rerank Retriever2's top-k results.The cross-encoder re-ranker uses a listwise loss and can provide soft scores for Retrieverdistill.Source: §3.2 · p. 4
  • Retrieverdistill learns from re-ranker scores on positive pairs and mined negatives, combining KL-divergence distillation with the hard-label contrastive loss.The final loss linearly interpolates the two losses.Source: §3.2 · p. 4
  • SimLM initializes the biencoder retrievers, not the cross-encoder re-ranker, and affects initialization only, allowing integration into other training pipelines.The described stages take the preceding stage's outputs as inputs and can be trained independently.Source: §3.2 · pp. 4–5, Figure 2 · p. 4

4 Experiments

The experiments evaluate SimLM on MS-MARCO, TREC Deep Learning, and Natural Questions; reported results include a new MS-MARCO state of the art and competitive NQ recall.

4.1 Setup

  • Evaluation covers MS-MARCO passage ranking, TREC DL 2019/2020, and NQ, using dataset-specific MRR, Recall, and nDCG metrics.Pretraining initializes the encoder from BERTbase and uses a two-layer Transformer decoder with a frozen ELECTRA generator.Source: §4.1 · p. 5

4.2 Main Results

  • On TREC DL 2019, SimLM is slightly below coCondenser, but the authors deem the gap insignificant given 1%–2% nDCG@10 seed variation.The TREC DL datasets have fewer than 100 labeled queries.Source: §4.2 · pp. 5–6
  • 41.1 MRR@10 establishes a new MS-MARCO state of the art, outperforming ColBERTv2 despite its 6× index-storage cost.Table 3 compares index storage cost and index-search complexity.Source: §4.2 · p. 5, Table 3 · p. 6
  • 85.2 R@20 and 89.7 R@100 on NQ are comparable to or better than other methods; end-to-end QA evaluation remains future work.SimLM's listed NQ results use Retrieverdistill.Source: Table 4 · p. 6, §4.2 · p. 6
  • SimLM initialization improves MS-MARCO reranker performance by 0.6% over BERTbase but remains behind ELECTRAbase.Table 5 reports reranker performance on the MS-MARCO development set.Source: Table 5 · p. 6, §4.2 · p. 6
  • With BM25 hard negatives alone, MRR@10 reaches 38.0; mined negatives and reranker distillation add gains, while another mining round adds no meaningful improvement.The authors identify simplifying the training pipeline while maintaining competitive performance as an open direction.Source: Table 6 · p. 6, §4.2 · p. 6

5 Analysis

The analyses compare pre-training objectives and settings, finding advantages for SimLM across training efficiency, corpus choice, and qualitative retrieval examples. They also identify limits to decoder replacement and show that matching pre-training and fine-tuning corpora yields the largest gains.

5.1 Variants of Pre-training Objectives

  • Enc-Dec MLM outperforms Enc-Dec RTD, Condenser is only slightly better than MLM, and naive auto-encoding underperforms BERTbase.The comparisons use MS-MARCO dev-set MRR@10 with official BM25 hard negatives; Table 7 summarizes the results.Source: §5.1 · p. 7, Table 7 · p. 7

5.2 Effects of Replace Rate

  • SimLM is robust across replace rates, with 30%–40% encoder replacement performing slightly better; 100% decoder replacement leaves no meaningful context and harms the encoder.Table 8 reports MS-MARCO passage-ranking performance across token replace rates.Source: §5.2 · p. 7, Table 8 · p. 7

5.3 Effects of Pre-training Steps

  • SimLM reaches competitive results after 10k training steps and converges at 60k, while MLM continues to improve slowly; Figure 3 reports dev-set MRR@10.The objective computes loss over all input tokens and has less input-distribution mismatch than MLM.Source: Figure 3 · p. 7, §5.3 · p. 8

5.4 On the Choice of Pre-training Corpus

  • Matching pre-training and fine-tuning corpora yields the largest gains, while pre-training on MS-MARCO and fine-tuning on NQ—or vice versa—still improves over baseline.The NQ setting has 21M candidate passages but fewer than 80k training question-answer pairs; simple target-corpus MLM also brings large gains.Source: §5.4 · p. 8, Table 10 · p. 8

5.5 Case Analysis

  • BERTbase retrieves passages with high lexical overlap that miss subtle query semantics, while SimLM pre-training helps retrieve passages that answer such queries.The examples include a query about Winnie the Pooh and one requiring a routing number; Table 9 presents selected dev-set examples.Source: §5.5 · p. 8, Table 9 · p. 8

6 Conclusion

SimLM pre-trains a representation bottleneck so an encoder compresses passage semantics into a dense vector, then uses that model to initialize dense-retriever training. The paper reports competitive results on large-scale passage-retrieval datasets and identifies scaling and unsupervised or multilingual retrieval as future directions.

6 Conclusion

  • SimLM uses an encoder-decoder architecture with a representation bottleneck, training the encoder to compress passage semantics into a dense vector.The decoder predicts original tokens from the bottleneck representation during replaced language modeling pre-training.Source: §6 · p. 9
  • Used to initialize dense-retriever training, SimLM achieves competitive results on several large-scale passage-retrieval datasets.Source: §6 · p. 9
  • Future work includes examining larger models and corpora and exploring pre-training for unsupervised dense and multilingual retrieval.Source: §6 · p. 9

Limitations

SimLM has a scope boundary: it is not a zero-shot dense retriever, and labeled-data fine-tuning is needed for a high-quality model.

Limitations

  • SimLM cannot serve as a zero-shot dense retriever because its pre-training lacks a contrastive objective; labeled-data fine-tuning is needed for a high-quality model.Pre-training still requires additional computational resources.Source: Limitations · p. 9

Ethical Considerations

Offensive or biased retrieval-corpus text could reach users through queries; the paper proposes classifiers or manual inspection to exclude it.

Ethical Considerations

  • Offensive or biased corpus texts may reach users through retrieval, so the paper proposes toxic-text classifiers or manual inspection to exclude them.Source: Ethical Considerations · p. 9

A Details on Table 1

The appendix qualifies the GLUE comparison and specifies the MS-MARCO fine-tuning setup used to compare pre-trained models.

A Details on Table 1

  • For MS-MARCO, all pre-trained models were fine-tuned using BM25 hard negatives only.BERT and RoBERTa used Section 4.1 hyperparameters, while ELECTRA trained for 6 epochs with a peak learning rate of 4 × 10−5 because it converged more slowly.Source: A Details on Table 1 · p. 12
  • The GLUE leaderboard comparison is not entirely fair because BERT’s submission did not use an ensemble, but the authors say their conclusion is unchanged.They conclude that BERT generally performs worse than RoBERTa and ELECTRA on NLP tasks.Source: A Details on Table 1 · p. 12

B Implementation Details

The implementation appendix documents the corpus, training and inference setup, and selected dataset-specific fine-tuning details.

B Implementation Details

  • The pre-training Wikipedia corpus comes from DPR rather than the corpus used for original BERT pre-training.Source: Table 11 · p. 12
  • Supervised fine-tuning uses one shared encoder for queries and passages, begins with BM25 hard negatives, then switches to mined hard negatives.At inference, brute-force search ranks all passages for comparison with previous work.Source: B Implementation Details · p. 12
  • All reported results are from a single run, and the authors report that values are stable across random seeds.Source: B Implementation Details · p. 12
  • For NQ, fine-tuning runs for 20k steps at learning rate 5×10−6, with maximum passage length 192.Mined hard negatives are top-100 predictions that do not contain a correct answer.Source: B Implementation Details · p. 12

C Variants of Generators

The appendix reports that tested generator-training variants perform similarly, while the frozen generator is faster to train; it also notes sensitivity to generator strength.

C Variants of Generators

  • Using a generator that is too strong or too weak hurts the discriminator’s learnability and generalization in ELECTRA pre-training.Source: C Variants of Generators · p. 12
  • The tested frozen, jointly trained, and randomly initialized joint-training generator variants show no significant performance difference on MS-MARCO dev with BM25 negatives only.The authors use the frozen generator for faster training.Source: C Variants of Generators · p. 12, Table 12 · p. 12
Loading 2207.02578v2…