PaperlayerView exact revision on arXiv

Source-linked AI summary

Unsupervised Dense Information Retrieval with Contrastive Learning

Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, Edouard Grave

arXiv:2112.09118v4cs.IRcs.AIcs.CL

TL;DR

Dense retrievers transfer poorly when annotated data is unavailable, motivating unsupervised alternatives. The paper trains dense retrievers with contrastive learning and evaluates them across BEIR, few-shot, fine-tuning, and multilingual settings. The approach is competitive with BM25 and improves retrieval after fine-tuning, although it still lags BM25 on MRR@100.

  • Problem

    Dense retrievers do not transfer well to new applications without training data and can be outperformed by unsupervised term-frequency methods such as BM25.

  • Method

    The paper uses contrastive learning to pre-train dense retrievers without annotated data, including multilingual pre-training and document-based positive-pair construction.

  • Results

    The unsupervised model is competitive with BM25, while pre-training followed by fine-tuning yields strong BEIR performance, including average Recall@100 improving from 65.0 to 67.1.

  • Takeaways & Limitations

    Contrastive pre-training provides strong unsupervised and multilingual retrieval performance and transfers across languages, including cross-lingual retrieval between different scripts.

  • Takeaways & Limitations

    The unsupervised model still lags behind BM25 on MRR@100.

Abstract

from arXiv · show

Recently, information retrieval has seen the emergence of dense retrievers, using neural networks, as an alternative to classical sparse methods based on term-frequency. These models have obtained state-of-the-art results on datasets and tasks where large training sets are available. However, they do not transfer well to new applications with no training data, and are outperformed by unsupervised term-frequency methods such as BM25. In this work, we explore the limits of contrastive learning as a way to train unsupervised dense retrievers and show that it leads to strong performance in various retrieval settings. On the BEIR benchmark our unsupervised model outperforms BM25 on 11 out of 15 datasets for the Recall@100. When used as pre-training before fine-tuning, either on a few thousands in-domain examples or on the large MS~MARCO dataset, our contrastive model leads to improvements on the BEIR benchmark. Finally, we evaluate our approach for multi-lingual retrieval, where training data is even scarcer than for English, and show that our approach leads to strong unsupervised performance. Our model also exhibits strong cross-lingual transfer when fine-tuned on supervised English data only and evaluated on low resources language such as Swahili. We show that our unsupervised models can perform cross-lingual retrieval between different scripts, such as retrieving English documents from Arabic queries, which would not be possible with term matching methods.

1 Introduction

Dense retrievers perform strongly when annotated training data is available, but transfer poorly to data-scarce domains where BM25 often performs better. The paper asks whether contrastive learning can train competitive dense retrievers without supervision and presents contributions across unsupervised, few-shot, fine-tuned, and multilingual retrieval.

  • Motivation: Neural retrievers achieve state-of-the-art performance when large training datasets are available, but collecting query-document annotations is difficult for massive collections.This leaves many applications with few or no in-domain examples.
  • Motivation: Dense retrievers transferred from MS MARCO are often outperformed by unsupervised term-frequency methods such as BM25 in new domains.Large annotated datasets are also generally unavailable for languages other than English.
  • Research question: The paper investigates whether contrastive learning can train dense retrievers without supervision and match BM25.The proposed setup generates synthetic query-document training pairs from individual documents.
  • Contributions: The model achieves Recall@100 competitive with BM25 on most BEIR datasets and improves with few-shot training or MS MARCO pre-training.The paper also reports multilingual and cross-lingual retrieval results.

2 Related work

Prior retrieval work spans sparse term-frequency methods, neural bi-encoders and cross-encoders, and self-supervised objectives for learning text representations and retrievers.

  • Term-frequency based information retrieval: Term-frequency systems represent documents and queries as sparse vocabulary vectors, with BM25 extending TF-IDF but relying on near-exact token matches.Latent semantic analysis introduced lower-dimensional dense representations as an alternative.
  • Neural network based information retrieval: Neural retrieval methods encode queries and documents independently or jointly, using relevance scores such as representation dot products.Cross-encoders and strong pretrained models improved performance on MS MARCO, while bi-encoders offered alternatives to reranking.
  • Self-supervised learning for NLP: Self-supervised retrieval research includes sentence-level objectives, contrastive techniques, inverse Cloze pre-training, and jointly trained retriever-generator models.Prior work compared retrieval pre-training tasks and applied MoCo-style learning to text.

3 Method

The method trains a dense retriever without supervision by encoding queries and documents with a transformer, forming positive views from one document, and contrasting them against many negatives.

  • Retriever architecture: The retriever scores a query-document pair by the dot product of their encoded representations, using a transformer encoder for both inputs.Sharing the encoder generally improves robustness in zero-shot and few-shot settings.
  • Contrastive learning: Contrastive training creates positive and negative document pairs and optimizes InfoNCE so positives score higher than negatives.The loss can be interpreted as retrieving the positive key from a pool of negative keys.
  • Building positive pairs from a single document: Inverse Cloze Task forms a positive pair from a sampled token span and its complementary document context.The span is used as the query and the complement as the key.
  • Building positive pairs from a single document: Independent random cropping samples two spans independently from one document, producing symmetric views that may overlap and encouraging exact matching.The paper contrasts this with ICT, whose views are mutually exclusive and asymmetric.
  • Building large set of negative pairs: In-batch negatives use views from other examples in the batch, but effective training may require extremely large batches.Prior work reported improvements with up to 8192 negatives.
  • Building large set of negative pairs: MoCo stores representations from previous batches in a queue, enabling smaller batch sizes while updating query and key encoders asymmetrically.The key encoder follows the query encoder through an exponential moving average controlled by momentum m.

4 Experiments

The experiments evaluate Contriever’s unsupervised contrastive training across question answering, BEIR, and few-shot retrieval settings. Results show competitive or improved retrieval against BM25 and other dense retrievers, including after pre-training and fine-tuning.

  • Method: Contriever uses MoCo contrastive learning with randomly cropped positive pairs, token deletion, queued negatives, and Wikipedia–CCNet training data.These choices distinguish the method from ICT and enable a large pool of negatives.
  • Datasets and metrics: Contriever is evaluated on NaturalQuestions and TriviaQA using open-domain collections and top-k retrieval accuracy.The evaluation counts questions whose answers occur in at least one of the top-k retrieved passages.
  • Unsupervised retrieval: 3 points improvement in recall@100 on NaturalQuestions makes Contriever competitive with BM25 and stronger than prior ICT- and salient-span-trained dense retrievers.The comparison concerns fully unsupervised models without fine-tuning on MS MARCO or other annotated data.
  • Unsupervised retrieval: 11 out of 15 BEIR datasets favor Contriever over BM25 for recall@100, while Contriever remains competitive with BM25 on nearly all datasets.The exceptions identified are TREC-COVID and Tóuche-2020.
  • BEIR fine-tuning: 67.1 average recall@100 versus 65.0 shows that contrastive pre-training improves MS MARCO-fine-tuned BEIR performance.Contriever achieves the best results among dense bi-encoders for nDCG@10 and state-of-the-art average recall@100; cross-encoder reranking reaches state-of-the-art nDCG@10 on 8 datasets and on average.
  • Few-shot retrieval: Contriever pre-training outperforms BM25 and BERT-based alternatives when only 729 to 5,500 in-domain queries are available.The comparison includes BERT fine-tuned on MS MARCO as an intermediate step.

5 Multilingual retrieval

Unsupervised multilingual dense retrieval addresses limited labeled data beyond English and supports both same-language and cross-lingual retrieval. The approach uses multilingual contrastive pre-training and benefits from English-only fine-tuning, although BM25 remains stronger on MRR@100.

  • 5.1 Multilingual pre-training: mContriever is jointly pre-trained on 29 languages for multilingual retrieval.The model is initialized from mBERT, uniformly samples training examples across languages, and uses a queue size of 32768.
  • 5.2 Fine-tuning on English data: English-only MS MARCO fine-tuning generally improves mContriever's performance across languages and can be followed by Mr. TyDi fine-tuning for state-of-the-art results.The paper evaluates whether English retrieval data can transfer to multilingual retrieval.
  • 5.4 Cross-lingual retrieval: The MKQA-derived evaluation contains 6619 queries, and term-matching methods are intrinsically limited when query and document languages use different terms or scripts.The benchmark retrieves English Wikipedia documents from questions posed in other languages.
  • 5.3 Evaluation: mContriever outperforms BM25 on Recall@100 in Mr. TyDi, reaches state-of-the-art performance after MS MARCO fine-tuning, and improves further with Mr. TyDi fine-tuning.The same results report that English-only fine-tuning improves performance on all languages.
  • 5.5 Results: mContriever still lags behind BM25 on MRR@100, which emphasizes the quality of the first retrieved documents.The limitation concerns the unsupervised model's performance on this metric.
  • 5.5 Results: On the MKQA-derived benchmark, MS MARCO-fine-tuned mContriever outperforms the CORA retriever using only supervised English training data.Multilingual contrastive pre-training improves performance over the corresponding model without contrastive pre-training.

6 Ablation studies

The ablations examine contrastive-learning design choices, including negative-example count, data augmentation, training data, and MS MARCO fine-tuning. Results favor scalable negatives and random cropping, while data-source effects vary by dataset and pre-training contributes substantially to final performance.

  • MoCo vs. in-batch negatives: MoCo and in-batch negatives show only a small performance difference, so MoCo is preferred because it scales negatives without increasing batch size.MoCo uses a momentum encoder for queued keys and avoids backpropagating through them.
  • Number of negative examples: Increasing MoCo queue size from 2,048 to 131,072 improves average BEIR retrieval performance, especially without supervised fine-tuning, although gains vary across datasets.The comparison uses nDCG@10 with and without MS MARCO fine-tuning.
  • Data augmentations: Random cropping outperforms the inverse Cloze task for generating positive pairs in this setting.The authors suggest that identical key and query distributions may make MoCo training more stable.
  • Training data: Neither Wikipedia nor CCNet is uniformly superior: CCNet helps on diverse-domain datasets such as FiQA and Quora, whereas Wikipedia performs better on FEVER.The study also compares 50/50% and union-uniform mixtures of the two sources.
  • Impact of fine-tuning on MS MARCO: Fine-tuning alone produces below-state-of-the-art results when applied to BERT, suggesting that contrastive pre-training accounts for most gains over state-of-the-art retrievers.The comparison reports nDCG@10 after fine-tuning BERT and the authors’ model on MS MARCO.

7 Discussion

The discussion presents contrastive pre-training with MoCo as a route to unsupervised dense retrieval that is competitive with BM25. Fine-tuning and cross-encoder reranking further strengthen performance, while random cropping emerges as an alternative to ICT.

  • 7 Discussion: MoCo-based contrastive pre-training yields unsupervised neural retrievers competitive with BM25, though not state-of-the-art.Fine-tuning on MS MARCO further improves retrieval, particularly Recall@100.
  • 7 Discussion: Cross-encoder reranking of documents retrieved by the model leads to new state-of-the-art performance on BEIR.The discussion also identifies independent random cropping as a strong alternative to the inverse Cloze task.

A.1 Contrastive pre-training

The pre-training setup combines MoCo contrastive learning with random cropping over mixed Wikipedia and CCNet data. MS MARCO fine-tuning instead uses in-batch negatives and a two-stage hard-negative procedure.

  • Contrastive pre-training: The fine-tuned model uses MoCo with a 131,072-element queue, momentum 0.9995, temperature 0.05, and random-cropped spans from mixed Wikipedia and CCNet data.Documents contain 256 tokens, with crop spans sampled between 5% and 50% of document length.
  • Contrastive pre-training: MS MARCO fine-tuning uses in-batch negatives, first trains with random negatives, then mines hard negatives and retrains a second model.Hard negatives are used 10% of the time in the second phase.

A.3 Few-shot training

The few-shot evaluation trains separately on each dataset and uses development-set early stopping. The BEIR table reports capped or standard Recall@100 for bi-encoder methods across datasets.

  • Few-shot training: Few-shot models train for 500 epochs per dataset with batch size 256 and in-batch random negatives, evaluating every 100 updates for early stopping.SciFact reserves a random 10% of training data for development, leaving 729 training samples.
  • Few-shot training: The BEIR benchmark table reports Recall@100 on test sets for bi-encoder methods, using capped Recall@100 for Trec-COVID.It also reports average performance and the number of datasets where each method is best.

B.1 Hyperparameters for multilingual contrastive pre-training

mContriever uses a 500,000-step multilingual contrastive pre-training schedule with a large queue, low temperature, momentum encoder, and AdamW optimization.

  • mContriever is pre-trained for 500,000 steps with a queue of size 32768, temperature 0.05, and momentum 0.999.AdamW is used with a learning rate of 5 · 10−5, followed by linear warmup and decay.

B.2 Hyperparameters for multilingual fine-tuning

Multilingual fine-tuning uses in-batch negatives, AdamW, temperature 0.05, and task-dependent training schedules, with hard negatives applied only for Mr. TyDi.

  • Fine-tuning uses in-batch negatives, AdamW, learning rate 10−5, batch size 1024, and temperature τ = 0.05.
  • The model is trained for 20k gradient steps on MS MARCO and Mr. TyDi, but only 1k steps on NaturalQuestions because of overfitting.All fine-tuning uses 1000 warmup steps followed by linear decay.
  • Hard negatives are mined on Mr. TyDi using a model trained on MS MARCO, without significant gains on MS MARCO or NaturalQuestions.
  • For non-contrastively pre-trained mBERT and XLM-R models, temperatures 1 and 5 were selected after testing several values.Lower temperatures decreased performance, while mContriever retained τ = 0.05 from pre-training.

B.3 Curse of multilinguality

The multilinguality analysis compares retrievers pre-trained on 11 versus 29 languages and reports performance on Mr. TyDi, including after MS MARCO and Mr. TyDi fine-tuning.

  • The authors generally observe performance deterioration when scaling pre-training to more languages.This comparison motivates reporting Mr. TyDi results for models trained on 11 versus 29 languages.
  • Table 12 lists the languages used for multilingual retrieval.
  • Tables 13 and 14 report cross-lingual retrieval performance on MKQA using Recall@100 and Recall@20, respectively.
  • Table 15 reports MRR@100 and R@100 on the Mr. TyDi test set for models pre-trained on 11 or 29 languages.It also reports results after MS MARCO fine-tuning, potentially followed by final Mr. TyDi fine-tuning.
Loading 2112.09118v4…