PaperlayerView exact revision on arXiv

Source-linked AI summary

GPL: Generative Pseudo Labeling for Unsupervised Domain Adaptation of Dense Retrieval

Kexin Wang, Nandan Thakur, Nils Reimers, Iryna Gurevych

arXiv:2112.07577v3cs.CLcs.IR

TL;DR

Dense retrievers require substantial training data and are sensitive to domain shifts, limiting their use in specialized domains. GPL adapts them without labeled target-domain data by generating queries and cross-encoder pseudo-labels, improving retrieval across six domain-specific datasets by up to 9.3 nDCG@10 points over an MS MARCO-only model.

  • Problem

    Dense retrievers require large training datasets and perform poorly under domain shifts, limiting their use across specialized domains.

  • Method

    GPL generates target-domain queries and trains dense retrievers on cross-encoder pseudo-labels for generated query–passage pairs.

  • Results

    GPL improves performance across six domain-specific datasets, by up to 9.3 points nDCG@10 over an MS MARCO-only state-of-the-art model.

  • Takeaways & Limitations

    GPL provides an easy, fast, and data-efficient approach to unsupervised domain adaptation for dense retrieval.

  • Takeaways & Limitations

    GPL assumes access to an unlabeled target-domain passage collection and pre-existing T5 and cross-encoder models trained on MS MARCO.

Abstract

from arXiv · show

Dense retrieval approaches can overcome the lexical gap and lead to significantly improved search results. However, they require large amounts of training data which is not available for most domains. As shown in previous work (Thakur et al., 2021b), the performance of dense retrievers severely degrades under a domain shift. This limits the usage of dense retrieval approaches to only a few domains with large training datasets. In this paper, we propose the novel unsupervised domain adaptation method Generative Pseudo Labeling (GPL), which combines a query generator with pseudo labeling from a cross-encoder. On six representative domain-specialized datasets, we find the proposed GPL can outperform an out-of-the-box state-of-the-art dense retrieval approach by up to 9.3 points nDCG@10. GPL requires less (unlabeled) data from the target domain and is more robust in its training than previous methods. We further investigate the role of six recent pre-training methods in the scenario of domain adaptation for retrieval tasks, where only three could yield improved results. The best approach, TSDAE (Wang et al., 2021) can be combined with GPL, yielding another average improvement of 1.4 points nDCG@10 across the six tasks. The code and the models are available at https://github.com/UKPLab/gpl.

1 Introduction

Dense retrieval addresses the lexical gap but requires substantial training data and is highly sensitive to domain shifts. GPL adapts dense retrievers without labeled target-domain data by generating queries, mining negatives, and assigning cross-encoder pseudo-labels, improving results across six domain-specific datasets.

  • Motivation: Dense retrieval maps queries and passages into a shared vector space for nearest-neighbor search, addressing lexical methods’ inability to recognize synonyms and disambiguate words.Traditional lexical methods suffer from the lexical gap.
  • Motivation: Dense retrieval requires large training datasets and is extremely sensitive to domain shifts, causing MS MARCO-trained models to perform poorly on COVID-19 scientific questions.MS MARCO predates COVID-19 and lacks related topics.
  • Generative Pseudo Labeling: GPL generates target-domain queries, mines similar passages as negatives, and uses a cross-encoder to pseudo-label query-passage pairs for training a domain-adapted dense retriever.The method uses publicly available models for query generation, negative mining, and cross-encoder scoring.
  • Results: 9.3 points nDCG@10 is GPL’s maximum improvement over a state-of-the-art model trained solely on MS MARCO across six representative domain-specific BeIR datasets.The evaluation uses six representative domain-specific datasets from the BeIR benchmark.
  • Pretraining Analysis: 2.5 points nDCG@10 is TSDAE’s average advantage over Masked Language Modeling, while combining TSDAE with GPL yields another average improvement of 1.4 point nDCG@10.The comparison covers six recent pretraining methods for domain adaptation in retrieval tasks.

2 Related Work

Prior work explored pre-training, query generation, domain-adversarial adaptation, and cross-encoder knowledge transfer for dense retrieval. This work extends query generation with cross-encoder pseudo-labels and hard negatives, while testing pre-training for unsupervised domain adaptation.

  • Pre-Training based Domain Adaptation: Domain-adaptive pre-training continues transformer pre-training on in-domain data before supervised fine-tuning, but retrieval often lacks in-domain labeled data.Dense-retrieval-specific strategies include ICT, ConDensor, SimCSE, CT, and TSDAE.
  • Pre-Training based Domain Adaptation: Whether these pre-training approaches support unsupervised domain adaptation for dense retrieval was unclear; this work pre-trains on the target corpus, then trains on labeled MS MARCO data.Performance is measured on the target corpus.
  • Query Generation: Query-generation methods improve retrieval: Doc2query expands passages with predicted queries for BM25, while QGen synthesizes domain-targeted queries from general-domain training.The passage identifies Doc2query’s strong performance on the zero-shot BeIR benchmark and QGen’s target-corpus setup.
  • Query Generation: Several points nDCG@10 can be gained by extending query generation with cross-encoder pseudo-labels and hard negatives beyond in-batch cross-entropy negatives.Previous query-generation methods provide coarse-grained relevance through in-batch negatives.
  • Other Methods: MoDIR applies domain-adversarial training to generate domain-invariant representations, but minimizing cross-domain representation distance can produce poor embedding spaces and unstable performance.Cross-encoders generally perform better and are less prone to domain shifts, but their computational overhead limits production suitability.

3 Method

GPL adapts dense retrievers to a target domain using only unlabeled passages by generating synthetic queries, retrieving negatives, and training on cross-encoder-derived score margins. Its MarginMSE objective addresses noisy queries and false negatives by soft-labeling query–passage pairs.

  • GPL pipeline: GPL generates three queries per target passage, retrieves 50 negative passages for each query, and forms query–positive–negative tuples for training.Queries are generated with a T5 encoder-decoder, and pre-existing dense retrieval is slightly more effective than BM25 for retrieving negatives.
  • GPL pipeline: The method requires only an unlabeled target-domain passage collection and pre-existing T5 and cross-encoder models trained on MS MARCO.This enables adaptation without labeled target-domain training data.
  • MarginMSE loss: MarginMSE improves on coarse relevance labeling by assigning soft scores that account for unanswerable generated queries and potentially relevant negatives.A low cross-encoder score reduces pressure to align a badly generated query with its passage, while a high score prevents forcing a false negative far away.
  • MarginMSE loss: MarginMSE trains the dense retriever to mimic the cross-encoder’s score margin between positive and negative query–passage pairs.The student margin is computed from the difference between positive and negative embedding dot products.

4 Experiments

Experiments evaluate dense retrievers and domain-adaptation methods on six specialized BeIR retrieval tasks, using MS MARCO as the source domain and nDCG@10 for evaluation. The comparison includes zero-shot, prior adaptation, pre-training, generation-based, and cross-encoder re-ranking approaches.

  • Datasets: MS MARCO provides the source-domain training data, with 8.8M passages and 532.8K labeled relevant query-passage pairs.Its state-of-the-art dense retriever achieves 33.2 MRR@10 on MS MARCO but performs poorly on the selected domain-specific datasets relative to BM25.
  • Datasets: The six target tasks cover financial, scientific, biomedical, COVID-19, StackExchange, and news domains.They are FiQA, SciFact, BioASQ, TREC-COVID, CQADupStack, and Robust04.
  • Evaluation: Table 1 evaluates methods with nDCG@10 and reports that GPL significantly outperforms other domain-adaptation methods.The table also investigates TSDAE pre-training for domain adaptation and finds significant improvements for both QGen and GPL.

5 Results

TSDAE is the strongest pre-training method, while GPL outperforms the tested domain-adaptation methods and reaches a new average state of the art when combined with TSDAE. Despite not fully closing the cross-encoder gap, TSDAE + GPL offers substantially lower inference cost.

  • Pre-Training based Domain Adaptation: TSDAE improves over the zero-shot MS MARCO model by 4.0 points nDCG@10 on average, whereas CD, CT, and SimCSE perform worse.MLM and ICT also improve performance when target-corpus pre-training precedes supervised MS MARCO training.
  • Pre-Training based Domain Adaptation: Without supervised MS MARCO training, all tested pre-training methods fail miserably.These results are reported in Appendix E.
  • Generative Pseudo Labeling (GPL, proposed method): GPL outperforms QGen by up to 4.5 points on BioASQ and by 2.7 points on average, with TREC-COVID as the exception.The passage notes that TREC-COVID has only 50 test queries, so the difference may be due to the small test set.
  • Generative Pseudo Labeling (GPL, proposed method): 52.9 nDCG@10 points in average is achieved by TSDAE + GPL, which improves performance on all datasets and exceeds the out-of-the-box MS MARCO model by 7.7 points on average.This combination is described as the new state-of-the-art result.
  • Re-ranking with Cross-Encoders: TSDAE + GPL narrows but does not fully close the cross-encoder performance gap, while its much lower inference cost makes it preferable for production.Cross-encoders perform well zero-shot but have significant inference costs.

6 Analysis

GPL performance saturates after about 100K training steps, while TSDAE pre-training continues improving performance throughout training. The method remains effective with limited target-domain data, requires corpus-dependent query-generation budgets, is robust to generation temperature and initialization, and can outperform the zero-shot baseline by 2.4 NDCG@10 points with more than 10K passages.

  • Training steps: GPL performance begins to saturate after around 100K training steps, whereas TSDAE pre-training improves performance consistently throughout training.Models were evaluated every 10K steps through 140K steps.
  • Corpus size: 2.4 NDCG@10 points is GPL’s improvement over the zero-shot baseline with more than 10K Robust04 passages.Performance begins to saturate with more than 50K passages.
  • Query generation: GPL requires around 50, 5, and 1 generated queries per passage on SciFact, FiQA, and Robust04, respectively, to achieve the reported optimum.Smaller corpora require more generated queries per passage than larger corpora.
  • Query generation: GPL performance peaks at a generation temperature of 1.0, while higher temperatures produce more diverse but lower-quality queries.On FiQA, the evaluation used 3 generated queries per passage across temperatures from 0.1 to 10.
  • Initialization: MS MARCO initialization changes GPL’s average performance by only 0.3 points, while QGen depends more strongly on initialization, with a 1.9-point average difference.GPL yields similar performance when starting from different checkpoints.

7 Case Study: Fine-Grained Labels

GPL’s continuous cross-encoder pseudo labels provide finer-grained information than QGen’s simple 0–1 labels. A FiQA case study shows GPL distinguishing passages by how completely they explain the generated query’s concept.

  • Case Study: Fine-Grained Labels: Continuous pseudo labels from a cross-encoder provide more fine-grained information than QGen’s simple 0–1 labels.The section examines this distinction through a case study.
  • Case Study: Fine-Grained Labels: 8.2: GPL labels a negative passage highly when it gives the required definition of “futures contract”.The generated query asks for the definition of “futures contract”.
  • Case Study: Fine-Grained Labels: 6.9: GPL assigns a medium label to a negative passage that gives only a partial explanation of “futures contract”.Other negatives merely mention the term and receive low GPL labels below 2.0.

8 Conclusion · A Performance of Using Different Retrievers for Negative Mining in GPL · B Performance of the Zero-Shot Retrievers in Hard-Negative Mining

GPL combines generated target-domain queries, cross-encoder pseudo-labeling, and hard-negative training to achieve strong domain adaptation, though its training pipeline is relatively complex. Dense retrievers are the best hard-negative miners, while zero-shot retriever choices have limited impact on GPL’s domain-adapted performance.

  • 8 Conclusion: GPL generates target-corpus queries and uses cross-encoder pseudo-labels to denoise training data and enable mined hard negatives.Pseudo-labeling addresses variable query quality and supports efficient denoising.
  • 8 Conclusion: GPL performs well across all evaluated datasets and significantly outperforms other approaches.The authors identify simplifying GPL’s relatively complex training setup as future work.
  • 8 Conclusion: 4 nDCG@10 points on average was the improvement from TSDAE, whereas ICT and MLM improved results by <=1.5 nDCG@10 points on average.The pre-training setup first adapts to the target domain and then performs supervised training on MS MARCO; other approaches degraded performance.
  • A Performance of Using Different Retrievers for Negative Mining in GPL: GPL performs best when dense retrievers mine hard negatives, compared with BM25 or BM25 + dense retrievers.Table 6 reports these comparisons using nDCG@10 performance.
  • B Performance of the Zero-Shot Retrievers in Hard-Negative Mining: The zero-shot retrievers msmarco-distilbert-base-v3 and msmarco-MiniLM-L-6-v3 are much worse than the MS MARCO MarginMSE baseline on zero-shot generalization across each dataset.Table 7 evaluates these two retrievers when used for GPL hard-negative mining.
  • B Performance of the Zero-Shot Retrievers in Hard-Negative Mining: GPL can train powerful domain-adapted dense retrievers with minimum reliance on the choice of retriever used for hard-negative mining.This conclusion follows from comparing direct use of the zero-shot retrievers with the strong MS MARCO baseline.

C Target Datasets

The experiments use six target datasets spanning financial, scientific, biomedical, community question-answering, and news retrieval. Their corpora range from 5.2K to 1M passages after dataset-specific cleanup, with binary or 3-level relevance labels.

  • FiQA: FiQA contains 648 opinion-question queries and 5.8K financial passages from Investment-topic StackExchange posts, with binary labels.Each query has 2.6 passages labeled relevant on average.
  • SciFact: SciFact contains 300 scientific-claim queries and 5.2K passages from S2ORC abstracts, with binary labels.Each query has 1.1 passages labeled relevant on average.
  • BioASQ: BioASQ covers biomedical question answering with 500 queries and a final corpus of 1M passages after removing irrelevant passages from 15M PubMed articles.The labels are binary, with 4.7 passages labeled relevant per query on average.
  • TREC-COVID: TREC-COVID is an ad-hoc scientific-article search dataset with 50 queries and a cleaned corpus of 129.2K documents from CORD-19.Its labels have three levels, with 430.8 passages labeled 1 or 2 per query on average.
  • CQADupStack: CQADupStack retrieves duplicate question posts across 12 StackExchange subforums using 13.1K queries and 457.2K passages.The labels are binary, with 1.4 passages labeled relevant per query on average.
  • Robust04: Robust04 targets poorly performing news-retrieval topics with 249 queries and 528.2K passages under 3-level relevance labels.There are 69.9 passages labeled relevant per query on average.

D Results on full BeIR

Across all 18 BeIR datasets, DocT5Query and BM25 outperform several baselines, while QGen often fails or harms performance. Applying GPL to TAS-B yields the strongest overall results, improving nDCG@10 by up to 21.5 points and 4.6 points on average.

  • Evaluation setup: 18 BeIR datasets are evaluated using DocT5Query, BM25 (Anserini), TAS-B, and TAS-B enhanced with QGen or GPL.TAS-B serves as both the base model and negative miner for the enhanced variants.
  • Baseline results: DocT5Query and BM25 (Anserini) outperform MS MARCO, TSDAE, and QGen in both average performance and average rank.QGen also significantly harms performance on datasets including TREC-COVID, FEVER, HotpotQA, and NQ.
  • GPL with TAS-B: 21.5 nDCG@10 points is the largest gain from applying GPL to TAS-B, achieved on TREC-COVID.The average gain across datasets is 4.6 nDCG@10 points.
  • GPL with TAS-B: 3.2 is the averaged rank achieved by TAS-B + GPL, the best-performing retriever model among those evaluated.Applying QGen on top of TAS-B does not improve overall performance and harms individual performance.

E Performance of Unsupervised Pre-Training · F Examples of Generated Queries under Different Temperatures

Without MS MARCO data, ICT achieves the highest scores across all datasets, although no unsupervised pre-training method improves over the zero-shot baseline. Higher query-generation temperatures produce longer, less-duplicated queries but increase the risk of nonsensical text.

  • E Performance of Unsupervised Pre-Training: ICT achieves the highest scores on all datasets among the evaluated unsupervised pre-training methods.These results are reported without access to MS MARCO data.
  • E Performance of Unsupervised Pre-Training: Table 9 reports nDCG@10 across all 18 original BeIR datasets and includes results for MS MARCO, TSDAE, QGen, TSDAE + QGen, GPL, and TSDAE + GPL.It also reports BM25, DocT5Query, BM25 + CE, and average rank.
  • E Performance of Unsupervised Pre-Training: No unsupervised pre-training method directly improves performance over the zero-shot baseline.The comparison uses methods trained without MS MARCO data.
  • E Performance of Unsupervised Pre-Training: Table 10 reports nDCG@10 for unsupervised pre-training methods trained using only the target corpus.The zero-shot MS MARCO baseline scores are copied from Table 1.
  • F Examples of Generated Queries under Different Temperatures: Generation temperature controls the sharpness of the next-token distribution.Examples are provided for one FiQA passage.
  • F Examples of Generated Queries under Different Temperatures: Higher temperatures generate longer, less-duplicated queries while increasing the risk of nonsensical text.The examples under different temperature values are shown for a passage from FiQA.
Loading 2112.07577v3…