Source-linked AI summary
MedCPT: Contrastive Pre-trained Transformers with Large-scale PubMed Search Logs for Zero-shot Biomedical Information Retrieval
Qiao Jin, Won Kim, Qingyu Chen, Donald C. Comeau, Lana Yeganova, W. John Wilbur, Zhiyong Lu
TL;DR
Biomedical information retrieval lacks abundant query–article annotations for training semantic models. MedCPT uses large-scale PubMed click logs for contrastive training of an integrated retriever and re-ranker, achieving state-of-the-art performance across six biomedical IR tasks.
Problem
Biomedical semantic retrieval is limited by scarce query–article annotations, motivating methods that can learn from readily available biomedical search data.
Method
MedCPT contrastively trains an integrated biomedical retriever and re-ranker using large-scale PubMed user click logs.
Results
MedCPT achieves the highest performance across six biomedical information retrieval tasks and produces strong article and sentence representations.
Takeaways & Limitations
PubMed search logs can provide high-quality training data for general-purpose biomedical information needs, with representations useful beyond retrieval.
Takeaways & Limitations
MedCPT is less controllable and explainable than sparse retrievers, and may return semantically similar but lexically mismatched articles.
Abstract
from arXiv · showhide
Information retrieval (IR) is essential in biomedical knowledge acquisition and clinical decision support. While recent progress has shown that language model encoders perform better semantic retrieval, training such models requires abundant query-article annotations that are difficult to obtain in biomedicine. As a result, most biomedical IR systems only conduct lexical matching. In response, we introduce MedCPT, a first-of-its-kind Contrastively Pre-trained Transformer model for zero-shot semantic IR in biomedicine. To train MedCPT, we collected an unprecedented scale of 255 million user click logs from PubMed. With such data, we use contrastive learning to train a pair of closely-integrated retriever and re-ranker. Experimental results show that MedCPT sets new state-of-the-art performance on six biomedical IR tasks, outperforming various baselines including much larger models such as GPT-3-sized cpt-text-XL. In addition, MedCPT also generates better biomedical article and sentence representations for semantic evaluations. As such, MedCPT can be readily applied to various real-world biomedical IR tasks.
Sparse retrievers · Dense retrievers · Large language model retrievers
The large language model retriever baselines span OpenAI cpt-text-S through cpt-text-XL, with parameter counts from 300M to 175B and differing reported evaluation values.
- Large language model retrievers: OpenAI cpt-text-S has 300M parameters and reports 0.679, 0.332, -, 0.672, -, -.
- Large language model retrievers: OpenAI cpt-text-M has 1.20B parameters and reports 0.585, 0.367, -, 0.704, -, -.
- Large language model retrievers: OpenAI cpt-text-L has 6.00B parameters and reports 0.562, 0.380, -, 0.744, -, -.
- Large language model retrievers: OpenAI cpt-text-XL has 175B parameters and reports 0.649, 0.407, -, 0.754, -, -.
MedCPT
MedCPT generalizes from PubMed query-article click logs to state-of-the-art biomedical retrieval and representation tasks. Its encoders achieve strong article and sentence similarity performance, while semantic retrieval remains less controllable and explainable than sparse retrieval.
- Article representations: The MedCPT article encoder outperforms all other models, including SPECTER and SciNCL, on the RELISH article similarity task.RELISH contains 196k expert-annotated article-article relevance annotations for 3.2k query articles.
- Sentence representations: 0.893 vs. 0.847: On BIOSSES, MedCPT performs best among compared models, surpassing SciNCL by 5% relative performance.The query encoder is evaluated on BIOSSES and MedSTS sentence-similarity datasets.
- Discussions: PubMed query-article pairs generalize as high-quality training data for general-purpose biomedical information needs, despite MedCPT using only click data.The encoders also achieve state-of-the-art performance on sentence and article similarity without explicit training on those data types.
- Limitations: MedCPT can return more comprehensive results than sparse retrievers such as BM25, but its semantic results and similarity scores are less controllable and explainable.For example, a search for “MAP3K3” may return articles containing only “MAP3K7”.
- Discussions: MedCPT achieves the highest performance on six biomedical information retrieval tasks, including query-to-article retrieval and semantic article and sentence representation.The model is trained contrastively using large-scale PubMed logs as an integral retriever-reranker.
Appendix A: MedCPT Inference
MedCPT inference uses an offline corpus-encoding stage followed by query encoding, maximum inner product search, and cross-encoder re-ranking. The implementation uses PyTorch and Hugging Face Transformers with a 768-dimensional hidden representation and specified Adam optimization settings.
- Inference pipeline: Each query is encoded by QEnc, then MIPS retrieves the top-K most similar articles from the saved corpus representations.MIPS denotes maximum inner product search.
- Inference pipeline: The CrossEnc scores relevance between the query and retrieved article candidates, which are sorted by relevance from highest to lowest.The sorted articles are returned as the final results.
- Inference pipeline: Corpus articles are encoded once offline with DEnc, and their representations are saved for downstream zero-shot retrieval.The saved corpus matrix is denoted 𝒟 ∈ ℝH×&.
- Implementation: MedCPT is implemented with PyTorch and Hugging Face Transformers using a 768-dimensional hidden representation, Adam optimization, learning rate 2e-5, and epsilon 1e-8.Training uses no weight decay; the retriever uses B=32, α=0.8, and gradient accumulation over 8 steps.
Appendix B: Compared methods
Appendix B groups the compared retrieval methods into sparse retrievers, dense retrievers, and large language model retrievers. The comparison spans lexical matching methods, embedding-based and BERT-based models, and larger T5- and GPT-3-based encoders.
- Sparse retrievers: Sparse retrievers match queries and documents through overlapping terms, including BM25, DeepCT, and SPARTA.BM25 uses bag-of-words representations with term-frequency and inverse-document-frequency scoring; DeepCT predicts contextualized term weights with BERT, while SPARTA precomputes contextualized matching weights.
- Dense retrievers: Dense retrievers encode queries and documents as low-dimensional dense vectors before nearest-neighbor search.The methods are broadly classified into non-BERT embedding models and BERT-based dense retrievers.
- Dense retrievers: Non-BERT comparisons include BioWordVec, FastText, Sent2vec, BioSentVec, LDA, and doc2vec, alongside biomedical-domain BERT models.The biomedical-domain BERT models include PubMedBERT, BioBERT, SPECTER, and SciNCL.
- Large language model retrievers: Large language model retrievers include Google’s GTR and OpenAI’s cpt-text, which use substantially larger encoders than typical BERT-base retrievers.GTR is based on T5 and its largest variant has 4.8B parameters, while cpt-text is based on GPT-3.