Source-linked AI summary
BioSentVec: creating sentence embeddings for biomedical texts
Qingyu Chen, Yifan Peng, Zhiyong Lu
TL;DR
Biomedical and clinical NLP lacked publicly available domain-specific sentence embeddings for semantic modeling. BioSentVec trains embeddings on PubMed and MIMIC-III text and evaluates them on sentence similarity and multi-label classification. It achieves the best reported performance in both tasks and has been applied to sentence-level biomedical search.
Problem
Biomedical and clinical NLP lacked publicly available domain-specific sentence embeddings, leaving researchers to train them independently or use general-domain alternatives.
Method
BioSentVec applies sent2vec to PubMed articles and MIMIC-III clinical notes, then evaluates the resulting embeddings in sentence similarity and multi-label classification tasks.
Results
BioSentVec achieved the highest performance among evaluated methods in both sentence similarity and multi-label classification.
Takeaways & Limitations
BioSentVec is publicly available and has been adapted for sentence-level search across large PubMed and PMC collections.
Abstract
from arXiv · showhide
Sentence embeddings have become an essential part of today's natural language processing (NLP) systems, especially together advanced deep learning methods. Although pre-trained sentence encoders are available in the general domain, none exists for biomedical texts to date. In this work, we introduce BioSentVec: the first open set of sentence embeddings trained with over 30 million documents from both scholarly articles in PubMed and clinical notes in the MIMIC-III Clinical Database. We evaluate BioSentVec embeddings in two sentence pair similarity tasks in different text genres. Our benchmarking results demonstrate that the BioSentVec embeddings can better capture sentence semantics compared to the other competitive alternatives and achieve state-of-the-art performance in both tasks. We expect BioSentVec to facilitate the research and development in biomedical text mining and to complement the existing resources in biomedical word embeddings. BioSentVec is publicly available at https://github.com/ncbi-nlp/BioSentVec
I. INTRODUCTION
Biomedical sentence semantics matter for text mining, but publicly available sentence embeddings were unavailable for biomedical and clinical domains. BioSentVec addresses this gap with domain-trained embeddings.
- Motivation: Biomedical sentence semantics are important for text mining, while bag-of-words can inadequately represent meaning under linguistic variation.Different biomedical sentences may describe similar findings despite using different words.
- Research gap: Before BioSentVec, researchers lacked publicly available sentence embeddings trained for biomedical and clinical text.They had to train embeddings themselves or use general-domain encoders vulnerable to out-of-domain issues.
- Proposed solution: BioSentVec provides pretrained sentence vectors for arbitrary input sentences using biological and clinical text at scale.The embeddings are created with sent2vec and evaluated on sentence similarity and multi-label classification.
II. METHODS AND MATERIALS
BioSentVec is trained on large biomedical and clinical corpora using sent2vec. Combining PubMed and MIMIC-III provides broad textual coverage for the embeddings.
- Training corpora: PubMed articles and MIMIC-III clinical notes together provide over 30 million documents, approximately 223 million sentences, and 5 billion tokens.The two sources cover scholarly and clinical text genres.
- Embedding training: The authors preprocess both corpora with sentence splitting and tokenization before training BioSentVec with sent2vec.sent2vec adapts Continuous Bag-of-Words to sentence-level training and extends it with sentence n-grams.
- Embedding training: The selected BioSentVec configuration produces 700-dimensional vectors using a bigram model.This setting was chosen empirically across vector dimension, window size, and negative-sample parameters.
A. Task 1: Biological and clinical sentence similarity
The first task evaluates BioSentVec for sentence similarity across biomedical and clinical datasets. Both unsupervised and supervised approaches are compared using Pearson correlation with annotated similarity scores.
- Datasets: BIOSSES contains 100 PubMed sentence pairs, while MedSTS contains 1,068 clinical-note sentence pairs.MedSTS uses 750 training pairs and 318 test pairs; BIOSSES is evaluated with 10-fold stratified cross-validation.
- Datasets: Similarity labels range continuously from 0 to 5 by averaging curator annotations for each sentence pair.The underlying annotation categories range from not similar to very similar.
- Compared methods: The study compares prior unsupervised baselines, averaged biomedical word vectors, pretrained sentence encoders, and a supervised neural model.The supervised model combines two BioSentVec vectors, their absolute differences, multiplications, and dot product before fully connected layers.
- Evaluation: Pearson correlation with the gold standard is used for evaluation across both datasets and modeling settings.BIOSSES uses stratified cross-validation, whereas MedSTS reports performance on its official test set.
B. Task 2: Biomedical multi-label text classification
The second task tests sentence embeddings for multi-label cancer-hallmark classification. A CNN combines word-level and sentence-level representations to predict ten labels.
- Dataset: The Hallmarks of Cancer corpus contains 14,919 manually annotated sentences with one or more of ten hallmark labels.The data are split randomly into 60% training, 20% development, and 20% test sets.
- Model: The CNN takes word and sentence embeddings as input and outputs a ten-element probability vector for the cancer hallmarks.The architecture uses convolution, max pooling, and fully connected layers.
- Model: Each word representation concatenates its own embedding with part-of-speech and chunk features before convolution and pooling.The resulting local and global features are combined with sentence-generated embeddings for classification.
- Evaluation: Six models are evaluated, including a CNN without sentence embeddings and five alternatives using different sentence embeddings.Example-based precision, recall, and F1-score measure multi-label classification performance.
III. RESULTS AND DISCUSSIONS
BioSentVec achieved the strongest reported performance across sentence similarity and multi-label text classification, with benefits from combining PubMed and clinical-note training data. Its practical use in LitSense and accompanying notebook support adoption, while the study lacks a biomedical sentence-embedding baseline.
- Sentence similarity: BioSentVec achieved the highest performance in sentence similarity using both supervised and unsupervised methods.It outperformed averaged word vectors and sentence vectors trained from the general domain.
- Training sources: Training on both PubMed and clinical notes generally improved performance over either single source, while clinical-notes-only embeddings were insufficient for PubMed applications.The authors report one exception to the general improvement pattern.
- Multi-label text classification: BioSentVec trained on both PubMed and MIMIC-III achieved the best results among methods for multi-label text classification.Using BioSentVec features doubled the approximately 50% F1-score achieved by the deep learning approach alone.
- Limitation: The study does not evaluate sentence embeddings trained from the biomedical domain because no publicly available biomedical sentence-embedding resource was known.Biomedical word embeddings were evaluated instead as alternatives.
- Practical application: An adapted BioSentVec version improved sentence-retrieval effectiveness in LitSense when combined with traditional term-matching methods.LitSense searches approximately 29 million PubMed abstracts and 3 million PMC full-text articles at sentence level.
- Usability: A Jupyter Notebook provides preprocessing, model-application, and sentence-similarity examples for using BioSentVec.The notebook summarizes fundamental usage steps and includes example functions.
IV. CONCLUSION
The paper introduced publicly available sentence embeddings pretrained on biomedical and clinical corpora and demonstrated superior benchmarking performance. The authors anticipate that BioSentVec will support biomedical deep-learning and text-mining applications while planning broader real-world evaluation.
- Conclusion: The authors introduced BioSentVec embeddings pretrained on two biomedical and clinical corpora and demonstrated superior benchmarking performance.Embeddings trained from different sources and the deep learning models were made publicly available.