Source-linked AI summary
Clinical Concept Embeddings Learned from Massive Sources of Multimodal Medical Data
Andrew L. Beam, Benjamin Kompa, Allen Schmaltz, Inbar Fried, Griffin Weber, Nathan P. Palmer, Xu Shi, Tianxi Cai, Isaac S. Kohane
TL;DR
Medical embedding research lacks broad pre-trained resources because healthcare data are private and multimodal. The paper develops cui2vec by combining claims, clinical notes, and full-text biomedical articles in a common concept space, achieving state-of-the-art performance in many instances and releasing the embeddings for reuse.
Problem
Healthcare lacks pre-trained medical resources because large non-imaging datasets are difficult to access and healthcare data are too varied for standard text embedding algorithms to apply directly.
Method
cui2vec combines claims, clinical notes, and full-text biomedical articles by mapping concepts into a common co-occurrence space and embedding them together.
Results
cui2vec achieves state-of-the-art performance in many instances relative to previous results and provides embeddings for 108,477 medical concepts.
Takeaways & Limitations
The downloadable cui2vec embeddings are intended to reduce barriers created by unlabeled or weakly labeled and difficult-to-share healthcare data.
Takeaways & Limitations
Ranking-based evaluation can penalize embeddings when genuinely related concepts are absent from the database of known relationships.
Abstract
from arXiv · showhide
Word embeddings are a popular approach to unsupervised learning of word relationships that are widely used in natural language processing. In this article, we present a new set of embeddings for medical concepts learned using an extremely large collection of multimodal medical data. Leaning on recent theoretical insights, we demonstrate how an insurance claims database of 60 million members, a collection of 20 million clinical notes, and 1.7 million full text biomedical journal articles can be combined to embed concepts into a common space, resulting in the largest ever set of embeddings for 108,477 medical concepts. To evaluate our approach, we present a new benchmark methodology based on statistical power specifically designed to test embeddings of medical concepts. Our approach, called cui2vec, attains state-of-the-art performance relative to previous methods in most instances. Finally, we provide a downloadable set of pre-trained embeddings for other researchers to use, as well as an online tool for interactive exploration of the cui2vec embeddings
1. Introduction
Word embeddings efficiently represent sparse data and support transfer learning, but healthcare lacks comparable pre-trained resources because data are private and multimodal. This work addresses that gap by constructing cui2vec from large multimodal healthcare sources.
- Word embeddings provide compact representations of sparse, high-dimensional data and are widely used in machine learning and NLP.
- Transfer learning pretrains embeddings on massive unlabeled text and reuses them as inputs for supervised models.
- Privacy concerns limit researchers’ access to large non-imaging healthcare datasets, leaving fewer pre-trained medical resources than in NLP and other fields.
- The work constructs cui2vec by combining extremely large sources of multimodal healthcare data into embeddings for medical concepts.
2. Overview of word2vec and GloVe
word2vec and GloVe can be understood through co-occurrence statistics and matrix factorization, enabling their application beyond text. The paper uses this framework to construct embeddings from multimodal medical concept co-occurrences.
- word2vec: Skip-gram learns target and context vectors so frequently co-occurring pairs receive high conditional probability.
- word2vec: The skip-gram softmax models the probability of observing a context word within a fixed window given a target word.
- word2vec: Negative sampling approximates the intractable softmax denominator, enabling bounded-memory, parallel training on very large corpora.
- GloVe: GloVe fits a weighted log-linear model to word-context co-occurrence statistics using stochastic-gradient-descent least-squares optimization.
- Embeddings as a Factorization of a Modified Co-occurrence Matrix: SGNS implicitly factorizes a shifted positive PMI matrix, so SPPMI factorization with methods such as SVD yields lower-dimensional word embeddings.
- Embeddings as a Factorization of a Modified Co-occurrence Matrix: Because both methods begin with co-occurrence matrices, the framework extends them to non-textual data where co-occurrences can be materialized.
Related Work
Prior clinical concept embeddings used word2vec-like methods and achieved mixed results, while later work established strong human-assessment performance. This paper expands scale, unifies sources, and broadens evaluation.
- Early medical concept embedding studies using word2vec reported mixed results, although De Vine et al. achieved state-of-the-art human-assessment performance.
- Choi et al. created UMLS CUI embeddings from a 4-million-patient claims database and clinical notes.
- Compared with prior work, this study uses larger claims and full-text article collections to produce a larger embedding set.
- Unlike source-specific embeddings, the paper maps concepts into one common co-occurrence space for tasks involving different clinical data types.
- The study introduces an expanded evaluation methodology and reports embeddings for 108,477 medical concepts.
3. Materials and Methods
The study combines claims, clinical-note, and biomedical-article data, normalizes textual concepts to UMLS CUIs, and prepares a unified representation for medical concept embeddings.
- The claims source contains data from 60 million members of a nationwide US health insurance plan collected during 2008–2015.
- The clinical-note source contains concept co-occurrences from 20 million Stanford notes.
- The biomedical-text source is an open-access collection of 1.7 million full-text articles from PubMed Central.
- Text is normalized by collapsing multi-word concepts with NILE and mapping SNOMED-CT identifiers to UMLS concept unique identifiers.
- The source description includes a clinical sentence about bronchopulmonary dysplasia and its normalized CUI representation.
Benchmarks and Evaluation
The study evaluates medical concept embeddings against curated and human-assessed relationships using cosine similarity and a statistical-power framework, with standardized comparison settings for baseline methods.
- Evaluation methodology: Cosine similarity compares vectors for concept pairs, while known relationships determine whether the embeddings identify related concepts.The evaluation uses established concept relationships rather than relying only on unsupervised similarity rankings.
- Evaluation methodology: Statistical power is estimated by comparing each known relationship’s observed cosine score with a 10,000-sample bootstrap null distribution.A relationship is significant when its score exceeds the null distribution’s 95th percentile; reported power estimates the fraction of true relationships discovered at a 5% false-positive rate.
- Benchmark suite: The benchmark suite includes comorbid conditions, causative relationships, drug-treatment and prevention relationships, semantic types, and physician-assessed concept similarity.These benchmarks draw on curated clinical resources, UMLS relationships, NDF-RT, UMLS semantic types, and resident-physician judgments.
- Benchmark suite: The comorbidity benchmark uses curated relationships for Addison’s disease, autism, heart disease, obesity, schizophrenia, and type 1 and type 2 diabetes.Sources include the Mayo Clinic’s Encyclopedia of Diseases and Conditions, Wikipedia, and the Merck Manuals.
- Implementation details: Baseline experiments standardize word2vec settings, smooth singleton frequencies with exponent α = 0.75, and construct SVD embeddings from symmetrically scaled word and context vectors.Additional comparisons use gensim word2vec, text2vec GloVe, and raw co-occurrence SVD reported as PCA.
4. Results
The study selected 500-dimensional word2vec embeddings as cui2vec, estimated embeddings for 108,477 concepts, and found stronger performance than prior embeddings across most benchmarks.
- Benchmark Results: 500-dimensional word2vec achieved the highest performance across nearly all benchmarks and was selected for the combined-data cui2vec embeddings.Embedding dimension had only a modest effect on power-based benchmarks.
- Combined-data embeddings: 108,477 unique medical concepts were embedded from the combined data, with 16,299 concepts appearing in multiple sources.Most concepts appeared in only one corpus.
- Comparison to previous results: cui2vec again performed better in most benchmarks against clinical-note embeddings, with large margins in some cases.The comparison used 21,789 concepts shared between the two embedding sets.
- Comparison to previous results: cui2vec was uniformly better across all benchmarks against PubMed-abstract embeddings, with a huge relative improvement.This comparison used 33,376 concepts in common.
- Discussion: The study reports state-of-the-art performance in many instances and presents cui2vec as a broadly useful resource for clinically oriented machine-learning tasks.The embeddings were created from extremely large, multimodal medical-data sources.