Source-linked AI summary
Multi-layer Representation Learning for Medical Concepts
Edward Choi, Mohammad Taha Bahadori, Elizabeth Searles, Catherine Coffey, Jimeng Sun
TL;DR
Healthcare EHRs contain both sequential visit structure and within-visit code co-occurrence, creating a challenge for efficient, interpretable representations. Med2Vec learns code- and visit-level representations from large EHR datasets, and experiments report improved performance over popular baselines alongside clinically meaningful interpretation.
Problem
Healthcare representation learning must capture temporally ordered visits and unordered within-visit code sets while remaining interpretable and scalable.
Method
Med2Vec is a scalable two-layer neural network that learns lower-dimensional code and visit representations using both code co-occurrence and visit sequence information.
Results
Med2Vec demonstrated superior performance in two predictive tasks, scaling to nearly 30K codes across datasets with 3 million and 5.5 million visits.
Takeaways & Limitations
The learned representations support healthcare applications while providing clinical interpretation validated through clinical experts.
Takeaways & Limitations
Code-representation conformity to medical code groupers does not necessarily measure hidden relationships, so grouper evaluation is recommended for comparisons rather than absolute performance.
Abstract
from arXiv · showhide
Learning efficient representations for concepts has been proven to be an important basis for many applications such as machine translation or document classification. Proper representations of medical concepts such as diagnosis, medication, procedure codes and visits will have broad applications in healthcare analytics. However, in Electronic Health Records (EHR) the visit sequences of patients include multiple concepts (diagnosis, procedure, and medication codes) per visit. This structure provides two types of relational information, namely sequential order of visits and co-occurrence of the codes within each visit. In this work, we propose Med2Vec, which not only learns distributed representations for both medical codes and visits from a large EHR dataset with over 3 million visits, but also allows us to interpret the learned representations confirmed positively by clinical experts. In the experiments, Med2Vec displays significant improvement in key medical applications compared to popular baselines such as Skip-gram, GloVe and stacked autoencoder, while providing clinically meaningful interpretation.
1 Introduction
Med2Vec addresses healthcare representation learning by modeling the structured relationships in EHR visits while targeting interpretability and scalability. The paper evaluates its representations in healthcare applications and validates their clinical interpretability.
- Motivation: Healthcare feature engineering often requires expert knowledge and ad-hoc design, limiting scalability and failing to represent relationships between medical codes.One-hot coding treats related diagnoses such as pneumonia and bronchitis as unrelated.
- Challenges: EHR data combine temporally ordered visits with unordered sets of medical codes within each visit.Aggregating code-level representations alone cannot capture sequential relationships among visits.
- Challenges: Learned healthcare representations must be interpretable, although some state-of-the-art methods such as RNNs are difficult to interpret.The paper frames interpretability as an essential requirement in the clinical domain.
- Approach: Med2Vec learns succinct code- and visit-level representations from real-world EHR data without depending on expert medical knowledge.The proposed algorithm is described as simple and robust.
- Evaluation: The study validates the interpretability of Med2Vec representations through a detailed user study with clinical experts.The contribution emphasizes clinically meaningful interpretation in addition to performance.
- Evaluation: Med2Vec scales to nearly 30K medical codes across datasets containing 3 million and 5.5 million visits.Experiments assess scalability and application to real-world health problems against popular baselines.
2 Preliminaries and Related Work
Prior work learns representations for words, paragraphs, and healthcare concepts using co-occurrence or longitudinal records. These approaches motivate Med2Vec but leave challenges around jointly representing structured visits and interpretability.
- Word Representation Learning: Skip-gram learns word vectors from co-occurrence within a context window by predicting neighboring words.Its objective maximizes the average log probability of context words given a center word.
- Word Representation Learning: GloVe learns representations from a global word co-occurrence matrix and can be computationally less demanding than Skip-gram when that matrix is sparse.Its weighting function may require considerable tuning effort.
- Hierarchical Representations: Hierarchical representation methods jointly learn representations for multiple text levels, such as paragraphs and words.Paragraph vectors are treated as part of the word-learning process, but the algorithm assigns fixed vectors to training paragraphs and words.
- Healthcare Representation Learning: Medical text studies apply Skip-gram to medical terms or UMLS concepts extracted from biomedical and patient-record text.These studies do not study longitudinal structured visit sequences.
- Healthcare Representation Learning: Structured visit-record studies learn medical-code representations with Skip-gram and aggregate them to form visit representations for prediction.RNNs instead model longitudinal records in temporal order but are difficult to interpret.
3 Method
Med2Vec learns non-negative code representations and visit representations from structured EHR data by combining intra-visit code co-occurrence with inter-visit sequence information. Its architecture incorporates demographic information and supports interpretation of representation dimensions through their strongest associated codes.
- 3.1 Med2Vec architecture: Each visit becomes a binary code vector, which is transformed into an intermediate representation and combined with demographic information to produce the final visit representation.The final representation is trained to predict medical codes in neighboring visits.
- 3.2 Learning from the visit-level information: Med2Vec learns visit representations from sequential context by training them to predict medical codes occurring in nearby visits.The visit-level objective captures inter-visit sequence information, while the code-level objective captures co-occurrence within visits.
- 3.3 Learning from the code-level information: The model uses ReLU to produce non-negative and sparse code representations, improving their interpretability relative to representations with positive and negative values.When the input is one-hot, the intermediate visit representation corresponds to a code representation.
- 3.4 Unified training: The unified objective jointly exploits intra-visit code co-occurrence and inter-visit sequential information from the same patient visit records.This joint training learns code and visit representations in one framework.
- 3 Method: Med2Vec learns both code-level and visit-level representations from longitudinal EHR visit sequences.The code embedding maps medical codes to non-negative m-dimensional vectors, while the visit embedding maps code sets to n-dimensional vectors.
- 3.5 Interpretation of learned representations: Representation dimensions are interpreted by identifying the codes with the strongest values, allowing coordinates to be associated with disease groups.Visit coordinates are interpreted through the code coordinates that contribute most strongly, enabled by the non-negative intermediate representation.
4 Experiments
Experiments evaluate Med2Vec’s code- and visit-level representations on CHOA and CMS datasets using expert judgments, grouper-based clustering, prediction tasks, convergence, and scalability analyses. Med2Vec generally performs strongly while maintaining interpretability, and grouped codes reduce computational cost without materially changing CRG prediction.
- Datasets and evaluation: Med2Vec is evaluated on CHOA and CMS datasets containing diagnosis, medication, and procedure codes, with CMS used specifically for scalability testing.CHOA supplies the main evaluation setting, while CMS differs in patient, code, and visit composition.
- Code-level evaluation: Medical experts assess whether nearest code representations capture clinically related diagnoses, medications, and procedures.The relatedness test selects 100 frequent diagnosis codes and five nearest concepts for expert labeling.
- Code-level evaluation: Med2Vec captures medical-code relationships, although medication representations can produce superficially incorrect but potentially deeper clinical associations.The paper illustrates this with Ofloxacin and sensorineural hearing loss.
- Code-level evaluation: GloVe achieves the strongest conformity to clinical groupers, while Med2Vec is roughly comparable to Skip-gram and SVD is weakest.The authors caution that grouper conformity does not necessarily measure hidden clinical relationships.
- Visit-level evaluation: Med2Vec achieves the highest future-code Recall@30 across experiments and remains robust across a wide range of hyperparameters.Including visit information improves and stabilizes performance relative to the more volatile Skip-gram results; larger representation sizes can cause overfitting.
- Visit-level evaluation: Grouped-code training preserves nearly the same CRG-prediction AUC while improving training speed by reducing softmax computation.The comparison indicates that grouping alleviates the curse of dimensionality with only a marginal difference in CRG prediction AUC.
- Convergence and scalability: All models except the single-layer stacked autoencoder converge after about 10 epochs, while Med2Vec converges similarly to or better than Skip-gram despite greater complexity.On CHOA, Skip-gram and Med2Vec take longer per epoch than the stacked autoencoder and GloVe because of softmax training.
- Convergence and scalability: On CMS, Med2Vec and Skip-gram take training times similar to the stacked autoencoder because the smaller number of codes per visit dominates computation.GloVe is faster with fewer unique codes, whereas the stacked autoencoder takes longer because CMS has more visits.
5 Interpretation
Med2Vec’s interpretability analysis links code and visit coordinates to clinically meaningful patterns and uses those representations to identify influential factors in CRG prediction. The paper combines coordinate-level inspection with an approximate activation-maximization method and expert-oriented clinical interpretation.
- Code and visit representation interpretation: Med2Vec annotates embedding dimensions by examining the top five medical codes associated with each code coordinate.This provides a coordinate-level view of the clinical concepts represented in the code embedding space.
- CRG prediction interpretation: Equation (5) approximately identifies the code coordinate that maximizes output activation, enabling influential code coordinates to be found without analyzing the visit representation.The approximation removes the ReLU term because ReLU is increasing, and the resulting vector represents each code coordinate’s maximum possible influence on CRG prediction.
- Code and visit representation interpretation: Table 5 reports ten medical codes with the strongest values for each of six coordinates in the 200-dimensional code embedding space.The table uses ICD9 codes for shortened diagnoses and marks medications and procedures with (R) and (P).
- Code and visit representation interpretation: Coordinates 112 and 152 group clinically related codes, including sickle-cell disease with organ transplant and sports-related injuries.The analysis also connects blood in feces and joint pain through Henoch–Schönlein purpura.
- Code and visit representation interpretation: Visit coordinate 50 is strongly activated by sickle-cell disease and sports-related injuries through its strongest connections to code coordinates 112 and 152.Visit coordinate 41 is also among the coordinates most strongly connected to the CRG logistic-regression model.
- CRG prediction interpretation: Code coordinates 190 and 199 are the strongest CRG influencers, corresponding to congenital chromosome anomalies and congenital paralysis associated with severe states.The method uses logistic-regression weights together with each coordinate’s maximum activation ability rather than connection strength alone.
6 Conclusion
The paper proposes Med2Vec as a scalable neural network for lower-dimensional medical concept representations. It combines code co-occurrence and visit sequence information, improves code and visit representations, and achieves superior performance in two predictive tasks while supporting clinical interpretation.
- Conclusion: Med2Vec is a scalable two-layer neural network for learning lower-dimensional representations of medical concepts.The model represents both medical codes and visits in longitudinal EHR data.
- Conclusion: Med2Vec incorporates code co-occurrence and visit sequence information to improve the accuracy of code and visit representations.These information sources correspond to relationships within visits and across the ordered sequence of visits.
- Conclusion: Med2Vec demonstrates superior performance in two predictive tasks and provides clinical interpretation of the learned representations.The conclusion presents predictive performance and interpretability as complementary outcomes of the approach.