Source-linked AI summary
HCOE: Hyperbolic Clinical Ontology Embeddings from Biomedical Language Models
Yixuan Li, Weihao Li, Ziyang Song
TL;DR
Biomedical language models encode medical-concept semantics but do not explicitly preserve clinical-code hierarchies. HCOE combines frozen LM representations with ontology-guided hyperbolic contrastive learning and coarse-to-fine path aggregation. It reports the best performance on clinical relation prediction and improved MIMIC-IV prediction results across mortality, readmission, and medication recommendation tasks.
Problem
Biomedical LMs provide transferable textual semantics but do not explicitly capture structural relations among clinical concepts organized in clinically meaningful coarse-to-fine hierarchies.
Source: §I · p. 1Method
HCOE combines frozen biomedical LM code embeddings with ontology-guided hyperbolic contrastive learning and coarse-to-fine ontology-path aggregation.
Source: §I · p. 2, B. Clinical Ontology Paths and LM… · p. 3Results
HCOE achieves the best F-score across four clinical relation prediction tasks and improves MIMIC-IV mortality, readmission, and drug recommendation metrics over the strongest baseline.
Source: A. HCOE Preserves Clinical Ontology… · p. 5, C. HCOE Improves Clinical Prediction · p. 6Takeaways & Limitations
The reported results support HCOE for clinical ontology representation and downstream EHR modeling across the evaluated tasks.
Source: §VII · p. 7The study focuses on common clinical ontologies and standard EHR prediction tasks, leaving broader biomedical knowledge sources for future exploration.
Source: §VII · p. 7
Abstract
from arXiv · showhide
Biomedical language models (LMs) encode textual semantics but do not explicitly preserve medical code hierarchies. We present Hyperbolic Clinical Ontology Embeddings (HCOE) for hierarchy-aware clinical concept representation. HCOE maps frozen BioBERT embeddings into a Poincare ball, combining parent-side and child-side ontology-guided contrastive learning with coarse-to-fine ontology-path aggregation. It uses International Classification of Diseases (ICD) codes organized by Clinical Classifications Software (CCS) and Anatomical Therapeutic Chemical (ATC) medication hierarchies. Evaluations show that HCOE performs best on ICD/ATC clinical relation prediction and CCS-to-PheCode hierarchy transfer. On the MIMIC-IV dataset, HCOE also achieves the best performance on mortality prediction, readmission prediction, medication recommendation, and rare drug prediction.
I. INTRODUCTION
Biomedical LMs provide transferable textual semantics but do not explicitly preserve clinical-code hierarchies, whose coarse-to-fine relations matter for clinical concept representation. HCOE combines biomedical LM representations with ontology-guided hyperbolic learning and path aggregation to address this gap.
I. INTRODUCTION
- Biomedical LMs offer transferable medical-concept semantics, but their pretraining does not explicitly capture structural relations among clinical concepts organized in multi-level ICD and ATC hierarchies.These ontologies encode clinically meaningful coarse-to-fine relationships.Source: §I · p. 1
- 58.6%: BioBERT accuracy on ICD parent–child prediction at the finest hierarchy level, motivating embeddings that preserve fine-grained relations.BioBERT accuracy declines as the hierarchy becomes more fine-grained; HCOE maintains higher accuracy across levels.Source: §I · p. 1
- Existing approaches motivate integrating pretrained biomedical LM representations with clinical ontologies in hyperbolic space, addressing limitations of task-specific training, hierarchy-unaware LMs, Euclidean hierarchy-aware LMs, and LM-independent hyperbolic embeddings.These limitations are described across related approaches.Source: §I · p. 1
- HCOE learns parent–child relations from CCS-organized ICD codes and ATC hierarchies with ontology-guided contrastive learning, then aggregates embeddings along coarse-to-fine ontology paths.The framework uses frozen biomedical LM representations.Source: §I · p. 2
II. RELATED WORK
Prior clinical representation methods use patient-record patterns, ontologies, biomedical language models, or hyperbolic geometry, but do not combine pretrained biomedical semantics with expert-curated clinical ontologies in hyperbolic space.
II. RELATED WORK
- Task-specific code-embedding methods use co-occurrence, longitudinal records, or ontology and graph structure, but do not fully exploit transferable knowledge from large-scale corpora.Examples include cui2vec and GRAM.Source: §II · p. 2
- Biomedical LMs provide transferable textual semantics but are not explicitly optimized for clinical-code hierarchies; hierarchy-aware LM methods use Euclidean space, while hyperbolic methods generally do not integrate pretrained LM semantics.The paper presents HCOE as integrating biomedical LM semantics and clinical ontologies in hyperbolic space.Source: §II · p. 2
III. BACKGROUND OF HYPERBOLIC REPRESENTATION
- Hyperbolic representation learning uses negatively curved space, whose exponential volume growth is described as naturally fitting tree-like structures.The paper names Poincaré embeddings, Poincaré GloVe, and entailment cones as hierarchy-preserving methods.Source: §III · p. 2
IV. METHODOLOGY
The study maps diagnosis, procedure, and medication codes to five-level ontology paths that define the parent–child relations used to train HCOE.
IV. METHODOLOGY
- Diagnoses and procedures use five-level CCS-organized ICD hierarchies, while medications use the five-level ATC hierarchy from anatomical group to chemical substance.For diagnoses, the levels run from major category through decimal-level ICD-CM code; procedure levels end in the full ICD-PCS code.Source: A. ICD and ATC Clinical Ontologies · p. 2
B. Clinical Ontology Paths and LM Initialization
HCOE represents clinical concepts within an expert-curated, multilevel ontology and uses ontology paths to combine frozen biomedical-LM embeddings into hierarchy-aware representations. Its overview also depicts hyperbolic contrastive learning that draws parent–child concepts together and separates unrelated concepts.
B. Clinical Ontology Paths and LM Initialization
- HCOE organizes clinical concepts across L ontology granularity levels, with each concept linked to an ordered ancestor path from coarse to fine.The overview depicts these as levels in an expert-curated medical ontology.Source: B. Clinical Ontology Paths and LM… · p. 2, Figure 2 · p. 3
- The overview depicts hyperbolic contrastive learning that pulls parent–child concepts closer and pushes unrelated concepts apart.For a target concept, it also depicts aggregation along the ontology path from coarse to fine.Source: Figure 2 · p. 3
- The ICD-10-CM code E11.2 follows the path [3, 3.3, 50, E11, E11.2], from endocrine disease to the specific code.The listed levels correspond to Major category, Sub-category, CCS code, integer-level ICD, and decimal-level ICD.Source: B. Clinical Ontology Paths and LM… · p. 3
- HCOE aggregates level-specific embeddings along each concept’s ontology path to form a hierarchy-aware semantic embedding.The ontology paths also provide parent–child relations for hyperbolic representation learning.Source: B. Clinical Ontology Paths and LM… · p. 3
- A frozen biomedical language model encodes each clinical code’s textual description, using pooled LM output to obtain its embedding.These LM-initialized embeddings provide semantic initialization for HCOE.Source: B. Clinical Ontology Paths and LM… · p. 3
C. Hyperbolic Clinical Ontology Embeddings
HCOE maps biomedical language-model embeddings into a shared Poincaré ball and learns hierarchy-aware representations through ontology-guided contrastive learning and coarse-to-fine path aggregation.
C. Hyperbolic Clinical Ontology Embeddings
- Level-specific linear projections and the exponential map place biomedical language-model embeddings in a shared Poincaré ball for direct parent–child distance comparisons.All ontology nodes are mapped into the same-dimensional ball.Source: C. Hyperbolic Clinical Ontology… · p. 3
- Parent-side and child-side hyperbolic triplet losses use ontology parent–child relations to bring related concepts closer while separating negatives.Child anchors pair with parents; parent anchors pair with children to preserve local descendant structure.Source: C. Hyperbolic Clinical Ontology… · p. 3
- Sequential Möbius addition aggregates node embeddings from coarsest to finest along each ontology path, producing a representation that integrates language-model semantics with ontology structure.The aggregation keeps the representation inside the Poincaré ball.Source: C. Hyperbolic Clinical Ontology… · p. 4
A. MIMIC-IV Dataset and Preprocessing
The MIMIC-IV cohort comprises inpatient records with chronological visits and unordered clinical codes within each visit. Preprocessing yields 223,452 patients and 546,028 admissions.
A. MIMIC-IV Dataset and Preprocessing
- The processed cohort contains 223,452 patients and 546,028 hospital admissions.It includes 9,143 ICD-9 and 19,440 ICD-10 diagnosis codes, 2,557 ICD-9 and 12,354 ICD-10 procedure codes, and 1,286 ATC level-5 medication codes.Source: A. MIMIC-IV Dataset and Preprocessing · p. 4
- Each patient’s clinical codes are represented as five-level codes, treated as an unordered set within each visit, while visits remain chronologically ordered.The inpatient data include diagnoses, procedures, prescriptions, and admission and discharge timestamps.Source: A. MIMIC-IV Dataset and Preprocessing · p. 4
- The study uses MIMIC-IV inpatient hospitalizations collected between 2008 and 2022 from emergency department and inpatient admissions.MIMIC-IV is described as a publicly available, de-identified EHR database from Beth Israel Deaconess Medical Center.Source: A. MIMIC-IV Dataset and Preprocessing · p. 4
B. ICD/ATC Relation Prediction and PheCode Transfer
The evaluation covers four clinical relation settings and MIMIC-IV prediction tasks. The downstream setup uses shared patient representations and prediction heads to compare embedding methods.
B. ICD/ATC Relation Prediction and PheCode Transfer
- Relation prediction evaluates ICD→ICD, ATC→ATC, ICD→ATC, and CCS→PheCode links, including multihop hierarchy relations and transfer to an unseen PheCode hierarchy.Negative candidates include siblings or unrelated concepts, and positives rank against negatives by hyperbolic distance.Source: B. ICD/ATC Relation Prediction and… · p. 4
C. MIMIC-IV Downstream Prediction Setup
- Mortality and readmission are binary classification tasks evaluated with AUPRC and AUROC; drug recommendation is multi-label prediction, and Recall@15 is computed for 98 rare ATC codes with frequencies between 10 and 20.Drug recommendation is also evaluated with AUPRC, F1-score, and Jaccard.Source: C. MIMIC-IV Downstream Prediction Setup · p. 5
- Downstream tasks use a 70%/10%/20% patient split and the same mean-pooling strategy with a single linear prediction head for all methods.The best model is selected by validation loss on the target task.Source: C. MIMIC-IV Downstream Prediction Setup · p. 5
- Downstream baselines include structure-aware embeddings, biomedical language models, and structure-aware language models.The compared methods include GRAM, cui2vec, RotatE, BioBERT, G-BERT, ClinicalBERT, BEHRT, SapBERT, HiTs, and OnT.Source: C. MIMIC-IV Downstream Prediction Setup · p. 5
D. Implementation Details
HCOE combines frozen BioBERT embeddings with hyperbolic ontology learning and path aggregation. The reported results include cross-ontology transfer and a radial arrangement of endocrine codes by granularity.
D. Implementation Details
- BioBERT's 768-dimensional pooled outputs are projected to 256-dimensional hyperbolic embeddings; training uses a frozen backbone, curvature κ = 1/256, and margin α = 5.Training uses Adam at 1 × 10^-4 learning rate, 1 × 10^-5 weight decay, batches of 64, and up to 20 epochs with early stopping.Source: D. Implementation Details · p. 5
A. HCOE Preserves Clinical Ontology Relations
- HCOE reaches F-scores of 56.7% on ICD→ATC and 80.6% on CCS→PheCode, exceeding the strongest baselines HiTs at 53.2% and 76.5%, respectively.PheCode labels are not used during training; the transfer result concerns an unseen PheCode-based hierarchy.Source: A. HCOE Preserves Clinical Ontology… · p. 5
B. Qualitative Visualization of HCOE Representations
- In the endocrine-code visualization, coarse concepts lie near the Poincaré-ball center while fine-grained ICD codes lie nearer its boundary.The figure covers descendants of endocrine disease category 3 and presents the embeddings as a radial hierarchy.Source: B. Qualitative Visualization of HCOE… · pp. 5–6, Figure 3 · p. 5
- The supplied table captions identify MIMIC-IV prediction results and four-task relation-prediction ablations, reporting mean (standard error) and F-score, respectively.The captions specify bootstrap estimates for Table II and highlight the best results in each column in both tables.Source: Table II · p. 6, Table III · p. 6
C. HCOE Improves Clinical Prediction
On MIMIC-IV, HCOE improves the reported clinical prediction metrics. Ablations find the full model performs best across four relation-prediction tasks, with removing hyperbolic contrastive learning producing the largest drops.
C. HCOE Improves Clinical Prediction
- HCOE improves MIMIC-IV mortality and readmission metrics and achieves the best reported results for drug recommendation and rare-drug prediction.Mortality AUPRC is 44.3% versus 43.5% and AUROC 87.2% versus 86.2%; readmission AUPRC is 48.7% versus 47.3% and AUROC 72.3% versus 71.5%. Rare-drug Recall@15 is 44.2% versus 42.7%.Source: C. HCOE Improves Clinical Prediction · p. 6
D. Ablation Studies
- The ablations test path aggregation and contrastive-learning variants, reporting F-score on ICD→ICD, ATC→ATC, ICD→ATC, and CCS→PheCode.Variants include removing path aggregation, using only parent-side or child-side supervision, and removing contrastive learning.Source: D. Ablation Studies · p. 6
- Removing hyperbolic contrastive learning drops F-score from 75.6 to 47.6 on ICD→ICD and from 76.3 to 51.6 on ATC→ATC.The full model performs best overall across all four tasks; the parent-side objective consistently outperforms the child-side objective, while both together outperform either alone.Source: D. Ablation Studies · pp. 6–7
E. Sensitivity Analysis of Dimension
Among the tested hyperbolic dimensions, 256 performs best overall on the reported relation-prediction tasks. The conclusion frames HCOE's evidence around common ontologies and standard EHR prediction tasks.
E. Sensitivity Analysis of Dimension
- With hyperbolic dimension 256, HCOE achieves the best performance on ICD→ICD, ICD→ATC, and CCS→PheCode among dimensions 128, 256, and 768.The authors use dimension 256 for all experiments because it gives the best overall performance.Source: E. Sensitivity Analysis of Dimension · p. 7
- Dimension 768 adds no gains and may add unnecessary complexity, while dimension 128 appears to limit representation capacity; ICD→ATC varies only slightly across dimensions.The tested tasks are ICD→ICD, ICD→ATC, and CCS→PheCode.Source: E. Sensitivity Analysis of Dimension · p. 7
VII. CONCLUSION
- The study focuses on common clinical ontologies and standard EHR prediction tasks, leaving richer biomedical knowledge graphs and multimodal clinical inputs for future work.Examples named for future exploration include UMLS and clinical notes.Source: §VII · p. 7