Source-linked AI summary
Multi-Label Classification of Patient Notes a Case Study on ICD Code Assignment
Tal Baumel, Jumana Nassour-Kassis, Raphael Cohen, Michael Elhadad, No`emie Elhadad
TL;DR
Automated ICD coding must assign multiple labels to long clinical notes drawn from very large code sets, while making predictions transparent. The paper compares four models using MIMIC II and III and introduces HA-GRU, whose hierarchical attention achieves the strongest reported performance while highlighting relevant text. The authors identify discourse-level structure as a direction for future work, and note limitations of bag-of-words models.
Problem
Automated diagnosis coding from EHR notes involves very large label sets, multiple labels, long documents, and a need to identify text supporting predictions.
Method
The paper evaluates SVM, CBOW, CNN, and HA-GRU models on MIMIC II and III, using hierarchical token- and sentence-level attention for multi-label classification.
Results
55.86% F1 on MIMIC III is reported for HA-GRU on rolled-up codes, approximately 2.8% above the best SVM baseline.
Takeaways & Limitations
HA-GRU provides the strongest reported performance while its attention highlights decision-relevant text and preserves the full input.
Takeaways & Limitations
The approach does not yet exploit discourse-level structure in discharge summaries, and bag-of-words models ignore word order and multi-word expressions.
Abstract
from arXiv · showhide
In the context of the Electronic Health Record, automated diagnosis coding of patient notes is a useful task, but a challenging one due to the large number of codes and the length of patient notes. We investigate four models for assigning multiple ICD codes to discharge summaries taken from both MIMIC II and III. We present Hierarchical Attention-GRU (HA-GRU), a hierarchical approach to tag a document by identifying the sentences relevant for each label. HA-GRU achieves state-of-the art results. Furthermore, the learned sentence-level attention layer highlights the model decision process, allows easier error analysis, and suggests future directions for improvement.
Introduction
Automated ICD coding from long EHR notes is a large-scale, multi-label classification problem where transparency is also important. The paper compares four models and finds HA-GRU most promising while preserving the full text and highlighting evidence for predictions.
- EHR diagnosis coding selects multiple labels from very large ICD code sets, with 14,000 ICD9 and 68,000 ICD10 codes.
- The task involves 6,500 unique ICD9 codes, up to 20 labels per instance, and discharge summaries averaging 1,900 words.
- Long clinical documents make transparency critical because the model should highlight text supporting each predicted label.
- The study evaluates SVM, CBOW, CNN, and bidirectional GRU models with hierarchical attention on publicly available MIMIC datasets.
- HA-GRU yields the most promising results over SVM, CBOW, and CNN while preserving the full input text and providing effective transparency.
Previous Work
Prior work spans automated coding, patient-record labeling, predictive modeling, and extreme multi-label learning, but comparisons are hindered by differing datasets. This paper uses public MIMIC data to study large label sets and transparent classification from complete notes.
- Healthcare multi-label classification includes diagnosis code assignment, patient record labeling, and predictive modeling.
- Diagnosis Code Assignment: Diagnosis coding methods range from rules and conventional machine learning to approaches using ICD hierarchy or code co-occurrence.
- Diagnosis Code Assignment: Many prior coding systems restrict labels to rolled-up ICD codes or subsets because of the large label space.
- The paper uses public MIMIC data to evaluate both all ICD9 codes and rolled-up codes with models that highlight supporting input text.
- Prior patient-record and predictive models commonly assign phenotypes or future outcomes using limited label sets and, often, structured inputs.
- Extreme Multi-Label Learning: Extreme multi-label methods address very large label spaces through shared embeddings, ranking, local label representations, or partial-label training.
Dataset and Preprocessing
The study uses publicly available MIMIC II and III ICU records with a shared unseen-patient test set, ICD9 labels, and preprocessing designed for noisy, long discharge summaries. Sentence segmentation preserves full documents and supports hierarchical modeling, while discourse structure remains future work.
- MIMIC Datasets: MIMIC II and III provide publicly available, de-identified ICU records for evaluating the proposed methods.
- MIMIC Datasets: The common test set contains 2,282 discharge summaries from unseen patients, with 20,533 summaries used for training in the original MIMIC II split.
- MIMIC Datasets: Discharge summaries were concatenated with MIMIC III addenda because the addenda contained vital ICD-coding information absent from the main summaries.
- MIMIC Datasets: MIMIC III is larger than MIMIC II in training data, vocabulary size, and overall label count.
- ICD9 Codes: The label set contains diagnosis-only ICD9 codes, represented as five digits with three primary and two secondary digits.
- ICD9 Codes: The task qualifies as extreme multi-label classification because label density is extremely low and label counts are comparable in order to record counts.
- Input Texts: Preprocessing tokenizes text, normalizes non-alphabetical characters, builds a vocabulary from tokens appearing at least five times, and maps out-of-vocabulary words by edit distance.
- Input Texts: Sentence-level segmentation avoids truncating long documents and supplies shorter sequences for hierarchical processing, while discourse-level structure is reserved for future work.
Methods
The study evaluates four ICD-coding models with Micro-F, emphasizing preprocessing, hierarchical document encoding, and label-specific attention for long clinical notes.
- Evaluation: Performance was evaluated with Micro-F on full and rolled-up ICD code settings across MIMIC II and MIMIC III.The compared models were SVM, CBOW, CNN, and HA-GRU.
- Baseline models: The SVM uses one-vs-all binary classifiers with TF-IDF bag-of-words features for each ICD code.Chi-square feature filtering was also tested but did not improve performance.
- Baseline models: CBOW averages embedded words before a fully connected sigmoid output layer assigns labels using a fixed 0.5 threshold.Its bag-of-words representation ignores word order and cannot identify multi-word expressions, limiting negation and lexical disambiguation.
- Baseline models: CNN replaces CBOW averaging with one-dimensional convolution and max pooling over embedded words, treating filters as n-gram features.Deeper convolutional networks and an inception module did not improve results; the reported configuration used 300 channels and filter size 3.
- HA-GRU: HA-GRU uses token- and sentence-level bidirectional GRUs with attention to encode long documents hierarchically.Label-specific document attention focuses on relevant sentences for each class, while shared sentence attention supports transparency and error analysis.
Results
HA-GRU provides label-specific attention for interpreting patient-note classifications, while supporting error analysis at sentence and word levels. Results compare its performance with SVM, CBOW, and CNN models across ICD9 settings and examine label-frequency effects.
- Model Comparison: 7.4% and 3.2% improvement over CNN and SVM, respectively, was achieved by HA-GRU in rolled-up ICD9 coding on MIMIC II and III.In the full ICD9 setting on MIMIC III, CNN narrowly surpassed HA-GRU on Micro-F.
- Model Comparison: One-vs-all SVM outperformed the flat and hierarchy classifiers of Perotte et al. (2014), with improvements attributed to one-vs-all modeling and tokenization.
- Label Frequency: Label-frequency analysis evaluates HA-GRU precision and recall across bins of 50 ICD9 labels ranked by their MIMIC-III training frequency.The most frequent bin averages 12% occurrence, while the eighth averages 0.2%.
- Model Explaining Power: HA-GRU attention identifies the sentence and word contributing most to each label, such as “failure” for the Heart failure label.The model can also surface the most attended sentence for a label, while numeric tokens are replaced by pseudo-tokens during preprocessing.
- Model Explaining Power: HA-GRU supports more precise sentence-level analysis of false positives, including diagnoses mentioned in past medical history rather than as current problems.This error pattern suggests adding discourse-level structure to the coding architecture.
- Model Explaining Power: HA-GRU can analyze false negatives when attention identifies a relevant sentence even though the model assigns more frequent comorbid codes.For cellulitis, the attended sentence was relevant, but diabetes and chronic ischemic heart disease were preferred as false-positive codes.
Conclusion
The study finds HA-GRU performs best across the evaluated MIMIC coding tasks while supporting transparency and potential domain adaptation.
- 55.86% F1 on MIMIC III for rolled-up codes represents HA-GRU’s best reported performance, exceeding the best SVM baseline by approximately 2.8% absolute.
- Tokenization that maps rare variants using edit distance improves CBOW and CNN results by approximately 0.5%.
- HA-GRU achieves approximately 7% absolute F1 improvement on MIMIC II, suggesting strong performance with less training data.
- The model’s decision highlighting supports medical-expert adoption and suggests future use of discourse-level structure in medical notes.