Source-linked AI summary
Explainable Prediction of Medical Codes from Clinical Text
James Mullenbach, Sarah Wiegreffe, Jon Duke, Jimeng Sun, Jacob Eisenstein
TL;DR
Automatic medical-code assignment is laborious and error-prone, and the link between codes and supporting text is typically unannotated. The paper presents CAML, a convolutional network with per-code attention and description-guided parameters, achieving stronger code-prediction performance and physician-rated meaningful explanations. Its evaluations focus on ICD-9 coding in MIMIC ICU discharge summaries.
Problem
Manual medical-code annotation is laborious and error-prone, while the connection between codes and clinical text is not annotated.
Method
CAML uses convolutional representations, per-label attention over document snippets, and code-description-guided parameter regularization for ICD-code prediction.
Results
CAML substantially outperforms previous medical code prediction results on MIMIC-II and MIMIC-III while producing physician-evaluated informative explanations.
Takeaways & Limitations
Per-label attention provides text snippets that help explain individual code assignments in this clinical coding setting.
Takeaways & Limitations
The study focuses on ICD-9 because that is the coding version used in the MIMIC datasets.
Abstract
from arXiv · showhide
Clinical notes are text documents that are created by clinicians for each patient encounter. They are typically accompanied by medical codes, which describe the diagnosis and treatment. Annotating these codes is labor intensive and error prone; furthermore, the connection between the codes and the text is not annotated, obscuring the reasons and details behind specific diagnoses and treatments. We present an attentional convolutional network that predicts medical codes from clinical text. Our method aggregates information across the document using a convolutional neural network, and uses an attention mechanism to select the most relevant segments for each of the thousands of possible codes. The method is accurate, achieving precision@8 of 0.71 and a Micro-F1 of 0.54, which are both better than the prior state of the art. Furthermore, through an interpretability evaluation by a physician, we show that the attention mechanism identifies meaningful explanations for each code assignment
1 Introduction
Automatic ICD coding from clinical notes is difficult because the label space is enormous and the text is noisy. CAML combines convolutional representations with per-label attention to predict codes and provide text-based explanations.
- Motivation: ICD coding is laborious and error-prone, while clinical notes must be mapped to taxonomies containing over 15,000 ICD-9 codes and over 140,000 ICD-10-CM/PCS codes.Clinical text also contains irrelevant information, misspellings, non-standard abbreviations, and a large medical vocabulary.
- Approach: CAML uses convolutional neural networks and per-label attention to learn distinct document representations for each ICD code.The approach assumes code-relevant snippets may occur anywhere in a document and differ across labels.
- Approach: CAML uses textual descriptions of codes to guide parameters for rarely observed labels toward those of codes with similar descriptions.This design addresses the large label space by sharing information through code descriptions.
- Results: The approach substantially outperforms previous medical code prediction results on both MIMIC-II and MIMIC-III.The evaluations use open ICU medical-record datasets containing narrative notes about diagnoses and procedures.
- Interpretability: CAML attention assigns importance values to input n-grams, enabling extracted snippets to explain why each code was predicted.A physician evaluated the informativeness of automatically generated explanations.
2 Method
CAML treats ICD-9 prediction from discharge summaries as multilabel classification, using convolutional representations and label-specific attention to identify relevant text. DR-CAML additionally regularizes label-prediction parameters toward embeddings derived from code descriptions.
- 2 Method: CAML models ICD-9 code assignment as multilabel text classification over the code set.The model produces binary decisions for every possible code.
- 2.1 Convolutional architecture: A convolutional layer transforms adjacent word embeddings into a document representation matrix, with padding preserving one representation per document position.Filter width, embedding size, and output dimension determine the convolutional features.
- 2.2 Attention: Per-label attention selects the document locations most relevant to each code instead of pooling one shared vector across all labels.Attention weights form label-specific document vectors from the convolutional representation.
- 2.3 Classification: Each label-specific representation is passed through a linear layer and sigmoid transformation to estimate that code’s probability.The output layer uses label-specific prediction weights and an offset.
- 2.5 Embedding label descriptions: DR-CAML uses code descriptions to build label embeddings and regularizes prediction parameters toward those embeddings, with λ controlling the tradeoff between objectives.The description module is a max-pooling CNN, and the regularization targets rarely observed codes.
3 Evaluation of code prediction
The evaluation tests CAML and neural and logistic-regression baselines across MIMIC-III and MIMIC-II code-prediction settings, using multiple multilabel metrics. CAML achieves the strongest results in the main full-label MIMIC-III evaluation and improves over prior work across several settings.
- Datasets: MIMIC-III contains ICU discharge summaries tagged with 8,921 ICD-9 diagnosis and procedure codes, with patient-level splitting to prevent overlap across partitions.The full-label setting uses 47,724 training summaries, 1,632 validation summaries, and 3,372 test summaries.
- Datasets: The evaluation covers full-label MIMIC-III, the 50 most frequent MIMIC-III codes, and MIMIC-II using established train/test settings.The 50-label setting uses 8,067 training, 1,574 validation, and 1,730 testing summaries; the MIMIC-II split uses 20,533 training and 2,282 testing examples.
- Systems: CAML is compared with a one-dimensional CNN, bag-of-words logistic regression, and Bi-GRU, with neural models using the same pretrained word2vec initialization.Hyperparameters are selected with Bayesian optimization using precision@8 on the MIMIC-III full-label validation set.
- Evaluation Metrics: The evaluation reports micro- and macro-averaged F1 and AUC, plus precision at 5, 8, and, for full-label MIMIC-III, 15.Micro metrics treat each text-code pair separately, while macro metrics average per-label metrics and emphasize rare labels more strongly.
- Results: CAML gives the strongest results on all metrics in full-label MIMIC-III, while attention improves substantially over the vanilla CNN and logistic regression performs substantially worse than the neural models.DR-CAML performs worse than CAML on most metrics with tuned λ=0.01.
- Results: 5.5 of the eight most confident predictions are correct on average, and CAML improves over prior work across the 50-label MIMIC-III and MIMIC-II evaluations.The authors identify precision@8 as especially informative for returning a small high-confidence subset of codes; an alternative MIMIC-II training protocol is left for future work.
4 Evaluation of Interpretability
The evaluation compares CAML’s attention-based explanations with three heuristics using physician judgments of extracted text snippets. CAML produces the strongest interpretability results, although cosine similarity lacks explanations for some note-label pairs.
- Evaluation setup: A physician evaluated explanations for 100 randomly sampled predicted codes from the MIMIC-III full-label test set.Each method supplied its most important 4-gram with five words of context on either side.
- Explanation methods: CAML extracts label-specific snippets by selecting the 4-gram with the highest attention weight.Its attention mechanism identifies the most influential text span for each predicted label.
- Explanation methods: The max-pooling CNN, logistic regression, and code-description baselines use activation-based, coefficient-based, and similarity-based snippet selection, respectively.The code-description baseline removes note-label pairs with no positive similarity score.
- Results: CAML selects the greatest number of highly informative explanations and more informative explanations than the CNN and logistic-regression baselines.These judgments are reported in the qualitative evaluation table.
- Results: Cosine similarity performs well, but it provides no explanation for about 12% of test note-label pairs.Those cases occur when no document 4-gram has nonzero similarity with the code description.
5 Related Work
Prior ICD-coding work spans classical, structured, and neural approaches, often evaluating restricted label spaces or limited datasets. CAML is compared with published neural and feature-based systems and reports stronger results across the evaluated conditions.
- Attentional convolution: CNNs and attention mechanisms have been used separately and together for text classification, while CAML combines convolution with label-specific attention.The attention selects relevant parts of ICU discharge summaries.
- Automatic ICD coding: Automatic ICD coding has used machine-learning and handcrafted methods, with recent systems drawing primarily on unstructured text or combining it with structured data.The task is longstanding in medical informatics.
- Automatic ICD coding: Earlier studies often evaluated restricted ICD label subsets, narrower medical scenarios, or datasets that were not publicly available.These choices make direct comparison difficult.
- Neural approaches: Neural baselines include hierarchical attention GRUs, character-aware LSTMs, memory networks, and grounded recurrent networks.These systems vary in their use of subsections, external knowledge, and label-specific representations.
- Neural approaches: CAML yields stronger results across the reported experimental conditions than the compared prior approaches.The paper attributes these improvements to attention focusing on critical text.
- Explainable text classification: Explainability research has used latent rationales, gradient-based word salience, and feature-selection methods to approximate classification decisions.These approaches provide alternative ways to identify text relevant to predictions.
6 Conclusions and Future Work
The paper concludes that CAML combines strong ICD-9 prediction with satisfactory explanations and can extend to other multilabel tagging tasks. Future work targets document structure, non-standard clinical writing, code hierarchies, and prediction for future visits.
- Conclusions: CAML adaptively pools convolutional representations for each label to identify highly predictive document locations.The model is presented as a convolutional neural network for multilabel document classification.
- Conclusions: CAML improves previous metrics on several ICD-9 prediction formulations while providing satisfactory explanations for its predictions.The conclusion covers both predictive performance and interpretability.
- Conclusions: The architecture is extensible without modification to other multilabel document-tagging tasks, including ICD-10 coding.The paper’s experiments focus on a clinical setting and ICD-9 data.
- Future work: Future work will incorporate discharge-summary structure and improve handling of non-standard writing and out-of-vocabulary tokens.These are identified as linguistic directions.
- Future work: Application-focused future work includes exploiting ICD-code hierarchies and predicting diagnosis and treatment codes for future visits.The latter task uses discharge summaries as input.