PaperlayerView exact revision on arXiv

Source-linked AI summary

An Empirical Evaluation of Deep Learning for ICD-9 Code Assignment using MIMIC-III Clinical Notes

Jinmiao Huang, Cesar Osorio, Luke Wicent Sy

arXiv:1802.02311v2cs.CL

TL;DR

The paper evaluates automatic ICD-9 assignment from clinical notes, a labor-intensive task for which reliable benchmark comparisons are limited. It compares end-to-end deep-learning systems with traditional classifiers on MIMIC-III; deep models outperform baselines for top-10 codes and categories, while top-50 code results do not outperform the baseline.

  • Problem

    Manual ICD-9 assignment from clinical notes is labor-intensive, and existing studies are difficult to compare because standard comparison metrics are lacking.

    Source: §1 · p. 2, §5 · p. 14

  • Method

    The study evaluates deep neural networks and traditional classifiers for multilabel ICD-9 assignment, representing discharge summaries as sequential word2vec features.

    Source: §1 · p. 2, §3.3.2 · p. 7

  • Results

    Deep models outperform traditional baselines for top-10 codes and categories, but top-50 code F1 is 0.3263 for GRUs versus 0.3662 for Logistic Regression.

    Source: §6 · p. 16

  • Takeaways & Limitations

    The study uses standard metrics to compare ICD-9 assignment systems on MIMIC-III and presents the evaluation as a baseline for further research.

    Source: §1 · p. 2

    Input sequences may still be too long for LSTMs and GRUs to retain useful information.

    Source: §5 · p. 15

Abstract

from arXiv · show

Background and Objective: Code assignment is of paramount importance in many levels in modern hospitals, from ensuring accurate billing process to creating a valid record of patient care history. However, the coding process is tedious and subjective, and it requires medical coders with extensive training. This study aims to evaluate the performance of deep-learning-based systems to automatically map clinical notes to ICD-9 medical codes. Methods: The evaluations of this research are focused on end-to-end learning methods without manually defined rules. Traditional machine learning algorithms, as well as state-of-the-art deep learning methods such as Recurrent Neural Networks and Convolution Neural Networks, were applied to the Medical Information Mart for Intensive Care (MIMIC-III) dataset. An extensive number of experiments was applied to different settings of the tested algorithm. Results: Findings showed that the deep learning-based methods outperformed other conventional machine learning methods. From our assessment, the best models could predict the top 10 ICD-9 codes with 0.6957 F1 and 0.8967 accuracy and could estimate the top 10 ICD-9 categories with 0.7233 F1 and 0.8588 accuracy. Our implementation also outperformed existing work under certain evaluation metrics. Conclusion: A set of standard metrics was utilized in assessing the performance of ICD-9 code assignment on MIMIC-III dataset. All the developed evaluation tools and resources are available online, which can be used as a baseline for further research.

1. Introduction

The paper addresses labor-intensive manual ICD-9 coding from clinical notes and the need for reliable benchmarks. It evaluates learning-based systems on MIMIC-III, comparing deep neural networks with conventional classifiers and contributing standard-metric results as a baseline for further research.

1. Introduction

  • More than 80% of health-record data are unstructured text, including clinical notes with patient history and clinicians’ observations.EHRs also contain structured information such as blood pressure.Source: §1 · p. 2
  • Manual ICD-9 assignment from free-text EHRs is labor-intensive, and the codes support billing and patient-care history records.The paper frames automation as a way to improve efficiency and support clinical semantic analysis.Source: §1 · p. 2
  • Prior machine-learning approaches include SVM, Naive Bayes, and Boosting, while the authors argue accuracy could improve with deep learning trained end-to-end without hand-designed features.The paper situates the task among NLP applications where deep learning had shown improvements.Source: §1 · p. 2
  • The paper evaluates deep neural networks against Logistic Regression, Random Forests, and FNNs, including experiments with MIMIC-III-trained and medical-domain word embeddings.The stated aim is a baseline for learning-based ICD-9 assignment on MIMIC-III.Source: §1 · p. 2
  • The study develops deep-learning algorithms for ICD-9 coding and compares machine-learning and deep-learning methods on MIMIC-III using standard metrics intended as a baseline for further research.The implementation outperformed existing work under certain evaluation metrics.Source: §1 · p. 2

2. Related Work

Prior ICD-9 coding work includes both expert-designed rule systems and learning-based methods, while clinical deep learning has been applied to structured records and free-text notes. The paper positions its comparison with existing free-text coding work, especially Prakash et al..

2. Related Work

  • Rule-based ICD-9 systems have outperformed other methods in many cases, but depend heavily on medical-professional intervention and are difficult to maintain and scale.Learning-based systems instead rely on algorithms to learn from the supplied datasets without requiring medical experts’ domain knowledge.Source: §2 · p. 3
  • Clinical RNN research includes heart-failure prediction, diagnosis classification from structured measurements [20], and structured-EHR systems DoctorAI and RETAIN.Source: §2 · p. 3
  • For unstructured text, Bai used MeSH knowledge for ICD-9 coding [22], while Prakash et al. used Wikipedia text and condensed memory networks on MIMIC-III.The paper compares its findings with Prakash et al. because their task is similar.Source: §2 · p. 3

3. Methodology

The methodology preprocesses MIMIC-III discharge summaries, extracts text features, and trains and tests baseline and deep-learning models for multi-label ICD-9 assignment. It evaluates the models using precision, recall, F-score, and accuracy.

3. Methodology

  • The pipeline combines Spark preprocessing, Spark, Sklearn, and Gensim feature extraction, and Spark ML and Keras model training and testing.Experiments ran on Azure NC24 virtual machines with K80 GPUs.Source: §3 · p. 3

3.1. Data Preprocessing

  • The study uses MIMIC-III discharge summaries from the noteevents table, then evaluates separate top-10 and top-50 code and category datasets.The top 10 and top 50 codes cover 76.9% and 93.6% of the dataset, respectively; filtered data are split 50-25-25 for training, validation, and testing.Source: §3.1 · pp. 3–4

3.2. Feature Extraction

  • Feature extraction compares TF-IDF with word2vec, using both MIMIC-III-trained and PubMed-pretrained word vectors.The word2vec model uses CBOW to predict a target word from its surrounding context.Source: §3.2 · pp. 4–5

3.3. Model Training and Testing

  • Because each note can have multiple ICD-9 labels, the study uses problem transformation for linear and random-forest baselines and algorithm adaptation for neural networks.Problem transformation trains a separate binary model for each label, while algorithm adaptation trains one classifier for all labels.Source: §3.3 · p. 6

3.3.1. Baseline Models

  • Logistic regression and random forests use one classifier per label, while feed-forward neural networks handle multiple labels through their output layer.The baseline experiments tune model configurations, including iteration counts, tree depth, and neural-network layers and neuron sizes.Source: §3.3.1 · p. 6

3.3.2. Deep Neural Network Models

  • Deep models process sequential word2vec features with CNNs or recurrent networks, including LSTMs and GRUs.CNNs apply convolution filters over word windows and max-over-time pooling; recurrent models use the final time-step output, and LSTMs and GRUs use memory mechanisms for long-term dependencies.Source: §3.3.2 · pp. 7–9

3.4. Metrics

  • Models are evaluated for multi-label classification using precision, recall, F-score, and accuracy.Precision measures the proportion of predicted labels that are correct, recall the proportion of actual labels correctly predicted, and F1 their harmonic mean.Source: §3.4 · p. 9

4. Results

The results compare model configurations and benchmark performance across code and category tasks, then assess precision-recall behavior and compare selected models with prior work.

4.1. Model Performance under Different Configurations

  • 0.532 F1 (Logistic Regression) and 0.322 F1 (Random Forests) were best with 20k-feature tfidf; FNNs peaked at 0.528 F1 with 600-feature word2vec m3.For non-sequential models, tfidf generally performed better, while word2vec retained a satisfactory representation with far fewer features; averaging word vectors may lose information.Source: §4.1 · pp. 10–11
  • 0.637 F1 (CNNs), 0.696 F1 (GRUs), and 0.683 F1 (LSTMs) were the best with self-trained word2vec; simple RNNs reached only 0.0–0.08 F1.Self-trained word2vec outperformed PubMed word2vec in most configurations.Source: §4.1 · p. 11

4.2.1. Overview

  • 0.6957 F1 (GRUs) for top-10-code and 0.7233 F1 (GRUs) for top-10-cat; categories performed slightly better, with more samples per label and less-specific labels.Logistic Regression and Random Forests overfit in these tasks.Source: §4.2.1 · p. 11
  • 0.6328 F1 (GRUs) for the first 10 top-50-code labels and 0.6738 F1 (LSTMs) for the first 10 top-50-cat labels; GRUs led precision and accuracy.Figure 7 displays the top-50 results for the first 10 labels.Source: Figure 7 · p. 12, §4.2.1 · p. 12
  • 0.3662 F1 (Logistic Regression) for top-50-code and 0.4301 F1 (Logistic Regression) for top-50-cat; category results were slightly better.The baseline models also overfit.Source: §4.2.1 · p. 12

4.2.2. Precision-Recall Curve

  • 0.90 AP for atrial fibrillation was highest among top-10 codes; top-50-code results also included a 0.92 AP for neonatal jaundice label 44.The top-10 results also included coronary atherosclerosis at 0.89 AP; label 44 had 2,183 training samples.Source: Figure 8 · p. 13, §4.2.2 · p. 13

4.2.3. Results Comparison

  • Our models outperformed Prakash and Zhao [23] on macro AUC for GRUs and on top-five precision for all models, while their hamming loss was better.The comparison uses top-50 codes; top-100 labels were not compared.Source: §4.2.3 · p. 14, Table 9 · p. 14

5. Discussion

The discussion favors sequential RNNs for code assignment while identifying architecture, representation, and label-count limitations. It also points to future work on interpretation and model design.

5. Discussion

  • RNNs performed better than other machine-learning models and previously published systems under certain metrics, with sequential processing incorporating information from earlier words.Each RNN node receives input from the previous node through a feedback loop.Source: §5 · p. 14
  • Simple RNNs performed worst, with F1 no higher than 0.08; bidirectional RNNs were comparable to or slightly worse than RNNs on top-10 coding.The authors cite extra computation cost and suggest future context may not benefit this task.Source: §5 · p. 15
  • CNNs trained faster than recurrent models, but the authors identify Temporal Convolutional Networks as a future architecture to explore.CNN training took 30 minutes, versus 6 hours for GRUs and 18 hours for LSTMs.Source: §5 · p. 15
  • Averaged word2vec features underperformed tfidf in baseline models, while MIMIC-III-trained word2vec generally performed slightly better than PubMed vectors.The two word2vec sources showed comparable performance overall.Source: §5 · p. 15
  • Long input sequences may prevent LSTMs and GRUs from retaining useful information; the authors suggest shorter text representations and note memory networks perform better on some metrics.Sentence2vec or paragraph2vec are proposed as ways to shorten sequences.Source: §5 · p. 15
  • Top-50 models did not succeed, and labels 11–50 had few positive samples relative to negative ones, which may leave insufficient data for learning useful representations.The authors propose balancing or increasing those samples and suggest five parallel top-10 models as a possible design.Source: §5 · pp. 15–16
  • The custom word2vec model used CBOW rather than skip-gram, whose advantages in biomedical tasks motivate testing embedding parameters in future work.The paper reports that skip-gram was not used in the custom model.Source: §5 · p. 16
  • The authors propose studying which words influence prediction probabilities as a route to understanding symptom–diagnosis relationships and improving preprocessing or feature extraction.Source: §5 · p. 16

6. Conclusion

The study evaluates learning-based ICD-9 assignment on MIMIC-III and reports stronger top-10 results for deep learning than baselines, but not for top-50 labels.

6. Conclusion

  • GRUs achieved the best top-10 F1 scores over Logistic Regression for codes (0.6957 vs 0.5320) and FNNs for categories (0.7233 vs 0.6313), while top-50 GRU scores trailed their baselines.The study presents its standard-metric evaluation as a baseline for further research.Source: §6 · p. 16

Conflict of Interest Statement

The authors declare no conflict of interest.

Conflict of Interest Statement

A. Model Performance for Top 10 Label Codes

The supplied caption identifies a table on top-10-code model performance.

A. Model Performance for Top 10 Label Codes

  • Table 10 is identified as reporting model performance for top-10-code assignment.Source: Table 10 · p. 19

B. Model Performance for Top 50 Codes

This section identifies performance tables for the top 50 codes and a first-10 subset, and notes that aresult nan values were replaced with zero.

B. Model Performance for Top 50 Codes

C. Model Performance for Top 10 Label Categories

The section presents performance tables for top-10 and top-50 label categories, alongside precision-recall-curve panels for LSTMs, GRUs, and CNNs across code and category settings.

C. Model Performance for Top 10 Label Categories

  • Table 13 reports model performance for the top 10 label categories.Source: Table 13 · p. 20

D. Model Performance for Top 50 Label Categories

E. Best Performance Models (LSTMs, GRUs, CNNs) Precision-Recall Curve

Related papers

Loading 1802.02311v2…