PaperlayerView exact revision on arXiv

Source-linked AI summary

Automated Medical Coding on MIMIC-III and MIMIC-IV: A Critical Review and Replicability Study

Joakim Edin, Alexander Junge, Jakob D. Havtorn, Lasse Borgholt, Maria Maistro, Tuukka Ruotsalo, Lars Maaløe

arXiv:2304.10909v1cs.LGcs.IR

TL;DR

Manual medical coding is costly and prone to error, while differences in prior model setups make comparisons difficult. This study reproduces and compares automated coding models using revised evaluation and stratified splits, finding that rare codes remain challenging and long documents have negligible impact.

  • Problem

    Comparing automated coding models is difficult because prior studies use different experimental setups, and in-depth analyses of cited challenges are limited.

    Source: §1 · p. 1

  • Method

    The study reproduces selected models and compares them using corrected evaluation, stratified splits, and shared experimental setups with tuning.

    Source: §3.3 · p. 4, §6 · p. 9

  • Results

    Correcting macro F1 approximately doubles reproduced-model scores on MIMIC-III full; analyses find rare codes harm performance while long documents have negligible impact.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    The authors release code, model parameters, and new MIMIC-III and MIMIC-IV pipelines to support fair future comparisons.

    Source: §1 · p. 2

    The findings may not generalize beyond MIMIC-III and MIMIC-IV data from one hospital’s emergency department and ICU.

    Source: §5.3 · p. 9

Abstract

from arXiv · show

Medical coding is the task of assigning medical codes to clinical free-text documentation. Healthcare professionals manually assign such codes to track patient diagnoses and treatments. Automated medical coding can considerably alleviate this administrative burden. In this paper, we reproduce, compare, and analyze state-of-the-art automated medical coding machine learning models. We show that several models underperform due to weak configurations, poorly sampled train-test splits, and insufficient evaluation. In previous work, the macro F1 score has been calculated sub-optimally, and our correction doubles it. We contribute a revised model comparison using stratified sampling and identical experimental setups, including hyperparameters and decision boundary tuning. We analyze prediction errors to validate and falsify assumptions of previous works. The analysis confirms that all models struggle with rare codes, while long documents only have a negligible impact. Finally, we present the first comprehensive results on the newly released MIMIC-IV dataset using the reproduced models. We release our code, model parameters, and new MIMIC-III and MIMIC-IV training and evaluation pipelines to accommodate fair future comparisons.

1 INTRODUCTION

The paper addresses difficult model comparisons in automated medical coding by reproducing prior systems and revising evaluation and dataset splits. Its analyses identify rare codes as a performance challenge, while long documents have negligible impact.

1 INTRODUCTION

  • Automated coding aims to predict or rank diagnosis and procedure codes from clinical text, while manual coding is costly and prone to error.Medical codes also support appropriate care and provider compensation.Source: §1 · p. 1
  • Model comparisons are difficult because experimental setups differ, and in-depth analyses of commonly cited challenges such as rare codes and long documents remain limited.Table 1 compares MIMIC-III full and 50, the proposed clean split, and MIMIC-IV ICD-9 and ICD-10 splits after preprocessing.Source: §1 · p. 1, Table 1 · p. 2
  • The authors report that correcting macro F1 doubles its scores, rare codes harm performance, and long documents have only a negligible impact.They also report that previously low-performing models improve considerably under shared experimental setups and that the findings generalize to MIMIC-IV.Source: §1 · p. 2
  • The study creates a stratified MIMIC-III split and compares models under identical training, evaluation, and experimental setups, including hyperparameter and decision-boundary tuning.It also reports first results for reproduced models on MIMIC-IV.Source: §1 · p. 2

2 PREVIOUS WORK

The review describes the MIMIC datasets and common automated-coding architectures, then surveys differences in prior training and evaluation practices. These differences complicate comparisons between models.

2 PREVIOUS WORK

  • The review covers datasets, architectures, training, and evaluation for the models selected for comparison.Source: §2 · p. 2

2.1 Datasets

  • MIMIC-III contains ICD-9 documents and common full and top-50 splits, while MIMIC-IV includes emergency-department and ICU data labeled with ICD-9 or ICD-10.ICD-9 and ICD-10 differ substantially in code-set size.Source: §2.1 · p. 2

2.2 Model architectures

  • Most recent models use an encoder-decoder architecture: token sequences become hidden representations, which the decoder converts to code probabilities for ranking or threshold-based classification.Source: §2.2 · p. 2

2.2.1 Encoders:

  • Encoders commonly combine pretrained word embeddings with a context-encoding network, while masked language models such as BERT have gained popularity.The MIMIC-III training set or PubMed articles are common pretraining sources.Source: §2.2.1 · pp. 2–3

2.2.2 Decoders:

  • Label-wise attention replaces pooling in some decoders by forming label-specific representations; CAML and LAAT use distinct versions of this attention module.Source: §2.2.2 · p. 3

2.3 Training and evaluation methods

  • Prior MIMIC-III work commonly used a random split that left 54% of ICD codes absent from the test set, complicating evaluation.Training practices also varied in truncation length, decision-boundary tuning, hyperparameter search, and learning-rate scheduling.Source: §2.3 · p. 3

3 METHODS

The study selects models with publicly available source code and excludes approaches whose additional inputs or missing implementation details complicate comparison.

3 METHODS

  • The comparison includes models trained from scratch and models with externally pretrained components, while excluding multimodal inputs such as code descriptions, synonyms, hierarchies, or Wikipedia articles.The authors also excluded works without public source code because their experiment descriptions often lacked implementation details.Source: §3.1 · p. 3
  • Models using additional modalities were excluded because they added complexity without evidence of significant performance improvements.Source: §3.1 · p. 3
  • The selected approaches include both models trained from scratch and models with components pretrained on external corpora.Source: §3.1 · p. 3

3.2 Evaluation metrics

The evaluation uses established ranking and classification metrics alongside measures for exact document-level predictions and ranked relevance. The authors revise macro F1 computation and address missing test classes in their comparisons.

3.2 Evaluation metrics

  • The evaluation adds exact match ratio, R-precision, and mean average precision to AUC-ROC, F1, and precision@k.Exact match ratio measures documents with every code predicted correctly; R-precision uses the true number of relevant codes, and MAP accounts for their ranks.Source: §3.2 · p. 3
  • 54% of MIMIC-III full codes were absent from the test set, so the revised split reduces missing classes and ignores the few that remain.The authors compute macro F1 as the arithmetic mean of class-level F1 scores rather than the harmonic mean of macro precision and recall.Source: §3.2 · pp. 3–4
  • Table 3 compares original and optimized hyperparameters, document-length limits, learning-rate schedules, and decision-boundary tuning strategies.The caption defines starred settings as optimal settings found in this paper and says untuned decision boundaries were set to 0.5.Source: Table 3 · p. 4

3.3 Definition of splits

The authors create new stratified splits for MIMIC-III and MIMIC-IV, addressing missing-code coverage in the original MIMIC-III evaluation.

3.3 Definition of splits

  • MIMIC-III clean removes codes occurring fewer than ten times and uses multi-label stratified sampling with a doubled test set to represent most codes across training and test.The split is based on MIMIC-III v1.4 and keeps patients separated between training and test.Source: §3.3 · p. 4
  • The authors create MIMIC-III clean, MIMIC-IV ICD-9, and MIMIC-IV ICD-10 splits.The MIMIC-IV splits use version 2.2 and include documents labeled with the respective code system.Source: §3.3 · p. 4
  • The split design responds to the original MIMIC-III full set having 54% of codes absent from its test set, biasing evaluation metrics.Source: §3.3 · p. 4

3.4 Reproducibility experiments

The reproducibility experiments test whether reported model results can be reproduced and validate the authors’ reimplementations on MIMIC-III full.

3.4 Reproducibility experiments

  • The authors run reproducibility experiments on MIMIC-III full to assess whether prior results can be reproduced and validate their reimplementations.Source: §3.4 · p. 4
  • The experiments use the hyperparameters reported in each original paper.The hyperparameter settings are referenced in Table 3.Source: §3.4 · p. 4
  • The authors report both original and revised macro F1 scores for the reproducibility experiments.Source: §3.4 · p. 4

3.5 Revised comparison

The revised comparison uses shared experimental settings across new MIMIC-III and MIMIC-IV splits, with ablations examining truncation, training configuration, and decision-boundary tuning.

3.5 Revised comparison

  • All models are trained for 20 epochs with AdamW and a shared learning-rate schedule, and decision boundaries are tuned to maximize validation micro F1.The schedule uses linear warmup for the first 2K updates followed by linear decay; documents are truncated at 4,000 words.Source: §3.5 · p. 4
  • The comparison evaluates models on MIMIC-III clean and MIMIC-IV ICD-9 and ICD-10 using randomized hyperparameter search and ten seeds per dataset.The best settings found for each model generalized across datasets.Source: §3.5 · p. 4
  • Ablations vary one factor at a time: document truncation, original training settings, or a fixed 0.5 decision boundary instead of a tuned threshold.The truncation ablation uses a 2,500-word maximum; the original-settings ablation restores the original hyperparameters, epoch count, and learning-rate schedule.Source: §3.5 · p. 4

3.6 Error analysis

The error analysis examines training-data quantity, document length, and code-level difficulty to assess commonly cited challenges in automated medical coding.

3.6 Error analysis

  • The error analysis is designed to validate or falsify claims that limited training data, long documents, and rare codes challenge automated medical coding.It also examines code attributes associated with prediction difficulty.Source: §3.6 · pp. 4–5

3.6.1 Amount of training data:

  • On MIMIC-IV ICD-9, models are evaluated with training subsets of 25k, 50k, 75k, 100k, and 125k examples, using micro and macro F1 on a fixed test set.Multi-label stratified sampling preserves code distributions across training subsets.Source: §3.6.1 · p. 5

3.6.2 Document length:

  • Document-length analysis on MIMIC-IV ICD-9 correlates word counts with each model’s micro F1 using Pearson and Spearman measures.The analysis uses each model’s best seed from the revised comparison.Source: §3.6.2 · p. 5

3.6.3 Code analysis:

  • Code analysis correlates per-code performance with training frequency and examines ICD-10 diagnosis performance by classification chapter.Chapter-level grouping is used to investigate attributes of challenging codes; ICD-10 is selected as the system currently used at most hospitals.Source: §3.6.3 · p. 5

4 RESULTS

The revised comparison places PLM-ICD ahead across datasets and metrics, while the code analysis finds that models struggle with rare codes and document length has negligible average impact.

4.1 Reproduced results

  • Correcting macro F1 nearly doubled scores on MIMIC-III full; differences were smaller on MIMIC-III 50, where all included codes appear in the test set.The reproduced results use the original papers’ hyperparameters.Source: §4.1 · p. 5, Table 4 · p. 6

4.2 Revised comparison

  • PLM-ICD outperformed the other models on every reported metric and dataset, while Bi-GRU outperformed CNN across all metrics.The models previously reported as least performant improved the most.Source: §4.2 · p. 5, Table 5 · p. 6
  • Not tuning the decision boundary had the largest negative ablation effect on all models except MultiResCNN; truncating notes from 4,000 to 2,500 words had little impact.LAAT and MultiResCNN perform similarly at a threshold of 0.5, but LAAT performs considerably better when the threshold is tuned.Source: §4.2 · p. 5, Table 6 · p. 7

4.3.1 Amount of training data:

  • Increasing training data usually improved macro F1 more than micro F1, indicating larger gains for rare than common codes.The macro-F1 curve was less smooth because the decision boundary was tuned on micro F1.Source: §4.3.1 · p. 5, Figure 3 · p. 7

4.3.2 Document length:

  • Document length had a very small negative correlation with micro F1, and its average effect was negligible because few documents were long.Documents shorter than 1,000 words often lacked information needed to predict their labeled codes; the study could not validate prior claims that long documents impair performance.Source: §4.3.2 · p. 5, Figure 4 · p. 8

4.3.3 Code analysis:

  • Although overall performance was worse on ICD-10 than ICD-9, PLM-ICD performed comparably on codes with similar training frequencies, suggesting ICD-10’s larger rare-code fraction explains the gap.The frequency-binned comparison reports macro F1 for PLM-ICD on both datasets.Source: §4.3.3 · p. 5, Table 5 · p. 6, Figure 5 · p. 8
  • No model correctly predicted more than 50% of ICD-10 codes; PLM-ICD missed 2,928 of 5,794 diagnosis codes, of which only 110 had over 100 training examples.Of the missed codes with over 100 examples, 58 belonged to two ICD-10 chapters.Source: §4.3.3 · pp. 5–6
  • PLM-ICD missed similar nicotine codes under class imbalance, including Z72.0 at 0% F1, and the authors hypothesize that removing digits during preprocessing makes some codes difficult to predict.The model scored 53% F1 on Z87.891 and 51% on F17.210; examples of digit-bearing codes it missed include BMI codes and Z3A.34.Source: §4.3.3 · p. 7

5 DISCUSSION

The discussion finds that evaluation and setup choices materially alter model comparisons, while rare codes remain difficult and document length has little effect. It recommends more consistent comparisons and better handling of coding cases that remain challenging, while noting that MIMIC-based findings have limits.

5.1 Lessons learned

  • Under a shared, tuned setup, previously low-performing models improved, the micro F1 gap narrowed by 5.8 percentage points, and Bi-GRU outperformed CNN across three datasets; reproduced results still differed from earlier reports.The authors hypothesize that prior works may have used incorrect hyperparameters or an earlier MIMIC-III version.Source: §5.1 · p. 8
  • With a learning-rate schedule and appropriate hyperparameters, all models trained in 20 epochs without sacrificing performance, making CAML faster to train than MultiResCNN in this setup.Source: §5.1 · p. 8
  • 22.8% macro F1 for PLM-ICD on MIMIC-III full with the corrected calculation, versus the previously highest reported 12.7%.The paper attributes prior underestimation to the poorly sampled split and assigning zero F1 to codes absent from the test set.Source: §5.1 · p. 8
  • More than half of ICD-10 codes were never predicted correctly, and adding training data improved rare-code performance more than common-code performance; prior rare-code conclusions may be affected by macro-F1 miscalculation.The authors interpret the findings as evidence that limited training data gives rise to many rare codes.Source: §5.1 · p. 9
  • Document length and model performance had only a weak correlation, and the small number of very long documents was insufficient to affect average dataset performance.Source: §5.1 · p. 9

5.2 Future work

  • Future work should use stratified splits, corrected metrics, hyperparameter search, and decision-boundary tuning, while improving rare-code performance and identifying cases for manual coding.The authors also plan to investigate why pre-trained transformers underperform in medical coding.Source: §5.2 · p. 9

5.3 Limitations

  • Findings may not generalize beyond MIMIC's single-hospital emergency department and ICU data; the analysis considered 20 chapters with over 100 training occurrences per code, where Z00-Z99 and V00-Y99 were hardest.The discussion also notes that MIMIC labels can be error-prone and subjective, making training and evaluation challenging; human-in-the-loop evaluation could increase reliability.Source: §5.3 · p. 9, Figure 6 · p. 9

6 CONCLUSION

The paper reproduces selected state-of-the-art models, corrects evaluation and comparison procedures, and reports results on MIMIC-IV. Its analyses find persistent difficulty with rare codes but negligible negative impact from long documents, and it releases code, parameters, and dataset splits.

6 CONCLUSION

  • The study reproduces selected unimodal models, corrects macro-F1 computation and missing-class treatment, and compares models with a clean MIMIC-III split and shared tuned setups.It also provides the first results for these models on MIMIC-IV ICD-9 and ICD-10 subsets.Source: §6 · p. 9
  • Models underperform severely on rare codes, whereas long documents have only a negligible negative performance impact.Source: §6 · p. 9
  • The paper releases source code, model parameters, and new MIMIC-III clean and MIMIC-IV ICD-9 and ICD-10 splits.Source: §6 · p. 9

Related papers

Loading 2304.10909v1…