PaperlayerView exact revision on arXiv

Source-linked AI summary

A Label Attention Model for ICD Coding from Clinical Text

Thanh Vu, Dat Quoc Nguyen, Anthony Nguyen

arXiv:2007.06351v1cs.CLcs.LG

TL;DR

Manual ICD coding is labor-intensive and error-prone, and fixed-window CNN approaches face difficulty with variable-length, interdependent code-related text. The paper introduces label-specific attention with hierarchical joint learning and reports state-of-the-art results across three benchmark MIMIC datasets, with improved performance for infrequent codes.

  • Problem

    Manual ICD coding is labor-intensive and error-prone, while fixed-window CNN models struggle to select windows for variable-length, interdependent code-related fragments amid long-tailed code usage.

    Source: §1 · p. 1

  • Method

    The model uses a BiLSTM and label-specific attention vectors for code prediction, extended with hierarchical joint learning that uses ICD-code relationships.

    Source: §1 · p. 1, §3.1 · p. 2, §3.2 · p. 3

  • Results

    Across three benchmark MIMIC datasets, LAAT achieves new state-of-the-art performance, while hierarchical joint learning improves performance for infrequent codes and higher macro-averaged metrics.

    Source: §1 · p. 2, §6 · p. 6

  • Takeaways & Limitations

    The reported results support hierarchical joint learning as a way to improve infrequent-code performance within the evaluated benchmark setting.

    Source: §6 · p. 6

Abstract

from arXiv · show

ICD coding is a process of assigning the International Classification of Disease diagnosis codes to clinical/medical notes documented by health professionals (e.g. clinicians). This process requires significant human resources, and thus is costly and prone to error. To handle the problem, machine learning has been utilized for automatic ICD coding. Previous state-of-the-art models were based on convolutional neural networks, using a single/several fixed window sizes. However, the lengths and interdependence between text fragments related to ICD codes in clinical text vary significantly, leading to the difficulty of deciding what the best window sizes are. In this paper, we propose a new label attention model for automatic ICD coding, which can handle both the various lengths and the interdependence of the ICD code related text fragments. Furthermore, as the majority of ICD codes are not frequently used, leading to the extremely imbalanced data issue, we additionally propose a hierarchical joint learning mechanism extending our label attention model to handle the issue, using the hierarchical relationships among the codes. Our label attention model achieves new state-of-the-art results on three benchmark MIMIC datasets, and the joint learning mechanism helps improve the performances for infrequent codes.

1 Introduction

ICD coding assigns diagnosis and procedure codes from patient-visit data, but manual coding is labor-intensive and error-prone, while automatic coding faces many long-tailed labels. This paper proposes label-specific attention and hierarchical joint learning, reporting state-of-the-art performance on three benchmark MIMIC datasets and gains for infrequent codes.

1 Introduction

  • Manual ICD coding relies on inspection and experience-based judgment, making it time- and labor-intensive and prone to human error.ICD codes represent diagnoses and procedures assigned using patient-visit data such as clinical notes.Source: §1 · p. 1
  • The large code inventory and long-tailed usage make automatic ICD coding highly imbalanced, with most codes used infrequently.The passage gives approximate inventories of 17,000 ICD-9-CM codes and 140,000 ICD-10-CM/PCS codes.Source: §1 · p. 1
  • CNN-based state-of-the-art models use fixed window sizes, but clinical-text fragments relevant to a code vary in length and may be interdependent, making window selection challenging.The example links “past medical history” and “prostate cancer” to a single code despite their distance in the note.Source: §1 · p. 1
  • The proposed label attention model uses a BiLSTM and label-specific vectors to represent relevant text fragments, while hierarchical joint learning uses code relationships to address imbalance.Each label-specific vector supports a binary classifier for its code.Source: §1 · p. 1
  • Across three standard benchmark MIMIC datasets, the model achieves new state-of-the-art performance, and hierarchical joint learning improves performance for infrequent codes.The conclusion reports substantial improvements across evaluation metrics and higher macro-averaged metrics for infrequent-code gains.Source: §1 · pp. 1–2

2 Related Work

Prior ICD-coding research spans conventional classifiers and deep learning, including attention-based and hierarchical approaches. CNN-based models use convolution and attention, while later work adds dense, multi-scale, or residual designs.

2 Related Work

  • Earlier ICD-coding work used feature-based ensembles, code-description similarity, and flat or hierarchical SVM classifiers across discharge summaries, death certificates, and radiology reports.The cited examples include inpatient summaries, MIMIC-II discharge summaries, and cancer-related coding from death certificates.Source: §2 · p. 2
  • Deep-learning approaches include character-level LSTM with attention, joint label-word embeddings, and convolutional attention for clinical text.The joint embedding model predicts labels using cosine similarity in a shared vector space.Source: §2 · p. 2
  • Later CNN-based models add densely connected or multi-filter residual convolutions, multi-scale feature attention, and graph convolutional regularization for code relationships.The multi-filter residual CNN combines multi-filter and residual convolutional layers.Source: §2 · p. 2

3 Approach

LAAT builds a separate clinical-text representation for each ICD code, while JointLAAT adds hierarchical prediction to address infrequent codes. The approach combines contextual token encoding, label-specific attention, and code-level classification.

3.1 Our Label Attention Model

  • LAAT encodes clinical text with a BiLSTM, creates a label-specific text representation for each code, and uses a binary classifier to predict code assignment.Its four layers are word embeddings, a BiLSTM, label-specific attention, and label-specific binary classifiers.Source: §3.1 · p. 2

Embedding Layer

  • Each input token is represented by a pretrained word embedding before contextual encoding.The embedding size is denoted de.Source: Embedding Layer · p. 2

Bidirectional LSTM Layer

  • The BiLSTM captures context across input words, and concatenated forward and backward states form each token's latent vector.The token vectors are combined into a matrix H representing the document.Source: Bidirectional LSTM Layer · p. 3

Attention Layer

  • Label attention transforms H into a distinct document representation for each ICD label by weighting token states.The weight matrix has one row per label, and multiplying the attention weights by H produces the label-specific vectors.Source: Attention Layer · p. 3

Output Layer

  • A single-layer feed-forward network with a sigmoid output predicts each label's probability, and training minimizes binary cross-entropy.A predefined threshold, such as 0.5, converts the probability into a binary prediction.Source: Output Layer · p. 3

Discussion

  • Unlike structured self-attention's single text embedding for classification, LAAT uses each attention hop separately for a label-specific classifier.The paper describes single aggregated text embeddings as suitable for single-label classification but not for multi-label ICD coding.Source: Discussion · p. 3

3.2 Hierarchical Joint Learning Mechanism

  • The hierarchical mechanism uses shared first-three-character code prefixes to group raw codes and targets the challenge of infrequent-code prediction.For example, several meningitis codes share the higher-order category 322.Source: §3 · p. 2, §3.2 · p. 3
  • JointLAAT predicts normalized codes from the first three characters, then uses those predictions to help predict raw ICD codes.The normalized-code prediction is embedded and concatenated with each raw-code label vector before the final prediction; training sums binary cross-entropy losses at both levels.Source: §3.2 · pp. 3–4, Figure 2 · p. 3

4 Experimental Setup

The evaluation follows prior ICD-coding work and compares the models across MIMIC datasets, using established metrics, training procedures, and baselines.

4.1 Datasets

  • MIMIC-III-full uses 52,722 discharge summaries and 8,929 codes, with patient-level separation between training, validation, and test sets; MIMIC-III-50 and MIMIC-II-full provide additional settings.The MIMIC-III-50 subset contains the 50 most frequent codes, while MIMIC-II-full contains 5,031 unique codes.Source: §4.1 · p. 4

4.2 Evaluation Metrics

  • Evaluation reports macro- and micro-averaged F1 and AUC, plus precision at k for k values 5, 8, and 15.Macro-averaging takes a simple average over labels; micro-averaging pools text-code decisions.Source: §4.2 · p. 4

4.3 Implementation and Hyper-parameter Tuning

  • Models use AdamW, dropout of 0.3, early stopping on validation micro-F1, and scores averaged across 10 runs with different random seeds.The batch size is 8 and training runs for up to 50 epochs; JointLAAT uses projection size p=128.Source: §4.3 · p. 4

4.4 Baselines

  • Baselines include conventional classifiers and neural models, including CAML, multi-scale CNN approaches, hierarchical SVM, and logistic regression.CAML uses a CNN with label-dependent attention; MSATT-KG and MultiResCNN use convolutional features at multiple scales.Source: §4.4 · pp. 4–5

5 Experimental Results

LAAT outperforms prior models across several benchmark settings, while JointLAAT improves rare-code performance on the full datasets. Ablations favor the proposed label attention and BiLSTM encoder.

MIMIC-III-full

  • LAAT improves macro-AUC by 0.9%, macro-F1 by 0.9%, micro-F1 by 2.2%, P@8 by 0.4% and P@15 by 0.7% over MASATT-KG and MultiResCNN on MIMIC-III-full.Its P@5 is 81.3%.Source: MIMIC-III-full · p. 5
  • JointLAAT improves macro-F1 over LAAT by 0.8% (p < 0.01) on MIMIC-III-full, indicating better performance on infrequent codes.JointLAAT also produces a better macro-AUC score; its P@k scores are slightly lower, but the differences are not significant.Source: MIMIC-III-full · p. 5

MIMIC-III-50

  • LAAT improves macro-AUC by 1.1%, micro-AUC by 1.0%, macro-F1 by 2.8%, micro-F1 by 3.1% and P@5 by 3.1% over MSATT-KG on MIMIC-III-50.LAAT outperforms all baselines across the reported metrics; LAAT and JointLAAT show no significant score differences, possibly because the dataset has no infrequent codes.Source: MIMIC-III-50 · pp. 5–6, Table 2 · p. 6

MIMIC-II-full

  • LAAT improves macro-AUC by 1.8%, micro-AUC by 0.5%, macro-F1 by 0.7%, micro-F1 by 2.2% and P@8 by 0.6% over MultiResCNN on MIMIC-II-full.JointLAAT improves macro-F1 over LAAT by 0.9% (p < 0.01), indicating better performance on infrequent codes.Source: Table 3 · p. 6, MIMIC-II-full · p. 6

5.2 Ablation Study

  • LAAT’s label attention performs better than CAML’s label attention when both use the BiLSTM encoder.The ablation evaluates the variants on the MIMIC-III-full validation set; all score differences from LAAT are significant (p < 0.01).Source: §5.2 · p. 6, Table 4 · p. 6
  • With the same attention mechanism, the BiLSTM-based LAAT scores better than the CNN-based CAMLLAAT.All score differences from LAAT are significant (p < 0.01).Source: §5.2 · p. 6, Table 4 · p. 6
  • The BiLSTM-based LAAT scores higher than the BiGRU-based LAATGRU.The authors suggest LSTM’s separate memory cells may make it more suitable for long clinical text; all score differences from LAAT are significant (p < 0.01).Source: §5.2 · p. 6, Table 4 · p. 6

6 Conclusions

The paper concludes that LAAT achieves new state-of-the-art performance on three benchmark MIMIC datasets. Its hierarchical joint learning architecture significantly improves performance for infrequent codes, raising macro-averaged metrics.

6 Conclusions

  • LAAT achieves new state-of-the-art performance with substantial improvements across various metrics over competitive baselines on three standard benchmark MIMIC datasets.The hierarchical joint learning architecture also significantly improves performance for infrequent codes, resulting in higher macro-averaged metrics.Source: §6 · p. 6
  • MIMIC documents average about 1,500 word tokens and can exceed 6,500.These document lengths contextualize the paper’s focus on modeling clinical text.Source: §6 · p. 6

Related papers

Loading 2007.06351v1…