PaperlayerView exact revision on arXiv

Source-linked AI summary

ICD Coding from Clinical Text Using Multi-Filter Residual Convolutional Neural Network

Fei Li, Hong Yu

arXiv:1912.00862v1cs.CLcs.LG

TL;DR

Automated ICD coding is labor-intensive, while fixed-length convolution may miss the varied patterns in clinical text. MultiResCNN combines multi-filter and residual convolutions, achieving state-of-the-art performance across MIMIC-III and MIMIC-II evaluations.

  • Problem

    Manual ICD coding is labor-intensive, and fixed-length convolution may inadequately represent clinical text patterns that vary in length and grammar.

  • Method

    MultiResCNN combines multi-filter convolutions for varied-length patterns with residual convolutions that enlarge the receptive field.

  • Results

    Across MIMIC-III and MIMIC-II experiments, MultiResCNN achieved state-of-the-art performance, outperforming baselines in all metrics for top-50 MIMIC-III and full-code MIMIC-II settings.

  • Takeaways & Limitations

    The results support deep and diversified text representations as a strong baseline for ICD coding and other text classification tasks.

  • Takeaways & Limitations

    The multi-filter kernel sizes and residual-layer channel sizes were chosen empirically and require further study and optimization.

Abstract

from arXiv · show

Automated ICD coding, which assigns the International Classification of Disease codes to patient visits, has attracted much research attention since it can save time and labor for billing. The previous state-of-the-art model utilized one convolutional layer to build document representations for predicting ICD codes. However, the lengths and grammar of text fragments, which are closely related to ICD coding, vary a lot in different documents. Therefore, a flat and fixed-length convolutional architecture may not be capable of learning good document representations. In this paper, we proposed a Multi-Filter Residual Convolutional Neural Network (MultiResCNN) for ICD coding. The innovations of our model are two-folds: it utilizes a multi-filter convolutional layer to capture various text patterns with different lengths and a residual convolutional layer to enlarge the receptive field. We evaluated the effectiveness of our model on the widely-used MIMIC dataset. On the full code set of MIMIC-III, our model outperformed the state-of-the-art model in 4 out of 6 evaluation metrics. On the top-50 code set of MIMIC-III and the full code set of MIMIC-II, our model outperformed all the existing and state-of-the-art models in all evaluation metrics. The code is available at https://github.com/foxlf823/Multi-Filter-Residual-Convolutional-Neural-Network.

Introduction

Automated ICD coding supports diagnostic and billing workflows, but existing CNN-based approaches may inadequately represent variable clinical text patterns. MultiResCNN addresses this limitation by combining multi-filter and residual convolutions, and it shows strong performance on MIMIC data.

  • ICD codes encode diagnostic and procedural information in patient records and support healthcare diagnosis and billing.
  • Prior automated ICD-coding research spans traditional machine learning, deep learning, multiple data domains, modalities, and full or partial code sets.
  • The state-of-the-art CNN-attention model uses one convolutional layer, which may inadequately capture variable ICD-related text spans and patterns.
  • MultiResCNN combines multi-filter convolution to capture patterns of different lengths with residual convolution to enlarge the receptive field.
  • On the widely used MIMIC dataset, MultiResCNN outperformed five existing and state-of-the-art models in nearly all reported evaluation metrics.The metrics include macro- and micro-AUC, macro- and micro-F1, and precision at K.

Related Work

Prior automated ICD-coding research progressed from early supervised classifiers and single-code assignment toward multi-code prediction and neural models. Deep-learning approaches introduced recurrent networks, attention, code-description structures, adversarial learning, and memory mechanisms.

  • Early automated coding: Larkey and Croft (1996) combined K-nearest-neighbor, relevance feedback, and Bayesian independence classifiers for ICD9 coding, but assigned only one code per discharge summary.Pestian et al. (2007) organized a shared task requiring models to assign a large set of codes to radiology reports.
  • Supervised machine learning: Early studies primarily used supervised machine learning, including flat and hierarchical SVMs for ICD9 coding of MIMIC-II discharge summaries.Other work evaluated supervised multi-label classification and learning-to-rank approaches on 71,463 electronic medical records.
  • Supervised machine learning: Supervised approaches also included cascaded SVMs for cancer-related causes of death and Random Forests with coverage-based feature selection for ICD9 prediction.The cascaded model identified cancer presence first and cancer type second using ICD-10 classification, while Random Forests were evaluated across seven medical specialties.
  • Neural approaches: Deep-learning studies applied LSTM and attention mechanisms, tree structures with adversarial learning, and condensed memory neural networks to automated ICD coding.These methods used diagnosis descriptions, code descriptions, and free-text medical notes from MIMIC-III.

Method

MultiResCNN represents clinical text with multi-filter convolutions that capture patterns of different lengths, followed by residual convolutional blocks that expand the representation while preserving sequence length. Per-label attention and a sigmoid output layer support multi-label ICD prediction, trained with binary cross entropy and Adam.

  • Multi-filter convolution: Multi-filter convolutions use different kernel sizes to capture text patterns with different lengths.The convolutions preserve the input sequence length by using padding and stride 1, while all filters share the same output size.
  • Residual convolution: Each multi-filter output feeds a residual convolutional layer composed of p residual blocks.Each residual block contains three convolutional filters; the first two use the corresponding multi-filter kernel size, while the third uses kernel size 1.
  • Residual convolution: The residual layer concatenates the final output from all m filter-specific residual paths into H ∈ R^n×(m×d_p).Residual-block outputs retain n rows, matching the input sequence length; the first residual block receives d_f channels and the last produces d_p channels.
  • Attention and output: Per-label attention lets each ICD code attend to different parts of document representation H before producing attention outputs V.The attention weights are defined for each ICD-code and word pair, with l denoting the number of ICD codes.
  • Training and output: The model treats ICD coding as multi-label classification, minimizes binary cross entropy, and trains with back-propagation and Adam.A linear layer and sum pooling produce code scores, which are converted to probabilities with the sigmoid function.

Experiments · Datasets

The experiments evaluated MultiResCNN on MIMIC-III using full and top-50 ICD-9 code settings, and on MIMIC-II for comparison with prior work. The datasets comprised discharge summaries or clinical notes with specified splits, code counts, and preprocessing procedures.

  • MIMIC-III: MIMIC-III served as the primary evaluation dataset, using discharge summaries split by patient IDs.Experiments covered both the full code set and the 50 most frequent codes.
  • MIMIC-III: 47,719, 1,631, and 3,372 MIMIC-III discharge summaries were allocated to training, development, and testing, respectively.The dataset used 8,921 ICD-9 codes.
  • MIMIC-II: MIMIC-II was additionally used to compare the models with previous work under the prior experimental setting.The dataset contained 5,031 unique ICD-9 codes.
  • MIMIC-II: 20,533 and 2,282 MIMIC-II clinical notes were used for training and testing, respectively.The split followed the experimental setting of previous work.
  • Preprocessing: Text preprocessing followed Mullenbach et al. (2018), including tokenization, lowercasing, and removal of tokens without alphabetic characters.Numbers and punctuation were among the removed tokens.
  • Preprocessing: Token sequences were capped at 2,500 tokens, with sequences exceeding that length truncated.This maximum-length rule was part of the reported preprocessing procedure.
  • Preprocessing: The preprocessing used scripts provided by Mullenbach et al. (2018).The scripts were identified as the preprocessing implementation.

Evaluation Metrics

The evaluation used macro- and micro-averaged AUC and F1, with precision-at-k metrics varying by dataset and code-set experiment.

  • MIMIC-III full codes: MIMIC-III full-code experiments used macro- and micro-averaged AUC, macro- and micro-averaged F1, P@8, and P@15.Macro-averaged metrics average per-label performance, whereas micro-averaged metrics aggregate across labels.
  • MIMIC-III top-50 codes: MIMIC-III top-50-code experiments used macro- and micro-averaged AUC, macro- and micro-averaged F1, and P@5.
  • MIMIC-II full codes: MIMIC-II full-code experiments used the same metrics as MIMIC-III top-50-code experiments, replacing P@5 with P@8.

Hyper-parameter Tuning

Hyper-parameter tuning compared convolutional variants and configurations on MIMIC-III development data, finding that multi-filter convolution benefits from increasing kernel diversity while residual convolution performs best with one block.

  • Fixed hyper-parameters: The word embedding size is 100, filter out-channel size is 100, learning rate is 0.0001, batch size is 16, and dropout rate is 0.2.These values were chosen empirically or following prior work because exhaustive tuning of all hyper-parameters was infeasible.
  • Evaluation protocol: Each configuration was evaluated in three randomly initialized runs, with table results reported as the means.Experiments used the MIMIC-III development set under full and top-50 code settings.
  • Configuration comparison: MultiCNN outperforms CNN, with performance consistently increasing as the number of kernels grows in both full and top-50 code settings.CNN uses one convolutional filter, whereas MultiCNN uses only the multi-filter convolutional layer.
  • Multi-filter convolution: Performance peaks with kernel sizes 3,5,9,15,19,25.These sizes capture text patterns at different granularities while preserving sequence length after convolution.
  • Residual convolution: ResCNN outperforms CNN, but performance deteriorates as the residual block number increases and is best with one block.ResCNN contains only the residual convolutional layer, while CNN uses one convolutional filter.

Baselines

The paper compares MultiResCNN with convolutional, recurrent, memory-based, and SVM baselines for automatic ICD coding on MIMIC datasets. These baselines include CAML and its description-regularized extension, models targeting top-50 codes, and hierarchical approaches.

  • CAML: CAML uses one convolutional layer and one attention layer to generate label-aware features for multi-label classification.It achieved state-of-the-art results on MIMIC-III and MIMIC-II among models using unstructured text.
  • CAML: DR-CAML extends CAML by incorporating each code’s text description to regularize the model.
  • Memory-based baseline: The Condensed Memory Neural Network uses iterative condensed memory representations and achieved competitive results for predicting the top-50 MIMIC-III ICD codes.
  • Recurrent baseline: A Characteraware LSTM-based Attention model uses LSTM language models for note and code representations and attention to address their mismatch.It focuses on predicting the top-50 ICD codes in MIMIC-III.
  • Traditional and recurrent baselines: Hierarchy SVM models ICD9-code dependencies and outperforms flat SVM, achieving 29.3% f1-measure on MIMIC-II.HA-GRU instead encodes sentences and documents in separate layers and reports results on MIMIC-II using Perotte et al.’s data split.

Results

MultiResCNN outperformed strong ICD-coding baselines across several evaluation settings, with especially consistent gains on one setting and all-metric gains on another. On MIMIC-II, it also exceeded prior SVM- and attention/GRU-based methods.

  • Comparison with existing work: MultiResCNN improved macro-AUC by 0.013, micro-F1 by 0.013, precision@8 by 0.025, and precision@15 by 0.023 over CAML and DR-CAML.It achieved comparable micro-AUC and slightly worse macro-F1.
  • Comparison with existing work: MultiResCNN outperformed C-MemNN, C-LSTM-Att, CAML, and DR-CAML in all evaluation metrics.Its improvements in macro-AUC, micro-AUC, macro-F1, micro-F1, and precision@5 were 0.015, 0.012, 0.030, 0.037, and 0.023, respectively.
  • MIMIC-II results: On the MIMIC-II full code set, MultiResCNN exceeded Perotte et al.’s SVM by 0.171 in micro-F1 and Baumel et al.’s attention-and-GRU model by 0.098.It also outperformed CAML or DR-CAML by 0.024, 0.002, 0.003, 0.007, and 0.021 across all evaluation metrics.

Discussion

The discussion finds that MultiResCNN has higher computational costs than CAML but converges in fewer epochs, while performance is relatively insensitive to discharge-summary length limits. The authors attribute gains to deep, diversified text representations and identify BERT integration as future work constrained by hardware and fixed-length context.

  • Computational cost: MultiResCNN uses approximately 1.9 times as many parameters and about 2.3 times more training time than CAML.The authors attribute these costs to MultiResCNN’s greater number of filters and layers.
  • Computational cost: CAML is approximately 1.5 times faster than MultiResCNN at inference, although MultiResCNN needs much fewer epochs to converge.Overall, the authors characterize MultiResCNN’s computational cost as larger than CAML’s.
  • Length limitation: Varying the discharge-summary truncation limit from 2,500 to 6,500 tokens produces no significant performance differences.Across settings, P@8 ranges from 0.736 to 0.741, while the reported micro-F1 range is truncated in the supplied passage.
  • Limitations and future work: The authors attribute most performance improvement to deep and diversified text representations and plan to explore effective, efficient BERT integration.Preliminary BERT experiments performed poorly because of hardware limitations and fixed-length context; recurrent Transformer and hierarchical BERT are suggested as potential solutions.

Conclusions

The paper proposes MultiResCNN for ICD coding and evaluates it on MIMIC-III and MIMIC-II, finding state-of-the-art performance against competitive baselines. Both multi-filter and residual convolution improve performance with acceptable computational cost.

  • The proposed multi-filter residual convolutional neural network targets automated ICD coding.The model combines multi-filter and residual convolutional components.
  • Experiments on MIMIC-III and MIMIC-II show state-of-the-art performance compared with several competitive baselines.The study reports three experiments across the two widely used datasets.
  • Both multi-filter convolution and residual convolution improve performance with acceptable computational cost.The findings suggest that deep and diversified text representations benefit ICD coding.
Loading 1912.00862v1…