PaperlayerView exact revision on arXiv

Source-linked AI summary

Improving Chest X-Ray Report Generation by Leveraging Warm Starting

Aaron Nicolson, Jason Dowling, Bevan Koopman

arXiv:2201.09405v2cs.CV

TL;DR

Current CXR report generators lack the diagnostic accuracy needed for clinical deployment. This paper evaluates warm-started CV and NLP checkpoints on MIMIC-CXR and IU X-Ray, finding that CvT-21 ImageNet-21K and DistilGPT2 are the strongest encoder and decoder choices, respectively, with CvT2DistilGPT2 outperforming M^2 Transformer Progressive across reported metrics.

  • Problem

    Current CXR report generators lack the diagnostic accuracy required for clinical deployment.

  • Method

    The study evaluates recent publicly available CV and NLP checkpoints for warm starting the encoder and decoder of a CXR report generator on MIMIC-CXR and IU X-Ray.

  • Results

    CvT2DistilGPT2 improved over M^2 Transformer Progressive by 8.3% for CE F-1, 1.8% for BLEU-4, 1.6% for ROUGE-L, and 1.0% for METEOR.

  • Takeaways & Limitations

    CvT-21 ImageNet-21K and DistilGPT2 were identified as the best CV and NLP checkpoints for warm starting the encoder and decoder, respectively.

  • Takeaways & Limitations

    The study did not consider every encoder-decoder checkpoint combination or CXRs in DICOM format.

Abstract

from arXiv · show

Automatically generating a report from a patient's Chest X-Rays (CXRs) is a promising solution to reducing clinical workload and improving patient care. However, current CXR report generators -- which are predominantly encoder-to-decoder models -- lack the diagnostic accuracy to be deployed in a clinical setting. To improve CXR report generation, we investigate warm starting the encoder and decoder with recent open-source computer vision and natural language processing checkpoints, such as the Vision Transformer (ViT) and PubMedBERT. To this end, each checkpoint is evaluated on the MIMIC-CXR and IU X-Ray datasets. Our experimental investigation demonstrates that the Convolutional vision Transformer (CvT) ImageNet-21K and the Distilled Generative Pre-trained Transformer 2 (DistilGPT2) checkpoints are best for warm starting the encoder and decoder, respectively. Compared to the state-of-the-art ($\mathcal{M}^2$ Transformer Progressive), CvT2DistilGPT2 attained an improvement of 8.3\% for CE F-1, 1.8\% for BLEU-4, 1.6\% for ROUGE-L, and 1.0\% for METEOR. The reports generated by CvT2DistilGPT2 have a higher similarity to radiologist reports than previous approaches. This indicates that leveraging warm starting improves CXR report generation. Code and checkpoints for CvT2DistilGPT2 are available at https://github.com/aehrc/cvt2distilgpt2.

1. Introduction

The paper investigates warm starting encoder-to-decoder CXR report generators with publicly available CV and NLP checkpoints to produce reports more similar to radiologists’ reports. It evaluates checkpoint choices on MIMIC-CXR and IU X-Ray, motivated by dataset deficiencies and the need for improved diagnostic accuracy.

  • Current CXR report generators need substantial diagnostic-accuracy improvements before clinical adoption.
  • Warm starting initializes model parameters from a related pre-trained checkpoint, potentially transferring knowledge and improving performance.The approach is especially effective when the target dataset is limited or lower quality and the pre-training domain is similar.
  • Checkpoint selection involves balancing general-domain data scale against domain similarity to CXR reporting.The paper notes that ICU EHR vocabulary is closer to CXR reports, while general-domain corpora are larger.
  • The authors aim to identify suitable checkpoints, understand how pre-training domain, dataset size, and architecture affect effectiveness, and recommend improvements.
  • The study evaluates general-domain and domain-specific encoder and decoder checkpoints on MIMIC-CXR and IU X-Ray using several objective metrics.Different encoder-only and decoder-only checkpoints can be combined, creating many possible warm-start configurations.

2. Background

CXR report generation could reduce reporting burden and support care, but current systems require better diagnostic accuracy. The background reviews encoder-to-decoder architectures and the growing range of CV and NLP checkpoints available for warm starting.

  • CXRs support cardiovascular and respiratory disease diagnosis, while fatigue and inconsistent reporting can contribute to sub-optimal patient care.
  • CXR report generation may provide editable pre-filled reports, reduce radiology workload, and support clinicians with lower diagnostic confidence.
  • Current approaches are promising but need significant diagnostic-accuracy improvements before clinical adoption.
  • Encoder-to-decoder systems extract visual features with an encoder and generate reports with a Transformer decoder conditioned on image features and prior words.
  • The study considers newer CV checkpoints, including ViT, DeiT, and CvT, alongside general-domain and biomedical NLP checkpoints.DistilGPT2 is a general-domain NLG checkpoint, while BERT-family checkpoints represent NLU alternatives.
  • Warm starting the decoder with biomedical NLU checkpoints is presented as a promising but insufficiently investigated direction for CXR report generation.

3. Related work

Prior CXR report-generation studies improve language and clinical-efficacy metrics through architectural, knowledge-based, contrastive, and reinforcement-learning methods. The reviewed literature establishes encoder-to-decoder models and warm-started checkpoints as important foundations for comparison.

  • The M^2 Transformer Progressive approach combines abnormality prediction, GCN-derived labels, and BART report generation.
  • Earlier work also used warm-started ResNet, CheXNet, and DistilGPT2 components, but investigated different conditioning mechanisms.
  • NLG metrics measure report similarity but do not always capture diagnostic accuracy, motivating Clinical Efficacy metrics.CE scores compare CheXpert observations extracted from generated and ground-truth reports.
  • SCST uses a reward derived from clinical-efficacy scores and improved both NLG metrics and diagnostic accuracy over earlier approaches.
  • Knowledge-enhanced and differentiable CheXpert-based models outperformed earlier approaches on multiple language or clinical-efficacy evaluations.
  • R2Gen, CMN, Contrastive Attention, and PPKED each improved over earlier RNN-decoder approaches using combinations of NLG and CE metrics.

4. Contributions

The paper systematically studies publicly available CV and NLP checkpoints for CXR report generation, including previously unexplored model families and domain-specific checkpoints. It also adds interpretability and fine-grained abnormality evaluations.

  • The study investigates publicly available checkpoints for warm starting CXR report generators.
  • It evaluates whether Transformer-based CV encoders outperform CNN encoders and whether NLP checkpoints can model visual features.
  • It compares NLU checkpoints such as BERT with NLG checkpoints such as GPT2 for decoder warm starting.
  • Previously unexamined CV models include EfficientNet and Transformer encoder architectures, alongside domain-specific CV and NLP checkpoints.
  • The final model includes cross-attention visualizations to show what the model attends to during report generation.
  • The study evaluates individual abnormalities at fine granularity, addressing a gap in recent work.

5. Datasets

The study uses MIMIC-CXR and IU X-Ray with dataset splits adopted from Chen et al. (2020). MIMIC-CXR provides single-view examples, whereas IU X-Ray uses paired frontal and lateral views.

  • MIMIC-CXR contains 377,110 CXRs and 227,835 English radiology reports associated with 64,588 patients.
  • IU X-Ray contains 3,955 English radiology reports and 7,470 CXRs, with each report associated with one patient.
  • The experiments adopt the MIMIC-CXR and IU X-Ray dataset splits and labels established by Chen et al. (2020).
  • MIMIC-CXR uses 270,790 training, 2,130 validation, and 3,858 testing CXRs.
  • IU X-Ray uses 2,069 training, 296 validation, and 590 testing reports.
  • Each IU X-Ray example contains frontal and lateral views, while each MIMIC-CXR example contains either a frontal or lateral view; patients appear in only one subset.

6. Problem formulation

The task maps one or more CXRs to an automatically generated report using an encoder-to-decoder architecture. The encoder extracts visual features, which are projected into the decoder and generated autoregressively.

  • Given a set of N CXRs, the encoder-to-decoder model aims to generate a report whose target is the ground-truth report.
  • The encoder processes each CXR independently and produces spatial visual features from its last hidden state.
  • Visual features from multiple CXRs are concatenated along the spatial-position axis before decoder projection.
  • A learned projection matrix maps the visual features to the decoder hidden-state size.
  • Projected visual features enter a randomly initialized multi-head cross-attention module inserted into each decoder layer.
  • The decoder generates reports autoregressively; N=1 for MIMIC-CXR and N=2 for IU X-Ray.

7. Methodology

The methodology evaluates publicly available computer-vision and language checkpoints for warm-starting CXR report generators, using standard report-generation and clinical-efficacy metrics. Training, preprocessing, decoding, and confidence estimation are specified consistently across experiments.

  • The study evaluates publicly available CV checkpoints for encoder warm starting and NLP checkpoints for decoder warm starting.
  • Generated reports are evaluated with NLG word-overlap metrics and clinical efficacy metrics.
  • BLEU measures word n-gram overlap, while METEOR incorporates unigram Fβ, stemming, and synonym matching.
  • ROUGE-L uses the longest common n-gram, and CIDEr averages TF-IDF-weighted cosine similarities over 1- to 4-grams.
  • Word-overlap measures do not necessarily capture diagnostic accuracy.
  • CheXbert extracts 14 observations and supports example-based and label-based precision, recall, and F-1 scores.
  • The pipeline standardizes inputs, fine-tunes with teacher forcing and categorical cross-entropy, and uses beam search during testing.
  • Models are selected by validation CIDEr, repeated across training runs, and evaluated on the matching dataset test set.

8. Results and discussion

The evaluation compares CvT2DistilGPT2 with existing generators, examines its diagnostic behavior, and identifies suitable computer-vision and NLP checkpoints for warm starting.

  • Comparison with existing generators: CvT2DistilGPT2 attained the highest mean scores for all NLG metrics on the MIMIC-CXR test set.
  • Comparison with existing generators: CvT2DistilGPT2 attained the highest precision and second-highest recall, leading to the highest example-based CE F-1 score.
  • Comparison with existing generators: On the IU X-Ray test set, CvT2DistilGPT2 attained the highest mean BLEU-3, BLEU-4, METEOR, and CIDEr scores.Contrastive Attention attained the highest mean ROUGE-L score.
  • Case study: The case study shows that CvT2DistilGPT2 predicted mild cardiomegaly but incorrectly described it as unchanged based on an unobserved previous study.The model had learned to describe abnormalities in the context of disease progression.
  • Encoder checkpoint selection: CvT-21 was selected over DeiT for encoder warm starting because their metric differences were not statistically significant and CvT-21 was more parameter-efficient.CvT-21 achieved higher mean BLEU-4, METEOR, ROUGE-L, CIDEr, and macro-averaged precision scores, whereas DeiT achieved higher recall-related scores.
  • Encoder checkpoint selection: CheXNet showed a clear advantage for warm starting the encoder with a domain-specific computer-vision checkpoint over a general-domain checkpoint.CvT-21 also consistently outperformed EfficientNet among the evaluated Transformer-based checkpoints.
  • Decoder checkpoint selection: GPT2 and DistilGPT2 were the most suitable decoder checkpoints, with no significant NLG-score difference after distillation reduced GPT2 to 66% of its parameters.DistilGPT2 was selected because of its parameter efficiency.
  • Decoder checkpoint selection: GPT2 outperformed BERT, while domain proximity alone did not ensure better performance for EHR NLU checkpoints.The authors suggest pre-training dataset size and domain-specific vocabularies may also matter.

9. Limitations and future recommendations

The investigation identifies limitations involving checkpoint selection, image format and resolution, single-view inputs, missing prior studies, class imbalance, and newer unavailable checkpoints. It recommends future work addressing these constraints and expanding the model’s inputs and evaluation.

  • Limitations: The study did not evaluate every encoder–decoder checkpoint combination, assuming encoder choice does not affect decoder choice and vice versa.The authors acknowledge this assumption may be wrong and recommend investigating it further.
  • Limitations: JPEG and PNG inputs used 8-bit pixel depth instead of the 12 bits available in DICOM, so reducing quantisation error could improve performance.
  • Limitations: Downsampling CXRs to 384×384 increases the risk of missing fine details; future inputs should be closer in resolution to the originals.
  • Limitations: Single-CXR report generation omits information from additional views, although frontal and lateral views can make some abnormalities easier to detect.
  • Future recommendations: Variable numbers of CXRs could improve performance, while incorporating previous studies may improve generation and support disease-progression monitoring.
  • Future recommendations: Under- or over-sampling could reduce the impact of class imbalance in the MIMIC-CXR training set, and newer non-open-source checkpoints may outperform CvT.

8. As demonstrated by CheXNet outperforming the DenseNet-121 ImageNet-21K checkpoint, a domain-specific

The section recommends investigating newer computer-vision checkpoints and large language model decoders as potential alternatives for CXR report generation.

  • Large language models have produced Transformer decoder checkpoints with considerable capabilities, motivating their investigation as CXR report-generation decoders.

10. This study indicates that the following modifications to DistilGPT2 checkpoint should be investigated: i) pre-

The section proposes modifying DistilGPT2 and emphasizes that clinical deployment requires diagnostic accuracy to be established against an appropriate threshold. It also identifies incomplete clinical evaluation as a limitation.

  • DistilGPT2 modifications to investigate include pre-training on PubMed and PMC from scratch before distillation and using a domain-specific vocabulary.
  • Diagnostic accuracy must reach an appropriate threshold before retrospective or prospective clinical trials are considered appropriate.
  • Current CXR report generators show poor diagnostic accuracy for multiple abnormalities, indicating that further research and development is required before reaching the threshold.
  • The study did not include Receiver Operating Characteristics curves and plans to consider them in future work for a more thorough investigation.

13. Metrics that evaluate the clinical decisions inferred from the generated reports should be considered—as clinical

Clinical metrics should evaluate decisions inferred from generated reports because clinical decisions ultimately dictate patient care.

  • Clinical evaluation should include metrics for decisions inferred from generated reports because those decisions ultimately dictate patient care.

10. Conclusion

The study finds that warm starting with suitable computer-vision and natural-language checkpoints improves CXR report generation. CvT-21 ImageNet-21K and DistilGPT2 are identified as the strongest encoder and decoder initializations, respectively, with CvT2DistilGPT2 outperforming previous approaches on diagnostic and language metrics.

  • Findings: CvT-21 ImageNet-21K, combining convolutional layers with Transformer-based vision modeling, was the best checkpoint for warm starting the encoder.The study reports that a Transformer-based computer-vision checkpoint incorporating convolutional layers performs better than a CNN checkpoint.
  • Findings: DistilGPT2 was the best checkpoint for warm starting the decoder, outperforming BERT-based alternatives.The findings support effective fine-tuning of an NLP checkpoint to model both natural language and visual features.
  • Findings: CvT2DistilGPT2 generated reports with higher diagnostic accuracy and greater similarity to radiologist reports than previous approaches.The conclusion attributes these outcomes to the warm-started model configuration.
  • Implications: The best checkpoint depends on the pre-training task, dataset size and quality, vocabulary, and model architecture.Domain-specific checkpoints are better when the pre-training dataset has sufficient size or quality and a similar domain to the target task.
  • Implications: Clinically validated and regulation-compliant CXR report generation could support more consistent, reliable, and lower-cost radiology reporting.The paper also identifies possible applications in retrospective report analysis, question answering, and population research.
Loading 2201.09405v2…