PaperlayerView exact revision on arXiv

Source-linked AI summary

A Vision-Language Foundation Model to Enhance Efficiency of Chest X-ray Interpretation

Zhihong Chen, Maya Varma, Justin Xu, Magdalini Paschali, Dave Van Veen, Andrew Johnston, Alaa Youssef, Louis Blankemeier, Christian Bluethgen, Stephan Altmayer, Jeya Maria Jose Valanarasu, Mohamed Siddig Eltayeb Muneer, Eduardo Pontes Reis, Joseph Paul Cohen, Cameron Olsen, Tanishq Mathew Abraham, Emily B. Tsai, Christopher F. Beaulieu, Jenia Jitsev, Sergios Gatidis, Jean-Benoit Delbrouck, Akshay S. Chaudhari, Curtis P. Langlotz

arXiv:2401.12208v2cs.CVcs.CL

TL;DR

Task-specific CXR models have narrow scope, while the high volume of CXR interpretation burdens radiologists. This paper builds CheXinstruct, trains CheXagent, and evaluates it across diverse tasks, reporting cross-task performance advantages and resident time savings in a reader study.

  • Problem

    Task-specific CXR models address narrow predefined tasks, while the high volume of imaging and interpretation places a significant burden on radiologists.

    Source: Introduction · p. 2

  • Method

    The authors curated CheXinstruct from public datasets, trained the vision-language model CheXagent, and evaluated it across diverse CXR interpretation tasks using CheXbench.

    Source: Introduction · p. 2, Creating CheXinstruct, CheXagent, and… · p. 4, Discussion · p. 12

  • Results

    CheXagent consistently outperformed baselines, including significantly larger FMs like GPT-4V, across evaluations; residents using its drafted reports saved 36% of their time compared with writing from scratch.

    Source: Discussion · pp. 12–13

  • Takeaways & Limitations

    CheXagent showed potential to improve CXR interpretation efficiency while maintaining quality, and CheXinstruct and CheXbench provide resources for future CXR FM research and standardized evaluation.

    Source: Discussion · p. 13

    CheXagent has 3.1 billion parameters, and future work could explore how model scaling affects CXR FM performance.

    Source: Discussion · p. 13

Abstract

from arXiv · show

Over 1.4 billion chest X-rays (CXRs) are performed annually due to their cost-effectiveness as an initial diagnostic test. This scale of radiological studies provides a significant opportunity to streamline CXR interpretation and documentation. While foundation models are a promising solution, the lack of publicly available large-scale datasets and benchmarks inhibits their iterative development and real-world evaluation. To overcome these challenges, we constructed a large-scale dataset (CheXinstruct), which we utilized to train a vision-language foundation model (CheXagent). We systematically demonstrated competitive performance across eight distinct task types on our novel evaluation benchmark (CheXbench). Beyond technical validation, we assessed the real-world utility of CheXagent in directly drafting radiology reports. Our clinical assessment with eight radiologists revealed a 36% time saving for residents using CheXagent-drafted reports, while attending radiologists showed no significant time difference editing resident-drafted or CheXagent-drafted reports. The CheXagent-drafted reports improved the writing efficiency of both radiology residents and attending radiologists in 81% and 61% of cases, respectively, without loss of quality. Overall, we demonstrate that CheXagent can effectively perform a variety of CXR interpretation tasks and holds potential to assist radiologists in routine clinical workflows.

Introduction

The introduction frames high-volume CXR interpretation as a burden and argues that task-specific models have narrow scope, while foundation-model development lacks large, diverse datasets and holistic evaluation. It presents CheXinstruct, CheXagent, and CheXbench, then reports benchmark and reader-study results.

Introduction

  • CXR interpretation creates a substantial workload for radiologists, and the high volume may contribute to burnout and risks of misidentification or delayed reporting.CXRs comprise approximately 40% of 3.6 billion diagnostic X-ray examinations performed worldwide each year.Source: Introduction · p. 2
  • Task-specific models address narrow predefined tasks and may miss opportunities to leverage complementary knowledge across tasks.The introduction gives report generation and disease localization as distinct tasks whose joint training can support reports sensitive to fine-grained disease information.Source: Introduction · p. 2
  • Foundation models are presented as promising for diverse CXR tasks, but development is hindered by limited curated training data and holistic benchmarks.The field had primarily focused on report generation without robust evaluation of other capabilities critical to CXR interpretation.Source: Introduction · p. 2
  • The study curated CheXinstruct from 32 publicly available datasets, comprising 8.5 million training samples across 35 tasks, used it to train CheXagent, and introduced CheXbench as a comprehensive benchmark.Source: Introduction · p. 2
  • CheXagent outperformed prior medical foundation models, general-domain foundation models, and task-specific models across the evaluated tasks.The benchmark spans image perception, image-text reasoning, and text generation.Source: Introduction · p. 2
  • 36% average time savings were achieved by residents using CheXagent-drafted reports rather than writing reports from scratch.Attending radiologists showed no significant time difference when editing CheXagent-drafted versus resident-drafted reports.Source: Introduction · p. 2

Results

CheXagent was trained using CheXinstruct and evaluated across CXR perception, image-text reasoning, text generation, and a clinical reader study. It outperformed other models on perception and reasoning tasks, showed competitive text-generation performance, and reduced residents’ report-drafting time.

Creating CheXinstruct, CheXagent, and CheXbench

  • CheXinstruct curates 8.5 million triplets across 35 tasks from 32 public datasets; CheXagent’s three-stage training ends with joint instruction tuning.Some tasks, including findings summarization, are text-only and do not include images in their triplets.Source: Creating CheXinstruct, CheXagent, and… · p. 4

Performance on Image Perception

  • CheXagent consistently outperformed other FMs on image-perception tasks and other FMs and task-specific models on image-text reasoning, while achieving competitive text-generation performance.The clinical reader study also found significantly shorter report-production times for residents editing CheXagent drafts than for residents drafting from scratch.Source: Performance on Image Perception · p. 4, Performance on Image-Text Reasoning · p. 7, Performance on Text Generation · p. 10
  • CheXagent achieved 0.993 accuracy on view classification on both MIMIC-CXR and CheXpert, and 0.694 accuracy on MS-CXR-T40 temporal classification versus 0.387 for QwenVL and 0.419 for GPT-4V.OpenI was held out during training and uses a different labeling scheme, supporting assessment of out-of-distribution generalization.Source: Performance on Image Perception · pp. 4, 7

Performance on Image-Text Reasoning

  • On held-out OpenI, CheXagent scored 0.788, 0.793, and 0.776 accuracy on side, region, and severity reasoning; VQA accuracy was 0.967 on SLAKE and 0.687 on RadRestruct.OpenI was held out from training, and CheXagent was not specifically optimized for fine-grained reasoning.Source: Performance on Image-Text Reasoning · p. 7, Figure 4 · p. 8
  • CheXagent achieved 0.627 mIOU and 0.810 mAP on phrase grounding, outperforming four previously developed approaches, including task-specific models.Figure 4 also illustrates predicted and ground-truth bounding boxes and relates phrase grounding to iterative VQA.Source: Performance on Image-Text Reasoning · pp. 7–8, Figure 4 · p. 8

Performance on Text Generation

  • CheXagent outperformed proprietary models on all four CheXbert-F1 variants for findings generation and scored 0.450 ROUGE-L on findings summarization, competitive with baselines.For findings generation, publicly available baselines were outperformed on MIMIC-CXR and CheXpert.Source: Figure 5 · p. 9, Performance on Text Generation · p. 10

Clinical Evaluation: Reader Study

  • Residents edited CheXagent-drafted reports in 99.9 ± 97.3 seconds versus 156.4 ± 115.9 seconds to draft from scratch; attending editing times were comparable at 79.7 ± 54.6 versus 83.0 ± 36.3 seconds.The reader study involved four residents and four attending radiologists.Source: Clinical Evaluation: Reader Study · p. 10
  • Writing efficiency improved in 81.2% of resident and 61.3% of attending cases with CheXagent drafts; indication-response agreement ratios were 0.788 for residents and 0.738 for attendings.Report content prompted edits in 52.5% of residents’ and 51.3% of attendings’ reports; style prompted edits in 32.5% and 27.5%, respectively.Source: Clinical Evaluation: Reader Study · pp. 10, 12

Discussion

CheXagent extends CXR foundation models beyond report generation, with broad benchmark capabilities and reader-study evidence of efficiency gains. CheXinstruct and CheXbench provide resources for further model development and standardized evaluation.

Discussion

  • CheXagent outperformed baselines, including significantly larger FMs like GPT-4V, across view identification, progression monitoring, classification, reasoning, visual question-answering, and localization.Source: Discussion · p. 12
  • Fine-grained reasoning was out-of-distribution because CheXinstruct lacked explicit examples, yet CheXagent performed strongly; it also outperformed multiple approaches on phrase grounding.Source: Discussion · p. 12
  • With OpenI excluded from training despite its substantially different labeling schema, disease-identification trends closely mirrored MIMIC-CXR and CheXpert, indicating generalization to unseen data.Source: Discussion · p. 13
  • 36% time savings for radiology residents using CheXagent-drafted reports, while maintaining quality.Source: Discussion · p. 13
  • At 3.1 billion parameters, CheXagent is lightweight, but scaling effects remain future work; the reader study also leaves education impacts and ASR-dictated-report comparisons open.Source: Discussion · p. 13
  • CheXinstruct’s 32-source, 8.5-million-sample collection and CheXbench support future FM development and standardized CXR evaluation.Source: Discussion · pp. 12–13

Figure Legends

The legends trace CheXinstruct curation and CheXagent training, then show benchmark evaluations across perception, reasoning, and text generation alongside a clinical reader study.

Figure 1

  • Figure 1 traces CheXinstruct construction from 35 CXR tasks and 32 public datasets through data engineering to compiled training samples.The process includes manual quality control and automated data engineering.Source: Figure Legends · p. 14

Figure 2

  • Figure 2 shows language-model and image-encoder training followed by joint training, and lays out eight evaluation tasks across perception, reasoning, and text generation.The image encoder is trained using paired text before joint training on CheXinstruct triplets.Source: Figure Legends · p. 14

Figure 3

  • Figure 3 covers view, disease, and temporal classification, with OpenI evaluations described as testing generalization because OpenI was unseen during training.Its panels include model comparisons, confusion matrices, and an example temporal-classification prediction.Source: Figure Legends · p. 14

Figure 4

  • Figure 4 covers fine-grained reasoning, visual question answering, and phrase grounding, including examples that connect grounding with iterative question answering.The panels report mean accuracy with 95% confidence intervals for reasoning and VQA, and mIOU and mAP for phrase grounding.Source: Figure Legends · p. 14

Figure 5

  • Figure 5 compares findings generation and findings summarization, using CheXbert-F1, BERTScore, RadGraph-F1, and ROUGE-L as evaluation metrics.The legend distinguishes publicly available and proprietary model comparisons for findings generation.Source: Figure Legends · p. 14

Figure 6

  • Figure 6 depicts a reader study modeled on resident drafting and attending review, measuring report-production time, indication applicability, editing reasons, and perceived efficiency.It also presents time distributions and reader ratings and opinions.Source: Figure Legends · pp. 14–15

Method

The methods curate a large, multi-task CXR dataset, train CheXagent, and assess it with a broad benchmark and a simulated clinical workflow.

Description of the CheXinstruct dataset

  • CheXinstruct was compiled from 32 public datasets to support 35 defined tasks, yielding 8,466,352 instruction-response triplets after quality control and report restructuring.The source collection contained 1,077,494 unique images; dataset splits were retained to prevent leakage.Source: Description of the CheXinstruct dataset · pp. 15–16

Training CheXagent

  • CheXagent combines a medical-text-trained decoder, a CXR-adapted image encoder, and a projector, then trains on image-text pairs and CheXinstruct instruction triplets.The image encoder and language model are frozen while the projector is trained; later instruction training includes periods with the encoder frozen and unfrozen.Source: Training CheXagent · p. 16

Building CheXbench

  • CheXbench assesses image perception, image-text reasoning, and text generation across eight clinically relevant CXR interpretation tasks.The benchmark includes view and temporal classification, disease identification, fine-grained reasoning, VQA, phrase grounding, findings generation, and findings summarization.Source: Building CheXbench · p. 16
  • Perception tests include view and temporal classification and three disease-identification subtasks, evaluated by accuracy on multiple-choice responses.Tasks use class balance to the extent possible; the benchmark includes 600 view-classification samples, 62 temporal-classification samples, and 2,684 disease-identification samples.Source: Building CheXbench · pp. 16–17
  • Reasoning tests use hard-negative fine-grained choices, yes/no VQA, and phrase grounding; text generation evaluates report findings and text-only impression summarization.Reasoning and VQA are scored by accuracy, phrase grounding by bounding-box accuracy, and generated text with standard natural-language metrics.Source: Building CheXbench · pp. 17–18
  • CheXbench compares CheXagent with general, medical, and proprietary models, plus task-specific and adapted language-model baselines for selected tasks.Comparisons use accuracy for classification and reasoning, CheXbert-F1, BERTScore, and RadGraph-F1 for findings generation, and ROUGE-L for summarization.Source: Building CheXbench · p. 18

Setup of the Reader Study for Clinical Evaluation

  • The reader study simulated resident report drafting followed by attending review, comparing reports written from scratch or drafted by CheXagent among four residents and four attendings.Fifty MIMIC-CXR test cases were sampled and 30 cases distributed to each reader; measures included time, indication applicability, editing reasons, and efficiency feedback.Source: Setup of the Reader Study for Clinical… · p. 18

Statistics and reproducibility

  • Analyses used bootstrapped 95% confidence intervals, paired t-tests for model comparisons, and Mann–Whitney U tests for reader-study settings.Readers were blinded to report source and saw cases in random order; publicly accessible datasets, CheXinstruct, model weights, and CheXbench were slated for release before publication.Source: Statistics and reproducibility · p. 19, Data Availability · p. 19

Extended Data

The extended-data legends describe qualitative reader-study cases and comparative technical evaluations; a further table lists CheXagent training hyperparameters.

Extended Data Figure 1

  • Extended Data Figure 1 uses blue for accurate findings, red for false predictions, and green for missed findings across three reader-study cases.The cases illustrate reports judged to improve both interpretation and writing efficiency, writing efficiency alone, or neither; one missed finding was a left pleural effusion.Source: Figure 1 · p. 24

Extended Data Figure 2

  • Extended Data Figure 2 compares CheXagent with BLIP-2, InstructBLIP, MedFlamingo, and XrayGPT on view classification, disease identification, VQA, and fine-grained reasoning.The plots report mean accuracy with 95% confidence intervals; the supplied caption does not state which model performs best.Source: Figure 2 · p. 25

Extended Data Table 1

  • Extended Data Table 1 is identified as a table of CheXagent training hyperparameters.The supplied passage gives no individual hyperparameter values.Source: Table 1 · p. 26
Loading 2401.12208v2…