PaperlayerView exact revision on arXiv

Source-linked AI summary

CheXpert Plus: Augmenting a Large Chest X-ray Dataset with Text Radiology Reports, Patient Demographics and Additional Image Formats

Pierre Chambon, Jean-Benoit Delbrouck, Thomas Sounack, Shih-Cheng Huang, Zhihong Chen, Maya Varma, Steven QH Truong, Chu The Chuong, Curtis P. Langlotz

arXiv:2405.19538v2cs.CLcs.AIcs.CVcs.LG

TL;DR

Radiology vision-language research needs extensive datasets pairing images and text, alongside growing interest in linked reports and patient demographics. CHEXPERT PLUS adds de-identified reports, patient information, and other resources to CheXpert. It contains 36 million text tokens, compared with MIMIC’s 34 million.

  • Problem

    Training increasingly sophisticated radiology models requires extensive datasets containing both images and text, while researchers also seek linked reports and demographic data.

    Source: §1 · p. 2

  • Method

    CHEXPERT PLUS pairs chest X-rays with de-identified, pre-parsed reports and adds patient metadata, pathology labels, and RadGraph annotations.

    Source: §1 · p. 2, §5 · p. 9

  • Results

    36 million text tokens versus MIMIC’s 34 million; CHEXPERT PLUS has greater textual depth, and its 13,351,758 impression tokens exceed MIMIC’s 7,986,317.

    Source: §4.2 · p. 7

  • Takeaways & Limitations

    Patient metadata supports analysis of distribution shifts and efforts to diminish bias, while the released resources serve key radiology machine-learning tasks.

    Source: §5 · p. 9

    Fracture, lung lesion, and pleural other are under-represented, limiting performance for models studying those pathologies.

    Source: §4.6 · p. 9

Abstract

from arXiv · show

Since the release of the original CheXpert paper five years ago, CheXpert has become one of the most widely used and cited clinical AI datasets. The emergence of vision language models has sparked an increase in demands for sharing reports linked to CheXpert images, along with a growing interest among AI fairness researchers in obtaining demographic data. To address this, CheXpert Plus serves as a new collection of radiology data sources, made publicly available to enhance the scaling, performance, robustness, and fairness of models for all subsequent machine learning tasks in the field of radiology. CheXpert Plus is the largest text dataset publicly released in radiology, with a total of 36 million text tokens, including 13 million impression tokens. To the best of our knowledge, it represents the largest text de-identification effort in radiology, with almost 1 million PHI spans anonymized. It is only the second time that a large-scale English paired dataset has been released in radiology, thereby enabling, for the first time, cross-institution training at scale. All reports are paired with high-quality images in DICOM format, along with numerous image and patient metadata covering various clinical and socio-economic groups, as well as many pathology labels and RadGraph annotations. We hope this dataset will boost research for AI models that can further assist radiologists and help improve medical care. Data is available at the following URL: https://stanfordaimi.azurewebsites.net/datasets/5158c524-d3ab-4e02-96e9-6ee9efc110a1 Models are available at the following URL: https://github.com/Stanford-AIMI/chexpert-plus

1 Introduction

CHEXPERT PLUS responds to demand for large image-text resources by adding reports and patient information to CheXpert, with reports de-identified for research use. It also releases models and annotations for downstream machine-learning tasks.

1 Introduction

  • Recent chest X-ray vision-language work spans report generation, self-supervised learning, stable diffusion, and broad reasoning models (Hyland et al., 2023; Chaves et al., 2024; Chambon et al., 2022a; Wu et al., 2023; Chen et al., 2024).The introduction presents extensive image-and-text datasets as important for training increasingly sophisticated models.Source: §1 · p. 2
  • CHEXPERT PLUS adds corresponding radiology reports and patient information to CheXpert and de-identifies reports to remove patient PHI.The paper frames these additions as resources for training increasingly sophisticated models.Source: §1 · p. 2
  • The dataset provides images in DICOM and PNG, 14 automatically extracted pathology labels, and RadGraph annotations for Findings and Impression sections.The released models are trained on these data sources for key machine-learning tasks.Source: §1 · p. 2

2 Related Work

Prior chest X-ray datasets vary in scale, language, metadata, and modality; the section describes PadChest, MIMIC, and several smaller datasets.

2 Related Work

  • PadChest contains 109,931 studies and 168,861 images from a Spanish hospital, with Spanish-language reports and comprehensive metadata.Its images and reports were de-identified, and images failing stated quality or modality criteria were excluded.Source: §2 · p. 2
  • MIMIC pairs 377,110 de-identified DICOM chest radiographs with free-text reports from 227,835 studies at Beth Israel Deaconess Medical Center.Its de-identification process used random patient and study identifiers while preserving chronological integrity.Source: §2 · p. 2
  • Open-I, BIMCV-COVID19, and CANDIDPTX are cited as other, smaller datasets available to researchers.Source: §2 · p. 2

3 Dataset Composition

CHEXPERT PLUS pairs radiology reports with chest X-rays and adds patient demographics, pathology labels, RadGraph annotations, and pretrained models. It provides the data in DICOM and PNG formats, with report and image metadata.

3.1 General Composition

  • CHEXPERT PLUS pairs 223,228 unique radiology-report/chest-X-ray pairs from 187,711 studies and 64,725 patients, comprising 223,228 images, 187,711 reports, eight demographic data points, and 187,711 CheXbert annotations for 14 chest pathologies.Source: §3.1 · pp. 2–3

3.2 Images

  • CHEXPERT PLUS comprises 223,228 unique images available in both DICOM and PNG formats, with up to 47 DICOM metadata elements released for medical image processing and interpretation.Source: §3.2 · p. 3

3.3 Reports

  • 187,711 reports correspond to studies, may cover multiple images, and include a chronological study-order field for each patient.Reports may contain up to 11 sections because radiologists used section names inconsistently; impressions are almost always present.Source: §3.3 · pp. 3–4

3.4 Demographics

  • Patient demographics include age, sex, race, ethnicity, insurance type, BMI, deceased status, and interpreter need.The authors state that these data allow researchers to account for patient subgroups when training models.Source: §3.4 · p. 5

3.5 Pathology Labels

  • Fourteen pathology labels are extracted from reports, with CheXbert labels generated from the full report, individual sections, or section concatenations.Inputs exceeding CheXbert's 512-token limit are truncated.Source: §3.5 · p. 5

3.6 RadGraph Annotations

  • RadGraph annotations cover Findings and Impression; Anatomy and definitely-present observations are common, and Modify relations dominate other relation types.Entity counts are 1,581,863 in Findings and 3,999,559 in Impression; relation counts are 1,106,659 and 2,772,337, respectively.Source: §3.6 · pp. 5–6

3.5 Pathology Labels

  • 0.93 F1-score for CheXbert-impression labels on the text-based test set contrasts with 0.44 for CheXbert-findings labels on the image-based test set.The authors caution that report-derived labels might not reliably train image pathology classifiers.Source: §3.5 · p. 5, Table 4 · p. 6

3.7 Model Zoo

  • Released pretrained models include LLaMA, CLIP, VQ-GAN, DINOv2, and architecture-agnostic models for report generation and summarization.The collection spans language processing, image recognition, and generative modeling.Source: §3.1 · p. 3, §3.7 · p. 6

4 Dataset Release and Analysis

CHEXPERT PLUS expands CheXpert with richer image, report, demographic, and annotation resources, while documenting its scale, de-identification procedures, uses, and dataset limits. Its impression text volume exceeds MIMIC’s, but some studies and pathologies remain underrepresented.

4.1 Comparison with CheXpert 1.0

  • CHEXPERT PLUS adds DICOM images and parsed reports, patient demographics, CheXbert labels, and RadGraph annotations to the original CheXpert resource.186 studies from the original release are missing because their reports could not be recovered for CHEXPERT PLUS.Source: §4.1 · p. 7

4.2 Head-to-head Comparison with other Radiology Datasets

  • 36 million total text tokens versus MIMIC’s 34 million, and 13,351,758 impression tokens versus 7,986,317, make CHEXPERT PLUS largest for impression-only tasks despite fewer unique reports.MIMIC has 227,821 unique reports versus CHEXPERT PLUS’s 187,711; CHEXPERT PLUS reports retain structure, including de-identified accession sections, unlike MIMIC.Source: §4.2 · p. 7

4.3 De-identification and release of DICOMs

  • DICOM headers were automatically de-identified and human-reviewed, while released pixels remain identical to the PHI-cleared CheXpert 1.0 images.Source: §4.3 · pp. 7–8

4.4 De-identification and release of reports

  • 853,878 true PHI spans were reviewed during a year-long, four-step report de-identification effort supported by 25–30 annotators.The process used automatic replacement, human review and correction of missed spans, and radiologist review of the true-to-synthetic PHI mapping; 23 spans were fully missed (0.002%) and 841 partially missed (0.01%).Source: §4.4 · pp. 8–9

4.5 Applicability, Usage and Future Work

  • CHEXPERT PLUS nearly doubles publicly available English radiology image-text pairs, supports cross-institution testing and training, and adds demographics for fairness analyses.Models trained on it may still reflect biases, so researchers should audit distribution shifts and performance disparities across patient attributes.Source: §4.5 · p. 9

4.6 Limitations of the dataset

  • The dataset emphasizes detailed impressions over findings, and underrepresents fracture, lung lesion, and pleural-other pathologies, limiting model performance studies of those conditions.Source: §4.6 · p. 9

5 Conclusion

CHEXPERT PLUS combines paired images and reports with patient demographics, pathology labels, and RadGraph annotations. It also releases image formats, parsed reports, and pretrained models for radiology tasks.

  • CHEXPERT PLUS is a multi-sourced dataset pairing hundreds of thousands of images with texts, patient demographics, pathology labels, and RadGraph annotations.The release includes DICOM and PNG images, de-identified parsed reports, patient metadata, improved pathology labels, RadGraph annotations for findings and impressions, and models spanning text-to-image generation to text-to-text summarization.Source: §5 · p. 9

A From CHEXPERT PLUS DICOM to CheXpert 1.0 JPG

The appendix describes converting CHEXPERT PLUS DICOM files to CheXpert 1.0 JPG images. The code reads and rescales image pixels, applies image adjustments, and saves the result.

B DICOM metadata

The appendix includes a table titled as a combination of metadata contained in CHEXPERT PLUS DICOM files.

Loading 2405.19538v2…