PaperlayerView exact revision on arXiv

Source-linked AI summary

Chest X-ray Foundation Model with Global and Local Representations Integration

Zefan Yang, Xuanang Xu, Jiajin Zhang, Ge Wang, Mannudeep K. Kalra, Pingkun Yan

arXiv:2502.05142v2eess.IVcs.CV

TL;DR

Task-specific CXR models can require costly labels, remain narrow in scope, and generalize poorly to out-of-distribution data. CheXFound uses large-scale self-supervised pretraining and a downstream head integrating local and global features; the authors report improved disease classification, label efficiency, and performance across varied tasks.

  • Problem

    Task-specific CXR classifiers are limited in scope, require costly labeled datasets, and struggle to generalize to out-of-distribution data.

    Source: §I · p. 1

  • Method

    CheXFound is pretrained on a large CXR dataset using self-supervision, and its GLoRI head combines disease-specific local features with global image features for multilabel classification.

    Source: §I · p. 2, B. Global and Local Representations… · p. 3

  • Results

    CheXFound outperformed prior state-of-the-art models on 40 CXR-LT 24 disease findings and showed strong results in label-limited and out-of-distribution tasks.

    Source: §I · p. 2

  • Takeaways & Limitations

    The reported results support CheXFound's generalization across varied downstream CXR tasks and its use in adaptations with improved label efficiency.

    Source: §I · p. 2, C. Conclusion · p. 12

Abstract

from arXiv · show

Chest X-ray (CXR) is the most frequently ordered imaging test, supporting diverse clinical tasks from thoracic disease detection to postoperative monitoring. However, task-specific classification models are limited in scope, require costly labeled data, and lack generalizability to out-of-distribution datasets. To address these challenges, we introduce CheXFound, a self-supervised vision foundation model that learns robust CXR representations and generalizes effectively across a wide range of downstream tasks. We pretrain CheXFound on a curated CXR-1M dataset, comprising over one million unique CXRs from publicly available sources. We propose a Global and Local Representations Integration (GLoRI) module for downstream adaptations, by incorporating disease-specific local features with global image features for enhanced performance in multilabel classification. Our experimental results show that CheXFound outperforms state-of-the-art models in classifying 40 disease findings across different prevalence levels on the CXR-LT 24 dataset and exhibits superior label efficiency on downstream tasks with limited training data. Additionally, CheXFound achieved significant improvements on new tasks with out-of-distribution datasets, including opportunistic cardiovascular disease risk estimation and mortality prediction. These results highlight CheXFound's strong generalization capabilities, enabling diverse adaptations with improved label efficiency. The project source code is publicly available at https://github.com/RPIDIAL/CheXFound.

I. INTRODUCTION

CheXFound is presented as a CXR foundation model intended to address limited task scope, costly labeling, and poor out-of-distribution generalization. The paper reports gains across disease classification and other downstream tasks, using large-scale self-supervised pretraining and a head that combines local and global features.

I. INTRODUCTION

  • Task-specific CXR classifiers are often narrow, require costly labeled datasets, and struggle to generalize to out-of-distribution data.The introduction identifies these constraints as motivation for self-supervised models that learn representations usable across diverse tasks.Source: §I · p. 1
  • Earlier CXR foundation models were limited by model capacity or pretraining-data scale and often relied on global features rather than disease-specific patch features.The paper frames scale and local-feature use as limitations relevant to long-tail classification and clinical tasks spanning many findings.Source: §I · p. 1
  • CheXFound was pretrained on approximately 987K CXRs, and its GLoRI head combines fine- and coarse-grained disease-specific local features with global image features.GLoRI uses attention to compute local features and integrates them with the global image representation for multilabel classification.Source: §I · p. 2
  • CheXFound outperformed prior state-of-the-art models on 40 disease findings in CXR-LT 24 and showed strong results across label-limited and out-of-distribution tasks.The evaluated tasks also included opportunistic risk estimation, tube detection, and anatomical segmentation.Source: §I · p. 2

A. Self-supervised Visual Representation Learning

The section situates CheXFound among self-supervised representation-learning approaches and CXR foundation models, then describes its large-scale CXR pretraining setup. It emphasizes self-distillation, data scale, and two complementary training objectives.

A. Self-supervised Visual Representation Learning

  • Self-supervised visual learning includes masked-image reconstruction, contrastive objectives, and self-distillation; CheXFound follows the self-distillation approach.The cited literature describes self-distillation methods as strong on linear probing and generalizable in medical applications.Source: A. Self-supervised Visual… · p. 2

B. Foundation Models for Medical Applications

  • Earlier supervised CXR approaches either trained from scratch or pretrained and fine-tuned models, while contrastive CXR methods adapted that two-stage paradigm.The cited examples include CheXNet, CheXpert, CheXtransfer, MedAug, and MoCo-CXR.Source: B. Foundation Models for Medical… · p. 2
  • CheXFound scales vision-centric CXR pretraining to a ViT-Large model and 987K CXRs from 12 sources, while prior models commonly used up to 88M parameters and 838K images from five datasets.Compared with RayDINO, the dataset adds eight sources and 124K images; the paper also identifies multimodal vision-language models as a separate foundation-model category.Source: B. Foundation Models for Medical… · pp. 2–3

A. Pretraining CheXFound with Large-scale Dataset

  • CXR-987K contains 987,733 unique CXRs from 12 public datasets, including frontal and lateral views; CXR-207K and CXR-744K are subsets for data-scaling evaluation.CXR-207K draws from MIMIC-CXR, while CXR-744K draws from five datasets.Source: A. Pretraining CheXFound with… · p. 3
  • CheXFound uses DINOv2 teacher–student self-distillation with masked image modeling and [CLS] token alignment objectives.Masked image modeling trains reconstruction of masked patch tokens from teacher-generated tokens; [CLS] alignment encourages similarity between teacher and student class tokens.Source: A. Pretraining CheXFound with… · p. 3

B. Global and Local Representations Integration for Multilabel Classification

GLoRI addresses the limited local detail of a linear probe based on the global [CLS] token by combining disease-specific local features with global image features. Its fine-grained branch uses adaptive temperatures to tune attention for each disease query.

B. Global and Local Representations Integration for Multilabel Classification

  • A linear probe using one global [CLS] token can lack local detail for multilabel CXR classification, while patch tokens can provide disease-specific features.GLoRI combines these local features with the global representation through a skip connection.Source: B. Global and Local Representations… · p. 3

1) Encoding Fine-grained Local Features with Adaptive Temperatures:

  • GLoRI projects frozen CheXFound patch tokens into its feature space and uses disease queries with patch-derived keys and values to compute local features.Its adaptive temperature module generates one temperature per disease query and adjusts attention to focus on small or diffuse abnormalities.Source: 1) Encoding Fine-grained Local… · p. 3

2) Encoding Coarse-grained Local Features with Pyramid Patch Merging:

GLoRI builds coarse-grained local features by merging patch tokens at multiple scales, then combines its local features with the global [CLS] representation. The two local-feature branches distinguish GLoRI from related attention-pooling approaches.

2) Encoding Coarse-grained Local Features with Pyramid Patch Merging:

  • The pyramid patch merging module average-pools adjacent patch tokens at 8×8, 4×4, and 2×2 scales to form multiscale representations for coarse-grained local features.The resulting embeddings are projected, upsampled, concatenated, and normalized before attention encodes the coarse-grained features.Source: 2) Encoding Coarse-grained Local… · p. 4
  • GLoRI combines fine- and coarse-grained local features with the [CLS] token, then classifies each output token using a linear layer and binary cross-entropy loss.Source: 3) Integrating Global Image Features · p. 4

3) Integrating Global Image Features:

  • Unlike related attention-pooling approaches, GLoRI uses two local-feature branches to capture both fine- and coarse-grained features.The paper reports that these features are empirically relevant to disease abnormalities in CXRs.Source: 3) Integrating Global Image Features · p. 4

IV. EXPERIMENTAL DESIGN

CheXFound was pretrained on a curated, multi-source CXR dataset and evaluated across in- and out-of-distribution datasets, with test data kept out of pretraining. The design combines varied downstream tasks with metrics suited to class-imbalanced multilabel classification.

1) Self-supervised pretraining details:

  • CheXFound was self-supervised pretrained on the curated CXR-987K dataset using a ViT-L architecture with 16×16-pixel patches, varying global/local crop sizes across four specified resolution pairs.Training used 100 epochs, two global and eight local crops, and masked image modeling; crop-size pairs were (512, 144), (448, 128), (336, 128), and (224, 96).Source: 1) Self-supervised pretraining details · p. 4

A. Implementation details

  • Downstream evaluation trained GLoRI on frozen CheXFound features, using the concatenated representations from its last four layers.The GLoRI head used multihead cross-attention, and was trained for 10 epochs on CXR-LT 24, CheXpert, and PLCO and 100 epochs on Shenzhen, Montgomery, and JSRT.Source: 2) Downstream evaluation details · p. 5, Figure 2 · p. 5

B. Experimental datasets

  • The benchmark covered in-distribution CXR-LT 24 and CheXpert and out-of-distribution datasets spanning disease detection, tube malposition, and opportunistic risk estimation.The out-of-distribution set included VinDr-CXR, CXR-Pneumonia, Shenzhen, Montgomery, JSRT, RANZCR-CLiP, and PLCO.Source: B. Experimental datasets · pp. 5–6
  • To avoid data contamination, test images were excluded from pretraining, and none of the out-of-distribution dataset images were used for pretraining.CXR-LT 24 included 40 disease findings; performance was assessed with average AUROC and AUPRC and 95% confidence intervals from 1,000 bootstrap samples.Source: B. Experimental datasets · pp. 5–6

V. EXPERIMENTAL RESULTS

CheXFound performed strongly across in- and out-of-distribution classification benchmarks, and GLoRI generally improved on linear probes. Its results also held across disease-prevalence levels and against end-to-end and vision-language comparators.

A. Overall performance comparison

  • CheXFound outperformed other foundation models across the in- and out-of-distribution classification tasks using a linear probe, including 9.5% over the next best model in CXR-LT 24 AUPRC.It also exceeded the next best model by 3.5% in CheXpert AUROC and 25.1% in VinDr-CXR AUROC.Source: A. Overall performance comparison · p. 6, Table III · p. 7, Table IV · p. 7
  • CheXFound with GLoRI improved AUROC over its linear-probe baseline by 4.1%, 3.2%, 3.4%, 1.1%, 1.0%, and 11.9% on six datasets.The six datasets were CXR-LT 24, CheXpert, VinDr-CXR, Shenzhen, Montgomery, and JSRT; on CXR-Pneumonia, its AUROC was 0.993, on par with its linear-probe baseline.Source: A. Overall performance comparison · p. 7
  • CheXFound also outperformed the end-to-end trained ConvNeXt model, while CheXFound with GLoRI outperformed CLIP-based models in zero-shot classification.The comparisons include end-to-end and vision-language foundation models.Source: A. Overall performance comparison · pp. 7–8, Table V · p. 7

B. Malpositioned tube detection

CheXFound performed well on malpositioned-tube detection across three classes despite their unequal prevalence. The authors suggest high-resolution image encoding and representation of nuanced structures as possible explanations for the gains.

B. Malpositioned tube detection

  • CheXFound achieved AUPRCs of 0.857, 0.529, and 0.445 for normal, borderline, and abnormal tube classes, respectively.It outperformed the next best models by 16.4%, 16.1%, and 29.2% for those classes, respectively; class proportions were 59.4%, 28.8%, and 11.8%.Source: B. Malpositioned tube detection · p. 8, Table VI · p. 8
  • The authors suggest that high-resolution CXR encoding and effective representations of nuanced structures could account for the performance gains.The dataset's class distribution was unbalanced.Source: B. Malpositioned tube detection · p. 8

C. Opportunistic predictive power

On PLCO, CheXFound outperformed comparators in cardiovascular-risk and all-cause-mortality estimation, with predicted mortality-risk groups showing different survival distributions. For segmentation, it performed particularly well on rib masks, while lung segmentation results were close to the other methods.

C. Opportunistic predictive power

  • CheXFound achieved AUROCs of 0.749 for cardiovascular disease risk and 0.786 for all-cause mortality estimation, exceeding ConvNeXt by 3.5% and 4.1%, respectively.The low- and high-risk groups had different survival distributions (p < 0.001), with endpoint survival probabilities of 78.4% and 38.4%.Source: C. Opportunistic predictive power · p. 8, Table VII · p. 9, Figure 5 · p. 9

D. CXR foundation models for semantic segmentation

  • On VinDr-RibCXR, CheXFound achieved a Dice score of 0.793, exceeding ConvNeXt by 7.1% for rib segmentation.On Montgomery lung segmentation, its Dice score was 0.980, 0.2% above RAD-DINO; the authors describe the task as relatively simple, with all methods performing satisfactorily.Source: D. CXR foundation models for semantic… · pp. 8–9

E. Benchmarking self-supervised pretraining methods

On CXR-987K, CheXFound outperformed other self-supervised pretraining methods on CXR-LT 24. Its performance also improved with larger pretraining datasets and ViT-L rather than ViT-B backbones.

E. Benchmarking self-supervised pretraining methods

F. Scalability of self-supervised vision encoders

G. Impact of CXR resolution

Higher-resolution pretraining improved downstream performance, while low-resolution initialization and increased masked-image-modeling loss weight improved pretraining results. Together, the latter modifications yielded a 4.7% gain over the original DINOv2 settings.

G. Impact of CXR resolution

  • Pretraining from scratch at higher resolution did not yield meaningful downstream representations, so 224^2-pretrained weights initialized subsequent higher-resolution training.The tested resolutions were 224^2, 336^2, 448^2, and 512^2.Source: G. Impact of CXR resolution · p. 10
  • Raising pretraining resolution from 224^2 to 512^2 improved AUROC by 3.3% on CXR-LT 24, 3.5% on CheXpert, 3.7% on Shenzhen, 10.7% on Montgomery, and 35.8% on JSRT (all p < 0.001).The resolution evaluation covered these five datasets.Source: G. Impact of CXR resolution · p. 10

H. Ablation studies on pretraining

  • Increasing the MIM loss weight from 1 to 3 raised AUROC from 0.822 to 0.840.The authors motivate this change by diagnostic cues residing in nuanced local features of medical images.Source: H. Ablation studies on pretraining · p. 10
  • Low-resolution initialization followed by high-resolution pretraining and increased MIM loss weight produced a 4.7% gain over original DINOv2 settings.The original settings achieved AUROC 0.793; the modified settings used a low-resolution warm-up and MIM loss weight of 3.Source: H. Ablation studies on pretraining · p. 10

I. Ablation studies on GLoRI head

GLoRI combines disease-specific local and global image features for multilabel classification, improving CXR-LT 24 performance over an attention-pooling baseline and linear probing. Its attention maps localize findings but may cover only part of an abnormality.

I. Ablation studies on GLoRI head

  • GLoRI integrates global image features, adaptive temperatures, and pyramid patch merging into multi-head attention, alongside fine- and coarse-grained local features.The single-layer attention-pooling baseline achieved AUROC 0.826; adding the three components incrementally improved performance.Source: §I · pp. 10–11
  • GLoRI achieved AUROC 0.84, improving on an attention-pooling baseline by 1.4% and outperforming linear probing by 4.1% on CXR-LT 24.The ablation table evaluates architecture variants on CXR-LT 24; the reported comparisons are statistically significant (p < 0.001).Source: §I · pp. 10–11, Table XI · p. 11

A. Interpretation of disease-specific local features

  • Attention maps for five pathologies localized abnormalities, but some maps covered only partial regions, including parts of the heart or inferior lung boundaries.For edema, individual maps scattered across the lungs while the global map covered extensive regions of both lungs.Source: A. Interpretation of disease-specific… · p. 11

B. Generalizability of foundation models

CheXFound generalized across in-distribution and out-of-distribution CXR datasets and showed gains on low-prevalence findings. The conclusion also reports label-efficient adaptation to several downstream tasks, while noting that the larger ViT-giant model was not evaluated.

B. Generalizability of foundation models

  • CheXFound performed better on in-distribution CXR-LT 24 and CheXpert and out-of-distribution Shenzhen, Montgomery, and JSRT; it also improved results on PLCO opportunistic interpretation tasks.The authors attribute generalizability to representations learned from large-scale, diverse CXRs.Source: B. Generalizability of foundation models · p. 11
  • CheXFound achieved significant gains over comparison methods on pathologies occurring at less than 1% frequency.The authors describe this as superior label efficiency for low-prevalence findings.Source: B. Generalizability of foundation models · p. 11
  • The study did not evaluate DINOv2's 1.1B-parameter ViT-giant; the authors expect stronger generalization but note its greater data and computational demands.This is an author-stated scope boundary, not a measured comparison.Source: B. Generalizability of foundation models · p. 11

C. Conclusion

  • The conclusion reports label-efficient adaptation to opportunistic CXR interpretation, malpositioned-tube detection, and anatomical structure segmentation.Disease-specific local features were characterized as interpretable through attention maps.Source: C. Conclusion · pp. 11–12

Related papers

Loading 2502.05142v2…