PaperlayerView exact revision on arXiv

Source-linked AI summary

Vision Transformer using Low-level Chest X-ray Feature Corpus for COVID-19 Diagnosis and Severity Quantification

Sangjoon Park, Gwanghyun Kim, Yujin Oh, Joon Beom Seo, Sang Min Lee, Jin Hwan Kim, Sungjun Moon, Jae-Kwang Lim, Jong Chul Ye

arXiv:2104.07235v1eess.IVcs.CVcs.LG

TL;DR

COVID-19 CXR diagnosis and severity quantification are challenging because large labeled COVID-19 datasets are difficult to collect, while existing ViT embeddings are not tailored to CXR. The paper uses a backbone trained on common CXR findings to provide features for a Transformer, reporting strong external generalization and performance across diagnosis and severity quantification.

  • Problem

    Large labeled COVID-19 datasets are difficult to collect, and standard ViT pixel-patch or ResNet embeddings may not be optimal for CXR diagnosis.

    Source: §1 · pp. 1–2

  • Method

    A backbone trained on large public CXR datasets extracts common-finding features that serve as the corpus for a Transformer used for COVID-19 diagnosis and severity quantification.

    Source: §1 · p. 3, §3.2 · p. 4, §3.3 · p. 6

  • Results

    The model reportedly outperforms CNN- and Transformer-based models, generalizes across external institutions, and achieves severity-quantification performance similar to clinical experts.

    Source: §6 · p. 12

  • Takeaways & Limitations

    The framework extends COVID-19 CXR analysis beyond diagnosis to severity scoring and localization, with the authors describing potential support for clinical treatment decisions.

    Source: §1 · p. 3, §6 · p. 12

    Severity training labels came from three institutional datasets, and differences in Brixia labeling details and anatomy between races may constrain the training data.

    Source: §5.5.7 · p. 11

Abstract

from arXiv · show

Developing a robust algorithm to diagnose and quantify the severity of COVID-19 using Chest X-ray (CXR) requires a large number of well-curated COVID-19 datasets, which is difficult to collect under the global COVID-19 pandemic. On the other hand, CXR data with other findings are abundant. This situation is ideally suited for the Vision Transformer (ViT) architecture, where a lot of unlabeled data can be used through structural modeling by the self-attention mechanism. However, the use of existing ViT is not optimal, since feature embedding through direct patch flattening or ResNet backbone in the standard ViT is not intended for CXR. To address this problem, here we propose a novel Vision Transformer that utilizes low-level CXR feature corpus obtained from a backbone network that extracts common CXR findings. Specifically, the backbone network is first trained with large public datasets to detect common abnormal findings such as consolidation, opacity, edema, etc. Then, the embedded features from the backbone network are used as corpora for a Transformer model for the diagnosis and the severity quantification of COVID-19. We evaluate our model on various external test datasets from totally different institutions to evaluate the generalization capability. The experimental results confirm that our model can achieve the state-of-the-art performance in both diagnosis and severity quantification tasks with superior generalization capability, which are sine qua non of widespread deployment.

1. Introduction

COVID-19 CXR screening and follow-up could be useful, but scarce labeled cases and weak generalization constrain existing approaches. The paper proposes a ViT using common CXR finding features for diagnosis and severity quantification, and reports stronger performance and generalization than CNN- and Transformer-based models.

1. Introduction

  • CXR offers practical screening and follow-up advantages, while RT-PCR results may take hours or days and CT carries cost and cross-contamination concerns.The introduction also notes that most patients with confirmed COVID-19 present positive radiological findings.Source: §1 · pp. 1–2
  • Existing CXR deep-learning studies face poor generalization on unseen data, while large labeled COVID-19 datasets are difficult to construct and proposed mitigations remain suboptimal.The paper also identifies CNN locality as a potential mismatch for disease classification requiring global patterns, multiplicity, and distribution.Source: §1 · p. 2
  • The proposed ViT uses a CNN feature extractor trained on large CXR datasets to provide a corpus of common findings, which the Transformer uses for COVID-19 diagnosis.This design addresses the concern that standard pixel-patch or ResNet embeddings may not be optimal for CXR diagnosis.Source: §1 · pp. 2–3
  • The framework also quantifies COVID-19 involvement from weak array-based labels, producing a global score from 0 to 6 and a lesion-likelihood severity map.The authors describe this as supporting serial follow-up and clinicians’ treatment decisions.Source: §1 · p. 3
  • The method reportedly outperforms CNN- and Transformer-based models, with strong generalization across external institutions and severity-quantification performance similar to clinical experts.The authors present generalization across external settings as important for wider adoption.Source: §1 · pp. 2–3

2. Related work

Related work describes ViT’s use of self-attention for long-range image relationships, PCAM pooling for CXR feature localization and classification, and simplified severity labels that avoid pixel-level annotation.

2.1. Vision Transformer

  • ViT uses self-attention to model long-range dependencies, while hybrid CNN-backbone designs let the Transformer focus on global attention with relatively less computation.The section also notes ViT variants across vision tasks.Source: §2.1 · p. 3

2.2. Probabilistic Class Activation Map Pooling

  • PCAM pooling converts class activation probabilities into normalized attention weights for weighted feature maps, leveraging localization during training for CXR classification.The related work reports that this operation can improve localization and diagnostic performance and achieved first place in the 2019 CheXpert Challenge.Source: §2.2 · pp. 3–4

2.3. COVID-19 Severity Quantification

  • Severity-labeling approaches include regional scores and array-based labels; Toussie et al. (2020) assign each of six lung regions a binary involvement label, summed to a 0–6 score.The paper adopts this array-based annotation approach for COVID-19 severity quantification.Source: §2.3 · p. 4

3. The Proposed Framework

The framework uses a backbone trained on large public CXR datasets to produce low-level feature representations for a Transformer. It applies this design to COVID-19 classification and severity quantification.

3.1. Pre-training Backbone Network for Low-level Feature Corpus

  • The backbone is pretrained to classify 10 common CXR findings, and its common embedding before PCAM provides useful Transformer input, with PCAM important for optimal intermediate features.The backbone is a modified network proposed by Ye et al. (2020).Source: §3.1 · p. 4

3.2. Vision Transformer for Classification

  • For classification, the Transformer encodes low-level CXR feature vectors, and a linear classifier uses the attended [class] token representation to predict the diagnosis.The feature vectors are embedded representations of low-level features at encoded blocks.Source: §3.2 · pp. 4–5
  • For classification interpretability, the model uses Transformer relevance propagation based on local relevance computed through deep Taylor decomposition.The relevance is propagated through the layers to address self-attention operations and skip connections.Source: §3.2 · p. 5

3.3. Vision Transformer for Severity Quantification

  • For severity quantification, non-[class] Transformer outputs feed a map head whose masked output is ROI-max-pooled over six lung zones and trained against weak severity labels.The zones divide each lung into upper, middle, and lower regions.Source: §3.3 · pp. 5–6
  • The lung segmentation mask comes from a shared generator that uses AdaIN codes to segment both abnormal and normal lung areas.The cited approach is introduced by Oh and Ye (2021).Source: §3.3 · p. 6

4. Dataset

The study uses separate datasets for backbone pre-training, classification, and severity quantification, then evaluates across external institutional datasets. The implementation combines CXR feature maps with a Transformer and evaluates classification and severity using distinct metrics.

4. Dataset

  • The study separates data into backbone pre-training, classification, and severity-quantification datasets.This organization defines the data pipeline for the experiments.Source: §4 · p. 6

4.1. Dataset for Pre-training

  • CheXpert provides 224,316 CXRs with 10 finding labels for backbone pre-training, after excluding lateral views and retaining PA and AP images.The retained data comprise 29,420 PA and 161,427 AP images.Source: §4.1 · p. 6

4.2. Dataset for Classification

  • Classification combines public and four-hospital CXR datasets into normal, other-infection, and COVID-19 classes, with institutional hospitals held out for external testing.For PA images, CNUH, YNU, and KNUH are external test datasets; for AP images, CNUH is the sole external test dataset because it is the only dataset with all three classes.Source: §4.2 · p. 6

4.3. Dataset for Severity Quantification

  • Severity data combine PA and AP views, with two radiologists applying array-based severity labels to three institutional datasets.The study also adds Brixia data after translating its scores and uses 12 BIMCV cases to compare severity maps with clinical expert annotations.Source: §4.3 · p. 6, Figure 6 · p. 7, Table 3 · p. 7

4.4. Details of Implementation and Evaluation

  • A 16 × 16 × 1024 backbone feature map feeds a 12-layer, 12-head Transformer; classification and severity models use separate training setups and evaluation metrics.Classification is evaluated with AUC, sensitivity, specificity, and accuracy; severity uses MSE as the main metric, alongside MAE, correlation coefficient, and R2 score.Source: §4.4 · p. 7

5.1. Diagnostic Performance on External Test Datasets

Across three external institutions, the model maintains strong diagnostic performance on PA-view images, while AP-view performance is lower but remains fair on the external test dataset.

5.1. Diagnostic Performance on External Test Datasets

  • PA-view mean AUCs were 0.932, 0.947, and 0.928 across three external institutions, with sensitivities of 83.4%, 88.4%, and 85.4%.Specificities were 85.3%, 84.2%, and 86.8%, and accuracies were 83.8%, 84.9%, and 86.9%, respectively; the authors report stable performance with AUC ≥0.900.Source: §5.1 · p. 7, Table 4 · p. 8
  • AP-view mean AUCs were 0.890, 0.880, and 0.828 for the three label classes, remaining fair (AUC ≥0.800) on the external test dataset.The authors note that diagnosis from AP-view images is not standard and usually deteriorates diagnostic performance.Source: Table 5 · p. 8, §5.1 · p. 8

5.2. Model Interpretability Results

Saliency maps localized focal infected areas in bacterial infection and tuberculosis examples, and multifocal peripheral lower-lung lesions in a COVID-19 example.

5.2. Model Interpretability Results

  • The external-test examples localize focal bacterial-infection and tuberculosis areas, and multifocal peripheral lesions in both lower lungs for COVID-19.The authors identify the bilateral lower-lung peripheral pattern as typical of COVID-19 pneumonia.Source: §5.2 · p. 8

5.3. Severity Quantification Results on External Test Datasets

Severity estimates on external institutional datasets were reported as comparable to experienced radiologists, and the model produced maps that localized involved areas.

5.3. Severity Quantification Results on External Test Datasets

  • MSEs of 1.657, 1.696, and 1.676 across three external institutions were reported as comparable to experienced radiologists’ ratings.For the Brixia consensus subset, the average MSE between majority-vote severity scores and individual radiologist ratings was 1.683.Source: §5.3 · p. 8
  • The model generated severity maps that highlight affected areas and can be used to contour lesions.Examples include predicted scores, arrays, maps, and lesion contours; BIMCV comparisons show the model generally localized involved areas.Source: §5.3 · p. 8

5.4. Comparison with CNN and Transformer-based Models

The model is compared with CNN- and Transformer-based alternatives on external datasets, with examples also showing severity quantification and localization. It outperformed the tested alternatives across external datasets for both classification and severity quantification.

5.4. Comparison with CNN and Transformer-based Models

  • Fig. 7 compares localization results in BIMCV, marking radiologist annotations in green and thresholded model predictions in yellow.The caption describes the comparison but does not state its outcome.Source: Figure 7 · p. 9
  • The proposed model outperformed CNN- and Transformer-based models across external test datasets for both classification and severity quantification.The comparison included ResNet-50, ResNet-152, DenseNet-121, standard ViT, and hybrid ViT; other models used ImageNet-pretrained weights, with other settings held constant.Source: §5.4 · pp. 8–9
  • Table 6 reports severity quantification performance on external test datasets from different institutions.Fig. 6 shows example severity quantification results on the CNUH external dataset.Source: Table 6 · p. 9, Figure 6 · p. 9

5.5. Ablation Studies

The ablations test Transformer size, feature-map choices, backbone training, input resolution, self-supervised pre-training, pooling, and severity-training data. The results favor a 12-layer, 12-head Transformer, pre-PCAM features, a trainable backbone, lower input resolution, ROI max-pooling, and inclusion of institutional data.

5.5.1. Network Size

  • The 12-layer, 12-head Transformer performed best among the tested configurations of 2 layers and 4 heads, 4 layers and 8 heads, and 12 layers and 12 heads.Source: §5.5.1 · p. 9

5.5.2. Intermediate Feature Map

  • The common feature map before PCAM operation performed best among the tested intermediate feature-map options.The alternatives included weighted maps after PCAM using all findings, infectious-disease-related findings, or pneumonia alone.Source: §5.5.2 · pp. 9–10, Table 9 · p. 11

5.5.3. Freezing Backbone Weights

  • Training the backbone weights performed better than freezing them on all three external test datasets.The authors suggest that a trainable backbone may increase capacity and counter overfitting.Source: §5.5.3 · p. 10

5.5.4. Image Size

  • Increasing input CXR size from 512 × 512 to 1024 × 1024 deteriorated performance, causing severe overfitting and poor generalization.The intermediate feature map remained 16 × 16 × 768 by increasing the patch-embedding stride twofold.Source: §5.5.4 · p. 10, Table 9 · p. 11

5.5.5. Self-supervised Contrastive Pre-training

  • Self-supervised contrastive pre-training offered no prominent benefit for the proposed model or hybrid ViT, while slightly improving standard ViT.The proposed model still outperformed the other Transformer models using self-supervised pre-training.Source: §5.5.5 · p. 10, Table 9 · p. 11

5.5.6. ROI Pooling Type

  • ROI max-pooling outperformed ROI average-pooling when converting the severity map into the array.The authors link max-pooling to the labeling rule, which assigns a positive value to a subdivision when abnormality is present rather than averaging opacity degree.Source: §5.5.6 · p. 10, Table 10 · p. 11

5.5.7. Training Data for Severity Quantification

  • Excluding all institutional datasets degraded severity quantification performance, indicating that at least one institutional dataset was needed for stable performance.The authors suggest differences in severity-labeling details, including horizontal anatomical landmarks, and anatomical differences between populations as possible reasons.Source: §5.5.7 · pp. 10–11, Table 10 · p. 11

6. Discussion and Conclusion

The paper presents a two-stage ViT that uses a backbone trained on a large CXR dataset to produce low-level feature corpora for Transformer-based COVID-19 diagnosis and severity quantification. It reports better external performance than CNN and Transformer comparators, expert-like severity performance, and potential clinical and emerging-pathogen applications.

6. Discussion and Conclusion

  • The framework pre-trains a backbone on a large dataset to embed low-level CXR findings, then uses those features as Transformer input for high-level disease diagnosis.The authors state that the pre-training dataset contains more than 220,000 CXR images.Source: §6 · p. 11
  • The model outperformed other CNN- and Transformer-based state-of-the-art models across external institutions and showed severity-quantification performance similar to clinical experts.Source: §6 · p. 12
  • The authors describe possible uses in patient triage, treatment guidance, and evaluation of response using consecutive CXRs.They note potential value in settings with limited experienced radiologists or unavailable higher-sensitivity examinations.Source: §6 · p. 12
  • The authors propose that aggregating low-level features available in pre-built datasets could help develop algorithms for emerging pathogens that share common CXR features with existing diseases.Source: §6 · p. 12

Related papers

Loading 2104.07235v1…