PaperlayerView exact revision on arXiv

Source-linked AI summary

JCS: An Explainable COVID-19 Diagnosis System by Joint Classification and Segmentation

Yu-Huan Wu, Shang-Hua Gao, Jie Mei, Jun Xu, Deng-Ping Fan, Rong-Guo Zhang, Ming-Ming Cheng

arXiv:2004.07054v3eess.IVcs.CVcs.LG

TL;DR

Existing COVID-19 CT datasets were too small for training strong CNN models, while RT-PCR sensitivity and limited medical staffing motivated complementary automated diagnosis. The paper constructs the COVID-CS dataset and develops an explainable JCS system, achieving strong classification and segmentation results.

  • Problem

    Existing COVID-19 CT datasets contain at most hundreds of CT images, while RT-PCR sensitivity is insufficient and experienced medical staff are scarce.

  • Method

    The paper constructs the COVID-CS dataset with patient-level and pixel-level annotations and integrates explainable classification with fine-grained lesion segmentation in a JCS diagnosis system.

  • Results

    95.0% sensitivity and 93.0% specificity were achieved for COVID-19 classification, while segmentation achieved a 78.5% Dice score and surpassed previous methods.

  • Takeaways & Limitations

    JCS provides COVID-19 diagnosis with visual explanations and complementary pixel-level predictions of opacification lesions.

Abstract

from arXiv · show

Recently, the coronavirus disease 2019 (COVID-19) has caused a pandemic disease in over 200 countries, influencing billions of humans. To control the infection, identifying and separating the infected people is the most crucial step. The main diagnostic tool is the Reverse Transcription Polymerase Chain Reaction (RT-PCR) test. Still, the sensitivity of the RT-PCR test is not high enough to effectively prevent the pandemic. The chest CT scan test provides a valuable complementary tool to the RT-PCR test, and it can identify the patients in the early-stage with high sensitivity. However, the chest CT scan test is usually time-consuming, requiring about 21.5 minutes per case. This paper develops a novel Joint Classification and Segmentation (JCS) system to perform real-time and explainable COVID-19 chest CT diagnosis. To train our JCS system, we construct a large scale COVID-19 Classification and Segmentation (COVID-CS) dataset, with 144,167 chest CT images of 400 COVID-19 patients and 350 uninfected cases. 3,855 chest CT images of 200 patients are annotated with fine-grained pixel-level labels of opacifications, which are increased attenuation of the lung parenchyma. We also have annotated lesion counts, opacification areas, and locations and thus benefit various diagnosis aspects. Extensive experiments demonstrate that the proposed JCS diagnosis system is very efficient for COVID-19 classification and segmentation. It obtains an average sensitivity of 95.0% and a specificity of 93.0% on the classification test set, and 78.5% Dice score on the segmentation test set of our COVID-CS dataset. The COVID-CS dataset and code are available at https://github.com/yuhuan-wu/JCS.

I. INTRODUCTION

The paper addresses limited RT-PCR sensitivity, slow CT interpretation, and inadequate existing datasets by introducing a large annotated dataset and an explainable joint classification-segmentation system. On COVID-CS, JCS reports high classification sensitivity and specificity alongside segmentation performance.

  • Motivation: RT-PCR sensitivity is insufficient, while CT interpretation takes about 21.5 minutes per case and experienced radiologists are scarce.These constraints motivate automatic diagnosis systems that can identify potentially infected patients in time.
  • Motivation: Existing COVID-19 CT datasets are small and generally lack fine-grained pixel-level annotations, limiting robust and explainable CNN diagnosis.Most datasets contain at most hundreds of CT images from tens of cases and provide only patient-level labels.
  • Contributions: JCS combines a classification branch for suspected COVID-19 identification and visual explanations with a segmentation branch for locating and measuring lung infection.The system uses activation mapping for explanations and fine-grained segmentation for lesion areas.
  • Results: 22.0 seconds are required per infected case and 1 second per uninfected case, with radiologist assistance taking 54.4 seconds per infected case.The reported timing is presented as faster than RT-PCR testing and CT analysis by experienced radiologists.
  • Contributions: The COVID-CS dataset contains 3,855 pixel-level labeled CT images from 200 COVID-19 patients, 64,771 patient-level annotated images from 200 other patients, and 75,541 images from 350 uninfected cases.The dataset combines patient-level and pixel-level annotations for classification and lesion segmentation.
  • Results: 95.0% sensitivity, 93.0% specificity, and 78.5% Dice score are reported for COVID-19 classification and segmentation on the COVID-CS test sets.The reported classification metrics apply to classification, while Dice applies to segmentation.

II. RELATED WORKS

Related work includes publicly released COVID-19 radiology datasets and manual diagnostic practice combining RT-PCR with CT. These resources remain constrained by limited scale, while CT offers complementary early-stage sensitivity and supports illness monitoring.

  • Public datasets: Public COVID-19 radiology datasets include X-ray and CT collections assembled from websites, papers, and other public sources.The COVID-CT dataset contains 746 CT images: 349 from COVID-19 patients and 397 from non-COVID-19 cases.
  • Public datasets: The Cohen et al. X-ray dataset contains 122 frontal-view images, including 100 COVID-19, 11 SARS, and 11 other pneumonia images.These datasets were publicly collected and selected for COVID-19-related abnormalities.
  • Manual diagnosis: RT-PCR is the main manual diagnostic tool, but CT is used as a complementary test because combining both can identify early-stage COVID-19 patients with high sensitivity.Some RT-PCR-positive cases have normal CT scans, supporting use of both tests.
  • Manual diagnosis: CT is also necessary for monitoring illness severity, while shortages of experienced medical staff make timely identification difficult.These constraints contribute to the stated demand for automatic diagnosis systems.

C. Automatic COVID-19 Diagnosis Systems

The paper reviews automatic COVID-19 imaging systems and introduces JCS, which integrates classification and segmentation to provide explainable diagnostic evidence.

  • C. Automatic COVID-19 Diagnosis Systems: JCS integrates learning-based classification and segmentation networks to provide explainable diagnostic evidence and improve user interaction.
  • C. Automatic COVID-19 Diagnosis Systems: The segmentation branch discovers opacification areas while the classification branch identifies COVID-19 opacifications.The classifier uses patient-level annotations and some pixel-level annotations for improved activation mapping.
  • C. Automatic COVID-19 Diagnosis Systems: The classification branch uses Res2Net to predict COVID-19 infection probabilities from CT images and aggregates image-level predictions into a patient diagnosis.A patient is diagnosed as positive when the number of infected CT images exceeds a threshold.

2) Explanation by Activation Mapping:

The system explains classification predictions through activation mapping and reduces source-related bias using image mixing and pixel-level supervision.

  • 2) Explanation by Activation Mapping:: Activation mapping restores spatial information lost after global average pooling and makes the classifier's predictions more transparent.The last convolutional layer is followed by global average pooling and a fully connected layer.
  • 2) Explanation by Activation Mapping:: Larger gradients assign larger activation-mapping weights to feature-map channels.
  • 2) Explanation by Activation Mapping:: Activation mapping accurately locates COVID-19 opacification areas in patient scans.
  • 3) Alleviating Data Bias by Image Mixing:: Image mixing helps the classifier focus on decisive lesion areas rather than unwanted source-dependent regions.The mixing process also weakens label confidence, alleviating overfitting.
  • 4) Pixel-level Supervision for Activation Mapping:: The segmentation loss directly supervises the COVID-19 activation map with binary pixel-level annotations when such labels are available.The loss is omitted for images without ground-truth pixel-level annotations and improves opacification localization.

B. Accurate Segmentation

The segmentation branch uses a VGG-16 encoder, enhanced multi-scale features, and attentive fusion in a U-shaped decoder to predict lesion masks.

  • B. Accurate Segmentation: The segmentation model consists of an encoder and a decoder.
  • 1) Encoder-Decoder Architecture:: The VGG-16 encoder produces multi-scale feature maps, with an Enhanced Feature Module added to improve representation at the deepest block.The module uses two Grouped Atrous Modules to obtain stronger features with larger receptive fields.
  • 1) Encoder-Decoder Architecture:: The architecture can feed either the standalone encoder feature or combined classification features into the decoder and uses deep supervision during training.
  • 1) Encoder-Decoder Architecture:: The encoder produces six feature-map levels with strides of 1, 2, 4, 8, 16, and 32 for the U-shaped decoder.
  • 1) Encoder-Decoder Architecture:: The decoder uses five side outputs and selects the full-resolution output as the final prediction.No side output is predicted from the coarsest stride-32 feature map.
  • 1) Encoder-Decoder Architecture:: Attentive Feature Fusion emphasizes smaller top-level feature maps and filters useful information from bottom feature maps.

2) Enhanced Feature Module:

The Enhanced Feature Module enriches encoder features with multiscale atrous convolutions, channel recalibration, and attentive feature fusion. These operations expand receptive fields and prioritize smaller feature maps during aggregation.

  • Enhanced Feature Module: The module uses two sequential GAM modules with max pooling between them after the VGG-16 encoder’s final E5 layer.Each GAM expands channels, divides features into four groups, and applies atrous convolutions with different rates.
  • Enhanced Feature Module: Different atrous rates provide varied receptive fields while retaining the computational cost of normal convolution.The formulation uses a 3 × 3 kernel, with n = 1 corresponding to normal convolution.
  • Enhanced Feature Module: Squeeze-Excitation blocks recalibrate channel-wise convolutional responses using attention-derived channel factors.The block uses two linear layers and a sigmoid function after global average pooling.
  • Enhanced Feature Module: Attentive Feature Fusion values the smaller input feature map more strongly when aggregating multiscale features.Inputs are reduced, resized, concatenated, and enhanced with SE blocks before transition convolutions.

4) Combination with the Classification Model:

The system combines classification and segmentation features to enhance segmentation. Features from corresponding encoder and classification-backbone stages are resized, concatenated, recalibrated, and passed to the decoder.

  • Combination with the Classification Model: The proposed combination merges feature maps from each segmentation-encoder stage with corresponding classification-backbone features.The classification model uses the last feature map from each of five Res2Net stages.
  • Combination with the Classification Model: At each stage, the smaller classification feature map is resized to match the larger segmentation feature map before concatenation.A 1 × 1 convolution then reduces the merged feature channels to the segmentation feature width.
  • Combination with the Classification Model: A reduction-rate-4 SE block and a 3 × 3 transition convolution further process each merged feature representation.The resulting enhanced encoder features are fed into the segmentation decoder.

5) Deep Supervision Loss:

The segmentation branch uses deep supervision across all side-outputs, combining binary cross-entropy and Dice losses after resizing predictions to the ground-truth resolution. The joint system uses classification for screening and segmentation for detailed lesion analysis.

  • Deep Supervision Loss: Deep supervision applies losses to all side-outputs, not only the final prediction.Each side-output is up-sampled to the ground-truth map size before loss computation.
  • Deep Supervision Loss: The segmentation loss combines standard binary cross-entropy with Dice loss.The displayed objective is L = BCE(P, G) + 1 − P · G / (∥P∥1 + ∥G∥1).
  • Deep Supervision Loss: The binary cross-entropy term averages pixelwise confidence errors over H × W pixels, while P and G denote predicted and ground-truth maps.The dot product and ℓ1 norms define the Dice-loss component.
  • Joint Diagnosis: The classification model first predicts COVID-19 positivity, after which the segmentation model analyzes positive cases for complete opacification areas.The classifier provides lesion localization through activation mapping, while segmentation discovers whole opacification regions.

B. Professional Labeling

Professional labeling supplies pixel-level annotations for selected infection-containing CT slices. Radiologists mark opacification areas, refine the labels through review, and produce a segmentation subset alongside larger classification data.

  • Professional Labeling: The COVID-CS dataset is divided into 400 COVID-19 cases and 350 uninfected cases for joint classification and segmentation tasks.The labeling process focuses on pixel-level annotations for COVID-19 opacification areas.
  • Professional Labeling: The segmentation set summarizes lesion counts, opacification dimensions, and relationships between opacification areas and locations.These statistics are presented for 200 COVID-19 cases.
  • Professional Labeling: Radiologists select at most 30 discrete CT images per patient in which infection is observed for further annotation.This limits labeling to slices containing visible infections while targeting every opacification area.
  • Professional Labeling: 3,855 CT images receive pixel-level labels after one radiologist marks opacifications and a senior radiologist repeatedly refines them for cross-validation.The review process fixes some inaccurate labels.

C. Dataset Statistics

The COVID-CS dataset includes diverse patient, lesion, and image-level annotations, with separate protocols for classification and segmentation evaluation.

  • The 400 COVID-19 patients range from 14 to 89 years, with an average age of 48.9 years.
  • Lesion counts range from 1 to 10 in each CT scan image.
  • Opacification widths range from 7 to 191 and heights from 8 to 271, showing diverse distributions.
  • Normalized opacification areas range from 35/28452 to 28452/28452 pixels and are evenly distributed across lung locations.
  • The segmentation protocol uses 2,794 training images from 150 patients and 1,061 test images from 50 other patients.
  • Specificity and sensitivity evaluate classification, while Dice, IoU, and Eφ evaluate segmentation.

C. Results

The JCS system improves explainability and segmentation quality while maintaining robust classification and efficient inference across COVID-19 CT cases.

  • Explainable classification: Image mixing and segmentation loss make activation maps more accurate and specific in locating opacifications.
  • Explainable classification: 95.0% sensitivity and 93.0% specificity are achieved at a classification threshold of 25.
  • Segmentation ablation: 3.3% and 4.9% Dice improvements result from applying EFM and AFF separately, while combining them yields 6.5% higher Dice.
  • Segmentation ablation: 1.0% Dice improvement results when classification-model features are added to the segmentation model.
  • Segmentation comparison: 8.8%, 10.5%, and 8.8% improvements in Dice, IoU, and Eφ over PoolNet make the proposed model best on all three metrics.
  • Segmentation stability: 95.9% of images have Dice scores in [0.6, 1], while 0.8% are failure cases, indicating stable segmentation across lesion counts.
  • Inference speed: 22.0 seconds are required for each infected case and 1.0 second for each uninfected case on a single RTX 2080Ti.

VI. FUTURE WORKS

Future work proposes strengthening the diagnosis system with transformer backbones, neural architecture search, and improved CNN visualization techniques.

  • Transformer backbones are proposed to enhance the diagnosis system through stronger global-information modeling.
  • Neural architecture search is proposed to automatically optimize the framework’s detailed architecture.
  • Novel CNN visualization techniques are proposed to provide better CNN explanations.
Loading 2004.07054v3…