PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Data-Efficient Learning: A Benchmark for COVID-19 CT Lung and Infection Segmentation

Jun Ma, Yixin Wang, Xingle An, Cheng Ge, Ziqi Yu, Jianan Chen, Qiongjie Zhu, Guoqiang Dong, Jian He, Zhiqiang He, Yuntao Zhu, Ziwei Nie, Xiaoping Yang

arXiv:2004.12537v2eess.IVcs.CVcs.LG

TL;DR

COVID-19 CT segmentation lacks accessible annotated data and standardized comparisons, motivating a data-efficient benchmark. The paper constructs three benchmarks from annotated COVID-19 cases, standardizes evaluation, and releases more than 40 baseline models. The resources support comparable research and practical segmentation with limited data, although the dataset is relatively small and methodological innovation is limited.

  • Problem

    COVID-19 CT segmentation commonly depends on large private datasets, while inconsistent splits and metrics make methods difficult to compare.

  • Method

    The paper builds three benchmarks covering few-shot learning, domain generalization, and knowledge transfer, with standardized settings and nnU-Net baselines.

  • Results

    More than 40 publicly available trained baseline models and segmentation results are provided for the benchmark tasks.

  • Takeaways & Limitations

    Unified task settings make comparisons more feasible, while public baseline models can save training time for researchers developing limited-data COVID-19 segmentation methods.

  • Takeaways & Limitations

    The dataset contains a relatively small number of cases, and the paper's innovative methodology contribution is limited.

Abstract

from arXiv · show

Purpose: Accurate segmentation of lung and infection in COVID-19 CT scans plays an important role in the quantitative management of patients. Most of the existing studies are based on large and private annotated datasets that are impractical to obtain from a single institution, especially when radiologists are busy fighting the coronavirus disease. Furthermore, it is hard to compare current COVID-19 CT segmentation methods as they are developed on different datasets, trained in different settings, and evaluated with different metrics. Methods: To promote the development of data-efficient deep learning methods, in this paper, we built three benchmarks for lung and infection segmentation based on 70 annotated COVID-19 cases, which contain current active research areas, e.g., few-shot learning, domain generalization, and knowledge transfer. For a fair comparison among different segmentation methods, we also provide standard training, validation and testing splits, evaluation metrics and, the corresponding code. Results: Based on the state-of-the-art network, we provide more than 40 pre-trained baseline models, which not only serve as out-of-the-box segmentation tools but also save computational time for researchers who are interested in COVID-19 lung and infection segmentation. We achieve average Dice Similarity Coefficient (DSC) scores of 97.3\%, 97.7\%, and 67.3\% and average Normalized Surface Dice (NSD) scores of 90.6\%, 91.4\%, and 70.0\% for left lung, right lung, and infection, respectively. Conclusions: To the best of our knowledge, this work presents the first data-efficient learning benchmark for medical image segmentation and the largest number of pre-trained models up to now. All these resources are publicly available, and our work lays the foundation for promoting the development of deep learning methods for efficient COVID-19 CT segmentation with limited data.

I. INTRODUCTION

COVID-19 CT segmentation research is constrained by limited public annotations, inconsistent evaluation settings, and unavailable trained baselines. The paper addresses these gaps with an annotated dataset, three data-efficient benchmarks, and publicly available models.

  • Existing COVID-19 segmentation solutions mainly rely on large private datasets that may be inaccessible because of privacy and intellectual-property concerns.
  • Public COVID-19 3D datasets, standardized benchmarks, and publicly available trained baselines were identified as major unresolved needs.
  • The paper provides a COVID-19 CT dataset with left-lung, right-lung, and infection annotations from chest radiologists.
  • Three benchmark tasks target few-shot learning, domain generalization, and knowledge transfer for segmentation with limited training cases.
  • More than 40 trained state-of-the-art models and segmentation results are publicly available as baselines and out-of-the-box tools.

II. MATERIALS

The materials include public lung CT annotations from datasets with different diseases and annotation scopes. These resources support investigation of data-efficient segmentation and cross-dataset reuse.

  • Existing public lung CT annotations with other diseases are used to assist COVID-19 CT segmentation because COVID-19 annotations are scarce.
  • Figure 1 presents examples of five lung CT datasets, including original CT images, ground truth, and three-dimensional renderings.
  • The StructSeg lung organ dataset contains 50 lung cancer patient CT scans from one medical center, with six annotated organs including both lungs.

2) NSCLC left and right lung segmentation:

The NSCLC-related materials provide publicly available CT datasets with lung, tumor, and pleural-effusion annotations. A separate 50-case MosMed dataset is used for independent testing of generalization.

  • NSCLC left and right lung segmentation:: The NSCLC Radiomics dataset provides left- and right-thoracic-volume segmentations delineated on 402 CT scans.
  • NSCLC left and right lung segmentation:: A Stanford NSCLC dataset contains 63 labelled CT scans with tumors annotated by an expert thoracic radiologist.
  • NSCLC left and right lung segmentation:: The corresponding 50 lung cancer scans include gross target-volume tumor annotations in each case.
  • NSCLC left and right lung segmentation:: Pleural effusion is delineated for 78 cases from the same scans used in the NSCLC lung-segmentation dataset.
  • NSCLC left and right lung segmentation:: A separate dataset contains 50 annotated COVID-19 CT scans from municipal hospitals in Moscow and serves as an independent testing set.

C. Our COVID-19-CT-Seg dataset

The COVID-19-CT-Seg benchmark provides annotated COVID-19 CT data and standardized experimental settings for data-efficient lung and infection segmentation. It evaluates limited-data, cross-domain, and heterogeneous-data strategies with unified splits and metrics.

  • Dataset construction: 20 public COVID-19 CT scans were collected and annotated for left lung, right lung, and infection segmentation.Annotations were refined and verified by radiologists and manually produced slice by slice on axial images.
  • Benchmark tasks: Three tasks evaluate segmentation with pure but limited COVID-19 scans, annotated non-COVID-19 lung scans, and heterogeneous COVID-19 and non-COVID-19 datasets.These settings target few-shot learning, domain generalization, and knowledge transfer.
  • Standardization: Unified training, validation, and testing splits, experimental settings, and evaluation metrics standardize segmentation protocols for fair comparison.The benchmark settings are summarized across Tasks 1–3.
  • Task 1: Task 1 uses the COVID-19-CT-Seg dataset to assess learning with limited annotations for lung, infection, and joint segmentation.Its experiments use predefined five-fold cross-validation settings with four training cases and sixteen validation cases per fold.

B. Task 2: Learning to segment COVID-19 CT scans from non-COVID-19 CT scans

Task 2 benchmarks domain generalization by training on non-COVID-19 lung datasets and evaluating COVID-19 CT segmentation. The benchmark uses complementary region- and boundary-based metrics to assess performance.

  • Task design: Task 2 evaluates domain generalization when only out-of-domain non-COVID-19 data are available for training.It includes separate subtasks for lung segmentation and infection segmentation.
  • Training data: The lung subtask trains on StructSeg Lung and NSCLC Lung, while the infection subtask uses MSD Lung Tumor, StructSeg Gross Target, and NSCLC Pleural Effusion.For both subtasks, 80% of the data are randomly selected for training and 20% for the remaining evaluation split.
  • Target definition: Infection is defined as all visibly affected regions of the lungs.This operational definition applies to the benchmark’s infection-segmentation target.
  • Evaluation metrics: Dice similarity coefficient measures region overlap, whereas Normalized Surface Dice measures surface closeness at a specified tolerance τ.Higher scores are better for both metrics, and 100% denotes perfect segmentation.

2) Boundary-based measure:

The paper evaluates segmentation with complementary region- and boundary-based metrics, while standardizing its nnU-Net baselines across tasks and datasets.

  • Boundary-based measure:: Normalized Surface Dice evaluates how closely segmentation and ground-truth surfaces align within a specified tolerance τ.The paper sets τ to 1 mm for lung segmentation and 3 mm for infection segmentation, based on inter-rater variation.
  • Boundary-based measure:: Surface Dice ignores small boundary deviations because inter-observer errors are unavoidable and often clinically irrelevant.
  • E. U-Net baselines: oldies but goldies: The nnU-Net baseline automatically adapts preprocessing and network architecture to each 3D medical dataset.The authors manually equalize patch sizes and architectures across Tasks 1–3 for comparability.
  • E. U-Net baselines: oldies but goldies: Figure 2 encodes feature-map sizes near convolutional blocks and convolutional-kernel strides inside the 3D U-Net.
  • E. U-Net baselines: oldies but goldies: Training uses Z-score normalization, a combined cross-entropy and Dice loss, stochastic gradient descent, PolyLR scheduling, and fixed-size patch sampling.Patches are 192 × 192 × 64, training runs for 1000 epochs, and testing uses sliding-window inference.

IV. RESULTS AND DISCUSSION

Task 1 evaluates limited-annotation segmentation using five-fold cross-validation and compares separate versus joint lung-and-infection training.

  • IV. RESULTS AND DISCUSSION: Five-fold cross-validation is necessary because average DSC and NSD vary greatly across folds with different testing-case difficulty.The authors warn that one-fold results may be biased.
  • IV. RESULTS AND DISCUSSION: Promising left- and right-lung segmentation is achieved with as few as four training cases.
  • IV. RESULTS AND DISCUSSION: Separate lung segmentation models significantly outperform models trained jointly for lung and infection segmentation.Figure 3 visualizes the separate and union training strategies; separate training especially improves left- and right-lung results.
  • IV. RESULTS AND DISCUSSION: Infection segmentation still has substantial room for improvement when annotations are limited.Union training can confuse the left and right lungs, adversely affecting infection segmentation.

B. Results of Task 2: Learning to segment COVID-19 CT scans from non-COVID-19 CT scans

Task 2 tests domain generalization from non-COVID-19 CT data to COVID-19 CT scans, revealing strong in-domain lung segmentation but substantial out-of-domain degradation, especially for infection segmentation.

  • B. Results of Task 2: Learning to segment COVID-19 CT scans from non-COVID-19 CT scans: 3D U-Net achieves excellent DSC performance on the in-domain lung segmentation set, but lower NSD values indicate predominantly boundary-related errors.
  • B. Results of Task 2: Learning to segment COVID-19 CT scans from non-COVID-19 CT scans: Performance drops significantly on the out-of-domain testing set for both lung segmentation subtasks.The model trained on NSCLC Lung performs worse than the model trained on StructSeg lung, possibly because StructSeg and COVID-19-CT have more similar lung-appearance distributions.
  • B. Results of Task 2: Learning to segment COVID-19 CT scans from non-COVID-19 CT scans: In-domain infection segmentation performs worse than lung segmentation, showing that lesion segmentation remains challenging.
  • B. Results of Task 2: Learning to segment COVID-19 CT scans from non-COVID-19 CT scans: Models almost fail to predict COVID-19 infections on the testing set.The authors relate this result to significant differences in lesion appearances across lung cancer, pleural effusion, and COVID-19 infections.

C. Results of Task 3: Learning with both COVID-19 and non-COVID-19 CT scans

Task 3 combines COVID-19 and non-COVID-19 CT cases to evaluate heterogeneous-data training for lung and infection segmentation. It improves lung segmentation but remains limited for infection segmentation because of domain gaps and sensitivity to small or weak-boundary infections.

  • Even large non-COVID-19 lesion annotations produce obvious five-fold variance, making cross-validation reporting necessary for reliable and robust evaluation.
  • On testing infection segmentation, performance drops about 4%-14% in DSC and 11%-16% in NSD when COVID-19 and non-COVID-19 cases are simply fused.
  • Task 3 achieves up to 97.3% DSC and 90.6% NSD for left lung segmentation, and 97.7% DSC and 91.4% NSD for right lung segmentation.
  • Adding out-of-domain StructSeg and NSCLC annotations boosts left and right lung segmentation, supporting their use when COVID-19 annotations are scarce.
  • Including COVID-19 cases improves lung segmentation on both subtasks, with NSCLC gaining up to 34% in DSC for left lung segmentation.
  • Using only out-of-domain cases cannot predict COVID-19 infections, whereas adding a few COVID-19 cases significantly boosts performance; infection segmentation remains weaker than lung segmentation.
  • Models can segment infections with good contrast and clear boundaries from four training cases, but often miss small or weak-boundary infections.

E. Limitation

The dataset’s relatively small number of cases is a stated limitation, although the authors argue it is appropriate for limited-data benchmark tasks. The testing-set sizes are also comparable to recent MICCAI 2020 segmentation challenges.

  • The dataset contains a relatively small number of cases, which the authors identify as a possible limitation.
  • The authors consider the training-case count acceptable because the benchmark focuses on learning from limited training cases.
  • The benchmark is presented as applicable to general small-sample learning problems.
  • 16 or 20 cases in COVID-19-CT-Seg and 50 cases in MosMed are described as comparable to recent MICCAI 2020 segmentation challenges.StructSeg used 10 testing cases, while ASOCA used 20 testing cases.

V. CONCLUSION

The paper addresses limited-data COVID-19 image analysis by creating a CT dataset, three segmentation benchmarks, and more than 40 baseline models. The related results are publicly available.

  • The paper creates a COVID-19 CT dataset and establishes three segmentation benchmark tasks for research with limited data.
  • More than 40 baseline models are provided based on state-of-the-art segmentation architectures.
  • The benchmark’s related results are publicly available through the project repository.
Loading 2004.12537v2…