PaperlayerView exact revision on arXiv

Source-linked AI summary

A 3D Probabilistic Deep Learning System for Detection and Diagnosis of Lung Cancer Using Low-Dose CT Scans

Onur Ozdemir, Rebecca L. Russell, Andrew A. Berlin

arXiv:1902.03233v3cs.CVcs.LG

TL;DR

Lung CT screening needs automated detection and diagnosis that can communicate uncertainty reliably. The paper builds a coupled end-to-end 3D-CNN system with uncertainty-aware probabilities and evaluates it on public detection and malignancy datasets. It reports strong benchmark performance, calibrated probabilities, and improved referral decisions, while noting dataset limitations.

  • Problem

    Lung CT screening creates a need for accurate automated detection and diagnosis that can communicate prediction uncertainty reliably.

  • Method

    The system jointly develops 3D-CNN CADe and CADx modules, ranking nodule candidates before top-k multiple-instance patient classification and incorporating model uncertainty.

  • Results

    The system achieves state-of-the-art performance on LUNA16 nodule detection and Kaggle malignancy classification while producing calibrated classification probabilities.

  • Takeaways & Limitations

    Calibrated probabilities support utility- or risk-based diagnostic decisions, including patient referral strategies that further improve performance.

  • Takeaways & Limitations

    Performance is bounded by dataset limitations, including incomplete large-nodule annotations, and missed malignant lesions cannot be recovered by downstream CADx.

Abstract

from arXiv · show

We introduce a new computer aided detection and diagnosis system for lung cancer screening with low-dose CT scans that produces meaningful probability assessments. Our system is based entirely on 3D convolutional neural networks and achieves state-of-the-art performance for both lung nodule detection and malignancy classification tasks on the publicly available LUNA16 and Kaggle Data Science Bowl challenges. While nodule detection systems are typically designed and optimized on their own, we find that it is important to consider the coupling between detection and diagnosis components. Exploiting this coupling allows us to develop an end-to-end system that has higher and more robust performance and eliminates the need for a nodule detection false positive reduction stage. Furthermore, we characterize model uncertainty in our deep learning systems, a first for lung CT analysis, and show that we can use this to provide well-calibrated classification probabilities for both nodule detection and patient malignancy diagnosis. These calibrated probabilities informed by model uncertainty can be used for subsequent risk-based decision making towards diagnostic interventions or disease treatments, as we demonstrate using a probability-based patient referral strategy to further improve our results.

I. INTRODUCTION

The paper addresses the burden of accurate lung CT screening and unreliable uncertainty communication by introducing an end-to-end probabilistic 3D-CNN system. It jointly develops detection and diagnosis components, evaluates them on public benchmarks, and uses model uncertainty for calibrated probabilities and referral decisions.

  • Annual low-dose CT screening could reduce lung cancer mortality by at least 20% after 7 years versus annual chest radiography.
  • The system targets radiologist workload and unreliable communication of algorithmic uncertainty in large-scale CT screening.
  • The CADe module detects and segments suspicious nodules, while CADx performs nodule-level assessment and patient-level malignancy classification.
  • The paper reports comparable or better performance than published systems on LUNA16 and Kaggle Data Science Bowl benchmarks using only their available data and labels.
  • The system integrates model uncertainty to produce calibrated probabilities and proposes probability-based referral strategies for uncertain patients or nodules.
  • CADe and CADx are developed together because CADe recall-precision trade-offs directly affect downstream diagnosis performance.

II. RELATED WORK

Related work largely optimizes lung nodule detection or diagnosis independently, despite their operational coupling. The paper contrasts this practice with an end-to-end system that avoids a separate false-positive reduction stage and matches published benchmark performance without extra annotations.

  • Earlier deep-learning CADe methods moved from 2D slices toward 3D volumetric images because 3D features can improve performance.
  • Most CADe systems extract nodule candidates and then apply a separate 3D-CNN false-positive reduction step.
  • The combined system does not require a second false-positive reduction stage because its CADx model is relatively insensitive to false positives.
  • LUNA16 CADe performance reached 0.921, compared with 0.897 for a prior single-step 3D dense-CNN model.
  • LIDC-IDRI malignancy labels reflect radiologist opinions rather than pathology confirmation and exhibit inter-observer variability.
  • On the Kaggle test set, a prior system reported 0.87 AUROC after adding hand annotations, whereas this work achieved equally good CADx performance without additional annotations.

III. CAD SYSTEM MODEL

The system converts a raw 3D lung CT scan into patient-level malignancy probabilities, nodule scores, and segmented candidates through coupled CADe and CADx modules. CADx ranks candidates before top-k multiple-instance classification, while joint tuning addresses dataset and candidate-quality dependencies.

  • Overall system: The system outputs per-patient malignancy probabilities, per-nodule malignancy scores, and segmented lung nodule candidates from raw 3D CT scans.
  • CADe: CADe performs 3D segmentation for high-recall nodule localization before passing identified nodules to CADx.
  • CADx: CADx first ranks candidates by malignancy risk, then classifies the top-k candidates with a multiple-instance model to diagnose each patient.
  • Joint development: The full system is tuned simultaneously because CADx performance depends on CADe performance.
  • Joint development: Training CADx with high-FPR candidates makes it more robust to false positives and improves performance across underlying CADe false-positive rates.
  • Datasets: The LUNA16 data lack very large nodules, so the system uses original-resolution and downsampled candidate sets to address this limitation.

B. Data Preprocessing and Augmentation

Preprocessing standardizes CT volumes, while extensive training-time augmentation reduces overfitting and broadens variation. CADe uses voxelwise 3D segmentation to generate candidates through thresholding, filtering, connected-region labeling, and center-of-mass estimation.

  • Preprocessing: CT scans are clipped to -1000–400 Hounsfield units, resampled to 1 mm isotropic voxels, and normalized to zero mean and unit variance.
  • Augmentation: Training augmentation applies random 3D rotations, reflections, scaling, translations, gamma changes, blur or sharpening, and Gaussian noise.
  • Augmentation: Aggressive 3× scaling augmentation is used for CADe to improve recall of large nodules beyond the annotated LIDC-IDRI size range.
  • Candidate extraction: CADe combines a 3D voxelwise segmentation network with a 3D scoring network to identify and rank pulmonary nodule candidates.
  • Segmentation model: The segmentation model is a V-Net-based 3D fully convolutional network with encoder-decoder blocks, skip connections, and residual connections.
  • Candidate extraction: Candidates are extracted by thresholding voxel scores, applying binary opening, labeling connected regions, and computing score-weighted centers of mass.

B. False Positive Reduction for LUNA16 Evaluation

The CADe scoring network operates on 3D candidate-centered blocks and supplies candidates to CADx, while incorporating MC-dropout uncertainty into candidate scores.

  • False Positive Reduction for LUNA16 Evaluation: All generated candidates are supplied to the CADx system, while the LUNA16 evaluation uses a separate scoring network on candidate-centered blocks.
  • False Positive Reduction for LUNA16 Evaluation: The CADe scoring network operates on 32^3-shaped blocks around candidate centers and is trained to distinguish true nodules from false positives.True candidates receive twice the weight of false positives in the cross-entropy loss.
  • False Positive Reduction for LUNA16 Evaluation: MC-dropout quantifies model uncertainty in candidate scores, making the resulting nodule probabilities directly interpretable on the LUNA16 data distribution.

VI. CADX

The CADx pipeline ranks suspicious candidates, then classifies each CT scan using attention-based multiple-instance learning and uncertainty-aware probabilistic outputs. It also uses original and downsampled candidate sets to address the LUNA16 size-labeling limitation.

  • CADx: Two consecutive 3D CNNs first rank candidate malignancy risks, then classify each CT scan from the top-k ranked candidates using multiple-instance learning.The ranking network reduces noise by focusing classification on candidates more likely to be malignant.
  • CADx: The classification network outputs a probabilistic patient-level malignancy score and candidate-level malignancy risk scores.
  • CADx: Attention-based MIL assigns permutation-invariant weights to candidate features, whose weighted average feeds a sigmoid patient-level malignancy prediction.The attention weights sum to 1; alternative combination methods performed worse on this dataset.
  • CADx: Because LUNA16 annotations ignored nodules larger than 30 mm, the system adds candidates from 2 mm^3/voxel downsampled scans to detect large nodules.These candidates are ranked alongside candidates from the original 1 mm^3/voxel scans.
  • CADx: The best CADx model processes four candidates: the top two from each of the original and downsampled candidate lists.
  • CADx: Five models combining MC dropout and deep ensembles provide uncertainty estimates intended to support calibrated malignancy probabilities and clinical risk-utility decisions.

VII. RESULTS

The evaluation tests CADe on LUNA16 and CADx on the independent Kaggle Stage-2 set, assessing generalization across datasets and image quality.

  • RESULTS: CADe is evaluated on LUNA16, while CADx is tested on Kaggle Stage-2 after training and validation on Kaggle Stage-1.Kaggle Stage-2 is described as more recent and having different image quality.

A. CADe Results

The CADe system achieves high sensitivity while reducing false positives, with LUNA16 performance comparable to or better than published state-of-the-art results. A standard 3D U-Net comparison produces similar outcomes.

  • CADe Results: 96.5% sensitivity was achieved at 19.7 average false positives during candidate selection from segmented test scans.The false-positive average was dominated by a small number of patients with large regions of uncertain segmentation.
  • CADe Results: 0.921 average LUNA16 metric was comparable to or better than published state-of-the-art results, despite no optimization for the high-precision limit.
  • CADe Results: CADe performance was strongest for pulmonary nodules larger than 5 mm because false positives were dominated by small candidates.
  • CADe Results: The 3D U-Net comparison architecture produced CADe and CADx results similar to the primary results.

B. CADx Results

The CADx system uses CADe-generated candidates for malignancy classification and achieves strong performance on Kaggle while remaining robust to candidate-generation choices.

  • The best CADx model used candidates generated at approximately 8 false positives per scan, with a CADe segmentation F1 score of 0.40, precision of 0.25, and recall of 0.93.
  • The CADe FROC was also evaluated using a 3D U-Net comparison architecture alongside the primary system.
  • CADe sensitivity was significantly worse for 3–5 mm nodules than for 5–30 mm nodules at lower thresholds corresponding to fewer false positives.
  • 0.87 average ROC AUC was achieved on the Kaggle Stage-2 test set.
  • The Kaggle performance matched the winning solution without hand-labeling Stage-1 nodules or ensembling different CADx solutions.

C. CADe-CADx Interaction Studies

The interaction studies show that CADx performance depends on how CADe candidates are generated and filtered, with false-positive reduction not uniformly beneficial.

  • Table I reports CADe and CADx results across CADe threshold false-positive rates.
  • False-positive reduction steps added no value because the CADx system was relatively insensitive to CADe false-positive candidates.
  • Restricting training and testing candidates to 1 or 1/8 false positives per scan significantly degraded CADx AUROC and reduced robustness to false positives.
  • False-positive candidates benefited CADx training by improving its internal CADe capabilities and robustness to false positives.

VIII. CALIBRATION & REFERRAL

The system incorporates model uncertainty to produce calibrated nodule and malignancy probabilities, then uses those probabilities to prioritize uncertain cases for referral. Referral improves performance on selected remaining cases, although highly confident errors remain.

  • Model uncertainty is incorporated into CADe and CADx probability estimates, which are reported as well calibrated on test data.
  • MC dropout approximates Bayesian inference for CADe scoring, while CADx malignancy uncertainty combines MC dropout with deep ensembles.
  • Highest-entropy referral produces a dramatic CADe improvement even below a 10% referral rate, with remaining-nodule performance approaching 1 as referral increases.
  • Patient referral raises remaining-data malignancy-classification performance to a maximum AUC of 0.885.
  • At higher referral rates, AUC decreases because some incorrect CADx decisions occur when the system is highly confident, often after CADe misses or fails to detect a malignant nodule.

IX. CONCLUSIONS

The system combines 3D CNN-based detection and diagnosis, incorporates model uncertainty for calibrated probabilities, and achieves state-of-the-art performance on lung nodule detection and malignancy classification. Joint development, probability-based referral, and broader datasets are identified as important for reliable end-to-end diagnosis.

  • The cascade of 3D CNNs achieves state-of-the-art performance on lung nodule detection and malignancy classification using the LUNA16 and Kaggle datasets.
  • Quantified model uncertainty from Monte Carlo dropout and deep ensembles enables calibrated classification probabilities for diagnostic decision-making.
  • Calibrated probabilities can support patient referral decisions that further improve system performance.
  • CADe and CADx should be developed jointly because detection operating points directly affect downstream diagnosis performance.
  • Performance remains bounded by training-data limitations, including insufficient large-nodule annotations and limited coverage of nodule varieties, patients, and scanning equipment.
  • If CADe misses a malignant lesion, CADx cannot classify the scan as malignant; future work includes training models with an explicit reject option.
Loading 1902.03233v3…