Source-linked AI summary
Towards automatic pulmonary nodule management in lung cancer screening with deep learning
Francesco Ciompi, Kaman Chung, Sarah J. van Riel, Arnaud Arindra Adiyoso Setio, Paul K. Gerke, Colin Jacobs, Ernst Th. Scholten, Cornelia Schaefer-Prokop, Mathilde M. W. Wille, Alfonso Marchiano, Ugo Pastorino, Mathias Prokop, Bram van Ginneken
TL;DR
Lung cancer screening will create many CT scans requiring nodule assessment, whose management depends strongly on nodule type and size. The paper introduces a multi-stream, multi-scale convolutional system that classifies six relevant nodule types from raw CT data without segmentation or size estimation. Its performance is within the variability of experienced radiologists, although underrepresented classes and large solid nodules remain challenging.
Problem
Screening requires automatic classification of six nodule types relevant to management, but radiologists show substantial inter- and intra-observer variability.
Method
A multi-stream, multi-scale ConvNet processes multiple orthogonal 2D patch views of raw CT data to estimate probabilities for six nodule classes without segmentation or size estimation.
Results
Performance was within the inter-observer variability of four experienced radiologists, with human agreement κ between 0.59 and 0.75 and system variability in the same range.
Takeaways & Limitations
The system reaches human-level variability for nodule-type classification, supporting its suitability for integration into lung cancer screening workflows.
Takeaways & Limitations
Performance is lower for part-solid and spiculated nodules, and large solid nodules are sometimes misclassified as spiculated because of training-data associations.
Abstract
from arXiv · showhide
The introduction of lung cancer screening programs will produce an unprecedented amount of chest CT scans in the near future, which radiologists will have to read in order to decide on a patient follow-up strategy. According to the current guidelines, the workup of screen-detected nodules strongly relies on nodule size and nodule type. In this paper, we present a deep learning system based on multi-stream multi-scale convolutional networks, which automatically classifies all nodule types relevant for nodule workup. The system processes raw CT data containing a nodule without the need for any additional information such as nodule segmentation or nodule size and learns a representation of 3D data by analyzing an arbitrary number of 2D views of a given nodule. The deep learning system was trained with data from the Italian MILD screening trial and validated on an independent set of data from the Danish DLCST screening trial. We analyze the advantage of processing nodules at multiple scales with a multi-stream convolutional network architecture, and we show that the proposed deep learning system achieves performance at classifying nodule type that surpasses the one of classical machine learning approaches and is within the inter-observer variability among four experienced human observers.
Introduction
Lung cancer screening will greatly increase CT-reading demands, while nodule management depends heavily on nodule type and size. The paper proposes a multi-stream, multi-scale deep learning system to classify all six nodule types relevant to screening workup without segmentation or size estimation.
- Motivation: 20% mortality reduction was demonstrated for heavy-smoker screening with low-dose CT versus chest X-rays.This result motivated implementation of lung cancer screening programs and their associated CT-reading workload.
- Clinical context: Nodule follow-up decisions in Lung-RADS and the PanCan model depend chiefly on nodule type, alongside size, growth, imaging findings, and patient data.The relevant classes are solid, non-solid, part-solid, calcified, perifissural, and spiculated nodules.
- Design rationale: 10 mm, 20 mm, and 40 mm patches provide multiple spatial scales across axial, coronal, and sagittal views.Different scales address complementary cues, such as surrounding fissures and tiny solid cores within lesions.
- Existing limitations: Radiologists show substantial inter- and intra-observer variability when classifying nodule type.Earlier automated approaches relied on handcrafted descriptors and, in some cases, nodule segmentation or size estimation.
- Proposed approach: The proposed multi-stream multi-scale ConvNet analyzes multiple 2D views of raw CT data to classify six nodule types without segmentation or nodule-size estimation.The framework is designed to learn representations directly from raw data rather than requiring an ad-hoc descriptor.
Results
The system was trained on MILD data and evaluated independently on DLCST data, including comparisons with observers and classical machine-learning baselines. Its three-scale architecture achieved observer-level agreement, while performance remained weaker for underrepresented nodule classes.
- Evaluation data: The system was trained on baseline low-dose CT scans from the MILD trial and assessed independently on DLCST data.
- System development: The three-stream samples combined axial, coronal, and sagittal patches, with rotation and patch-level augmentation producing approximately 0.5M training samples.
- Baseline comparison: Performance improved progressively from intensity-based SVM features and unsupervised features to the proposed 3-scale deep-learning approach in accuracy and F-measure.
- Observer comparison: κ values of 0.58–0.67 for computer-observer agreement with the 3-scale architecture fell within human agreement of 0.59–0.75.
- Observer comparison: 72.9% versus 69.6% average accuracy was reported for human observers versus computer-observer comparisons, respectively, with similar trends for other parameters.
- Per-class performance: The system classified solid, calcified, and non-solid nodules with high performance, whereas part-solid and spiculated nodules had low precision and recall in testALL.
Methods
The framework takes a chest CT scan and nodule position, extracts orthogonal 2D patch triplets at multiple resolutions, and classifies the nodule into six classes using a multi-stream architecture.
- The input is a chest CT scan and nodule position q, while the output is a probability for each of six nodule classes.
- Three triplets of orthogonal patches are cropped at 10 mm, 20 mm, and 40 mm scales and processed simultaneously.
- The architecture uses convolutional, pooling, fully connected, and soft-max layers as its principal building blocks.
Generation of triplets of 2D patches
The system generates rotated triplets of orthogonal 2D views through the nodule and crops patches at multiple scales to capture both local appearance and broader context.
- A triplet of orthogonal planes passes through q, and rotating all three planes by the same angle preserves their orthogonality.
- Each plane triplet intersects the CT scan to generate three 2D views, from which square patches centered on q are cropped.
- Increasing N produces more patches per nodule and increases coverage of the nodule volume in 3D.
- Patch scales of 10, 20, and 40 mm provide information ranging from local content to more global context.
Deep learning network
The deep learning network combines nine ConvNet streams across three scales, shares parameters within each scale, and aggregates angle-dependent predictions for nodule classification.
- Nine ConvNet streams are grouped into three sets, with each set processing orthogonal patch triplets at one scale.
- Each patch is 64×64 pixels, covering approximately 40 mm at the used in-plane resolution of 0.67 mm/px.
- Parameters are shared across the three streams within each multi-stream network, whose architecture follows a VGG-net-inspired convolutional and pooling design.
- The three scale-specific multi-stream networks have independently optimized parameters and merge in a final fully connected layer before six-way soft-max classification.
- Training used MILD data split into non-overlapping 75% training and 25% validation sets with matching class proportions.
- Angle-dependent class probabilities are combined to classify each nodule, with validation using 30 samples per nodule.
Nodule classification using Support Vector Machines
Classical machine-learning comparisons used intensity-based and unsupervised learned features to represent pulmonary nodules before classification.
- The classical approaches used two feature sets: 2D patch pixel intensities and representations learned automatically with K-means.
- Each 64×64-pixel patch was vectorized into a 4,096-dimensional feature vector of raw Hounsfield Unit intensities.
- The unsupervised representation adapted the Coates method by doubling the receptive field to 12 pixels and using 1,600 centroids.
Author contributions statement
The statement assigns responsibilities across experimentation, annotation, technical development, data selection, analysis, and manuscript preparation.
- F.C. conceived and conducted the experiments, analysed the results, and wrote the manuscript.
- K.C. trained students for annotating training data, reviewed annotations, and participated in the observer study.
- S.v.R. reviewed training set annotations, while E.T.S. reviewed annotations and participated in the observer study.
- A.S. and P.G. assisted with technical development of the deep learning system, and C.J. assisted in data selection.
- C.S.P. participated in the observer study.
Additional information
The additional information reports financial relationships involving two authors and MeVis Medical Solutions, as well as one author’s connection to Thirona.
- Colin Jacobs received a research grant from MeVis Medical Solutions AG, while Bram van Ginneken receives research support from MeVis Medical Solutions.
- Bram van Ginneken is co-founder and stockholder of Thirona.