Source-linked AI summary
Validation, comparison, and combination of algorithms for automatic detection of pulmonary nodules in computed tomography images: the LUNA16 challenge
Arnaud Arindra Adiyoso Setio, Alberto Traverso, Thomas de Bel, Moira S. N. Berens, Cas van den Bogaard, Piergiorgio Cerello, Hao Chen, Qi Dou, Maria Evelina Fantacci, Bram Geurts, Robbert van der Gugten, Pheng Ann Heng, Bart Jansen, Michael M. J. de Kaste, Valentin Kotov, Jack Yu-Hung Lin, Jeroen T. M. C. Manders, Alexander Sónora-Mengana, Juan Carlos García-Naranjo, Evgenia Papavasileiou, Mathias Prokop, Marco Saletta, Cornelia M Schaefer-Prokop, Ernst T. Scholten, Luuk Scholten, Miranda M. Snoeren, Ernesto Lopez Torres, Jef Vandemeulebroucke, Nicole Walasek, Guido C. A. Zuidhof, Bram van Ginneken, Colin Jacobs
TL;DR
LUNA16 addresses the limited objective comparison of pulmonary-nodule CAD systems by providing a standardized evaluation framework on 888 annotated CT scans. It evaluates individual and combined systems, finding that combining leading approaches achieved over 95.0% sensitivity at fewer than 1 FP/scan.
Problem
Objective comparison of different CAD systems on a common database has been limited, despite the need to analyze large volumes of screening CT images.
Method
LUNA16 provides a web-based standardized evaluation framework using 888 LIDC-IDRI CT scans, evaluates complete detection and false-positive-reduction tracks, and combines false-positive-reduction outputs by averaging probabilities.
Results
Combining multiple false-positive-reduction algorithms outperformed individual systems, with over 95.0% detection sensitivity at fewer than 1 FP/scan.
Takeaways & Limitations
LUNA16 supports objective benchmarking and indicates that combining complementary algorithms can improve nodule-detection performance; the challenge and additional CAD-found nodules were released for future development.
Takeaways & Limitations
Public annotations allowed teams to tune algorithms on the LIDC-IDRI data, while cross-validation could introduce positive bias because training and test data came from similar sources.
Abstract
from arXiv · showhide
Automatic detection of pulmonary nodules in thoracic computed tomography (CT) scans has been an active area of research for the last two decades. However, there have only been few studies that provide a comparative performance evaluation of different systems on a common database. We have therefore set up the LUNA16 challenge, an objective evaluation framework for automatic nodule detection algorithms using the largest publicly available reference database of chest CT scans, the LIDC-IDRI data set. In LUNA16, participants develop their algorithm and upload their predictions on 888 CT scans in one of the two tracks: 1) the complete nodule detection track where a complete CAD system should be developed, or 2) the false positive reduction track where a provided set of nodule candidates should be classified. This paper describes the setup of LUNA16 and presents the results of the challenge so far. Moreover, the impact of combining individual systems on the detection performance was also investigated. It was observed that the leading solutions employed convolutional networks and used the provided set of nodule candidates. The combination of these solutions achieved an excellent sensitivity of over 95% at fewer than 1.0 false positives per scan. This highlights the potential of combining algorithms to improve the detection performance. Our observer study with four expert readers has shown that the best system detects nodules that were missed by expert readers who originally annotated the LIDC-IDRI data. We released this set of additional nodules for further development of CAD systems.
1. Introduction
Lung cancer screening creates a large CT-reading burden, motivating CAD systems for pulmonary-nodule detection. LUNA16 addresses the limited objective comparison of existing systems by providing a common, large evaluation framework and studying algorithm combinations.
- Low-dose CT screening reduces lung cancer mortality but creates an enormous volume of images for radiologists to analyze.
- CAD systems aim to make CT interpretation faster and more accurate through preprocessing, candidate detection, and false positive reduction.
- Published CAD performances have been difficult to compare objectively because few studies evaluated systems on a common database.
- ANODE09 provided a shared evaluation set but included only 50 scans from one center and one scanner protocol, motivating a larger, more diverse database.
- LUNA16 provides 888 annotated CT scans, a web-based comparison framework, and analyses of combining candidate-detection and false-positive-reduction methods.
- Combining classical candidate detectors or deep-learning systems produced better results than individual systems, while also identifying nodules missed in the original annotations.
2. Data
The study uses a heterogeneous subset of the LIDC-IDRI database, filtered to 888 thin-slice CT scans with multi-reader annotations. Evaluation focuses on nodules at least 3 mm, while smaller nodules, non-nodules, and insufficiently agreed findings are treated as irrelevant.
- LIDC-IDRI contains 1,018 CT scans annotated in XML files by four experienced radiologists across heterogeneous institutions, scanners, and acquisition parameters.
- 888 scans remained after excluding studies with slice thickness above 3 mm, inconsistent slice spacing, or missing slices.
- Radiologists independently marked suspicious lesions as nodules ≥3 mm, nodules <3 mm, or non-nodules before reviewing the other readers’ marks without forced consensus.
- Only annotations categorized as nodules ≥3 mm were considered relevant lesions for lung-cancer-screening evaluation.
- Findings annotated by fewer than three of four radiologists, nodules <3 mm, and non-nodules were excluded from true-positive and false-positive analysis.
3. LUNA16 challenge
LUNA16 is an open challenge for evaluating pulmonary-nodule CAD systems on a shared database through standardized submissions and cross-validation. It separates complete detection from false-positive reduction and scores results with FROC-based measures.
- LUNA16 evaluates automatic pulmonary-nodule detection algorithms using a provided dataset and reference annotations in an open challenge.
- The challenge has complete nodule detection and false positive reduction tracks.
- The complete-detection track requires a full CAD system whose only input is a CT scan.
- The false-positive-reduction track supplies candidate locations for participants to classify as nodules or non-nodules.
- Participants use patient-level 10-fold cross-validation when training and testing on the LIDC-IDRI data.
- Submissions are CSV files containing each CAD mark’s image identifier, coordinates, and score, with higher scores indicating greater likelihood of a true nodule.
- Performance uses FROC sensitivity versus false positives per scan, with an overall CPM averaging sensitivity at seven predefined false-positive rates.
4. Methods
The methods section surveys the algorithms submitted to both LUNA16 tracks as of 31 October 2016. It organizes candidate-generation methods, complete detection systems, and false-positive-reduction systems for subsequent comparison.
- As of 31 October 2016, seven systems had entered the complete nodule detection track and five had entered the false positive reduction track.
- The paper describes candidate-detection algorithms used for false-positive reduction and systems submitted to the complete-detection track.
- The submitted methods are presented as the basis for evaluating algorithm performance in the LUNA16 challenge.
4.1. Candidate detection
The challenge assembled complementary candidate-detection algorithms, merged their outputs, and supplied the resulting candidates for false-positive reduction. The methods use diverse image-processing, morphology, enhancement, multi-scale, and ant-colony strategies.
- Candidate combination: Candidate lists from multiple published detectors were merged because their detections were expected to be complementary.Candidates closer than 5 mm were merged, their positions averaged, and candidates outside the lung region discarded.
- ISICAD: Shape Index and curvedness were thresholded to generate seed points, which were expanded into clusters and recursively merged within 3 voxels.The cluster center of mass became the point of interest.
- Subsolid nodules: Subsolid-nodule detection used HU values from -750 HU to -300 HU with morphological opening to address partial-volume effects.The approach targeted less common subsolid nodules described as more likely to be cancerous.
- Large solid nodules: Large-solid-nodule detection applied a -300 HU threshold and multiple morphological operations before connected-component clustering.The method targeted nodules at least 10 mm whose properties were not captured by two earlier detectors.
- Additional detectors: Other detectors used isotropic resampling, enhancement filters, divergence features, multi-scale seed detection, thresholding, and candidate merging for varied nodule contexts.These approaches addressed isolated, juxtavascular, juxtapleural, and pleural nodules.
- M5L detectors: LungCAM used ant-colony pheromone maps and iterative thresholding, while VBNA used multi-scale dot enhancement and lung-wall surface normals.Together, the approaches targeted internal and pleural nodule candidates.
4.2. Complete nodule detection system
Seven complete nodule-detection systems combined candidate generation with false-positive reduction using conventional features, classifiers, or convolutional networks. The systems included both research algorithms and commercial CAD tools.
- Overview: Seven methods were submitted to the complete nodule detection track.The section describes systems spanning candidate detection and false-positive reduction.
- ZNET: ZNET used ConvNets for both candidate detection and false-positive reduction after resampling CT images to 0.5 mm isotropic resolution.Candidate detection used a slice-based U-Net probability map followed by thresholding and connected-component analysis.
- Aidence: LUNAAidence used end-to-end ConvNets trained on National Lung Screening Trial studies with additional in-house radiologist annotations.LUNA16 was used only for validation, not training.
- JianPeiCAD: JianPeiCAD used multi-scale rule-based candidate screening and 3D ConvNets with wide channels for false-positive reduction.Data augmentation was used during training to prevent overfitting.
- Multi Opening and Threshold CAD: Multi Opening and Threshold CAD used gray-level thresholding, morphological processing, dedicated segmentation, and 15 geometric and intensity features.The system was fully automatic and included procedures for separating nodules from vascular structures.
- Visia: Visia CT Lung CAD was an FDA-approved adjunct designed to alert radiologists to solid nodules potentially overlooked during initial reading.The system was commercially available and intended for multidetector chest CT review.
- Other complete systems: ETROCAD used invariant, shape, and regional features with an RBF-kernel SVM, while M5LCAD used neural networks for LungCAM and VBNA candidates.M5LCAD classified LungCAM features with a feed-forward neural network and VBNA raw voxels with a three-layer network.
4.3. False positive reduction systems
Five false-positive reduction systems classified supplied nodule candidates using multi-context, multi-view, multi-slice, or residual convolutional architectures. Their designs addressed variation in nodule size, type, geometry, and viewpoint.
- Overview: Five methods were applied to the false positive reduction track.These systems classified supplied candidate locations rather than performing complete candidate detection.
- CUMedVis: CUMedVis used three 3D ConvNets with different receptive fields to incorporate multiple levels of contextual information.The architectures were designed for variation in nodule sizes, types, and geometry.
- Modified contextual ConvNet: A modified multi-level contextual 3D ConvNet concatenated fully connected layers from three architectures before classification.The method used exponential activation units and combined architectural outputs through concatenation rather than linear prediction fusion.
- Multi-view ConvNets: Multi-view ConvNets processed nine 65 × 65 patches from different symmetry-plane views and fused their fully connected representations late.Shared convolutional parameters allowed the streams to learn 3D characteristics by comparing view outputs.
- Multi-view training: The multi-view system used zooming and rotation augmentation, balanced positive and negative sampling, and validation based on ROC area.Training was stopped according to validation performance.
- ZNET: ZNET used wide residual networks on axial, sagittal, and coronal 64 × 64 patches, averaging the three view predictions.The network comprised four sets of convolutional layers with skip connections and global average pooling.
- Multi-slice ConvNets: The multi-slice system concatenated patches from three anatomical views into 52 × 52 × 3 mm arrays centered on each candidate.Training used augmentation and random negative patches, while testing sampled translated patches from each view.
4.4. Combining false positive reduction systems
The study combined false-positive reduction systems by averaging their predicted probabilities, enabled by their shared candidate set. This provided a direct way to evaluate ensemble performance.
- Ensemble combination: False-positive reduction outputs were combined by simply averaging the probabilities predicted by the participating systems.Because all systems used the same candidate set, the effect of combining classifiers could be evaluated directly.
4.5. Observer study
An observer study used four expert readers to examine CAD marks classified as false positives, seeking nodules missed by human readers and characterizing CAD false positives.
- Four expert readers independently assessed CAD marks from the combined false positive reduction systems.
- CAD marks were extracted at 0.25 FPs/scan for further expert analysis.
- Research scientists first removed obvious false positives, including vessels, ribs, and diaphragm findings, to reduce reader workload.
- Marks near annotated lesions that failed the hit criteria were discarded before the expert assessment.
5. Results
The results show that combining detection systems improves performance across candidate detection and false positive reduction, while complete-track and false-positive-reduction leaders achieved CPM scores of 0.811 and 0.908. Additional analyses characterized performance by nodule type and identified nodules accepted by expert readers among CAD marks.
- 5.1. Candidate detection: 98.3% sensitivity (1,166/1,186 nodules) was achieved by combining candidate detection algorithms, exceeding the 92.9% sensitivity of ETROCAD.
- 5.2. Complete nodule detection track: 0.811 CPM was the best score in the complete nodule detection track, achieved by ZNET.
- 5.3. False positive reduction track: 0.908 CPM was the best average score in the false positive reduction track, achieved by CuMedVis.
- 5.3. False positive reduction track: Combining ConvNets with different configurations further improved overall sensitivity in the false positive reduction track.
- 5.4. Performance on different nodule types: The LUNA16 data set contained 64 non-solid, 189 part-solid, and 933 solid nodules for subtype-specific performance analysis.
- 5.5. Observer study: 108 of 127 CAD marks were accepted as nodules ≥3 mm by at least one reader, while 41 were accepted by all four readers.
6. Discussion
LUNA16 shows that combining candidate-detection and false-positive-reduction algorithms can improve pulmonary-nodule detection, while highlighting evaluation and generalizability boundaries. The discussion also identifies missed nodules and remaining false-positive challenges.
- Evaluation analyses: The evaluation tables compare candidate combinations, false-positive-reduction combinations, nodule types, and reader acceptance of false positives.Table 1 and Table 2 report system combinations; Table 3 stratifies by nodule morphology; Table 4 summarizes expert-reader assessments.
- Complete nodule detection track: The complete-track systems achieved 69.1%–91.5% sensitivity at 1 and 8 FPs/scan, with ConvNets among the top three systems.Performance variability was also affected by the training data, motivating standardized training data for comparison.
- False positive reduction track: Combining false-positive-reduction ConvNets achieved over 95.0% sensitivity at fewer than 1 FP/scan.Differences in architectures, initialization, and input patches made the systems complementary despite their shared ConvNet basis.
- Observer study: Some CAD false positives were nodules missed during LIDC-IDRI annotation, particularly small or less visible nodules.Examples include ground-glass or nonsolid nodules, and early detection could provide follow-up information such as growth rate.
- Remaining challenges: Large vessels, scar tissue, spinal abnormalities, and mediastinal structures remained major sources of false positives.The combined system detected varied nodule morphologies at 1 FP/scan, while abnormalities in irregular lungs remained challenging.
- Limitations: Because LIDC-IDRI annotations were public and cross-validation was used, independent validation remains important for assessing generalizability.The open setup allowed parameter tuning on the publicly available database and introduced risks of unnoticed cross-validation errors.
7. Conclusions
The paper presents a web-based framework for fair, automated evaluation of nodule detection algorithms and finds that combining classical candidate detection with convolutional-network analysis yields excellent results. LUNA16 also updates the reference standard with additional CAD-detected nodules and remains open for future benchmarking.
- The web-based framework enables fair, automated evaluation of nodule detection algorithms using a large CT dataset annotated by multiple expert readers.
- Combining classical candidate detection algorithms with convolutional networks yields excellent results.
- LUNA16 updates the LIDC-IDRI reference standard with additional nodules found by CAD.
- LUNA16 remains open for new submissions as a benchmarking framework for future CT nodule CAD development.