Source-linked AI summary
An Interpretable Deep Hierarchical Semantic Convolutional Neural Network for Lung Nodule Malignancy Classification
Shiwen Shen, Simon X. Han, Denise R. Aberle, Alex A. T. Bui, Willliam Hsu
TL;DR
Deep-learning CADx models are difficult for radiologists to interpret and do not incorporate prior domain knowledge. The paper introduces HSCNN, jointly predicting radiologist semantic features and nodule malignancy from CT images. On LIDC, the model produces interpretable predictions and significantly improves malignancy-classification performance compared with common 3D CNN approaches.
Problem
Deep-learning CADx models are difficult to interpret and do not incorporate prior domain knowledge, limiting understanding by radiologists.
Method
HSCNN uses raw CT nodule cubes to jointly predict intermediate semantic features and a final malignancy score, training both tasks with a global loss.
Results
On LIDC, HSCNN significantly improves nodule malignancy classification compared with common 3D CNN approaches while producing interpretable predictions.
Takeaways & Limitations
Intermediate semantic predictions connect the model’s malignancy decision to radiologist-recognized nodule characteristics and can generate semantic-feature labels for unseen cases.
Takeaways & Limitations
The study omits some informative semantic labels because of LIDC labeling errors and uses radiologist suspicion levels rather than pathological diagnosis as malignancy truth.
Abstract
from arXiv · showhide
While deep learning methods are increasingly being applied to tasks such as computer-aided diagnosis, these models are difficult to interpret, do not incorporate prior domain knowledge, and are often considered as a "black-box." The lack of model interpretability hinders them from being fully understood by target users such as radiologists. In this paper, we present a novel interpretable deep hierarchical semantic convolutional neural network (HSCNN) to predict whether a given pulmonary nodule observed on a computed tomography (CT) scan is malignant. Our network provides two levels of output: 1) low-level radiologist semantic features, and 2) a high-level malignancy prediction score. The low-level semantic outputs quantify the diagnostic features used by radiologists and serve to explain how the model interprets the images in an expert-driven manner. The information from these low-level tasks, along with the representations learned by the convolutional layers, are then combined and used to infer the high-level task of predicting nodule malignancy. This unified architecture is trained by optimizing a global loss function including both low- and high-level tasks, thereby learning all the parameters within a joint framework. Our experimental results using the Lung Image Database Consortium (LIDC) show that the proposed method not only produces interpretable lung cancer predictions but also achieves significantly better results compared to common 3D CNN approaches.
1. Introduction and Background
Lung cancer screening with CT can reduce mortality but also produces substantial false positives and variable interpretation, motivating interpretable CADx methods. The paper proposes HSCNN, which combines radiologist-oriented semantic features with deep representations to predict malignancy jointly.
- Clinical motivation: 20% lung cancer mortality reduction was demonstrated by the NLST for high-risk subjects screened with low-dose CT.
- Clinical motivation: False positive rates for low-dose CT are reported as upwards of 20%, creating unnecessary medical, economic, and psychological costs.
- Limitations of conventional CADx: Segmentation-dependent handcrafted features can be inaccurate because nodule delineation and computational segmentation models vary.
- Semantic features: Diagnostic semantic features such as spiculation, lobulation, texture, and shape are intuitive to radiologists and can be characterized numerically.
- Proposed approach: HSCNN takes raw CT nodule cubes and produces intermediate semantic-feature outputs alongside a final malignancy score, using jump connections between levels.
- Proposed approach: The model is trained with a joint global cost function incorporating low- and high-level task losses, while integrating semantic and deep features for malignancy prediction.
2. Materials and Methods
The study uses LIDC-IDRI annotations and CT nodule cubes to train an HSCNN that jointly predicts radiologist semantic features and malignancy. Its architecture combines shared 3D convolutional representations with five semantic branches and a high-level malignancy classifier optimized by a global weighted loss.
- Dataset: LIDC-IDRI provides CT scans, radiologist annotations, nodule locations, and reference ratings for training and testing.The dataset includes 1,018 cases from seven academic centers and eight medical imaging companies, with annotations by four radiologists.
- Dataset selection: The study selects nodules identified by at least three radiologists on scans with slice thickness below 3 mm, yielding 4,252 nodule annotations.
- Label construction: LIDC ratings are binarized for malignancy and four semantic characteristics, while internal structure, spiculation, and lobulation are excluded for limited discrimination or annotation inconsistency.Malignancy likelihood serves as a proxy for truth because actual nodule diagnoses are unavailable.
- Preprocessing: Each candidate is represented by a normalized 40 × 40 × 40 mm CT cube centered on the nodule.CT intensities are converted to Hounsfield units and normalized to the range (0, 1).
- HSCNN architecture: The HSCNN learns shared 3D image features, predicts five semantic features, and feeds those branch representations together with shared features into malignancy prediction.The semantic tasks are margin, texture, sphericity, subtlety, and calcification.
- Feature learning: The feature-learning module uses stacked 3D convolutions, normalization, ReLU activations, and pooling to reduce spatial dimensions and control overfitting.The convolution kernels are 3 × 3 × 3, and pooling uses a 2 × 2 × 2 window with stride 2.
- Optimization: Training jointly optimizes semantic and malignancy tasks with a global loss combining weighted cross-entropy terms.Class weights are proportional to reciprocal class counts to reduce bias from imbalanced labels.
3. Experimental Results
The experiments compare HSCNN with a matched 3D CNN baseline and assess malignancy performance, semantic-feature prediction, and interpretability. HSCNN achieved significantly better malignancy AUC and produced semantic outputs that could explain representative predictions, although semantic and malignancy correctness were sometimes inconsistent.
- Malignancy Prediction Results: The baseline 3D CNN matched HSCNN’s feature-learning and high-level components but omitted the low-level semantic task component.Both models used the same four-fold cross-validation process and data splits.
- Malignancy Prediction Results: HSCNN achieved a mean AUC of 0.856 and mean accuracy of 0.842 for lung nodule malignancy classification.The supplied passage also reports mean sensitivity and specificity, but their values are truncated.
- Malignancy Prediction Results: 0.009 mean AUC improvement, with p-value 0.005 and confidence interval [0.0051, 0.0129], indicated significantly better HSCNN performance than 3D CNN.The paired t-test compared corresponding fold-level AUC scores.
- Semantic Feature Prediction Results and Model Interpretability: Semantic-feature mean AUCs were 0.930, 0.776, 0.803, 0.850, and 0.568 for calcification, margin, subtlety, texture, and sphericity, respectively.The passage also reports corresponding accuracy, sensitivity, and specificity for each feature.
- Semantic Feature Prediction Results and Model Interpretability: For representative benign and malignant nodules, predicted semantic labels aligned with malignancy predictions and provided expert-interpretable characteristics.Examples linked benign predictions to five semantic characteristics and malignant predictions to poorly defined margins, ground-glass consistency, and nonround shape.
- Semantic Feature Prediction Results and Model Interpretability: Semantic and malignancy correctness were inconsistent in representative failures: one benign nodule had four incorrect semantic predictions but correct malignancy, while another had correct semantics but incorrect malignancy.The authors note that further semantic features could augment the model.
4. Discussion
The discussion presents HSCNN as an integrated architecture that combines radiologist semantic features with learned deep features for malignancy prediction and interpretation. It also identifies omitted labels, annotation variability, nonpathological malignancy labels, binarization, and limited architecture search as constraints.
- Discussion: HSCNN integrates five semantic features—calcification, margin, subtlety, texture, and sphericity—with the primary task of malignancy diagnosis.The architecture incorporates domain knowledge into model design while predicting semantic characteristics and malignancy together.
- Discussion: Semantic labels quantify nodule characteristics in one data-driven model and map network features used for malignancy prediction to established pulmonary-nodule knowledge.The semantic outputs are presented as an interpretation mechanism for the high-level prediction.
- Discussion: Further architecture optimization was not fully searched because of limited computation power, with densely connected and residual designs proposed for future work.The authors frame these designs as potentially improving performance.
- Discussion: The study omitted size, margin spiculation, lobulation, and anatomic location despite their reported association with malignancy.Known LIDC labeling errors made lobulation and spiculation unsuitable for use.
- Discussion: Semantic labels have moderate inter-reader variability, while LIDC malignancy labels represent radiologists’ suspicion levels rather than pathological diagnosis.These annotation properties constrain interpretation of the reported results.
- Discussion: Binarizing original 5- or 6-level semantic labels may lose information, although the authors used binary labels to address data sparsity.Changing the binary threshold would also affect results.
5. Conclusion
The proposed HSCNN jointly predicts lung nodule malignancy and five radiologist semantic characteristics, using the semantic outputs to explain its diagnosis. Its shared, jointly optimized architecture improves prediction performance and interpretability and can generate semantic features.
- HSCNN simultaneously predicts malignancy and five semantic characteristics: calcification, margin, subtlety, texture, and sphericity.
- Semantic predictions serve as intermediate outputs associated with the final malignancy prediction, providing an explanation for the diagnosis.
- Jump connections incorporate low-level semantic predictions into malignancy prediction and encourage shared convolutional modules to learn features generalizable across tasks.
- A joint global loss function trains malignancy and semantic-feature prediction together within the unified model.
- Extensive experiments and statistical tests show significantly improved malignancy classification performance, while semantic predictions improve interpretability and support semantic feature generation.