PaperlayerView exact revision on arXiv

Source-linked AI summary

Blended Multi-Modal Deep ConvNet Features for Diabetic Retinopathy Severity Prediction

J. D. Bodapati, N. Veeranjaneyulu, S. N. Shareef, S. Hakak, M. Bilal, P. K. R. Maddikunta, O. Jo

arXiv:2006.00197v1eess.IVcs.CVcs.LG

TL;DR

Diabetic retinopathy diagnosis is costly and time-consuming when it relies on manual grading, while retinal-image feature quality strongly influences prediction. This work blends features from multiple pre-trained ConvNets with pooling methods and trains a DNN for DR identification and severity recognition, achieving superior performance to existing methods on APTOS 2019.

  • Problem

    Manual retinal-scan grading for DR identification and severity assessment is expensive and time-consuming, motivating automated diagnosis.

  • Method

    The approach extracts features from pre-trained ConvNets, blends them through 1-D and cross-pooling methods, and trains a DNN for DR identification and severity-level prediction.

  • Results

    The blended feature representations trained with a DNN achieve superior performance to existing methods on the APTOS 2019 benchmark for both DR identification and severity-level recognition.

  • Takeaways & Limitations

    Blending local lesion features with global image representations provides the paper’s compact retinal-image representation for DR severity prediction.

Abstract

from arXiv · show

Diabetic Retinopathy (DR) is one of the major causes of visual impairment and blindness across the world. It is usually found in patients who suffer from diabetes for a long period. The major focus of this work is to derive optimal representation of retinal images that further helps to improve the performance of DR recognition models. To extract optimal representation, features extracted from multiple pre-trained ConvNet models are blended using proposed multi-modal fusion module. These final representations are used to train a Deep Neural Network (DNN) used for DR identification and severity level prediction. As each ConvNet extracts different features, fusing them using 1D pooling and cross pooling leads to better representation than using features extracted from a single ConvNet. Experimental studies on benchmark Kaggle APTOS 2019 contest dataset reveals that the model trained on proposed blended feature representations is superior to the existing methods. In addition, we notice that cross average pooling based fusion of features from Xception and VGG16 is the most appropriate for DR recognition. With the proposed model, we achieve an accuracy of 97.41%, and a kappa statistic of 94.82 for DR identification and an accuracy of 81.7% and a kappa statistic of 71.1% for severity level prediction. Another interesting observation is that DNN with dropout at input layer converges more quickly when trained using blended features, compared to the same model trained using uni-modal deep features.

1. Introduction

DR can progress from small retinal lesions to severe vision loss, making early and affordable automated recognition important. The paper proposes blended feature representations from multiple pre-trained ConvNets and evaluates them with DNNs for DR identification and severity prediction.

  • Motivation: DR damages retinal blood vessels and can cause permanent blindness, while lesion types indicate disease severity.The paper describes five stages, from No DR to Proliferative DR, with retinal detachment and complete vision loss at the advanced stage.
  • Motivation: Manual retinal grading is expensive and time consuming because it requires human expertise for DR identification and severity assessment.Automated identification is presented as a way to make diagnosis more affordable.
  • Motivation: Existing machine-learning approaches include lesion-focused segmentation and classification methods, but many are described as sub-optimal or complex.The cited approaches target hard exudates, microaneurysms, hemorrhages, or regions of interest.
  • Proposed approach: The paper proposes uni-modal and multi-modal approaches to obtain better retinal-image representations using features from pre-trained ConvNets.The multi-modal approach blends features from models including VGG16, NASNet, Xception, and Inception ResNetV2.
  • Proposed approach: Pooling-based blending of deep features is evaluated with a DNN for DR identification and severity-level recognition.The authors report that blended representations outperform existing methods and reduce false positives.

2. Related Work

Related work progresses from conventional lesion-based and handcrafted-feature methods toward deep learning and transfer learning. The paper positions its approach as a robust model using pre-trained ConvNets and early-layer dropout for limited-data settings.

  • Conventional methods: Conventional DR methods use classifiers, ensembles, segmentation, and lesion-specific processing for identification and severity prediction.Examples include SVM, fuzzy C-means, GMM-SVM, random forests, and segmentation-based approaches.
  • Deep-learning methods: Deep-learning studies apply CNNs, data augmentation, fuzzy models, Hough transforms, and other hybrid techniques to DR severity identification.The cited literature includes a large retinal-image dataset and CNN-based approaches.
  • Deep feature methods: Deep feature methods use representations from pre-trained ConvNets and may apply PCA or SVD for dimensionality reduction.The paper notes robustness and data requirements as differing concerns across earlier model classes.
  • Feature extraction: Handcrafted features such as GIST, HOG, and SIFT provide global or local image representations for conventional machine-learning models.The paper contrasts these separate feature-learning algorithms with deep models that learn image characteristics.
  • Feature extraction: Deep learning reduces the need for separate handcrafted feature extraction, although training generally requires extensive data.Transfer learning is presented as a way to use knowledge from pre-trained CNNs when application datasets are limited.
  • Position of this work: The proposed model addresses limited data and computational resources by extracting features with pre-trained ConvNets and training a DNN with early-layer dropout.The approach uses transfer learning to obtain prominent fundus-image features.

3. Proposed Methodology

The proposed methodology extracts retinal-image features from multiple pre-trained ConvNets, blends them through pooling-based fusion, and trains DNNs for DR identification and severity recognition.

  • Overall architecture: The pipeline comprises feature extraction, model training, and evaluation modules for automated DR recognition.Retinal images are processed into deep features, used to train DNN classifiers, and evaluated on test data.
  • Uni-modal feature extraction: Uni-modal recognition uses features from one pre-trained ConvNet as input to a DNN for DR identification or classification.The uni-modal pipeline extracts final-layer features and feeds them to a DNN with dropout at the input layer.
  • Blended multi-modal feature extraction: Multi-modal recognition blends features from multiple ConvNets because their differing architectures and training data provide complementary retinal-image representations.The blended features are used for both DR identification and severity recognition.
  • Feature fusion: The fusion module applies 1-D pooling to VGG16 fc1 and fc2 features, then cross pooling to combine the pooled representation with Xception features.1-D pooling selects prominent local features, while cross pooling aggregates them with Xception’s global representation.
  • Model training and evaluation: The fused features train DNNs using two hidden layers for binary DR identification and three hidden layers for severity classification, with 0.2 input dropout.The identification network uses 256 and 128 ReLU units; the severity network uses 512, 256, and 128 ReLU units.

4. Experimental Results

The experiments evaluate blended feature representations on the APTOS 2019 retinal-image benchmark, whose images are manually graded by five DR severity levels. The dataset contains varied imaging conditions and an imbalanced class distribution, with 80% used for training and 20% for validation.

  • Dataset: The experiments use the APTOS 2019 Kaggle benchmark dataset from a blindness-detection challenge.The dataset contains retinal images captured using fundus photography under varied imaging conditions.
  • Dataset: Images are manually graded from 0 to 4, representing No, Mild, Moderate, Severe, and Proliferative DR.
  • Dataset: The dataset is imbalanced, with more normal images and very few images in class 3.
  • Experimental setup: The experiments use 80% of the data for training and the remaining 20% for validation.

4.2. Performance Measures:

Performance is assessed with classification metrics and the Kappa statistic, which compares observed accuracy with expected accuracy. Observed accuracy counts correct classifications, while expected accuracy reflects class and prediction-label distributions.

  • Metrics: The evaluation reports Accuracy, Precision, Recall, F1 Score, and Kappa statistic.
  • Kappa statistic: Kappa statistic compares observed accuracy with expected accuracy.
  • Kappa statistic: Observed accuracy is the number of correctly classified samples, whereas expected accuracy depends on class and correctly matched prediction counts.

4.3. DR Identification and Severity level Prediction:

The study separates DR presence detection from five-level severity prediction. Task 1 uses binary classification after merging all DR-positive classes, while task 2 predicts one of five severity levels.

  • Task definitions: The experiments comprise two tasks: identifying DR presence or absence and predicting severity level.
  • Task 1: DR identification: Task 1 merges all DR-affected images into one positive class and retains normal images as the negative class.The resulting dataset contains 1857 positive and 1805 negative images.
  • Task 1: DR identification: Task 1 uses binary cross entropy loss and the Adam optimizer for binary DR identification.
  • Task 2: Severity prediction: Task 2 categorizes each retinal image into one of five severity levels using categorical cross entropy loss and Adam optimization.

4.4. Experimental studies to show the representative nature of uni-modal features for task1

Uni-modal experiments compare machine-learning classifiers trained on features from several pre-trained ConvNets. DNN performs best across the compared classifiers, while feature source affects performance across accuracy, agreement, precision, recall, F1, loss, and convergence.

  • Experimental design: Uni-modal features are extracted from VGG16, Xception, NASNet, and Inception ResNetV2 for DR identification experiments.
  • Classifier comparison: DNN outperforms the other evaluated machine-learning models when using VGG16 and Xception features.This finding motivates using DNN alone in the remaining experiments.
  • Feature-source comparison: Xception features yield better accuracy for DR identification, although accuracy and Kappa differences among uni-modal feature sources are nominal.
  • Convergence comparison: VGG16-fc1 reaches the minimum loss, while Xception converges in 16 epochs and Inception ResNetV2 achieves the strongest reported performance in the convergence comparison.
  • Task 1 summary: VGG16-fc1 provides better Kappa scores, VGG16-fc2 provides better precision, recall, and F1 scores, and DNN consistently outperforms other classifiers especially on Kappa.The paper attributes the strong VGG16 and Xception results to lesion information useful for discriminating DR-positive from unaffected images.

4.5. Experimental studies to show the representative nature of uni-modal features for task2

Task2 severity prediction is more challenging because it involves multiple classes. Among uni-modal features, VGG16-fc1 provides the strongest performance, while Xception converges faster.

  • Task2 is more challenging than task1 because severity prediction involves multiple classes.
  • VGG16-fc1 features show superior severity-prediction performance among the evaluated uni-modal models.This superiority is reported across all metrics.
  • VGG16-fc1 yields superior performance with minimum loss among the pre-trained features.
  • Xception converges in fewer epochs than the other evaluated pre-trained feature models.

4.6. Performance evaluation of the proposed blended multi-modal features

The study evaluates pooling-based fusion of multiple deep features for DR identification and severity prediction. Average pooling provides the strongest blended representation and faster convergence across both tasks.

  • Different uni-modal features are blended because they extract complementary feature sets from retinal images.
  • The fusion experiments combine VGG16 fully connected features and extend the representation to include Xception features.
  • Maximum, sum, and average pooling are evaluated as alternative methods for blending multiple deep features.
  • Task1: Average pooling produces superior DR-detection performance, faster convergence in fewer than 50 epochs, and minimum loss relative to uni-modal features.
  • Task2: Average pooling gives the best severity-prediction fusion, with the three-feature combination outperforming the VGG16-fc1 and VGG16-fc2 combination.
  • Task2: For severity prediction, average pooling converges faster and achieves better accuracy and kappa scores than the other fusion approaches.

4.7. Comparison of proposed Blended feature extraction with existing methods

The proposed blended-feature DNN is compared with existing DR-prediction models and uses complementary local and global representations from VGG16 and Xception. It achieves 80.96% accuracy for severity prediction, although proliferative DR is often classified as moderate.

  • 80.96% accuracy is achieved by the proposed blended-feature DNN for DR severity prediction, outperforming existing literature models.
  • The proposed DNN uses three hidden layers with 512, 256, and 128 units each.
  • Most proliferative DR images are misclassified as moderate in the severity-prediction confusion matrix.
  • VGG16 captures lesion information while Xception represents the entire retinal scan, yielding a compact combined representation.

5. Conclusion

The work develops a compact retinal-image representation by pooling features from VGG16 and Xception, then trains a DNN with input dropout for severity prediction. On APTOS 2019, average pooling gives better performance with minimum loss in fewer epochs than the other pooling approaches.

  • The final representation pools VGG16-fc1, VGG16-fc2, and Xception features to combine lesion and global image information.
  • A DNN with dropout at the input layer is trained on the blended features for diabetic retinopathy severity-level prediction.
  • Experiments on the benchmark APTOS 2019 dataset show superiority over existing models.
  • Average pooling of penultimate-layer features gives better performance with minimum loss in fewer epochs than the other pooling approaches.
Loading 2006.00197v1…