PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep image mining for diabetic retinopathy screening

Gwenolé Quellec, Katia Charrière, Yassine Boudi, Béatrice Cochener, Mathieu Lamard

arXiv:1610.07086v3cs.CV

TL;DR

The paper addresses how to detect lesions from image-level diagnoses while making ConvNet decisions more interpretable. It uses modified sensitivity heatmaps and training-time sparsity optimization, achieving strong referable-DR detection and lesion-detection performance across retinal datasets. The method still produces architecture-related false alarms and can be further improved with expert segmentations or richer labels.

  • Problem

    ConvNets can classify retinal images from diagnosis labels without manual segmentations, but their local lesion evidence is difficult to interpret and existing heatmaps contain artifacts.

  • Method

    The method uses ConvNets trained for image-level referable-DR detection, modified sensitivity analysis to generate image-sized heatmaps, and sparsity optimization to improve heatmap quality.

  • Results

    The detector outperformed prior lesion-specific algorithms at image level except for red lesions and outperformed recent heatmap methods for every lesion type at lesion level.

  • Takeaways & Limitations

    Diagnosis-only supervision supports lesion detection without expert knowledge or segmentations and may enable image mining for new biomarkers.

  • Takeaways & Limitations

    Sensitivity-based heatmaps can amplify nearby confounders into drafting artifacts, and final performance benefited from expert segmentations used for fine-tuning.

Abstract

from arXiv · show

Deep learning is quickly becoming the leading methodology for medical image analysis. Given a large medical archive, where each image is associated with a diagnosis, efficient pathology detectors or classifiers can be trained with virtually no expert knowledge about the target pathologies. However, deep learning algorithms, including the popular ConvNets, are black boxes: little is known about the local patterns analyzed by ConvNets to make a decision at the image level. A solution is proposed in this paper to create heatmaps showing which pixels in images play a role in the image-level predictions. In other words, a ConvNet trained for image-level classification can be used to detect lesions as well. A generalization of the backpropagation method is proposed in order to train ConvNets that produce high-quality heatmaps. The proposed solution is applied to diabetic retinopathy (DR) screening in a dataset of almost 90,000 fundus photographs from the 2015 Kaggle Diabetic Retinopathy competition and a private dataset of almost 110,000 photographs (e-ophtha). For the task of detecting referable DR, very good detection performance was achieved: $A_z = 0.954$ in Kaggle's dataset and $A_z = 0.949$ in e-ophtha. Performance was also evaluated at the image level and at the lesion level in the DiaretDB1 dataset, where four types of lesions are manually segmented: microaneurysms, hemorrhages, exudates and cotton-wool spots. The proposed detector outperforms recent algorithms trained to detect those lesions specifically, as well as competing heatmap generation algorithms for ConvNets. This detector is part of the Messidor system for mobile eye pathology screening. Because it does not rely on expert knowledge or manual segmentation for detecting relevant patterns, the proposed solution is a promising image mining tool, which has the potential to discover new biomarkers in images.

1. Introduction

DR screening aims to identify patients needing ophthalmologist referral while reducing human interpretation workload. Earlier systems required manually segmented lesions, whereas newer image-level approaches use diagnoses alone and enable larger-scale training.

  • Early DR diagnosis can slow disease progression and help prevent blindness by identifying patients needing ophthalmologist referral.
  • Earlier computer-aided detection algorithms learned pixel-level lesions from manually segmented images, followed by image-level pathology diagnosis.
  • Multiple-instance learning and deep-learning methods can learn pathology detection directly from image-level diagnoses without manual segmentations.
  • Diagnosis-only supervision permits training on much larger datasets, including anonymized examination archives, and may support discovery of new image biomarkers.
  • The paper’s secondary objective is to use local lesion detections to improve DR detection at the image level.

2. State of the Art

Deep learning has been applied to retinal segmentation, lesion detection, and image-level pathology classification, but visualization methods differ in lesion coverage and spatial resolution. This paper motivates optimized heatmaps that handle multiple and small lesions while reducing ConvNet artifacts.

  • 2.1. Deep Learning for Retinal Image Analysis: Deep learning has been used for retinal landmark segmentation and lesion detection at the pixel level, and pathology detection at the image level.
  • 2.1. Deep Learning for Retinal Image Analysis: Prior retinal applications include pixel-level detection of vessels, optic discs, microaneurysms, hemorrhages, exudates, and other lesions.
  • 2.1. Deep Learning for Retinal Image Analysis: Other deep-learning systems classify glaucoma, AMD, ROP, and referable DR directly or through segmented anatomical structures.
  • 2.2. Visualizing what ConvNets are Learning: Visualization methods can produce heatmaps the size of the input image, unlike region-highlighting methods that may lack pixel-level information for small lesions.The cited visualization approach increased map definition from 7 × 7 to 31 × 31 pixels but still could not provide pixel-level information.
  • 2.2. Visualizing what ConvNets are Learning: The proposed task requires handling multiple occurrences and small lesions, while addressing artifacts in heatmaps generated by existing visualization methods.
  • 2.2. Visualizing what ConvNets are Learning: Sensitivity analysis was selected because it can be differentiated for joint optimization of ConvNet predictions and heatmap quality.

3. Heatmap Generation

The paper defines pixel-level sensitivity heatmaps for ConvNet predictions and modifies the criterion to preserve hue, while addressing disguise and drafting artifacts that can create false alarms.

  • 3.2. Sensitivity Criterion: The sensitivity criterion measures how a pixel’s color-channel value affects the ConvNet’s image-level prediction for a selected class.It uses the absolute partial derivative of the class prediction with respect to one pixel color channel, computed through the network by the chain rule.
  • 3.2. Sensitivity Criterion: For referable DR, each pixel’s overall contribution is aggregated across color channels to identify local changes that would modify the prediction.Lesions should receive nonzero contributions because removing them could downgrade a positive diagnosis, while enhancing them could strengthen it.
  • 3.4. Disguise Artifacts: The criterion can produce disguise artifacts because it indicates prediction-increasing directions in RGB space rather than whether a pixel supports or opposes the prediction.Confounders and healthy tissue near lesions may therefore receive nonzero contributions when color changes make them resemble or enlarge lesions.
  • 3.5. Hue-Constrained Sensitivity Criterion: Hue-constrained sensitivity forces the three color components of each pixel to change by the same factor, focusing analysis on pattern enhancement or attenuation.A mask tensor with one channel is multiplied entrywise with the input, so all color components at a pixel share the same multiplier.
  • 3.6. Drafting Artifacts: A second limitation is drafting artifacts: downsampling causes nearby confounders to share derivative terms with true lesions, artificially boosting their perceived influence.These artifacts affect backpropagation-based methods, including deconvolution and layer-wise relevance propagation.
  • 3.6. Drafting Artifacts: A brute-force artifact-reduction strategy repeatedly records the largest sensitivity and masks that pixel before recomputing the heatmap.This seeks to measure each pixel independently of more influential neighbors, but requires repeated sensitivity computations.

4. Heatmap Optimization

The paper adds sparsity regularization to reduce drafting artifacts in heatmaps and introduces backward-forward propagation to optimize ConvNets whose loss depends on backpropagated quantities.

  • 4.2. Sparsity-Enhanced Sensitivity Criterion: An additional regularization term L0 maximizes heatmap sparsity while preserving classification or regression accuracy, encouraging filters that separate lesions from nearby confounders.The method uses L1-norm minimization as a tractable surrogate for L0-norm minimization.
  • 4.3. Backward-Forward Propagation Method: Standard backpropagation cannot optimize these networks because the heatmaps depend on backpropagated quantities.The proposed procedure therefore adds a forward pass that propagates second-order derivatives.
  • 4.3. Backward-Forward Propagation Method: The backward-forward training iteration computes image-level loss by forward propagation, its parameter gradients by backward propagation, and the regularization gradient by forward propagation through first-order derivatives.Each parameter tensor is then updated using the combined gradients.
  • 4.3. Backward-Forward Propagation Method: The proposed algorithm can be implemented with deep learning libraries using automatic differentiation, but each network operator needs a forward second-order derivative.The paper provides these derivatives for the operators used in the experiments.
  • Experiments: The heatmap generation and optimization method is evaluated for diabetic retinopathy screening with task-specific ConvNets.These ConvNets produce image-level predictions and, after the proposed modifications, pixel-level predictions.

5. Experiments

Experiments adapt competition-derived ConvNets to produce image- and pixel-level predictions, evaluating them across Kaggle, DiaretDB1, and e-ophtha datasets. Visualization results identify drafting artifacts, while lesion-level evaluation highlights substantial false alarms for subtle lesions.

  • 5.1. ConvNets: The baseline ConvNets are adapted from successful 2015 Kaggle competition solutions and produce both image-level and pixel-level predictions.The study focuses particularly on ‘net B’, while also evaluating AlexNet for generality.
  • 5.3. Datasets: The experiments use Kaggle for image-level training and testing, DiaretDB1 for lesion- and image-level testing, and e-ophtha for image-level testing.The datasets contain 88,702 Kaggle photographs, 89 DiaretDB1 photographs, and 107,799 e-ophtha photographs.
  • 5.4. Visualization Artifacts: Hue-constrained sensitivity maps prevent lesion detections from growing, whereas drafting effects cause nearby blood vessels to appear as false detections.These false detections are not necessarily connected to true lesions and therefore cannot be removed easily through image post-processing.
  • 5.5. Image- and Pixel-Level Performance: Image-level performance is assessed with ROC analysis using Az, while lesion-level performance in DiaretDB1 is assessed with free-response ROC analysis.Figure 5 tracks ‘net B’ across training checkpoints and compares hue-constrained sensitivity with and without sparsity maximization.
  • 5.5. Image- and Pixel-Level Performance: False alarms are particularly numerous for microaneurysm detection because experts primarily segmented obvious lesions while screening algorithms also target subtle lesions.Consequently, many true subtle lesions are counted as false alarms in the evaluation.

6. Discussion and Conclusions

The proposed framework uses image-level supervision and optimized ConvNet heatmaps to detect referable diabetic retinopathy and lesions, achieving strong performance across retinal datasets while supporting mobile screening.

  • Proposed framework: Image-level labels supervise ConvNets that generate heatmaps for detecting referable DR and related lesions.A modified sensitivity analysis identifies pixels contributing to image-level predictions, while sparsity enhancement reduces heatmap artifacts.
  • Lesion detection: Lesion detection at the image level outperformed previous pixel-supervised algorithms for all lesions or lesion groups except ‘red lesions’.The ConvNets were evaluated on DiaretDB1 without retraining after training on Kaggle-train.
  • Lesion detection: All lesion types achieved higher lesion-level performance than recent competing heatmap-generation algorithms.Performance was summarized using the limited FROC area in Table 3.
  • Heatmap quality: Sparsity maximization strongly reduced false alarms on vessels near true lesions, improving the quality of the produced heatmaps.The comparison involved unoptimized π0 maps and optimized πHE, πRDR, and averaged maps.
  • Practical implications: Very good detections in blurry images from a low-cost handheld retinograph support automated mobile screening, while lesion-level performance depends on strong image-level ConvNets.AlexNet produced poor lesion-level results despite heatmap optimization.
  • Referable DR detection: Az = 0.9542 was achieved on Kaggle-test with ‘net B’, outperforming the main baseline at Az = 0.9512.The reported 95% confidence interval for the Kaggle result was [0.9515, 0.9569].
  • Referable DR detection: Az = 0.9490 was achieved on e-ophtha-test, compared with Az = 0.8440 for the previous solution based partly on multiple-instance learning.Extending the ensemble to multiple network structures did not significantly improve performance and increased complexity.
  • Scope and limitations: The framework is general for multiple-instance problems governed by a limited number of relevant pixels and requires referral decisions rather than expert knowledge or segmentations.Expert segmentations nevertheless helped fine-tune and further improve performance.

Appendix A. Popular ConvNet Operators

The appendix introduces the ConvNet operators used in the evaluated networks and points to derivative specifications in Appendix B.

  • Popular ConvNet Operators: The appendix describes the operators used by the ConvNets evaluated in this paper.Their backward first-order and forward second-order derivatives are provided in Appendix B.

Appendix A.1. Convolutional Layer (Conv)

A convolutional layer computes activation-map values by cross-correlating local input windows with learned filters, adding bias, and applying a nonlinear activation.

  • Convolutional Layer: Each output activation is computed from a sliding-window cross-correlation between the previous activation maps and learned filters.The filters have dimensions wl × hl × Cl−1, and the window offset is controlled by stride sl.
  • Convolutional Layer: A bias is added to the filter product before the nonlinear activation function is applied.Biases may be tied or untied across activation-map locations.

Appendix A.2. Activation Functions

The appendix describes rectifier and leaky-rectifier activations as computationally efficient alternatives to sigmoid functions, with α controlling leakage.

  • Activation Functions: Rectifiers and leaky rectifiers are computationally less intensive than traditional sigmoid activations.The leaky-rectifier parameter α is usually small, commonly 0.01 or 0 for the original rectifier.
  • Activation Functions: Large α values, such as 0.33, define ‘very leaky rectifiers’.The parameter α lies in [0, 1[.

Appendix A.3. Dense or Fully-Connected Layers (Dense)

Dense layers are a special case of convolutional layers, with filters spanning the full spatial dimensions of the input.

  • Dense layers use filters whose width and height equal the preceding layer’s spatial dimensions.This makes the output tensor spatially 1 × 1.

Appendix A.4. Pooling Layer (MaxPool, MeanPool or RMSPool)

Pooling layers summarize values within sliding windows, optionally down-sampling the data, while preserving the number of activation maps.

  • Pooling layers replace each sliding-window region with a single output value.Maximum, mean, and root mean square are typical pooling operators.
  • A stride greater than 1 performs down-sampling during pooling.
  • MaxPool and MeanPool are defined by aggregating values over each sliding-window neighborhood.The supplied passages introduce their respective computations but do not include the displayed formulas.
  • Pooling leaves the number of activation maps unchanged.The number of maps satisfies C_l = C_{l−1}.

Appendix A.5. Dropout and Maxout Layers

Dropout temporarily removes selected filters and connections during training, while the derivative treatment applies the ConvNet’s layer-specific backpropagation rules to the thinned network and maximum-based operators.

  • Appendix A.5. Dropout and Maxout Layers: Dropout randomly drops filters from the next layer during each training iteration by temporarily removing their input and output connections.The aim is to train multiple thinned versions of the ConvNet.
  • Appendix A.5. Dropout and Maxout Layers: Maxout returns the maximum output among subsets of filters and can be placed after dropout for improved performance.
  • Appendix B.1. Forward Second-Order Derivatives for Cross-Correlation: Cross-correlation requires separate backward derivatives because it has both data and filter-weight inputs.The relevant inputs are D^(l−1) and Ω^(l).
  • Appendix B.1. Forward Second-Order Derivatives for Cross-Correlation: In the backward first-order derivative, cross-correlation becomes a convolution between the filter weights and the derivative with respect to the layer output.
  • Appendix B.1. Forward Second-Order Derivatives for Cross-Correlation: The loss gradient with respect to filter weights is a cross-correlation between the output derivative and the input data.During the forward pass, the corresponding input derivative plays the role of the input data.
  • Appendix A.5. Dropout and Maxout Layers: Dropout uses the same first- and second-order derivative rules on the temporarily thinned network, while Maxout is handled like other maximum-based operators.These operators include leaky rectifiers and MaxPool.
Loading 1610.07086v3…