PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Learning Ensembles for Melanoma Recognition in Dermoscopy Images

Noel Codella, Quoc-Bao Nguyen, Sharath Pankanti, David Gutman, Brian Helba, Allan Halpern, John R. Smith

arXiv:1610.04662v2cs.CV

TL;DR

Early melanoma detection depends on highly trained specialists, whose recognition performance drops without adequate training. This paper combines deep and classical machine-learning methods in ensembles for dermoscopic lesion segmentation and classification, achieving state-of-the-art performance, including an AUC of 0.843 versus 0.783 previously.

  • Problem

    Accurate early melanoma detection requires highly trained specialists, while recognition performance drops significantly when clinicians are inadequately trained.

  • Method

    The system ensembles deep residual, convolutional, fully convolutional, sparse-coding, and hand-coded feature methods for lesion segmentation and melanoma classification.

  • Results

    The system achieved state-of-the-art performance across evaluation metrics, including an AUC of 0.843 versus 0.783 previously.

  • Takeaways & Limitations

    Combining multiple deep-learning and classical computer-vision techniques can outperform systems relying on a single method or deep learning alone.

  • Takeaways & Limitations

    Comparisons lacked statistical significance because evaluations used a fixed dataset partition and unavailable comparison implementations prevented repeated n-fold evaluations.

Abstract

from arXiv · show

Melanoma is the deadliest form of skin cancer. While curable with early detection, only highly trained specialists are capable of accurately recognizing the disease. As expertise is in limited supply, automated systems capable of identifying disease could save lives, reduce unnecessary biopsies, and reduce costs. Toward this goal, we propose a system that combines recent developments in deep learning with established machine learning approaches, creating ensembles of methods that are capable of segmenting skin lesions, as well as analyzing the detected area and surrounding tissue for melanoma detection. The system is evaluated using the largest publicly available benchmark dataset of dermoscopic images, containing 900 training and 379 testing images. New state-of-the-art performance levels are demonstrated, leading to an improvement in the area under receiver operating characteristic curve of 7.5% (0.843 vs. 0.783), in average precision of 4% (0.649 vs. 0.624), and in specificity measured at the clinically relevant 95% sensitivity operating point 2.9 times higher than the previous state-of-the-art (36.8% specificity compared to 12.5%). Compared to the average of 8 expert dermatologists on a subset of 100 test images, the proposed system produces a higher accuracy (76% vs. 70.5%), and specificity (62% vs. 59%) evaluated at an equivalent sensitivity (82%).

Introduction

Melanoma requires early detection, yet accurate dermoscopic diagnosis depends on highly trained clinicians whose expertise is unevenly available. This work addresses that gap by combining classical and deep learning methods into ensembles for melanoma recognition and lesion segmentation using the ISBI 2016 dataset.

  • Motivation: Melanoma causes approximately 100,000 new U.S. cases and over 9,000 deaths annually, while skin cancer costs the U.S. healthcare system more than $8 billion.Skin cancer exceeds 5 million U.S. cases each year.
  • Motivation: 75-84% diagnostic accuracy has been demonstrated by specially trained clinicians using dermoscopy, making early detection dependent on specialized expertise and equipment.Dermoscopy reduces skin-surface reflectance and reveals deeper underlying structures.
  • Motivation: 340 registered dermatologists in Australia and 16 in New Zealand illustrate the limited supply of expert consultation outside the United States.Restricted access to experts creates challenges in providing adequate care to at-risk populations.
  • Related work: Earlier deep learning work achieved state-of-the-art performance on over 2,000 dermoscopy images but required lesions to be manually pre-segmented.The work combined representations learned from natural photos with unsupervised and hand-coded features.
  • Contribution: The proposed system combines hand-coded feature extractors, sparse-coding methods, SVMs, deep residual networks, and fully convolutional neural networks into ensembles for melanoma recognition and segmentation.Evaluation uses the ISBI 2016 dataset, enabling comparison with prior algorithms and future studies.

Dataset

The study uses the ISIC dataset from the ISBI 2016 “Skin Lesion Analysis toward Melanoma Detection” challenge, with annotated training and held-out test images. It covers lesion segmentation and classification, while dermoscopic feature extraction is reserved for future work.

  • Dataset: 900 annotated dermoscopic images, including 173 melanomas, were used for training, while 379 held-out images, including 75 melanomas, formed the test set.The dataset was released by the International Skin Imaging Collaboration for the ISBI 2016 challenge.
  • Lesion Segmentation: Segmentation outputs were binary masks, with lesion pixels assigned 255 and background skin pixels assigned 0.Participants were ranked by the Jaccard index (JACC), with pixel-wise accuracy (ACC), sensitivity (SENS), and specificity (SPEC) also reported.
  • Study scope: The paper addresses Part 1, Lesion Segmentation, and Part 3, Lesion Classification, of the ISBI challenge.The challenge also included Part 2, Lesion Dermoscopic Feature Extraction.
  • Study scope: Part 2, including the potential influence of dermoscopic feature extraction on disease detection, was left for a future study.The paper therefore excludes dermoscopic feature extraction from its scope.

Visual Recognition System

The visual recognition system has two primary components: segmentation and classification. Segmentation identifies the skin lesion and enables subsequent analysis within the lesion and across the entire image.

  • System components: The system consists of segmentation and classification components.Segmentation identifies the lesion and distinguishes it from healthy background skin.
  • Segmentation: Segmentation identifies the skin lesion in the dermoscopic image.It distinguishes the lesion from background healthy skin.
  • Analysis contexts: The system analyzes two contexts: the lesion itself and the entire image.The lesion context covers potentially diseased tissue, while the whole-image context includes surrounding areas.

Segmentation

The segmentation system used a fully convolutional network similar to U-Net to map dermoscopic images to lesion masks. Its design combined skip connections, six-channel color inputs, augmentation, and validation-based training procedures.

  • Model architecture: The model was a fully convolutional network, similar to U-Net, that learned a mapping from each input image to a lesion segmentation mask.The implementation used Theano, Lasagne, and Nolearn packages.
  • Model architecture: Skip connections linked pre-pooling convolutional data to deconvolution layers, providing higher-resolution information and modeling functional residuals.These connections were intended to improve performance compared with networks without skip connections.
  • Model architecture: The architecture used three convolution-and-pooling stages before a fully connected layer, followed symmetrically by three unpooling-and-deconvolution stages.Each pre-fully-connected stage contained three convolution operations followed by one pooling operation.
  • Input processing: Six color channels combining RGB and HSV were used, with images resized to 128-by-128 dimensions before network input.Training-data performance improved empirically with six rather than three color channels.
  • Input processing: Training batches used rotations, flips, rescaling, shifts, crops, and sinusoidal non-linear distortions to augment dermoscopic images.The non-linear distortions were motivated as a way to model variation in soft-tissue biological structures.
  • Training and evaluation: 80% (720) of the segmentation data was used for training and 20% (180) for validation, with early stopping and best-validation-weight saving.A grid search selected the network configuration minimizing validation loss before final test-set segmentation.

Classification

The classification system ensembles recent deep-learning and established machine-learning methods for melanoma recognition. It uses automated segmentation and multi-contextual analysis to improve on previous literature methods.

  • Methods: The ensemble combines deep residual networks, convolutional neural networks, and fully convolutional U-Net architecture.It also includes sparse coding and hand-coded feature representations.
  • Methods: Established methods include sparse coding and hand-coded feature representations alongside the deep-learning models.
  • Advantages: Compared with previous literature methods, the approach uses automated segmentation and multi-contextual analysis.
  • Feature extraction: The approach extracts features from dermoscopic images using the described classification methods.A visual overview is provided in Figure 3.

Hand-coded Feature Extraction

The system combined hand-crafted visual descriptors with learned representations for melanoma recognition and lesion segmentation. Features included color, edge, multiscale color LBP, sparse-coding, deep residual, and U-Net shape descriptors.

  • Hand-crafted visual features: Hand-crafted features comprised HSV color histograms, Sobel-based edge histograms, and multiscale color LBP descriptors.These feature types had previously been used in high-performing medical and dermoscopic image systems.
  • Hand-crafted visual features: 166 dimensions described the HSV color histogram, while the Sobel-based edge histogram used 64 dimensions.The edge histogram contained 8 direction bins and 8 magnitude bins.
  • Sparse coding: SPAMS sparse coding was employed to learn dictionaries using an efficient online optimization algorithm based on stochastic approximations.The method was selected because it had been used in prior state-of-the-art melanoma recognition systems.
  • Deep learned descriptors: 101 layers characterized the Deep Residual Network used to extract a 1000-dimensional whole-image concept detector vector.The feature was extracted from dermoscopy images at the whole-image context level.
  • U-Net shape descriptor: 1024 dimensions limited the U-Net fully connected layer used as a compact shape descriptor alongside its lesion-segmentation role.An independent U-Net network was trained, and the layer was extracted for every dataset image.

Experimental Results · Segmentation

Segmentation experiments compared an optimized single U-Net with an ensemble of 10 independently trained networks using varied architectures and dropout settings. The approach achieved competitive performance, agreement with ground truth within the range of human experts, and supported subsequent disease classification.

  • Segmentation: The U-Net Ensemble comprised 10 independently trained networks with varied network topology and dropout parameters.The major topology and dropout parameters were configured at least two different ways across the networks.
  • Segmentation: Segmentation residual error was partly associated with variability in the ground truth annotations.Contributing factors included intrinsic differences in human annotation, low contrast, hair, and visual adjacency of lesions to surrounding skin.
  • Segmentation: 0.786 was the average Jaccard index among three clinical experts segmenting 100 held-out test images.The three pairwise Jaccard indices were 0.743, 0.754, and 0.861.
  • Segmentation: The segmentation network produced competitive state-of-the-art performance and agreement with ground truth within the range of human experts.These results were considered satisfactory for subsequent disease classification processing steps.
  • Segmentation: The network served both as a segmenter and as a visual descriptor of lesion shape.Outputs from its most compressed fully connected layer were saved for this descriptor role.

Classification

The proposed classification system improved average precision, AUC, and clinically relevant specificity over the prior state of the art, with gains especially pronounced at high-sensitivity operating points. It also outperformed dermatology experts on a 100-image subset, while component-selection algorithms reduced held-out performance and removing the DRN concept detector decreased average precision.

  • Performance versus prior state of the art: 7.7% relative AUC improvement was achieved for Part 3B, increasing AUC from 0.783 to 0.843 over the prior state of the art.Part 3 showed a 4.2% improvement, from 0.804 to 0.838.
  • Performance versus prior state of the art: 4% average-precision improvement was achieved for Part 3B, increasing performance from 0.624 to 0.649 over the prior state of the art.Part 3 showed a 1.3% improvement, from 0.637 to 0.645.
  • Performance versus prior state of the art: 36.8% specificity at 95% sensitivity represented a 194.4% relative improvement over the prior state of the art’s 12.5% for Part 3B.Part 3 specificity improved by 43.6%, from 22.7% to 32.6%.
  • Segmentation context: 0.649 average precision using ground-truth segmentations was only slightly higher than 0.645 using automatically generated segmentations.The prior state of the art instead achieved 0.637 with automatically generated masks versus 0.624 with ground-truth segmentations.
  • ROC analysis: Performance improvements occurred across multiple ROC operating points, particularly those corresponding to the highest sensitivity levels.These high-sensitivity operating points were identified as clinically the most relevant system operating points.
  • Ensemble component selection: 0.638 and 0.646 average precision from greedy model selection and 0.614 and 0.612 from forward model selection showed that both selection algorithms reduced held-out test performance.The values correspond respectively to Parts 3 and 3B.
  • DRN concept detector contribution: 0.645 to 0.632 and 0.649 to 0.633 average-precision decreases followed removal of the 1000-dimension DRN concept detector vector.The first comparison used automatically generated segmentations; the second used ground-truth segmentations.

Discussion

The discussion highlights that clinical utility may not be fully captured by average precision, while ensembles, complex augmentation, and ImageNet-derived concept features contributed meaningfully. The study also notes limitations from fixed data partitioning, unavailable comparison implementations, and limited statistical significance assessment.

  • Evaluation methodology: Average precision may not fully reflect a system’s value to clinical workflow, despite being useful for retrieval-based applications.The challenge organizers ranked participants using average precision, but the authors question its adequacy for clinical evaluation.
  • Algorithm design: Ensembling multiple techniques remained valuable despite the field’s focus on optimizing increasingly powerful deep-learning network structures.The authors specifically describe ensemble-based algorithm design as a fundamental insight from their results.
  • Data augmentation: 543,888,390 parameters were trained using only 900 examples in the fully convolutional U-Net, supporting complex non-linear augmentation for limited datasets.The discussion presents this as evidence that augmentation can help large neural networks train with few examples.
  • Transfer learning: 1,000 ImageNet concept-detector outputs provided meaningful diagnostic information from dermoscopic images when used as a descriptive image vector.The authors state that this finding represents the second published report showing positive results with such a technique.
  • Limitations: Fixed dataset partitioning and unavailable comparison implementations limited statistical significance assessment through repeated n-fold evaluations.The fixed partition was required by the public challenge’s held-out blind test design.

Conclusion and Future Work

The proposed system performs melanoma lesion segmentation and classification on dermoscopic images, achieving new state-of-the-art benchmark performance. Future work may combine dermoscopic pattern information with disease classification and explore nonlinear image warping for augmentation.

  • Conclusion: The proposed system performs segmentation and classification of melanoma from dermoscopic images.It was evaluated on the largest publicly available benchmark for melanoma recognition.
  • Conclusion: 7.5% improvement in area under receiver operating characteristic curve (0.843 vs. 0.783) and 4% improvement in average precision (0.649 vs. 0.624) demonstrate new state-of-the-art performance.The system also improved specificity at the clinically relevant 95% sensitivity operating point.
  • Future Work: Future work may learn a joint pattern-disease classification model or construct a semantic descriptor vector of dermoscopic patterns for use with other disease-classification approaches.These directions depend on an expanding ISIC dataset and additional dermoscopic pattern annotations.
  • Future Work: Non-linear image warping may provide a useful data augmentation technique for classification, alongside exploration of other machine learning approaches.The passage also identifies other machine learning methods as a potential source of additional performance.

Bios

The section identifies John R. Smith as an IBM Fellow and manager leading IBM multimedia and computer-vision research. It also notes his recent participation in the IEEE Distinguished Visitor and ACM Distinguished Speaker programs.

  • John R. Smith: John R. Smith is an IBM Fellow and Manager of the Multimedia and Vision team at IBM’s T. J. Watson Research Center.He is based in Yorktown Heights, NY, and is listed with IBM Research Division.
  • John R. Smith: Smith leads research on IBM Multimedia Analysis and Retrieval System, Intelligent Video Analytics, and IBM Watson Developer Cloud Visual Recognition.His leadership also covers other industry projects related to multimedia and computer vision.
  • Professional service: Smith most recently volunteered as part of the IEEE Distinguished Visitor and ACM Distinguished Speaker programs.The passage describes this involvement as occurring after decades of related activity.
Loading 1610.04662v2…