PaperlayerView exact revision on arXiv

Source-linked AI summary

RECOD Titans at ISIC Challenge 2017

Afonso Menegola, Julia Tavares, Michel Fornaciali, Lin Tzy Li, Sandra Avila, Eduardo Valle

arXiv:1703.04819v1cs.CV

TL;DR

Skin-lesion segmentation is subjective, with imperfect agreement between human annotators. RECOD Titans evaluated segmentation models and averaged four models for its final submission, which scored 0.793 on official validation.

  • Problem

    Skin-lesion segmentation is subjective, and inter-human agreement for the task is far from perfect.

    Source: B. Data and Framework · p. 1

  • Method

    The team evaluated U-net-style segmentation models with online image augmentation and submitted the averaged outputs of four models.

    Source: D. Experiments · pp. 1–2, C. Data Augmentation · p. 1

  • Results

    0.793 was the official validation score of the final submission averaging four models.

    Source: D. Experiments · p. 2

  • Takeaways & Limitations

    The authors report that their best individual segmentation network was Codella-based, using 8,192 fully connected neurons and 0.5 dropout.

    Source: D. Experiments · p. 1

    The authors caution that they performed no significance or equivalence tests and did not pursue factors with apparently small effects diligently.

    Source: F. Failures to Thrive · p. 3

Abstract

from arXiv · show

This extended abstract describes the participation of RECOD Titans in parts 1 and 3 of the ISIC Challenge 2017 "Skin Lesion Analysis Towards Melanoma Detection" (ISBI 2017). Although our team has a long experience with melanoma classification, the ISIC Challenge 2017 was the very first time we worked on skin-lesion segmentation. For part 1 (segmentation), our final submission used four of our models: two trained with all 2000 samples, without a validation split, for 250 and for 500 epochs respectively; and other two trained and validated with two different 1600/400 splits, for 220 epochs. Those four models, individually, achieved between 0.780 and 0.783 official validation scores. Our final submission averaged the output of those four models achieved a score of 0.793. For part 3 (classification), the submitted test run as well as our last official validation run were the result from a meta-model that assembled seven base deep-learning models: three based on Inception-V4 trained on our largest dataset; three based on Inception trained on our smallest dataset; and one based on ResNet-101 trained on our smaller dataset. The results of those component models were stacked in a meta-learning layer based on an SVM trained on the validation set of our largest dataset.

I. PART 1: LESION SEGMENTATION

The segmentation work began from a U-net baseline and used the 2,000-image official challenge dataset. Trials with extra data produced worse validation scores, with annotation disagreement and training duration offered as possible explanations.

A. Baseline

  • The baseline was a public implementation inspired by U-net, a convolutional network intended for biomedical-image segmentation [5].The team’s first segmentation effort marked its first experience with the task.Source: A. Baseline · p. 1

B. Data and Framework

  • Training used the 2,000-image official ISIC 2017 dataset; tentative trials with extra data produced worse official validation scores.The authors suggest differing segmentation criteria or insufficient training time could explain the extra-data results.Source: B. Data and Framework · p. 1
  • The authors note that segmentation judgments have imperfect inter-human agreement, potentially limiting the benefit of masks from other sources.They also offer insufficient training time as an alternative explanation for the worse extra-data validation scores.Source: B. Data and Framework · p. 1

C. Data Augmentation

The segmentation experiments used online augmentation and evaluated U-net-style models. Averaging four models produced the final submission score of 0.793 on official validation.

C. Data Augmentation

  • Online augmentation applied shifts up to 10%, zoom up to 20%, and rotations up to 270°; 128×128 resizing was faster and performed similarly to 256×256.Transforming images before resizing was slower and did not improve results.Source: C. Data Augmentation · p. 1

D. Experiments

  • The initial VGG-based U-shaped model used ImageNet-pretrained layers and Dice loss, scoring 0.753 on official validation.Training froze the original VGG layers for 100 epochs, then unfroze all weights for another 100 epochs.Source: D. Experiments · p. 1
  • The best individual Codella-based network used 8,192 fully connected neurons and 0.5 dropout, reaching an official validation score of 0.783 after 220 epochs.Reducing the fully connected layer to 4,092 or 2,048 neurons worsened results; the batch-normalized model without that layer scored 0.774 and was faster and smaller, but was not used.Source: D. Experiments · pp. 1–2
  • 0.793 was the official validation score of the final submission, which averaged four models trained with different sample splits and epoch counts.Two models used all 2,000 samples for 250 and 500 epochs; two used different 1,600/400 splits for 220 epochs.Source: D. Experiments · p. 2
  • The authors report that several alternatives did not improve scores, including extra HSV channels, other losses, alternative normalization, and 500-epoch training on all masked archive images.They note that time constraints led them to try new ideas rather than train the same models longer, so longer training might have changed results.Source: D. Experiments · p. 2
  • The team wanted to explore how lesion classification and segmentation could interact and help one another.This was the team’s first contact with skin-lesion segmentation.Source: D. Experiments · p. 2

A. Strategy

The team pursued deep learning to maximize challenge performance, prioritizing data, model depth, and computational resources. It also sought AUC gains without accepting complexity that lacked proportional benefit.

A. Strategy

  • The team identified training-data volume, model depth, and computational power as performance bottlenecks and sought to secure more of each.This motivated attempts to use the deepest possible models.Source: A. Strategy · p. 2
  • The stated goal was the highest challenge rank and extracting as much AUC as possible from the models.The paper contrasts this aim with an earlier work whose goal was not to push model accuracy.Source: A. Strategy · p. 2
  • Added complexity had to bring proportional AUC improvements; otherwise the team preferred a simpler model.Aesthetic considerations tempered the goal of maximizing AUC.Source: A. Strategy · p. 2

B. Training data

The team assembled multiple public skin-lesion datasets, deduplicated images, and excluded some categories to avoid annotation clashes. Dataset choice varied by task: the 7,544-image semi subset performed best for melanoma on official validation, while the full dataset performed better for keratosis.

B. Training data

  • The team was prompted by the data demands of deep models and the use of 129,000 images in Esteva et al..The paper notes that most of those images were unavailable to other researchers.Source: B. Training data · p. 2
  • The initial strategy combined available images while excluding categories that could clash with Challenge annotations.Examples excluded were undiagnosed ISIC Archive images, the Atlas miscellaneous class, IRMA images marked benign, and PH2 atypical nevi.Source: 1) Data debiasing (or rebiasing?) · p. 2
  • A suspicious cluster of benign images from 15-year-old patients was found in the ISIC Archive, and validation numbers slightly improved after its removal.The authors report that the all-data strategy was not optimal for melanoma by official validation AUC.Source: 1) Data debiasing (or rebiasing?) · p. 2
  • Near-duplicates within and across the ISIC Challenge and Archive biased internal validation AUCs, prompting a procedure to keep duplicates within the same split.This was intended to avoid train–test contamination.Source: 1) Data debiasing (or rebiasing?) · p. 2
  • The six-source deploy dataset contained 9,640 images after exclusions and deduplication.The collected sources included the ISIC Challenge, ISIC Archive, Interactive Atlas, Dermofit, IRMA, and PH2.Source: 1) Data debiasing (or rebiasing?) · p. 2, B. Training data · p. 2
  • The 7,544-image semi dataset gave the best official validation AUC for melanoma, while the full dataset performed better for keratosis.The semi subset combined the Challenge, ISIC Archive, and Interactive Atlas, with stated restrictions and exclusions.Source: 1) Data debiasing (or rebiasing?) · p. 2
  • For melanoma, the larger dataset was systematically better on internal validation, leaving uncertainty about whether semi’s official-validation advantage reflected dataset quality or validation-set bias.The authors also considered whether carcinomas in Dermofit could confound classification when added to the negative class, and retained results from models trained on both datasets for aggregation.Source: 1) Data debiasing (or rebiasing?) · pp. 2–3

C. Candidate Models

The team focused on ImageNet-pretrained ResNet-101 and Inception-v4 models, then shifted from separate class-specific models to one three-class model to reduce training and evaluation costs.

C. Candidate Models

  • ImageNet pretraining followed by fine-tuning for skin-lesion classification was considered a sound strategy based on the team’s prior experience.That experience also suggested that deeper, stronger ImageNet models tend to perform better after fine-tuning.Source: C. Candidate Models · p. 3
  • ResNet-101 [12] and Inception-v4 [13] were selected for their strong ImageNet results and availability as pretrained models.The team judged the larger Inception–ResNet’s ImageNet improvement too small for its much greater memory footprint.Source: C. Candidate Models · p. 3
  • Experiments conducted in Keras/Theano and Slim/TensorFlow were ultimately consolidated into a single Slim/TensorFlow codebase.Source: C. Candidate Models · p. 3
  • The team replaced separate melanoma and seborrheic-keratosis models with one three-class model because duplicating the full training and evaluation plan was prohibitive.Source: C. Candidate Models · p. 3

D. Computational Resources

The team sought large-memory CUDA GPUs for deep-learning trials but could not find a rented service combining affordability, public availability, and sufficient computing and memory capacity. LIP6/UPMC/Paris support and reduced summer demand helped supply the needed resources.

D. Computational Resources

  • The team sought large-memory CUDA-compatible GPUs to run many deep-learning trials.Source: D. Computational Resources · p. 3
  • No cloud service combined affordability, public availability, and the computing and memory capability the team needed.AWS was cheap and available, SoftLayer available and capable, and Google Cloud cheap and capable; RECOD had only about six large-memory GPUs shared by over 60 students.Source: D. Computational Resources · p. 3
  • LIP6/UPMC/Paris provided part of the needed resources, while Southern Hemisphere summer reduced competition for RECOD GPUs.Source: D. Computational Resources · p. 3
  • The team announced that its part-3 code would be available soon at the linked repository.Source: D. Computational Resources · p. 3

E. Experimental Design Tactics

The team balanced the goal of maximizing challenge performance with reproducibility and sound decisions, while time constrained statistical testing and a full factorial design. It tested a broad agenda mostly sequentially and investigated a melanoma validation-AUC drop that appeared after the complete dataset became available.

E. Experimental Design Tactics

  • The team aimed to balance scientific rigor and reproducibility with the competitive goal of maximizing its chance of winning.Source: E. Experimental Design Tactics · p. 3
  • Slack, Google Docs, and Google Sheets supported communication and experiment tracking, while git was used to aid reproduction of intermediate steps.Source: E. Experimental Design Tactics · p. 3
  • The team tested ten planned factors, including model depth, resolution, weighting, curriculum, augmentation, SVMs, patient data, optimizers, normalization, and ensembling.Source: E. Experimental Design Tactics · p. 3
  • Time pressure ruled out significance tests, ANOVAs, and the usual incomplete factorial design, so the team tested hypotheses mostly sequentially.They revisited surprising findings and crossed factors only when they strongly expected interactions.Source: E. Experimental Design Tactics · p. 3
  • A delay in obtaining the complete dataset prompted semi-dataset trials after melanoma validation AUC sharply dropped, while keratosis AUC did not.The team decided to investigate the discrepancy.Source: E. Experimental Design Tactics · p. 3

F. Failures to Thrive

Several expected improvements did not materialize: larger-resolution inputs and more complex weighting did not help, while curriculum training performed worse than training on all images at once. The authors caution that small effects were not rigorously tested and that time constrained segmentation-information experiments.

F. Failures to Thrive

  • The authors caution against definitive claims that the tested factors were useless because small effects were not pursued diligently and no significance or equivalence tests were performed.Source: F. Failures to Thrive · p. 3
  • Neither larger-input VGG-16 nor larger-image pre-cropping for Inception-v4 improved results.Source: F. Failures to Thrive · p. 4
  • More complex class- and sample-weighting schemes produced worse AUCs; no weighting was best.The schemes addressed class imbalance and annotation reliability.Source: F. Failures to Thrive · p. 4
  • With one exception, the two early-stopping rules had no impact; the team could not afford the several-week fine-tuning recommended for some Inception applications.The authors note that validation-based early stopping might matter in such super-long training scenarios.Source: F. Failures to Thrive · p. 4
  • The three-step curriculum schedule performed worse than simply training with all images at once.The schedule progressed from Atlas easy images, to easy and moderate images, and then to all images.Source: F. Failures to Thrive · p. 4
  • Patient-data encodings yielded inconsistent results, while the segmentation-information experiments were too quick-and-dirty to assess properly under the time constraints.The authors still believed properly encoded segmentation information could help.Source: F. Failures to Thrive · p. 4

G. Success Factors

The authors identify larger datasets combined with deeper models, augmentation, per-image normalization, and model fusion as factors behind stronger validation results. A meta-learning SVM achieved the best official validation AUC.

G. Success Factors

  • A meta-learning SVM that learned from multiple models' decisions gave the best results on the official validation AUC.The authors note they had not conducted a factorial analysis to quantify each factor's contribution.Source: G. Success Factors · p. 4
  • Combining deeper models with larger datasets boosted results more than the transition to deeper models alone.The smaller semi dataset was already several times larger than the official challenge training set.Source: G. Success Factors · p. 4
  • Data augmentation was reported as critical for training and testing, and simple average pooling worked best across the pooling schemes tried.Augmentation also improved results with an SVM decision layer and with stacked models.Source: G. Success Factors · p. 4
  • Subtracting the average image pixel improved Inception results, while additionally dividing by the standard deviation performed worse than no per-image normalization.The authors did not test this factor on ResNet.Source: G. Success Factors · p. 4

H. Assembling the Final Submission

The team selected models using both internal and official validation results because their AUCs were not perfectly correlated, especially for melanoma. The final classification system stacked seven models with an SVM meta-learning layer and augmentation.

H. Assembling the Final Submission

  • The internal-to-official validation AUC correlations were R=0.58 for melanoma, R=0.77 for average AUC, and R=0.79 for seborrheic keratosis AUC.The plotted models were a subset of 48 from more than a hundred experiments; large symbols mark models chosen for the final stacking.Source: H. Assembling the Final Submission · p. 4, Figure 1 · p. 5
  • Because the best models differed between melanoma and keratosis, the team chose one shared model set and left task-specific adjustments to the meta-learning layer.They selected models that performed well on both internal and official validation axes.Source: H. Assembling the Final Submission · p. 4
  • For each sample, the meta-learning layer concatenated the chosen models' decisions as features for two binary SVMs, one for melanoma and one for keratosis.The SVMs were trained on the internal validation set, preventing evaluation with the internal validation AUC; the scheme attained the best official validation AUC.Source: H. Assembling the Final Submission · p. 5
  • The most successful meta-learning variation evaluated each internal-validation training sample thrice per model and average-pooled the resulting replicas.The same approach was applied to official validation and test samples.Source: H. Assembling the Final Submission · p. 5
  • The seven-model stack used three Inception models trained on deploy, three Inception models trained on semi, and one ResNet model trained on semi.An SVM meta-learning layer was trained on the deploy validation set.Source: H. Assembling the Final Submission · p. 5

Related papers

Loading 1703.04819v1…