Skin Cancer Segmentation and Classification Using Vision Transformer for Automatic Analysis in Dermatoscopy-based Non-invasive Digital System

• U.S. skin cancer incidence rose from 1992 to 2019, and ACS statistics reported 58,120 cases among males and 39,490 among females [2].Source: §1 · p. 1
• More than 90% of skin cancers are linked to direct sunlight exposure and occur on visible skin, which can be imaged non-invasively with a dermatoscope.The passage also notes that high-resolution smartphone cameras can capture affected areas.Source: §1 · p. 2
• Primary diagnosis traditionally uses the ABCDE criteria, while definitive pathological diagnosis typically requires a costly, time-consuming, and painful biopsy.The paper describes excisional, incisional, punch, and shave biopsies; it also says diagnostic delays can make it difficult to handle patient volume in time.Source: §1 · pp. 2–3
• The proposed workflow uses SAM to segment suspected areas in HAM10000 images, then classifies lesions as benign or malignant for potential smart-device use.The authors manually annotated suspected areas before applying SAM for semantic segmentation.Source: §1 · p. 3
• Google’s ViT patch-32 outperformed the evaluated VAN, ResNet50, MAE, DiT, and BEiT models for skin cancer classification.The evaluated models are identified in the paper as Google’s ViT [14], VAN [15], ResNet50 [14], MAE, DiT [17], and BEiT [18].Source: §1 · p. 3
• The selected ViT configuration used Adam, a learning rate of 0.00002, and Cross Entropy Loss after evaluating alternative optimizers, learning rates, and loss functions.The authors also evaluated Adamax, RMSProp, and SGD, plus Hinge loss.Source: §1 · pp. 3–4

Diagnostic context

• Conventional evaluation can take nearly a month, from specialist examination through biopsy and histopathology, motivating efforts to reduce diagnostic time.The review describes dermoscopy taking 10–15 minutes, biopsy typically taking 2–3 weeks, and histopathological examination taking 23 days.Source: §2 · p. 4

Classification literature

• Prior classification research spans varied datasets and methods, including pretrained CNNs, traditional machine learning, ensembles, feature engineering, and transformer-based models.The review covers studies using datasets such as HAM10000, PH2, and ISIC, with varied preprocessing, augmentation, and model modifications.Source: §2 · pp. 4–7

Segmentation literature

• Segmentation literature includes U-Net reviews and variants, contour methods, and hybrid architectures evaluated across skin-lesion datasets; the paper chooses SAM for segmentation.The review reports dataset-specific performance for several segmentation approaches and notes the use of Detectron2 in medical segmentation research.Source: §2 · p. 7

Research gap and approach

• The paper identifies limited ViT experimentation and no prior SAM use in cancer detection, then proposes SAM segmentation followed by HAM10000 classification with several ViT models.The authors aim to select the best ViT model for classification after detecting regions of interest with SAM.Source: §2 · p. 7

3.1. Dataset Description

• From 10,015 HAM10000 images, 15 were excluded; images were resized from 600×450 to 224×224, classes balanced at 5,000 each using augmentation, and split 60/20/20.Source: §3.1 · p. 7

3.2. Experimental Procedures

• Six classifiers—ViT-Google, ViT-MAE, ViT-ResNet50, ViT-VAN, ViT-BEiT, and ViT-DiT—were compared, with ViT-Google reported as outperforming the others.Source: §3.2 · p. 9
• The pipeline resizes 20× dermatoscope images to 224×224, applies SAM segmentation and binary masks, then sends processed images to ViT for benign/malignant classification.Source: §3.2 · pp. 8–9
• ViT converts masked images into patches and positional embeddings, uses encoder self-attention and feed-forward layers, and applies a fully connected SoftMax classification head; Google patch-32 was used for feature extraction.Source: §3.2 · p. 9

3.3. Experimental Setup

• SAM segmentation ran through TensorFlow and ViTs through PyTorch on separate Windows 11 computers: one with an RTX 3090 and 32 GB RAM, the other an RTX 3080 Ti and 64 GB RAM.Hyperparameter optimization was performed for both segmentation and classification.Source: §3.3 · pp. 10–11

4.1.1. Segmentation

• IoU measures the intersection of predicted and ground-truth masks relative to their union, while Dice measures twice their intersection relative to the sum of their areas.Both metrics assess spatial overlap; pixel accuracy alone does not account for class distinctions or spatial relationships between pixels.Source: §4.1 · pp. 11–12
• SAM matched over 96% of areas on average under the IoU metric, with Figure 10 showing side-by-side examples of segmentation results.The study preferred IoU because it directly represents overlap between predicted and ground-truth masks.Source: §4.1 · p. 12, Figure 10 · p. 13

4.1.2. Classification

• Google ViT patch-32 was selected from six classifiers for the skin cancer detection system based on accuracy comparisons, and its loss scores were significantly better than those of the other five models.The compared models were ViT-Google, ViT-MAE, ViT-ResNet50, ViT-VAN, ViT-BEiT, and ViT-DiT.Source: §4.1 · p. 13, Table 5 · p. 13, Figure 11 · p. 14

ViT

• False negatives exceeded false positives among misclassifications for every tested model, while ViT-Google had the highest overall test accuracy.Source: ViT · p. 15, Figure 12 · p. 15

4.2. Accuracy Comparison

• ViT-Google combined a low training-to-test accuracy difference with higher test accuracy than the other models, leading to its selection for final implementation.The paper describes a lower training-to-test difference as indicating better training performance, while cautioning that the difference alone may be misleading when test accuracy is low.Source: §4.2 · pp. 15–16, Table 7 · p. 16, Figure 13 · p. 16
• 96.15% average validation accuracy across five-fold cross-validation and 99.49% ROC AUC were reported for ViT-Google.The authors selected ViT-Google for final implementation after finding its test accuracy higher than the other models and its training-to-test accuracy difference low.Source: §4.2 · p. 16, Figure 14 · p. 17, Figure 15 · p. 17

5. Conclusion

• SAM segmentation achieved 96.01% IOU and 98.14% Dice Coefficient, while Google ViT patch 32 classification achieved 96.15% accuracy and 99.49% ROC score.Source: §5 · p. 17
• The authors conclude that ViT-Google (patch 32) can serve as an efficient detection model on smart devices for swift detection and support for pathologists.Source: §5 · p. 18
• The method is more suitable for fair-skinned individuals because the Harvard University dataset predominantly represents white people.Future work will expand the dataset to include diverse ethnic backgrounds and use federated learning to update the model iteratively without retraining it entirely.Source: §5 · p. 18