PaperlayerView exact revision on arXiv

Source-linked AI summary

MDSkin-Net: Multi-Task Skin Lesion Analysis Driven by Pattern Analysis Priors and Spatial Alignment Regularization

Yijian Li, Saad Bedros, Paul Bigliardi, Mei Bigliardi Qi, Vassilios Morellas, Nikolaos Papanikolopoulos

arXiv:2609.30855v1cs.CV

TL;DR

Dermoscopic multi-task models have coupled segmentation and classification without clinical knowledge, while commonly injected ABCD priors were designed for macroscopic assessment rather than dermoscopic cues. MDSkin-Net adds Pattern Analysis-inspired priors and mask-guided spatial alignment to a hybrid CNN–Transformer; its ensemble reports zero-shot transfer across PH2 and ISIC 2018 and in-domain results on ISIC 2017.

  • Problem

    Large dermoscopic training corpora lack structure-level annotations, limiting direct adoption of Pattern Analysis despite its relevance to microscopic diagnostic cues.

    Source: §I · p. 1

  • Method

    MDSkin-Net combines cue-level Pattern Analysis priors in a hybrid CNN–Transformer with segmentation-mask-based spatial alignment for classification.

    Source: §I · p. 2

  • Results

    Across zero-shot PH2 and ISIC 2018 and in-domain ISIC 2017 evaluation, the ensemble achieved 97.84% melanoma AUC on PH2.

    Source: §VI · p. 12

  • Takeaways & Limitations

    The authors conclude that cue-level priors and alignment yield representations that generalize consistently across cohorts of different scales.

    Source: §V · p. 9

    ISIC 2017 and PH2 labels reflect clinical and dermoscopic assessment rather than histopathology, so AUC reflects agreement with expert consensus, not biological ground truth.

    Source: 2) Clinical-Level Limitations · p. 12

Abstract

from arXiv · show

Reliable skin lesion segmentation and classification are central to dermoscopic computer-aided diagnosis. Existing multi-task frameworks couple the two tasks architecturally without clinical knowledge, while knowledge-injecting approaches rely on the macroscopic ABCD rule, which was not designed for dermoscopy. Dermoscopic diagnosis is grounded in Pattern Analysis, a microscopic framework structured around dermoscopic features. We propose MDSkin-Net, which incorporates cue-level Pattern Analysis priors into a hybrid CNN-Transformer architecture. At its core is a Pattern Analysis-Guided Attention Module (PAGAM) comprising three priors motivated by distinct dermoscopic cues: an improved Efficient Channel Attention (iECA), a Multi-Scale Spatial Attention (MSSA), and a Biased Asymmetry Attention (BAA). We further introduce a multi-scale spatial alignment regularization (MSAR) that uses the segmentation ground-truth mask as hierarchical soft supervision, confining the classification head to lesion-localized evidence and coupling both task pathways through a shared spatial prior. Trained exclusively on the ISIC 2017 training split without external dermoscopy data, the MDSkin-Net ensemble transfers robustly under zero-shot evaluation, reaching a Dice Similarity Coefficient (DSC) of 92.38% and a melanoma AUC of 97.84%on PH2, and a DSC of 88.92% on the ISIC 2018 Task 1 test set. On the in-domain ISIC 2017 benchmark, the ensemble attains a mean Area Under the Curve (AUC) of 91.60% across the two classification tasks (melanoma and seborrheic keratosis vs. rest), and a DSC of 84.72% for segmentation. Classification remains competitive with baselines; in-domain segmentation trails single-task specialists, yet the proposed priors and alignment regularization yield representations that generalize consistently across cohorts of different scales.

I. INTRODUCTION

MDSkin-Net addresses the gap between architectural multi-task coupling and clinically grounded dermoscopic priors by bringing Pattern Analysis cues into a shared CNN–Transformer framework. Its encoder combines local texture processing with global context, while spatial alignment uses segmentation supervision to connect classification with lesion-localized evidence.

I. INTRODUCTION

  • Prior multi-task coupling lacked encoded clinical reasoning, while ABCD-based priors target macroscopic features rather than dermoscopic microscopic cues.Pattern Analysis offers an alternative based on microscopic structures, and cue-inspired architectural biases do not require structure-level supervision.Source: §I · p. 1

II. METHODOLOGY

  • The shared ConvNeXt-Tiny and MobileViT encoder preserves fine-grained dermoscopic texture processing while modeling lesion-wide global dependencies.The contribution description identifies ConvNeXt-Tiny as the stem and MobileViT as the bottleneck.Source: §I · pp. 1–2

I. INTRODUCTION

  • MDSkin-Net introduces cue-level Pattern Analysis priors and spatial alignment regularization to couple segmentation and classification under a shared spatial prior.The segmentation ground-truth mask supplies hierarchical soft supervision that confines classification to lesion-localized evidence.Source: §I · pp. 1–2

A. Hybrid CNN-Transformer Shared Encoder

  • The MobileViT bottleneck transforms the deepest encoder feature to capture local detail and global context before restoring and combining its spatial representation.The deepest feature is compressed from 512 to 256 channels; patch tokens undergo attention and are folded back to the spatial layout.Source: A. Hybrid CNN-Transformer Shared Encoder · p. 2
  • The bottleneck’s multi-head self-attention is the injection point for a patch-level structural asymmetry prior.This places Biased Asymmetry Attention in the shared encoder’s transformer stage.Source: A. Hybrid CNN-Transformer Shared Encoder · p. 2

B. Pattern Analysis-Guided Attention Module (PAGAM)

PAGAM uses architectural priors aligned with visual cue categories in Pattern Analysis rather than explicitly detecting canonical dermoscopic structures. Its iECA component responds to abrupt chromatic transitions by combining channel-wise mean and standard-deviation descriptors.

B. Pattern Analysis-Guided Attention Module (PAGAM)

1) Chromatic-Sensitive Channel Attention (iECA):

  • iECA pools channel-wise mean and standard deviation in parallel to represent abrupt local chromatic transitions.The module was motivated by regression areas and the blue-whitish veil.Source: 1) Chromatic-Sensitive Channel… · p. 2
  • iECA concatenates the two descriptors and applies a 1D channel convolution, whose fused response becomes channel-attention weights that rescale the input.The convolution merges the mean and variance statistics before the sigmoid produces the attention weights.Source: 1) Chromatic-Sensitive Channel… · p. 2

2) Multi-Scale Spatial Attention (MSSA):

MSSA uses receptive fields matched to dermoscopic pigment structures’ scale range. BAA builds a lesion-aware patch-pair bias from spatial layout and structural differences.

2) Multi-Scale Spatial Attention (MSSA):

  • MSSA aggregates features across receptive fields matched to pigment structures ranging from localized dots and globules to the lesion-wide network.Its parallel depthwise convolutions use dilation rates 1, 2, and 4.Source: 2) Multi-Scale Spatial Attention (MSSA) · p. 3

3) Asymmetry-Aware Self-Attention (BAA):

  • BAA estimates a saliency-weighted spatial centroid because lesions are rarely centered at the geometric image center.It uses this lesion-aware reference point to construct its asymmetry-related bias.Source: 3) Asymmetry-Aware Self-Attention (BAA) · p. 4
  • BAA’s geometric component softly weights patch pairs by their layout relative to the centroid, using a Gaussian kernel with learnable, head-specific bandwidth.A separate feature-level term represents structural discrepancy using squared ℓ2 distance between pixel-averaged patch tokens.Source: 3) Asymmetry-Aware Self-Attention (BAA) · p. 4
  • BAA couples the geometric and feature-level terms by Hadamard product, normalizes the resulting bias, and injects it into multi-head self-attention with learnable head-wise scaling.The patch-pair bias is broadcast across pixel positions.Source: 3) Asymmetry-Aware Self-Attention (BAA) · p. 4

C. Multi-Task Decoding Heads

The network branches into a progressive segmentation decoder and a spatial-decoupled classification head. The decoder modulates encoder skip features before recursively fusing them with upsampled decoder features.

C. Multi-Task Decoding Heads

1) Segmentation Decoder:

  • The segmentation decoder conditionally modulates encoder skip features before fusion to reconcile semantic differences between deep and shallow features.Source: 1) Segmentation Decoder · p. 4
  • At each stage, the decoder upsamples and refines the deeper feature, halves its channels, concatenates it with the modulated skip feature, and applies two Conv3×3–GN–GELU blocks.An ECA channel-attention block refines the upsampled feature before concatenation.Source: 1) Segmentation Decoder · p. 4

2) Classification Head:

The classification head decouples bottleneck features and combines average and max pooling for its diagnostic prediction. The supplied losses define the joint optimization, with deep supervision for segmentation and label-smoothed cross-entropy for classification.

2) Classification Head:

  • The classification head applies depthwise then pointwise convolutions to bottleneck features, then concatenates global average and max pooling summaries for an MLP diagnostic prediction.The MLP uses GELU activation and dropout.Source: 2) Classification Head · p. 4
  • The cohorts comprise 1,843 nevi, 386 seborrheic keratoses, and 521 melanomas in ISIC 2017, whose official split is 2,000 / 150 / 600.PH2 contains 80 common nevi, 80 atypical nevi, and 40 melanomas; ISIC 2018 Task 1 has no image-level diagnosis and supports segmentation scoring only.Source: 2) Classification Head · p. 5

1) Segmentation Loss:

  • The joint objective combines three components, and the segmentation loss uses equal-weight BCE and Dice terms with deep supervision at decoder stages.The segmentation objective applies the base loss to multi-scale predictions, including the final full-resolution mask and intermediate auxiliary outputs.Source: D. Optimization and Loss Formulation · p. 5, 1) Segmentation Loss · p. 5

2) Classification Loss:

3) Multi-Scale Spatial Alignment Regularization (MSAR):

MSAR provides hierarchical lesion-aligned guidance and couples the classification head to the segmentation decoder through a shared deepest gate. Multi-scale BCE supervision trains gates using soft versions of the segmentation mask.

3) Multi-Scale Spatial Alignment Regularization (MSAR):

  • MSAR combines a residual self-attentive spatial gate with auxiliary BCE supervision to provide hierarchical lesion-aligned guidance to the decoder.The deepest gate is shared by the classification head and segmentation decoder, while shallower gates impose a coarse-to-fine localization prior on segmentation.Source: 3) Multi-Scale Spatial Alignment… · p. 5
  • At each encoder stage, a projected skip feature produces a sigmoid-activated soft spatial gate that residually modulates the feature.The deepest gated feature feeds the MobileViT bottleneck and classification head; shallower gated features replace decoder skip inputs.Source: 3) Multi-Scale Spatial Alignment… · p. 5
  • Each gate is supervised by multi-scale BCE against a ground-truth mask downsampled with adaptive average pooling, preserving partial occupancy near lesion boundaries.Source: 3) Multi-Scale Spatial Alignment… · p. 5
  • The alignment loss aggregates per-stage terms with normalized weights that assign larger weights to deeper stages, and the overall objective combines all three task losses.Source: 3) Multi-Scale Spatial Alignment… · p. 5

III. EXPERIMENTS

Experiments use the official ISIC 2017 training split, reserving PH2 and ISIC 2018 for zero-shot evaluation. The section also describes augmentation, inference-time augmentation, and training configuration.

A. Datasets

  • PH2 and ISIC 2018 are zero-shot benchmarks without fine-tuning; ISIC 2018 is used only for segmentation because it has no image-level diagnostic labels.Training uses the official ISIC 2017 partition without external dermoscopy data or random splits.Source: A. Datasets · p. 5

B. Data Augmentation

  • The original training set is class-imbalanced; a WeightedRandomSampler rebalances sampling without adding external dermoscopy data.Results are reported over three independent runs on 1,000 images.Source: B. Data Augmentation · p. 5, 1) Hyper-Parameters Configuration · p. 7
  • Images are scaled to 384 × 384 with joint image-mask spatial transforms, then augmented photometrically and standardized with ImageNet statistics.Spatial transforms include flips, rotation, and elastic deformation; photometric transforms include gamma correction, Gaussian blurring, color jitter, and CLAHE.Source: B. Data Augmentation · p. 6
  • Inference averages logits from 12 passes spanning three resolutions and four horizontal/vertical flip states.The resolutions are 256 × 256, 384 × 384, and 512 × 512.Source: B. Data Augmentation · p. 6

1) Hyper-Parameters Configuration:

  • The training configuration uses a higher learning rate for BAA, a reduced rate for the pretrained encoder, and EMA weights for evaluation.BAA receives three times the base rate, the pretrained encoder 0.05 times the base rate, and validation and final evaluation use EMA weights.Source: 1) Hyper-Parameters Configuration · pp. 6–7, Table IV · p. 7

2) Evaluation Metrics:

The experiments report segmentation and classification metrics, compare zero-shot transfer and in-domain performance, and ablate the proposed components. Cross-dataset results are strong, while the ablation protocol disables two model-agnostic enhancements to isolate component effects.

2) Evaluation Metrics:

  • Segmentation is evaluated with DSC, IoU, sensitivity, specificity, and accuracy; classification uses sensitivity, accuracy, specificity, F1-score, and AUC averaged across two tasks.Checkpoint selection maximizes the mean of segmentation Dice and classification AUC.Source: 2) Evaluation Metrics · p. 7

1) PH2 and ISIC 2018:

  • Zero-shot PH2 evaluation reaches segmentation DSC 92.28% and melanoma-classification AUC 95.38%; ISIC 2018 segmentation reaches DSC 88.81%±0.45.PH2 segmentation sensitivity is 98.07%; ISIC 2018 IoU is 81.30% ±0.69, and all cross-seed standard deviations are below 1%.Source: 1) PH2 and ISIC 2018 · p. 7

2) ISIC2017:

  • On the official ISIC 2017 test set, segmentation DSC is 84.63% and mean classification AUC is 91.1% across melanoma-vs-rest and seborrheic-keratosis-vs-rest.Segmentation specificity is 99.36%, the highest among listed methods; some baselines use different protocols or additional archive images.Source: 2) ISIC2017 · p. 7

B. Ablation Study

  • Ablations use three random seeds, with mean and standard deviation reported; EMA and ColorJitter are disabled to isolate component effects.The ablation full model has slightly higher DSC and slightly lower AUC than the Table II configuration, reflecting the changed evaluation setup.Source: B. Ablation Study · p. 7

1) Architecture-Level Ablation:

The ablations examine the hybrid encoder, PAGAM components, BAA placement, alignment strategy, and loss weighting. Their results show task trade-offs across configurations, with multi-scale alignment and the 1:3 weighting ratio selected for the final framework.

1) Architecture-Level Ablation:

2) Module-Level Ablation:

  • Model II, with BAA restricted to the final layer, has the highest AUC among BAA variants at 84.48%, while Model VI peaks in segmentation at DSC 85.22%.Extending BAA to all transformer layers causes a slight segmentation regression; Model VII also has F1 variance of ±8.01.Source: 2) Module-Level Ablation · p. 8, Table VI · p. 8

3) Objective-Level Ablation:

  • On ISIC 2017, multi-scale alignment Model V leads segmentation with DSC 85.43% and IoU 77.16%, while deepest-bottleneck Model IX leads classification with AUC 86.43%.The comparison includes no alignment, deepest-bottleneck alignment, and multi-scale alignment across encoder stages.Source: 3) Objective-Level Ablation · p. 8, Table VII · p. 8
  • On unseen PH2, multi-scale alignment Model V ranks first in DSC 91.13%, IoU 84.42%, and AUC 95.14%, with the lowest standard deviation for all metrics.On ISIC 2017, the loss-weighting results follow a U-shaped trend, with Model XI (1:5) attaining DSC 85.48%, IoU 77.28%, and AUC 85.38%.Source: 3) Objective-Level Ablation · p. 8, Table VII · p. 8, Table VIII · p. 8
  • On PH2, Model V (1:3) outperforms Model XI (1:5) on DSC (91.13% vs. 89.91%), IoU (84.42% vs. 82.41%), and AUC (95.14% vs. 93.46%).The authors therefore adopt multi-scale alignment with the 1:3 weighting ratio in the final framework.Source: 3) Objective-Level Ablation · p. 8, Table VIII · p. 8

1) Qualitative Examples Alongside Representative Methods:

MDSkin-Net combines cue-level Pattern Analysis priors with spatial alignment in a shared CNN–Transformer framework. Its results show cross-cohort segmentation gains alongside in-domain trade-offs, while visual examples and ablations illustrate the model’s behavior and task coupling.

1) Qualitative Examples Alongside Representative Methods:

  • MDSkin-Net produced visibly smoother lesion boundaries with fewer high-frequency artifacts than representative methods, with SwinU-Net the closest baseline.Examples span ISIC 2017, ISIC 2018, and PH2.Source: 1) Qualitative Examples Alongside… · p. 9

2) Visual Ablation Analysis:

  • The visual ablation spans hair occlusion, asymmetry, blue-whitish veil, multiscale structures, and imaging artifacts across PH2 and ISIC examples.Fig. 3 columns progress from the input and ground truth through baseline and added attention modules to the full model and its MSAR spatial prior.Source: 2) Visual Ablation Analysis · p. 9, Figure 3 · p. 10

V. DISCUSSION

  • Pattern-level priors target chromatic transitions, scale-diverse pigment structures, and internal asymmetry, while the shared spatial prior guides classification toward lesion-relevant regions.The discussion links these design choices to consistent zero-shot transfer; alternative explanations include cleaner images or favorable class distributions, and no individual component is isolated.Source: §V · p. 9, A. Pattern-Level Priors and… · p. 10

A. Pattern-Level Priors and Generalization

  • Zero-shot segmentation reached DSC 92.38% on PH2 and 88.92% on ISIC 2018, exceeding the in-domain ISIC 2017 DSC of 84.72%, which remained below single-task specialists.The authors attribute the cross-dataset reversal partly to pattern-level priors, while noting that other explanations cannot be ruled out.Source: A. Pattern-Level Priors and… · p. 9

B. Spatial Alignment and Task Conflict

  • The proposed ensemble achieved AP 0.7047, compared with baseline ensemble AP 0.6788, on the ISIC 2017 test set.Fig. 4 presents the precision-recall curves.Source: Figure 4 · p. 11
  • Adding MSAR in Model V raised both ablation metrics to their joint optimum: DSC 85.43% and AUC 85.37%.The learnable residual gate softly modulates classification features using segmentation-derived alignment, and the learned prior concentrates on lesion regions across hierarchical scales.Source: B. Spatial Alignment and Task Conflict · p. 11

C. Layer-Specific Prior Placement

MDSkin-Net transfers zero-shot from ISIC 2017 to PH2 and ISIC 2018, while its classification operating point favors specificity over recall. The discussion also identifies limits in prior placement, dataset scope, clinical context, and label certainty.

C. Layer-Specific Prior Placement

  • Restricting BAA to the final transformer layer gave the most balanced trade-off; broader placement maximized segmentation in Model VI but lowered AUC to 83.86%, while Model VII raised F1 variance to ±8.01.The authors propose matching a clinical prior’s injection depth to its semantic level, placing BAA at the bottleneck and iECA and MSSA along skip connections.Source: C. Layer-Specific Prior Placement · p. 11

D. Operating Point and Clinical Use

  • At the 0.5 threshold, sensitivity was 52.1% and specificity 94.8%, a conservative operating point that prioritizes false-positive suppression over capturing every malignant case.The authors frame the system as a high-throughput triage tool followed by expert review, not a substitute for dermatopathological diagnosis.Source: D. Operating Point and Clinical Use · p. 11

1) Model-Level Limitations:

  • The evaluated datasets are limited in scale and lack real-world clinical long-tail distributions, constraining the model’s evidence base.The authors restrict Transformer components to the bottleneck and suggest larger multi-center datasets could support safely expanding their share.Source: 1) Model-Level Limitations · p. 11
  • MSAR constrains classification attention spatially but does not address gradient-level conflicts in the loss space.The authors identify dynamic gradient projection and uncertainty-based loss weighting as possible remedies.Source: 1) Model-Level Limitations · p. 11

2) Clinical-Level Limitations:

  • Single-image datasets omit longitudinal lesion evolution, and the framework was trained on dermoscopic images acquired under standardized conditions.The discussion notes that total-body photography can introduce distortion, scale variation, and lighting-related color shifts that may degrade PAGAM cues.Source: 2) Clinical-Level Limitations · pp. 11–12
  • ISIC 2017 and PH2 classification labels reflect clinical and dermoscopic assessment rather than histopathology, so reported AUC measures agreement with expert consensus rather than biological ground truth.The authors state that prospective cohorts with histology-confirmed labels are needed for definitive clinical validation.Source: 2) Clinical-Level Limitations · p. 12

VI. CONCLUSION

  • Zero-shot evaluation achieved melanoma AUC 97.84% and DSC 92.38% on PH2, plus DSC 88.92% on ISIC 2018; in-domain ISIC 2017 results were mean AUC 91.60% and DSC 84.72%.The model was trained solely on the ISIC 2017 training set without external dermoscopy data.Source: §VI · p. 12
Loading 2609.30855v1…