PaperlayerView exact revision on arXiv

Source-linked AI summary

XBound-Former: Toward Cross-scale Boundary Modeling in Transformers

Jiacheng Wang, Fei Chen, Yuxi Ma, Liansheng Wang, Zhaodong Fei, Jianwei Shuai, Xiangdong Tang, Qichao Zhou, Jing Qin

arXiv:2206.00806v1cs.CVcs.AI

TL;DR

Skin-lesion segmentation is difficult because lesions vary in size, shape, and color and can have ambiguous boundaries. XBound-Former combines boundary knowledge with self-attention through three boundary learners. It leads the reported skin-lesion comparisons and is also evaluated on polyp segmentation.

  • Problem

    Skin-lesion segmentation is challenging amid size, shape, and color variation and ambiguous boundaries, despite its importance to cancer diagnosis and treatment planning.

    Source: §I · p. 1

  • Method

    XBound-Former combines boundary prior knowledge with self-attention through implicit, explicit, and cross-scale boundary learners.

    Source: §I · p. 2

  • Results

    XBound-Former achieves the best reported ISIC-2016&PH2 performance and the highest IoU and shortest ASSD on all ISIC-2018 sets.

    Source: 1) Quantitative results for skin… · p. 7

  • Takeaways & Limitations

    Experiments on polyp segmentation indicate the model’s feasibility on additional tasks with similar characteristics.

    Source: §VI · p. 9

    In extremely challenging images, unclear boundary key points and false detections may provide harmful guidance to the lesion-segmentation branch.

    Source: §V · p. 9

Abstract

from arXiv · show

Skin lesion segmentation from dermoscopy images is of great significance in the quantitative analysis of skin cancers, which is yet challenging even for dermatologists due to the inherent issues, i.e., considerable size, shape and color variation, and ambiguous boundaries. Recent vision transformers have shown promising performance in handling the variation through global context modeling. Still, they have not thoroughly solved the problem of ambiguous boundaries as they ignore the complementary usage of the boundary knowledge and global contexts. In this paper, we propose a novel cross-scale boundary-aware transformer, \textbf{XBound-Former}, to simultaneously address the variation and boundary problems of skin lesion segmentation. XBound-Former is a purely attention-based network and catches boundary knowledge via three specially designed learners. We evaluate the model on two skin lesion datasets, ISIC-2016\&PH$^2$ and ISIC-2018, where our model consistently outperforms other convolution- and transformer-based models, especially on the boundary-wise metrics. We extensively verify the generalization ability of polyp lesion segmentation that has similar characteristics, and our model can also yield significant improvement compared to the latest models.

I. INTRODUCTION

Skin lesion segmentation must contend with substantial variation, hair occlusion, and ambiguous boundaries. XBound-Former combines boundary knowledge with transformer attention through three learners intended to address these challenges.

I. INTRODUCTION

  • Lesion segmentation is challenged by variation in size, shape, and color, hair occlusion, and low-contrast ambiguous boundaries.These difficulties matter because segmentation supports diagnosis and treatment planning.Source: §I · p. 1

II. RELATED WORK

  • CNNs improve on hand-crafted methods but lack sufficient global context, while transformers capture global context without fully exploiting boundary information across scales.The paper also notes that existing transformer approaches retain convolutional modules.Source: §I · pp. 1–2, B. Vision Transformers · p. 3

I. INTRODUCTION

  • XBound-Former combines boundary prior knowledge and self-attention through implicit, explicit, and cross-scale boundary learners.The design is motivated by coarsely locating lesions with context, then attending to ambiguous areas to refine boundaries.Source: §I · p. 2
  • Im-Bound focuses attention on boundary key points with large variation, strengthening local boundary modeling while retaining global context.The paper argues that these points contribute more to segmentation than other boundary points.Source: §I · p. 2
  • Ex-Bound extracts multiple boundary embeddings, each representing boundary knowledge at a unique scale, to refine local modeling and support cross-scale communication.Source: §I · p. 2

B. Vision Transformers

  • X-Bound uses a boundary embedding from one scale to guide boundary-aware attention at another, addressing ambiguous boundaries and lesion-size variation.The paper describes a pure attention-based design rather than a transformer-convolution fusion.Source: §I · p. 2, B. Vision Transformers · p. 3

C. Boundary-aware Prior Knowledge

The paper introduces boundary-aware supervision for transformers and applies boundary processing to multiscale features from a pyramid vision transformer. The resulting enhanced features feed segmentation heads.

C. Boundary-aware Prior Knowledge

  • XBound-Former adds boundary-aware prior knowledge to vision transformers and selects ambiguous boundary points in a key-patch map for supervision.The proposed generator converts selected boundary points into a map for transformer training.Source: C. Boundary-aware Prior Knowledge · p. 3
  • The key-patch map generator selects the most ambiguous boundary points to supervise transformer training, complementing earlier boundary losses, multitask supervision, and spatial attention approaches.Source: C. Boundary-aware Prior Knowledge · p. 3

III. METHOD

  • A pyramid vision transformer extracts four feature scales, which are enhanced through in-scale and cross-scale boundary aggregation before linear heads predict segmentation maps.The feature pyramid ranges from the lowest, largest-scale feature to the deepest, smallest-scale feature.Source: §III · p. 3

A. In-scale Boundary Modeling

In-scale modeling adds local boundary information to transformer features while retaining global context. Its implicit learner predicts high-variation boundary points and uses the resulting map to refine features before explicit boundary refinement.

A. In-scale Boundary Modeling

  • In-scale modeling combines global dependency with local boundary context to refine features at each scale for lesions with ambiguous boundaries.Source: A. In-scale Boundary Modeling · p. 3

1) Implicitly Boundary-wise Attention:

  • Im-Bound uses self-attention to identify boundary key points with large variation, which the paper treats as more consequential for segmentation than smoother boundary points.Source: 1) Implicitly Boundary-wise Attention · p. 3
  • The boundary key-point map is generated from the ground-truth segmentation map to supervise boundary learners.The cross-scale learner also uses an embedding at one scale to guide attention at another scale.Source: Figure 4 · p. 4, Figure 3 · p. 4
  • Cascaded MSA and MLP blocks gather global dependency, and a sigmoid predictor classifies patches as high-boundary-variation points under key-point-map supervision.The predicted map supports feature refinement; after the implicit blocks, the feature proceeds to ex-Bound.Source: 1) Implicitly Boundary-wise Attention · p. 4

2) Learn Explicit Boundary Embedding:

Ex-Bound learns explicit, scale-specific boundary embeddings, while cross-scale attention transfers complementary boundary information between features at different scales.

2) Learn Explicit Boundary Embedding:

  • Ex-Bound uses boundary key points as queries in a transformer decoder to learn explicit, scale-specific embeddings of high-level boundary semantics.The decoder’s global-context modeling refines randomly initialized vectors; the embeddings also support subsequent cross-scale communication.Source: 2) Learn Explicit Boundary Embedding · p. 4
  • Repeated Ex-Bound blocks refine features and boundary embeddings together before sending them to cross-scale boundary aggregation.Each block outputs an aggregated feature, an embedding, and a predicted key-point map.Source: 2) Learn Explicit Boundary Embedding · p. 4

B. Attention-based Cross-scale Boundary Fusion

  • Cross-scale boundary learning addresses lesion-size variation and ambiguous boundaries by combining information across scales.The approach is described as determining boundaries by zooming in and out and combining multi-perspective information.Source: B. Attention-based Cross-scale… · p. 4
  • A boundary embedding guides attention over features at another scale, transferring complementary boundary knowledge between local detail and high-level semantics.The method compares the higher-scale embedding with points in the lower-scale feature, then upsamples and fuses features before linear projection.Source: B. Attention-based Cross-scale… · p. 4

C. Boundary Key-point Generation Algorithm

The key-point generation algorithm selects boundary locations with greater local deviation to supervise boundary learning.

C. Boundary Key-point Generation Algorithm

  • For each contour point, the algorithm scores local boundary deviation as |p − 0.5|, where p is the lesion-area proportion within a radius-r circle.It applies non-maximum suppression to select points whose scores exceed those of neighboring k points, then marks their locations in a binary key-point map.Source: C. Boundary Key-point Generation… · p. 5
  • The resulting key-point map supervises boundary learners to focus on ambiguous regions and learn boundary knowledge.The map is formed by setting selected locations to one and all other locations to zero.Source: C. Boundary Key-point Generation… · p. 5

D. Objective Function

The joint objective trains segmentation and boundary key-point prediction, balancing the two losses with λ.

D. Objective Function

  • The joint objective combines lesion-segmentation loss with predicted boundary key-point-map loss, balanced by λ.The objective uses Dice loss for segmentation and cross-entropy for key-point maps, with ground-truth maps downsampled at multiple scales.Source: D. Objective Function · p. 5
  • The segmentation loss is Dice loss, while the key-point-map loss is cross-entropy.The displayed formulas define the Dice and cross-entropy terms for predicted and ground-truth maps.Source: D. Objective Function · p. 5

IV. EXPERIMENTS

Experiments use two skin-lesion datasets and a five-dataset polyp benchmark, with evaluation metrics tailored to overlap and boundary accuracy.

A. Dataset

  • ISIC-2016 training samples use the official train-validation split, and the model is tested on 200 PH2 samples.ISIC-2016&PH2 combines samples from two centers; ISIC-2016 has 900 training and 379 validation samples.Source: A. Dataset · p. 5
  • ISIC-2018 evaluation uses five-fold cross-validation because its public test set is unreleased.The dataset contains 2,594 labeled images, with image resolutions ranging from 720×540 to 6708×4439.Source: A. Dataset · p. 5
  • Polyp-seg combines five public datasets, training on portions of Kvasir-SEG and ClinicDB and testing on their remaining samples plus three other datasets.Kvasir-SEG and ClinicDB provide 548 and 900 training samples, respectively.Source: A. Dataset · p. 5

B. Evaluation Metrics

  • Skin-lesion performance is evaluated with Dice, IoU, ASSD, and HD95; polyp segmentation uses IoU and Fw.Higher Dice and IoU and lower ASSD and HD95 indicate better skin-lesion segmentation.Source: B. Evaluation Metrics · pp. 5–6

C. Implementation Details

The implementation uses a shared training setup and a PVTv2 backbone, with fixed defaults for learner counts and boundary-key-point generation.

C. Implementation Details

  • Images are resized to 512 × 512, augmented with flips and limited random scaling, and trained in batches of eight for 200 epochs using AdamW at an initial learning rate of 0.0003.The best validation model is saved; experiments use a single NVIDIA GeForce GTX 3090 with 24 GB of memory.Source: C. Implementation Details · p. 6
  • The network uses ImageNet-pretrained PVTv2 and sets Nim = 2, Nex = 2, r = 2, and k = 30 by default.The learner-count choice is discussed in Section IV-F.2.Source: C. Implementation Details · p. 6

D. Comparisons with state-of-the-art Methods

XBound-Former is compared with CNN- and transformer-based skin-lesion models and polyp-segmentation models. It reports strong performance across skin-lesion datasets and polyp datasets, including challenging boundary cases.

1) Quantitative results for skin lesion segmentation:

  • XBound-Former achieves the best performance on the ISIC-2016 validation and unseen PH2 test sets, and the highest IoU and shortest ASSD across ISIC-2018 sets.The authors also report lower ASSD (−1.83 and −1.93) and HD95 (−6.68 and −7.30) on ISIC-2016&PH2, and improved handling of false positives away from boundaries and ambiguous boundaries on ISIC-2018.Source: 1) Quantitative results for skin… · p. 7

2) Visualized Comparison for Skin Lesion Segmentation:

  • Visual comparisons report accurate predictions for hair-occluded lesions, the smallest and largest lesions, and lesions resembling neighboring tissue.The figure caption notes that IoU and ASSD scores appear at each image’s lower-right corner.Source: 2) Visualized Comparison for Skin… · p. 7, Figure 5 · p. 7

3) Extensive Evaluation for Polyp Segmentation:

  • Against Polyp-PVT, XBound-Former improves overall IoU by 1.5% and Fβ by 2.7%, and on ETIS by 4.5% and 7.0%, respectively.The authors report near-best scores across metrics and identify ETIS as the more challenging dataset, with poorer performance across experiments.Source: 3) Extensive Evaluation for Polyp… · pp. 7–8

E. Analytical Ablation Study

Ablations add the implicit, explicit, and cross-scale boundary learners step by step, comparing quantitative scores and visual predictions against a baseline.

E. Analytical Ablation Study

  • The ablation constructs Im-Bound-Former, Ex-Bound-Former, and XBound-Former by adding each learner in sequence to a baseline; Fig. 8 compares inference time per image across learner counts.The baseline retains the same linear prediction and up-sampling fusion as U-Net.Source: E. Analytical Ablation Study · p. 8, Figure 8 · p. 8

1) Quantitative Analysis:

  • The complete XBound-Former shows consistent improvement on validation and test sets, while Im-Bound adds 0.76% and 1.12% IoU and Ex-Bound adds 0.78% and 0.24%, respectively.The reported gains are relative to the baseline for Im-Bound and relative to the preceding model for Ex-Bound; the complete model’s cross-scale fusion is described as useful.Source: 1) Quantitative Analysis · p. 8

2) Visual Comparison on Lesion Boundaries:

  • In visual ablations, XBound-Former gives the best predictions for small and large lesions, while the baseline produces false positives and the individual learners reduce but do not fully resolve boundary errors.The comparison is shown in Fig. 6.Source: 2) Visual Comparison on Lesion… · p. 8

1) Boundary Supervision:

The section examines boundary-key-point supervision and learner-count choices, then discusses the model’s scope, limitations, and potential extensions.

1) Boundary Supervision:

  • Segmentation scores rise as λ increases from 0 to 2, then fall at 10; without boundary supervision, predictions locate coarse lesion regions but miss the most challenging boundaries.The boundary-key-point map concentrates on ambiguous boundaries, and the authors describe both too little supervision and excessive λ as harmful to training.Source: 1) Boundary Supervision · p. 8

2) Statistics of the Efficiency:

  • The authors select Nim = 2 and Nex = 2 as the final setting, balancing efficiency, accuracy, and generalization; validation IoU changes little at that setting and drops when Nim reaches 3.Increasing learner counts adds computation and may make optimization harder.Source: 2) Statistics of the Efficiency · pp. 8–9

V. DISCUSSION

  • XBound-Former combines global context with multi-scale boundary knowledge through three learners, and the authors report strong segmentation performance on skin lesions and polyp lesions.They present its generalization to unseen images and different tasks as verified within their experiments.Source: §V · p. 9
  • The authors frame segmentation of ambiguous-boundary targets beyond skin lesions as a possible extension, while describing that potential as preliminary.They connect the task to diagnosis, quality control, and treatment planning.Source: §V · p. 9, §VI · p. 9
  • Boundary-key-point detection can remain unclear in extremely challenging images, and false detections may provide harmful guidance to lesion segmentation.The authors suggest separate models for the two branches may help guarantee their accuracy because their representation needs differ.Source: §V · p. 9

VI. CONCLUSION

  • The conclusion reports failures on some extremely low-contrast lesions and suggests combining deep learning with a low-level feature extractor as a possible remedy.The paper also concludes that its boundary modeling improvements arise from implicit, explicit, and cross-scale modeling.Source: §VI · p. 9

Related papers

Loading 2206.00806v1…