Source-linked AI summary
SAUF-Net: Structure--Appearance Representation Learning with Uncertainty Feedback for Semi-Supervised Medical Image Segmentation
Qin Lu, Zheyang Jing, Yujie Yang, Jianwang Li, Chen Yi, Shaofeng Jiang
TL;DR
Semi-supervised medical image segmentation still needs better use of unlabeled data because prediction-level regularization often overlooks the reliability of internal representations and structure–appearance entanglement. SAUF-Net decomposes features, enforces appearance-swapped structural consistency, and supplies reliability-guided uncertainty feedback, with experiments on ISIC-2016 and Kvasir-SEG validating its effectiveness under low-label settings. The evaluation remains limited to 2D binary lesion segmentation without cross-dataset generalization assessment.
Problem
Most semi-supervised segmentation methods exploit unlabeled data mainly through prediction-level regularization, while internal feature-representation reliability and structure–appearance entanglement remain insufficiently addressed.
Method
SAUF-Net decomposes bottleneck features into structural and appearance representations, applies appearance-swapped consistency, and uses reliability-guided uncertainty feedback to regularize unlabeled representations.
Results
Experiments on ISIC-2016 and Kvasir-SEG validate SAUF-Net’s effectiveness under low-label settings.
Takeaways & Limitations
SAUF-Net provides a structure–appearance representation learning framework with reliability-guided uncertainty feedback for semi-supervised medical image segmentation.
Takeaways & Limitations
The evaluation is limited to 2D binary lesion segmentation and does not assess cross-dataset generalization.
Abstract
from arXiv · showhide
Semi-supervised learning has shown great potential for reducing annotation costs in medical image segmentation. However, most existing methods mainly exploit unlabeled data through prediction-level consistency, while the reliability of internal feature representations is often overlooked. In medical images, target-related structural cues are easily entangled with unstable appearance variations, which may lead to unreliable pseudo labels and error accumulation during training. To address these issues, we propose SAUF-Net, a Structure--Appearance Representation Learning with Uncertainty Feedback Network for semi-supervised medical image segmentation. SAUF-Net uses the Structure--Appearance Decomposition Module (SADM) to separate bottleneck features into structural and appearance representations. The Disentangled Guidance Module (DGM) injects these representations into the decoding process to enhance structure-aware segmentation. Meanwhile, the Auxiliary Decoder produces branch-specific predictions for reliability estimation and a fused prediction for appearance-swapped consistency. Furthermore, we introduce an Appearance-Swapped Consistency branch to encourage structural representations to remain stable under appearance variations. We also introduce a reliability-map-guided dual-head discriminator with a Validity Head and an Uncertainty Head to provide feature-level uncertainty feedback. Extensive experiments on ISIC-2016 and Kvasir-SEG demonstrate that SAUF-Net outperforms state-of-the-art semi-supervised methods, especially under low-label settings.
1 Introduction
Medical image segmentation has advanced substantially, but fully supervised approaches remain costly because they require expert pixel-level annotations. Semi-supervised methods reduce this burden, yet most exploit unlabeled data mainly through prediction-level regularization; SAUF-Net instead combines structure–appearance modeling with reliability-guided feature-level feedback.
- Expert pixel-level annotation is expensive and time-consuming, limiting fully supervised segmentation in label-scarce clinical settings.
- Semi-supervised learning jointly uses a small labeled set and a large unlabeled set to reduce dependence on extensive annotations.
- Existing semi-supervised methods commonly use consistency regularization, pseudo-label generation, uncertainty-aware selection, or distribution alignment to exploit unlabeled images.
- Most existing methods regularize training mainly at the prediction level, leaving internal feature-representation reliability less directly addressed.
- SAUF-Net separates structural cues from appearance variations and adds reliability-guided feature-level feedback for semi-supervised medical image segmentation.
- Experiments on ISIC-2016 and Kvasir-SEG validate SAUF-Net’s effectiveness under low-label settings.
2 Methods
SAUF-Net learns disentangled structural and appearance representations, injects them into decoding, and uses consistency and uncertainty feedback to improve semi-supervised segmentation.
- Auxiliary Decoding: The Auxiliary Decoder generates structural and appearance branch predictions for reliability estimation and a fused prediction for consistency learning.Its fusion branch combines Fs and Fa, while the main decoder produces the final prediction P.
- Structure–Appearance Representation Learning: SAUF-Net decomposes bottleneck features into structural representation Fs and appearance representation Fa, separating target-related cues from unstable appearance variations.Fs captures shape, location, continuity, and boundaries, whereas Fa models texture, intensity, color, and noise.
- Structure–Appearance Representation Learning: The DGM injects resized structural and appearance representations into multi-scale decoding through serial deformable guidance steps.Structural guidance first focuses decoding on structure-related regions, followed by appearance-based refinement, and the guided features are aggregated to produce P.
- Appearance-Swapped Consistency: ASC pairs a structural representation with a shuffled appearance representation so the fused prediction preserves structure despite appearance changes.The mixed feature Fmix is processed by the same fusion branch used for the auxiliary prediction.
- Reliability-Guided Uncertainty Feedback: The reliability map combines main–structural prediction agreement with structural–appearance response contrast, with larger values indicating more reliable regions.This map guides augmented prediction learning and supplies complementary uncertainty targets for unlabeled features.
- Reliability-Guided Uncertainty Feedback: A dual-head discriminator estimates feature validity and spatial uncertainty, encouraging augmented unlabeled features to be valid and low-uncertainty.The Validity Head distinguishes encoder-derived from generated features, while the Uncertainty Head uses labeled errors and unlabeled complementary reliability maps.
3 Experiments
Experiments evaluate SAUF-Net on two medical image segmentation datasets using standard metrics, low-label comparisons, visual cases, and component ablations. Results show improvements over AdaptFRCNet and gains from structure–appearance decomposition, reliability guidance, ASC, and uncertainty feedback.
- Datasets and metrics: The study evaluates ISIC-2016 and Kvasir-SEG using Dice, IoU, and pixel-level Accuracy.ISIC-2016 contains dermoscopy images, while Kvasir-SEG contains polyp images; both use train/test splits for evaluation.
- Comparison with state-of-the-art methods: 91.88% Dice and 85.88% IoU on ISIC-2016 at 10% labeled data outperform AdaptFRCNet by 0.51% and 0.58%.On Kvasir-SEG, SAUF-Net reaches 90.53% Dice and 84.94% IoU, improving over AdaptFRCNet by 1.18% and 1.58%.
- Comparison with state-of-the-art methods: 90.91% Dice on ISIC-2016 and 89.71% Dice on Kvasir-SEG are achieved with only 5% labeled data.At 20% labeled data, Dice still improves over AdaptFRCNet by 0.11% on ISIC-2016 and 1.27% on Kvasir-SEG.
- Ablation study: SADM raises Kvasir-SEG Dice from 82.16% to 89.91% and IoU from 74.69% to 84.77% versus SegFormer-B4 with 20% labeled data.The ablation attributes these gains to representations that emphasize shape, spatial location, and boundary continuity over unstable appearance changes.
- Ablation study: Adding Mrel increases Dice from 89.91% to 91.16% and IoU from 84.77% to 85.44%, while ASC further raises Dice to 91.38% and IoU to 85.86%.With all components enabled, SAUF-Net achieves 97.59% Accuracy, 91.98% Dice, and 86.69% IoU.
- Hyper-parameter sensitivity: The best hyper-parameter settings are λasc = 0.05, α = 0.10, and τ = 0.95.These settings correspond to the strongest reported ASC, uncertainty-feedback, and pseudo-label-filtering performance, respectively.
4 Conclusion
SAUF-Net combines structure–appearance decomposition, multi-scale disentangled guidance, appearance-swapped consistency, and reliability-guided uncertainty feedback for semi-supervised segmentation. Experiments on ISIC-2016 and Kvasir-SEG show competitive low-label performance, while evaluation remains limited to 2D binary lesion segmentation without cross-dataset generalization.
- Conclusion: SAUF-Net decomposes bottleneck features into structure and appearance representations, guides decoding with DGM, and regularizes unlabeled learning with ASC and uncertainty feedback.The framework is designed for semi-supervised medical image segmentation.
- Conclusion: Experiments on ISIC-2016 and Kvasir-SEG show competitive performance under low-label settings.The reported conclusion covers both evaluated datasets and the low-label regime.
- Limitations and future work: The current evaluation is limited to 2D binary lesion segmentation and does not assess cross-dataset generalization.Future work is described as extending evaluation to 3D, multiclass, and cross-domain settings.