Source-linked AI summary
Feature Reconfiguration With Visual Prior for Medical Lesion Segmentation
Yinan Liu, Jiankang Hong, Zhen Gao, Ye Lu
TL;DR
Lesion segmentation remains difficult because complex backgrounds and diverse lesion morphology challenge existing methods lacking early prior guidance and encoding-stage feature reconfiguration. FreNet uses SAM-derived visual priors for pixel-level input reconfiguration and dual-domain feature reconfiguration during encoding. Across nine benchmarks and three imaging modalities, it outperforms existing methods, including a 5.0% Dice improvement over the SOTA method on ETIS.
Problem
Complex backgrounds and diverse lesion morphology challenge lesion segmentation, while existing methods lack early prior guidance and feature reconfiguration during encoding.
Method
FreNet uses SAM-derived visual priors for pixel-level input reconfiguration before encoding and frequency-spatial feature reconfiguration during encoding.
Results
FreNet achieves superior segmentation accuracy and generalization across nine benchmarks and three imaging modalities, with a 5.0% Dice improvement over the SOTA method on ETIS.
Takeaways & Limitations
The framework jointly reconfigures inputs and backbone features to address background interference and diverse lesion morphology in lesion segmentation.
Abstract
from arXiv · showhide
Lesion segmentation in medical images plays a critical role in clinical diagnosis and treatment planning. Despite significant advances, lesion segmentation remains challenging due to two major factors: (1) complex background interference; (2) diverse lesion morphology. Existing encoder-decoder based methods mainly focus on enhancing feature extraction or redesigning decoding strategies. However, they lack early prior guidance and feature reconfiguration during the encoding stage, limiting their effectiveness in handling these challenges. To address these limitations, we propose FreNet, a feature reconfiguration framework with visual priors, which performs pixel-level reconfiguration before encoding and feature-level reconfiguration during encoding for precise medical lesion segmentation. To suppress background responses, we propose an Implicit Prior Neural Network (IPNN), which models a continuous spatial field and leverages visual prior from SAM to reconfigure input image before encoding stage. To better handle diverse lesion morphology, we design a Dual-domain Feature Reconfiguration (DFR) module to progressively reconfigure backbone features during encoding stage. Within DFR, the Frequency Decoupling Module (FDM) decouples backbone features in frequency domain to enhance foreground-background discriminability, while the Spatial Localization Module (SLM) spatially relocates and improving spatial stability after frequency decoupling. Extensive experiments on 9 medical image segmentation benchmarks across three imaging modalities demonstrate that FreNet significantly outperforms state-of-the-art (SOTA) methods. On the challenging ETIS dataset, our method achieves Dice improvements of 5.0% over SOTA method and 7.2% over SAM.
Introduction
Lesion segmentation is hindered by complex backgrounds and diverse lesion morphology, while existing methods lack coordinated prior guidance and feature reconfiguration during encoding. FreNet addresses these gaps with pixel- and feature-level reconfiguration using visual priors.
- Lesion segmentation supports disease diagnosis, clinical decision-making, therapy monitoring, and prognosis prediction.
- Complex backgrounds and diverse lesion morphology cause erroneous responses and make lesions difficult to distinguish from surrounding tissues.Examples include hair occlusions, ultrasound speckle and artifacts, irregular shapes, blurred boundaries, and subtle intensity or texture variations.
- Existing methods lack either early prior guidance or multi-scale feature reconfiguration during encoding.This limitation affects background suppression and handling of diverse lesion morphology.
- FreNet introduces visual priors before and during encoding to reconfigure representations at both pixel and feature levels.The framework uses SAM-derived priors before encoding and spatial-frequency feature updates during encoding.
- IPNN reconfigures raw inputs before encoding, while DFR progressively refines backbone features through frequency decoupling and spatial localization.FDM enhances lesion-background separability, and SLM preserves spatial continuity after frequency decoupling.
- Experiments across three imaging modalities demonstrate FreNet's superior segmentation accuracy and generalization.
Related Work
Related lesion-segmentation methods use multi-scale and hybrid architectures, prior-driven constraints, or adaptations of SAM. FreNet's overview combines dense visual-prior injection before encoding with dual-domain feature reconfiguration during encoding.
- Recent lesion-segmentation methods include multi-scale modeling, U-Net variants, and hybrid architectures.Examples combine multi-frequency attention, recurrent scanning, Transformers and convolutions, CNNs and MLPs, or KAN-enhanced attention.
- Prior-driven methods use channel, shape, textual, or diffusion-based priors to constrain or enhance segmentation.
- SAM-based medical segmentation methods adapt the foundation model through fine-tuning, lightweight adapters, or learnable prompts.
- FreNet's architecture injects dense SAM-derived visual priors into inputs before encoding and reconfigures multi-scale features in spatial and frequency domains during encoding.
Methodology
FreNet reconfigures lesion representations before and during encoding by injecting SAM-derived priors at the pixel level and progressively optimizing backbone features across frequency and spatial domains.
- Pixel-level reconfiguration: FreNet injects SAM-derived visual priors into raw images before encoding, reconfiguring input representations to enhance lesion perception and suppress background responses.The framework is designed for complex backgrounds and diverse lesion morphology.
- Pixel-level reconfiguration: IPNN models a continuous 2D spatial field conditioned on SAM masks to generate pixel-wise reconfiguration weights for the input image.Coordinates are encoded alongside a mask-conditioned vector before producing the reconfiguration weights.
- Feature-level reconfiguration: DFR is inserted between backbone stages to progressively update features through frequency decoupling and spatial localization.This targets lesion–background transition regions affected by diverse morphology and prior-mask misalignment.
- Frequency-domain processing: FDM transforms backbone features into the frequency domain, partitions magnitude maps into four ratio-thresholded bands, and uses the visual prior to constrain band extraction.The thresholds are based on ratios 2, 4, and 8, while trainable coefficients balance frequency components.
- Spatial-domain processing: SLM restores spatial coherence after frequency decoupling by aligning refined prior masks with frequency-optimized features through similarity-based fusion and residual connections.The resulting dual-domain feature is fused residually with the backbone representation.
- Training objective: Training jointly supervises PVT and SAM outputs using weighted binary cross-entropy and weighted intersection-over-union losses.The total loss sums the two branch-specific losses for end-to-end training.
Experiments
FreNet is evaluated on nine challenging 2D datasets across dermoscopy, ultrasound, and endoscopy, against 13 SOTA methods. It improves Dice over SOTA methods and SAM, while ablations and visual analyses examine its modules, robustness, and efficiency.
- Datasets: The evaluation covers 9 datasets spanning dermoscopy, ultrasound, and endoscopy.The datasets include ISIC2018, PH2, BUSI, STU, CVC-ColonDB, CVC-ClinicDB, Kvasir, ETIS, and CVC-300.
- Benchmark comparison: FreNet improves Dice by 0.7%–5.0% over SOTA methods and 0.7%–7.2% over SAM across 9 datasets.All Dice-based p-values are below 0.05.
- Module ablation: Adding SAM alone yields a slight improvement, whereas progressively adding IPNN and DFR substantially improves segmentation performance.The ablation baseline uses a PVT-based backbone.
- IPNN analysis: IPNN concentrates Grad-CAM activations in lesion regions and reduces activations in background areas.The comparison uses backbone features P 1 with and without IPNN.
- DFR analysis: DFR produces more compact intra-class and more separable inter-class foreground and background features, while strengthening lesion-region responses.The internal ablation finds the ratio-based frequency-decoupling strategy performs best.
- SAM mask quality: FreNet remains robust across low-, medium-, and high-quality SAM masks, achieving a 33.0% performance gain on ETIS with low-quality masks.Mask-quality categories are defined by Dice intervals [0, 0.5), [0.5, 0.8), and [0.8, 1.0].
- Efficiency: Although FreNet adds computational cost, its IPNN and DFR modules are lightweight, with 0.04 M and 2.97 M trainable parameters.The efficiency analysis compares accuracy-complexity trade-offs with competitive SOTA methods.
Conclusion
FreNet addresses complex background interference and diverse lesion morphology through feature reconfiguration with visual priors. Its pixel- and feature-level reconfiguration improves segmentation accuracy and generalization, but increases computational cost.
- Framework: FreNet uses IPNN for pixel-level input reconfiguration before encoding and DFR for progressive multi-scale feature reconfiguration during encoding.DFR contains Frequency Decoupling and Spatial Localization Modules.
- Problem addressed: The framework targets complex background interference and diverse lesion morphology in medical lesion segmentation.IPNN provides early lesion-aware activation and reduces background responses, while DFR improves lesion-background discriminability.
- Conclusion: Experiments demonstrate superior segmentation accuracy and generalization compared with existing methods.
- Limitation: FreNet incurs increased computational cost, motivating future work on knowledge distillation for improved efficiency.