Source-linked AI summary
InstEditSeg: Instruction-Driven Image Editing for Polyp and Skin Lesion Segmentation
Ziquan Liu, Zhewei Zhu, Xuyang Shi
TL;DR
Medical polyp and skin-lesion segmentation must handle low contrast, ambiguous boundaries, and domain discrepancies, while prior discriminative and diffusion methods commonly produce standalone binary masks. InstEditSeg instead performs instruction-driven color-coded image editing with DINOv3-guided features, achieving competitive accuracy and stronger unseen-domain generalization, multi-lesion completeness, and task control. Its scope is limited by color sensitivity and unsupported attribute-conditioned selection.
Problem
Polyp and skin-lesion segmentation remains difficult under low contrast, ambiguous boundaries, and cross-domain discrepancies, while many existing methods output standalone binary masks.
Method
InstEditSeg reformulates segmentation as instruction-driven editing, overlays color-coded regions on the original image, and injects DINOv3 multi-scale discriminative features into a diffusion U-Net.
Results
The framework is competitive in benchmark accuracy and shows better unseen-domain generalization, more complete multi-lesion segmentation, and instruction-conditioned task control.
Takeaways & Limitations
The editing formulation is most valuable for cross-domain generalization, multi-lesion completeness, and instruction-conditioned control rather than uniformly superior in-domain accuracy.
Takeaways & Limitations
Instruction control is limited to category selection, and the model does not support spatial or size modifiers or attribute-conditioned selection.
Abstract
from arXiv · showhide
Accurate segmentation of polyps and skin lesions is pivotal for clinical diagnosis, yet existing methods struggle with low contrast, ambiguous boundaries, and cross-domain distribution discrepancies. Discriminative networks and most diffusion-based segmentation approaches predict standalone binary masks, leaving the visual priors of large-scale pretrained generative models largely unexploited. We propose InstEditSeg, a unified generative framework that reformulates medical segmentation as an instruction-driven image editing problem. Instead of emitting a mask, the model renders a color-coded overlay on the original image, conditioned on a textual instruction, so that the edited output aligns with the natural image distribution learned by latent diffusion models and mitigates the domain gap between natural and medical imagery. To recover fine anatomical structures, we introduce DINOv3 as an auxiliary visual encoder and a DINO Feature Guidance Block that builds a multi-scale feature pyramid. The pyramid is fused into the diffusion U-Net by channel concatenation and zero-initialized convolution so that hierarchical discriminative priors can be injected without perturbing the pretrained weights. A dual-branch classifier-free guidance strategy requiring only two forward passes per denoising step reduces inference cost. On polyp and skin lesion benchmarks the framework achieves accuracy competitive with strong discriminative baselines, and it further demonstrates concrete advantages of the generative formulation: notably better cross-domain generalization on unseen data, more complete multi-lesion segmentation, instruction-conditioned task control, and sampling flexibility. We also analyze the strengths and limitations of the paradigm, including its color sensitivity and unsupported attribute-conditioned selection. Code is available at: https://github.com/wincharm001/InstEditSeg.
1 Introduction
Medical polyp and skin-lesion segmentation remains difficult because lesions can be low-contrast with ambiguous boundaries and domain shifts. InstEditSeg reframes segmentation as instruction-driven color-coded image editing, adding DINOv3 guidance and showing advantages in unseen-domain generalization, multi-lesion completeness, and task control.
- Motivation: Polyp and skin-lesion segmentation is clinically important but challenged by low contrast, ambiguous or fuzzy boundaries, morphology variation, and imaging artifacts.These factors hinder accuracy and generalization across colonoscopy and dermoscopic settings.
- Motivation: SAM and binary-mask diffusion methods retain substantial limitations from natural-to-medical domain gaps or task-specific outputs.SAM can struggle with small lesions and blurry boundaries, while diffusion segmentation methods generally remain specialized binary-mask predictors.
- Proposed framework: InstEditSeg reformulates segmentation as instruction-driven editing that preserves the original image while adding color-coded regions aligned with latent diffusion priors.The framework is designed to reduce mismatch between natural and medical imagery.
- Proposed framework: A DINOv3-guided multi-scale feature pyramid injects hierarchical discriminative priors into the diffusion U-Net for lesion localization and boundary delineation.The method also uses an instruction-based data pipeline with randomized colors and unified image-text samples.
- Results: The framework achieves competitive benchmark accuracy while improving unseen-domain generalization, multi-lesion completeness, and instruction-conditioned task control.The strongest discriminative baselines remain competitive in-domain, so the reported advantage is concentrated in these generative-formulation benefits.
2 Related Work
Related work spans discriminative medical segmentation, foundation-model adaptation, diffusion-based segmentation, and instruction-following visual generation. These lines of research motivate a generative approach that addresses domain shift and reduces reliance on manual prompts or task-specific heads.
- Discriminative segmentation: U-Net established the encoder-decoder standard, while attention, dense-convolution, specialized, and transformer designs advanced medical segmentation.Polyp-PVT and EMCAD represent transformer-based and efficient multi-scale attention approaches, respectively.
- Discriminative segmentation: Discriminative models remain sensitive to domain shifts and require extensive pixel-level annotations, motivating generative alternatives using pretrained diffusion priors.The stated goal is improved cross-domain generalization rather than replacement of all discriminative strengths.
- Foundation models: SAM enables strong natural-image zero-shot segmentation, but medical adaptations can be unstable and point or box prompts are cumbersome for low-contrast lesions.Textual instructions offer a direct alternative to manual prompting in the described framework.
- Diffusion segmentation: Diffusion research has progressed from iterative binary-mask segmentation to latent-space and general generation formulations for perception tasks.Examples include SegDiff, MedSegDiff, SDSeg, TSLDseg, Pix2Seq-D, SegGen, LDMSeg, and VPD.
- Instruction-driven generation: Instruction-following editors and unified visual generators use textual conditioning and spatial controls to unify segmentation, detection, and depth without task-specific heads.InstructPix2Pix, ControlNet, T2I-Adapter, Vision Banana, SenseNova Vision, and related models establish this broader interface.
3 Methodology
InstEditSeg reformulates medical segmentation as instruction-driven image editing, generating color-coded overlays with a Stable Diffusion backbone. DINOv3-derived multi-scale priors, auxiliary supervision, and dual-branch guidance support anatomical detail and efficient inference.
- Instruction-Based Segmentation Reformulation: The framework generates color-coded target-region overlays from medical images and textual instructions instead of directly predicting pixel-wise labels.The overlay preserves original content outside the target region while highlighting the target region.
- Diffusion-Based Segmentation Framework: Stable Diffusion performs conditional latent generation by combining the noisy latent with the VAE-encoded input image and instruction embedding.The U-Net predicts added noise from the noisy latent, image-condition latent, timestep, text embedding, and DINO-guided priors.
- DINO-Guided Multi-Scale Feature Projection: A frozen DINOv3 encoder and DFG Block transform patch tokens into hierarchical convolutional feature maps aligned with the U-Net decoding stages.The resulting multi-scale priors are injected into corresponding U-Net stages to provide discriminative semantic and boundary information.
- DINO-Guided Multi-Scale Feature Projection: Concatenation and zero-initialized convolution fuse heterogeneous DINOv3 and diffusion features while initially preserving pretrained U-Net behavior through a residual connection.Concatenation retains both feature branches, and zero initialization allows the guidance branch to take effect progressively.
- Training Objective: The model is trained end-to-end with diffusion noise prediction and auxiliary segmentation supervision from a lightweight decoder using Dice and cross-entropy losses.The balancing hyperparameter λ weights auxiliary segmentation supervision relative to denoising loss.
- Inference Strategy: Dual-branch classifier-free guidance uses instruction-conditioned and empty-text predictions, restricting text guidance to task specification while reducing inference latency.DINO feature injection is part of the framework but omitted from the inference figure for brevity.
4 Experiments
InstEditSeg is competitive in-domain and stronger under domain shift, while its editing formulation improves multi-lesion completeness and instruction-based task unification. Ablations support the contributions of DINOv3 guidance, overlay targets, concatenation fusion, and auxiliary supervision, while color behavior remains dependent on training coverage.
- Cross-Domain Generalization: 83.92% Dice and 77.50% IoU on unseen PolypGen were the best among compared methods, demonstrating strong cross-domain generalization.On unseen ISIC2017, the framework also achieved the best Dice of 83.14%.
- Multi-Lesion Completeness: 75.93% Dice on 119 multi-lesion PolypGen images exceeded EMCAD by 12.75 Dice points, indicating more complete detection of secondary lesions.The authors attribute this primarily to the position-agnostic editing formulation, with DINOv3 helping low-contrast boundaries.
- Unified Instruction-Conditioned Task Control: A single architecture handles polyp and skin lesion segmentation through textual category instructions without task-specific heads, although each task is trained separately.This provides unified instruction-conditioned task control within the reported scope.
- Sampling Flexibility and CFG Analysis: The generative pipeline is more costly: five DDIM steps reached 91.78% Dice at 711 ms per image, versus EMCAD’s 93.74% at 218 ms.Thus, sampling flexibility does not remove the inference-time cost of diffusion-based segmentation.
- Effectiveness of Components: 92.10% Dice with auxiliary segmentation loss followed 87.73% with DINO Feature Guidance alone, while the fine-tuned editing baseline reached 85.47%.The progressive gains support combining discriminative visual priors with explicit structural supervision.
- Overlay Representation: Overlay targets outperformed binary-mask and black-background alternatives, with binary masks reaching only 82.35% Dice on Kvasir-SEG and 52.60% on ISIC2016.The original-content overlay supplies visual context and preserves color-coded structure for diffusion denoising.
- Fusion Strategy: Concatenation fusion exceeded element-wise addition by 0.79% Dice on Kvasir-SEG and 0.81% on ISIC2016, preserving heterogeneous feature information.DINOv3 ViT-S/16 also outperformed alternative backbones; the no-branch variant fell to 86.23% Dice on Kvasir-SEG.
- Color Sensitivity Analysis: Unseen-color failures mix segmentation errors with color-mapping artifacts: corrected Dice exceeded 90% for lavender at Nc ≥5, while small color pools caused genuine failures.At Nc = 16, corrected brown Dice reached 92.55%, showing that broader color training can reduce these failures.
5 Discussion
InstEditSeg’s editing formulation performs best where cross-domain robustness, multi-lesion completeness, and instruction-conditioned control matter, while retaining important trade-offs in color handling, in-domain accuracy, and inference cost.
- The framework attains the best Dice on unseen PolypGen and ISIC2017, supporting stronger cross-domain generalization.The overlay retains original image content and aligns with the pretrained latent diffusion prior.
- A sufficiently rich palette largely removes color-binding artifacts and genuine segmentation failures under unseen colors.Corrected Dice remaps evaluation colors to the model’s painted colors, isolating color-mapping artifacts from true segmentation failures.
- In-domain discriminative baselines remain competitive or superior on Kvasir-SEG, CVC-ClinicDB, and ISIC2016.
- Inference is more expensive than a single discriminative pass, concentrating the formulation’s value in cross-domain generalization, multi-lesion completeness, and instruction-conditioned control.
6 Conclusion
InstEditSeg reframes medical segmentation as instruction-driven diffusion editing with color-coded outputs and DINOv3-guided discriminative features. It is competitive with strong baselines, strongest on unseen-domain Dice, and offers advantages in multi-lesion completeness, task control, and sampling flexibility, despite stated limitations.
- InstEditSeg recasts medical segmentation as instruction-driven diffusion editing that outputs a color-coded edited image rather than a standalone binary mask.The representation aligns with latent diffusion priors and reduces the natural–medical domain gap.
- A DINOv3-guided multi-scale feature pyramid injects hierarchical discriminative priors into denoising through concatenation and zero-initialized convolution.The design improves lesion localization and boundary delineation.
- The framework is competitive with strong discriminative baselines and achieves the best Dice on unseen PolypGen and ISIC2017.
- InstEditSeg improves multi-lesion completeness by 12.75 Dice points over EMCAD and retains about 99.7% of 25-step accuracy at five steps.