Source-linked AI summary
Diffusion Model is Secretly a Training-free Open Vocabulary Semantic Segmenter
Jinglong Wang, Xiawei Li, Jing Zhang, Qingyuan Xu, Qin Zhou, Qian Yu, Lu Sheng, Dong Xu
TL;DR
Open-vocabulary segmentation is sought as an alternative to annotation-heavy, category-limited methods, but contrastively pretrained models can lose localization and object-shape information. DiffSegmenter uses attention maps from off-the-shelf diffusion models for training-free segmentation, achieving impressive results across three benchmarks in zero-shot and image-level weakly supervised settings.
Problem
Open-vocabulary segmentation is sought as a more practical alternative to annotation-heavy, category-limited methods, but contrastively pretrained image-text models can lose localization and object-shape awareness.
Source: §1 · p. 1Method
DiffSegmenter enables training-free open-vocabulary segmentation by combining diffusion U-Net self-attention maps for object shapes with cross-attention maps for semantics, alongside designed prompts and category filtering.
Source: §1 · p. 2Results
Across three benchmarks, DiffSegmenter achieves impressive open-vocabulary segmentation results in both zero-shot and image-level weakly supervised settings.
Source: §1 · p. 2Takeaways & Limitations
The findings support text-to-image diffusion models as an alternative basis for open-vocabulary semantic segmentation.
Source: §1 · p. 2The diffusion process can blur fine-grained details and local pixel-level semantics, which is detrimental to segmentation results.
Source: §3.1 · p. 3
Abstract
from arXiv · showhide
The pre-trained text-image discriminative models, such as CLIP, has been explored for open-vocabulary semantic segmentation with unsatisfactory results due to the loss of crucial localization information and awareness of object shapes. Recently, there has been a growing interest in expanding the application of generative models from generation tasks to semantic segmentation. These approaches utilize generative models either for generating annotated data or extracting features to facilitate semantic segmentation. This typically involves generating a considerable amount of synthetic data or requiring additional mask annotations. To this end, we uncover the potential of generative text-to-image diffusion models (e.g., Stable Diffusion) as highly efficient open-vocabulary semantic segmenters, and introduce a novel training-free approach named DiffSegmenter. The insight is that to generate realistic objects that are semantically faithful to the input text, both the complete object shapes and the corresponding semantics are implicitly learned by diffusion models. We discover that the object shapes are characterized by the self-attention maps while the semantics are indicated through the cross-attention maps produced by the denoising U-Net, forming the basis of our segmentation results.Additionally, we carefully design effective textual prompts and a category filtering mechanism to further enhance the segmentation results. Extensive experiments on three benchmark datasets show that the proposed DiffSegmenter achieves impressive results for open-vocabulary semantic segmentation.
1 Introduction
DiffSegmenter turns pretrained text-to-image diffusion models into training-free open-vocabulary segmenters by combining shape information from self-attention with semantics from cross-attention. The paper motivates this approach as a way to reduce annotation dependence and reports results across three benchmarks and two settings.
Motivation
- Open-vocabulary segmentation can reduce reliance on extensive annotation and reach more categories, whereas prior generative approaches often require training, extra mask annotations, or complex data synthesis [Wu et al., 2023; Karazija et al., 2023; Baranchuk et al., 2021; Xu et al., 2023a].The introduction gives SegDiff and MedSegDiff as supervised examples, ODISE and SegDiff as requiring mask annotations, and DiffuMask and OVDiff as using data synthesis.Source: §1 · pp. 1–2
- Contrastively pretrained image-text models such as CLIP [Radford et al., 2021] may lose localization and object-shape information because they learn only the most discriminative visual features.Prior work investigated zero-shot [Ding et al., 2022; Liang et al., 2023] and weakly supervised [Lin et al., 2023] segmentation settings.Source: §1 · p. 1
- Diffusion models learn object shapes and semantics while modeling p(x|c), which the paper argues can be converted through Bayes’ theorem into pixel-level p(c|x) for segmentation.Large-scale image-text training can provide rich visual-semantic correspondence priors.Source: §1 · p. 2
DiffSegmenter
- DiffSegmenter produces open-vocabulary segmentation score maps without added learnable modules or parameter tuning, combining U-Net self-attention for object shapes with cross-attention for semantics.It also uses carefully designed textual prompts and BLIP-based category filtering to enhance results.Source: §1 · p. 2
Experiments
- Experiments on PASCAL VOC 2012, MS COCO 2014, and Pascal Context report impressive open-vocabulary segmentation results in zero-shot and image-level weakly supervised settings.The paper also notes potential for assisting downstream image editing.Source: §1 · p. 2
2 Related Work
Prior work adapts CLIP’s global image-text representations for pixelwise segmentation, while diffusion models have been used for perception and segmentation. Earlier diffusion segmentation methods have vocabulary or training requirements that DiffSegmenter avoids.
2.1 CLIP-based Open-vocabulary Segmentation
- CLIP aligns images with global category labels, but segmentation requires pixelwise predictions; adaptations use annotated or pseudo-label finetuning, external masks or grouping, or CAM refinement.Source: §2.1 · p. 2
- Contrastive alignment may discard localization information and object-shape awareness, contributing to unsatisfactory CLIP segmentation results.Source: §2.1 · p. 2
2.2 Diffusion Models for Perception Tasks
- Earlier diffusion segmentation methods are either not open-vocabulary [Baranchuk et al., 2021; Wu et al., 2023] or require training and a pretrained segmentation network [Xu et al., 2023a; Karazija et al., 2023], unlike DiffSegmenter.The proposed approach requires neither additional training nor a pretrained segmentation network.Source: §2.2 · p. 3
3 Methodology
DiffSegmenter uses a pretrained conditional diffusion model’s attention maps to produce training-free open-vocabulary segmentation, refining semantic localization with visual affinities and prompt/category strategies. Its Bayes-based diffusion baseline is computationally costly and can blur image structure.
3.1 Diffusion Model for Segmentation: A Baseline
- A Bayes-based diffusion segmentation baseline uses per-pixel ELBO estimates but requires many Monte Carlo samples and substantial computation.Diffusion can also compromise image structure, making fine-grained details and local pixel semantics less distinguishable or blurred.Source: §3.1 · p. 3
3.2 Proposed DiffSegmenter
- Cross-attention maps from a pretrained conditional diffusion model provide pixel-level class scores without additional learnable modules or parameter tuning.Under a uniform class prior, the method relates p(ci|x) to p(x|ci) and uses cross-attention activations as segmentation masks.Source: §3.2 · p. 4
3.3 Cross-attention-based Score Map Generation
- Cross-attention maps from multiple U-Net layers are fused because coarse maps localize objects while finer maps capture detail.Four layers at 8 × 8, 16 × 16, 32 × 32, and 64 × 64 are interpolated to a common size; their fused map initializes segmentation scores.Source: §3.3 · p. 4
3.4 Self-attention-based Score Map Completion
- Self-attention pixel affinities complete cross-attention score maps that may lack object boundaries or contain internal holes.The method multiplies cross-attention maps by affinities from selected encoder and decoder self-attention maps, producing final scores during pretrained-model denoising without additional modules or tuning.Source: §3.4 · pp. 4–5
3.5 Prompt Design for Semantic Enhancement
- Detailed object attributes and re-weighted class tokens are incorporated into prompts to improve target localization and suppress background.Adverbs and adjectives are fused with class-name attention maps, while the emphasized class token is denoted “[class]++”.Source: §3.5 · p. 5
3.6 Open-vocabulary Semantic Segmentation
- For zero-shot segmentation, the method assigns each pixel to the highest-scoring candidate among categories pre-filtered by BLIP captions and CLIP similarity above 0.97.It takes the union of BLIP-extracted nouns and CLIP-selected classes; for weakly supervised training, prompted ground-truth labels produce training segmentations for an off-the-shelf segmentation model.Source: §3.6 · p. 5
4 Experiments
DiffSegmenter is evaluated for zero-shot open-vocabulary and image-level weakly supervised segmentation, with additional analyses of its components and an image-editing application. The reported results show strong benchmark performance, while the analyses identify attention, prompt, and timestep choices that affect segmentation.
4.1 Datesets and Implementation Details
- Evaluations cover image-level weak supervision on VOC and zero-shot segmentation on VOC, Context, and Object; zero-shot evaluation uses validation sets and frozen Stable Diffusion v1.5.The validation sets contain 1,449 VOC, 5,105 Context, and 5,000 Object images; all experiments use mIoU.Source: §4.1 · p. 5
4.2 Results on Open-vocabulary Segmentation
- DiffSegmenter significantly improves mIoU over previous training-based approaches across VOC, Context, and Object, trailing concurrent OVDiff only on VOC and Context.The comparison is described as demonstrating strong zero-shot generalization; the paper characterizes DiffSegmenter as simpler and more efficient than OVDiff.Source: §4.2 · p. 5, Table 1 · p. 6
- Figure 4 shows DiffSegmenter producing more complete objects and clearer mask boundaries than the SegCLIP baseline in zero-shot segmentation.Additional qualitative results are reported in the supplementary material.Source: §4.2 · p. 5, Figure 4 · p. 6
4.3 Results on Weakly-supervised Segmentation
- DiffSegmenter achieves 70.5% mIoU on VOC training-set masks, and masks used to train DeepLabV2 yield 69.1% validation and 68.6% test mIoU.The 70.5% result is comparable to CLIP-ES, which also uses language supervision and synonym fusion, while DiffSegmenter uses only image-level supervision.Source: §4.3 · p. 6, Table 2 · p. 6, Table 3 · p. 6
4.4 Ablation Studies and Analyses
- On VOC training masks, cross-attention averaging reaches 61.25% mIoU; adding self-attention improves performance, and highest-resolution self-attention yields more complete boundaries.Weighted cross-attention across layers further improves results over averaging, while prompt completion and target-word reweighting help focus attention on target categories and suppress background or other objects.Source: §4.4 · pp. 6–7, Figure 5 · p. 7
- Segmentation is best at t=100 among the tested single timesteps, while averaging multiple timesteps gives the best results overall.The method uses multiple-timestep averaging for weak supervision and t=100 for zero-shot inference to accelerate inference.Source: §4.4 · p. 7, Table 5 · p. 7
4.5 Impacts on Image Editing
- DiffSegmenter’s more complete masks enable image editing that better preserves background and modifies foreground more accurately than directly replacing cross-attention with PTP.The comparison is illustrated for attention replacement in Figure 7.Source: §4.5 · p. 7, Figure 7 · p. 7
5 Conclusion
DiffSegmenter enables open-vocabulary semantic segmentation with readily available diffusion models, using denoising U-Net attention and BLIP-informed textual prompts without additional learnable modules or parameter tuning. Its reliance on latent input features may cause small objects to disappear, potentially addressable with diffusion models using original images or larger latent feature maps.
5 Conclusion
- DiffSegmenter uses diffusion-model denoising U-Net attention and BLIP-informed textual prompts for open-vocabulary segmentation without additional learnable modules or parameter tuning.The prompts are designed to enhance semantic quality.Source: §5 · p. 7
- Reliance on Stable Diffusion latent features may cause small objects to disappear in the latent space's small feature maps.The passage suggests using diffusion models based on original images or with larger latent feature maps as potential remedies.Source: §5 · p. 7