Source-linked AI summary
DiffuMask: Synthesizing Images with Pixel-level Annotations for Semantic Segmentation Using Diffusion Models
Weijia Wu, Yuzhong Zhao, Mike Zheng Shou, Hong Zhou, Chunhua Shen
TL;DR
Pixel-level segmentation data is costly to collect and annotate, motivating alternatives based on synthetic images. DiffuMask uses cross-attention in Stable Diffusion to generate high-resolution, class-discriminative masks automatically, and its synthetic-data-trained segmenters perform competitively with real-data counterparts while supporting open-vocabulary segmentation.
Problem
Semantic segmentation models require large amounts of pixel-level annotated data, but collecting images and producing pixel-wise masks are labor-intensive, expensive, and sometimes constrained by privacy or copyright.
Method
DiffuMask uses text-guided cross-attention from a text-supervised diffusion model to localize classes and generate high-quality pixel-level masks, with thresholding, noise learning, prompt retrieval, and augmentation.
Results
DiffuMask-trained segmentation methods perform competitively with real-data counterparts, achieve within a 3% gap for some classes, and reach new SOTA results on unseen VOC classes in open-vocabulary segmentation.
Takeaways & Limitations
DiffuMask demonstrates that text-driven diffusion models can automatically produce synthetic images with semantic annotations for training segmentation systems.
Abstract
from arXiv · showhide
Collecting and annotating images with pixel-wise labels is time-consuming and laborious. In contrast, synthetic data can be freely available using a generative model (e.g., DALL-E, Stable Diffusion). In this paper, we show that it is possible to automatically obtain accurate semantic masks of synthetic images generated by the Off-the-shelf Stable Diffusion model, which uses only text-image pairs during training. Our approach, called DiffuMask, exploits the potential of the cross-attention map between text and image, which is natural and seamless to extend the text-driven image synthesis to semantic mask generation. DiffuMask uses text-guided cross-attention information to localize class/word-specific regions, which are combined with practical techniques to create a novel high-resolution and class-discriminative pixel-wise mask. The methods help to reduce data collection and annotation costs obviously. Experiments demonstrate that the existing segmentation methods trained on synthetic data of DiffuMask can achieve a competitive performance over the counterpart of real data (VOC 2012, Cityscapes). For some classes (e.g., bird), DiffuMask presents promising performance, close to the stateof-the-art result of real data (within 3% mIoU gap). Moreover, in the open-vocabulary segmentation (zero-shot) setting, DiffuMask achieves a new SOTA result on Unseen class of VOC 2012. The project website can be found at https://weijiawu.github.io/DiffusionMask/.
1. Introduction
DiffuMask addresses the cost and difficulty of pixel-level semantic segmentation annotation by deriving masks from text-conditioned diffusion models. It combines cross-attention localization with mask refinement and data-generation techniques, achieving competitive segmentation results on real-data benchmarks.
- Motivation: Pixel-wise annotation is labor-intensive and expensive, while weakly supervised alternatives can have lower accuracy, complex training, and additional annotation or image-collection costs.Labeling one Cityscapes image can take up to 60 minutes.
- Core insight: Cross-attention maps localize text-specific image regions, with low-resolution maps providing class discrimination and higher-resolution maps capturing fine-grained details.The method combines maps from 8 × 8, 16 × 16, 32 × 32, and 64 × 64 resolutions.
- Method: Adaptive thresholding, noise learning, retrieval-based prompts, and data augmentation address mask precision and the domain gap between synthetic and real data.Noise learning filters noisy labels, while prompt diversity and splicing augmentations increase synthetic-data diversity.
- Method: DiffuMask automatically generates synthetic images and pixel-level semantic masks from a text-supervised diffusion model without manual mask annotation.It exploits cross-attention between text and image to extend text-driven image synthesis to semantic mask generation.
- Results: Segmentation methods trained on DiffuMask data perform competitively against counterparts trained on real data, including VOC 2012.For some classes, such as dog, performance is within a 3% gap of training with real data; the method also reports new SOTA results on unseen VOC classes in open-vocabulary segmentation.
2. Related Work
The related work reviews approaches for reducing segmentation annotation costs and for generating images or synthetic datasets. It positions synthetic data as an increasingly available source of automatically generated pixel-level ground truth.
- Reducing Annotation Cost: Weak or cheap supervision includes image-level labels, points, scribbles, and bounding boxes, but these approaches involve accuracy, training-complexity, or annotation-cost trade-offs.Image-level supervision has the lowest cost among the cited alternatives but unacceptable performance, while bounding boxes can remain expensive.
- Image Generation: Image generation spans GANs, VAEs, flow-based models, and diffusion probabilistic models, with Stable Diffusion operating in VAE latent space to reduce resource consumption.The related work highlights text-to-image systems such as GLIDE, DALL-E 2, and Imagen alongside Stable Diffusion.
3. Methodology
DiffuMask generates synthetic images and pixel-level semantic masks by exploiting Stable Diffusion’s text-guided cross-attention. It refines attention-derived masks with adaptive thresholds, affinity learning, noise filtering, and prompt engineering.
- Core framework: DiffuMask jointly generates synthetic images and semantic masks using cross-attention from a pre-trained text-to-image diffusion model.The approach does not require pixel-level annotations and can produce data for training segmentation architectures.
- Cross-attention: Cross-attention maps provide class-discriminative localization, while aggregating maps across U-Net layers and diffusion steps supports mask generation.The maps correspond to textual tokens and are extracted at resolutions from 8 × 8 to 64 × 64.
- Mask generation and refinement: Adaptive thresholding is used because the optimal binarization threshold varies across images and object categories.Observed examples include thresholds around 0.48 for Bottle and 0.39 for Dog; affinity matching produces thresholds usually close to optimal.
- Mask generation and refinement: AffinityNet estimates suitable thresholds by matching predicted affinity maps with binary attention masks across a threshold search space.This targets uncertainty in pixels with middle confidence scores while preserving high- and low-confidence regions.
- Noise learning: Noise Learning improves mask quality by estimating label noise, ranking examples by confidence, and pruning noisy synthetic annotations.The described procedure filters examples using out-of-sample segmentation predictions and class-specific confidence ranking.
4. Experiments
Experiments evaluate DiffuMask on semantic, open-vocabulary, and domain-generalization segmentation across VOC 2012, Cityscapes, and ADE20K, alongside ablations of its design choices. Synthetic-data training is competitive with real-data training, while adaptive thresholding, prompt engineering, augmentation, and stronger backbones improve performance.
- Experimental Setup: DiffuMask is evaluated on semantic segmentation, open-vocabulary segmentation, and domain generalization using VOC 2012, ADE20K, and Cityscapes.The evaluation uses mIoU, including seen, unseen, and harmonic-mean scores for open-vocabulary segmentation.
- Semantic Segmentation: 70.6% mIoU versus 84.3% with Swin-B shows competitive VOC 2012 semantic-segmentation performance for DiffuMask-trained models.For bird, cat, cow, horse, and sheep, performance is within a 5% gap of real-data training; fine-tuning 5.0k real images reaches 84.9% mIoU versus 83.4% with 10.6k real images.
- Semantic Segmentation: 79.6% versus 90.8% mIoU demonstrates competitive Cityscapes performance for Vehicle and Human segmentation.Cityscapes contains small objects and complex backgrounds, and the experiment evaluates these two driving-scene categories.
- Semantic Segmentation: 73.4% mIoU is achieved for car on ADE20K using 6k synthetic images, compared with training on 20.2k real images.ADE20K evaluates bus, car, and person categories.
- Open-vocabulary Segmentation: DiffuMask achieves a state-of-the-art result on unseen VOC 2012 classes in open-vocabulary segmentation using purely synthetic, text-supervised data.Prior methods use real images and manually annotated masks, while a contemporaneous pseudo-labeling approach uses a segmentation model pre-trained on COCO.
- Domain Generalization: 69.5% versus 68.0 mIoU on VOC 2012 validation shows effective domain generalization from DiffuMask compared with ADE20K training.For Motorbike, DiffuMask reaches 63.2% mIoU versus 28.9% from Cityscapes-only training; the paper attributes this to differing foreground and background domain shifts.
- Ablation Study: Adaptive thresholding is robust and close to the optimal threshold, while retrieval-based and subclass prompts improve performance and subclass prompts improve dog mIoU by 7.7%.Image splicing provides the largest augmentation gain, and stronger backbones improve classification, false-negative handling, and mask precision; mask precision and domain gap account for 6.4% and 4.5% mIoU gaps, respectively.
5. Conclusion
DiffuMask demonstrates that text-driven diffusion models can automatically produce accurate pixel-level semantic annotations for generated images. Models trained on this synthetic data perform competitively with real-data counterparts and show promise for open-vocabulary segmentation.
- DiffuMask automatically obtains accurate semantic masks for generative images using a text-driven diffusion model.
- Segmentation methods trained on DiffuMask synthetic data achieve competitive performance compared with counterparts trained on real data.
- DiffuMask shows promising performance for open-vocabulary segmentation on unseen categories.