PaperlayerView exact revision on arXiv

Source-linked AI summary

Dataset Diffusion: Diffusion-based Synthetic Dataset Generation for Pixel-Level Semantic Segmentation

Quang Nguyen, Truong Vu, Anh Tran, Khoi Nguyen

arXiv:2309.14303v4cs.CV

TL;DR

Pixel-level annotation for semantic segmentation is expensive, motivating synthetic-data generation beyond image-level labels. Dataset Diffusion uses Stable Diffusion prompts and attention maps to generate images with segmentation pseudo-labels, achieving strong results on VOC and COCO while retaining limits on complex scenes and intertwined objects.

  • Problem

    Pixel-level semantic-segmentation annotation is labor-intensive, while existing generative methods mainly provide image-level labels or single-object masks.

  • Method

    Dataset Diffusion uses Stable Diffusion with class-prompt appending, cross-attention, self-attention refinement, and uncertainty-aware training to synthesize images and pixel-level pseudo-labels.

  • Results

    64.8 mIoU on VOC and 34.2 mIoU on COCO demonstrate effective synthetic semantic-segmentation datasets, with VOC outperforming DiffuMask by 4.2 mIoU.

  • Takeaways & Limitations

    The framework advances synthetic dataset generation from image-level to pixel-level annotation and provides synth-VOC and synth-COCO benchmarks for evaluation.

  • Takeaways & Limitations

    Stable Diffusion may omit objects in prompts involving complex scenes and may produce poor masks when objects are closely intertwined.

Abstract

from arXiv · show

Preparing training data for deep vision models is a labor-intensive task. To address this, generative models have emerged as an effective solution for generating synthetic data. While current generative models produce image-level category labels, we propose a novel method for generating pixel-level semantic segmentation labels using the text-to-image generative model Stable Diffusion (SD). By utilizing the text prompts, cross-attention, and self-attention of SD, we introduce three new techniques: class-prompt appending, class-prompt cross-attention, and self-attention exponentiation. These techniques enable us to generate segmentation maps corresponding to synthetic images. These maps serve as pseudo-labels for training semantic segmenters, eliminating the need for labor-intensive pixel-wise annotation. To account for the imperfections in our pseudo-labels, we incorporate uncertainty regions into the segmentation, allowing us to disregard loss from those regions. We conduct evaluations on two datasets, PASCAL VOC and MSCOCO, and our approach significantly outperforms concurrent work. Our benchmarks and code will be released at https://github.com/VinAIResearch/Dataset-Diffusion

1 Introduction

Dataset Diffusion addresses the high cost of pixel-level annotation by generating realistic multi-object images with segmentation masks from Stable Diffusion. Its synthesized data supports semantic-segmenter training and achieves strong benchmark results on synth-VOC and synth-COCO.

  • Pixel-level semantic-segmentation annotation is labor-intensive, time-consuming, and costly for large-scale datasets.
  • Existing GAN-based approaches mitigate annotation needs but primarily generate object-centric images rather than complex real-world scenes.
  • Text-to-image diffusion models provide textual object guidance and spatial cues from cross- and self-attention for synthetic segmentation.
  • Dataset Diffusion generates realistic multi-object images and precise masks using class-prompt appending, class-prompt cross-attention, and self-attention exponentiation.Generated data is used to train a semantic segmenter with uncertainty-aware segmentation loss and self-training.
  • 64.8 mIoU on synth-VOC outperforms DiffuMask, while DeepLabV3 trained on synth-COCO reaches 34.2 mIoU against full-supervision real-image training.
  • The framework introduces synth-VOC and synth-COCO benchmarks for evaluating semantic-segmentation dataset synthesis.

2 Related Work

Prior work spans semantic-segmentation architectures, diffusion-based segmentation, GAN-generated datasets, and image-level synthetic data. Dataset Diffusion extends this direction toward pixel-level annotations for multi-object scenes without ground-truth segmentation labels.

  • Semantic segmentation methods include convolutional, transformer-based, and encoder-decoder architectures, with Dataset Diffusion designed to support any semantic segmenter.
  • Text-to-image diffusion models generate multi-object interactions more realistically than earlier single-object image generators.
  • Diffusion models used for segmentation still generally require ground-truth masks, whereas Dataset Diffusion generates semantic segmentation without GT labels.
  • GAN-based synthetic segmentation methods mainly target object-centric images with masks for salient objects or parts of common objects.
  • Synthetic-data diffusion methods for classification produce image-level annotations, while Dataset Diffusion targets the more challenging pixel-level annotations.
  • Concurrent Stable Diffusion methods generally segment one object per image, whereas Dataset Diffusion uses prompts containing multiple objects.

3 Dataset Diffusion

Dataset Diffusion synthesizes images and pixel-level semantic masks through a three-stage pipeline using prompts and Stable Diffusion attention maps. The resulting masks train a semantic segmenter with uncertainty handling and self-training.

  • Three-stage framework: The pipeline prepares prompts, generates images and segmentations with Stable Diffusion, then trains a semantic segmenter on the synthetic dataset.Generated images are I_i and masks are S_i, with background represented by class 0.
  • Prompt preparation: Class-prompt appending adds every target class to captions, addressing missing or mismatched class names in provided or generated captions.The augmented prompt is P′_i = [P_i; C_i].
  • Attention-based segmentation: Class-prompt cross-attention extracts attention maps using only the class-name portion of the prompt while preserving the original prompt for image generation.The resulting maps contain one channel for each of the M classes in the image.
  • Attention-based segmentation: Self-attention refinement propagates cross-attention responses to similar latent positions, improving coarse object localization before mask construction.Self-attention exponentiation powers the self-attention map by τ before multiplying it with cross-attention.
  • Uncertainty-aware masks: Objectness and class maps are converted into masks using lower and upper thresholds, with intermediate values assigned an uncertainty label.Locations with V_x ≤ α become background, α < V_x < β become uncertain, and the remaining locations retain their predicted class.
  • Training: Self-training further refines the generated masks by using the initially trained segmenter to produce pseudo-labels without uncertainty values.The final segmenter is trained again on these self-generated labels.

4 Experiments

Experiments evaluate Dataset Diffusion on synth-VOC and synth-COCO using mIoU, ablations, qualitative examples, and analyses of attention, prompt selection, feature scales, and mask-generation hyperparameters. The method reaches 64.8 mIoU on VOC and 34.2 mIoU on COCO, while ablations show benefits from prompt design, self-attention refinement, uncertainty-aware loss, self-training, and test-time augmentation.

  • Experimental setup: Dataset Diffusion is evaluated on PASCAL VOC 2012 and COCO 2017 using mIoU, with synth-VOC and synth-COCO constructed as benchmarks for semantic-segmentation dataset synthesis.The benchmarks use standardized text prompts and ground-truth segmentation evaluation.
  • Main results: 64.8 mIoU on VOC and 34.2 mIoU on COCO are achieved by Dataset Diffusion, compared with 79.9 and 54.9 mIoU from the corresponding real training sets.The remaining gaps are 15 mIoU on VOC and 20 mIoU on COCO, attributed to the greater complexity of COCO images.
  • Main results: 4.2 mIoU separates Dataset Diffusion from DiffuMask on VOC with the same ResNet50 backbone.The comparison uses synthetic datasets to train semantic segmenters.
  • Qualitative analysis: The qualitative results include accurate masks but failures involving numerous small objects, intertwined objects, and harmful self-training effects on small objects.Self-training can correct some mis-segmented objects but can also damage original masks for small objects.
  • Ablation study: Class-prompt appending improves mIoU by 11.2 over caption-only prompts, 4.6 over class-label-only prompts, and 7.3 over simple prompts.The results indicate that this prompt design supports both dataset diversity and accurate attention.
  • Ablation study: 44.8 mIoU from cross-attention alone increases to 61.0 mIoU with self-attention refinement, while uncertainty-aware loss, self-training, and TTA provide further incremental improvements.The cross-attention map is described as coarse and inaccurate before refinement.
  • Ablation study: Feature-scale analysis finds that overly small or large cross-attention scales hurt performance, while self-attention at scale 32 performs slightly better.Small scales lack details, whereas large scales focus on fine details instead of object shape.
  • Mask-generation analysis: Sensitivity analysis selects τ = 4 because τ = 5 causes the refined cross-attention map to spread across the image rather than remain on the object.The analysis also evaluates α and β ranges for mask generation.

5 Discussion and Conclusion

Dataset Diffusion generates synthetic semantic-segmentation datasets with realistic images and pixel-level masks, achieving 64.8 mIoU on VOC and 34.2 mIoU on COCO. Its main limitations are Stable Diffusion’s difficulty with complex or intertwined scenes and possible transfer of LAION-5B bias.

  • Limitations: Stable Diffusion may generate only two or three objects from prompts containing three or more objects, limiting complex-scene generation.Stronger guidance such as layout or box conditions is identified as a possible direction for addressing this issue.
  • Limitations: Dataset Diffusion may produce low-quality masks when objects are closely intertwined, and bias from the LAION-5B training data may transfer to generated datasets.These limitations are described as inherited or associated with the underlying Stable Diffusion model.
  • Conclusion: Dataset Diffusion generates synthetic semantic-segmentation datasets with realistic images and pixel-level annotations using Stable Diffusion.The conclusion reports mIoU values of 64.8 on VOC and 34.2 on COCO.
  • Conclusion: 64.8 mIoU on VOC and 34.2 mIoU on COCO are reported as superior results compared with the concurrent method DiffuMask.The conclusion presents these results as supporting future work on large-scale datasets with precise annotations.

6 Supplementary Material

The supplementary material reports implementation details, ablations, per-class evaluations, cross-domain results, and qualitative examples. It shows limited gains from larger synthetic datasets, class-specific weaknesses, and domain-dependent performance gaps.

  • Additional Ablation Studies: Averaging attention maps across all timesteps performs best, with alternative timestep ranges causing at most a 0.5 mIoU decrease.This ablation examines both self-attention and cross-attention aggregation.
  • Additional Ablation Studies: A fourfold increase from 10k to 40k generated images yields only a 1.0% mIoU gain, while higher-quality masks produce a 1.4 mIoU gain.The authors attribute the small gain primarily to mask quality rather than dataset size and stop at 40k images because of computation cost.
  • Detailed Per-class IoU on the VOC and COCO datasets: The “tv/monitor” class suffers when cross-attention focuses on object boundaries rather than the full object, and self-attention does not correct the resulting masks.The authors connect this behavior to a significant performance drop relative to training on the real dataset.
  • Other Image Domains: Synthetic-data performance trails real-data baselines by approximately 19 mIoU for driving scenes and around 16.5 mIoU for facial-part segmentation.Dataset Diffusion nevertheless performs comparably for selected driving-scene classes and reaches 78.2 versus 87.0 in the facial-part comparison.
  • Other Image Domains: Satellite/aerial segmentation has a 37.4 mIoU gap because generated images and segmentations do not match DroneDeploy’s standards.The authors attribute this discrepancy to limited aerial-image representation in Stable Diffusion’s training data and prompt-style mismatch.
  • More Qualitative Results: Qualitative results are strongest for single-object prompts, remain acceptable for some multi-object scenes, and degrade when complex prompts omit requested objects.The omitted objects consequently lack corresponding masks.
Loading 2309.14303v4…