PaperlayerView exact revision on arXiv

Source-linked AI summary

DatasetDM: Synthesizing Data with Perception Annotations Using Diffusion Models

Weijia Wu, Yuzhong Zhao, Hao Chen, Yuchao Gu, Rui Zhao, Yefei He, Hong Zhou, Mike Zheng Shou, Chunhua Shen

arXiv:2308.06160v2cs.CV

TL;DR

Perception models need large labeled datasets, but collecting and annotating them is costly and difficult. DatasetDM decodes pretrained diffusion-model latents into perception annotations using less than 1% labeled data, enabling large-scale synthetic dataset generation. It delivers strong results across tasks and datasets, while supporting domain generalization and flexible task composition.

  • Problem

    Perception models require large amounts of labeled data, but collecting images and pixel-level annotations is costly, time-consuming, and difficult in specialized or privacy-sensitive domains.

  • Method

    DatasetDM uses a unified perception decoder to decode latent codes from a pretrained text-to-image diffusion model into annotations, trained with less than 1% manually labeled images.

  • Results

    DatasetDM synthetic data performs strongly across six datasets and five downstream tasks, including gains of 13.3% mIoU for VOC 2012 semantic segmentation and 12.1% AP for COCO 2017 instance segmentation.

  • Takeaways & Limitations

    Text-guided data generation provides diverse synthetic data for domain generalization and supports flexible task composition such as image editing.

  • Takeaways & Limitations

    The study does not report experiments with the higher-resolution, more semantically aligned DeepFloyd IF model because of time constraints.

Abstract

from arXiv · show

Current deep networks are very data-hungry and benefit from training on largescale datasets, which are often time-consuming to collect and annotate. By contrast, synthetic data can be generated infinitely using generative models such as DALL-E and diffusion models, with minimal effort and cost. In this paper, we present DatasetDM, a generic dataset generation model that can produce diverse synthetic images and the corresponding high-quality perception annotations (e.g., segmentation masks, and depth). Our method builds upon the pre-trained diffusion model and extends text-guided image synthesis to perception data generation. We show that the rich latent code of the diffusion model can be effectively decoded as accurate perception annotations using a decoder module. Training the decoder only needs less than 1% (around 100 images) manually labeled images, enabling the generation of an infinitely large annotated dataset. Then these synthetic data can be used for training various perception models for downstream tasks. To showcase the power of the proposed approach, we generate datasets with rich dense pixel-wise labels for a wide range of downstream tasks, including semantic segmentation, instance segmentation, and depth estimation. Notably, it achieves 1) state-of-the-art results on semantic segmentation and instance segmentation; 2) significantly more robust on domain generalization than using the real data alone; and state-of-the-art results in zero-shot segmentation setting; and 3) flexibility for efficient application and novel task composition (e.g., image editing). The project website and code can be found at https://weijiawu.github.io/DatasetDM_page/ and https://github.com/showlab/DatasetDM, respectively

1 Introduction

DatasetDM addresses the cost and scarcity of pixel-level labels by extending text-to-image diffusion into text-to-data generation with perception annotations. Its synthetic data supports strong performance across downstream tasks, while text-guided generation adds diversity and flexibility.

  • Motivation: Large-scale perception datasets are expensive and time-consuming to collect and annotate, especially for pixel-level labels and privacy-sensitive domains.Annotating one Cityscapes urban image can take up to 60 minutes.
  • Approach: DatasetDM extends text-to-image diffusion into text-to-data generation by decoding rich diffusion-model latent codes into perception annotations.The approach uses a unified perception decoder built on a pretrained diffusion model.
  • Approach: Less than 1% manually labeled data, around 100 images, is used to train the perception decoder for generating annotated synthetic data.This training strategy is described as visual align/instruct tuning.
  • Results: Synthetic data performs strongly across six datasets and five downstream tasks, including semantic and instance segmentation.The reported gains include 13.3% mIoU on VOC 2012 semantic segmentation and 12.1% AP on COCO 2017 instance segmentation.
  • Results: Text-guided generation produces diverse data for domain generalization and supports novel task compositions such as image editing.The paper presents these capabilities as additional advantages of the generation framework.

2 Related Work

Prior perception-dataset methods rely on simulations or generative models, while diffusion models also appear in direct perception approaches. DatasetDM differs by using diffusion primarily for synthetic data augmentation through a unified decoder.

  • Diffusion models: Text-guided diffusion models generate diverse, high-fidelity images from prompts by combining visual and textual embeddings through cross-attention.The passage identifies Stable Diffusion, DALL-E2, and Imagen as examples.
  • Synthetic datasets: Simulation-based datasets such as Virtual KITTI target standard 2D vision tasks but are limited by their 3D model domains and scene categories.The cited examples cover detection, scene understanding, and optical flow estimation.
  • Synthetic datasets: Generative-model approaches support flexible open-world synthetic data for tasks including classification, face recognition, salient object detection, segmentation, and human pose.The passage contrasts GAN- and diffusion-generated data with simulation-based datasets.
  • Diffusion models: Some methods apply diffusion models directly to perception, including perception modeling and open-vocabulary panoptic segmentation.VPD and ODISE are cited as representative approaches.
  • DatasetDM: DatasetDM instead focuses on synthetic data augmentation and uses a unified transformer-based decoder to support more perception tasks.This distinguishes its data-generation focus from direct diffusion-based perception models.

3 Methodology

DatasetDM extends text-guided diffusion from image synthesis to annotated data generation by decoding diffusion representations with a generalized perception decoder. It trains this decoder from limited labeled data, then uses language-guided generation to produce diverse images and task-specific annotations.

  • Text-to-data formulation: The method extends text-to-image generation into text-to-data generation by decoding diffusion-model latent representations into perception annotations.Supported outputs include semantic and instance segmentation, depth, and human pose annotations.
  • Training: DatasetDM uses diffusion inversion to extract latent representations from real images and trains a perception decoder with corresponding annotations.The training stage obtains latent codes and text-image representations before fitting the decoder.
  • Hypercolumn representation: The extracted representation combines multi-scale U-Net feature maps with averaged cross-attention maps, fused using a 1 × 1 convolution.Feature maps come from four resolutions, while cross-attention maps are grouped by resolution before averaging.
  • Perception decoder: The P-Decoder is a generalized architecture whose minor layer changes support segmentation, depth estimation, and pose estimation.Generic segmentation uses both pixel and transformer decoders, whereas depth and pose estimation require only the pixel decoder.
  • Perception decoder: Open-vocabulary segmentation equips learned queries with class semantics by fusing query embeddings with text-encoder class embeddings.Adding a new class name enables DatasetDM to generate a corresponding open-vocabulary mask.
  • Text-guided data generation: During inference, GPT-4 supplies diverse prompts while extended denoising generates synthetic images whose annotations are produced by the P-Decoder.This differs from training, which uses template prompts and diffusion inversion of real images.

4 Experiments

DatasetDM is evaluated across segmentation, depth, pose, zero-shot, long-tail, and ablation settings using synthetic data generated from limited real training images. Results show consistent gains across downstream tasks, with performance depending on diffusion features, cross-attention, training-set size, and prompt design.

  • Evaluation settings: DatasetDM is evaluated on semantic segmentation, instance segmentation, depth estimation, human pose estimation, zero-shot segmentation, and long-tail segmentation.Experiments use datasets including Pascal-VOC 2012, Cityscapes, COCO 2017, NYU Depth V2, and DeepFashion-MM.
  • Semantic segmentation: 73.7% mIoU is achieved on VOC 2012 using purely synthetic data from DatasetDM, an 8.5% improvement over real data alone.Joint training with 100 real images further raises performance to 78.5% mIoU.
  • Instance segmentation: 12.1% AP is gained for COCO 2017 instance segmentation, reaching 26.5% with 800 training images and a Swin-B backbone.The improvement is approximately 10% across the three reported training settings.
  • Depth estimation: 10% improvement is obtained on NYU Depth V2 depth estimation when training with 50 images compared with training solely on real images.The comparison evaluates synthetic and real data under the reported downstream-task setup.
  • Human pose estimation: 5.1% improvement is achieved for COCO 2017 human pose estimation compared with a baseline trained on 800 real images.The result is reported for DatasetDM-generated synthetic data.
  • Zero-shot and long-tail segmentation: Up to 20% mIoU improvement is reported for zero-shot and long-tail segmentation by synthesizing substantial data for rare classes.The long-tail setting addresses rare-class representation through additional synthetic samples.
  • Ablation studies: 81% performance is reached with 1k training images, while increasing training data from 60 to 400 images produces the clearest improvement before saturation.The 1k-image training set represents roughly 10% of the original data.
  • Ablation studies: Approximately 4% enhancement follows increasing prompt length from 5 to 10 words, while longer prompts show no stated continued improvement.Cross-attention maps provide roughly 1% improvement at step 1, but can become detrimental beyond step 500.

5 Conclusion

The paper concludes that DatasetDM extends text-to-image diffusion into text-guided perception-data generation through a perception decoder. It reports strong downstream performance and broader flexibility, including domain generalization and image editing.

  • Conclusion: DatasetDM parses a diffusion model’s latent space with a perception decoder to extend text-to-image synthesis into text-guided data generation.The generated outputs include synthetic images paired with perception annotations.
  • Conclusion: The evaluated perception models show strong performance across six datasets and five downstream tasks using DatasetDM-generated synthetic data.The conclusion summarizes results across the paper’s downstream evaluations.
  • Conclusion: 13.3% mIoU improvement is reported for semantic segmentation on VOC 2012, alongside a 12.1% AP improvement for instance segmentation on COCO 2017.These are the representative headline results highlighted in the conclusion.
  • Conclusion: The paper reports more robust domain generalization and enhanced image-editing capabilities as additional advantages of text-guided data generation.These advantages are stated as broader applications of the proposed generation paradigm.

A.3 Training Setup for DatasetDM

DatasetDM training uses standard augmentation and task-specific downstream configurations. The appendix specifies the baseline models, data transformations, and evaluation-table metrics for semantic segmentation, depth estimation, and related tasks.

  • Data augmentation: DatasetDM training uses random cropping and random scaling for semantic and instance segmentation.The crop size is 512×512 pixels.
  • Data augmentation: Depth-estimation training uses random flipping, cropping, brightness-contrast adjustment, and hue-saturation manipulation.These are the four listed depth-estimation augmentations.
  • Data augmentation: Pose-estimation training uses random scaling, cropping, flipping, and rotation-related augmentation.The passage lists the pose-specific augmentation sequence.
  • Evaluation details: The reported appendix tables cover DeepFashion-MM semantic segmentation and NYU Depth V2 depth estimation with task-specific metric columns.The depth table measures the listed accuracy and error metrics on the NYU Depth V2 validation dataset.

A.4 Details for Training Data of DatasetDM

DatasetDM generally trains downstream perception models with less than 1% of the available real data, excluding the seen class in the zero-shot segmentation setting. This reduces the amount of real data required for the reported tasks.

  • Training-data quantities: Less than 1% of available real data is required for DatasetDM training in nearly all downstream tasks, except the seen class in zero-shot segmentation.The paper connects this data requirement to lower implementation costs and improved data utilization.

B.1 Comparison with Other Data Augmentation Methods.

DatasetDM functions as an effective data augmentation method, outperforming flip and color augmentation by around 10%.

  • Around 10% improvement: DatasetDM substantially outperforms flip and color augmentation.The comparison treats DatasetDM as an efficient data augmentation technique.

B.2 Ablation Study for Baseline of Downstream Tasks.

Across downstream perception settings, synthetic data from DatasetDM consistently improves performance, supports limited-data training, and generalizes across domains and tasks.

  • Approximately 10%: DatasetDM synthetic data enhances DeepLabV3 performance.The improvement is reported across benchmark models, supporting integration with existing downstream task models.
  • Approximately 5% AP improvement: DatasetDM improves human pose estimation across backbone and input-size settings.The method reaches 48.9% AP using 800 training images.
  • Approximately 5% mIoU improvement: joint training with synthetic data outperforms training using purely synthetic data.
  • 450 images: DatasetDM is evaluated for zero-shot semantic segmentation with 30 images for each of 15 seen classes.Testing covers all 20 categories.
  • 73.6% versus 69.5%: DatasetDM outperforms DiffuMask on VOC 2012 validation for domain generalization.The paper also reports enhanced robustness compared with real data.
  • 78.2 mIoU: DatasetDM achieves this performance with only 100 images using a more powerful backbone.The method also reports competitive performance with 60 images.
  • Unified synthesis spans human-centric and urban-city domains, supporting multi-task data generation and custom domain design through prompts.The paper highlights applications to bad-case scenarios and long-tail data distributions.

C Details on the Architecture of Perception Decoder

The P-Decoder combines text, pixel, and transformer decoders to convert diffusion representations into perception outputs across multiple tasks.

  • P-Decoder consists of a pixel decoder, text encoder, and transformer decoder.
  • Open-vocabulary inputs encode class names with CLIP, replicate the resulting token, and fuse it with learnable query embeddings through an MLP.
  • Multi-scale features and cross-attention maps are decoded into per-pixel and mask embeddings.
  • Two added convolutional layers extend the segmentation architecture to human pose and depth estimation outputs.

D Synthetic Dataset

DatasetDM uses GPT-4-guided prompts to generate large, customizable synthetic datasets while acknowledging ethical concerns inherited from the pretrained diffusion model.

  • GPT-4 expands a small human-provided prompt clue into massive numbers of downstream-task prompts.The stated prompt clue costs no more than 50 words.
  • Class-sensitive tasks receive around 100 generated descriptions per class, while class-insensitive tasks receive many descriptions at once.
  • Prompts are tailored to different data domains and released with the corresponding code and prompt files.
  • Synthetic image generation raises privacy and copyright concerns because DatasetDM uses Stable Diffusion trained on the 5-billion-image LAION dataset.The paper proposes prompt design and removal of privacy-sensitive concepts as mitigation strategies.

E.2 Limitation & Future Work

The study identifies limitations tied to synthesized-data quality and the choice of text-to-image model, while proposing stronger models and better domain-aligned prompts as future directions.

  • Limitation: Synthesized data still cannot match real data in quality and complexity.The authors state that organizations able to collect and manually annotate massive datasets could achieve better results.
  • Prompt generation: GPT-4 is used to generate diverse diffusion-model prompts from simple cues.The paper asks whether GPT-4 can produce good prompts and illustrates its familiarity with COCO2017 as a target benchmark.
  • Limitation: DeepFloyd IF could improve synthesis through stronger semantic alignment and 1024 resolution, but the paper reports no experiments with it.The authors attribute the missing experiments to time constraints after IF's release.
  • Future Work: Future work includes using more robust image-generation models and improving prompt-generation efficiency or target-domain alignment.The authors suggest designing prompts specifically for datasets such as COCO2017.
Loading 2308.06160v2…