Source-linked AI summary
DatasetGAN: Efficient Labeled Data Factory with Minimal Human Effort
Yuxuan Zhang, Huan Ling, Jun Gao, Kangxue Yin, Jean-Francois Lafleche, Adela Barriuso, Antonio Torralba, Sanja Fidler
TL;DR
Pixel-wise annotation is expensive and limits the scale of segmentation datasets. DatasetGAN decodes semantic labels from GAN features using a decoder trained on only a few labeled images, then generates arbitrarily many image-annotation pairs. It achieves performance comparable to fully supervised methods and sometimes exceeds baselines while using far less annotated data.
Problem
Pixel-wise labels are laborious and expensive to create, while deep networks benefit from large-scale datasets.
Method
DatasetGAN trains an ensemble of MLP classifiers on pixel-wise StyleGAN feature vectors from a few human-annotated images, then uses the trained interpreter to generate labeled image-annotation pairs.
Results
DatasetGAN outperforms semi-supervised baselines significantly and is on par with fully supervised methods, in some cases using two orders of magnitude less annotated data.
Takeaways & Limitations
The method enables detailed segmentation datasets and successful downstream computer vision models from very small annotated sets, including training with as few as 16 labeled examples.
Takeaways & Limitations
Image quality can interfere with labeling; blurry and unnatural synthesized bird legs led to sparse leg labels and affected test-time performance for that part.
Abstract
from arXiv · showhide
We introduce DatasetGAN: an automatic procedure to generate massive datasets of high-quality semantically segmented images requiring minimal human effort. Current deep networks are extremely data-hungry, benefiting from training on large-scale datasets, which are time consuming to annotate. Our method relies on the power of recent GANs to generate realistic images. We show how the GAN latent code can be decoded to produce a semantic segmentation of the image. Training the decoder only needs a few labeled examples to generalize to the rest of the latent space, resulting in an infinite annotated dataset generator! These generated datasets can then be used for training any computer vision architecture just as real datasets are. As only a few images need to be manually segmented, it becomes possible to annotate images in extreme detail and generate datasets with rich object and part segmentations. To showcase the power of our approach, we generated datasets for 7 image segmentation tasks which include pixel-level labels for 34 human face parts, and 32 car parts. Our approach outperforms all semi-supervised baselines significantly and is on par with fully supervised methods, which in some cases require as much as 100x more annotated data as our method.
1. Introduction
DatasetGAN addresses the cost and scale limits of pixel-wise annotation by using a few labeled examples to generate large, detailed synthetic segmentation datasets. The approach leverages semantic knowledge learned by GANs and supports downstream vision tasks with strong benchmark performance.
- Motivation: Pixel-wise dataset curation is laborious and expensive, with complex scenes taking 30 to 90 minutes to label.This annotation burden limits the scale of datasets that can be created.
- Approach: DatasetGAN uses GAN semantic knowledge and a shallow decoder trained on a few labeled images to produce pixel-level labels across the latent space.The resulting procedure is described as an infinite annotated dataset generator.
- Dataset scope: The generated datasets support rich object and part segmentation, including 34 human face parts and 32 car parts across 7 segmentation tasks.Only a few examples need manual labeling, enabling highly detailed annotations.
- Results: DatasetGAN outperforms semi-supervised baselines significantly and matches fully supervised methods while sometimes using two orders of magnitude less annotated data.The generated datasets can train computer vision architectures as real datasets are used.
- Downstream application: Successful computer vision models can be trained with as little as 16 labeled examples, supporting downstream applications such as 3D reconstruction of animatable objects.The showcased application exploits the detailed part labels produced by DatasetGAN.
2. Related Work
Related work includes synthetic data generation, semi-supervised segmentation, contrastive learning, and GAN-based semantic decoding. DatasetGAN differs by directly using GAN features to generate highly detailed labeled image datasets from few human-provided examples.
- Generative data synthesis: Earlier synthetic-data work mainly generated labeled images from 3D scene graphs and graphics-based rendering.DatasetGAN instead focuses on GANs that synthesize high-quality images after adversarial training.
- Generative data synthesis: Prior GAN methods supported domain adaptation, multi-view dataset generation, or zero-shot visual-feature synthesis rather than detailed direct image annotation.The cited StyleGAN work generated multi-view images for inverse graphics, while other methods synthesized features for unseen classes.
- Semi-supervised learning: Semi-supervised segmentation learns from small annotated sets plus unlabeled images using adversarial training, pseudo-labels, or consistency regularization.These approaches do not use generative modeling of images themselves in the way DatasetGAN does.
- GAN-based semantic decoding: DatasetGAN directly interprets disentangled per-pixel StyleGAN feature vectors with an ensemble of MLP classifiers to synthesize detailed labels and datasets.This contrasts with concurrent convolutional and residual decoders that project internal StyleGAN layers into segmentation maps.
- Contrastive learning: Contrastive learning improves representations with unsupervised similarity losses and can improve classification after fine-tuning on few labeled examples.The supplied related-work passage identifies classification as a prominent application and notes extension to segmentation.
3. Our Approach
DatasetGAN combines a StyleGAN image generator with a learned Style Interpreter that converts pixel-wise GAN features into labels. After training on a few annotated examples, it can generate arbitrarily many image-annotation pairs for downstream vision models.
- Core approach: DatasetGAN trains a Style Interpreter on StyleGAN pixel-wise feature vectors to match human-provided labels.The interpreter uses an ensemble of MLP classifiers with shared weights across pixels.
- Style Interpreter: The method upsamples and concatenates StyleGAN feature maps so each output pixel has a feature vector for label prediction.The resulting tensor is built at the highest feature-map resolution.
- Training: The training procedure samples feature vectors from labeled regions and does not backpropagate gradients into the StyleGAN backbone.An ensemble of 10 classifiers uses majority voting for segmentation and averaged heat values for keypoints.
- Core approach: A few annotated GAN-generated images can train feature classifiers that make accurate predictions.The paper reports this behavior in Figures 4 and 5 and validates it experimentally.
- Dataset generation: After training, the Style Interpreter becomes a label-synthesis branch, allowing StyleGAN to generate any desired number of image-annotation pairs.Generating each pair requires one StyleGAN forward pass, averaging 9 seconds; most experiments use 10k synthetic images.
- Annotation efficiency: For 10k highly detailed labels, crowdsourcing would take over 3200 hours, whereas DatasetGAN required roughly 5 hours of human annotation.The comparison concerns one synthetic dataset at that scale.
4. Collecting Fine-grained Annotations
The annotation study uses a single experienced annotator to create detailed part labels for small GAN-generated datasets. The figures illustrate synthesized images and labels across faces, cars, birds, cats, and bedrooms, while image quality affects annotation and label completeness.
- Annotation collection: A single experienced annotator labeled 40 bedrooms, 16 cars, 16 heads, 30 birds, and 30 cats for Style Interpreter training.The annotations contained 1109, 605, 950, 443, and 737 polygons respectively.
- Annotation collection: The study defines detailed partonomies for each class, while real images are reserved for evaluation.The reported annotation counts reflect the GAN-generated training examples.
- Annotation statistics: Head annotations averaged 58 parts for GAN images versus 55 for real images, with each fully detailed head taking 1159 seconds.Annotating only the head outline took 74 seconds on average.
- Annotation statistics: GAN-generated bedrooms contained 37.8 annotated objects on average versus 47.8 for real bedrooms, and took 629 versus 1583 seconds to annotate.The generated bedrooms had 256×256 resolution, while the real images had higher resolution.
- Synthesized datasets: Figures 4 and 5 show synthesized images with labels for faces, cars, birds, cats, and bedrooms using 16 to 40 annotated examples per class.The corresponding StyleGAN training resolutions range from 256×256 to 1024×1024.
- Limitations: Bird image quality made annotation difficult because legs were often invisible, blurry, or unnatural, causing sparse leg labels and reduced test-time performance.This limitation arises from relying on GAN-generated images for annotation.
5. Experiments
DatasetGAN is evaluated across part segmentation, keypoint detection, and a downstream 3D reconstruction application using synthetic training data and limited human annotation. It outperforms transfer-learning and semi-supervised baselines, while approaching fully supervised performance in some settings.
- Evaluation scope: The experiments cover part segmentation across five categories, keypoint detection for cars and birds, and single-image 3D asset creation for cars.The part-segmentation categories are Car, Face, Bird, Cat, and Bedroom.
- Experimental setup: The method trains segmentation models on synthesized image-annotation pairs generated from category-specific StyleGAN models.Higher-resolution StyleGAN models produce more accurate synthesized annotations.
- Part segmentation: 25 annotations produce performance comparable to a fully supervised ADE-Car-12 model trained on 2,600 images, although the method does not exceed the fully supervised baseline on ADE-Car-5.On PASCAL-Car-5, it outperforms the fully supervised baseline by 1.3% in the out-of-domain comparison.
- Part segmentation: Qualitative outputs are impressive in the few-shot regime, with most errors involving thin parts or parts lacking clear visual boundaries.Examples include wrinkles, bird legs, and cat necks.
- Keypoint detection and 3D application: The approach significantly outperforms fine-tuning for keypoint detection and supports detailed part and keypoint predictions for single-image 3D reconstruction of animatable assets.The 3D application targets richly detailed assets that can be animated realistically.
6. Conclusions
DatasetGAN uses StyleGAN’s learned latent space and a few human-annotated images to synthesize large labeled datasets for training vision models. It outperforms semi-supervised baselines and can surpass fully supervised approaches trained with far more data.
- DatasetGAN trains an effective classifier on StyleGAN’s learned latent space using only a few human-annotated images.
- The method manually labels tiny datasets for seven tasks, then synthesizes large labeled datasets for computer vision training.
- DatasetGAN significantly outperforms all semi-supervised baselines.
- In some cases, DatasetGAN surpasses fully supervised approaches trained with two orders of magnitude more data.