Source-linked AI summary
BigDatasetGAN: Synthesizing ImageNet with Pixel-wise Annotations
Daiqing Li, Huan Ling, Seung Wook Kim, Karsten Kreis, Adela Barriuso, Sanja Fidler, Antonio Torralba
TL;DR
Pixel-wise ImageNet annotation is costly, and existing GAN-based labeling has limited class diversity. BigDatasetGAN uses a small annotated set to turn BigGAN and VQGAN into labeled dataset generators. Across ImageNet segmentation tasks, synthesized data improves performance, and the datasets also benefit downstream dense prediction.
Problem
Adding pixel-wise labels to ImageNet could support large-scale segmentation and dense-task pre-training, but manually labeling masks for a million images is costly.
Source: §1 · p. 1Method
Results
Across seven ImageNet segmentation tasks, MoCo-v3 with VQGAN-sim improves mean performance by 10.2 over using only the manually labeled dataset.
Source: §5.2 · p. 7Takeaways & Limitations
Synthesized datasets benefit downstream dense prediction across supervised and self-supervised backbones, and the authors aim to extend annotations to object parts and keypoints.
Source: §6 · p. 8
Abstract
from arXiv · showhide
Annotating images with pixel-wise labels is a time-consuming and costly process. Recently, DatasetGAN showcased a promising alternative - to synthesize a large labeled dataset via a generative adversarial network (GAN) by exploiting a small set of manually labeled, GAN-generated images. Here, we scale DatasetGAN to ImageNet scale of class diversity. We take image samples from the class-conditional generative model BigGAN trained on ImageNet, and manually annotate 5 images per class, for all 1k classes. By training an effective feature segmentation architecture on top of BigGAN, we turn BigGAN into a labeled dataset generator. We further show that VQGAN can similarly serve as a dataset generator, leveraging the already annotated data. We create a new ImageNet benchmark by labeling an additional set of 8k real images and evaluate segmentation performance in a variety of settings. Through an extensive ablation study we show big gains in leveraging a large generated dataset to train different supervised and self-supervised backbone models on pixel-wise tasks. Furthermore, we demonstrate that using our synthesized datasets for pre-training leads to improvements over standard ImageNet pre-training on several downstream datasets, such as PASCAL-VOC, MS-COCO, Cityscapes and chest X-ray, as well as tasks (detection, segmentation). Our benchmark will be made public and maintain a leaderboard for this challenging task. Project Page: https://nv-tlabs.github.io/big-datasetgan/
1. Introduction
The paper scales GAN-based pixel-wise labeling to ImageNet’s 1,000 classes, aiming to reduce the cost of manually annotating masks while supporting segmentation benchmarks and downstream pre-training.
1. Introduction
- ImageNet’s million images and 1,000 classes make it valuable for classification and pre-training, but the paper targets adding pixel-wise labels at a fraction of the cost of manually labeling all images.The existing classification labels support broad downstream generalization, while dense masks would enable large-scale multi-class segmentation.Source: §1 · p. 1
- The paper scales DatasetGAN [82] to ImageNet, introduces a pixel-wise benchmark, and analyzes existing methods on it.DatasetGAN manually labels a small set of GAN-generated images and trains a shallow segmentation branch on generator features, but its StyleGAN [39] backbone is limited to single-class modeling.Source: §1 · p. 1
- BigDatasetGAN uses BigGAN samples manually labeled across ImageNet classes to train a feature interpreter, and VQGAN can generate labeled data using those annotations.The paper also labels 8,000 held-out real images for a segmentation benchmark.Source: §1 · p. 2
2. Related Work
Related work addresses the cost of pixel-wise labels through propagation, rendered synthetic data, and generative models; this paper extends GAN-based dataset generation to ImageNet-scale class diversity.
Reducing Annotation Cost
- Prior ImageNet mask propagation labels 10 images per class across 500 classes and iteratively transfers masks, but does not demonstrate the usefulness of the auto-labeled dataset.The paper contrasts that approach with GAN-based synthesis and modern deep learning methods.Source: §2 · p. 2
Synthetic Dataset Generation
- Graphics engines can provide rendered images with perfect labels, but their datasets typically have appearance and content gaps from real data, and ImageNet-scale diversity would require substantial 3D content acquisition.GAN-based translation and learned generation pipelines have been explored to reduce distribution gaps.Source: §2 · p. 2
Generative Models and Representation Learning
- Prior GAN labeling methods use a handful of annotated generated images, but mostly target single classes; this work scales the approach with conditional BigGAN and VQGAN.The paper also analyzes supervised and self-supervised pre-training with synthesized data.Source: §2 · p. 2
3. BigDatasetGAN
BigDatasetGAN extends DatasetGAN to ImageNet by training feature interpreters for BigGAN and VQGAN from a small manually annotated set, then sampling filtered labeled datasets for downstream training.
3. BigDatasetGAN
- BigDatasetGAN extends DatasetGAN [82] using ImageNet-pretrained BigGAN and VQGAN, chosen as high-performance conditional generators with different architectures and training approaches.The method manually labels BigGAN samples, trains interpreters for both generators from those annotations, and samples large datasets from each.Source: §3 · p. 3
3.1. Generative Models of Datasets
- A feature interpreter maps generator features and class labels to pixel-wise labels, enabling the generator and interpreter together to produce densely annotated data.For VQGAN, the annotations on BigGAN images are reusable because its encoder embeds those images; the authors report no satisfactory BigGAN encoder.Source: §3.1 · p. 3, §3 · p. 3
- The BigGAN interpreter groups features by resolution, progressively fuses them, and outputs segmentation logits; its mix-conv design reduces memory use versus DatasetGAN.VQGAN’s interpreter uses transformer features for semantic information together with decoder features, following the BigGAN architecture design.Source: §3.1 · pp. 3–4
3.2. Synthesizing Labeled Data
- The authors synthesize large BigGAN and VQGAN datasets using filtering steps intended to improve image and label quality.BigGAN uses truncation and rejection sampling plus an ensemble of 16 segmentation heads to filter the 10% most uncertain samples; VQGAN uses top-200 filtering and nucleus sampling with p = 0.92.Source: §3.2 · pp. 4–5
- Online BigGAN sampling improves ImageNet segmentation performance by 1–2% over a static 100k dataset and converges faster, but is slower per iteration and omits expensive filtering.The authors explore online sampling only with BigGAN because VQGAN’s autoregressive sampling is particularly slow; offline sampling is the default otherwise.Source: §3.2 · p. 5
4. Annotating ImageNet with Pixel-wise Labels
The annotation process uses BigGAN samples because BigGAN cannot encode images, while VQGAN can reuse annotated BigGAN images; a single annotator labels about five images per class and dataset. Table 2 compares segmentation methods and synthetic-data settings on the ImageNet pixel-wise benchmark.
4. Annotating ImageNet with Pixel-wise Labels
- Because BigGAN cannot encode images, the authors choose its samples for annotation; VQGAN can later leverage those same labeled images.The annotator labels about five images per class and dataset on average, ignoring low-quality images with unrecognizable objects.Source: §4 · p. 5
- Table 2 compares supervised and self-supervised methods across FG/BG and multi-class tasks, including synthetic-data ablations and offline versus online BigGAN sampling.FG/BG evaluates binary segmentation across all classes; MC-N denotes multi-class setups with N classes.Source: Table 2 · p. 5
5. Analysis and Experiments
Across ImageNet segmentation tasks, synthetic datasets improve performance, and the analyses compare dataset quality, scale, and model size. Transfer experiments also report gains on downstream vision and chest X-ray tasks, though Cityscapes semantic segmentation shows no significant boost.
5.1. Dataset Analysis
- The experiments analyze four datasets: 8k real-annotated images for benchmark testing, 5k synthetic-annotated images for training, and 100k each of BigGAN-sim and VQGAN-sim.The analysis compares image and label quality, label statistics, and polygon shape complexity and diversity.Source: §5.1 · p. 5
5.2. ImageNet Segmentation Benchmark
- With a fixed ResNet50 backbone, training on 22k synthetic samples outperforms 2k human-annotated labels, and increasing synthetic data from 22k to 220k adds another 7 points.BigGAN-sim supervision with a dataset 100× larger than the 2k human-annotated set leads to consistent improvements, especially for larger backbones.Source: Figure 8 · p. 6, Figure 9 · p. 6
5.1. Dataset Analysis
- BigGAN-sim has better masked-image FID/KID than VQGAN-sim (19.45/3.47 vs. 21.21/11.10), while VQGAN-sim averages 1.52 instances per image versus 1.33.BigGAN-sim also has higher Shape Complexity (33.9 vs. 30.8); the paper notes some VQGAN effects may reflect training on BigGAN data.Source: §5.1 · p. 6
5.3. Transfer Learning
- On PASCAL-VOC, synthetic-data pre-training converges significantly faster than contrastive learning alone; colored areas show variance over five trials.The transfer experiments use pre-trained backbones on downstream detection and segmentation tasks.Source: Figure 10 · p. 6, §5.3 · p. 7
5.2. ImageNet Segmentation Benchmark
- The MoCo-v3-pretrained backbone trained with VQGAN-sim improves mean performance by 10.2 over seven tasks; in MC-16, the DenseCL-pretrained backbone improves performance by 11.4 with the synthetic dataset.Source: §5.2 · p. 7
- Thin structures, occlusions, and unclear object boundaries can make classes hard to segment even when they are not hard to classify.The top-5 analysis compares best and worst FG/BG segmentation classes against ResNet-50 ImageNet classification accuracy.Source: Figure 7 · p. 6, §5.2 · p. 7
5.3. Transfer Learning
- Synthetic data for pre-training improves detection by 0.3 AP75 and segmentation by 0.5% on mIoU; chest X-ray reaches 67.6 mIoU versus 48.8 with the baseline using 1% training data.Cityscapes instance segmentation improves by 0.3 AP mk, while semantic segmentation has no significant performance boost.Source: §5.3 · p. 8
6. Conclusions
The paper builds an ImageNet segmentation benchmark for all 1k classes and uses BigGAN and VQGAN as high-quality dataset generators. The synthesized data benefits downstream tasks when paired with different supervised or self-supervised backbones.
6. Conclusions
- The collected benchmark targets segmentation across all 1k ImageNet classes.The approach aims to extend annotations to object parts and other dense labels, including keypoints.Source: §6 · p. 8
- Enhancing ImageNet-trained BigGAN and VQGAN with segmentation branches yields high-quality generators of large-scale, densely labeled datasets.The BigGAN annotations require minimal labeling effort, according to the conclusion.Source: §6 · p. 8
- Synthesized datasets benefit varied downstream tasks when combined with supervised or self-supervised pre-trained backbones.Source: §6 · p. 8
A. BigDatasetGAN Implementation Details
For BigGAN, the segmentation branch is trained on annotated latent codes and uncertainty filtering removes the most uncertain generated samples. For VQGAN, the branch combines transformer and decoder features and is trained using images sampled from BigGAN.
A.1. BigGAN
- BigGAN features and class information train the segmentation branch on human-annotated masks using binary cross-entropy.Training uses pre-recorded latent codes, Adam with learning rate 0.001, batch size 8, and 5k iterations.Source: §A.1 · p. 9
- Uncertainty filtering uses Jensen-Shannon divergence across 16 segmentation heads and removes the 10% most uncertain samples.Each model's output distribution contributes to the uncertainty score.Source: §A.1 · p. 9
A.2. VQGAN
- VQGAN features come from 12 transformer layers and 7 decoder layers, grouped by spatial resolution and reduced to 128 dimensions.Features are fused across resolution groups to produce segmentation logits.Source: §A.2 · p. 9
- VQGAN's encoder processes images previously sampled via BigGAN, and its class-conditional transformer supplies the class label for segmentation prediction.The branch uses the same foreground/background binary cross-entropy as BigGAN, with Adam at learning rate 0.001 and batch size 8.Source: §A.2 · p. 9
B.1. Dataset Statistics
The benchmark appendix specifies data splits for seven tasks and says detailed class mapping will accompany the public benchmark launch.
B.1. Dataset Statistics
- Table 7 gives the dataset split details for the benchmark’s seven tasks.The authors plan to provide a detailed class mapping when the public benchmark launches.Source: §B.1 · p. 9
B.2. Training Setup
The appendix describes a shared segmentation training setup, a special loss for the imbalanced MC-992 task, and slower online sampling. It also reports that object centers are generally central, with noisier corner distributions in the generated datasets.
B.2. Training Setup
- All methods use DeepLabv3 with a ResNet-50 backbone, SGD, batch size 64, 200 epochs, and the same stated augmentation and learning-rate schedule.Synthetic-data training follows the same schedule and augmentation policy.Source: §B.2 · p. 9
- Cross-entropy is used for all tasks except MC-992, where focal loss addresses the imbalanced 992-class pixel distribution and poor convergence with cross-entropy.The authors report that standard cross-entropy training did not converge well on MC-992.Source: §B.2 · pp. 9–10
B.3. Long-time Training with Online Sampling
- Online sampling is slower because it runs the generative model at each iteration; the main-paper BigGAN-on result for MC-992 was not fully converged, and Table 8 updates it.The updated table changes the BigGAN-on result for MC-992 only.Source: §B.3 · p. 10, Table 8 · p. 10
C.1. Center Distributions
- Most object centers lie near the image center, while BigGAN-sim and VQGAN-sim have noisier center distributions, especially near image corners.Centers are measured from tight bounding boxes around pixel-wise labels and normalized by image size.Source: §C.1 · p. 10
C.2. Geometry Metrics
The geometry metrics characterize polygon complexity and diversity after filtering and normalizing segmentation masks. The appendix also visualizes normalized object-center distributions across four datasets.
C.2. Geometry Metrics
- Figure 12 plots tight-label-box centers normalized to image size for the Real-annotated, Synthetic-annotated, BigGAN-sim, and VQGAN-sim datasets.The geometry metrics compare shape complexity and shape diversity between human-annotated and synthetic datasets.Source: Figure 12 · p. 10, §C.2 · p. 10
- The implementation rescales each extracted polygon to a unit square and applies Douglas-Peucker simplification with a 0.01 threshold.OpenCV contour extraction compresses horizontal, vertical, and diagonal segments; scaling is normalized separately by image direction.Source: §C.2 · p. 10
- Shape complexity is the number of points in the normalized simplified polygon; polygon length is also measured.Masks with fewer than 100 pixels are filtered, only the largest connected component is retained, and polygons are simplified before measurement.Source: §C.2 · p. 10
- Shape diversity is the mean pair-wise Chamfer distance between polygons within each class, averaged across classes.The compared point sets correspond to different polygons.Source: §C.2 · pp. 10–11
D.1. Pre-training Implementation
The pre-training implementation follows DenseCL’s setup before training the segmentation branch. Downstream detection, instance-segmentation, and semantic-segmentation evaluations use established framework configurations.
D.1. Pre-training Implementation
- Backbone pre-training follows DenseCL with SGD, a 65,536-entry dictionary, 224 × 224 augmentations, and 150 epochs, followed by 50 epochs of joint segmentation-branch training.The encoder momentum update rate is 0.999; the segmentation branch uses random resized crops and horizontal flips.Source: §D.1 · p. 11
D.2. Downstream Task Details
- Detection and instance segmentation use OpenSelfSup configurations with Detectron2, while semantic segmentation uses DenseCL configurations in MMSegmentation.These downstream tasks cover Pascal VOC, MS-COCO, and Cityscapes for detection and instance segmentation, and Pascal VOC and Cityscapes for semantic segmentation.Source: §D.2 · p. 11
E. Qualitative Results
The qualitative results show that BigGAN- and VQGAN-generated datasets contain diverse, realistic images with high-quality labels, while revealing differences in their backgrounds, sample diversity, and label quality.
E. Qualitative Results
- Both BigGAN- and VQGAN-generated datasets contain photorealistic, diverse images with high-quality labels, while real-annotated images have more complex backgrounds and structure.The section also presents random samples from the human-annotated and synthetic datasets.Source: E. Qualitative Results · p. 11
- VQGAN-sim samples vary more in object scale, pose, and background than BigGAN-sim, while BigGAN-sim labels are cleaner; some VQGAN-sim labels have holes or noise.The comparison uses the same classes for both datasets.Source: E. Qualitative Results · p. 11
- BigGAN-sim mean-shape visualizations show varied shapes and poses, especially for animal classes, though some classes lack clear shapes.Shapes are clustered into five modes per class after masks are cropped and resized to 32 × 32; an unclear shape may reflect a randomly selected, less meaningful mode.Source: E. Qualitative Results · p. 11