Source-linked AI summary
LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop
Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, Jianxiong Xiao
TL;DR
Deep visual models need large labeled datasets, but existing data are limited in density and manual annotation is costly. LSUN amplifies human effort by iteratively combining crowdsourced labels with classifier-based propagation, producing dense scene and object datasets. The authors report more than 90% average labeling precision and substantial classifier gains from the larger training set.
Problem
Deep models require massive human-labeled datasets, while available datasets lag in size and density and manual annotation remains costly.
Method
LSUN iteratively samples images for AMT labeling, trains classifiers, propagates confident positive and negative labels, and sends ambiguous images to later iterations.
Results
More than 90% average labeling precision was achieved, and popular deep models showed substantial performance gains when trained on LSUN’s larger dataset.
Takeaways & Limitations
LSUN demonstrates the potential of denser training data for visual recognition and provides a large labeled image resource.
Takeaways & Limitations
The pipeline trades computation time against performance, and category labeling is constrained to binary positive-or-negative decisions.
Abstract
from arXiv · showhide
While there has been remarkable progress in the performance of visual recognition algorithms, the state-of-the-art models tend to be exceptionally data-hungry. Large labeled training datasets, expensive and tedious to produce, are required to optimize millions of parameters in deep network models. Lagging behind the growth in model capacity, the available datasets are quickly becoming outdated in terms of size and density. To circumvent this bottleneck, we propose to amplify human effort through a partially automated labeling scheme, leveraging deep learning with humans in the loop. Starting from a large set of candidate images for each category, we iteratively sample a subset, ask people to label them, classify the others with a trained model, split the set into positives, negatives, and unlabeled based on the classification confidence, and then iterate with the unlabeled set. To assess the effectiveness of this cascading procedure and enable further progress in visual recognition research, we construct a new image dataset, LSUN. It contains around one million labeled images for each of 10 scene categories and 20 object categories. We experiment with training popular convolutional networks and find that they achieve substantial performance gains when trained on this dataset.
1 Introduction
Deep visual models require massive labeled datasets, but existing datasets are limited in size, density, and freshness while manual labeling remains costly. LSUN addresses this bottleneck with a human-in-the-loop labeling framework that produces denser datasets and improves classifier performance.
- Motivation: Deep models with millions of parameters require massive amounts of human-labeled training data.Their capacity enables complex representations but makes data collection a central constraint.
- Problem: Existing datasets lag behind model growth in size and are increasingly outdated.ImageNet is described as seven years old and heavily over-fitted, while Places is not much larger.
- Problem: Low category density causes deep networks trained on current datasets to learn noisy or unstable features.Millions of images are distributed across many categories, leaving relatively few examples per category.
- Contribution: The proposed alternative targets approximately 10^6 images per category, about 10 times denser than Places and 100 times denser than ImageNet.The goal is to increase the density of novel examples rather than only augment existing images with perturbations.
- Problem: Manual labeling remains the bottleneck because constructing ImageNet and Places each required more than a year of AMT effort.This cost makes substantially larger or denser supervised datasets difficult to build manually.
- Contribution: LSUN uses deep learning with humans in the loop to amplify manual labeling effort and contains 10 million scene images plus 59 million object images.The framework combines image selection, rapid label acquisition, verification, and label propagation.
- Results: More than 90% average labeling precision accompanies substantial classifier gains from the larger, noisy training set.The paper reports that larger noisy data produced better models than smaller noise-free datasets in its experiments.
2 Overview
LSUN’s overview describes an iterative cascade that combines crowdsourced labels with classifiers to resolve a large image pool. Confident predictions are accepted or discarded, while ambiguous images continue to later iterations or receive exhaustive manual labeling.
- 2 Overview: The pipeline begins by collecting millions of candidate images per category and repeatedly selecting small subsets for human labeling.Candidate pools typically contain approximately 10^7–10^8 images.
- 2 Overview: Each iteration trains a classifier on labeled examples, predicts labels and confidences, and selects ambiguous images for further consideration.The procedure alternates human labeling, classifier training, prediction, and candidate selection.
- 2 Overview: Unlike active learning aimed at generalizable classifiers, this process may over-fit locally because its goal is correct labels for a specified image set.Generalization outside the target set is not required for the labeling objective.
- 2 Overview: The cascade splits images into positive, negative, and unlabeled subsets, adding positives, discarding negatives, and passing unlabeled images forward.It terminates when the remaining unlabeled subset is small enough for complete manual labeling.
- 2 Overview: The system combines data collection, deep classifiers, manual-labeling interfaces, and label-quality controls before evaluating the dataset for image classification.The overview frames these components as parts of one integrated system.
- 2 Overview: Figure 1 depicts the same loop from random sampling and crowdsourcing through classifier scoring and iterative processing of the unknown set.High- or low-scoring images receive automatic labels, while ambiguous images continue to the next iteration.
3 Data Collection
The data-collection stage uses image-search engines and expanded keyword queries to build very large candidate pools. Basic size filtering retains a substantial portion of downloaded images while avoiding expensive early deduplication.
- 3 Data Collection: Google Images queries provide nearly 100 million relevant image URLs for each category.The collection process leverages existing image-search engines rather than gathering images manually.
- 3 Data Collection: Scene-category queries combine each category name with 696 manually selected adjectives to enlarge and diversify search results.The adjectives are drawn from a list of popular English adjectives relevant to scenes.
- 3 Data Collection: More than 1 billion images were downloaded, and approximately 60% remained after keeping only images whose smaller dimension exceeded 256 pixels.The initial quality check applies a size threshold before labeling.
- 3 Data Collection: The initial pool does not remove content duplicates because deduplication across tens of millions of non-duplicate images is considered too expensive.This choice prioritizes scalable collection before the labeling pipeline.
4 Deep Learning with Humans in the Loop
The labeling system treats each category as a binary classification problem and iteratively combines AMT labels with classifiers trained on sampled data. Confidence thresholds automatically resolve easy cases while ambiguous images receive further labeling, subject to computation–performance tradeoffs.
- 4 Deep Learning with Humans in the Loop: Each category is formulated as a binary problem in which every downloaded image receives either a positive or negative label.The formulation defines the labeling target for the semi-automatic procedure.
- 4 Deep Learning with Humans in the Loop: A sampled AMT-labeled set is split into training and testing data, then used to train a classifier and calibrate confidence ranges.Testing scores identify easy images and more difficult or ambiguous images.
- 4 Deep Learning with Humans in the Loop: Images above or below two confidence thresholds are labeled positive or negative, while intermediate-score images move to the next iteration.The upper threshold corresponds to 95% ground-truth positives, and the lower threshold captures a range containing only 1% of total positive images.
- 4 Deep Learning with Humans in the Loop: Later iterations combine new random samples with prior ambiguous examples, enabling classifiers to process increasingly difficult training cases.Images that still lack confident labels are passed onward.
- 4 Deep Learning with Humans in the Loop: Model selection balances classification performance against the runtime cost of processing tens of millions of images.More accurate and higher-capacity convolutional models can reduce iterations but may be too expensive to run.
- 4 Deep Learning with Humans in the Loop: AlexNet features with a two-hidden-layer MLP removed more than half of the initial image set, but that feature space became ineffective after two iterations.The pipeline then adapted pre-trained GoogLeNet for later processing.
- 4 Deep Learning with Humans in the Loop: Each iteration samples 40 thousand images, reserving subsets for testing and validation and using the remainder for training.Experiments with 80 or 120 thousand images per iteration did not significantly improve early classifier results.
5 Crowd Sourcing
The pipeline combines rapid, quality-controlled crowd annotation with category-specific instructions to obtain reliable labels at scale. Redundant labels, test images, and explicit corner-case guidance address variation and ambiguity among workers.
- The system uses Amazon Mechanical Turk to obtain many labels quickly and cheaply, while recognizing that worker quality varies.
- Category-specific rules exclude confusable examples, such as trucks and buses from the car category, and define which visible parts establish object identity.
- The process uses pilot hits, sampled conflict reviews, embedded ground-truth images, and redundant labeling to detect and control annotation errors.Each image receives labels from two workers, and only doubly confirmed labels are retained.
- Each HIT presents 205 images, including 150 target images and 55 quality-control images, and workers typically finish within five minutes.The authors describe this as a trade-off between efficiency and quality control.
- Workers receive category definitions, positive and negative examples, and instructions for ambiguous cases such as partial object visibility.
6 Results
LSUN substantially increases image density and uses the labeling pipeline to produce large collections with approximately 90% precision and strong evidence that additional related-category data benefits recognition models. The experiments also examine labeling-effort amplification and learned representations.
- Dataset Statistics: 59 million object images across 20 categories and 10 million scene images across 10 categories make LSUN substantially denser than ImageNet categories.Most LSUN object categories contain more images than the entire 1.4-million-image ImageNet classification challenge dataset.
- Label Precision: Approximately 90% labeling precision was observed across 11 tested object categories, with variation caused mainly by toys, rendered images, collages, and edited photos.
- Effort Amplification: Human effort was amplified 40 times on average, with people labeling less than 1/60th of the images in the train category.
- Impact on Model Performance: AlexNet classification errors were compared after training with PLACES alone versus PLACES and LSUN, with lower errors indicating better performance.The experiment used at most 200 thousand LSUN images per category and evaluated the 10 LSUN categories in the PLACES test set.
- Learned Image Representation: Pre-training with more images in related categories outperformed pre-training with more categories when representations were evaluated on PASCAL VOC 2012.
- Learned Image Representation: AlexNet filters trained on LSUN object images showed cleaner first-layer patterns than filters learned from ImageNet.
7 Conclusion
The paper presents LSUN as a working human-in-the-loop pipeline for acquiring large labeled image datasets. Its initial database contains around a million labeled images per scene category and more than a million per object category.
- The proposed pipeline combines deep learning with human labeling to acquire large image datasets with category labels.
- LSUN contains around a million labeled images in each scene category and more than a million in each of 20 object categories.