Source-linked AI summary
Places: An Image Database for Deep Scene Understanding
Bolei Zhou, Aditya Khosla, Agata Lapedriza, Antonio Torralba, Aude Oliva
TL;DR
Places addresses the need for broad, dense training data by introducing a quasi-exhaustive scene repository. The 10-million-image database supports CNN scene-classification baselines, while its diversity and coverage target difficult visual-recognition problems.
Problem
Supervised learning requires task-suitable algorithms and training datasets with sufficient coverage and density to represent class diversity.
Method
Places constructs a quasi-exhaustive database of 10 million scene photographs labeled with 476 semantic categories and attributes.
Results
Places-CNNs outperform the ImageNet feature+SVM baseline, with leaderboard results reaching 64.10% Top-1 and 90.65% Top-5 accuracy.
Takeaways & Limitations
Places provides high-coverage, high-diversity visual context for progress on difficult recognition problems involving scenes, actions, objects, and events.
Takeaways & Limitations
The ResNet performance may reflect fine-tuning or under-training because ResNet was not trained from scratch.
Abstract
from arXiv · showhide
The rise of multi-million-item dataset initiatives has enabled data-hungry machine learning algorithms to reach near-human semantic classification at tasks such as object and scene recognition. Here we describe the Places Database, a repository of 10 million scene photographs, labeled with scene semantic categories and attributes, comprising a quasi-exhaustive list of the types of environments encountered in the world. Using state of the art Convolutional Neural Networks, we provide impressive baseline performances at scene classification. With its high-coverage and high-diversity of exemplars, the Places Database offers an ecosystem to guide future progress on currently intractable visual recognition problems.
1 INTRODUCTION
Near-human visual recognition depends on both suitable algorithms and large datasets with broad class coverage and dense exemplar diversity. The Places Database addresses this data requirement with 10 million labeled scene photographs.
- Motivation: Supervised learning requires algorithms suited to the task and training datasets with appropriate coverage and density.Coverage means representing classes and exemplar variety, while density means sampling each class’s diversity sufficiently.
- Motivation: Multi-million-item datasets have accompanied near-human performance in chess, question answering, game playing, object recognition, and scene recognition.The cited examples include datasets of 700,000, 8.6 million, 30 million, 1.2 million, and 2.5 million items.
- Contribution: Places is a quasi-exhaustive repository of 10 million scene photographs labeled with 476 scene semantic categories and attributes.It is intended to represent visual environments encountered in the world.
- Contribution: The paper explains dataset-construction steps intended to enable the performance of machine-learning algorithms.The introduction frames dataset construction as part of enabling remarkable machine-learning feats.
2.1 Coverage of the categorical space
Places prioritizes expansive coverage of the categorical space by building on SUN’s quasi-exhaustive list of scene categories and environments.
- Coverage of the categorical space: Places treats expansive coverage of the categorical space as the primary asset of a high-quality dataset.Its target space is environments encountered in the world, bounded by spaces where a human body would fit.
- Coverage of the categorical space: The SUN dataset supplied an initial quasi-exhaustive list of scene categories organized around distinct environmental functionalities and identities in discourse.WordNet terms were used to form a corpus of scene, place, and environment concepts, later bundled into 900 categories.
2.2 Construction of the database
Places was constructed through large-scale web retrieval, crowdsourced verification, classifier-assisted relabeling, and targeted separation of similar categories. Four annotation stages produced over 10 million labeled exemplars across 434 place categories.
- 2.2.1 Step 1: Downloading images using scene category and adjectives: The dataset samples environments across Indoor, Nature, and Urban macroclasses and uses query-grouped examples to illustrate visual diversity.Figure 2 shows four scene categories with nine annotated images for each query.
- 2.2.1 Step 1: Downloading images using scene category and adjectives: Images were downloaded from Google Images, Bing Images, and Flickr using scene-category queries expanded with 696 common English adjectives.About 60 million color images at least 200×200 pixels with unique URLs were identified.
- 2.2.2 Step 2: Labeling images with ground truth category: Amazon Mechanical Turk workers verified category membership through instructed go/no-go judgments using positive and negative examples.Each HIT contained 750 images plus 30 positive and 30 negative controls, and only workers achieving at least 90% control accuracy were retained.
- 2.2.2 Step 2: Labeling images with ground truth category: 25.4% of images were relabeled as No during the second cleaning iteration, after which 7,076,580 images had one label across 476 categories.A third iteration was not pursued initially because its relabeling percentage was not significant.
- 2.2.3 Step 3: Scaling up the dataset using a classifier: A classifier-assisted bootstrapping stage ranked 53 million unassigned images, sending high-confidence candidates for further manual annotation.AlexNet reached 32% validation accuracy, and images with predicted class confidence above 0.8 were reviewed.
- 2.2.3 Step 3: Scaling up the dataset using a classifier: About 3 million images were added, yielding 401 categories with more than 5,000 images and 240 with more than 20,000 images.The third annotation round flattened the distribution of images across categories.
- 2.2.4 Step 4: Improving the separation of similar classes: Because some category boundaries were blurry, workers compared similar categories directly and could assign an image to either category or neither.The fourth-step interface used exemplar images from category pairs with shared content.
- 2.2.4 Step 4: Improving the separation of similar classes: After four annotation steps, Places contained 10,624,928 labeled exemplars from 434 place categories.The final database is described as containing over 10 million labeled exemplars.
2.3 Scene-Centric Datasets
Scene-centric datasets label images by places or environments rather than objects, and Places is the largest such dataset described here. The section establishes benchmarks for comparing Places with ImageNet and SUN, including differences in coverage, category definitions, and image diversity.
- Places is described as the largest scene-centric image dataset, containing images labeled with scene or place names rather than object names.
- Benchmarks: Places365-Standard and Places365-Challenge provide benchmarks built from 365 categories with more than 4,000 images per category.Places205 and Places88 are additional benchmark subsets.
- Benchmarks: Places365-Standard contains 1,803,460 training images, while Places365-Challenge expands the training set to 8 million images using the same categories and evaluation sets.
- Benchmarks: Places88, ImageNet88, and SUN88 contain 88 common scene categories and support performance comparisons across the three scene-centric databases.
- Category definitions: Category correspondences are imperfect because ImageNet uses WordNet object-oriented definitions, whereas Places can distinguish location-dependent meanings and indoor from outdoor settings.For example, “elevator” may mean an elevator door, interior, or lobby in Places.
- Dataset diversity: Relative diversity is defined by whether random image pairs from one dataset are more visually similar than pairs from another dataset.The experiment used human judgments across 88 shared categories, with four pairs per database in each trial.
- Dataset diversity: 0.83, 0.67, and 0.50 are the average relative diversity values for Places, ImageNet, and SUN, respectively, making Places the most diverse of the three.Users selected SUN pairs as closest 50% of the time, ImageNet pairs 33%, and Places pairs 17%.
3 CONVOLUTIONAL NEURAL NETWORKS FOR SCENE CLASSIFICATION
The paper trains CNNs on Places scene datasets and evaluates them for scene classification and generic visual recognition. Places-trained models outperform an ImageNet feature baseline on scene tasks, while scene- and object-centric features show complementary strengths.
- CNN training: Places-CNNs are trained with AlexNet, GoogLeNet, and VGG architectures on Places205 and Places365-Standard to establish scene-classification baselines.ResNet152 is additionally fine-tuned on Places365-Standard for comparison.
- Places205 and SUN205 results: Places-CNNs outperform the ImageNet feature+SVM baseline, with deeper GoogLeNet and VGG models surpassing AlexNet by a large margin.The comparison evaluates Top-1 and Top-5 accuracy on Places205 and SUN205.
- Places365 results: Places365-VGG and Places365-ResNet achieve similar top performance on Places365 validation and test sets.The ResNet result may reflect fine-tuning or under-training because it was not trained from scratch.
- Places365 results: With 160 more categories, Places365 reduces Places205-CNN Top-5 accuracy on the test set by only 2.5%.This compares the previous Places205-CNNs with the larger-category Places365 setting.
- Recognition limitations: Scene environments expose limitations of Top-1 accuracy because multiple labels may validly describe one image, motivating multi-ground-truth or free-form descriptions.Reported failure cases include atypical activities and images containing multiple scene parts.
- Generic visual features: Places-CNN deep features outperform ImageNet-CNN features on scene-related datasets, whereas ImageNet-CNN features perform better on object-related datasets.The comparison uses matching CNN architectures and linear SVM classifiers; Places365-VGG reaches 63.24% on SUN397.
4 CONCLUSION
Places combines broad category coverage and diverse exemplars to support near-human semantic classification and future visual recognition research. The database is positioned as an ecosystem for problems involving actions, inconsistent objects or behaviors, and future events in scenes.
- 4 CONCLUSION: Places offers high coverage and high diversity across visual environments to guide progress on currently intractable visual recognition problems.The paper identifies actions, inconsistent objects or behaviors, and future events or causes as example problems.