Source-linked AI summary
The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scale
Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, Tom Duerig, Vittorio Ferrari
TL;DR
Open Images V4 addresses the need for large, rich datasets for image understanding across classification, detection, and visual relationship detection. It constructs a unified, large-scale benchmark from CC-BY Flickr images, validates its annotations, and evaluates models and cross-annotation applications. The resulting dataset combines broad semantic coverage, dense localization, and relationship annotations intended to support further research.
Problem
Computer vision models need large, richly annotated datasets to identify objects, localize them, and determine their interactions.
Method
The paper constructs Open Images V4 with unified image-level, bounding-box, and visual-relationship annotations, then analyzes dataset statistics, annotation quality, model scaling, and unified-annotation applications.
Results
Open Images V4 contains 9.2 million images with 30.1 million image-level labels, 15.4 million bounding boxes, and 375 thousand visual relationship annotations.
Takeaways & Limitations
The dataset’s scale, quality, and variety support research in image classification, object detection, visual relationship detection, and related areas.
Takeaways & Limitations
Image-classification experiments include only classes with at least 100 positive human-verified training examples, covering 7,186 classes.
Abstract
from arXiv · showhide
We present Open Images V4, a dataset of 9.2M images with unified annotations for image classification, object detection and visual relationship detection. The images have a Creative Commons Attribution license that allows to share and adapt the material, and they have been collected from Flickr without a predefined list of class names or tags, leading to natural class statistics and avoiding an initial design bias. Open Images V4 offers large scale across several dimensions: 30.1M image-level labels for 19.8k concepts, 15.4M bounding boxes for 600 object classes, and 375k visual relationship annotations involving 57 classes. For object detection in particular, we provide 15x more bounding boxes than the next largest datasets (15.4M boxes on 1.9M images). The images often show complex scenes with several objects (8 annotated objects per image on average). We annotated visual relationships between them, which support visual relationship detection, an emerging task that requires structured reasoning. We provide in-depth comprehensive statistics about the dataset, we validate the quality of the annotations, we study how the performance of several modern models evolves with increasing amounts of training data, and we demonstrate two applications made possible by having unified annotations of multiple types coexisting in the same images. We hope that the scale, quality, and variety of Open Images V4 will foster further research and innovation even beyond the areas of image classification, object detection, and visual relationship detection.
1 Introduction
Open Images V4 addresses the need for large, rich annotations supporting image understanding by unifying image-level labels, bounding boxes, and visual relationships at unprecedented scale. Its Flickr-based, CC-BY collection and annotation procedures provide complex imagery, natural class statistics, and quality-oriented labeling workflows.
- Unified tasks: The dataset targets image classification, object detection, and visual relationship detection, covering what objects are present, where they are located, and how they interact.These tasks require increasingly structured interpretation of image contents.
- Dataset scale: 9,178,275 images contain 30,113,078 image-level labels, 15,440,132 bounding boxes, and 374,768 visual relationship triplets across 19,794 and 600 classes.Object detection includes 15.4 million boxes for 600 categories on 1.9 million images, more than 15× the next largest datasets, with 8 annotated boxes per image on average.
- Evaluation and applications: The paper validates annotation quality, studies model performance as training data increases, reports visual-relationship baselines, and demonstrates two applications enabled by unified annotations.The applications are fine-grained object detection without fine-grained box labels and zero-shot visual relationship detection.
- Image acquisition: Images are CC-BY Flickr photographs collected without predefined class names or tags, yielding complex scenes and natural class statistics while avoiding initial design bias.The collection also removes images appearing elsewhere on the internet, reducing bias toward web image search engines.
- Annotation: Bounding-box annotation combines class-specific guidance, automated annotator training, extreme clicking, and box verification series to improve consistency and efficiency.Annotators received perfect-box examples and common-mistake guidance; rejected verification boxes also constrain later candidate searches.
3 Statistics
Open Images contains 9,178,275 images with image-level labels spanning 19,794 classes, but label coverage differs across splits and is concentrated among frequent classes.
- 9,178,275 images are split into train, validation, and test, with human-verified image-level labels analyzed across these subsets.
- 19,794 classes receive positive or negative image-level labels, while unverified classes remain unknown for each image.
- 2.4 positive labels per image occur in train versus 8.8 in validation and test because candidate labels were generated more densely there.
- The ~300 most frequent classes cover the majority of image-level label samples in all three dataset splits.
- About 20% of train verified labels come from crowdsourced annotators, compared with less than 1% in validation and test.
General statistics
Open Images provides bounding boxes for 600 classes across its splits and contains substantially richer multi-object imagery than comparison detection datasets.
- 600 boxable classes receive bounding-box annotations across the whole validation and test splits and a subset of the train split.
- The comparison in Table 6 uses publicly available ground truth and notes that ILSVRC annotations are sparse outside a densely annotated subset.
- Open Images has significantly fewer single-object biases and more images across the full range of boxes per image than comparison datasets.
Box attributes statistics
Bounding boxes carry five attributes that capture visibility, depiction, grouping, and containment, with occlusion and truncation most prevalent.
- Five attributes are assigned to Open Images bounding boxes, and their frequencies are reported for the train set.
- Occluded and Truncated are the most common attributes, while GroupOf and Depiction are significant and Inside is rare.
- The examples illustrate Inside, Occluded, Depiction, and Truncated annotations through buildings, people, busts, and flowers.
Box class statistics
Open Images has broad and highly varied box-class distributions, with many more instances and richer class co-occurrence than older detection datasets.
- Open Images is generally an order of magnitude larger than comparison datasets, with 11 classes exceeding COCO’s largest class in sample count.
- 3,505,362 instances cover the agglomerated person-related classes in Open Images, compared with 257,253 person instances in COCO.
- 517 Open Images classes exceed COCO’s least frequent class by instance count, and 417 exceed ILSVRC’s least frequent class.
- Open Images and COCO contain richer distributions of images with co-occurring classes than ILSVRC and PASCAL.
- Open Images has about 1,000 images with 14 distinct classes, while COCO has 20, ILSVRC has none above 11, and PASCAL none above 4.
- The most frequent co-occurring pairs commonly link people or objects with their parts and accessories, such as Person–Human face or Car–Wheel.
Box size statistics
Open Images contains a high proportion of small annotated objects, with a box-size distribution similar to COCO and more small objects than PASCAL. This distribution makes object detection challenging.
- Box size statistics: 43% of Open Images bounding boxes occupy less than 1% of the image area.The cumulative density function reports the percentage of boxes below a given fraction of image area.
- Box size statistics: Figure 20 compares the cumulative distribution of annotated box areas across PASCAL, COCO, Open Images, and uniform-area or uniform-side-length baselines.The horizontal axis is percentage of image area, and the vertical axis is the percentage of boxes below that area.
- Box size statistics: Open Images has many more small objects than PASCAL and a distribution similar to COCO.The comparison covers Open Images, PASCAL, COCO, and ILSVRC.
Box center statistics
Open Images has a rich and diverse distribution of object centers, comparable in complexity to COCO and richer than the more centered distributions of PASCAL and ILSVRC.
- Box center statistics: Open Images train-set object centers show complexity in a similar ballpark to COCO.The train set contains most of the data and exhibits a rich, diverse spatial distribution.
- Box center statistics: PASCAL and ILSVRC exhibit simpler, more centered object-center distributions than Open Images.The comparison uses normalized image coordinates.
Validation and test V5
Open Images visual-relationship annotations are generated through controlled verification over annotated object pairs. Compared with VG and VRD, they trade relationship diversity for clearer definitions, lower noise, and more complete annotation.
- Visual Relationships: 375k visual-relationship annotations cover 329 distinct triplets and 57 object classes across the Open Images splits.The relationships were verified for pairs of annotated objects after image-level labels and bounding boxes were produced.
- Visual Relationships: Relationship annotations are highly imbalanced across triplets, including both rare and very frequent triplets.The paper suggests that detecting sparsely annotated triplets may require a compositional rather than monolithic detector.
- Comparison with related datasets: Open Images contains more annotations than VG or VRD for several shared semantic triplets and includes triplets absent from VG.Examples include man–play–flute, dog–inside of–car, and woman–holds–rugby ball.
- Visual Relationships: Open Images annotates all relationships between objects corresponding to each pair of positive image-level labels, allowing multiple instances of one triplet per image.GroupOf objects are excluded from this process.
- Comparison with related datasets: Open Images relationship annotations are less diverse than VG and VRD but are better defined, less noisy, and more completely annotated.The dataset also contains complementary triplets and more samples for some shared triplets.
4 Quality
Open Images reports high bounding-box quality and strong annotation agreement, while image-level label recall is lower because the dataset covers many classes. Quality varies substantially for some ambiguous or culturally dependent classes and attributes.
- Bounding-box quality: 97.7% precision and 98.2% recall were measured for bounding-box annotations.Precision errors were mainly geometrically imprecise boxes and boxes with incorrect semantic labels.
- Bounding-box quality: Bidet, cello, and coffee table had semantic error rates of 86%, 55%, and 35%, respectively.The first two errors were attributed to cultural differences, while coffee table was described as ambiguous.
- Attribute quality: Most attributes have precision and recall in the high nineties, but Inside has 67% precision and occurs in only 0.4% of boxes.The Inside errors were mostly incorrect assignments to bell peppers inside containers.
- Annotator agreement: Extreme-clicking boxes achieved 0.87 average IoU agreement between independent annotators.This was close to the reported PASCAL human agreement of 0.88 IoU.
- Annotator agreement: Box Verification Series boxes achieved 0.77 average IoU agreement with manually redrawn boxes.This exceeded the IoU > 0.7 threshold used to train annotators and was lower than extreme-clicking agreement.
- Image-level label recall: Image-level label recall was 43% for all objects and 59% when difficult objects were excluded.For positive image-level labels, bounding-box recall was 98.2%, and explicit negative labels help train discriminative classifiers despite incomplete recall.
5 Performance of baseline models
The paper evaluates baseline models for image classification and object detection on Open Images, using controlled studies of annotation availability, model setup, and training data.
- The experiments quantify baseline performance for image classification and object detection on Open Images.
- Image classification: The classification study trains Inception-ResNetV2 and measures effects of verified-label quantity, negative labels, and restricting evaluation to boxable classes.
- Image classification: Machine-generated labels are thresholded at confidence above 0.5 and combined with human-verified labels for training.
- Image classification: The classification analysis includes 7,186 classes with at least 100 positive human-verified training examples.
- Image classification: Models are pre-trained from scratch on machine-generated labels and fine-tuned using 90% human-verified and 10% machine-generated labels.
- Image classification: Average Precision is computed for 4,728 eligible classes while accounting for non-exhaustive test annotations.
Number of human-verified labels
Performance generally benefits from more verified labels and training boxes, while explicit negative labels help after positive-label gains begin to saturate. Open Images also requires evaluation adjustments for incomplete labels, class hierarchies, and group-of boxes.
- Number of human-verified labels: mAP increases as the fine-tuning set grows from 1% to 100% of images with human-verified labels.All models start from pre-training on machine-generated labels over the full training split.
- Number of human-verified labels: Explicit human-verified negative labels significantly improve classification once performance begins to saturate with many positive labels.The positive-only comparison treats unverified labels as implicit negatives.
- Number of human-verified labels: mAP is generally higher for 600 boxable classes than for the full class set because boxable categories are concrete and visually defined.
- Object detection: The detection metric modifies PASCAL VOC 2012 mAP to handle non-exhaustive labels, class hierarchy, and group-of boxes.
- Object detection: Under the final metric, group-of boxes use w = 1, requiring at least one detection inside each group.
- Object detection: Faster R-CNN with InceptionV2 improves through all 14.6M training boxes, whereas SSD with MobileNetV2 saturates at 1M boxes.
- Visual relationship detection: BAR-CNN outperforms frequency baselines, indicating that visual relationship detection needs image information beyond object detection and relationship priors.
6 The Power of a Unified Dataset
Unified image-level, box, and relationship annotations support two applications: fine-grained detection and zero-shot visual relationship detection. These experiments show useful transfer across annotation types, although zero-shot performance remains far below supervised performance.
- Unified annotations: Open Images co-locates image classification, object detection, and visual relationship annotations, enabling experiments that combine annotation types.
- Fine-grained object detection: Image-level labels can provide finer semantic classes than bounding-box labels within the shared class hierarchy.
- Fine-grained object detection: Fine-grained labels are transferred safely when an image contains one bounding box of the general class.
- Fine-grained object detection: The fine-grained detection study covers Car, Flower, Cat, and Dog using stratified 90% training and 10% test splits.
- Fine-grained object detection: Fine-grained detection reaches 0.231–0.594 mAP across the four general classes, while all baselines remain below 0.05 mAP.Reported subclass examples include Ferrari at 0.638 mAP, Land Rover at 0.620 mAP, and Schnauzer at 0.542 mAP.
- Zero-shot visual relationship detection: Zero-shot VRD combines relationship training data with bounding boxes for new subject or object classes absent from relationship annotations.
- Zero-shot visual relationship detection: The zero-shot evaluation contains 194 new triplets, 5,983 test annotations, and 48 new zero-shot classes.
- Zero-shot visual relationship detection: Zero-shot VRD is tractable to a reasonable degree, but its gap to supervised results remains very large.
7 Conclusions
Open Images V4 combines large scale, varied annotations, and unified ground truth across three vision tasks, with the authors aiming to support research beyond those tasks.
- Open Images V4 contains 9.2 million images with unified ground truth for image classification, object detection, and visual relationship detection.
- The paper reports dataset construction, annotation statistics, quality evaluation, and baseline performance for image classification and object detection.
- The authors hope the dataset’s scale, quality, and variety will foster research beyond image classification, object detection, and visual relationship detection.
8 Credits
The Open Images Dataset credits contributors across project coordination, image collection, annotation, infrastructure, challenge organization, and advising.
- Project coordination was led by Vittorio Ferrari, Tom Duerig, and Victor Gomes.
- Contributors handled image collection, image-level labels, bounding boxes, and visual relationship annotations.
- Additional contributors developed the website and visualizer, classes and hierarchy, and dataset challenge.
- The project also acknowledges advisers Andreas Veit, Serge Belongie, Abhinav Gupta, and Dhyanesh.