Source-linked AI summary
WebVision Database: Visual Learning and Understanding from Web Data
Wen Li, Limin Wang, Wei Li, Eirikur Agustsson, Luc Van Gool
TL;DR
Large-scale visual recognition needs abundant labeled data, but manual annotation is costly and existing web-data results are difficult to compare with human-annotated benchmarks. The paper builds WebVision from noisy web images aligned to ILSVRC 2012 concepts and finds robust recognition, effective transfer, and dataset bias. It also identifies limitations and open directions for handling noise and using metadata.
Problem
Manual annotation of large-scale training data is costly, while prior web-data studies provide limited direct evidence about noisy images relative to human-annotated datasets.
Method
The paper constructs WebVision from more than 2.4 million web images across the same 1,000 concepts as ILSVRC 2012, collecting metadata and providing human-annotated validation and test sets.
Results
Noisy WebVision images train robust CNNs, transfer comparably or better to new datasets and tasks than ILSVRC 2012 models, and reveal a domain difference between the datasets.
Takeaways & Limitations
WebVision provides a large-scale benchmark for visual recognition from noisy web data, visual domain adaptation, and multimodal learning with metadata.
Takeaways & Limitations
Web search results can contain significant outliers and label noise, motivating methods that better handle noisy labels and exploit metadata.
Abstract
from arXiv · showhide
In this paper, we present a study on learning visual recognition models from large scale noisy web data. We build a new database called WebVision, which contains more than $2.4$ million web images crawled from the Internet by using queries generated from the 1,000 semantic concepts of the benchmark ILSVRC 2012 dataset. Meta information along with those web images (e.g., title, description, tags, etc.) are also crawled. A validation set and test set containing human annotated images are also provided to facilitate algorithmic development. Based on our new database, we obtain a few interesting observations: 1) the noisy web images are sufficient for training a good deep CNN model for visual recognition; 2) the model learnt from our WebVision database exhibits comparable or even better generalization ability than the one trained from the ILSVRC 2012 dataset when being transferred to new datasets and tasks; 3) a domain adaptation issue (a.k.a., dataset bias) is observed, which means the dataset can be used as the largest benchmark dataset for visual domain adaptation. Our new WebVision database and relevant studies in this work would benefit the advance of learning state-of-the-art visual models with minimum supervision based on web data.
1. Introduction
The paper studies whether noisy web images can support visual recognition at scale and introduces WebVision to compare them with human-annotated data. It finds competitive recognition and transfer performance, while identifying dataset bias and the usefulness of web metadata as research directions.
- Motivation: Large-scale web data offers weak supervision through naturally accompanying metadata, avoiding the cost and time of comprehensive manual annotation.Web images commonly include surrounding text, titles, and tags that can provide supervision.
- Dataset and Study: WebVision contains more than 2.4 million images aligned with the 1,000 ILSVRC 2012 semantic concepts, plus metadata and two 50,000-image human-annotated evaluation sets.Images were crawled from Google Image Search and Flickr, with titles, descriptions, and tags collected where available.
- Findings: WebVision-trained CNNs achieve competitive results with ILSVRC 2012 models, because large image quantity can compensate for noisier labels.Varying the number of training images indicates a trade-off between label quality and data quantity.
- Findings: Transferred WebVision representations achieve comparable or better image-classification results on Caltech-256 and PASCAL VOC 2007, while cross-dataset evaluation reveals dataset bias.Performance drops when models are applied across the WebVision and ILSVRC 2012 validation sets.
- Implications: The database supports studying large-scale visual domain adaptation and the role of metadata in visual recognition.The authors identify metadata utility and dataset bias as explicit research questions and future applications.
2. Related Work
Prior work harvested Internet imagery for recognition, weak supervision, and domain adaptation, but typically studied smaller or heterogeneous settings. This paper positions WebVision as a large-scale testbed for learning directly from noisy web data without instance-level human annotations.
- Internet-based datasets: Many established vision datasets harvest Internet images but depend on human annotation, unlike this paper’s direct learning from noisy web data.The contrast concerns instance-level human annotations used after image collection.
- Web-data methods: Earlier web-data studies explored categorization, label-noise handling, domain adaptation, classifier harvesting, and metadata exploitation.The cited approaches include improved pLSA, multi-instance learning, domain adaptation, automatic harvesting, and metadata-based recognition.
- Open gap: Existing studies often used small-scale settings or differing tasks and methodologies, limiting direct understanding of noisy web-image capacity against human-annotated datasets.WebVision addresses this by conducting extensive experiments in a large-scale scenario.
- Deep learning from web data: Recent CNN studies reported comparable detection results, stronger fine-grained classification, and good generalization from web images.These works used web images for object detection, localization, fine-grained classification, and word prediction.
- Open gap: Web learning differs from purely supervised or unsupervised learning because web imagery naturally includes metadata that can provide weak supervision.The metadata includes surrounding text, titles, and tags.
3. WebVision Dataset
WebVision is constructed from web images gathered using the 1,000 ILSVRC 2012 concepts, with accompanying metadata and human-annotated validation and test sets. Its analysis shows varied category sizes, source-dependent visual differences, and substantial, uneven label noise.
- Dataset Construction: WebVision crawls images from Google Image Search and Flickr using queries derived from the 1,000 ILSVRC 2012 semantic concepts.The dataset uses these shared concepts to enable direct comparison with ILSVRC 2012.
- Dataset Construction: More than 2.4 million web images are collected, including about 1.6 million from Flickr and 1.1 million from Google Image Search.Flickr collection crawls up to 2,000 images per query, while Google typically returns 600–1,000 images per query.
- Dataset Construction: Each image receives a class label from the synset associated with its query, but web search results can introduce significant outliers.The paper identifies noisy labels as an important research issue when using web data.
- Dataset Construction: WebVision includes textual metadata such as titles, descriptions, and tags, with geographical and camera information also available for some Flickr images.Google images provide crawled titles and descriptions, while Flickr images provide more extensive textual information.
- Dataset Construction: The dataset provides human-annotated validation and test sets, while near-duplicate detection is used to prevent overlap with training images.The annotation process uses three users per image, and images are treated as inliers when more than two users agree.
- Dataset Analysis: Category sizes range from 300 to more than 10,000 images, and source domains differ: Google images are often cleanly framed, whereas Flickr images more often depict objects in varied backgrounds and smaller sizes.Category availability depends on the number of generated queries and the supply of images on the two websites.
- Dataset Analysis: About 20% of sampled images receive zero votes, while inlier images account for 66% overall and vary sharply across categories.The cleanest category has 199 inliers among 200 images, whereas the worst has only 24 inliers.
4. Experimental Studies
The experiments assess WebVision’s baseline recognition performance, image quantity versus label quality, and transferability against ILSVRC 2012. WebVision models are competitive with or better than ILSVRC-trained models across validation and transfer settings.
- Baseline Model and Dataset Bias: WebVision trains an AlexNet baseline by treating query concepts as semantic labels for web images.Models are evaluated on WebVision and ILSVRC 2012 validation data using single-crop testing.
- Baseline Model and Dataset Bias: 57.03% top-1 and 77.90% top-5 accuracy are achieved by the WebVision AlexNet on its validation set.Cross-dataset testing reduces WebVision top-5 accuracy to 70.36%, while ILSVRC 2012 falls from 79.77% to 74.64%.
- Quantity vs. Quality: WebVision models outperform ILSVRC 2012 models at all four sampled image percentages, but label noise weakens performance when training data is limited.Using 50% of WebVision images gives 52.22% versus 52.58% with 100% of ILSVRC 2012, while 10% versus 25% gives 30.47% versus 39.65%.
- Quantity vs. Quality: 55.11% versus 52.58% top-1 accuracy is obtained with 1.28M WebVision versus ILSVRC 2012 images, indicating quantity can offset noisy labels.With 128K images, WebVision’s advantage disappears and its recognition accuracy becomes worse than ILSVRC 2012.
- Transfer Learning: WebVision features reach 70.43 versus 70.44 classification accuracy on Caltech-256 and 77.78 versus 75.65 mAP on PASCAL VOC 2007 against ILSVRC features.Concatenating both representations further improves results to 73.61 and 78.46, respectively.
- Transfer Learning: The WebVision model achieves comparable object-detection results to the ILSVRC 2012 model after fine-tuning on PASCAL VOC 2007.This evaluates transfer to a new vision task using the pretrained models.
5. Conclusions and Future Work
WebVision demonstrates that large-scale noisy web data can support robust visual recognition and transfer across datasets, while exposing domain differences for adaptation research. The database also motivates further work on noisy-label learning and richer use of web metadata.
- WebVision’s validation and test resources facilitate algorithmic development, while its noisy labels and crawled metadata motivate improved label handling and metadata integration.The database includes crawled web images and metadata, and the paper identifies noisy-label learning and metadata use as open research issues.
- WebVision’s large number of noisy web images is sufficient to train a robust deep CNN for visual recognition.The experiments report that image quantity can overcome noisy labels in web data.
- Comparable or better transfer results were achieved on Caltech-256 and PASCAL VOC 2007 for classification and object detection.
- A domain difference between WebVision and ILSVRC 2012 was verified qualitatively and quantitatively, supporting large-scale visual domain adaptation studies.