Source-linked AI summary
Webly Supervised Learning of Convolutional Networks
Xinlei Chen, Abhinav Gupta
TL;DR
The paper addresses how to train CNNs from abundant but noisy and biased web data without extensive human supervision. It bootstraps from easy Google images, adapts to harder Flickr images using category relationships, and reports strong VOC results, including state-of-the-art VOC 2007 performance without VOC training images.
Problem
CNN training from noisy web labels remains difficult, while large-scale human supervision is costly and webly supervised systems have lagged supervised methods.
Method
A two-stage CNN is trained first on easy Google images, then fine-tuned on harder Flickr images using a fixed relationship graph between categories.
Results
The two-stage CNN outperforms an ImageNet-pretrained CNN on VOC 2012 and achieves state-of-the-art VOC 2007 performance without VOC training images.
Takeaways & Limitations
Web data can support CNN representations, object localization, and detection at scales beyond manually labeled datasets within the demonstrated tasks.
Takeaways & Limitations
Webly trained detectors suffer localization errors from background inclusion, multiple instances, spatial invariance, and semantic drift between web and PASCAL categories.
Abstract
from arXiv · showhide
We present an approach to utilize large amounts of web data for learning CNNs. Specifically inspired by curriculum learning, we present a two-step approach for CNN training. First, we use easy images to train an initial visual representation. We then use this initial CNN and adapt it to harder, more realistic images by leveraging the structure of data and categories. We demonstrate that our two-stage CNN outperforms a fine-tuned CNN trained on ImageNet on Pascal VOC 2012. We also demonstrate the strength of webly supervised learning by localizing objects in web images and training a R-CNN style detector. It achieves the best performance on VOC 2007 where no VOC training data is used. Finally, we show our approach is quite robust to noise and performs comparably even when we use image search results from March 2013 (pre-CNN image search era).
1. Introduction
The paper asks whether web data can train CNNs effectively despite noise, bias, and limited human supervision. It proposes exploiting larger-scale web data to improve visual recognition and detection without equivalent manual labeling.
- Motivation: Web sources offer enormous visual data, but human filtering and localization labeling are costly and can introduce inconsistency, incompleteness, and bias.Existing clean datasets such as ImageNet, PASCAL VOC, and MS COCO rely on human intelligence to remove noise and label object locations.
- Motivation: Webly supervised systems process far more images yet historically have not matched methods trained with extensive human supervision.The paper frames this performance gap as the central question motivating its approach.
- Challenges: Web data is problematic because it contains noise and bias, including centered objects, clean backgrounds, and canonical viewpoints in image-search results.Google image search is described as high-precision and low-recall, while Flickr images are more realistic but noisier.
- Approach: The paper presents a two-stage webly supervised CNN approach that starts with easy search-engine images and adapts to harder web images using learned relationships.The approach is motivated by using more accessible examples before broader, noisier data.
- Reported scope: The approach reports competitive or state-of-the-art results on relevant vision tasks, including VOC detection without using VOC training images and competitive scene classification.The authors position web data as a route toward much larger datasets without human labeling costs.
2. Related Work
Prior work studied web-data sources, data types, exploitation algorithms, and learning goals, but human supervision remained stronger despite vastly larger web datasets. This paper instead combines curriculum-inspired bootstrapping with CNN representations and relationship constraints for noisy data.
- Open challenge: Despite processing orders of magnitude more images, earlier web learners had not matched contemporary methods receiving extensive human supervision.This motivates a learning strategy that can better use noisy, large-scale data.
- Proposed direction: The approach assumes a CNN can learn a robust relationship graph from simple examples and use it to constrain learning from noisier data.The paper distinguishes this webly supervised setting from unsupervised learning because semantic supervision comes from text.
- Proposed direction: The paper uses curriculum learning by training first on easy Google images, then adapting to harder Flickr images with similarity constraints and a relationship graph.The graph provides constraints when the network encounters more challenging and noisier data.
3. Approach
The approach trains CNNs progressively from cleaner Google images to noisier Flickr images, using learned category relationships to adapt representations and localize objects for detection.
- Initial Network: Random noun-phrase web queries can fail to train a CNN, motivating curated category lists and a staged learning strategy.A CNN trained on approximately 5M Google images from 7,000 randomly sampled noun phrases did not converge.
- Initial Network: The first stage trains a CNN from scratch on easy Google image-search images, while the second adapts it to harder Flickr images.Flickr images are more realistic but noisier than Google results, creating a curriculum from cleaner to harder examples.
- Representation Adaptation with Graph: The method builds a category-relationship graph from the initial network’s confusion matrix and adds it above the seventh layer during fine-tuning.The graph models interrelated rather than mutually exclusive categories, and its initially learned structure is kept fixed to avoid semantic drift.
- Representation Adaptation with Graph: Flickr training is noisier than Google training, with entropy approximately 4.0 versus 2.8, while complete random noise yields approximately 7.3.The comparison uses the same approximately 1,500 categories and a close-to-uniform label distribution.
- Localizing Objects: For object detection, seed images provide bounding boxes; Exemplar-LDA and EdgeBox proposals find neighbors, which are clustered, denoised, and used to train R-CNN detectors.Positive examples are further expanded with overlapping EdgeBox regions and related categories from the relationship graph.
4. Experimental Results
The experiments evaluate webly trained CNN representations for recognition, detection, localization, and robustness. Across these tasks, the two-stage approach benefits from category relationships, realistic Flickr data, and scene-related categories, while revealing localization and semantic-drift errors.
- Experimental Setup: The experiments test webly learned CNNs through object detection, representation generalization, relationship discovery, localization, and scene classification.The evaluation includes VOC detection with and without fine-tuning, confusion-matrix analysis, web-only localization and detection, and scene classification.
- PASCAL VOC Object Detection: The two-stage CNN with graph regularization performs on par with the ImageNet-trained CNN on VOC 2007 despite web-search bias and noisy data.The reported comparison concerns VOC 2007 detection using the webly trained representation.
- PASCAL VOC Object Detection: 40.5 mAP is obtained when training directly on all Google and Flickr images, substantially below the proposed approach.The result is cited as evidence that directly training on noisy, hard Flickr images can hurt learning.
- PASCAL VOC Object Detection: 43.4 mAP is achieved without graph regularization, while adding graph regularization raises the final FlickrG network to ImageNet-level performance at 44.7 mAP.The two-stage strategy uses realistic Flickr data, and graph regularization improves the resulting detector.
- PASCAL VOC Object Detection: On VOC 2012, the webly supervised networks outperform the fine-tuned ImageNet pretrained network, with stronger performance on vehicles and weaker performance on some animals.The authors attribute the vehicle advantage to roughly 500 vehicle classes and the animal disadvantage to ImageNet’s larger animal data.
- Robustness Analysis: Using pre-CNN-era March 2013 image-search results causes only an approximately 1% performance drop, indicating robustness to noise and little dependence on Google’s underlying CNN.The older data was noisier and contained roughly 450 images per category.
- Web-only Detection and Localization: The webly CNN discovers category relationships and localizes positive instances, enabling R-CNN-style detectors without PASCAL training images.Qualitative localization is less homogeneous than prior subcategory discovery, while the framework is reported to achieve state-of-the-art VOC 2007 performance without VOC training data.
- Scene Classification: Scene-related categories raise scene-classification accuracy to 66.5, comparable to the Places-trained CNN’s 68.2, indicating generic web-learned features.The comparison is reported for the default train/test split.
5. Conclusion
The paper presents a two-stage method for training CNNs from noisy web data and reports strong transfer to recognition tasks without relying on VOC training images.
- The method first trains a CNN on easy Google image-search images, then fine-tunes it on realistic Flickr images using a relationship graph.
- The two-stage CNN comes close to an ImageNet-pretrained CNN on VOC 2007 and outperforms it on VOC 2012.
- The approach achieves state-of-the-art performance on VOC 2007 without using any VOC training image.
- Webly supervised learning is presented as suited to semantic tasks such as detection and classification, whereas unsupervised learning targets generic tasks that may not require semantic invariance.