Source-linked AI summary
Learning Visual Features from Large Weakly Supervised Data
Armand Joulin, Laurens van der Maaten, Allan Jabri, Nicolas Vasilache
TL;DR
Existing progress in visual representation learning depends on increasingly large manually labeled datasets, whose annotation burden limits scalability. The paper trains convolutional networks from scratch on 100 million Flickr images and captions, finding features nearly on par with Imagenet-trained features while also learning semantic and cross-language word structure.
Problem
Improving visual features appears to require larger manually labeled datasets, but manual annotation is time-consuming, non-scalable, and task-biased.
Method
The paper trains convolutional networks from scratch on massive weakly supervised Flickr images paired with captions, using words and hashtags as targets.
Results
Weakly supervised models learn features nearly on par with those from an image collection containing over a million manually defined labels, while capturing semantic structure from image-word co-occurrences.
Takeaways & Limitations
Massive weakly supervised image collections can support good visual representations and learn word similarity, analogies, and relationships across languages.
Takeaways & Limitations
The weakly supervised networks perform poorly on the flowers dataset, where Imagenet-trained networks produce better features.
Abstract
from arXiv · showhide
Convolutional networks trained on large supervised dataset produce visual features which form the basis for the state-of-the-art in many computer-vision problems. Further improvements of these visual features will likely require even larger manually labeled data sets, which severely limits the pace at which progress can be made. In this paper, we explore the potential of leveraging massive, weakly-labeled image collections for learning good visual features. We train convolutional networks on a dataset of 100 million Flickr photos and captions, and show that these networks produce features that perform well in a range of vision problems. We also show that the networks appropriately capture word similarity, and learn correspondences between different languages.
1. Introduction
Large supervised datasets have enabled strong visual features, but manual annotation is costly, non-scalable, and task-biased. This paper investigates whether massive image-caption collections can instead train useful visual representations and semantic word structure.
- Motivation: Large manually annotated datasets have fueled convolutional features that achieve state-of-the-art results across several vision problems.These include fine-grained classification, object detection, and segmentation.
- Motivation: Manual annotation of increasingly large image datasets is time-consuming and does not scale well to improving recognition performance.The paper notes that COCO development required more than 20,000 annotator hours over two years.
- Motivation: Manual image selection and annotation can introduce a strong bias toward a specific task.The authors contrast this with weakly supervised and unsupervised learning in human vision.
- Approach: The paper trains convolutional networks from scratch on 100 million Flickr images and their associated captions without fully supervised data.The weakly labeled collection is drawn from a publicly available photo-sharing dataset.
- Approach: Weakly supervised data offers abundant supply, reduced task bias, and greater similarity to how humans learn vision.The paper presents these as three advantages of image-caption training data.
- Results: Weakly supervised models learn image-relevant words, visual features nearly as good as Imagenet-trained features, semantic word structure, and cross-language correspondences.The reported semantic structure includes word similarity and analogies, while multilingual training relates words assigned to similar visual inputs.
2. Related Work
Prior work used weak supervision for curriculum learning, feature construction, label cleansing, and multimodal embedding. This paper differs by learning visual features through the full recognition pipeline from noisy data.
- Weak supervision: Earlier work combined easy Google Images examples with weakly labeled image-hashtag pairs in a two-stage curriculum-learning approach.Those results suggested outperforming models trained solely on image-hashtag data, with the authors attributing the comparison partly to dataset size.
- Weak supervision: Other studies used weakly supervised data for topic modeling, incremental object learning, multiple-instance learning, or image-hashtag embeddings with predefined visual features.These approaches generally operated on hand-crafted or pre-existing visual representations rather than learning the full visual pipeline.
- Noisy labels: Label-cleansing methods identify relevant image-hashtag pairs to construct cleaner datasets rather than training recognition pipelines directly on noisy labels.The paper instead studies whether noisy weak supervision can support end-to-end recognition learning.
- Multimodal embedding: Related multimodal-embedding work co-embeds images with words, sentences, or n-grams using methods including kernel CCA, restricted Boltzmann machines, topic models, and logbilinear models.The paper places its approach within this broader family while emphasizing end-to-end visual-feature learning.
3. Weakly Supervised Learning of Convnets
The models learn visual features from Flickr images paired with noisy word targets, using end-to-end convolutional networks optimized with logistic losses. The experiments compare one-versus-all and multiclass objectives while addressing class imbalance and approximate-loss behavior.
- Data and preprocessing: The training data consists of Flickr photos paired with titles, hashtags, and captions, converted into multi-label word targets.The vocabulary uses the 1,000, 10,000, or 100,000 most common words after preprocessing; each image averages 3.72, 5.62, or 6.81 targets, respectively.
- Network architecture: AlexNet and GoogLeNet map images to intermediate embeddings and jointly learn label mappings through end-to-end optimization.The architectures differ from their standard versions only in the size of the final output layer.
- Loss functions: The models optimize either one-versus-all logistic loss or multiclass logistic loss over the positive labels.The one-versus-all objective sums binary classifier losses, while the multiclass objective uses softmax probabilities.
- Loss-function selection: Pairwise ranking loss was abandoned because its sparse updates significantly slow end-to-end convolutional-network training.Each example updates only the columns corresponding to one positive and one negative label.
- Class balancing and training: Uniform-per-class sampling counters the Zipfian word distribution by selecting a word uniformly and then an associated image.Other words attached to the selected image are treated as negatives, potentially producing noisier gradients.
- Approximate loss: For multiclass training, the stochastic approximation can be arbitrarily far from the true loss in the worst case, but expected-value bounds show close practical correspondence.The lower bound differs by only the additive constant log(|C|/K), and is exact when |C| → K.
4. Experiments
The experiments evaluate weakly supervised Flickr-trained networks for word prediction, transferable visual features, visual organization, and semantic word embeddings. Results show benefits from large-scale training, competitive transfer features, learned visual taxonomies, and multilingual correspondences, with weaker performance on fine-grained flowers.
- Experimental design: The study evaluates word prediction, transfer learning across seven vision datasets, and the quality of learned word embeddings.The transfer-learning experiments use logistic regressors on penultimate-layer features from Flickr- and ImageNet-trained networks.
- Associated word prediction: 45 to 110% relative gain in precision@10: end-to-end Flickr training outperformed classifiers trained on ImageNet features for word prediction.The comparison uses Flickr-trained convolutional networks and L2-regularized logistic regressors on ImageNet-pretrained features.
- Associated word prediction: 16.43 versus 17.98 precision@10: multiclass logistic loss outperformed one-versus-all loss with K = 1,000 words.The authors attribute the difference to class imbalance and gradient spikes affecting one-versus-all training.
- Associated word prediction: ∼50 million images: word prediction performance continued improving beyond 1 million images and leveled out only after roughly this scale.The learning-curve experiment used AlexNet with K = 1,000.
- Visual and semantic structure: The learned features grouped diverse sports while keeping individual sports separable, and the embeddings captured semantic structure and English-French correspondences.The sports organization appeared in a t-SNE map of 20,000 Flickr test images; multilingual matches spanned objects, locations, and concepts.
- Transfer learning: Weakly supervised features performed well across transfer tasks, although ImageNet-trained features were better on the fine-grained Oxford Flowers dataset.The transfer experiments covered Indoor, SUN, Stanford 40 Actions, Flowers, Sports, ImageNet, and Pascal VOC; tens of millions of images were required for good weakly supervised features.
5. Discussion and Future Work
The study shows that convolutional networks trained from scratch on weakly supervised data can learn broadly useful visual features and semantic structure. It also identifies practical directions for improving weakly supervised learning and extending these models to language-based tasks.
- Weakly supervised networks learn good features without manual annotation, achieving results nearly on par with features learned from over a million manually defined labels across varied datasets.The paper notes that state-of-the-art results would require averaging predictions over many crops and models, which was outside its scope.
- Future models may need greater capacity because architectures that perform best on ImageNet may not be optimal for the more complex Flickr dataset.
- Uniform per-class sampling is essential because it prevents frequent classes from dominating features intended for transfer learning.
- Future work proposes combining weakly supervised vision models with language models for tasks such as visual question answering.