PaperlayerView exact revision on arXiv

Source-linked AI summary

Exploring the Limits of Weakly Supervised Pretraining

Dhruv Mahajan, Ross Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, Laurens van der Maaten

arXiv:1805.00932v1cs.CV

TL;DR

Visual perception pretraining has largely relied on ImageNet, while evidence about datasets orders of magnitude larger remains limited. This paper studies convolutional networks trained to predict hashtags on billions of Instagram images and finds excellent transfer performance, including improvements across classification and detection tasks and 85.4% ImageNet-1k single-crop top-1 accuracy.

  • Problem

    Relatively little is known about pretraining on datasets multiple orders of magnitude larger than ImageNet, despite pretraining's central role and the difficulty of collecting and annotating such data.

  • Method

    The paper trains large convolutional networks to predict thousands of social-media hashtags on billions of Instagram images, using weak labels without manual curation or sophisticated data cleaning.

  • Results

    85.4% single-crop, top-1 accuracy is achieved on ImageNet-1k, alongside improvements over ImageNet-1k pretraining on image classification and COCO object detection.

  • Takeaways & Limitations

    Hashtag prediction on large-scale wild data can provide excellent transfer learning performance and supports empirical study of how pretraining scale and label-space choices relate to transfer.

  • Takeaways & Limitations

    Other research groups cannot know exactly which images were used or download the datasets en masse, preventing replication at this time.

Abstract

from arXiv · show

State-of-the-art visual perception models for a wide range of tasks rely on supervised pretraining. ImageNet classification is the de facto pretraining task for these models. Yet, ImageNet is now nearly ten years old and is by modern standards "small". Even so, relatively little is known about the behavior of pretraining with datasets that are multiple orders of magnitude larger. The reasons are obvious: such datasets are difficult to collect and annotate. In this paper, we present a unique study of transfer learning with large convolutional networks trained to predict hashtags on billions of social media images. Our experiments demonstrate that training for large-scale hashtag prediction leads to excellent results. We show improvements on several image classification and object detection tasks, and report the highest ImageNet-1k single-crop, top-1 accuracy to date: 85.4% (97.6% top-5). We also perform extensive experiments that provide novel empirical data on the relationship between large-scale pretraining and transfer learning performance.

1 Introduction

Modern visual perception models commonly pretrain on manually annotated ImageNet before task-specific finetuning, but little is known about pretraining with datasets orders of magnitude larger. This paper studies billions of hashtag-labeled social-media images despite concerns about noisy labels and biased image distributions.

  • Most state-of-the-art visual perception algorithms pretrain convolutional networks on large manually annotated classification datasets before finetuning on smaller task-specific datasets.This approach has improved object detection, semantic segmentation, pose estimation, video recognition, and monocular depth estimation.
  • ImageNet is the de facto pretraining dataset, yet relatively little is known about pretraining on datasets multiple orders of magnitude larger.Large-scale studies are limited because few such datasets exist, constructing them is labor intensive, and experiments require substantial computational resources.
  • Billions of social-media images labeled with hashtags provide a large, continuously growing, manually annotation-free data source for studying this unexplored regime.The paper uses hashtags as labels for transfer-learning experiments.
  • Hashtag supervision may be too noisy, while biases in the image distribution could harm transfer learning, so its effectiveness is not obvious beforehand.These concerns motivate an empirical study rather than an assumption that scale alone will improve transfer.
  • 85.4% single-crop, top-1 accuracy on ImageNet-1k and 45.2% AP on COCO demonstrate excellent transfer performance from billions of Instagram images without manual curation or sophisticated cleaning.The same models trained or pretrained on ImageNet-1k reached 79.8% and 43.7%, respectively, on those metrics.

2 Scaling up Supervised Pretraining

The study scales hashtag-supervised pretraining to billions of public Instagram images using a simple hashtag-selection and canonicalization pipeline, then evaluates transfer across datasets and model configurations. It also examines dataset construction, deduplication, multi-label loss design, and large-scale distributed training.

  • Datasets: 3.5 billion public Instagram images are used for hashtag prediction, with datasets varied by selected hashtags, sampled image count, role, source, and label count.Dataset names encode role, source, approximate image count, and approximate label count.
  • Datasets: Hashtag datasets are built by selecting hashtags, downloading images carrying them, merging synonymous hashtags through WordNet synsets, and discarding unselected labels.Canonical hashtags become the labels used for training and evaluation.
  • Datasets: The study compares approximately 1.5k ImageNet-aligned hashtags with approximately 17k WordNet-synonym and 8.5k frequent-hashtag sets representing broader visual distributions.The first set is hypothesized to resemble ImageNet-1k visually, while the other sets are more general.
  • Dataset integrity: 0.41% and 0.12% duplicate rates are reported for specified validation sets, and the authors state that these small percentages do not impact their findings.Observed accuracy is reported, with a conservative duplicate-marked lower bound provided in supplementary material.
  • Dataset integrity: Other research groups cannot know exactly which images were used or download the datasets en masse, so the results cannot currently be replicated by others.The authors nevertheless argue that publishing the study is preferable to withholding its findings.
  • Models: ResNeXt-101 32×Cd models are evaluated across configurations ranging from 8B to 153B multiply-add FLOPs and 43M to 829M parameters.The configurations vary group width while keeping 101 layers and 32 groups.
  • Models: Each multi-label image target distributes probability uniformly across its k ≥1 hashtags, and the model minimizes cross-entropy against this target distribution.The model predicts probabilities over the hashtag vocabulary with a softmax activation.
  • Models: Sigmoid outputs with binary logistic loss perform significantly worse than the softmax cross-entropy formulation on the multi-label hashtag data.The authors note that successful sigmoid training may require label completion and additional hyperparameter search.

3 Experiments

Experiments show that weakly supervised pretraining on billions of Instagram images transfers effectively across classification and detection settings. Transfer quality depends on label-space alignment, data scale, model capacity, sampling, and robustness to noisy hashtags.

  • Image Classification Experiments: 84.2% top-1 accuracy on ImageNet-1k was achieved using nearly 1B Instagram images and 1.5k hashtags, improving 4.6% over the same architecture trained on IN-1k alone.This also represented a 1.5% improvement over the prior state of the art.
  • Hashtag Vocabulary: Label-space alignment matters: 1.5k hashtags matched to ImageNet classes outperform larger vocabularies on IN-1k, whereas 17k hashtags prevail as target classes and visual diversity increase.On IN-9k, the accuracy difference between 1.5k and 17k hashtags is approximately 7%; on CUB2011, 17k becomes better after roughly 100M images.
  • Feature Transfer: 83.3% and 83.6% top-1 accuracy were achieved on val-IN-1k with fixed features from IG-940M-1k and IG-3.5B-17k, respectively, using only a trained linear classifier.These results were nearly as good as full-network finetuning, and repeated 1B-image samples differed by less than 0.1% top-1 accuracy.
  • Label Noise: 10% injected label noise reduced classification accuracy by less than 1%, while 25% noise reduced it by around 2%, indicating substantial robustness.The evaluation trained only the final linear classifier so that pretraining noise effects were not masked by full finetuning.
  • Sampling: Uniform or square-root hashtag sampling improved ImageNet transfer accuracy by 5 to 6% across target class counts, while larger vocabularies increasingly helped as target classes grew.The comparison used IG-1B-17k pretraining and evaluated ResNeXt-101 32×4d models with linear classifiers.
  • Model Capacity: 85.4% top-1 and 97.6% top-5 accuracy on ImageNet-1k were obtained by increasing ResNeXt model capacity from 32×16d to 32×48d.The corresponding top-1 progression was 84.2% to 85.1% to 85.4% for the 32×16d, 32×32d, and 32×48d models.

4 Related Work

Prior work explored weakly supervised pretraining on datasets smaller than or comparable to ImageNet, while this study scales hashtag prediction to substantially larger image collections.

  • JFT-300M provided a weakly supervised benchmark, but the Instagram datasets here are an order of magnitude larger and required less manual annotation.The comparison concerns both dataset scale and annotation effort.
  • Earlier Flickr studies predicted words or n-grams from comments, whereas hashtag supervision is more structured and transferred features more effectively to ImageNet.Prior hashtag work on Flickr did not investigate transfer of the resulting representations.

5 Discussion

The study reports strong benchmark performance and identifies label-space alignment, model capacity, and benchmark visual variety as important factors in large-scale pretraining.

  • 85.4% single-crop top-1 and 97.6% single-crop top-5 accuracy were obtained on ImageNet-1k, alongside state-of-the-art results on several other vision tasks.The discussion frames these results as part of an exploration of the limits of supervised pretraining.
  • Label-space alignment can outweigh dataset scale: an ImageNet-matched hashtag vocabulary outperformed a vocabulary trained on twice as many images.The authors propose label-space engineering as a way to select weakly supervised labels for particular target tasks.
  • Larger networks may improve target-task accuracy because current architectures underfit when trained on billions of images.The paper also notes that architectures designed around ImageNet-1k may need reconsideration for hashtag supervision.
  • Increasing benchmark visual variety makes differences between learned feature quality more pronounced, especially as the number of target classes grows.Evaluations limited to ImageNet-1k can therefore understate differences between pretraining vocabularies.
  • Hashtag prediction works without additional manual annotation or data cleaning, illustrating the potential of publicly available “wild” data.The authors contrast this approach with manually designed and annotated datasets.

A.1 Hashtag Selection

The hashtag-selection pipeline filters and canonicalizes Instagram hashtags using WordNet synsets before using them as training and evaluation labels.

  • The dataset pipeline selects hashtags, samples tagged images, merges synonymous hashtags into canonical forms, and discards unselected labels.Canonical hashtags become the labels used for training and evaluation.
  • WordNet matching retains hashtags that match at least one synset in a chosen synset set, using the hashtag and hashtag-derived bigrams as query strings.The matching function returns the subset of selected synsets associated with each hashtag.
  • Two hashtags are treated as duplicates when they match exactly the same set of WordNet synsets.Using all WordNet synsets makes this a conservative deduplication rule across possible word senses.

A.2 Image Deduplication

The deduplication pipeline searches billions of Instagram images for near-duplicates, verifies candidates with uncompressed features, and reports both observed and conservative accuracy bounds.

  • A two-stage procedure first retrieves 128 potential duplicates per query image from 3.5 billion images, then computes pairwise distances and applies a conservative threshold.The pipeline uses approximate nearest-neighbor search followed by higher-quality feature comparisons.
  • The first stage uses truncated ResNet-50 R-MAC features, resized images, and regional maximum activations to support large-scale retrieval.R-MAC features are computed from fixed sub-regions of convolutional feature maps.
  • A 28-bit product-quantized representation indexes the 3.5 billion images after normalization, OPQ reduction, and coarse quantization.The compressed index enables efficient nearest-neighbor search over the full image collection.
  • Queries are assigned to 256 nearest sub-quantizers, whose residual distances select 128 nearest neighbors efficiently.Distance estimates are used with a max-heap implementation.
  • The second stage computes exact squared Euclidean distances between 2048-dimensional R-MAC features and applies a 0.6 threshold.This stage avoids feature compression when verifying the retrieved candidates.
  • Duplicate rates were low across validation sets, ranging from 0.12% on COCO to 0.41% on Places, and conservative lower bounds treated duplicates as incorrect.The reported main-paper accuracies included the detected duplicates.

A.3 Training Details

Training details specify consistent optimization settings across ImageNet and Instagram pretraining, with separate procedures for feature transfer, full-network finetuning, and detection. The authors also note an unperformed duplicate analysis for ImageNet-pretrained transfer.

  • SGD uses Nesterov momentum of 0.9 and weight decay of 0.0001, with weight decay excluded from batch-normalization scale and bias parameters.
  • Approximately 5% of val-CUB-6k-200 images were estimated to overlap train-IN-1M-1k, but the corresponding duplicate analysis was not performed.
  • ImageNet and Instagram pretraining share the same initialization, image rescaling, augmentation, and batch-normalization settings.
  • Feature transfer trains a convex L2-regularized logistic regressor, selecting penalty 0.001 for train-CUB-200 and 0.0001 for all other datasets.
  • Detection finetuning requires grid search for the initial learning rate because ImageNet defaults performed poorly with Instagram-pretrained models.
  • Full-network finetuning selects initial learning rate and weight decay by coarse grid search using a held-out validation set.

A.4 Data Resampling

The resampling procedure addresses the highly skewed hashtag distribution by comparing natural, uniform, and square-root sampling schemes. Images are replicated according to hashtag frequencies to create datasets with controlled size and visual distributions.

  • Hashtag frequencies are Zipfian: in the 17k vocabulary, #fineart appears over 1 million times as often as #shirtfront.
  • Uniform and square-root sampling use replication factors r(h) = max(1, φ(t/f(h))), with φ(x) equal to x or √x respectively.
  • For each image, the replication factor is the maximum factor among its hashtags, after which replicated images are randomly permuted into the training list.
  • The threshold t is selected so the final replicated list matches the desired training-schedule length.

A.5 Comparison with the State of the Art on ImageNet-1k

The comparison appends Instagram-hashtag-pretrained ResNeXt-101 results to a state-of-the-art ImageNet-1k validation table. Results use single image crops and evaluate finetuning on train-IN-1k.

  • Instagram hashtag pretraining is evaluated with ResNeXt-101 32×Cd models, using C ∈ {16, 32, 48}, followed by finetuning on train-IN-1k.
  • All reported ImageNet-1k validation results use a single squared image crop of the specified size.
  • The table reports that pretraining on billions of hashtag-labeled images significantly improves ImageNet-1k results relative to the appended state-of-the-art comparison.
Loading 1805.00932v1…