PaperlayerView exact revision on arXiv

Source-linked AI summary

Revisiting Weakly Supervised Pre-Training of Visual Perception Models

Mannat Singh, Laura Gustafson, Aaron Adcock, Vinicius de Freitas Reis, Bugra Gedik, Raj Prateek Kosaraju, Dhruv Mahajan, Ross Girshick, Piotr Dollár, Laurens van der Maaten

arXiv:2201.08371v2cs.CV

TL;DR

Weakly supervised pre-training must balance semantic signal against the scalability limits of manual labels and the limited semantics of self-supervision. This paper revisits hashtag supervision with larger models and a much larger dataset, evaluates transfer including zero-shot settings, and finds the approach highly competitive while documenting unresolved fairness and evaluation limitations.

  • Problem

    Pre-training needs scalable semantic supervision, but fully supervised and self-supervised approaches occupy different trade-offs in labeling cost and semantic information.

  • Method

    The paper trains modern image-recognition models on public Instagram images and canonicalized hashtags, then evaluates transfer, zero-shot performance, self-supervised comparisons, and harmful associations.

  • Results

    Weakly supervised models are very competitive across transfer settings, substantially outperform current self-supervised learners in the reported comparisons, and perform on par with state of the art across visual-perception tasks.

  • Takeaways & Limitations

    The results provide a compelling argument for using weakly supervised pre-training in developing visual-recognition systems.

  • Takeaways & Limitations

    The study lacks well-established datasets and protocols for rigorous evaluation of user-group accuracy and harmful stereotypes, making its initial analyses inconclusive.

Abstract

from arXiv · show

Model pre-training is a cornerstone of modern visual recognition systems. Although fully supervised pre-training on datasets like ImageNet is still the de-facto standard, recent studies suggest that large-scale weakly supervised pre-training can outperform fully supervised approaches. This paper revisits weakly-supervised pre-training of models using hashtag supervision with modern versions of residual networks and the largest-ever dataset of images and corresponding hashtags. We study the performance of the resulting models in various transfer-learning settings including zero-shot transfer. We also compare our models with those obtained via large-scale self-supervised learning. We find our weakly-supervised models to be very competitive across all settings, and find they substantially outperform their self-supervised counterparts. We also include an investigation into whether our models learned potentially troubling associations or stereotypes. Overall, our results provide a compelling argument for the use of weakly supervised learning in the development of visual recognition systems. Our models, Supervised Weakly through hashtAGs (SWAG), are available publicly.

1. Introduction

The paper frames pre-training as a trade-off among semantic supervision, scalability, and noise. It studies hashtag-based weak supervision as a scalable middle ground and evaluates its transfer performance and potential harmful associations.

  • Fully supervised pre-training provides strong semantic signals but scales poorly because manual labeling is time-consuming.
  • Self-supervised pre-training scales readily to billions of examples but provides little semantic information per training example.
  • Weakly supervised pre-training uses noisy semantic signals from hashtags or related text that can be collected at large scale.
  • The study trains hashtag-supervised models, evaluates standard, zero-shot, and few-shot transfer, and compares them with other pre-training approaches.
  • The authors find their weakly supervised models competitive with the state of the art across visual-perception tasks despite a relatively simple training pipeline.
  • The paper investigates whether weakly supervised models inherit or amplify harmful associations, but reports that its experiments do not provide conclusive answers.

2. Related Work

The paper situates hashtag-supervised pre-training within established supervised and emerging self-supervised approaches. It extends prior hashtag-based work with larger models, a larger dataset, broader transfer evaluations, and comparisons with self-supervised learning.

  • Visual-recognition pre-training includes fully supervised, self-supervised, and weakly supervised approaches, with ImageNet-1K the most common image pre-training dataset.
  • The study builds on prior work that trained convolutional networks on billions of images to predict associated hashtags.
  • Compared with prior hashtag-based work, this study uses larger models, more efficient architectures, and a much larger dataset.
  • The study adds zero-shot transfer evaluation, comparisons with state-of-the-art self-supervised learners, and an in-depth analysis of potentially harmful associations.
  • The best model achieves more than 3% higher ImageNet-1K validation accuracy than the result reported in the prior hashtag-based study.

3. Pre-Training using Hashtag Supervision

The paper constructs a large multi-label dataset from public Instagram images and canonicalized hashtags, then trains modern image-recognition models to predict those hashtags. Its pipeline addresses noisy, imbalanced supervision through filtering, canonicalization, resampling, and model selection.

  • Hashtag supervision trains image-recognition models to predict the hashtags assigned to each image by its uploader.
  • Hashtags provide noisy, incomplete semantic supervision because they may be unrelated to visual content, ambiguous, or absent for visible content.
  • The dataset pipeline selects and canonicalizes frequent hashtags, collects matching public images, forms labeled examples, and resamples their distribution.
  • Figure 1 separates canonical hashtags occurring with more than 5,000 images into the head and places the remaining hashtags in the tail.
  • Canonicalization produces approximately 27,000 canonical hashtags corresponding to approximately 75,000 raw hashtags, while filtering removes potentially offensive content.
  • The IG-3.6B dataset contains approximately 3.6 billion unique images, while resampling processes approximately 5 billion samples per epoch.
  • Preliminary experiments identify RegNetY and ViT as the most competitive architectures among the evaluated model families.
  • Models use a softmax classifier over approximately 27,000 hashtag classes and minimize cross-entropy against multi-label target distributions.

4. Experiments

The experiments evaluate hashtag-supervised models across standard, zero-shot, and few-shot transfer settings, comparing them with supervised and self-supervised pre-training. The models are competitive across five transfer datasets, substantially outperform self-supervised learners in low-shot settings, and show accuracy-throughput trade-offs across architectures.

  • Experimental setup: The study evaluates linear-classifier, finetuning, zero-shot, and few-shot transfer, comparing hashtag-supervised models with fully supervised and self-supervised pre-training.Experiments transfer models to image-classification tasks, including five datasets and ImageNet-1k low-shot settings.
  • Comparison with supervised pre-training: The weakly supervised models achieve the best or second-best accuracy on all five transfer datasets.The comparison reports pre-training data, resolution, throughput, compute, parameters, and transfer accuracy, while noting possible CUB-2011 overlap for ImageNet-pretrained models.
  • Accuracy-throughput trade-off: Among models trained on IG-3.6B, ViTs reach the highest classification accuracies, while RegNetYs provide a better accuracy-throughput trade-off at small to medium model sizes.RegNetY 128GF performs similarly to EfficientNet L2 on accuracy and throughput, while smaller RegNetYs offer a better trade-off.
  • Comparison with self-supervised pre-training: Weakly supervised pre-training substantially outperforms current self-supervised learners, particularly in low-shot transfer settings.The authors attribute this pattern to more learning signal per sample, while noting that observations may change if self-supervised learners are scaled further.
  • Zero-shot transfer: Without seeing ImageNet images, the best weakly supervised model reaches 75.3% ImageNet top-1 accuracy in zero-shot transfer.Platt scaling is essential for good zero-shot performance, and RegNetY models outperform ViTs in this setting.
  • Zero-shot transfer: Zero-shot comparisons are mixed: CLIP ViT L/14 slightly outperforms the authors’ model on IN-1k, smaller CLIP RN50×64 underperforms it, and the authors’ models perform best on ImageNet-v2.The system-level comparisons make the sources of performance differences difficult to determine.

5. Broader Impact

The paper examines harmful associations and geographic performance differences in hashtag-prediction models, finding troubling race-related associations and uneven country-level accuracy while cautioning that dataset bias complicates interpretation.

  • Association analysis: The models were evaluated for associations involving apparent skin tone, age, gender, and race using Instagram and UTK Faces datasets.The Instagram dataset contains 178,448 photos annotated with the Fitzpatrick skin tone scale; UTK Faces provides apparent age, gender, and race labels.
  • Association analysis: The models associated hashtags such as #redhead and #black with different skin tones, and #baby or #kid with apparent age.
  • Association analysis: The most troubling result was more frequent association of photos containing Black people with #mugshot and #prison.
  • Interpretation: The interpretation is confounded by UTK Faces containing many mug shots that disproportionately portray Black individuals.
  • Geographic fairness: Country-level accuracy differences remained substantial, but a larger proprietary dataset placed per-country accuracies between 65% and 70%.

6. Discussion

The discussion argues that weakly supervised pre-training has substantial potential, while emphasizing that uncontrolled system-level comparisons and inadequate fairness evaluation limit current conclusions and deployment readiness.

  • Overall assessment: The study presents an in-depth comparison of fully supervised, self-supervised, and weakly supervised pre-training and argues for weak supervision in visual perception.
  • Evaluation limitations: Controlled comparisons are increasingly difficult because data, architectures, training recipes, hardware, compute, and model availability vary across studies.
  • Evaluation limitations: System-level comparisons provide evidence about potential but cannot produce conclusive results, especially when benchmark accuracies are saturated.
  • Broader evaluation: The field’s focus on recognition accuracy and inference speed overlooks user-group performance and harmful-stereotype prevalence.
  • Broader evaluation: The paper reports that fairness analyses are not yet conclusive because well-established datasets and protocols for rigorous evaluation are lacking.
  • Outlook: Resolving evaluation and fairness issues could allow weakly supervised learning to improve visual-perception systems, in the authors’ view.

A. Additional Dataset Details

The dataset pipeline canonicalizes hashtags into WordNet-based labels, balances head and tail hashtags during sampling, and evaluates collection choices alongside model speed and family selection.

  • Hashtag filtering and canonicalization: Hashtags are mapped to WordNet synsets, empty mappings are removed, and equivalent synset sets become shared canonical labels.For example, #eggplant and #aubergine share a target, while #newyork is filtered out.
  • Image sampling: Images are sampled with importance proportional to the maximum inverse-square-root frequency factor of their associated canonical hashtags.This reduces the relative sampling weight of frequent hashtags.
  • Dataset ablations: Dataset ablations show that α=0.7 tail oversampling improves transfer, whereas α=0.3 worsens it; the reported collection changes boost ImageNet-1K transfer by 2.1% in one 100M-sample setting.The 2.1% result uses ResNeXt-101 32x8d and may change with dataset or model scale.
  • Image sampling: The sampling procedure uses a 70% head and 30% tail allocation, substantially upsampling infrequent hashtags.A tail image contains at least one tail hashtag; duplicates can therefore occur within an epoch.
  • Model family comparison: RegNetY outperforms the other evaluated convolutional families across ImageNet-1k, ImageNet-5k, and IG-1B transfer settings at similar training budgets.The comparison considers ResNeXt, RegNetY, DenseNet, and EfficientNet, with throughput used as the efficiency criterion.

C.2. Effect of Dataset Size

The study separates unique images from total image-label samples and examines how model scale and training-set scale affect ImageNet transfer under a fixed training regime.

  • Dataset size: The final dataset is characterized by both unique images and total image-label samples because tail upsampling creates duplicate samples.Table 7 holds total training samples constant at 2 billion while varying the number of unique images.
  • Scaling effects: Transfer accuracy improves with larger models and longer training, while larger models benefit more from additional pre-training samples.The experiments vary RegNetY models from 16 to 128 GFLOPs and measure ImageNet-1k linear-classifier accuracy.
  • Scaling effects: Figure 3 reports transfer accuracy against both total samples seen and total training time in GPU-days for four model sizes.Training time is calculated from total samples and measured training speeds.
  • Fine-tuning setup: Increasing synchronized batch-normalization batch size continues to improve ImageNet fine-tuning accuracy for RegNetY 128GF under a fixed mini-batch size of 512.The reported results omit EMA.

D. Training Details

Zero-shot transfer uses WordNet to create a many-to-many hashtag-to-ImageNet-class mapping, then aggregates hashtag predictions into class predictions with model-specific rules.

  • Training and evaluation details: Transfer results use specified base learning rates, with implementation and fine-tuning details reported alongside the zero-shot aggregation procedures.The models were trained with Classy Vision, while checkpoints from other works were obtained with timm.
  • Hashtag-to-class mapping: Each hashtag is mapped to WordNet synsets, and each ImageNet class is linked to the most similar hashtag or hashtags using maximum path similarity.Ambiguous hashtags can map to multiple classes, while a synset can correspond to multiple hashtags.
  • Prediction aggregation: RegNetY predictions are aggregated by summing mapped values with Platt scaling and averaging them without Platt scaling.Each hashtag prediction is first assigned to every ImageNet class linked to that hashtag.
  • Prediction aggregation: ViT predictions distribute each hashtag value evenly across its N mapped ImageNet classes and use the maximum resulting value for each class.This aggregation method performed better for the ViT models.

E. ImageNet Robustness Experiments

The robustness experiments evaluate fine-tuned models on ImageNet-like datasets and find weakly supervised models competitive across datasets, while differences in robustness remain inconclusive.

  • Experimental design: The experiments test ReaL ImageNet, ImageNet v2, and ObjectNet to assess robustness to visual-content, image-distribution, and labeling variations.Models are fine-tuned before evaluation on these ImageNet-like datasets.
  • Results: The largest weakly pretrained RegNetY and ViT models are runner-up on each evaluated ImageNet-like dataset, behind large ViTs trained on JFT 3B.Table 9 reports accuracy on ImageNet-1k, ReaL ImageNet, ImageNet v2, and ObjectNet.
  • Results: Validation accuracy on ImageNet-1k appears to predict accuracy on the other test sets across models, leaving robustness differences inconclusive.The passage does not establish a distinct robustness advantage for weakly supervised pre-training.
  • Additional analyses: Additional analyses examine performance on photos from non-English-speaking countries and associations involving people with varying characteristics.These analyses use hashtag-prediction models without fine-tuning.

F.1. Analyzing Hashtag Prediction Fairness

The zero-shot hashtag-prediction model shows country-level differences that depend on the dataset and evaluation measure. Accuracy is higher in the US and Europe than in many other countries, while the large proprietary dataset shows a relatively tight country range.

  • Dollar Street: 48.0% average accuracy was observed across countries on Dollar Street in zero-shot evaluation, with higher recognition accuracy in the US and Europe than in most other countries.The Dollar Street analysis used manually mapped hashtags and measured predictions per country.
  • Proprietary dataset: 63.3% accuracy was observed across all images in the proprietary dataset, whose country-level accuracy values varied by approximately 5%.The proprietary dataset contains millions of images labeled for visual concepts and country of origin.
  • Country comparisons: Up to 35% of classes were recognized substantially worse in a row country than a column country under the 80% accuracy-ratio rule.The heat map reports the percentage of classes with a country-accuracy ratio below 0.8.
  • Country comparisons: The heat map suggests better performance in the US and worse performance in Egypt and Nigeria, differing from the accuracy map’s lowest results for Brazil and Japan.The discrepancy may reflect differences in the distribution of per-class accuracies across countries.

F.2. Analyzing Associations in Hashtag Predictions

The analyses identify disparate hashtag associations across apparent skin tone, age, gender, and race, including stereotypes and potentially harmful associations. However, the evidence does not conclusively separate model-learned associations from biases in the evaluation datasets.

  • Scope of analyses: Hashtag predictions showed associations that varied by apparent skin tone, age, gender, and race, with some associations more difficult to explain than others.The analyses used separate datasets or subsets and examined the model’s highest-scoring hashtag predictions.
  • Apparent gender: The model associated men more frequently with #football and #basketball and women more frequently with #makeup and #bikini.The gender analysis was restricted to males and females because of evaluation-dataset limitations.
  • Potentially harmful associations: The analysis associated photos of Black people with #mugshot and #prison, prompting a deeper investigation of this potentially troubling pattern.Visual inspection found that a small percentage of UTK Faces images appeared to be mug shots.
  • Interpretation limits: The analyses cannot conclusively determine whether observed associations were learned by the model, reflected evaluation-data characteristics, or both without additional annotations.This uncertainty is explicitly noted for the UTK Faces and skin-tone analyses.
  • Potentially harmful associations: #mugshot was predicted 7 times for Fitzpatrick skin tone 1 images and once for skin tone 6 images, suggesting the UTK Faces pattern was more likely dataset-related than model-related.The authors emphasize that skin tone does not map closely to race and that the experiments are not fully conclusive.
Loading 2201.08371v2…