PaperlayerView exact revision on arXiv

Source-linked AI summary

ImageNet-21K Pretraining for the Masses

Tal Ridnik, Emanuel Ben-Baruch, Asaf Noy, Lihi Zelnik-Manor

arXiv:2104.10972v4cs.CVcs.LG

TL;DR

ImageNet-21K is underused because its inconsistent hierarchical labels complicate pretraining, despite its larger and publicly available dataset. The paper standardizes preprocessing, exploits WordNet structure with semantic softmax and distillation, and reports consistent downstream benefits across architectures relative to ImageNet-1K and earlier ImageNet-21K schemes.

  • Problem

    ImageNet-21K pretraining lacks a methodologically optimized process because its non-mutually-exclusive, inconsistently assigned labels complicate training and evaluation.

  • Method

    The paper preprocesses ImageNet-21K, uses WordNet to represent semantic labels, and trains with hierarchical semantic softmax supplemented by semantic knowledge distillation.

  • Results

    Across numerous downstream datasets and tasks, semantic-softmax ImageNet-21K pretraining benefits diverse architectures and outperforms standard ImageNet-1K and previous ImageNet-21K schemes.

  • Takeaways & Limitations

    The pipeline makes high-quality ImageNet-21K pretraining more accessible, including for small mobile-oriented models and prominent models such as ViT and Mixer.

  • Takeaways & Limitations

    Multi-label training can suffer from harder optimization and severe positive-negative and hierarchy-related class imbalance.

Abstract

from arXiv · show

ImageNet-1K serves as the primary dataset for pretraining deep learning models for computer vision tasks. ImageNet-21K dataset, which is bigger and more diverse, is used less frequently for pretraining, mainly due to its complexity, low accessibility, and underestimation of its added value. This paper aims to close this gap, and make high-quality efficient pretraining on ImageNet-21K available for everyone. Via a dedicated preprocessing stage, utilization of WordNet hierarchical structure, and a novel training scheme called semantic softmax, we show that various models significantly benefit from ImageNet-21K pretraining on numerous datasets and tasks, including small mobile-oriented models. We also show that we outperform previous ImageNet-21K pretraining schemes for prominent new models like ViT and Mixer. Our proposed pretraining pipeline is efficient, accessible, and leads to SoTA reproducible results, from a publicly available dataset. The training code and pretrained models are available at: https://github.com/Alibaba-MIIL/ImageNet21K

1 Introduction

ImageNet-21K is larger and publicly available but remains underused because its hierarchical, inconsistently assigned labels complicate training and evaluation. The paper develops an accessible preprocessing and semantic-softmax pipeline, showing benefits across architectures and downstream tasks.

  • ImageNet-21K contains 14,197,122 images across 21,841 classes, while ImageNet-1K is a 1.2M-image subset covering 1,000 mutually exclusive classes.
  • WordNet-derived labels are not mutually exclusive or consistently assigned, so images may receive either a class label or its semantic parent.For example, an image of a chair may be labeled “chair” or “furniture.”
  • The paper addresses the lack of systematic ImageNet-21K pretraining studies with an efficient, high-quality pipeline intended to be accessible to deep learning practitioners.
  • The pipeline preprocesses ImageNet-21K, converts labels through WordNet semantics, and introduces semantic softmax across hierarchical layers with balanced losses.
  • Semantic softmax consistently outperforms single-label and multi-label pretraining, with further gains from semantic knowledge distillation.
  • ImageNet-21K pretraining improves numerous downstream tasks for large, medium, and small mobile-oriented models, and outperforms earlier schemes for ViT and Mixer.

2 Dataset Preparation

The dataset-preparation process filters invalid classes, creates a standardized validation split, and resizes images before analyzing ImageNet-21K-P’s WordNet hierarchy. This hierarchy exposes inconsistent tagging and motivates methods designed to use semantic information during training.

  • Dataset Preparation: Preprocessing removes invalid classes, creates a standardized train-validation split, and resizes images to reduce the dataset’s memory footprint.
  • Dataset Preparation: The processed dataset, named ImageNet-21K-P, retains only valid classes and has a standardized train-validation split.
  • Utilizing Semantic Data: WordNet hypernym and hyponym relations transform each single label into a semantic multi-label representation containing the original label and its ancestors.
  • Utilizing Semantic Data: Images containing the same object can receive different labels, such as “animal” and “cow,” because tagging is not guaranteed at the highest hierarchy.
  • Utilizing Semantic Data: Incomplete tagging complicates training and motivates a dedicated scheme to address the resulting semantic inconsistencies.
  • Utilizing Semantic Data: ImageNet-21K-P contains 11 possible hierarchies, although most classes belong to lower hierarchies.

3 Pretraining Schemes

The paper compares single-label and multi-label pretraining on ImageNet-21K-P, then introduces semantic softmax to combine semantic coverage with more efficient optimization. Semantic softmax uses hierarchy-specific softmax layers and selective gradient propagation, with balanced loss aggregation and optional semantic knowledge distillation.

  • Single-label Training: Single-label training uses the original labels with one softmax and cross-entropy, offering balanced classes and efficient single-loss optimization.Its limitations are inconsistent tagging and the absence of semantic data during training.
  • Multi-label Training: Multi-label training converts each label into semantic labels, applies independent sigmoids, and aggregates binary losses across 11,221 classes.This exposes more information per image but creates an extreme multi-task optimization problem.
  • Multi-label Training: Multi-label training can suffer from extreme multi-tasking and positive-negative imbalance, producing optimization drawbacks despite richer labels.ASL loss significantly outperforms cross-entropy for this setting on upstream and downstream tasks.
  • Semantic Softmax Training Scheme: Semantic softmax replaces independent sigmoids with 11 hierarchy-specific softmax layers, activating gradients only for hierarchies relevant to each image.This presents available semantic labels while avoiding the 11,221 uncoupled losses of multi-label training.
  • Semantic Softmax Training Scheme: Semantic softmax aggregates hierarchy losses with balancing logic so frequently activated lower hierarchies do not dominate training.The proposed formulation uses hierarchy class counts to normalize contributions before forming the total loss.
  • Semantic Knowledge Distillation: Semantic knowledge distillation supplements semantic softmax by matching teacher and student distributions across hierarchies.The teacher can predict missing tags caused by inconsistent ImageNet-21K-P annotations; the implementation may use MSE or Kullback-Leibler divergence, with MSE reported to converge faster.

4 Experimental Study

The study evaluates single-label, multi-label, and semantic softmax pretraining through transfer learning across diverse datasets and tasks. Semantic softmax performs best broadly, while semantic knowledge distillation further improves downstream results.

  • Experimental setup: Transfer learning across varied datasets, domains, sizes, and computer-vision tasks is used to compare pretraining quality without overfitting one evaluation setting.The evaluation includes datasets ranging from thousands to more than a million images.
  • Comparing pretraining schemes: On 6 out of 7 datasets tested, semantic softmax pretraining outperforms both single-label and multi-label pretraining.
  • Comparing pretraining schemes: Single-label pretraining scores higher than multi-label pretraining on 5 out of 7 datasets tested.
  • Comparing pretraining schemes: Multi-label training provides more information per image but suffers from less efficient optimization caused by extreme multi-tasking and imbalance.
  • Semantic KD: Adding semantic knowledge distillation improves downstream results on all tested tasks and datasets.The paper attributes this to filling missing tags and producing smoother, more informative ground truth.

5 Results

The paper compares semantic softmax with established ImageNet-21K pretraining and ImageNet-1K pretraining across models and downstream tasks. It reports stronger transfer performance, including for small mobile-oriented models and publicly reproducible ImageNet-1K results.

  • Comparison to other ImageNet-21K schemes: Semantic softmax significantly outperforms official ImageNet-21K pretrained weights on all downstream tasks tested for ViT and Mixer.Transfer learning with semantic softmax is reported as more stable and robust for some MLP-based models.
  • Comparison to ImageNet-1K pretraining: Semantic softmax significantly outperforms standard ImageNet-1K pretraining on all datasets and models tested.On iNaturalist, average top-1 accuracy improves by 2.9%.
  • Model scale: Small mobile-oriented models, including MobileNetV3 and OFA-595, consistently and significantly benefit from large-scale ImageNet-21K pretraining.This challenges the cited assumption that large datasets primarily benefit large models.
  • ImageNet-1K SoTA results: Using ImageNet-21K-P semantic softmax pretraining, MobileNetV3 reaches 78.0% ImageNet-1K accuracy versus its reported 75.2%, while ResNet50 reaches 82.0% versus 76.0%.
  • Additional comparisons: Additional comparisons cover Open Images, non-classification vision tasks, and the impact of training-sample counts on upstream results.

6 Conclusion

The paper presents an end-to-end ImageNet-21K pretraining scheme combining standardized preprocessing, WordNet semantics, semantic softmax, and semantic knowledge distillation. Across varied architectures, datasets, and tasks, it reports consistent benefits over ImageNet-1K and previous ImageNet-21K schemes.

  • Conclusion: The end-to-end scheme standardizes preprocessing, uses WordNet to create semantic labels, and develops semantic softmax to exploit ImageNet-21K hierarchy.The paper reports that single-label training outperforms multi-label training despite less information per image.
  • Conclusion: Different architectures significantly and consistently benefit from the proposed pretraining scheme compared with ImageNet-1K and previous ImageNet-21K schemes.
  • Broader Impact: The pipeline is intended to make high-quality ImageNet-21K pretraining more accessible without massive computing resources or large-scale private datasets.The paper also identifies benefits for real-world products using small mobile-oriented models.

Appendices

The appendices provide hierarchy statistics, implementation details, benchmark assumptions, and metric guidance for the proposed pretraining pipeline.

  • A Number of Classes in Different Hierarchies: Figure 4 reports the number of classes assigned to different WordNet hierarchies.
  • Implementation details: Training uses label smoothing with factor 0.2, squish-resizing, input resolution 224, Adam, learning rate 3e-4, and a one-cycle policy.
  • Implementation details: Multi-label training differs from single-label training mainly through its loss function, while other training details and training times are similar.
  • Benchmarking: A standardized dataset with a fixed train-validation split enables future benchmark comparisons using training metrics.

C.1 Singe-label Upstream Results

Single-label ImageNet-21K-P training achieves substantially lower top-1 accuracy than ImageNet-1K, reflecting inconsistent semantic tagging. Multi-label loss comparisons identify ASL as superior, while semantic softmax provides accurate hierarchy-aware metrics.

  • Single-label upstream results: Top-1 accuracy remains difficult to interpret when semantically similar images have different ground-truth labels.Top-5 accuracy is described as more representative, though still limited.
  • Single-label upstream results: ImageNet-21K-P single-label top-1 accuracy is 37%−46%, versus 75%−85% on ImageNet-1K.The gap is mainly attributed to ImageNet-21K-P’s semantic structure and inconsistent tagging methodology.
  • Multi-label loss comparison: ASL loss outperforms cross-entropy and focal loss for multi-label classification on ImageNet-21K-P.This aligns with the identified positive-negative and class-imbalance optimization challenges.
  • Semantic softmax metrics: Semantic softmax enables top-1 accuracy calculation for each hierarchy and produces fully accurate training metrics.The method weights hierarchy-specific accuracies by the number of classes and excludes classes above the maximal hierarchy.
  • Semantic softmax metrics: Figure 5 reports hierarchy-level top-1 accuracies for models trained with semantic softmax and knowledge distillation.The figure compares accuracies across different hierarchy levels.

D Downstream Datasets Training Details

The downstream evaluation spans classification, multi-label recognition, and video action recognition under standardized fine-tuning settings. Semantic softmax pretraining is compared across dataset variants, with Winter21 causing a minor downstream performance reduction relative to Fall11.

  • Downstream datasets: Downstream evaluation covers ImageNet-1K, iNaturalist 2019, CIFAR-100, Food-251, MS-COCO, Pascal-VOC, and Kinetics-200.The datasets cover single-label classification, multi-label classification, and video action recognition.
  • Training details: Results for datasets with fewer than 150,000 images average three runs with different seeds.All results use 224-pixel inputs, cutout, RandAugment, true weight decay, and task-specific training settings.
  • Pretraining loss comparison: ASL-based multi-label pretraining produces significantly better downstream results than vanilla cross-entropy on all evaluated downstream datasets.The comparison is reported across single-label, multi-label, and action-recognition metrics.
  • Teacher confidence: Teacher confidence is averaged over an epoch and is generally higher for lower semantic hierarchies.The confidence calculation uses hierarchy-specific teacher predictions and semantic ground truth; only the top 5% probabilities are summed when needed.
  • Dataset variants: The processed Fall11 ImageNet-21K-P variant contains 11,221 classes, 11,797,632 training samples, and 561,052 test samples.The Winter21 variant contains 10,450 classes, 11,060,223 training samples, and 522,500 test samples.
  • Dataset variants: Winter21 contains 10% fewer classes and 6% fewer images than Fall11 and leads to a minor reduction in downstream performance.The comparison uses semantic softmax pretraining and the MTResNet model.

J.1 Comparison to Pretraining on Open Images Dataset

ImageNet-21K semantic-softmax pretraining consistently outperforms Open Images multi-label pretraining and improves non-classification transfer tasks. Additional experiments show that more training images continue to improve semantic accuracy.

  • Open Images comparison: ImageNet-21K pretraining consistently provides better downstream results than Open Images pretraining.The paper suggests Open Images may share multi-label optimization pitfalls caused by large class counts and imbalance.
  • Additional downstream tasks: ImageNet-21K pretraining yields better results than ImageNet-1K pretraining on object detection and image retrieval.The evaluated tasks use MS-COCO object detection and INRIA Holidays image retrieval.
  • Training-set scaling: More training images lead to better semantic accuracies without observed saturation in upstream results.The effect is evaluated by varying the number of training samples.
  • Implementation components: The paper provides PyTorch-style components for semantic-logit splitting, knowledge-distillation loss calculation, and teacher-confidence estimation.The implementation includes hierarchy scanning, semantic-logit conversion, weighted losses, and top-5%-based confidence estimation.

L Limitations

The proposed pipeline remains computationally demanding and omits some potentially beneficial training techniques. Its reported pretraining quality may also be below that achievable with larger private datasets.

  • Computational requirements: Reasonable training times still require an 8-GPU machine, while single-GPU training may take 2−3 weeks.This is the paper’s explicit hardware and runtime boundary.
  • Un tested techniques: Training tricks that could improve pretraining quality were omitted because they would significantly increase training time.The authors prioritize an efficient pipeline and staying within their computing budget.
  • Un tested techniques: GradNorm and PCGrad were not incorporated, although the authors suggest they could improve multi-label pretraining quality.These methods target extreme multi-tasking but would also substantially increase training time.
  • Un tested techniques: General semantic training techniques were not tested because the authors found most infeasible for large-scale efficient training.The paper argues that semantic softmax is tailored to ImageNet-21K’s specific characteristics.
  • Scope boundary: Pretraining on larger private datasets such as JFT-300M may still achieve higher pretraining quality.This marks a scope boundary relative to publicly available ImageNet-21K pretraining.
Loading 2104.10972v4…