PaperlayerView exact revision on arXiv

Source-linked AI summary

Self-Supervised Pretraining Improves Self-Supervised Pretraining

Colorado J. Reed, Xiangyu Yue, Ani Nrusimha, Sayna Ebrahimi, Vivek Vijaykumar, Richard Mao, Bo Li, Shanghang Zhang, Devin Guillory, Sean Metzger, Kurt Keutzer, Trevor Darrell

arXiv:2103.12718v2cs.CV

TL;DR

Self-supervised pretraining is computationally demanding and sensitive to augmentation, and transfer learning applied to the pretraining process has not been formally studied for computer vision. HPT progressively pretrains on datasets closer to the target, producing faster convergence and stronger, more robust representations across tasks.

  • Problem

    Self-supervised pretraining requires long runs on large datasets and is sensitive to augmentation, while transfer learning applied to self-supervised pretraining had not been formally studied for computer vision.

    Source: §1 · pp. 1–2

  • Method

    HPT initializes self-supervised pretraining with an existing model and progresses through datasets increasingly similar to the target, with an optional domain-specific source stage.

    Source: §1 · p. 2

  • Results

    HPT converges up to 80× faster and achieves better representations on 15 of 16 datasets across classification, detection, and segmentation, with greater resilience to augmentation and data-amount changes.

    Source: §1 · p. 2

  • Takeaways & Limitations

    HPT provides a simple framework that reduces resource requirements and achieves strong results without dataset-specific hyperparameter or augmentation-policy optimization.

    Source: §6 · p. 9

Abstract

from arXiv · show

While self-supervised pretraining has proven beneficial for many computer vision tasks, it requires expensive and lengthy computation, large amounts of data, and is sensitive to data augmentation. Prior work demonstrates that models pretrained on datasets dissimilar to their target data, such as chest X-ray models trained on ImageNet, underperform models trained from scratch. Users that lack the resources to pretrain must use existing models with lower performance. This paper explores Hierarchical PreTraining (HPT), which decreases convergence time and improves accuracy by initializing the pretraining process with an existing pretrained model. Through experimentation on 16 diverse vision datasets, we show HPT converges up to 80x faster, improves accuracy across tasks, and improves the robustness of the self-supervised pretraining process to changes in the image augmentation policy or amount of pretraining data. Taken together, HPT provides a simple framework for obtaining better pretrained representations with less computational resources.

1. Introduction

Self-supervised pretraining is costly and sensitive to augmentation, while practitioners often rely on generalist models rather than target-similar pretraining. The paper introduces Hierarchical Pre-Training (HPT), which progressively pretrains on datasets closer to the target and reports faster convergence, improved representations, and greater robustness.

1. Introduction

  • Self-supervised pretraining requires long runs on large datasets and is sensitive to augmentation policies, often requiring many trials to find effective augmentations.SimCLR showed gains out to 3200 ImageNet epochs on 1.2 million images.Source: §1 · p. 1
  • HPT progressively pretrains on datasets increasingly similar to the target, moving from a general base dataset through an optional domain-specific source dataset to target pretraining.The resulting model is supervised-finetuned on the target task.Source: Figure 1 · p. 1, §1 · p. 2
  • The work addresses the lack of formal study of transfer learning applied to self-supervised pretraining, despite its resource-intensive, sensitive specialist-pretraining process.Practitioners often use publicly available general-dataset models because domain-specific pretrained models are not commonly available.Source: §1 · pp. 1–2
  • HPT improves representations over generalist or specialist pretraining on 15 of 16 datasets across classification, detection, and segmentation, while converging up to 80× faster.The reported benefits also include greater resilience to changes in augmentation policy and pretraining data amount.Source: §1 · p. 2
  • Across datasets, metrics, and methods, the paper concludes that self-supervised pretraining improves self-supervised pretraining.It also presents practitioner guidelines for applying self-supervised pretraining to new datasets and downstream applications.Source: §1 · p. 2

2. Background and Related Work

This section situates HPT within transfer learning and self-supervised learning, distinguishing its transfer of pretrained weights from prior dataset-selection strategies. It also relates the work to progressive pretraining in NLP and label-efficient learning.

2. Background and Related Work

  • The paper studies transfer learning in which source-trained model weights initialize training on a target task, with performance generally scaling with source size and source-target similarity.Transfer learning seeks to leverage larger, more general or specialized source datasets for downstream tasks.Source: §2 · p. 2
  • Unlike prior approaches that subset or reweight base data for target adaptation, HPT does not revisit base data, modify the pretrained architecture, or require expert model selection or reweighting.The contrasted approaches are identified as [45] and [39].Source: §2 · p. 2
  • The study focuses on instance contrastive learning, which matches augmented views of the same image against views from different images, and uses MoCo [7].The paper also explores additional self-supervised methods in the appendix.Source: §2 · p. 2
  • Progressive self-supervised pretraining has also been explored in NLP [20] [44]; this work instead focuses on computer vision and varies pretraining pipelines and methods.The paper states that its approach allows adaptation with fewer parameter updates.Source: §2 · pp. 2–3
  • HPT is complementary to weak- and semi-supervised label-efficient learning, and its representations can be used in conjunction with those approaches.Weak supervision assumes imperfect but related labels, while semi-supervised methods use labels for only some available examples.Source: §2 · p. 3

3. Hierarchical pretraining

HPT transfers a pretrained model through stages of increasingly target-similar data, with source pretraining optional. The section describes base-model initialization, source selection, target pretraining, and downstream supervised finetuning.

3. Hierarchical pretraining

  • Base pretraining initializes HPT with a model trained from scratch on a large, general vision dataset; practitioners can often start from a publicly available model instead.The paper gives ImageNet models as an example and says practitioners commonly select high-performing models from large-domain tasks.Source: §3 · p. 3
  • A task-aware search selects a source dataset larger and more target-similar than the base dataset by briefly pretraining candidate sources, linearly evaluating frozen models on supervised target data, and choosing the highest-scoring source.The short source-pretraining search is approximately 5K steps; the selected source model is then used for additional target pretraining.Source: §3 · p. 3
  • A single, similar source dataset, such as one with more than 30K images, consistently improves target-task representations, and source pretraining is optional.The paper reports gains when target pretraining follows base pretraining directly, without a source stage.Source: §3 · p. 3
  • Target pretraining continues self-supervised learning on target data from the source model, or from the base model when no source is used.Freezing all parameters except batch-normalization modulation parameters performs strongly, particularly for targets with fewer than 10K images.Source: §3 · p. 3
  • After target self-supervised pretraining, the model is transferred to the downstream target task, such as classification or object detection, for supervised finetuning.Source: §3 · p. 3

4. Experiments

Experiments evaluate HPT across 16 diverse datasets using linear separability, transfer, and semi-supervised tests. They also examine sequential pretraining, robustness to augmentation and data availability, and domain adaptation.

4.1. Datasets

  • The 16-dataset suite spans aerial, driving, medical, natural multi-object, and assorted image domains.Source: §4.1 · pp. 3–4

4.3. Pretraining Quality Analysis

  • By 5k steps, HPT and HPT-BN matched or exceeded Base transfer or 400k-step Target pretraining on 15 of 16 datasets; QuickDraw favored Target.HPT typically converged by 5k steps regardless of target dataset size.Source: Figure 2 · p. 4, §4.3 · p. 6
  • HPT produced the best semi-supervised pretrained models on 15 of 16 datasets, while HPT-BN never surpassed HPT after full-model finetuning.The evaluation finetuned models using 1,000 randomly selected labels per dataset.Source: §4.3 · p. 6
  • On three target datasets, the full Base-Source-Target pipeline led the results; HPT variants beat all baselines in every full-finetuning case.For Pascal, the source dataset was COCO-2014; the study reports a reduced impact of an intermediate source model as target dataset size increases.Source: Figure 4 · p. 6, §4.3 · p. 7
  • Across image classification, object detection, and segmentation, Base-Target and Base-Target (BN) improved over direct Base transfer.For object detection and segmentation, the tables report median AP, AP50, and AP75 across three runs.Source: §4.3 · p. 7, Table 1 · p. 7, Table 2 · p. 7

4.4. HPT Robustness

  • HPT maintained strong performance as augmentation policies changed, although CheXpert performance decreased under policy changes.HPT performed better with cropping alone than other policies did with incomplete augmentation combinations.Source: Figure 5 · p. 7, §4.4 · p. 8
  • As pretraining data decreased, HPT’s accuracy gains increased; HPT-BN consistently outperformed other methods with fewer than 5k samples.HPT-BN did not reach HPT’s performance in every case.Source: §4.4 · p. 8, Figure 6 · p. 8

4.5. Domain Adaptation Case Study

  • In real-to-clipart and real-to-sketch adaptation, HPT outperformed the baseline at every target-label budget, with nearly 8% better accuracy at one shot per class.The gap shrank to 2% at 68 shots per class, equivalent to 23,603 samples.Source: §4.5 · p. 8, Table 3 · p. 9

5. Discussion

The discussion frames HPT as a first thorough empirical analysis of transfer learning applied to self-supervised pretraining in computer vision. It reports faster convergence, improved performance, and increased robustness, and outlines practical use and supervised-model results.

5. Discussion

  • HPT achieves faster convergence, improved performance, and increased robustness across data domains.The appendix reports that similar analyses using supervised ImageNet base models also improve performance across datasets and tasks.Source: §5 · p. 9
  • The work provides the first thorough empirical analysis of transfer learning applied to self-supervised pretraining in computer vision.The authors recommend including HPT as a baseline, which they say is surprisingly absent from current works.Source: §5 · p. 9
  • For existing codebases, HPT is usually as simple as downloading an existing model and updating a configuration.The authors provide code, documentation, and models to use HPT and reproduce their results.Source: §5 · p. 9

6. Conclusion and Implications

HPT improves self-supervised pretraining while reducing resource requirements. The conclusion reports faster convergence, improved accuracy and robustness across several domains, and a straightforward implementation that did not require dataset-specific hyperparameter or augmentation-policy optimization.

6. Conclusion and Implications

  • 80x faster convergence, improved accuracy, and increased robustness were observed across aerial, medical, autonomous-driving, and simulation domains.The paper describes this as its first empirical analysis of transfer learning applied to self-supervised pretraining for computer vision.Source: §6 · p. 9
  • HPT requires fewer data and computational resources than prior methods, which the authors say enables wider adoption for real-world applications.They characterize HPT as a simple framework for improving self-supervised pretraining while decreasing resource requirements.Source: §6 · p. 9
  • The reported strong results did not require optimizing hyperparameters or augmentation policies for each dataset.Code and pretrained models are available to use HPT and reproduce the results.Source: §6 · p. 9

A. Implementation details

The implementation uses specified training, evaluation, and tuning procedures across linear, fine-tuning, and detection tasks. Evaluation settings are selected using validation performance, with final performance reported on test data.

A. Implementation details

  • Validation performance selected the evaluation hyperparameter, schedule, and checkpoint, and the best settings were then evaluated on the test split.Pretraining used train and validation splits; evaluation-task training used the train split, with the validation split for selection.Source: A. Implementation details · p. 13
  • Linear analysis and fine-tuning used 224-pixel RandomResizedCrop and RandomHorizontalFlip during training, with resize-and-center-crop evaluation.Images were normalized by dataset channel-wise mean and variance; classification used top-1 accuracy and multilabel classification used AUROC.Source: A. Implementation details · p. 13
  • For semi-supervised fine-tuning, 1,000 randomly selected training examples covered every class, though class counts were unbalanced.The original validation and test splits were retained for evaluation consistency.Source: A. Implementation details · p. 13
  • Object-detection experiments used an R50-C4 backbone, with dataset-specific splits, training schedules, and batch sizes.PASCAL used train 2007+2012 and validation 2012; BDD used 70K training and 10K validation images; COCO used the 2017 splits and a 1× schedule. Semantic segmentation used Mask-RCNN with a C4 backbone.Source: A. Implementation details · p. 13
  • Table 4 summarizes parameters for pretraining, linear evaluation, and fine-tuning, while the learning-rate schedule decayed by a factor of 10 at one-third and two-thirds of training.Curly-braced table values indicate that all listed combinations were tested.Source: Table 4 · p. 13, A. Implementation details · p. 13

B. Datasets

The appendix lists datasets and evaluation metrics, then reports a learning-rate comparison used to set the HPT pretraining default.

B. Datasets

  • Dataset descriptions use x/y/z to indicate train/validation/test splits, with top-1 accuracy for single-label classification and AUROC for multilabel tasks.The reported metrics apply unless otherwise noted.Source: B. Datasets · p. 13, Table 5 · p. 14

C.1. HPT Learning Rate

  • The tested HPT learning rates were 0.1, 0.03, and 0.001; 0.03 outperformed the other configurations and was used for pretraining.The comparison covered RESISC, BDD, and CheXpert; the table caption identifies linear-evaluation performance as the comparison.Source: §C.1 · p. 14, Table 6 · p. 14

C.2. HPT with Supervised Base Model

With a supervised ImageNet base model, HPT generally improves results over direct transfer or target-only pretraining, and improves finetuning across the three evaluated datasets. Its augmentation robustness is less consistent when the base model was trained with fewer augmentations.

C.2. HPT with Supervised Base Model

  • HPT with a supervised ImageNet base model generally improved results over direct transfer or target-only pretraining, sometimes after 5K iterations.The passage lists improvements at 50K iterations on RESISC, BDD, DomainNet Sketch, DomainNet Quickdraw, xView, Chexpert, and CoCo-2014.Source: §C.2 · p. 14
  • HPT improved finetuning performance over target-pretrained models on all three evaluated datasets, with or without source training.For DomainNet, results were considerably worse than with the self-supervised base model, and adding DomainNet Clipart source training did not improve results.Source: §C.2 · pp. 14–15
  • 13/16 datasets had HPT results as good as or better than target-only pretraining without hyperparameter tuning.The comparison uses a supervised ImageNet base model and linear evaluation.Source: Figure 7 · p. 15
  • On RESISC, both supervised-base HPT and self-supervised-base HPT were robust to the tested augmentation changes, but supervised-base HPT varied more on BDD and Chexpert.The supervised base model used only cropping and flipping, unlike the self-supervised model, which used all augmentations in the shown policy.Source: §C.2 · p. 15

C.3. Basetrain Robustness

HPT’s performance is robust to changes in base-training duration and target-data amount, with benefits that vary by dataset and setting. Experiments also report similar HPT behavior with ResNet-18, though evaluation performance is lower than with ResNet-50.

C.3. Basetrain Robustness

  • Supervised-base HPT varied more across augmentation policies on BDD and Chexpert; the base model had been trained only with cropping and flipping.The reported robustness of further pretraining depends on augmentations present during base pretraining.Source: Figure 9 · p. 16
  • At 20 ImageNet epochs plus 5,000 target-pretraining steps, HPT exceeded the best target-only result on every dataset except Chest-X-ray-kids, at roughly equivalent computation to 100,000 target-only iterations.At 20 epochs, RESISC still exceeded the best target-only result despite performing below its 200- and 800-epoch HPT results.Source: §C.3 · p. 16, Figure 10 · p. 17
  • The benefit of source pretraining increased as target-data amount decreased across CheXpert, DomainNet Sketch, and UC-Merced.The tested target-data amounts were 1%, 10%, 25%, and 100%.Source: §C.4 · pp. 17–18, Figure 11 · p. 17

C.5. ResNet-18 Experiments

  • With ResNet-18, HPT showed behavior similar to ResNet-50 on BDD, Chexpert, and RESISC, although evaluation performance was lower.The passage reports improved performance at 50K iterations for all three datasets and expects broad applicability across architectures.Source: §C.5 · p. 18, Figure 12 · p. 18

C.6. BYOL Experiments

Experiments using BYOL report behavior similar to the paper’s MoCo results on RESISC, BDD, and Chexpert. The accompanying representation analysis compares models through prediction-error overlap and activation similarity at the final convolutional and linear layers.

C.6. BYOL Experiments

  • BYOL pretraining produced results similar to the reported MoCo results on RESISC, BDD, and Chexpert.The authors expect HPT to apply across self-supervised pretraining methods.Source: §C.6 · p. 18, Figure 13 · p. 18

C.7. Representational Similarity

  • The representation analysis examines HPT models initialized from different base models and compares their learned representations.It evaluates the final convolutional-layer activations and the linear layer finetuned on top of them.Source: §C.7 · p. 19, §C.7.1 · p. 19

C.7.1 Defining metrics

  • Intersection over Union (IoU) measures the overlap between two models’ misclassified-image sets relative to images misclassified by at least one model.The accompanying algorithm counts common errors and total errors across the labels and predictions.Source: §C.7.1 · p. 19
  • The activation metric RV2 compares outputs from two model layers computed on the same data.The analysis evaluates activations from the final convolutional layer and the finetuned linear layer.Source: §C.7.1 · p. 19

C.7.2 Effect of model initialization?

Models with different basetrains learn distinct representations, while runs sharing a basetrain are generally more similar before overfitting. The reported comparisons cover layer similarity and overlap in model errors.

C.7.2 Effect of model initialization?

  • Across different random seeds, models with the same basetrain and pretraining-step count had more similar errors and representations than other combinations.The reported IoU was typically 0.5–0.6; similarity was generally higher and varied by dataset.Source: §C.7.2 · p. 20, Figure 14 · p. 20
  • Before about 50,000 training iterations, models with the same basetrain were generally more similar in linear layers and errors; this pattern weakened after overfitting began.The linear-layer comparison is shown in Figure 15, and the IoU comparison follows a similar trend in Figure 17.Source: §C.7.2 · p. 20, Figure 15 · p. 21, Figure 17 · p. 22
  • Final convolutional layers trained from the same basetrain were generally more similar, with some deviations in a little under half of the tested datasets.The comparison is shown in Figure 16.Source: §C.7.2 · p. 20, Figure 16 · p. 21
  • Different-basetrain models had mean linear-layer similarity 0.78 and mean IoU 0.40, versus 0.98 and 0.61 for identical-basetrain models, respectively.The differences were significant: p = 2 × 10^-4 for similarity and p = 2 × 10^-6 for IoU.Source: §C.7.2 · p. 21
Loading 2103.12718v2…