PaperlayerView exact revision on arXiv

Source-linked AI summary

Pushing the limits of self-supervised ResNets: Can we outperform supervised learning without labels on ImageNet?

Nenad Tomasev, Ioana Bica, Brian McWilliams, Lars Buesing, Razvan Pascanu, Charles Blundell, Jovana Mitrovic

arXiv:2201.05119v2cs.CVcs.LGstat.ML

TL;DR

Self-supervised representations had not surpassed supervised learning on ImageNet, limiting their applicability in performance-critical settings. RELICv2 adds inductive biases through saliency masking and varied-size views; it is reported to outperform supervised baselines across ResNet architectures and to generalize better out of distribution than previous methods.

  • Problem

    Self-supervised representations had not exceeded supervised training of the same architecture on ImageNet, limiting their wide adoption; the paper hypothesizes that spurious features may contribute to this gap.

    Source: §1 · pp. 1–2

  • Method

    RELICv2 extends RELIC with saliency masking and contrastive and invariance losses over multiple data views of varying sizes.

    Source: §1 · p. 2

  • Results

    RELICv2 is reported to outperform a standard supervised model across a range of ResNet architectures and to generalize better out of distribution than previous methods on image classification and semantic segmentation.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The authors describe outperforming the supervised baseline across ResNet architectures as a first step toward making self-supervised learning a go-to approach for performant representations.

    Source: §6 · p. 11

    Saliency masking is imperfect at removing background, especially when foreground and background have little color contrast.

    Source: §3 · p. 5

Abstract

from arXiv · show

Despite recent progress made by self-supervised methods in representation learning with residual networks, they still underperform supervised learning on the ImageNet classification benchmark, limiting their applicability in performance-critical settings. Building on prior theoretical insights from ReLIC [Mitrovic et al., 2021], we include additional inductive biases into self-supervised learning. We propose a new self-supervised representation learning method, ReLICv2, which combines an explicit invariance loss with a contrastive objective over a varied set of appropriately constructed data views to avoid learning spurious correlations and obtain more informative representations. ReLICv2 achieves $77.1\%$ top-$1$ accuracy on ImageNet under linear evaluation on a ResNet50, thus improving the previous state-of-the-art by absolute $+1.5\%$; on larger ResNet models, ReLICv2 achieves up to $80.6\%$ outperforming previous self-supervised approaches with margins up to $+2.3\%$. Most notably, ReLICv2 is the first unsupervised representation learning method to consistently outperform the supervised baseline in a like-for-like comparison over a range of ResNet architectures. Using ReLICv2, we also learn more robust and transferable representations that generalize better out-of-distribution than previous work, both on image classification and semantic segmentation. Finally, we show that despite using ResNet encoders, ReLICv2 is comparable to state-of-the-art self-supervised vision transformers.

1 INTRODUCTION

Self-supervised representations had not surpassed supervised training of the same architecture on ImageNet linear evaluation, and the paper identifies spurious features as a possible reason. RELICv2 adds saliency masking and varied-size views, and is reported to outperform supervised baselines across ResNet architectures.

1 INTRODUCTION

  • RELICv2 is the first self-supervised method reported to outperform a standard supervised baseline across a range of ResNet architectures in like-for-like ImageNet linear evaluation.Figure 1 compares top-1 linear evaluation accuracy across ResNet50 width multipliers and ResNet200 2×.Source: Figure 1 · p. 2, §1 · p. 3
  • Spurious features—features not directly related to downstream tasks—are hypothesized to contribute to self-supervised representations’ subpar performance and may harm generalization.The paper argues that avoiding these features is important for learning informative representations.Source: §1 · p. 2
  • RELICv2 extends RELIC [Mitrovic et al., 2021] with unsupervised saliency masking and multiple views of varying sizes, using invariance and contrastive losses over those views.The proposed inductive biases target spurious background and style features.Source: §1 · p. 2
  • The authors present results across image classification, semi-supervised and transfer learning, robustness, and out-of-distribution generalization as evidence of RELICv2’s generality.They also report performance comparable to recent Vision Transformers despite using ResNet encoders.Source: §1 · pp. 2–3
  • 80.6% top-1 accuracy is reported for RELICv2 with ResNet200 2× on ImageNet.The paper also reports more transferable and robust representations that generalize better out of distribution than previous methods on image classification and semantic segmentation.Source: §1 · p. 3

2 METHOD

The method extends RELIC’s contrastive and invariance objectives with saliency masking and multiple view sizes. Saliency emphasizes foreground content, while varied views train invariance across styles and occlusions.

2.1 BACKGROUND

  • RELIC [Mitrovic et al., 2021] contrasts two differently augmented views and aligns their positive- and negative-similarity distributions across augmentations.Its encoder maps inputs to representations, and the objective weights the invariance loss relative to the contrastive loss.Source: §2.1 · pp. 3–4

2.2 RELICV2

  • RELICv2 extends RELIC with inductive biases intended to prevent learning spurious features that hinder informative representations.The method encodes these biases through unsupervised saliency masking and multiple views of varying sizes.Source: §2.2 · p. 4
  • The saliency pipeline uses no labels or data beyond ImageNet: it refines handcrafted masks with a self-supervised network, then applies masking with small probability during pretraining.The authors use the resulting masked images to focus representations on foreground features and improve robustness to background changes.Source: §2.2 · p. 4
  • Large views expose varied object styles, while small crops can train robustness to occlusion; too many small views risk omitting important features.The authors also hypothesize that small crops complement saliency masking by removing background regions.Source: §2.2 · pp. 4–5
  • RELICv2 combines contrastive and invariance losses across views, comparing online-network large and small views with target-network large views.Large views update both networks, while small views update only the online network.Source: Figure 2 · p. 5, §2.2 · p. 5

3 EVALUATING PRETRAINED REPRESENTATIONS ON IMAGENET

RELICv2 improves ImageNet linear-evaluation accuracy over prior self-supervised methods across ResNet architectures, and the paper reports outperforming supervised baselines in a like-for-like comparison. Its analysis examines scaling, class separation, view sizes, saliency masking, and invariance.

3 EVALUATING PRETRAINED REPRESENTATIONS ON IMAGENET

  • +1.5%: RELICv2 outperforms previous self-supervised approaches in ImageNet test top-1 accuracy under linear evaluation with a ResNet50.Table 1 reports top-1 accuracy for different representation-learning methods on the ImageNet test set.Source: Table 1 · p. 6, §3 · p. 6
  • Up to +2.3%: RELICv2 achieves state-of-the-art top-1 accuracy across varied ResNet encoders, and the paper reports outperforming supervised baselines across a wide range of architectures.The compared architectures include ResNet50, ResNet101, ResNet152, and ResNet200, with widths of 1×, 2×, and 4×.Source: Table 2 · p. 6, §3 · p. 6
  • At 70% accuracy, ResNet50 requires approximately twice as many pretraining iterations as ResNet295, which has about 3.6× as many parameters.The models have 24M and 87M parameters, respectively.Source: §3 · p. 7, Figure 3 · p. 7

3.1 ANALYSIS

  • RELICv2’s linear discriminant ratio distribution is shifted right of the supervised baseline, indicating a higher mean and more linearly separable representations.The ratio compares between-class and within-class embedding distances on the ImageNet test set.Source: §3.1 · p. 7, Figure 4 · p. 7
  • With four large views, adding two small views raises top-1 accuracy by +1.3%, while adding two large views raises it by +0.7%.The analysis reports a penalty beyond two small views and a plateau beyond six large views.Source: §3.1 · p. 7
  • Saliency masking raises top-1 accuracy from 74.8% to 75.3% with two large views, while invariance-based methods benefit more from multiple views and saliency masking than purely contrastive methods.The authors conclude that invariance plays a crucial role in learning better representations.Source: §3.1 · p. 8

4 ADDITIONAL EXPERIMENTAL RESULTS

The paper evaluates RELICv2 on robustness, out-of-distribution generalization, classification transfer, semantic segmentation, and pretraining on JFT-300M. It reports improvements over competing methods across these settings, with results varying by task and comparison.

4.1 ROBUSTNESS AND OOD GENERALIZATION

  • RELICv2 outperforms competing self-supervised models on ImageNetV2 and ImageNet-C robustness, exceeds the supervised baseline on robustness, and is on par with supervised performance on OOD generalization.Evaluation uses ImageNetV2 and ImageNet-C for robustness, and ImageNet-R, ImageNet-Sketch, and ObjectNet for OOD generalization.Source: §4.1 · p. 8, Table 3 · p. 8

4.2 TRANSFER TO OTHER CLASSIFICATION DATASETS AND SEMANTIC SEGMENTATION

  • RELICv2 improves upon the supervised baseline and competing methods on classification transfer, performing best on 7 of 11 tasks with an average relative improvement over +5%.The average relative improvement is more than double that of the closest competing self-supervised method, NNCLR.Source: §4.2 · p. 9, Figure 5 · p. 9
  • On semantic segmentation, RELICv2 significantly outperforms BYOL on PASCAL and Cityscapes, and exceeds DetCon on PASCAL with 77.3 mIoU.The section reports fine-tuning the pretrained model for both tasks.Source: §4.2 · p. 9, Table 4 · p. 9

4.3 PRETRAINING ON JOINT FOTO TREE (JFT-300M)

  • On JFT-300M pretraining, RELICv2 achieves better overall performance than competing methods after 1000 epochs while requiring fewer training epochs.JFT-300M contains 300 million images from more than 18k classes.Source: §4.3 · pp. 9–10, Table 5 · p. 9

5 RELATED WORK

Prior self-supervised methods encourage invariance in different ways, but the paper contrasts those approaches with ReLIC’s explicit invariance loss alongside a contrastive objective. ReLICv2 extends this direction with background-aware, varied-size views and invariance between them.

5 RELATED WORK

  • Contrastive and bootstrapping methods implicitly enforce invariance by maximizing similarity between two views, while clustering methods add an explicit step [Oord et al., 2018; Bachman et al., 2019; Chen et al., 2020a; He et al., 2019; Dwibedi et al., 2021; Grill et al., 2020; Caron et al., 2020].The paper notes that neither approach is directly linked theoretically to learning more compact representations.Source: §5 · p. 10
  • ReLIC [Mitrovic et al., 2021] combines an explicit invariance loss with a contrastive objective, which its authors link to guaranteed generalization.ReLICv2 builds on this approach.Source: §5 · p. 10
  • Unlike background-augmentation methods [Zhao et al., 2021; Ryali et al., 2021], ReLICv2 enforces invariance between views with and without background and uses multiple view sizes.It uses a small number of small views for robustness, a large number of large views, and invariance between large and small views; the cited methods use only two views.Source: §5 · p. 10

6 DISCUSSION

ReLICv2 combines invariance over differently sized views with saliency masking to learn representations robust to spurious correlations. It reports gains over prior self-supervised methods across ResNet architectures and competitive performance against recent vision transformers.

6 DISCUSSION

  • Up to +2.3% in ImageNet linear evaluation: ReLICv2 outperforms previous self-supervised methods across ResNet architectures and reports state-of-the-art transfer, semi-supervised, robustness, and out-of-distribution results.The paper also reports strong JFT-300M and semantic-segmentation results.Source: §6 · p. 10
  • The method enforces invariance across varied-size views and uses saliency masking in data augmentation to learn representations robust to spurious correlations.The discussion attributes its approach to improving robustness to spurious correlations.Source: §6 · p. 10
  • ReLICv2 is the first method reported to consistently outperform the standard supervised baseline on ImageNet across ResNet architectures in a like-for-like comparison.The authors describe this as a first step toward self-supervised learning becoming a go-to approach for performant representations.Source: §6 · p. 11
  • ReLICv2 outperforms recent self-supervised vision transformers DINO [Caron et al., 2021] and MoCov3 [Chen et al., 2021], and matches EsViT [Li et al., 2021] at comparable parameter counts; Figure 6 compares ImageNet top-1 accuracy.The compared transformer methods use more powerful architectures and more involved training procedures.Source: Figure 6 · p. 10, §6 · p. 11

A COMPARISON BETWEEN SELF-SUPERVISED METHODS

This comparison emphasizes how methods choose positive and negative examples and enforce invariance. It also describes ReLICv2’s ImageNet pretraining view setup.

A COMPARISON BETWEEN SELF-SUPERVISED METHODS

B PRETRAINING ON IMAGENET – IMPLEMENTATION DETAILS AND ADDITIONAL RESULTS

  • ImageNet pretraining uses four 224 × 224 views and two centered 96 × 96 views, each drawn from a differently augmented image.Models are trained for 1000 epochs with 10 warm-up epochs and batch size 4096 unless otherwise indicated.Source: B PRETRAINING ON IMAGENET –… · p. 18

B.1 LINEAR EVALUATION

Under ImageNet linear evaluation, ReLICv2 outperforms competing self-supervised methods and the reported supervised baselines across varied ResNets. It also reports strong semi-supervised and transfer results, with performance depending on label fraction and training conditions.

B.1 LINEAR EVALUATION

  • Linear evaluation freezes the pretrained encoder and batch statistics while training a classifier on top; the reported models span ResNet depths and layer widths.The test set is ImageNet; Table 8 reports top-1 and top-5 accuracy.Source: §B.1 · p. 18, Table 8 · p. 19
  • Up to 1.2% absolute margin: ReLICv2 outperforms all competing self-supervised methods and the reported supervised baselines across varied ResNets in ImageNet linear evaluation.Table 8 covers top-1 and top-5 accuracy across architectures; some supervised baselines use stronger augmentations and are not direct like-for-like comparisons.Source: §B.1 · p. 18, Table 8 · p. 19

B.2 SEMI-SUPERVISED LEARNING

  • With 10% of ImageNet labels, ReLICv2 outperforms the supervised baseline and prior self-supervised methods; with 1%, only C-BYOL performs better, and BYOL beats ReLICv2 on larger ResNets at 10%.The authors hypothesize that ReLICv2 may need more training than its 20 epochs for larger models; BYOL trains for 30 or 50 epochs.Source: §B.2 · pp. 19–20, Table 9 · p. 20

B.3 TRANSFER

  • ReLICv2 is best on 7 of 11 transfer datasets in linear evaluation and 8 of 11 in fine-tuning, outperforming the supervised baseline and state-of-the-art self-supervised method across a wide range of datasets.Table 11 reports accuracy for a ResNet50 pretrained on ImageNet.Source: §B.3 · p. 21, Table 11 · p. 21

B.4 ROBUSTNESS AND OOD GENERALIZATION

The section evaluates frozen ResNet50 representations with a linear classifier trained on ImageNet, then tests them zero-shot on robustness and out-of-distribution datasets. It reports that RELICv2 outperforms supervised and competing self-supervised methods on ImageNetV2 and ImageNet-C.

B.4 ROBUSTNESS AND OOD GENERALIZATION

  • ImageNetV2 and ImageNet-C assess robustness, while ImageNet-R, ImageNet-Sketch, and ObjectNet assess generalization to altered styles, sketches, viewpoints, and backgrounds.Source: §B.4 · p. 21
  • Frozen ResNet50 representations are evaluated with a linear classifier trained on labelled ImageNet data, with zero-shot testing on the robustness and OOD datasets.Source: §B.4 · p. 21
  • RELICv2 outperforms the supervised baseline and competing self-supervised methods on ImageNetV2 and ImageNet-C under linear evaluation.The ImageNet-C results are broken down across corruption types in Table 12.Source: §B.4 · pp. 21–22

C ANALYSIS

The analysis compares nearest-neighbour representations learned by RELICv2 and BYOL for related ImageNet classes. RELICv2’s neighbours are closer and show less class and superclass confusion.

C ANALYSIS

  • RELICv2 learns closer nearest-neighbour representations with less confusion between related classes and super-classes than BYOL.The analysis uses Euclidean distances for two dog breeds and two cat breeds; a block-diagonal pattern would indicate neighbours from the same class.Source: C ANALYSIS · p. 23, Figure 7 · p. 23

D.1 LINEAR EVALUATION

The appendix reports JFT-300M evaluation and transfer results, robustness and OOD comparisons, and implementation details for RELICv2’s views and saliency masking. JFT-pretrained RELICv2 improves on DnC’s ImageNet accuracy, while its robustness and OOD performance is lower than for ImageNet-pretrained RELICv2.

D.1 LINEAR EVALUATION

  • More than 2%: RELICv2 improves over DnC after 1000 JFT-300M ImageNet-equivalent epochs and achieves better overall performance with fewer epochs.The comparison evaluates ImageNet top-1 accuracy under linear evaluation.Source: §D.1 · p. 24, Table 13 · p. 24

D.2 TRANSFER

  • Longer JFT-300M pretraining benefits RELICv2 transfer performance, which is comparable to DnC in the number of datasets with self-supervised state-of-the-art results.Source: §D.2 · p. 25, Table 14 · p. 25

D.3 ROBUSTNESS AND OOD GENERALIZATION

  • JFT-pretrained RELICv2 has worse robustness and OOD performance than ImageNet-pretrained RELICv2, and longer JFT pretraining does not necessarily improve those results.The authors hypothesize that JFT-300M has a larger distribution discrepancy from the evaluation datasets than ImageNet does.Source: §D.3 · p. 26, Table 15 · p. 26

E RELICV2 PSEUDO-CODE IN JAX

  • The pseudo-code applies saliency masking to large views, sends large views through online and target networks, and computes RELIC losses across large- and small-to-large-view pairs.Source: E RELICV2 PSEUDO-CODE IN JAX · p. 27

F.1 AUGMENTATIONS

  • Augmentation uses random crops, flips, colour changes, grayscale, blur, and solarization; training uses four large 224 × 224 views and two small 96 × 96 views.Source: §F.1 · p. 28, Table 16 · p. 28

F.2 SALIENCY MASKING

  • Saliency masks remove image backgrounds, which are replaced with randomly sampled grayscale, and are applied to large views with small probability during training.The mask is applied only when it covers at least 5% of the image; masked images are used only during training.Source: §F.2 · pp. 28–29, Figure 8 · p. 29

F.2.1 TRAINING THE SALIENCY DETECTION NETWORK TO OBTAIN SALIENCY MASKS

The saliency-mask ablations compare masking probabilities and mask-estimation training data. Masking 10% of the time performs best among tested probabilities, while MSRA-B-trained masks yield a slight ImageNet accuracy gain.

F.2.1 TRAINING THE SALIENCY DETECTION NETWORK TO OBTAIN SALIENCY MASKS

  • The fully unsupervised saliency detector refines pseudo-labels from four handcrafted methods, then trains a network to predict masks for all ImageNet training images.The handcrafted methods are RBD, MR, DSR, and MC, applied initially to a random subset of 2500 ImageNet images.Source: §F.2.1 · p. 29
  • Applying saliency masks 10% of the time gives the best performance and significantly improves over no masking.Source: §F.2.1 · p. 30

G ABLATIONS

  • The ablation study uses 300 pretraining epochs unless otherwise noted and begins by examining saliency-mask design choices.Source: G ABLATIONS · p. 31

G.1 USING DIFFERENT DATASETS FOR OBTAINING THE SALIENCY MASKS

  • 77.2% top-1 and 93.3% top-5: RELICv2 with MSRA-B-trained saliency masks after 1000 ImageNet pretraining epochs.The authors suggest the slight gain may reflect the greater variety of MSRA-B images used to train the saliency detector.Source: §G.1 · p. 31
  • Training the saliency detector on 5000 ImageNet images overfits, degrading mask quality; RELICv2 then reaches 76.7% top-1 and 93.3% top-5 after 1000 epochs.Source: §G.1 · p. 31

G.2 ANALYSIS AND ABLATIONS FOR SALIENCY MASKS

The saliency-mask ablations find that mask quality matters: saliency masks outperform random masks, and performance declines as masks are corrupted. Other hyperparameter tests find strong performance with relatively few negatives and robustness to EMA decay rates from 0.9 to 0.996.

G.2 ANALYSIS AND ABLATIONS FOR SALIENCY MASKS

  • Saliency masking also improves BYOL’s ImageNet linear-evaluation performance across tested mask-application settings.The comparison used 300-epoch models and varied the probability of applying masks to the two large augmented views.Source: §G.2 · p. 31, Table 17 · p. 32
  • Using MSRA-B rather than ImageNet to train the saliency detector yields better masks and better RELICv2 performance when masks are used.The experiments vary mask-application probability from 0 to 1 and use a ResNet50 (1x) trained for 300 epochs.Source: Table 18 · p. 32, §G.2 · p. 32
  • Random masks perform more than 1% below saliency masks; centered rectangles outperform randomly positioned rectangles, and rectangles outperform random points.The masks cover varying image areas and are applied with probability 0.1 during training.Source: §G.2 · p. 32, Table 19 · p. 33
  • Small saliency-mask corruptions are tolerated, but performance declines as mask quality degrades; bounding-box masks yield 74.5% top-1 and 92.2% top-5 accuracy.The bounding-box result is for a ResNet50 (1x) trained for 300 epochs with mask-application probability 0.1, and is lower than using the detector’s saliency masks.Source: §G.2 · p. 33, Table 20 · p. 33

G.3 OTHER MODEL HYPERPARAMETERS

  • Performance varies less after 1000-epoch pretraining than after 300 epochs, though variation is already quite low at 300 epochs.The passage reports this pattern for RELICv2 pretraining.Source: §G.3 · p. 34
  • With randomly subsampled negatives, 10 negatives give the best performance, while using the whole batch as negatives is among the worst settings.The comparisons use a ResNet50 (1x) trained for 300 epochs.Source: §G.3 · p. 33, Table 21 · p. 34
  • EMA decay rates from 0.9 to 0.996 yield similar top-1 accuracy, while rates of 0.999 and higher quickly degrade performance.The target network updates too slowly at the higher decay rates; RELICv2 is more robust at γ = 1 than the cited Grill et al. (2020) result.Source: Table 22 · p. 34, §G.3 · p. 34

Related papers

Loading 2201.05119v2…