PaperlayerView exact revision on arXiv

Source-linked AI summary

Self-Supervised Learning of Pretext-Invariant Representations

Ishan Misra, Laurens van der Maaten

arXiv:1912.01991v1cs.CVcs.LG

TL;DR

Semantic image representations are difficult to learn without scalable annotations, and common transformation-prediction tasks can make representations covariant rather than invariant. PIRL addresses this by contrasting original and transformed images, achieving state-of-the-art results across self-supervised benchmarks and surpassing supervised pre-training for object detection. The paper studies PIRL mainly with Jigsaw and also evaluates Rotation, while identifying richer transformation sets as future work.

  • Problem

    Pre-defined semantic annotations scale poorly, while transformation-prediction pretext tasks encourage covariance even though semantic recognition benefits from transformation invariance.

  • Method

    PIRL learns invariant representations by contrasting each image with its transformed counterpart and with representations of other images.

  • Results

    PIRL achieves state-of-the-art results across multiple self-supervised image-classification and object-detection benchmarks and outperforms supervised ImageNet pre-training on object detection.

  • Takeaways & Limitations

    Learning invariance to pretext-task transformations improves representation quality across image classification and object detection settings.

  • Takeaways & Limitations

    The study uses PIRL with Jigsaw and Rotation transformations and leaves extension to richer transformation sets for future work.

Abstract

from arXiv · show

The goal of self-supervised learning from images is to construct image representations that are semantically meaningful via pretext tasks that do not require semantic annotations for a large training set of images. Many pretext tasks lead to representations that are covariant with image transformations. We argue that, instead, semantic representations ought to be invariant under such transformations. Specifically, we develop Pretext-Invariant Representation Learning (PIRL, pronounced as "pearl") that learns invariant representations based on pretext tasks. We use PIRL with a commonly used pretext task that involves solving jigsaw puzzles. We find that PIRL substantially improves the semantic quality of the learned image representations. Our approach sets a new state-of-the-art in self-supervised learning from images on several popular benchmarks for self-supervised learning. Despite being unsupervised, PIRL outperforms supervised pre-training in learning image representations for object detection. Altogether, our results demonstrate the potential of self-supervised learning of image representations with good invariance properties.

1. Introduction

Self-supervised learning avoids poorly scalable semantic annotations, but transformation-prediction pretext tasks often produce covariant representations that retain less semantic information. PIRL instead learns transformation-invariant representations and improves self-supervised performance across recognition settings.

  • Pre-defined semantic annotations scale poorly to the long tail of visual concepts, limiting further improvements in image recognition.
  • Self-supervised learning addresses annotation limitations by learning from pixels through pretext tasks based on image transformations such as rotations, affine transformations, and jigsaw permutations.
  • Transformation-prediction tasks encourage covariance with the transformation, although semantic recognition generally benefits from invariance because transformations do not alter visual semantics.
  • PIRL makes representations of an image and its transformed version similar while separating them from representations of other images, adapting Jigsaw to invariant learning.
  • PIRL improves on covariant representations across vision tasks, sets a new self-supervised state of the art, and outperforms supervised pre-training for object detection.

2. PIRL: Pretext-Invariant Representation Learning

PIRL trains representations to remain similar across transformed views while contrasting them against other images. Its contrastive objective uses negative samples from a memory bank and combines transformed-view alignment with additional representation comparisons.

  • PIRL trains a convolutional network to produce representations invariant to a specified set of image transformations, including patch reshuffling and rotations.
  • Unlike covariant pretext losses that preserve transformation information, PIRL uses contrastive learning to align an image with its transformed counterpart and separate it from other images.
  • The contrastive objective uses positive image-transformation pairs and N negative samples drawn from other images.
  • A memory bank stores an exponential moving average representation for every dataset image, providing many cached negatives without enlarging the training batch.
  • PIRL combines two NCE losses to compare transformed and untransformed representations, with λ = 0 recovering the loss used by NPID.
  • For Jigsaw, PIRL uses ResNet-50 features, nine independently represented patches, random patch ordering, and 128-dimensional projected image representations.

3. Experiments

PIRL is evaluated across object detection, linear image classification, semi-supervised classification, and pre-training data distributions. Across these settings, invariant representations generally outperform covariant and competing self-supervised representations, with strong results against supervised baselines.

  • 3.1. Object Detection: PIRL outperforms all alternative self-supervised methods on VOC object detection across APall, AP50, and AP75, improving AP by 5 points over Jigsaw.PIRL also outperforms NPID++, isolating benefits associated with pretext invariance.
  • 3.1. Object Detection: PIRL surpasses supervised ImageNet pre-training on the conservative APall and AP75 object-detection metrics without extra pre-training data or architectural changes.The comparison uses the same backbone, finetuning epochs, and pre-training data, but PIRL does not use labels.
  • 3.2. Image Classification with Linear Models: PIRL improves ImageNet linear-classification accuracy by over 15% relative to covariant Jigsaw representations and achieves the highest single-crop top-1 accuracy among single-ResNet-50 self-supervised learners.Across other datasets, PIRL sets new self-supervised state-of-the-art results on VOC07, Places205, and iNaturalist2018.
  • 3.2. Image Classification with Linear Models: PIRL substantially outperforms NPID++ in linear classification, despite NPID++ reaching 59% single-crop top-1 accuracy on ImageNet.This comparison examines the effect of adding pretext invariance through λ > 0.
  • 3.3. Semi-Supervised Image Classification: Finetuning PIRL with 1% (∼13,000) labeled ImageNet images achieves 57% single-crop top-5 accuracy and outperforms Jigsaw and NPID++.PIRL performs at least as well as S4L and better than VAT in this semi-supervised setting.
  • 3.4. Pre-Training on Uncurated Image Data: Changing pre-training data produces mixed results: ImageNet pre-training is better for ImageNet classification, whereas YFCC-1M pre-training is better for Places205 classification.PIRL also outperforms Jigsaw and DeeperCluster trained on 100× more data from the same distribution.

4. Analysis

The analysis examines whether PIRL learns transformation-invariant representations and how its design choices affect representation quality. PIRL representations are more invariant than Jigsaw representations, with performance depending on layer, λ, transformations, and negative samples.

  • Representation invariance: PIRL representations of images and their transformed versions are generally similar, whereas Jigsaw representations have larger mean and variance in their distances.Distances are computed between normalized representations across transformations.
  • Layer analysis: PIRL representations achieve their best quality at the ResNet-50 res5 layer, while Jigsaw quality improves through res4 before sharply decreasing at res5.The paper attributes PIRL’s res5 advantage to invariance allowing greater focus on semantic information.
  • Loss analysis: PIRL performance is sensitive to λ, with the best performance obtained at λ=0.5.λ trades off two NCE losses; λ=0 removes the pretext-invariance term.
  • Transformation complexity: PIRL outperforms Jigsaw for every tested number of patch permutations and particularly benefits from very large transformation sets.PIRL can use all 9! ≈3.6 million permutations because it does not predict the permutation explicitly.
  • Negative samples: Increasing the number of negative samples tends to improve the quality of PIRL representations.Quality is measured using top-1 accuracy of linear ImageNet classifiers on fixed PIRL representations.
  • Other pretext tasks: PIRL’s invariance principle generalizes beyond Jigsaw: PIRL with Rotation yields +11% top-1 accuracy on ImageNet, and combining Jigsaw and Rotation transforms further improves representations.The experiments use rotations by {0°, 90°, 180°, 270°}.

5. Related Work

The paper builds on reconstruction, pretext-task, and invariant-representation methods in self-supervised learning. PIRL differs by learning invariance to both data augmentations and pretext image transformations rather than predicting transformations or missing data.

  • Prior self-supervision: Earlier self-supervised approaches reconstruct images or model limited data-distribution properties through tasks such as frame ordering, tracking, and cross-modal prediction.These approaches span image and video data.
  • Image pretext tasks: Image-based pretext tasks include colorization, orientation and affine-transform prediction, contextual patches, patch reordering, visual-primitive counting, and combinations of these tasks.PIRL instead learns representations invariant to the applied image transformations.
  • Invariant representations: Invariant-representation methods use contrastive learning, clustering, or mutual-information maximization, often targeting invariance under standard data augmentation.PIRL additionally targets invariance to pretext image transformations.
  • Predictive learning: PIRL differs from predictive-learning approaches using contrastive loss because it learns invariances rather than predicting missing data.Prior predictive methods may predict future video frames or operate on multiple views.

6. Discussion and Conclusion

The paper concludes that PIRL learns transformation-invariant representations that preserve semantic information and achieves strong results across self-supervised benchmarks. Its experiments are limited to Jigsaw and Rotation, motivating future work with richer transformations and clustering combinations.

  • Conclusion: PIRL achieves state-of-the-art results on multiple self-supervised image-classification and object-detection benchmarks.The paper attributes its rationale to preserving semantic information through invariance to image transformations.
  • Conclusion: PIRL outperforms supervised ImageNet pre-training on object detection.This is reported as a result despite PIRL being unsupervised.
  • Future work: The study uses PIRL with Jigsaw and Rotation transformations, while future work will investigate richer transformation sets and combinations with clustering-based approaches.The proposed clustering combination is described as potentially improving image representations.

A. Training architecture and Hyperparameters

PIRL uses ResNet-50 backbones with linear pre-training heads and standardized image augmentations. Main experiments use 800 epochs and 32,000 negatives, while analysis experiments reduce training to 400 epochs and 4,096 negatives.

  • Architecture: The standard PIRL architecture is a 25.6-million-parameter ResNet-50, with linear heads f(·) and g(·) used during pre-training.A larger PIRL-c2x variant has 98 million parameters and doubles channels in each ResNet stage.
  • Training hyperparameters: Section 3 models train for 800 epochs on ImageNet using 32 GPUs, batch size 32 per GPU, SGD with cosine learning-rate decay, and 32,000 negatives.The optimizer uses momentum 0.9 and weight decay 10^-4.
  • Training hyperparameters: Section 4 uses the same hyperparameters as Section 3 except for 400 epochs and 4,096 negatives, producing lower absolute performance for analysis experiments.The altered setup is intended to make training the larger number of analysis models feasible.
  • Data preprocessing: All methods use common geometric and photometric preprocessing, including random resized crops, horizontal flips, and randomized RGB color alterations.The preprocessing is implemented with PyTorch and the Python Imaging Library.

A.1. Details for PIRL with Jigsaw

The Jigsaw implementation builds nine independently augmented patches from a resized crop, while the Rotation implementation applies one of four rotations to an augmented image. Their architectures map transformed and original inputs to 128-dimensional features.

  • Jigsaw preprocessing: Jigsaw preprocessing extracts a random resized crop covering at least 60% of the image, divides it into a 3 × 3 grid, and takes one 64×64 patch from each cell.Each patch receives independent photometric augmentation before forming the nine-patch transformed input.
  • Jigsaw architecture: The Jigsaw network average-pools each patch’s ResNet-50 res5 features, projects them to 128 dimensions, concatenates them, and produces a final 128-dimensional feature.The nine patch features form a 1152-dimensional vector before the final projection.
  • Rotation preprocessing: Rotation preprocessing applies standard geometric and photometric augmentation, then randomly rotates the resulting 224 × 224 image by 0°, 90°, 180°, or 270°.The original image uses the same standard augmentation without the additional random rotation.
  • Rotation architecture: The Rotation architecture average-pools ResNet-50 res5 features for transformed and original images and linearly projects each to 128 dimensions.The transformed and original images are processed separately to obtain vIt and vI.

B.1. VOC07 train+val set for detection

This evaluation finetunes Faster R-CNN C4 models on the 5K-image VOC07 train+val set and reports detection AP on VOC07. PIRL outperforms supervised ImageNet pre-training on AP75 and APall without extra pretraining data or architecture changes.

  • Dataset and protocol: The experiment finetunes Faster R-CNN C4 detection models using the smaller VOC07 train+val set containing 5K images.Models use Detectron2 and hyperparameters from prior work.
  • Results: PIRL outperforms ImageNet supervised pre-training on the stricter AP75 and APall detection metrics.The comparison is made after finetuning on VOC07 train+val.
  • Results: The PIRL detection advantage requires neither extra pretraining data nor changes to the network architecture.The table compares ResNet-50-backbone models under the stated finetuning setup.

B.2. VOC07+12 train set for detection

The VOC07+12 detection experiment uses the established training split and VOC07 test set, with models finetuned for 25K iterations under specified learning-rate schedules.

  • Dataset and evaluation: The experiment uses the VOC07+12 training split and VOC07 test set, following prior work’s evaluation setup.The passage specifies this configuration for Table 1.
  • Optimization: Models are finetuned for 25K iterations, with the learning rate reduced by 0.1 at 17K iterations.The protocol uses batch size 2 per GPU across 8 GPUs.
  • Optimization: Supervised and Jigsaw baselines start at learning rate 0.02, whereas NPID++ and PIRL start at 0.003.Both groups use linear warmup with slope 1/3, but the supplied passage truncates the latter warmup duration.

C. Linear Models for Transfer

Linear evaluation trains classifiers on intermediate ResNet-50 representations across several datasets and reports classification accuracy or VOC07 mean average precision. Results are organized by layer in Table 7.

  • Evaluation protocol: Intermediate ResNet-50 features are average-pooled to approximately 9,000 dimensions per layer before training linear models.The linear models use mini-batch SGD with learning rate decay, momentum 0.9, and weight decay 5×10−4.
  • Evaluation protocol: ImageNet and iNaturalist2018 linear models are trained for 2 and 84 epochs, respectively, while updates are kept roughly constant across datasets.The supplied passage states 2 epochs for ImageNet and 84 epochs for iNaturalist2018; the preceding dataset is truncated.
  • Metrics: The evaluation reports center-crop top-1 accuracy on ImageNet, Places205, and iNaturalist2018, and mean average precision for VOC07.VOC07 uses linear SVMs following prior work.
  • Per-layer results: Table 7 reports performance for all ResNet-50 layers on ImageNet and Places205 under the established linear-evaluation protocol.Most values are adopted from the respective papers, except NPID++ and Ours.
Loading 1912.01991v1…