PaperlayerView exact revision on arXiv

Source-linked AI summary

Selfie: Self-supervised Pretraining for Image Embedding

Trieu H. Trinh, Minh-Thang Luong, Quoc V. Le

arXiv:1906.02940v3cs.LGcs.CVeess.IVstat.ML

TL;DR

Selfie addresses the challenge of improving neural-network data efficiency with unlabeled data and adapts masked language modeling to continuous images by classifying masked patches among same-image distractors. It reports consistent accuracy gains across three benchmarks, including an 11.1-point absolute gain on ImageNet 224 × 224 with 5% labeled data, and reduced training variability.

  • Problem

    Using unlabeled data to improve neural-network data efficiency remains an open challenge, and images are continuous rather than discrete like the words used by masked language modeling.

    Source: §1 · pp. 1–2

  • Method

    Selfie pretrains image representations by classifying the correct masked patch among distractor patches sampled from the same image.

    Source: §1 · p. 2

  • Results

    Selfie improves test accuracy across CIFAR-10 and both ImageNet resolutions; with 5% labeled data, ImageNet 224 × 224 mean accuracy rises from 35.6% to 46.7%, an 11.1-point absolute gain.

    Source: §1 · p. 2, §3.3 · p. 6

  • Takeaways & Limitations

    Pretraining also reduces run-to-run test-accuracy variability, with reported reductions of 3× on CIFAR-10 and 8× on 5% ImageNet 32 × 32.

    Source: §1 · p. 2

    Transfer across tasks such as ImageNet to CIFAR is difficult, and ImageNet 224 × 224 results for the full labeled subset require additional pretraining tuning on a development set.

    Source: §3.3 · p. 7

Abstract

from arXiv · show

We introduce a pretraining technique called Selfie, which stands for SELFie supervised Image Embedding. Selfie generalizes the concept of masked language modeling of BERT (Devlin et al., 2019) to continuous data, such as images, by making use of the Contrastive Predictive Coding loss (Oord et al., 2018). Given masked-out patches in an input image, our method learns to select the correct patch, among other "distractor" patches sampled from the same image, to fill in the masked location. This classification objective sidesteps the need for predicting exact pixel values of the target patches. The pretraining architecture of Selfie includes a network of convolutional blocks to process patches followed by an attention pooling network to summarize the content of unmasked patches before predicting masked ones. During finetuning, we reuse the convolutional weights found by pretraining. We evaluate Selfie on three benchmarks (CIFAR-10, ImageNet 32 x 32, and ImageNet 224 x 224) with varying amounts of labeled data, from 5% to 100% of the training sets. Our pretraining method provides consistent improvements to ResNet-50 across all settings compared to the standard supervised training of the same network. Notably, on ImageNet 224 x 224 with 60 examples per class (5%), our method improves the mean accuracy of ResNet-50 from 35.6% to 46.7%, an improvement of 11.1 points in absolute accuracy. Our pretraining method also improves ResNet-50 training stability, especially on low data regime, by significantly lowering the standard deviation of test accuracies across different runs.

1 Introduction

The section frames data-efficient learning from unlabeled data as an open challenge and identifies the difficulty of applying masked language modeling to continuous images. Selfie adapts masked-patch prediction into a classification task, with reported accuracy and training-stability gains, especially in low-label settings.

1 Introduction

  • Unlabeled-data representation learning remains an open challenge for improving neural-network data efficiency because many systems still rely on supervised learning.The passage notes that neural networks often require large amounts of labeled data.Source: §1 · p. 1
  • Selfie adapts masked language modeling to continuous images by classifying the correct masked patch among same-image distractors rather than regressing exact pixels.The passage motivates classification as less sensitive than regression to small image perturbations.Source: §1 · p. 2
  • Selfie consistently improves accuracy across CIFAR-10 and both ImageNet resolutions as labeled data ranges from 5% to 100%; on ImageNet 224 × 224, it raises mean accuracy from 35.6% to 46.7% with 5% labeled data.The reported absolute gain is 11.1 points, and gains tend to be larger with smaller labeled sets.Source: §1 · p. 2
  • Pretraining also reduces run-to-run test-accuracy variability, including a 3× reduction on CIFAR-10 with 400 examples per class and an 8× reduction on 5% ImageNet 32 × 32.The reported variability is the standard deviation of final test accuracy across five runs.Source: §1 · p. 2

2 Method

Selfie pretrains the first three ResNet-50 blocks on image patches, then reuses those weights when finetuning on the target task. During pretraining, attention pools unmasked-patch features into an image representation used to classify masked patches at their locations.

2 Method

  • The method has two stages: pretrain the first three ResNet-50 blocks on unlabeled data, then initialize those blocks from the pretrained patch network and finetune ResNet-50 on full images.The final convolution block uses standard initialization.Source: §2 · p. 2, §2.3 · p. 5

2.1 Pretraining Details

  • During pretraining, shared patch processing produces feature vectors; attention pools encoder patches into u, while decoder features and a location embedding score candidate patches for cross-entropy classification.Same-image distractors are masked in the input, and the encoder and decoder are trained jointly. Predicting multiple masked locations at once reuses encoder-decoder computation.Source: §2 · p. 2, §2.1 · pp. 3–4

2.2 Attention Pooling

  • Transformer layers pool patch vectors into u, and for 224 × 224 images, each positional embedding is formed by summing separate row and column embeddings.This reduces the number of positional embeddings from 49 to 14 and is described as reducing parameters and helping regularization.Source: §2.2 · p. 5
  • The attention-pooling network uses BERT-style self-attention blocks with fully connected layers, GeLU, dropout, residual connections, and layer normalization.A learned vector u0 is fed with patch vectors through the Transformer layers, and the output corresponding to u0 is the pooling result.Source: §2.2 · p. 5

2.3 Finetuning Details

  • During finetuning, the pretrained first three convolution blocks initialize ResNet-50, which is then applied to full images and finetuned end-to-end.The last convolution block uses standard initialization.Source: §2.3 · p. 5

3 Experiments and Results

The experiments test whether unlabeled-data pretraining improves ResNet-50 across three image benchmarks and labeled-data budgets. Selfie consistently improves accuracy, with stronger gains in low-data settings, and also improves training stability.

3.1 Datasets

  • Experiments use CIFAR-10, ImageNet 32 × 32, and ImageNet 224 × 224, with approximately 5%, 10%, 20%, or 100% labeled data and the full training set for pretraining.The CIFAR-10 10% setting is replaced by 4,000 examples (8%).Source: §3.1 · p. 5, §3 · p. 5

3.2 Experimental setup

  • Models match standard ResNet-50 size and depth (about 25M parameters, 50 layers), train for 120,000 steps, and use 8 × 8 or 32 × 32 pretraining patches, predicting 50% or 75%.Three attention blocks are added for attention pooling, each with hidden size 1024, intermediate size 640, and 32 attention heads.Source: §3.2 · p. 6
  • Hyperparameters are tuned using 10% of the training data for validation, then each model is retrained on all training data across five random seeds.Reported results are the mean and standard deviation across those five runs.Source: §3.2 · p. 6

3.3 Results

  • 11.1% absolute accuracy gain on ImageNet 224 × 224 with 5% labeled data; Selfie consistently improves test accuracy across all three benchmarks and labeled-data sizes.The reported comparison is between ResNet-50 with and without pretraining.Source: §3.3 · p. 6, Table 1 · p. 7
  • Pretrained models generalize better on the test set despite usually converging to higher training loss; the paper describes this as a regularization effect.Figure 3 compares loss and accuracy during supervised training from random and pretrained initializations on 10% of ImageNet 224 × 224.Source: §3.3 · p. 6, Figure 3 · p. 6
  • Pretraining enhances training stability, as shown by reduced standard deviation in almost all experiments; when the unlabeled and labeled datasets are the same (Labeled Data Percentage = 100%), the gain is small.Source: §3.3 · p. 6
  • Finetuning ResNet-36 with attention pooling slightly outperforms finetuning ResNet-50 in some cases.The usual ResNet-50 finetuning setup drops the attention-pooling network used during pretraining.Source: §3.3 · p. 7
  • Transfer across tasks can be difficult, and the 100% ImageNet 224 × 224 result requires additional pretraining-phase tuning using a development set.The authors also note that pretraining processes patches independently, whereas finetuning processes the whole image.Source: §3.3 · p. 7

4 Analysis

On ImageNet 224 × 224, Selfie’s accuracy gains shrink as the labeled set grows. With pretrained weights, ResNet-36 plus attention pooling also outperforms ResNet-50 in limited-data settings and gives a small full-data gain.

4.1 Pretraining benefits more when there is less labeled data

  • At 20% labeled data, Selfie improves mean accuracy by 1.4%, while at 100% the gain is 0.1%.Figure 4 compares mean accuracy across five runs for supervised training and pretraining, and shows the mean gain.Source: §4.1 · p. 8, Figure 4 · p. 8

4.2 Self-attention as the last layer helps finetuning performance.

  • With pretraining, ResNet-36 plus attention pooling outperforms ResNet-50 on both datasets with limited data.Table 2 reports the two architectures after finetuning from Selfie-pretrained weights.Source: §4.2 · p. 8, Table 2 · p. 9
  • On the full ImageNet 224 × 224 training set, ResNet-36 plus attention pooling gives a 0.5% gain.The gain reflects using attention pooling in place of the last convolution block.Source: §4.2 · p. 8, Table 2 · p. 9

5 Related Work

Related work spans context-prediction objectives in NLP, several image pretraining tasks, and semi-supervised consistency methods. Selfie is related to denoising autoencoders and Contrastive Predictive Coding, but predicts missing patches by selecting among distractors.

Unsupervised representation learning for text

  • Language-model and masked-language-model pretraining use context prediction: inferring missing words from adjacent data and their locations.The cited approaches include word, sentence, and paragraph representation methods.Source: §5 · p. 9

Unsupervised representation learning for images and audio

  • Image representation methods include rotation prediction, exemplar classification, patch-location prediction or jigsaw solving, and inpainting.The passage says their success is limited to small datasets or settings, with some methods requiring expensive joint training to surpass supervised counterparts.Source: §5 · p. 9
  • Selfie differs from denoising autoencoders by predicting only missing patches and selecting the correct patch among distractors.It also uses Contrastive Predictive Coding, which employs negative sampling to classify continuous objects in a sequence.Source: §5 · p. 9

Semi-supervised learning

  • Semi-supervised learning uses labeled data and commonly relies on consistency training, unlike unsupervised representation learning’s separate finetuning stage.The passage names several consistency-training approaches.Source: §5 · p. 9

6 Conclusion

Selfie generalizes masked language modeling to continuous data by selecting masked image patches from same-image negative samples. The paper reports gains when labeled data is scarce, reduced run-to-run variability, and promise for a hybrid convolution-attention architecture.

6 Conclusion

  • Selfie generalizes masked language modeling to continuous data by selecting masked patches from negative samples drawn from the same image.This classification objective avoids predicting exact target-patch pixel values.Source: §6 · p. 9
  • Selfie achieves significant gains when the labeled set is small compared with the unlabeled set.The conclusion also reports reduced standard deviation across runs, attributed to improved initialization from pretraining.Source: §6 · pp. 9–10
  • The paper’s analysis finds that a hybrid convolution-attention architecture shows promise.Source: §6 · p. 10

Related papers

Loading 1906.02940v3…