Source-linked AI summary
Unsupervised Representation Learning by Predicting Image Rotations
Spyros Gidaris, Praveer Singh, Nikos Komodakis
TL;DR
Learning semantic visual features usually depends on large manually labeled datasets, motivating scalable unsupervised alternatives. This paper trains ConvNets to recognize image rotations and achieves state-of-the-art results across unsupervised benchmarks, narrowing the gap with supervised feature learning.
Problem
Learning powerful ConvNet representations typically requires massive manually labeled datasets, motivating annotation-free semantic feature learning.
Method
The method trains ConvNets to classify four image rotations: 0, 90, 180, and 270 degrees.
Results
The approach achieves state-of-the-art performance across evaluated unsupervised benchmarks; PASCAL VOC 2007 detection reaches 54.4% mAP, 2.4 points below supervised learning.
Takeaways & Limitations
The rotation-based self-supervised task significantly narrows the gap between unsupervised and supervised feature learning across several vision tasks.
Takeaways & Limitations
Rotation recognition is well defined for typical upright-object images but can be ambiguous for images depicting only round objects.
Abstract
from arXiv · showhide
Over the last years, deep convolutional neural networks (ConvNets) have transformed the field of computer vision thanks to their unparalleled capacity to learn high level semantic image features. However, in order to successfully learn those features, they usually require massive amounts of manually labeled data, which is both expensive and impractical to scale. Therefore, unsupervised semantic feature learning, i.e., learning without requiring manual annotation effort, is of crucial importance in order to successfully harvest the vast amount of visual data that are available today. In our work we propose to learn image features by training ConvNets to recognize the 2d rotation that is applied to the image that it gets as input. We demonstrate both qualitatively and quantitatively that this apparently simple task actually provides a very powerful supervisory signal for semantic feature learning. We exhaustively evaluate our method in various unsupervised feature learning benchmarks and we exhibit in all of them state-of-the-art performance. Specifically, our results on those benchmarks demonstrate dramatic improvements w.r.t. prior state-of-the-art approaches in unsupervised representation learning and thus significantly close the gap with supervised feature learning. For instance, in PASCAL VOC 2007 detection task our unsupervised pre-trained AlexNet model achieves the state-of-the-art (among unsupervised methods) mAP of 54.4% that is only 2.4 points lower from the supervised case. We get similarly striking results when we transfer our unsupervised learned features on various other tasks, such as ImageNet classification, PASCAL classification, PASCAL segmentation, and CIFAR-10 classification. The code and models of our paper will be published on: https://github.com/gidariss/FeatureLearningRotNet .
1 INTRODUCTION
The paper addresses the dependence of ConvNet representation learning on massive manually labeled datasets by proposing a self-supervised task based on recognizing applied geometric transformations. It reports state-of-the-art results across diverse vision benchmarks and a narrowed gap between unsupervised and supervised feature learning.
- Motivation: ConvNets learn powerful visual representations from object-recognition or scene-classification training, but these tasks require massive amounts of manually labeled data.This motivates annotation-free alternatives for learning high-level representations.
- Self-supervised learning: Self-supervised learning uses annotation-free pretext tasks based only on visual information to provide surrogate supervision for feature learning.The rationale is that solving such tasks forces ConvNets to learn semantic image features useful for other vision tasks.
- Method: The proposed method trains ConvNets to recognize which geometric transformation was applied to each image, using a small set of discrete transformations.The illustrated transformations are random multiples of 90 degrees: 0, 90, 180, or 270 degrees.
- Contribution: The paper presents this simple self-supervised task as a powerful supervisory signal for semantic feature learning.This is stated as a central contribution of the work.
- Evaluation: The method is evaluated across semi-supervised and transfer-learning settings and across CIFAR-10, ImageNet, Places, and PASCAL classification, detection, and segmentation tasks.The evaluation spans various settings and vision tasks.
- Results: The self-supervised formulation achieves state-of-the-art results with dramatic improvements over prior unsupervised approaches and significantly narrows the gap with supervised feature learning.These outcomes are reported across the evaluated vision tasks.
2 METHODOLOGY
The method trains a ConvNet to predict which geometric transformation was applied to an input image, using rotations by 0, 90, 180, and 270 degrees as a self-supervised task. This rotation-recognition objective is designed to encourage semantic feature learning while avoiding low-level artifacts and retaining supervised-like computational efficiency.
- Self-supervised formulation: The method trains a ConvNet to estimate the unknown label of a geometric transformation applied to an input image.The model outputs a probability distribution over all possible transformations, with learnable parameters θ.
- Rotation task: The transformation set consists of image rotations by 0, 90, 180, and 270 degrees.These four rotations define the classification task used for semantic feature learning.
- Semantic feature learning: Rotation recognition is intended to force the ConvNet to localize salient objects, recognize their orientation and type, and relate orientation to object-specific visual patterns.The paper argues that effective prediction requires learning semantic parts and object classes rather than relying only on superficial cues.
- Design advantages: Using rotations by multiples of 90 degrees avoids easily detectable low-level visual artifacts that could produce trivial, practically useless features.The rotations can be implemented with flip and transpose operations.
- Design advantages: The task is generally well posed because human-captured images usually depict objects upright, making the applied rotation identifiable except for images showing only round objects.The paper contrasts this with object scale, which varies substantially in human-captured images.
- Computational efficiency: Around 2 days on a single Titan X GPU is the reported AlexNet training time, while the task retains supervised learning’s computational cost and similar convergence speed.The authors also state that it converges significantly faster than image-reconstruction-based approaches and supports supervised-learning parallelization schemes.
3 EXPERIMENTAL RESULTS
The experiments evaluate rotation-recognition features across standard image datasets, vision tasks, transfer-learning, and semi-supervised settings against state-of-the-art methods. On CIFAR-10, four discrete rotations provide the strongest object-recognition performance, while transfer to PASCAL VOC remains competitive with or better than prior unsupervised approaches.
- Evaluation scope: The evaluation spans CIFAR-10, ImageNet, PASCAL, and Places205 across object detection, segmentation, classification, transfer learning, and semi-supervised learning.The approach is compared with corresponding state-of-the-art methods in all settings.
- CIFAR-10 evaluation: On CIFAR-10, features are evaluated by training supervised non-linear classifiers on representations from different depths and RotNet model depths.The experiments use RotNet models with 3, 4, and 5 convolutional blocks and classifiers trained on features from each block.
- Rotation-task design: Four discrete rotations outperform the 8-rotation and 2-rotation variants on object recognition.The 2-rotation task provides fewer recognition classes, while the 8-rotation task introduces a less favorable geometric-transformation setting.
- CIFAR-10 comparisons: The CIFAR-10 experiments compare RotNet features with supervised, unsupervised, and hand-crafted feature-learning methods using non-linear and convolutional classifiers.The compared RotNet representations come from the second convolutional block of a four-block model.
- Semi-supervised learning: The method is also evaluated semi-supervisedly by pretraining RotNet on all CIFAR-10 images and training object classifiers with only a subset of labels.The classifiers use features from the second convolutional block of a four-block RotNet model.
- Transfer learning: Fine-tuning on PASCAL VOC classification, detection, and segmentation outperforms competing unsupervised methods by significant margins and narrows the gap with supervised learning.The method also surpasses or matches prior state-of-the-art unsupervised approaches on classes unseen during unsupervised training.
4 CONCLUSIONS
The paper proposes self-supervised feature learning by training ConvNets to recognize image rotations. Despite the task’s simplicity, the learned semantic features support object recognition, detection, and segmentation.
- 4 CONCLUSIONS: The method trains a ConvNet to recognize the rotation applied to each input image.This is presented as a novel formulation for self-supervised feature learning.
- 4 CONCLUSIONS: The simple rotation-prediction task forces the ConvNet to learn semantic features useful across visual perception tasks.The passage specifically identifies object recognition, object detection, and object segmentation.
- 4 CONCLUSIONS: The learned features are useful for object recognition, object detection, and object segmentation.These are the visual perception tasks explicitly named in the conclusion.
APPENDIX A VISUALIZING ATTENTION MAPS OF ROTATED IMAGES
Attention maps from a rotation-recognition AlexNet remain roughly consistent across all rotated copies of an image. This equivariance indicates that the network focuses on the same object parts regardless of image rotation.
- Attention-map behavior: Attention maps for all rotated copies of an image are roughly the same, showing equivariance with respect to image rotations.The visualizations concern Conv3 and Conv5 feature maps from an AlexNet trained to recognize image rotations.
APPENDIX B PER CLASS BREAKDOWN OF DETECTION AND CLASSIFICATION PERFORMANCE
This appendix reports per-class performance for the unsupervised learning method on PASCAL VOC 2007 detection and CIFAR-10 classification. The corresponding results appear in Tables 8 and 9, using average precision for detection and accuracy for classification.
- PASCAL VOC 2007 detection: PASCAL VOC 2007 detection results are reported with the average precision metric.Table 8 compares the unsupervised results with a supervised ImageNet-labels entry from Doersch et al. (2015).
- CIFAR-10 classification: CIFAR-10 results are reported as per-class classification accuracy.This breakdown is presented in Table 9.