Source-linked AI summary
ScaleNet: An Unsupervised Representation Learning Method for Limited Information
Huili Huang, M. Mahdi Roozbahani
TL;DR
Large labeled datasets are costly to collect, motivating methods that learn visual representations with limited information. ScaleNet trains rotation prediction on progressively larger image scales and transfers parameters between models. It outperforms RotNet in limited-information settings, improves limited-data SimCLR, and transfers useful parameters to larger-data classification.
Problem
Collecting and annotating large-scale labeled datasets is time-consuming and expensive, motivating self-supervised representation learning when information is limited.
Method
ScaleNet trains rotation prediction on resized images, transfers the learned parameters to larger-scale models, and applies the approach to existing self-supervised models.
Results
ScaleNet outperforms RotNet by about 7% on limited CIFAR-10 classification and improves larger-dataset RotNet classification by 6.49% after parameter transfer.
Takeaways & Limitations
ScaleNet improves self-supervised representation learning under limited information and also enhances SimCLR and larger-dataset RotNet through transferred parameters.
Abstract
from arXiv · showhide
Although large-scale labeled data are essential for deep convolutional neural networks (ConvNets) to learn high-level semantic visual representations, it is time-consuming and impractical to collect and annotate large-scale datasets. A simple and efficient unsupervised representation learning method named ScaleNet based on multi-scale images is proposed in this study to enhance the performance of ConvNets when limited information is available. The input images are first resized to a smaller size and fed to the ConvNet to recognize the rotation degree. Next, the ConvNet learns the rotation-prediction task for the original size images based on the parameters transferred from the previous model. The CIFAR-10 and ImageNet datasets are examined on different architectures such as AlexNet and ResNet50 in this study. The current study demonstrates that specific image features, such as Harris corner information, play a critical role in the efficiency of the rotation-prediction task. The ScaleNet supersedes the RotNet by ~7% in the limited CIFAR-10 dataset. The transferred parameters from a ScaleNet model with limited data improve the ImageNet Classification task by about 6% compared to the RotNet model. This study shows the capability of the ScaleNet method to improve other cutting-edge models such as SimCLR by learning effective features for classification tasks.
1 Introduction
ScaleNet addresses the cost and scarcity of large labeled datasets by improving existing self-supervised models under limited information. It uses multi-scale learning to improve representations and reports gains over RotNet and SimCLR.
- Motivation: Large-scale ConvNet datasets are time-consuming and expensive to collect and annotate, limiting their practicality in several scientific fields.The passage specifically identifies neuroscience, medical diagnosis, material science, and chemistry applications.
- Related approaches: Researchers have used augmentation and transfer learning to obtain effective visual representations from limited labeled data.Examples include scaling, rotating, cropping, synthetic samples, and transferring ConvNet-based representations.
- Self-supervised learning: Self-supervised learning trains models on annotation-free pretext tasks using automatically generated pseudo-labels or transformed views.The related methods include colorization, patch-location prediction, jigsaw solving, contrastive learning, and image clustering.
- Contribution: ScaleNet improves existing self-supervised models under limited training samples, missing corner information, and lacking color information rather than introducing a new architecture.The stated target models include RotNet and SimCLR.
- Contribution: ScaleNet resizes images, trains rotation prediction at smaller scale, then trains a larger-scale model using transferred parameters.The method is evaluated with AlexNet and ResNet50.
- Results: 7.03%: ScaleNet outperforms RotNet on the classification task with a limited CIFAR-10 dataset.The same passage also reports a 1.23% gain without image corner information and an approximately 4% improvement for limited-data SimCLR with small batches.
2 ScaleNet
ScaleNet performs rotation prediction across progressively larger image scales, transferring parameters from smaller-scale models to larger-scale models. Its pretext task uses automatically generated geometric-transformation labels and four rotation classes.
- Self-supervised learning: Self-supervised learning replaces human labels with automatically generated pseudo-labels for training ConvNets on images.The model is trained with images X and pseudo-labels Y* rather than human-labeled pairs X and Y.
- Scale construction: The resize parameter α ≤1 maps images of dimensions R×C to a rescaled dataset with dimensions αR×αC.The resized sample is denoted xαi.
- Geometric transformations: The geometric transformation framework generates pseudo-labels for transformations such as resizing and rotation, and Fα(.) predicts the transformation distribution.The transformed resized dataset is labeled automatically by the ConvNet Fα(.).
- Rotation prediction: Rotation prediction uses four classes corresponding to 0, 90, 180, and 270 degrees.The rotations are implemented through transposing and horizontal or vertical flipping operations.
- Representation learning: The rotation pretext task is intended to encourage recognition of high-level object structures rather than trivial low-level visual features.Prior RotNet analyses associate the learned features with texture, shape, and high-level semantic structure.
- Multi-scale training: ScaleNet first trains a ConvNet on resized images for rotation prediction, then transfers its parameters to a larger-scale ConvNet.The multi-scale pretext task applies this progression before training on the original image size.
- Multi-scale training: ScaleNet can chain more than one pretrained model before the final original-scale model, such as resize operators α = [0.5, 0.75, 1].A two-model example trains F0.5(.) before F1(.) using the original input dataset.
3 Experiments
The experiments evaluate ScaleNet across resize settings, limited-information conditions, SimCLR, and ImageNet transfer. Results show gains over RotNet, especially when corner information or labeled data are limited, while transferred ScaleNet parameters improve larger-data classification.
- Experimental design: ScaleNet uses multi-scale rotation prediction, transferring parameters from smaller-input images to train the original-size model.The experiments compare two- and three-model ScaleNet variants with RotNet across resize operators and learning-rate combinations.
- CIFAR-10 limited information: ~7%: ScaleNet outperforms RotNet on the limited-information CIFAR-10 classification task.The limited-information setting combines 4K randomly selected images with missing corner information; removing corners reduces both pretext and classification accuracy.
- CIFAR-10 limited information: ScaleNet parameters are associated with rotation-invariant and scale-invariant features, whereas RotNet parameters are described as rotation-invariant.The comparison is based on the dependence of rotation prediction on Harris corner features and the classification behavior with 4K data.
- Multi-scale SimCLR: Multi-scale SimCLR is evaluated with different epochs, batch sizes, and classification-data amounts on CIFAR-10.The selected multi-scale SimCLR configuration uses resize operators [0.5, 1] and learning rates [0.001, 0.001].
- ImageNet experiments: 0.93%: ScaleNet accuracy is higher than RotNet accuracy when trained on 65,000 ImageNet images.The ImageNet experiment uses AlexNet and evaluates downstream classification with a logistic regression layer.
- ImageNet experiments: 6.49%: RotNet initialized from ScaleNet parameters outperforms RotNet initialized from RotNet parameters after transfer to 240,000 ImageNet samples.Both pre-trained models were trained on 65,000 samples before being transferred to the larger-data RotNet classification setting.
- Attention map analysis: ScaleNet attention maps indicate greater emphasis on edge information and specific semantic features than RotNet.The comparison uses ResNet50 attention maps across the first three stages.
4 Conclusion
ScaleNet improves self-supervised representation learning with limited information and extends benefits across architectures, datasets, and models.
- ScaleNet outperforms RotNet with ResNet50 and AlexNet, especially when information is limited.
- Training ScaleNet on small datasets improves RotNet performance on larger datasets.
- ScaleNet enhances other self-supervised learning models, including SimCLR.
- ScaleNet trains multi-scale images to extract high-quality representations and learn detailed features.
- Future work will examine additional affine transformations, image information beyond color and corners, and other self-supervised models.