Source-linked AI summary
Boosting Self-Supervised Learning via Knowledge Transfer
Mehdi Noroozi, Ananth Vinjimoor, Paolo Favaro, Hamed Pirsiavash
TL;DR
SSL transfer is limited by requiring the pretext and target models to share an architecture. The paper decouples them by converting learned features into clustered pseudo-labels for training a target model, and reports strong benchmark performance with a smaller architecture.
Problem
Requiring the same architecture for SSL pre-training and target-task fine-tuning limits exploration of deeper models, varied data, and challenging pretext tasks.
Method
The framework clusters features from an arbitrary SSL model, assigns cluster IDs as pseudo-labels, and trains a smaller target network to predict them.
Results
56.5% mAP was achieved on PASCAL VOC 2007 object detection with Jigsaw++, versus 59.1% mAP with supervised ImageNet pre-training.
Takeaways & Limitations
The framework supports deeper pre-training, comparison across incompatible models and tasks, and knowledge transfer from deep models to shallower target models.
Abstract
from arXiv · showhide
In self-supervised learning, one trains a model to solve a so-called pretext task on a dataset without the need for human annotation. The main objective, however, is to transfer this model to a target domain and task. Currently, the most effective transfer strategy is fine-tuning, which restricts one to use the same model or parts thereof for both pretext and target tasks. In this paper, we present a novel framework for self-supervised learning that overcomes limitations in designing and comparing different tasks, models, and data domains. In particular, our framework decouples the structure of the self-supervised model from the final task-specific fine-tuned model. This allows us to: 1) quantitatively assess previously incompatible models including handcrafted features; 2) show that deeper neural network models can learn better representations from the same pretext task; 3) transfer knowledge learned with a deep model to a shallower one and thus boost its learning. We use this framework to design a novel self-supervised task, which achieves state-of-the-art performance on the common benchmarks in PASCAL VOC 2007, ILSVRC12 and Places by a significant margin. Our learned features shrink the mAP gap between models trained via self-supervised learning and supervised learning from 5.9% to 2.6% in object detection on PASCAL VOC 2007.
1. Introduction
The paper addresses SSL’s reliance on the same architecture for pre-training and fine-tuning by introducing knowledge transfer that decouples them. This enables deeper pre-training, cross-model comparison, and transfer to shallower target models.
- Motivation: SSL commonly transfers pre-trained features to target tasks using the same model architecture, limiting pretext-task and model design choices.The limitation is especially relevant for tasks using different data domains, formats, dataset sizes, or deeper networks.
- Method: Knowledge transfer reduces a learned representation to pseudo-labels by clustering features, then trains a smaller target network to classify those labels.The procedure uses cluster IDs as pseudo-labels for unlabeled images.
- Contributions: The framework transfers knowledge between models with different architectures and training settings, including from complex pre-training models to final task models.This decoupling permits deeper models during pre-training while retaining a chosen target-task architecture.
- Contributions: It also enables comparison of learned and handcrafted features under a common reference model, dataset, and task.The stated example uses AlexNet, PASCAL VOC, and object detection.
- Results: 56.5% mAP was achieved by Jigsaw++ with self-supervised pre-training, compared with 59.1% mAP from supervised ImageNet pre-training on PASCAL VOC 2007 object detection.Both final models use AlexNet.
2. Prior Work
Prior SSL work learns representations from varied pretext tasks and evaluates them through transfer learning on common architectures and datasets. The paper identifies same-architecture transfer as a barrier to exploring deeper models and more challenging tasks.
- Self-supervised learning: SSL pretext tasks include image reconstruction, video-temporal ordering, spatial relations, jigsaw solving, and foreground-background segmentation.These tasks obtain supervisory signals from image structure, video temporal information, or compact signals derived from data.
- Prior limitations: Existing pretext tasks are commonly benchmarked through transfer learning using standard architectures such as AlexNet and datasets such as PASCAL.This practice leaves the broader design space for SSL methods unclear.
- Prior limitations: Using the same model for pretext training and target transfer restricts exploration of architectures, data formats, domains, and more challenging pretext tasks.The paper reports that deeper models can learn better representations from challenging tasks than shallower models.
3. Transferring Knowledge
The proposed framework transfers knowledge from an SSL model to a target-task model by clustering learned features into pseudo-labels. This removes the requirement that both stages share an architecture.
- Motivation: Standard SSL fine-tuning requires the final supervised model and SSL model to use the same architecture, constraining deeper pretext-task models.The paper motivates a new transfer method because the final supervised-task architecture is intended to remain unchanged.
- Knowledge transfer: The method clusters feature-space representations so semantically similar images receive the same pseudo-label.This follows the stated intuition that semantically similar data points are close in a good visual representation.
- Knowledge transfer: The target-architecture classifier is trained to predict cluster assignments, producing a new representation for the final task.The pipeline is summarized as pre-training, feature clustering, nearest-center pseudo-label assignment, and target-model training.
4. The Jigsaw++ Pretext Task
The Jigsaw++ pretext task makes the original jigsaw problem harder by replacing up to two tiles with tiles from another image, while the framework transfers representations from deeper pretraining models to AlexNet. Experiments examine architecture capacity, cluster-center choices, and data domains used for pseudo-label transfer.
- Architecture comparison: The framework separates SSL pretraining architecture from the final target architecture, enabling deeper pretraining while retaining AlexNet for transfer tasks.This addresses comparisons that are difficult when SSL and fine-tuning must use the same architecture.
- Task design: Jigsaw++ replaces up to two tiles in a 3 × 3 image puzzle with tiles from another random image.The occluding tiles are randomly selected in number and location, and the task uses 701 permutations after removing similar permutations.
- Task design: The occlusions require the model to detect altered tiles and solve the puzzle using the remaining patches.The task is designed to avoid ambiguity by enforcing a minimum Hamming distance of 3 between permutations.
- Ablations: The experiments vary the number of cluster centers and the datasets used for clustering and pseudo-label assignment.The data-domain study reports only a little reduction in VOC2007 classification when clustering and pseudo-label extraction use different datasets.
- Architecture comparison: VGG16 handles the increased Jigsaw++ complexity better than AlexNet and yields better representations and corresponding pseudo-labels.The models are trained on Jigsaw++, then features are clustered and AlexNet is trained to classify the resulting pseudo-labels.
5. Experiments
The experiments evaluate knowledge transfer and Jigsaw++ across multiple transfer-learning benchmarks, architectures, data domains, and feature types. The results show strong performance, robustness to clustering choices, and useful visual evidence about transferred representations.
- Transfer Learning Evaluation: Jigsaw++ features outperform current self-supervised learning methods across the evaluated transfer-learning benchmarks.Evaluations include PASCAL VOC fine-tuning, nonlinear ImageNet classification, and linear classification on Places and ImageNet.
- Transfer Learning Evaluation: All PASCAL VOC fine-tuning rows use AlexNet at fine-tuning, while CC+vgg-Jigsaw++ transfers knowledge from VGG16-trained Jigsaw++ through cluster classification.The table also compares classification, detection, and semantic segmentation, and includes CC+HOG based on handcrafted features.
- Ablation Studies: Increasing the number of clusters does not make the network highly sensitive to the clustering choice.The analysis notes a possible trade-off: more clusters provide finer categorization but fewer samples per cluster may increase overfitting.
- Ablation Studies: The transferred model shows only a small performance reduction when cluster centers are learned on Places and pseudo-labels are assigned on ImageNet.This experiment tests flexibility across pre-training, clustering, and pseudo-label datasets.
- Transfer Learning Evaluation: The supervised-learning gap shrinks from 5.9% to 2.6% on PASCAL VOC 2007 object detection.The reported reductions are 1.6% in detection and 2.6% in semantic segmentation in the corresponding benchmark results.
- Transfer Learning Evaluation: Pre-training Jigsaw++ with VGG16 provides an average improvement of almost 2% for linear classification after transfer to AlexNet.The transferred CC+Jigsaw++ model also benefits from the knowledge-transfer procedure, while Jigsaw++ is comparable to prior state-of-the-art methods.
- Visualizations: Filters learned after transfer reflect properties of the pre-training task, including the absence of color after Color-Dropping pre-training.Cluster samples for CC+vgg-Jigsaw++ are organized by rows around cluster centers, with same-category images expected within each row.
6. Conclusions
The paper decouples self-supervised pre-training from final-task fine-tuning through clustering-based knowledge transfer. Using VGG16 for the harder Jigsaw++ task and transferring to AlexNet yields state-of-the-art results while narrowing the supervised-versus-self-supervised gap.
- 6. Conclusions: Clustering learned features decouples the pre-training model from the final task model, enabling knowledge transfer to a different architecture.The paper increases Jigsaw++ complexity, trains it with VGG16, and transfers the resulting knowledge back to AlexNet.
- 6. Conclusions: CC+vgg-Jigsaw++ outperforms state-of-the-art self-supervised models and narrows the PASCAL VOC 2007 object-detection gap from 5.9% to 2.6%.The final task continues to use AlexNet despite the deeper VGG16 pre-training model.