Source-linked AI summary
Multi-task Self-Supervised Visual Learning
Carl Doersch, Andrew Zisserman
TL;DR
The paper asks how to train visual representations without manual labels when individual self-supervised tasks remain weaker than supervised pre-training. It fairly compares four tasks, combines them in a shared network, and studies lasso regularization and input harmonization. Combining tasks improves performance, nearly closes the ImageNet-pretraining gap on VOC detection, and completely closes it on NYU depth.
Problem
Self-supervised methods provide automatically measurable objectives but remained less effective than ImageNet pre-training for semantic visual tasks.
Method
The paper fairly compares four self-supervised tasks using ResNet-101, jointly trains them in a shared trunk, and evaluates lasso feature combinations and input harmonization.
Results
Combining self-supervised tasks always improves performance; the four-task model nearly closes the VOC detection gap and completely closes the NYU depth gap relative to ImageNet pre-training.
Takeaways & Limitations
Multi-task self-supervision can substantially narrow or close the evaluated gap between self-supervised and fully supervised pre-training.
Takeaways & Limitations
Depth prediction was not very informative, and the authors suggest replacing it with a more reliable shape measure such as surface normal prediction.
Abstract
from arXiv · showhide
We investigate methods for combining multiple self-supervised tasks--i.e., supervised tasks where data can be collected without manual labeling--in order to train a single visual representation. First, we provide an apples-to-apples comparison of four different self-supervised tasks using the very deep ResNet-101 architecture. We then combine tasks to jointly train a network. We also explore lasso regularization to encourage the network to factorize the information in its representation, and methods for "harmonizing" network inputs in order to learn a more unified representation. We evaluate all methods on ImageNet classification, PASCAL VOC detection, and NYU depth prediction. Our results show that deeper networks work better, and that combining tasks--even via a naive multi-head architecture--always improves performance. Our best joint network nearly matches the PASCAL performance of a model pre-trained on ImageNet classification, and matches the ImageNet network on NYU depth prediction.
1. Introduction
The paper addresses the limits of self-supervised visual learning by fairly comparing diverse tasks and combining them into a shared representation. It targets conflicts between tasks and reports that multi-task self-supervision narrows the gap to supervised pre-training across evaluations.
- Motivation: Self-supervised learning uses automatically obtainable objectives, but existing methods remained weaker than ImageNet-pretrained networks for semantic vision tasks.The motivation is abundant unlabeled visual data alongside limited effectiveness of unsupervised training for tasks such as detection and geometry estimation.
- Contributions: The paper implements four self-supervised tasks and compares them on ImageNet classification, PASCAL VOC detection, and NYU v2 depth prediction.The tasks are relative position, colorization, exemplar, and motion segmentation.
- Multi-task learning: A shared multi-task trunk combines the tasks, despite different learning rates, to test whether multiple tasks outperform individual tasks.The paper also investigates which task combinations provide the largest boost.
- Task conflicts: Input channels and learning objectives can conflict, so the paper studies input harmonization and lasso-regularized feature combinations as targeted remedies.Colorization and exemplar learning use different input channels, while semantic categorization and instance matching favor different invariances.
- Experimental design: A common ResNet-101-v2 enables apples-to-apples comparison, while distributed training supports experiments across multiple tasks and large datasets.Prior studies often used different architectures, evaluation tasks, or fine-tuning procedures, limiting cross-paper comparison.
- Results: Combining multiple self-supervision tasks further closes the gap between self-supervised and fully supervised pre-training across all three evaluation measures.The reported conclusion concerns the combined representation rather than any single self-supervised task.
2. Self-Supervised Tasks
The study selects four conceptually simple but diverse self-supervised tasks and adapts them for a common multi-task setting. They derive supervision from spatial relations, color, image-instance transformations, or motion without manual labels.
- Task selection: The four selected tasks are relative position, colorization, exemplar, and motion segmentation, chosen to be simple and diverse.The authors note that they cannot evaluate every possible task combination and modify procedures where necessary for multi-task training.
- Relative Position: Relative position predicts one of eight neighboring spatial relationships between two randomly sampled patches from the same image.Color dropping is used because chromatic aberration can provide a low-level shortcut unrelated to semantic tasks.
- Colorization: Colorization predicts quantized Lab ab color categories from a grayscale L-channel image at lower spatial resolution.The implementation uses 313 categories for each 8-by-8 pixel region.
- Exemplar: Exemplar learning uses a triplet loss to bring together two augmented patches from one pseudo-class and separate a patch from another image.The margin is set to 0.5, avoiding a classification layer whose size would grow linearly with the dataset.
- Motion Segmentation: Motion segmentation predicts which pixels in a frame will move later, using masks extracted from dense tracking and an alternative Improved Dense Trajectories procedure.Pixels not labeled as camera motion are treated as foreground in the described procedure.
- Datasets: The image-based tasks use ImageNet, whereas motion segmentation uses the SoundNet dataset; the effect of alternative or larger datasets remains open.This dataset choice is explicitly identified as an open problem for improving performance.
3. Architectures
The architectures share a deep ResNet-101 trunk while assigning each task a separate head, then add lasso feature selection and input harmonization to address task-specific conflicts. Training is distributed across machines that compute task-specific gradients for the shared network.
- Common Trunk: The naive architecture uses a common ResNet-101-v2 trunk with one task-specific head per self-supervised objective.The trunk ends at block 3 and supplies a 1024-channel representation shared across tasks and evaluations.
- Common Trunk: Separate task losses are optimized through alternating active heads, with gradients averaged across iterations so the overall objective sums task losses.This enables a single shared trunk to receive updates from different self-supervised tasks.
- Separating features via Lasso: Lasso feature selection addresses potential conflicts because different tasks require different features, such as fine-grained information for colorization but less of it for detection.The architecture lets the network choose which layers provide each task head’s representation.
- Separating features via Lasso: The lasso architecture feeds each head a task-specific weighted sum of residual-unit activations and applies an L1 penalty to encourage sparse layer combinations.Sparse combinations are intended to concentrate information useful for each task into a small number of layers.
- Harmonizing network inputs: Input harmonization modifies preprocessing so tasks receive more similar low-level image statistics, reducing burdens on shared feature detectors.Relative position adopts colorization’s Lab preprocessing by retaining replicated L-channel input instead of random channel dropping.
- Distributed training: Distributed training assigns machines to individual tasks while sharing trunk parameters and synchronizing task gradients.The setup uses several machines per task and aggregates gradients with separate RMSProp optimizers.
4. Training the Network
The network is trained across distributed task-specific workers sharing a ResNet-101 trunk, using hybrid gradient aggregation and task-specific RMSProp optimizers. Training uses substantial parallel compute and periodic checkpoints for evaluation.
- Distributed architecture: Task-specific workers share the ResNet-101 trunk while retaining heads shared only among workers performing the same task.Each worker repeatedly computes losses and backpropagates gradients.
- Gradient aggregation: Hybrid aggregation applies each task’s accumulated gradients without synchronizing updates across different tasks.Gradients are accumulated among workers for one task, then applied when ready.
- Gradient aggregation: The hybrid approach learns faster than purely synchronous or asynchronous training and is more stable than asynchronous training.
- Optimization: RMSProp rescales gradients so loss-scale differences do not change learning speed, with separate moving averages maintained for each task.Task-specific optimizers scale each task’s accumulated updates before applying them to shared parameters.
- Compute: Experiments run on 64 GPUs, with checkpoints saved approximately every 2.4K NVIDIA K40 GPU hours for evaluation initialization.
5. Evaluation
The learned representations are evaluated through image classification, object detection, and pixel-wise depth prediction. These evaluations test raw features, object categories and locations, and geometric representation at pixel accuracy.
- Evaluation tasks: The evaluation tasks are image classification, object category detection, and pixel-wise depth prediction.
- Image classification: Frozen-weight ImageNet evaluation adds a linear softmax classifier after block 3 and measures performance on the validation set.Pre-trained weights remain fixed while the classifier is trained on the full ImageNet training set.
- Object detection: PASCAL VOC detection uses Faster-RCNN with heads for proposals, box classification, and localization, while fine-tuning all network weights.Strong performance requires representing both object categories and locations.
- Depth prediction: NYU V2 depth prediction measures geometric representation and pixel-level localization using modified up-projection layers and reverse Huber loss.The setup feeds block 3 outputs directly into three up-projection levels.
- Depth prediction: For lasso experiments, the lasso combination of block 3 layers is fed into the up-projection layers instead of block 3’s final output.
6. Results: Comparisons and Combinations
Using a common ResNet-101-v2 setup, the paper compares individual self-supervised tasks and finds that relative position and colorization are strongest across evaluations. Combining tasks further improves performance, while harmonization and lasso produce mixed results.
- Individual-task comparisons: The study uses a full ResNet-101 model as an ImageNet-trained comparison baseline across its evaluations.The baseline is also compared with prior results, though ImageNet pre-training performance is lower than a reported ResNet-101 result because block 4 is removed.
- Individual-task comparisons: Relative position leads on PASCAL and NYU, while colorization leads on ImageNet with frozen features.Relative position is on par with ImageNet pre-training for depth, and its PASCAL gap is 7.5% mAP.
- Training over time: After 16.8K GPU hours, performance is plateauing but not fully saturated, while exemplar training behaves unlike the other methods.Most algorithms improve slowly with training time, but exemplar performance falls on ImageNet and undulates on PASCAL and NYU.
- Individual-task comparisons: Exemplar and motion segmentation underperform the other self-supervised methods, with exemplar worst on PASCAL and NYU and motion segmentation worst on ImageNet.Exemplar performance also falls on ImageNet and varies across PASCAL and NYU checkpoints, possibly because its features remain sensitive to texture details.
- Multi-task combinations: Adding any other task to relative position improves ImageNet and PASCAL, with colorization or exemplar each producing more than 6 points of ImageNet gain.Adding both colorization and exemplar gives a further 2% boost, and the best-performing method combines all four self-supervised tasks.
- Multi-task combinations: The best multi-task representation reaches 43.91% mAP at 75% IoU versus 44.27% for the standard ImageNet model, while depth results remain on par with ImageNet pre-training.Combining relative position with exemplar or motion segmentation improves over those tasks alone, although those combinations slightly reduce depth performance relative to relative position.
7. Summary and extensions
The paper finds that deeper networks and combining self-supervision tasks improve representation learning, while harmonization and lasso weighting have minimal effects. It identifies several directions for extending the approach and suggests self-supervision may eventually augment or replace fully supervised pre-training.
- Deeper networks improve self-supervision over shallow networks.
- Combining self-supervision tasks always improves performance over individual tasks.
- Four-task self-supervision nearly closes the ImageNet pre-training gap on VOC detection and completely closes it on NYU depth.
- Harmonization and lasso weightings have only minimal effects.
- Future work includes broader augmentation, additional tasks and evaluation measures, and dynamic task weighting.
- Self-supervision may eventually augment or replace fully supervised pre-training.
A. Additional metrics for depth prediction
Additional depth metrics largely reproduce the main paper’s comparisons across self-supervised methods and combinations. However, the effects of harmonization and lasso on depth prediction remain uncertain.
- Depth prediction accuracy includes a strict relative-depth threshold of 1.25, with relaxed thresholds of 1.25^2 and 1.25^3 also available.
- Mean Absolute Error and Mean Relative Error provide alternative pixel-level error measures with different weighting of prediction errors.
- Additional depth measures generally tell the same story as the main reported depth measure.
- Exemplar and relative position perform best, while color and motion segmentation perform worse across the additional depth measures.
- Combinations perform as well as the best individual methods on the additional depth measures.
- The additional tables leave it uncertain whether harmonization or lasso improves depth prediction.