Source-linked AI summary
Colorization as a Proxy Task for Visual Understanding
Gustav Larsson, Michael Maire, Gregory Shakhnarovich
TL;DR
The paper asks whether self-supervised colorization can replace ImageNet pretraining without labels and studies how its formulation and training choices affect representations. It reports state-of-the-art downstream results without ImageNet labels and finds colorization on par with several annotated-data pretraining methods.
Problem
Human annotation limits scaling to new and expensive domains despite abundant unlabeled visual data, motivating alternatives to label-dependent pretraining.
Method
The paper develops colorization self-supervision and empirically studies its loss, architecture, training details, and relation to ImageNet pretraining.
Results
Colorization achieves state-of-the-art semantic segmentation and small-sample classification results without ImageNet labels and is on par with several annotated-data pretraining methods.
Takeaways & Limitations
Colorization provides a similarly powerful supervisory signal to several forms of ImageNet pretraining within the evaluated downstream-task settings.
Takeaways & Limitations
Training ResNet from scratch with a hypercolumn is an outlier experiment that may combine disruptive choices, as suggested by its low results.
Abstract
from arXiv · showhide
We investigate and improve self-supervision as a drop-in replacement for ImageNet pretraining, focusing on automatic colorization as the proxy task. Self-supervised training has been shown to be more promising for utilizing unlabeled data than other, traditional unsupervised learning methods. We build on this success and evaluate the ability of our self-supervised network in several contexts. On VOC segmentation and classification tasks, we present results that are state-of-the-art among methods not using ImageNet labels for pretraining representations. Moreover, we present the first in-depth analysis of self-supervision via colorization, concluding that formulation of the loss, training details and network architecture play important roles in its effectiveness. This investigation is further expanded by revisiting the ImageNet pretraining paradigm, asking questions such as: How much training data is needed? How many labels are needed? How much do features change when fine-tuned? We relate these questions back to self-supervision by showing that colorization provides a similarly powerful supervisory signal as various flavors of ImageNet pretraining.
1. Introduction
The paper targets the limits of label-dependent pretraining by improving self-supervision through colorization, using unlabeled data while evaluating representation quality on downstream vision tasks.
- Human annotation limits scaling to new and expensive domains, whereas unlabeled visual data is abundant and essentially free.
- Self-supervision applies a discriminative loss to unlabeled data by partitioning each input and predicting the association between its parts.
- Colorization splits each image into intensity and color components, using intensity to predict color.
- The paper reports state-of-the-art VOC 2007 classification and VOC 2012 segmentation results among methods that do not use ImageNet labels.
- Its colorization analysis shows that loss, network architecture, and training details importantly influence results.
- The study empirically compares ImageNet pretraining formulations with self-supervision.
2. Related work
Related work spans initialization, unsupervised and self-supervised learning, and temporal or spatial proxy tasks. Colorization differs by predicting color at the same pixel and is evaluated as a representation-learning signal for segmentation.
- Unsupervised pretraining can facilitate optimization or expose networks to much larger quantities of unlabeled data.
- Self-supervised feature learning methods broadly exploit temporal or spatial structure in natural visual data.
- Temporal: Temporal methods use relationships between adjacent video frames, including prediction, similarity, ordering, and optical-flow-based signals.
- Spatial: Spatial methods divide single images into patches or remove pixels and train representations by predicting spatial arrangements or missing content.
- Colorization: Colorization predicts color over the same input pixel without discarding spatial information, making it potentially suited to semantic segmentation.
- Colorization: Colorization improved prior PASCAL VOC segmentation results from 50.2% to 60.0% mIU, while earlier work provided little analysis.
- Feature reuse/repurpose: Fine-tuning can reuse, specialize, or replace colorization features for semantic segmentation, with quantitative feature-shift analysis provided separately.
3. Colorization as the target task
The paper trains colorization networks for representation learning rather than image aesthetics, comparing regression and histogram losses while using large-scale unlabeled data and architectural choices such as hypercolumns.
- The colorization setup is adapted from prior work, with Caffe used for colorization training and TensorFlow used for downstream tasks.
- Loss: The study compares L*a*b regression with KL-divergence prediction of hue/chroma histograms, evaluating their representation-learning ability rather than image colorization quality.
- Hypercolumn: Hypercolumns are computed sparsely to reduce memory requirements and enable training on larger images.
- Dataset: The pretraining corpus contains 3.7M unlabeled images from ImageNet and Places205.
- Training: Training uses momentum SGD, Xavier initialization, batch normalization, random mirroring and scaling, and 352-by-352 desaturated patches.
4. Colorization as a proxy task
The paper uses colorization pretraining to initialize downstream classification and segmentation models, while examining architectural and training choices that affect transfer. It also addresses practical design issues including receptive field, normalization, padding, and input color handling.
- Colorization representations initialize downstream classification and segmentation models.
- Receptive field: Additional receptive-field blocks expand the field of view by 160 pixels per block for segmentation.
- Hypercolumn: Hypercolumn use during segmentation is independent of hypercolumn use during colorization pretraining, and post-hypercolumn weights are not reused.
- Batch normalization: Downstream training absorbs batch-normalization statistics into weights and biases, except for ResNet where batch normalization helps in their experience.
- Padding: Padding correction makes activations well behaved but does not measurably improve downstream tasks.
- Self-supervision loss: The loss choice significantly affects downstream performance, whereas hypercolumn pretraining does not appear beneficial.
5. Results
The paper evaluates colorization-based representations on established PASCAL VOC classification and segmentation benchmarks. It reports leading results among methods without ImageNet-label pretraining, including 60.0% mIU for segmentation and 77.3% mAP for classification.
- VOC 2012 Semantic Segmentation: 60.0% mIU is achieved on VOC 2012 semantic segmentation without supervised pretraining.This exceeds the reported AlexNet-based FCN result of 48.0% and the VGG-16-based FCN result of 59.4%, both trained on ImageNet.
- VOC 2007 Classification: 77.3% mAP is achieved on VOC 2007 classification, attaining state-of-the-art performance when no ImageNet labels are used.
6. Experiments
Experiments show that colorization pretraining benefits from careful choices of loss, architecture, data supervision, and training schedule. Across VOC and ImageNet-pretraining comparisons, the learned representations remain useful for downstream tasks and perform competitively with label-based alternatives.
- Loss: 52.9% versus 48.0% shows that hue/chroma histogram prediction outperforms L*a*b regression for representation learning.The authors attribute the gap to the loss formulation and color uncertainty handling.
- Network architecture: Colorization pretraining improves more with larger networks: gains rise from +9.1 to +20.6 percentage points across AlexNet, VGG-16, and ResNet-152.The corresponding scratch results are 39.1%, 43.2%, and 42.5%, respectively.
- ImageNet pretraining: Colorization performs similarly to ImNt-100k pretraining for VGG-16, 56.0% versus 57.1%, corresponding to roughly 3.7 million unlabeled samples replacing 0.1 million labeled samples.Reducing the label space to two classes performs similarly to using 1/100th of the training data, and richer labels appear important for representation learning.
- ImageNet pretraining: 50% label noise causes a 2.6-point downstream penalty, while 10% noise has little impact and still yields a competitive representation.The comparison evaluates modified ImageNet supervision for VOC 2012 segmentation.
- Training time and learning rate: Reducing the learning rate during pretraining is crucial, and extending training from 10 epochs to 37M samples improves VOC 2012 segmentation from 52.9% to 56.0% mIU.A ResNet-152 model reaches 60.0% mIU after four months on a single GPU.
- Feature analysis: Colorization learns both color-specific and semantic features, but fine-tuning changes these features more than label-based features.Colorization features alone perform similarly to end-to-end training from random initialization, although fine-tuning is more important than with supervised pretraining.
7. Conclusion
The paper presents self-supervised colorization as a competitive replacement for ImageNet pretraining and identifies key factors governing its effectiveness.
- Self-supervised colorization achieves state-of-the-art semantic segmentation and small-sample classification without ImageNet labels.
- The analysis shows that loss formulation, network architecture, and training details are important for achieving competitive results.
- Colorization provides a supervisory signal on par with several methods using annotated data.
A. Document changelog
The document changelog records the camera-ready release, updated VOC results, added experiments, renamed configurations, and an additional reference.
- The v2 release updated ResNet-152 results after additional VOC training.VOC 2007 Classification changed from 76.9 to 77.3, and VOC 2012 Segmentation from 59.0 to 60.0.
- The revision added label noise experiments and renamed C2/C16 to H2/H16.
- The v2 release added references to concurrent work, and v3 added an overlooked reference.