PaperlayerView exact revision on arXiv

Source-linked AI summary

Colorful Image Colorization

Richard Zhang, Phillip Isola, Alexei A. Efros

arXiv:1603.08511v5cs.CV

TL;DR

Automatic colorization must produce plausible, diverse colors despite the ambiguity of recovering color from grayscale images. This paper uses classification with class rebalancing and fools humans on 32% of trials while achieving state-of-the-art self-supervised feature-learning performance on several metrics.

  • Problem

    Grayscale colorization is underconstrained, while regression losses can average plausible colors into desaturated or implausible results.

  • Method

    The method frames colorization as rebalanced classification over color classes and uses it as a cross-channel self-supervised representation-learning task.

  • Results

    32% of trials fooled human participants, and the learned representation achieved state-of-the-art performance on several self-supervised learning metrics.

  • Takeaways & Limitations

    Colorization produces realistic graphics outputs and a representation useful for object classification, detection, and segmentation.

  • Takeaways & Limitations

    Natural-image color distributions are strongly biased toward low-ab, desaturated values, requiring explicit consideration of color-class rarity.

Abstract

from arXiv · show

Given a grayscale photograph as input, this paper attacks the problem of hallucinating a plausible color version of the photograph. This problem is clearly underconstrained, so previous approaches have either relied on significant user interaction or resulted in desaturated colorizations. We propose a fully automatic approach that produces vibrant and realistic colorizations. We embrace the underlying uncertainty of the problem by posing it as a classification task and use class-rebalancing at training time to increase the diversity of colors in the result. The system is implemented as a feed-forward pass in a CNN at test time and is trained on over a million color images. We evaluate our algorithm using a "colorization Turing test," asking human participants to choose between a generated and ground truth color image. Our method successfully fools humans on 32% of the trials, significantly higher than previous methods. Moreover, we show that colorization can be a powerful pretext task for self-supervised feature learning, acting as a cross-channel encoder. This approach results in state-of-the-art performance on several feature learning benchmarks.

1 Introduction

The paper presents automatic colorization as an inherently uncertain, multimodal prediction problem, using semantic cues and large-scale self-supervision to generate diverse colors. It evaluates perceptual realism with a colorization Turing test and studies colorization as a cross-channel encoder for downstream representation learning.

  • Problem: Grayscale colorization is underconstrained, but scene semantics and surface texture provide cues for plausible colors in many image regions.Examples include typically green grass, blue sky, and red ladybugs.
  • Method: The system predicts CIE Lab a and b channels from the lightness channel L, using color photographs as freely available supervisory data.Any color image supplies an L-channel input and corresponding ab-channel supervision.
  • Method: The method models multimodal color uncertainty by predicting distributions of possible pixel colors and reweighting training loss toward rare colors.This objective is designed to avoid the desaturated results common in earlier CNN-based approaches and to encourage greater color diversity.
  • Evaluation: The paper introduces a colorization Turing test that asks participants to distinguish real from synthesized colors, directly measuring perceptual realism.The authors report that the test can be difficult for participants and supports the realism of many generated colorizations.
  • Representation Learning: Colorization is also studied as self-supervised representation learning through a cross-channel encoder, with colorizations tested for usefulness in object classification.The approach uses raw data as its own supervision and evaluates representations with an off-the-shelf VGG network.

2 Approach

The approach predicts quantized color distributions from grayscale images using multinomial classification, addressing ambiguity and class imbalance through soft encoding and reweighted training. An annealed-mean operation converts predictions into vibrant yet spatially coherent color estimates.

  • Color prediction: The CNN maps grayscale inputs to a probability distribution over 313 in-gamut ab color bins quantized with grid size 10.The output distribution spans all pixels and the Q = 313 quantized ab values.
  • Color prediction: Euclidean loss can average distinct plausible colors, producing grayish, desaturated, or implausible results, so the method instead uses multinomial classification.The problem is inherently ambiguous and multimodal, making regression averages unsuitable.
  • Training objective: Ground-truth colors are soft-encoded using the 5 nearest quantized neighbors with a Gaussian kernel of σ = 5, then optimized with multinomial cross entropy.Soft encoding allows the network to learn relationships among elements in the output space.
  • Class rebalancing: Natural-image colors are heavily biased toward desaturated values, so training reweights pixels by color rarity to prevent the loss from being dominated by common classes.The empirical distribution was estimated from 1.3M ImageNet training images; λ = 1/2 and σ = 5 worked well for smoothing and rebalancing.
  • Point estimation: Temperature T = 0.38 produces a final prediction that preserves the mode’s vibrancy while maintaining the mean’s spatial coherence.The annealed-mean operation is applied independently per pixel as part of the CNN’s feed-forward pass.

3 Experiments

Experiments evaluate colorization realism, accuracy, grayscale classification, and self-supervised representation learning. The method fools humans on 32% of trials, improves grayscale classification without retraining, achieves state-of-the-art accuracy across three PASCAL VOC classification tests, and colorizes legacy photographs effectively.

  • Experimental setup: The network trains on 1.3M ImageNet images, validates on 10k validation images, and tests on a separate 10k-image validation set.The evaluation reports quantitative results on three metrics and qualitative success and failure cases.
  • Colorization realism: 32% of trials fooled participants, significantly outperforming every compared algorithm except Larsson et al., for which the difference was not significant.The comparison used p < 0.05 for each significant case and p = 0.10 against Larsson et al.
  • Colorization accuracy: 68.3% fell to 52.7% after ablating input colors, then rose to 56.0% after recolorization, while Larsson et al. reached 59.4%.This demonstrates that colorizing grayscale images can improve an off-the-shelf classifier without additional training or fine-tuning.
  • Colorization accuracy: The full method outperforms all variants and compared algorithms on the class-balanced AuC metric, indicating that class rebalancing achieved its intended effect.The metric re-weights pixels inversely by color-class probability to emphasize saturated, perceptually interesting regions.
  • Representation learning: The cross-channel encoder achieves state-of-the-art accuracy across all three PASCAL VOC 2007 classification tests and competitive self-supervised performance from conv2 onward.Its conv1 representation initially trails competing methods, but the gap is immediately bridged at conv2.
  • Representation learning: 47.9% detection with color inputs exceeds the k-means baseline of 45.6%, while 35.6% segmentation with color exceeds other tested algorithms.Detection remains below ImageNet semantic supervision at 56.8%, and grayscale segmentation reaches 35.0%.

4 Conclusion

The paper presents image colorization as a difficult pixel-prediction problem and shows that a deep CNN with a well-chosen objective can approach photorealistic results. The method also provides graphics output and serves as a pretext task for representation learning.

  • Image colorization is framed as a difficult pixel prediction problem in computer vision.
  • A deep CNN with a well-chosen objective function produces colorizations closer to being indistinguishable from real color photographs.
  • The method provides useful graphics output while also functioning as a pretext task for representation learning.

Appendix

The Appendix reports analyses from the original arXiv v1 model that were omitted from the ECCV 2016 camera-ready paper, while noting that the main-paper networks were retrained as v2 models. It covers representation learning, semantic interpretability, low-level cues, multimodality, architecture, and comparisons with prior approaches.

  • Appendix: The Appendix contains analyses generated from a previous v1 model because space constraints prevented their inclusion in the ECCV 2016 camera-ready paper.All main-paper networks were retrained from scratch and referred to as v2; all models are publicly available.
  • Appendix: Section 5 presents additional representation learning experiments, while Section 6 analyzes the VGG semantic interpretability test.
  • Appendix: Sections 7 and 8 examine how low-level cues affect outputs and investigate multimodality learned in the network.
  • Appendix: Sections 9 and 10 define the network architecture and compare the algorithm with previous approaches, including additional legacy grayscale-image examples.The cited previous approaches are and.

5 Cross-Channel Encoding as Self-Supervised Feature Learning (continued)

The paper evaluates colorization representations using nonlinear classifiers trained on frozen internal layers, with performance summarized in Table 3. The comparison also accounts for architectural differences among models.

  • Cross-Channel Encoding as Self-Supervised Feature Learning: Nonlinear classifiers are trained by freezing each internal layer and all preceding layers, then randomly reinitializing and training the layers above for classification.This extends the representation-learning evaluation beyond linear classifiers on ImageNet classifiers.
  • Cross-Channel Encoding as Self-Supervised Feature Learning: Table 3 compares ImageNet classification with nonlinear layers while reporting parameter count, features per image, and runtime relative to unmodified AlexNet through pool5.These measurements characterize architectural differences between the evaluated models.
  • Cross-Channel Encoding as Self-Supervised Feature Learning: Noroozi et al. performs best on all layers, with denser feature maps resulting from a smaller conv1 stride and modified LRN and pooling order.The table attributes these differences to architectural modifications.

6 Semantic Interpretability of Colorizations

The paper evaluates semantic interpretability by measuring VGG classification performance after colorization and analyzing categories that improve or deteriorate relative to grayscale. It also examines recurrent recolorization-induced confusions, including errors caused by plausible but incorrect colors.

  • 6.1 Categories with improved or degraded performance: Colorization improves VGG classification most for some color-diagnostic categories, including green snake, orange, and goldfinch.The analysis compares VGG object-classification accuracy on colorized images with accuracy on grayscale images.
  • 6.1 Categories with improved or degraded performance: Colorization hurts performance for errors such as clothing assigned incorrect or inconsistent colors and animals given plausible but incorrect colors.The affected categories are those for which color information is highly diagnostic, according to Figure 9.
  • 6.2 Commonly confused categories: Recolorization commonly causes minibus images colored yellow to be misclassified as school buses and animals colored differently from ground truth to shift toward related species.These examples illustrate classification confusions that occur after recolorization but not with the original ground-truth image.
  • 6.2 Commonly confused categories: Added top-5 confusion is quantified as A = Crecolor − Corig by subtracting confusion rates with ground-truth colors from those after recolorization.The confusion rates are defined over category pairs, and off-diagonal entries are sorted to identify newly added confusions.

7 Is the network exploiting low-level cues?

The experiments indicate that colorization is not explained by simple pixel-lightness cues alone: the network can recover colors from recognizable, roughly isoluminant vegetables but fails on a grayscale Macbeth chart. Predictions are relatively stable to lightness and contrast changes, while blurring has a larger effect, possibly because it removes diagnostic zucchini texture.

  • Low-level cue hypothesis: The network failed to recover Macbeth chart colors despite considerable variation in patch lightness, but recovered colors for recognizable, roughly isoluminant vegetables.This tests whether prediction relies on a simple low-level lightness–color relationship.
  • Robustness to image changes: Predictions were somewhat stable to low-level lightness and contrast changes.
  • Robustness to image changes: Blurring had a larger effect on predictions, possibly because it removed the zucchini’s diagnostic texture pattern.

8 Does our model learn multimodal color distributions?

The model formulates color prediction as multinomial classification to capture the inherent ambiguity and multimodal distributions of natural-object colors. Figure 13 demonstrates that the network predicts distinct probability distributions for background vegetation and a foreground bird.

  • Multimodal color distributions: Multinomial classification enables the system to predict multimodal color distributions and represent ambiguity in natural-object colors.This formulation treats color prediction as a classification problem over output color bins.
  • Multimodal color distributions: The network outputs probability maps bZ_q ∈ [0, 1]^(H,W) for color bin q, subsampled and coarsely quantized for visualization.The figure displays only a subset of the Q total output bins.
  • Multimodal color distributions: Figure 13(a) shows different predicted distributions for background vegetation and the foreground bird.The system output bY is visualized in the top-left of Figure 13.

9 Network architecture

The network uses a convolutional architecture whose testing pipeline incorporates temperature adjustment, softmax, averaging, and bilinear upsampling as feed-forward layers. Its effective convolutional dilation increases from conv1 through conv5, then decreases from conv6 through conv8.

  • Network architecture: During testing, temperature adjustment, softmax, mean, and bilinear upsampling are implemented as subsequent layers in a feed-forward network.The architecture’s training-time layers are listed in Table 4.
  • Network architecture: Effective dilation increases through convolutional blocks conv1–conv5 and decreases from conv6–conv8.Effective dilation is computed as the product of accumulated stride and layer dilation, representing the spacing between evaluated kernel elements relative to input pixels.

10 Colorization comparisons on held-out datasets

The model generalizes effectively to held-out SUN scene photographs despite differences in grayscale space and training data, outperforming the LEARCH-based method on accuracy and perceptual realism. Compared with Deep Colorization, it achieves similar qualitative performance while offering a simpler, faster, and broader evaluation setup.

  • SUN dataset comparison: Despite training on object-centric ImageNet, the model remains effective on scene-centric SUN photographs and outperforms Deshpande et al. on raw accuracy and perceptual realism.The comparison uses the same grayscale input for fairness, although the input space differs nonlinearly from the training L channel.
  • Deep Colorization comparison: On 23 test images from Deep Colorization, the results are qualitatively about the same.The images were manually cropped from the original paper’s test set.
  • Deep Colorization comparison: Deep Colorization’s comparison setting favors it because its images come from an unseen SUN dataset and were hand-selected from 1344 images.The authors could not obtain results on the full 1344-image test set.
  • Deep Colorization comparison: Compared with Deep Colorization, the method is faster, easier to train and understand, uses fewer hand-tuned components, and covers more diverse test images.The comparison spans algorithm pipeline, learning, dataset, and run-time.
Loading 1603.08511v5…