Source-linked AI summary
Learning Representations for Automatic Colorization
Gustav Larsson, Michael Maire, Gregory Shakhnarovich
TL;DR
The paper addresses the challenge of automatically assigning plausible colors to grayscale images while capturing both semantic structure and multimodal color uncertainty. It uses an end-to-end deep architecture with per-pixel color-histogram prediction and evaluates it in fully and partially automatic settings. The method achieves state-of-the-art performance across those tasks and datasets, while also supporting a colorization benchmark and self-supervised representation-learning exploration.
Problem
Fully automatic colorization requires visual understanding of scene composition and object locations, while many objects admit multiple plausible colors.
Method
An end-to-end deep convolutional system uses semantically meaningful multilayer features and predicts per-pixel color histograms for direct or reference-biased image formation.
Results
The method achieves state-of-the-art performance across fully and partially automatic colorization tasks and all tested datasets, with 0.293 RMSE and 24.94 dB PSNR on ImageNet test.
Takeaways & Limitations
The work provides a new large-scale ImageNet colorization benchmark and a proof of concept for learning visual representations through colorization.
Takeaways & Limitations
The method has significant failures associated with gaps in semantic interpretation, incorrect segmentation, unfamiliar objects, and naturally uncertain colors.
Abstract
from arXiv · showhide
We develop a fully automatic image colorization system. Our approach leverages recent advances in deep networks, exploiting both low-level and semantic representations. As many scene elements naturally appear according to multimodal color distributions, we train our model to predict per-pixel color histograms. This intermediate output can be used to automatically generate a color image, or further manipulated prior to image formation. On both fully and partially automatic colorization tasks, we outperform existing methods. We also explore colorization as a vehicle for self-supervised visual representation learning.
1 Introduction
The paper presents fully automatic colorization as both a challenging visual-understanding problem and a proxy for visual understanding. Its system combines deep semantic and localized representations with per-pixel color-distribution prediction, and is evaluated across automatic settings.
- Motivation: Fully automatic colorization remains challenging because successful color assignment requires high-level scene comprehension.The paper motivates colorization through the need to interpret what objects are present and where they are located.
- Motivation: The paper treats automatic colorization as a proxy measure for visual understanding and connects it to deep architectures used in classification and detection.This makes colorization relevant beyond graphics applications.
- Approach: The system processes grayscale images with a deep convolutional architecture and spatially localized multilayer hypercolumns as per-pixel descriptors.The model is trained end-to-end to predict hue and chroma distributions from these descriptors.
- Approach: The model predicts a color histogram at every image location rather than a single color, accommodating objects with multiple suitable colors.Predicted distributions determine color assignment at test time.
- Evaluation: The paper evaluates fully automatic and partially automatic colorization and reports best performance across every metric and dataset.The fully automatic setting uses grayscale input only, while the partially automatic setting also provides a reference global color histogram.
- Contributions: The work introduces an ImageNet colorization benchmark and explores colorization as an objective for learning visual representations from scratch.This experiment considers replacing ImageNet pretraining in a traditional semantic-labeling task.
2 Related work
Prior colorization methods include scribble-based, transfer-based, and automatic direct-prediction approaches. The paper distinguishes its method through fully automatic, end-to-end learning with histogram-based color prediction and reports superior performance across evaluated regimes.
- Prior methods: Scribble-based methods require users to specify desired colors in selected regions before propagating them under luminance-based assumptions.Their optimization relies on Normalized Cuts.
- Prior methods: Transfer-based methods obtain color from related reference images, but reference selection is at least partially manual in most methods.They establish source-target mappings using local-descriptor correspondences, sometimes combined with manual intervention.
- Automatic methods: Earlier fully automatic methods include linear-system inference and fixed-feature neural regression, with reported limitations in inference or feature optimization.Deshpande et al. use iterative inference, while Cheng et al. optimize only the final component over fixed representations.
- Proposed approach: The proposed system is trained end-to-end, avoids hand-crafted features, and formulates color prediction as histogram estimation rather than regression.The paper presents these choices as distinctions from earlier automatic methods.
- Concurrent work: A concurrent network combines separate global and local paths, whereas this work uses a 16-layer hypercolumn spanning low- and high-level features.The compared system jointly trains classification and colorization, while this work permits colorization fine-tuning on unlabeled datasets.
- Concurrent work: Another concurrent method also predicts color histograms, while this work uses a fully convolutional architecture with implicit deep supervision and spatially sparse, memory-efficient training.The cited differences include up-convolutional layers, explicit deep supervision, and dense training in the concurrent method.
3 Method
The method combines fully convolutional deep features with color-space and loss designs that represent multimodal per-pixel color distributions. It supports automatic inference and optional histogram transfer from a reference color distribution.
- Problem formulation: The colorizer learns a function that maps a grayscale image patch to the color of its center pixel, with the patch defining the receptive field.The network implements this image-to-image prediction function using a convolutional architecture.
- Network representation: Deep convolutional networks and hypercolumns connect semantic and low-level image features for per-pixel color prediction.The architecture uses localized slices from multiple layers as per-pixel descriptors, following image-to-image prediction designs.
- Color spaces: The method evaluates multiple color representations, including hue/chroma and Lab or αβ, while retaining lightness as a separate channel where possible.For hue/chroma, chroma-dependent weighting compensates for hue instability, and the color bicone avoids instabilities associated with hue-based cylinders.
- Color distributions: The system predicts color histograms over quantized bins rather than single colors, using KL divergence and a softmax output to model multimodal distributions.Ground-truth histograms are empirical distributions in a neighborhood around the center pixel; with R = 1, they become one-hot vectors and the loss becomes log loss.
- Inference: At inference, histogram predictions can be converted to colors by sampling, mode, median, or expectation; Lab performs best qualitatively and quantitatively with expectations.For hue/chroma, chroma uses the median, while circular hue uses a complex expectation followed by remapping and optional chromatic fading.
- Histogram transfer: For partially automatic colorization, the method transfers a target histogram using lightness-normalized quantile matching or energy minimization.Quantile matching adjusts source color-channel histograms to target histograms, while energy minimization balances prediction fidelity against histogram proximity.
4 Experiments
Experiments show strong colorization performance across ImageNet and SUN benchmarks, including fully automatic settings, while also demonstrating self-supervised representation learning and interactive color control.
- ImageNet evaluation: 0.293 RMSE and 24.94 dB PSNR on ImageNet/c test10k improve over the 0.333 RMSE and 23.27 dB PSNR baseline.The selected hue/chroma model was trained for 10 epochs and uses chromatic fading during image generation.
- Ablations: Higher hypercolumn layers alone are insufficient for good colorization, although some lower layers can be removed without much performance loss.Table 2 evaluates systems using different subsets of layers for hue and chroma prediction.
- SUN evaluation: On SUN-6, quantile matching outperforms the prior method, while the proposed energy minimization procedure further improves partially automatic results.The comparison uses ground-truth histograms for the partially automatic setting.
- SUN evaluation: The fully automatic method dominates competing approaches on SUN-6 and SUN-A error distributions, while ground-truth histograms improve its own results further.On SUN-A, the method shows clear separation from in per-image PSNR.
- Representation learning: Colorization trained from scratch on unlabeled images generalizes to other visual tasks, more than halfway bridges supervised-pretraining performance, and yields an 18-point mIU improvement over baseline.The approach converges more slowly but requires no more than twice as many epochs; 50.2% is reported without additional annotated data.
5 Conclusion
The system combines a deep neural architecture with per-pixel color-histogram prediction to colorize grayscale images, handle ambiguity, and support creative post-processing. It achieves strong automatic-colorization results and motivates colorization for self-supervised visual learning.
- The system combines semantically meaningful features of varying complexity with color-histogram prediction in an end-to-end architecture.The architecture is designed to handle uncertainty and ambiguity while preventing jarring artifacts.
- The color-histogram framework supports creative control through histogram transfer and uncertainty-driven color sampling.
- The fully automatic colorizer improves upon previously leading methods by large margins on all tested datasets.
- The work proposes a large-scale benchmark and establishes a baseline for future automatic-colorization comparisons.
- The results also identify colorization as a promising avenue for self-supervised visual learning.
A.2 Color space αβ
This subsection specifies the color-space constraints used for colorization and illustrates predicted hue/chroma histograms. It also records how a prior method computed a related quantity.
- A.2 Color space αβ: The color-space variables satisfy ϵ = 0.0001, R, G, B ∈[0, 1], and L = R+G+B.
- A.2 Color space αβ: Deshpande et al.’s calculation is identified from their released code.
- A.2 Color space αβ: Figure 11 shows examples of predicted hue/chroma histograms.
A.3 Error metrics
The error metrics are computed across multiple images and pixels, with a final correction ensuring predicted color channels remain consistent with the input lightness.
- A.3 Error metrics: For M images, each image m with N_m pixels, the error metrics are calculated across the image set.
- A.3 Error metrics: A final L−L̂ correction is added to every predicted RGB color channel when predicted and input lightness differ.This compensates for subtle differences in how lightness is defined.
B.1 Validation
The validation subsection compares histogram-inference methods for hue and chroma using image-quality metrics. The table notes that mode/mode performs fairly well but introduces severe visual artifacts.
- B.1 Validation: Table 7 compares hue/chroma histogram-inference methods on ImageNet/cval1k.
- B.1 Validation: Mode/mode performs fairly well but produces severe visual artifacts.
- B.1 Validation: The reported evaluation metrics include CF, RMSE, and PSNR for hue/chroma inference.CF denotes chromatic fading.
B.2 Examples
The paper provides additional qualitative examples spanning global biasing, benchmark comparisons, historical photographs, ImageNet results, failure cases, and histogram predictions.
- Additional samples illustrate global biasing and SUN-6 results.
- Figures 14 and 15 compare the method with Charpiat et al.
- Further examples cover old-photograph colorization, ImageNet ctest10k results, failure cases, and histogram predictions.
C Document changelog
The document revisions include a camera-ready ECCV 2016 version, discussion of concurrent work, new representation-learning experiments, and an added reference, alongside extensive qualitative figures.
- Document revisions: The v2 ECCV 2016 camera-ready version includes discussion about concurrent work.
- Document revisions: New experiments use colorization to learn visual representations in Section 4.1.
- Document revisions: The v3 revision added an overlooked reference.
- Additional figures: Additional figures show multiple sampled colorizations, SUN-6 comparisons, transfer comparisons, portraits, historical photographs, ImageNet results, and failure cases.
- Additional figures: Figures 18–20 provide further fully automatic ImageNet/ctest10k results, while Figure 21 presents failure cases.