Source-linked AI summary
PixColor: Pixel Recursive Colorization
Sergio Guadarrama, Ryan Dahl, David Bieber, Mohammad Norouzi, Jonathon Shlens, Kevin Murphy
TL;DR
Colorizing grayscale images is ambiguous because many color images can correspond to the same grayscale image. PixColor samples low-resolution color with a conditional PixelCNN and refines it with the grayscale input; the paper reports better average performance than published methods in a crowdsourced human evaluation.
Problem
Grayscale-to-color restoration is one-to-many because removing chromaticity leaves multiple possible color images for the same input.
Source: §1 · p. 1Method
PixColor samples a low-resolution color image conditioned on grayscale with PixelCNN, then a refinement CNN combines it with the high-resolution grayscale input.
Source: §1 · p. 2, §3.2 · p. 6Results
PixColor significantly outperforms previous state-of-the-art methods, with an average VTT score of 33.9%.
Source: §5 · p. 6Takeaways & Limitations
PixColor outputs more closely match marginal color distributions than other methods, while exhibiting failure modes left for future work.
Source: §6 · pp. 7–8PixelCNN likelihood scores did not correlate well with human judgments when used to select the best sample; training a separate ranking model is left to future work.
Source: §5.1 · p. 7
Abstract
from arXiv · showhide
We propose a novel approach to automatically produce multiple colorized versions of a grayscale image. Our method results from the observation that the task of automated colorization is relatively easy given a low-resolution version of the color image. We first train a conditional PixelCNN to generate a low resolution color for a given grayscale image. Then, given the generated low-resolution color image and the original grayscale image as inputs, we train a second CNN to generate a high-resolution colorization of an image. We demonstrate that our approach produces more diverse and plausible colorizations than existing methods, as judged by human raters in a "Visual Turing Test".
1 INTRODUCTION
Colorizing grayscale images is one-to-many, so a useful system should produce multiple plausible outputs rather than a single prediction. PixColor uses a PixelCNN to sample low-resolution color and a refinement CNN to combine it with the grayscale image, and the paper reports outperforming existing methods.
1 INTRODUCTION
- Grayscale-to-color restoration is one-to-many because removing chromaticity loses information, making a conditional distribution appropriate for multiple possible outputs.The paper frames this ambiguity as p(y | x).Source: §1 · p. 1
- A PixelCNN models a coherent joint distribution over color images given grayscale input, capturing pixel dependencies with exact log-likelihood and stable training.The paper notes that PixelCNN sampling is slow because of its sequential structure.Source: §1 · p. 1
- PixColor samples a low-dimensional, low-resolution color image with PixelCNN, then uses a refinement CNN to combine that sample with the high-resolution grayscale input.The paper describes the low-resolution color as a latent variable and the refinement CNN as mapping it and the grayscale image to the output.Source: §1 · p. 2
- PixColor produces diverse, high-quality colorizations and significantly outperforms existing methods, according to the paper's reported results.Figure 2 shows three model samples for each grayscale input.Source: §1 · p. 2, Figure 2 · p. 2
2 RELATED WORK
Prior automated colorization methods include single-output CNNs, approaches for generating multiple outputs, and generative models, each with reported limitations. PixColor instead samples low-resolution color with PixelCNN and refines it with a CNN; the paper contrasts this pipeline with a concurrent approach that uses the networks in the opposite order.
2 RELATED WORK
- Most recent CNN methods predict a single color image; L1 or L2 training can yield washed-out averages, while independent per-pixel predictions fail to enforce consistent regional colors.Discretizing color and upweighting rare colors can produce more colorful pictures, but does not by itself capture the one-to-many structure.Source: §2 · p. 2
- Earlier diversity approaches include CRFs, mixtures of experts, VAEs with latent codes, and GANs, with reported issues including slow inference, sepia outputs, mode collapse, or limited evaluation datasets.The CRF and GAN discussion also notes that methods capable of multiple solutions may nevertheless report only one output or ignore noise.Source: §2 · pp. 2–3, Table 1 · p. 3
- PixColor uses PixelCNN to generate multiple low-resolution color images and a CNN refinement network to convert them deterministically into high-resolution color images.The paper says the multiple low-resolution color hints capture the task's one-to-many nature and prevent sepia-toned outputs.Source: §2 · p. 3
- A concurrent method reverses PixColor's network order, passing CNN output into PixelCNN; it generates 32×32 color images and deterministically upscales them, unlike PixColor's learned refinement from 28×28.The paper says the concurrent work did not perform human evaluation, so it provides no quantitative comparison.Source: §2 · p. 3
- Figure 3 shows the original color image above true chroma downsampled to a smallest size of 28 pixels, then bilinearly upsampled and combined with the original grayscale image.Source: Figure 3 · p. 4
3 PIXEL RECURSIVE COLORIZATION (PIXCOLOR)
PixColor uses low-resolution color hints, which can retain plausible chrominance because color varies more slowly across space than intensity, then combines them with the original grayscale image for high-resolution refinement. Its PixelCNN generates the hints, and a second CNN refines them.
3 PIXEL RECURSIVE COLORIZATION (PIXCOLOR)
- PixColor samples low-resolution color images and deterministically converts each into a high-resolution colorization with a CNN refinement network.The approach relies on chrominance having lower spatial frequency than luminance; downsampled and then upsampled ground-truth chroma can still look close to the original.Source: §3 · p. 4
3.2 FEEDFORWARD CNN FOR HIGH-RESOLUTION REFINEMENT
- 19.9% to 33.9% VTT score: the refinement network raises the generated-sample score, while GT-Refined reaches 43.6% against a 50% maximum.Figure 5 compares ground-truth and generated chroma with and without refinement.Source: Figure 5 · p. 5, §3.2 · p. 6
3.1 PIXELCNN FOR LOW-RESOLUTION COLORIZATION
- The conditional PixelCNN generates multiple 28×28 color images sequentially, conditioning each pixel on the grayscale input and previously generated pixels.It uses YCbCr with 32 discrete bins for each chroma channel and per-pixel cross-entropy training.Source: §3.1 · p. 5
- The PixelCNN models Cr before Cb at each pixel, with Cb conditioned on Cr and both channels conditioned on earlier pixels and the grayscale input.Preliminary logistic-mixture outputs did not meaningfully improve on the simpler multinomial model.Source: §3.1 · p. 5
3.2 FEEDFORWARD CNN FOR HIGH-RESOLUTION REFINEMENT
- Low-resolution chroma hints are combined with high-resolution grayscale so the refinement network can use color cues alongside grayscale edge boundaries.It is trained on 28×28 downsampled ground-truth chroma, rather than end-to-end with PixelCNN, to discourage ignoring potentially irrelevant generated hints.Source: §3.2 · p. 6
4 EVALUATION METHODOLOGY
The evaluation uses human judgments because grayscale-to-color mapping allows multiple plausible outputs, making direct comparison with one ground-truth image unsuitable. Raters choose which of two displayed color versions has the “true colors.”
4 EVALUATION METHODOLOGY
- The Visual Turing Test presents ground truth and generated colorizations and asks raters to choose the image with “true colors”; a method always producing ground truth scores 50%.The paper calls the fraction of times the generated image is preferred the VTT score.Source: §4 · p. 6
- Each image is shown sequentially for one second each to five raters, and the VTT score is the fraction preferring the generated image to ground truth.The evaluation uses 500 images from the ILSVRC-CLS validation split; training uses 1.2M ILSVRC-CLS images.Source: §4 · p. 6
5 RESULTS
On ImageNet, PixColor significantly outperforms previous state-of-the-art methods in the Visual Turing Test (VTT), while matching empirical color distributions more closely. Its samples are mostly similar but vary in details, and human selection of the best sample yields a higher VTT score; PixelCNN likelihood did not reliably identify human-preferred samples.
5 RESULTS
- PixColor significantly outperforms previous state-of-the-art methods on ImageNet, with an average VTT score of 33.9%.Figure 6 reports the VTT study on 500 test images, with five raters per image.Source: §5 · p. 6, Figure 6 · p. 7
- PixColor matches the empirical Lab color distribution more closely than other methods, without explicitly reweighting color bins as prior work did.Figure 7 presents method histograms against the ImageNet test-set histogram and histogram intersections.Source: §5 · pp. 6–7, Figure 7 · p. 7
5.1 SAMPLE DIVERSITY
- Most PixColor sample pairs have multiscale SSIM scores of 0.95–0.99, differing in subtle details, while the lowest scores correspond to large objects receiving different colors.Figure 8 shows the SSIM-distance distribution and representative pairs.Source: §5.1 · p. 7
- Selecting the most-liked of three samples raises the VTT score to 38.3%, suggesting algorithmic sample ranking could improve results.The selected samples were evaluated by a different set of raters.Source: §5.1 · p. 7
- PixelCNN likelihood scores did not correlate well with human judgments in preliminary sample-selection experiments; a separate ranking model is left to future work.Source: §5.1 · p. 7
6 CONCLUSION
PixColor produces diverse colorizations that perform better on average than other published methods in a crowdsourced human evaluation. It samples low-resolution color channels with PixelCNN and refines them alongside the high-resolution grayscale input; the paper also notes varied failure modes for future work.
6 CONCLUSION
- PixColor produces diverse colorizations and performs better on average than other published methods in a crowdsourced human evaluation.Source: §6 · p. 7
- To avoid slow PixelCNN inference, PixColor samples low-resolution color channels and uses an image-to-image CNN to refine the result.The refinement network combines sampled color with the high-resolution grayscale input.Source: §6 · p. 7
- The model's outputs match color distributions more closely than other methods, but it exhibits varied failure modes left for future work.Source: §6 · p. 8
APPENDIX
The appendix provides network hyperparameters, selected high-resolution and non-square examples, VTT-score illustrations, and a qualitative comparison of colorization methods.
APPENDIX
- The PixelCNN and refinement networks were trained independently; Table 2 lists their hyperparameters.Source: Table 2 · p. 11
- Figure 9 presents selected high-resolution and non-square samples.Source: Figure 9 · p. 12
- Figure 10 illustrates the possible VTT scores.Source: Figure 10 · p. 12
- Table 3 shows randomly sampled ImageNet test examples colorized by LTBC, pix2pix, VAE, LRAC, CIC, and PixColor alongside the ground truth.Source: Table 3 · p. 13