Source-linked AI summary
Learning Diverse Image Colorization
Aditya Deshpande, Jiajun Lu, Mao-Chuang Yeh, Min Jin Chong, David Forsyth
TL;DR
The paper tackles the ambiguity of colorization, where one grey-level image can have multiple realistic outputs. It learns low-dimensional color-field embeddings with a VAE, models their multimodal conditional distribution with an MDN, and reports more diverse colorizations than CVAE and cGAN. Evaluation uses variance as a proxy for diversity because multiple ground-truth colorizations are unavailable.
Problem
Colorization admits multiple realistic outputs for one grey-level image, while prior methods generally produce a single output and independent pixel sampling lacks spatial coordination.
Method
A VAE embeds color fields in a low-dimensional space, and an MDN models the multimodal conditional distribution from grey-level features to embeddings for sampling and decoding.
Results
The method produces better diverse colorizations than CVAE and cGAN, with realistic samples and reliably high variance at comparable error-of-best.
Takeaways & Limitations
Low-dimensional embeddings let multimodal conditional models generate diverse, spatially coherent colorizations, and the approach can extend to other ambiguous problems.
Takeaways & Limitations
Diversity cannot be reliably evaluated against a single ground-truth image, so the paper uses variance as a weaker proxy and qualitative naturalness judgments.
Abstract
from arXiv · showhide
Colorization is an ambiguous problem, with multiple viable colorizations for a single grey-level image. However, previous methods only produce the single most probable colorization. Our goal is to model the diversity intrinsic to the problem of colorization and produce multiple colorizations that display long-scale spatial co-ordination. We learn a low dimensional embedding of color fields using a variational autoencoder (VAE). We construct loss terms for the VAE decoder that avoid blurry outputs and take into account the uneven distribution of pixel colors. Finally, we build a conditional model for the multi-modal distribution between grey-level image and the color field embeddings. Samples from this conditional model result in diverse colorization. We demonstrate that our method obtains better diverse colorizations than a standard conditional variational autoencoder (CVAE) model, as well as a recently proposed conditional generative adversarial network (cGAN).
1. Introduction
Colorization is ambiguous because one grey-level image can support multiple realistic color fields, but independent pixel sampling loses long-scale spatial coherence. The paper addresses this by embedding color fields in a low-dimensional space and learning a multimodal conditional model that enables coherent diverse samples.
- A single grey-level image can have multiple acceptable colorizations, including different sky, building, skin, and other colors.
- Independent sampling from per-pixel color distributions produces spatially incoherent and unrealistic color fields.
- Directly modeling P(C|G) is difficult because color fields and grey-level features are high-dimensional and their distributions are scattered.
- The method learns a low-dimensional VAE embedding z for color fields and an MDN-based multimodal model P(z|G) from grey-level CNN features.
- At test time, sampling embeddings from P(z|G) and decoding them with the VAE produces diverse, spatially coherent colorizations.
- The authors report better diverse colorizations than CVAE and cGAN baselines.
2. Background and Related Work
Earlier automatic colorization methods generally produced one output, while independent sampling from per-pixel distributions created speckled, spatially uncoordinated results. VAEs provide low-dimensional embeddings but commonly yield blurry, over-smooth outputs, motivating specialized decoder losses.
- Earlier automatic colorization methods were designed to produce a single color image rather than multiple outputs.
- A VAE learns a low-dimensional color-field embedding through encoder and decoder networks with reconstruction and KL-divergence objectives.
- Independent sampling of predicted per-pixel color distributions creates speckled noise and lacks spatial coordination.
- VAEs are known to produce blurry and over-smooth outputs, so the paper designs losses discouraging blurry, greyish color fields while incorporating specificity and colorfulness.
3. Embedding and Decoding a Color Field
The paper augments the VAE decoder beyond standard L2 reconstruction to preserve color-field specificity, emphasize rare colors, and match image gradients. These losses produce more vivid and realistic colorizations, while decoder fidelity receives greater emphasis than embedding regularization.
- The decoder losses are designed to avoid the over-smooth and washed-out fields produced by standard L2 loss.
- Specificity: The specificity loss compares generated and ground-truth fields along the top 20 principal components and includes a residual term.
- Colorfulness: Colorfulness reweights squared pixel errors by inverse empirical color probability, giving rarer colors higher weight than common colors.
- Gradient: A gradient loss encourages generated color fields to have the same horizontal and vertical gradients as the ground truth.
- The overall decoder objective combines histogram, Mahalanobis, and gradient losses: Ldec = Lhist + λmahLmah + λgradLgrad.
- The encoder KL loss is weighted by 10^-2 relative to decoder loss, relaxing embedding regularization to prioritize color-field fidelity.
4. Conditional Model (G to z)
The conditional stage models the one-to-many relationship between grey-level features and low-dimensional color-field embeddings with a Gaussian-mixture MDN. A closest-component approximation stabilizes training while retaining multimodal sampling for diverse colorizations.
- An MDN models the multimodal conditional distribution P(z|G), allowing multiple embedding values for the same grey-level input.
- The MDN minimizes conditional negative log likelihood for a Gaussian mixture whose weights and means are predicted from G.
- Because the mixture likelihood can suffer numerical underflow early in training, the method selects the Gaussian component whose mean is closest to ground-truth z.
- Optimizing only the closest component resolves mixture-component symmetry and lets clustered grey-level features jointly optimize the mixture model.
- The MDN-based strategy produces better diverse colorizations than CVAE and cGAN baselines.
5. Baseline
The baselines model diverse colorization using conditional generative architectures: CVAE conditions both embedding generation and decoding on the grey-level image, while cGAN injects variation through dropout. The paper compares these approaches with its VAE–MDN strategy for diverse colorization.
- CVAE generates embeddings from the color and grey-level inputs, then conditions its decoder on the grey-level image and embedding.
- At test time, CVAE samples embeddings randomly while keeping the grey-level input fixed.
- cGAN uses an encoder-decoder with skip connections, patch-based adversarial and L1 losses, and dropout as its noise source.
- The cGAN baseline generates diverse colorizations at test time by applying dropout and clusters 256 outputs into 5 colorizations.
- The paper compares CVAE and cGAN with its VAE–MDN strategy for diverse colorization.
6. Architecture and Implementation Details
The implementation uses a VAE encoder-decoder to map between color fields and low-dimensional embeddings, and an MDN to model grey-level features into a multimodal embedding distribution. At test time, sampled or selected embeddings are decoded into diverse colorizations, with CVAE and an optional skip-connected variant providing comparison architectures.
- VAE: The VAE follows DCGAN-style design choices, including strided convolutions, batch normalization, ReLU intermediate activations, and tanh output activation.
- VAE: The VAE encoder maps a 64 × 64 × 2 color field to a d-dimensional embedding, while the decoder reconstructs a 64 × 64 × 2 Lab color field.The decoder’s two output channels are the a and b channels of Lab color space.
- VAE: The model uses d = 64 for all three datasets.
- MDN: The MDN takes 28 × 28 × 512 grey-level features and outputs an 8-component Gaussian mixture over the d-dimensional embedding.Its output contains 8 × d means and 8 mixture-weight activations, using fixed spherical variance .1.
- MDN: At test time, embeddings sampled from the MDN are decoded by the VAE into diverse colorizations; the reported procedure instead selects the top-k means by mixture weight with k = 5.
- CVAE: The CVAE adds a grey-level encoder and conditions the decoder on the resulting features and randomly sampled embeddings.At test time, 256 embeddings are clustered into 5 colorizations.
- Implementation: The implementation code is provided at the project website.
7. Results
Experiments across LFW, LSUN-Church, and ImageNet-Val evaluate decoder losses and diverse colorization against CVAE and cGAN baselines. The proposed method produces vivid, realistic, diverse colorizations, with high variance and comparable error-of-best, while diversity is assessed only through a proxy.
- Effect of loss terms: Using all decoder loss terms produces plausible, realistic, vivid colorizations instead of the greyish outputs obtained with standard L2 loss.The combined losses include Mahalanobis, colorfulness, and gradient terms; examples include yellow, brown, and green regions.
- Effect of loss terms: The specificity loss provides subtler benefits, including more contained skin colors, while all loss terms improve colorizations over standard L2 loss.On mean absolute error, all loss terms outperform standard L2 on LFW and Church despite not being trained specifically for that metric.
- Effect of loss terms: For all datasets, the proposed loss terms achieve better weighted absolute error per pixel than standard L2 loss.The weighted metric uses the colorfulness-loss weights and evaluates both all pixels and a central 8 × 8 grid.
- Comparison to baselines: Compared with CVAE and cGAN, the strategy generates higher-quality diverse colorizations that remain realistic across the evaluated datasets.LFW samples vary in skin tones, hair, clothing, and backgrounds; LSUN-Church samples vary in brick, sky, and grass colors.
- Comparison to baselines: The method reliably produces high variance with comparable error-of-best to other methods, but variance is only a proxy for diversity because ground truth lacks multiple colorizations per image.Naturalness of the predicted colorization pool is therefore checked qualitatively.
8. Conclusion
The conclusion presents the loss terms and multimodal conditional model as producing realistic, diverse colorizations. It also identifies limited high spatial detail as the main boundary and a focus for future work.
- The proposed loss terms support a VAE that generates high-fidelity color fields.
- The multimodal conditional model produces embeddings that decode into realistic diverse colorizations more diverse than those of CVAE and cGAN.
- The low-dimensional embeddings enable diversity prediction but do not encode high spatial detail, motivating future work on improving detail alongside diversity.