Source-linked AI summary
Unsupervised Diverse Colorization via Generative Adversarial Networks
Yun Cao, Zhiming Zhou, Weinan Zhang, Yong Yu
TL;DR
Grayscale images can correspond to multiple realistic colorizations, whereas earlier methods generally seek one color image matching the original. The paper uses conditional GANs with a fully convolutional, noise-conditioned generator to produce diverse realistic outputs, receiving more than 62.6% positive human feedback versus 70.0% for ground-truth images. The authors also identify adding item-color or global-tone constraints as future work.
Problem
Because many colors share the same gray value, deterministic colorization methods can produce averaged colors instead of diverse realistic alternatives.
Method
Conditional GANs generate colorizations from a grayscale image and noise using fully convolutional layers, multi-layer noise, multi-layer condition concatenation, and stride 1.
Results
More than 62.6% of human judgments rated generated images positively, compared with 70.0% for ground-truth images; the reported p-value was 0.1359 > 0.05.
Takeaways & Limitations
The model produces various colorization schemes while maintaining good reality and keeping color parts within their correct components.
Takeaways & Limitations
The current method uses only corresponding grayscale images, leaving specified item colors and global color schemes as future constraints.
Abstract
from arXiv · showhide
Colorization of grayscale images has been a hot topic in computer vision. Previous research mainly focuses on producing a colored image to match the original one. However, since many colors share the same gray value, an input grayscale image could be diversely colored while maintaining its reality. In this paper, we design a novel solution for unsupervised diverse colorization. Specifically, we leverage conditional generative adversarial networks to model the distribution of real-world item colors, in which we develop a fully convolutional generator with multi-layer noise to enhance diversity, with multi-layer condition concatenation to maintain reality, and with stride 1 to keep spatial information. With such a novel network architecture, the model yields highly competitive performance on the open LSUN bedroom dataset. The Turing test of 80 humans further indicates our generated color schemes are highly convincible.
1 Introduction
Traditional colorization methods aim to reproduce an original image, but grayscale inputs can support multiple realistic colorings. The paper addresses this ambiguity with conditional GANs designed to generate diverse, realistic colorizations.
- Existing colorization methods: Prior methods use reference-image transfer or supervised regression/classification to model grayscale–color correspondence.Non-parametric methods transfer colors from analogous reference regions, while parametric methods learn prediction functions from color-image datasets.
- Motivation: Many colors share the same gray value, so deterministic mappings can produce weighted-average brownish colors instead of diverse plausible results.The paper gives clothing and bedroom walls as examples whose colors cannot be determined purely from grayscale information.
- Proposed approach: Conditional GANs generate diverse colorizations for one grayscale image while maintaining realism by modeling a color distribution controlled by input noise.The generator captures the data distribution, while the discriminator distinguishes generated images from real ones.
- Proposed approach: The generator uses fully convolutional layers, stride 1, multi-layer condition concatenation, and multi-layer noise to preserve spatial information and increase diversity.Changing the input noise alters the colorization without training a separate model for each color scheme.
- Evaluation: More than 62.6% of human judgments rated generated images positively, compared with 70.0% for ground-truth images.The reported significance test gave p-value 0.1359 > 0.05, indicating no significant difference between generated and real color images in the test.
2 Related Work
Related work includes diverse-colorization approaches and conditional GANs for image translation. This paper distinguishes its solution through a generic conditional-GAN design with architectural choices focused on diversity, realism, and spatial preservation.
- Diverse colorization: Earlier diverse-colorization research used an additional model, class re-balancing, or a variational-autoencoder embedding of color fields.These approaches addressed particular scene changes, colorfulness, or learned low-dimensional color representations.
- Diverse colorization: The paper uses conditional GANs for unsupervised diverse colorization with little domain knowledge of the images.This is presented as a generic alternative to the earlier approaches described in the related work.
- Conditional GANs: Conditional GANs have been applied to image generation and image-to-image translation tasks including grayscale-to-color conversion.Related examples include text-to-image generation and translations such as labels to scenes, aerial images to maps, and day to night.
- Conditional GAN architecture: Unlike encoder–decoder generators, the proposed generator uses only convolutional layers, avoids downsampling, adds multi-layer noise, and repeatedly injects conditional information.These choices target spatial preservation, color diversity, and realism.
- Conditional GAN architecture: The conditional GAN framework trains a generator with grayscale information and noise against a discriminator distinguishing real from generated color images.Both networks are trained adversarially.
3 Methods
The method formulates colorization as conditional adversarial generation: a generator maps a grayscale image and noise to a color image, while a discriminator evaluates whether outputs are real.
- Conditional GAN formulation: Conditional GANs learn a mapping from observed grayscale image y and random noise vector z to color image x.The generator is trained to produce outputs that an adversarially trained discriminator cannot distinguish from real images.
- Adversarial objective: The GAN objective trains the discriminator to maximize discrimination between real data and generated outputs.The objective is expressed through the adversarial value function introduced for the GAN formulation.
- Adversarial objective: The conditional GAN objective incorporates the observed grayscale image as conditioning information alongside the generated and real color images.The conditional objective follows the GAN formulation while conditioning generation and discrimination on the grayscale input.
- Adversarial objective: The generator minimizes the adversarial objective against a discriminator that attempts to maximize it.This opposing optimization defines the adversarial training procedure.
D LcGAN(G, D). (3)
The paper’s conditional GAN uses a fully convolutional generator that preserves spatial information while injecting grayscale conditions and noise throughout the network. Its training and testing procedures are designed to support diverse colorization and evaluate different noise responses.
- Without noise z, the generator maps each grayscale image to deterministic outputs, limiting suitability for diverse colorization.
- The generator uses convolution layers throughout, stride 1, and no pooling to preserve spatial information during generation.This contrasts with encoder-decoder designs that downsize and later upsize data representations.
- Multi-layer noise is introduced across generator layers because noise supplied only at the input can be attenuated during continuous data transformation.
- The model combines a generator and discriminator in a conditional GAN, with training iterations alternating discriminator and generator updates on minibatches of noise, grayscale, and color images.The stated default parameters are kD = 1, kG = 1, m = 64, sz = 100, and s = 64.
- Because layer shapes remain consistent, grayscale conditional information can be concatenated throughout the generator to provide sustained conditional supervision.
- Testing rearranges repeated noise responses for the same image through multi-round testing to assess different generated colorizations.
4 Experiments
Experiments use the LSUN bedroom dataset to test diverse colorization, comparing representation choices, noise and condition injection, and GAN variants. The study evaluates how these design choices affect diversity, structural stability, and training behavior.
- 4.1 Dataset: 503,900 LSUN bedroom images were randomly selected, center-cropped, and reshaped to 64 × 64 pixels for experiments.Bedrooms were chosen because indoor items have varied colors, unlike outdoor scenes with typically blue skies and green trees.
- 4.2 Comparison Experiments: YUV representation avoids the structural discontinuities observed with RGB representation and additional L1 loss.YUV predicts two chrominance channels while using the grayscale Y channel as fixed conditional information, making training more stable.
- 4.2 Comparison Experiments: Multi-layer noise produces more colorful and diverse results than single-layer noise for the same grayscale inputs.The comparison uses 8 × 8 generated images per grayscale input at the same epoch.
- 4.2 Comparison Experiments: Multi-layer condition injection makes generated results more stable, whereas single-layer conditioning can cause colorization deviation.The multi-layer setting provides the generator with more structural information and produces smoother transitions.
- 4.2 Comparison Experiments: Wasserstein GAN produces some comparable results but also failed results despite training for 40 epochs versus 20 epochs for GAN.The authors attribute the longer training requirement to the large dataset and optimization constraints, and therefore use the standard GAN.
5 Results and Evaluation
The final model generates varied color schemes while keeping colors within appropriate image components. Evaluation uses human judgments rather than distance metrics because realistic alternatives may differ substantially from the original colors.
- 5.1 Colorization Results: The fully convolutional generator with multi-layer noise and condition concatenation produces diverse colorization schemes while maintaining good reality.The reported results keep almost all color parts within their correct components without deviation.
- 5.2 Evaluation via Human Study: Distance metrics such as RMSE and PSNR are unsuitable because reasonable diverse colorizations can differ substantially from the original image.The evaluation therefore focuses on whether generated images appear realistic rather than on pixel distance to ground truth.
- 5.2 Evaluation via Human Study: 80 participants judged 62.6% of generated color images convincing, compared with 70.0% for ground-truth images.Participants answered 20 questions each, choosing whether any displayed image had poor reality.
- 5.2 Evaluation via Human Study: p-value 0.1359 > 0.05 indicates no significant difference between generated and ground-truth images in human realness judgments.The test compares the percentages of human judges rating each image as real.
- 5.2 Evaluation via Human Study: 37.5% of generated results were more convincing than the true images within their comparison groups.This credibility rank statistic is computed from the reported average rank range.
6 Conclusion
The paper concludes that conditional GANs can automatically generate diverse, realistic colorizations from grayscale images. It identifies added color or global-tone constraints as future directions for controlling the generated schemes.
- 6 Conclusion: Conditional GANs generate diversified color images for each grayscale input while addressing the sepia-toned tendency of deterministic colorization models.The evaluation uses a questionnaire-based Turing test with 80 participants, whose feedback indicates the results are highly convincing.
- 6 Conclusion: The proposed generator combines a fully convolutional non-stride structure, multi-layer noise, and multi-layer condition concatenation.These choices are presented as supporting diversity, spatial information, and realism.
- 6 Conclusion: Future work could constrain item colors or global color schemes while retaining varied vivid colorizations.Examples include specifying a blue bed and white wall or requesting warm or cool tones.