Source-linked AI summary
ChromaGAN: Adversarial Picture Colorization with Semantic Class Distribution
Patricia Vitoria, Lara Raad, Coloma Ballester
TL;DR
Grayscale image colorization is ill-posed because multiple colorizations can be correct. ChromaGAN uses a self-supervised adversarial generator conditioned on semantic information, and the paper reports realistic, high-quality colorization with state-of-the-art quantitative comparisons.
Problem
Automatic image colorization remains challenging because grayscale images can correspond to multiple plausible color solutions.
Method
ChromaGAN uses an end-to-end adversarial generator to predict chrominance while learning semantic class distributions from grayscale images without labeled data.
Results
ChromaGAN produces photorealistic colorful images, outperforms state-of-the-art methods in PSNR and perceptual quality, and achieves perceptually more realistic results in a user study.
Takeaways & Limitations
Semantic information coupled with adversarial learning supports vivid, diverse, and perceptually realistic colorizations.
Takeaways & Limitations
Quantitative evaluation is challenging because several colors may be valid for the same object, making ground-truth closeness an imperfect measure.
Abstract
from arXiv · showhide
The colorization of grayscale images is an ill-posed problem, with multiple correct solutions. In this paper, we propose an adversarial learning colorization approach coupled with semantic information. A generative network is used to infer the chromaticity of a given grayscale image conditioned to semantic clues. This network is framed in an adversarial model that learns to colorize by incorporating perceptual and semantic understanding of color and class distributions. The model is trained via a fully self-supervised strategy. Qualitative and quantitative results show the capacity of the proposed method to colorize images in a realistic way achieving state-of-the-art results.
1. Introduction
Automatic image colorization remains challenging because grayscale images admit multiple plausible color solutions. ChromaGAN addresses this with an end-to-end adversarial model that combines semantic information with color and perceptual learning.
- Colorization adds plausible color information to monochrome photographs or videos and remains challenging despite substantial progress.
- ChromaGAN is a fully automatic end-to-end adversarial approach that combines generative adversarial networks with semantic class distribution learning.
- ChromaGAN produces vibrant and diverse colorizations from semantic understanding, including different colors for objects in the same category.
- A user-based perceptual study reports that adversarial learning is key to obtaining vivid colorizations.
- The model uses a three-term loss combining color information, perceptual information, and semantic class distribution.
2. Related Work
Earlier colorization methods relied on user guidance, reference images, or learned mappings from grayscale/color data. The related work positions semantic information within an adversarial colorization framework as the paper's distinctive direction.
- Colorization methods are grouped into scribble-based, exemplar-based, and deep learning-based approaches.
- Scribble-based methods: Scribble-based methods propagate user-provided local color hints across an image using assumptions about neighboring pixels with similar intensities.
- Exemplar-based methods: Exemplar-based methods transfer colors from a reference image, but early approaches could lack spatial coherency and produce unsatisfactory results.
- Deep learning methods: Deep learning methods learn automatic colorization from grayscale/color image pairs, using semantic descriptors, regression, autoencoders, CNNs, or local and global features.
- Deep learning methods: Hybrid CNN methods combine learned colorization with user interactions or exemplar references to provide multiple plausible solutions, rare colors, or fallback predictions.
- GAN-based methods: ChromaGAN incorporates semantic class distributions into an adversarial approach coupled with color regression.
3. Proposed Approach
ChromaGAN learns to colorize grayscale images by jointly predicting chrominance and semantic class distributions within an adversarial framework. Its objective combines color accuracy, perceptual realism, and semantic consistency, using self-supervised training on color images.
- The generator maps grayscale input L to Lab chrominance channels (a, b), producing a geometrically, perceptually, and semantically plausible color image.
- ChromaGAN’s generator also predicts a semantic class distribution vector y, learned without labeled data to provide scene and object information for colorization.The target distribution is obtained from a pretrained VGG-16 applied to a reshaped grayscale image.
- The Objective Function: The WGAN component uses gradient penalty to constrain the discriminator toward 1-Lipschitz functions, supporting stable training and reducing vanishing-gradient and mode-collapse problems.
- The adversarial loss favors sharp, colorful outputs and perceptually realistic palettes, while the L2 color loss alone tends to produce ochreish results.
- The Objective Function: The objective combines a per-pixel L2 chrominance loss, a KL-based class-distribution loss, and a WGAN adversarial loss.The respective terms target color accuracy, semantic interpretation, and perceptual realism.
- Detailed Model Architecture: The generator contains chrominance and classification subnetworks with shared modules, while a discriminator distinguishes real color images from generated colorizations.Both generator subnetworks are jointly trained with one-step backpropagation.
4. Experimental Results and Discussion
The experiments evaluate ChromaGAN qualitatively, quantitatively, and through a perceptual realism study, including ablations of adversarial and semantic components. Results indicate vivid, realistic colorizations, while PSNR comparisons and cross-dataset examples provide complementary evidence.
- Evaluation protocol: The evaluation combines qualitative comparisons, quantitative PSNR, and a user-based perceptual realism study because multiple colors may be valid for one grayscale image.The authors note that ground-truth proximity is not always the best measure of colorization quality.
- Ablation design: Three variants isolate the loss terms: full ChromaGAN, ChromaGAN without classification, and ChromaNet without the adversarial term.The variants are trained end-to-end on 1.3M ImageNet images resized to 224 × 224 pixels.
- Perceptual realism: The perceptual study finds ChromaGAN most realistic among the compared methods, with adversarial learning contributing more than semantic information to natural color generation.The study used 50 randomly selected images and was performed 62 times.
- Quantitative comparison: ChromaGAN generally achieves higher PSNR than, [22], and [39], but ChromaNet obtains the highest PSNR among the three proposed variants.The PSNR experiment averages results over 1000 ILSVRC2012 challenge images.
- Qualitative comparisons: ChromaGAN produces more lively and natural colors than muted outputs from Iizuka et al. and ChromaNet, while retaining vivid colorization across many ImageNet examples.The comparison uses publicly available online demos and includes ChromaGAN without classification, Larsson et al., and Zhang et al.
- Cross-dataset evaluation: On Places validation images, ChromaGAN and Iizuka et al. produce comparable results despite training on different datasets and different numbers of training images.Iizuka et al. uses Places, while ChromaGAN uses ImageNet.
5. Conclusions
ChromaGAN is presented as a novel adversarial colorization method, with quantitative comparisons reporting improvements over state-of-the-art methods and qualitative comparisons showing high visual quality.
- ChromaGAN uses an adversarial strategy to capture geometric, perceptual, and semantic information for colorization.
- Quantitative comparisons report that ChromaGAN outperforms state-of-the-art methods in PSNR and perceptual quality.
- Qualitative comparisons show high visual quality for the resulting colorized images.