Source-linked AI summary
Image Colorization with Generative Adversarial Networks
Kamyar Nazeri, Eric Ng, Mehran Ebrahimi
TL;DR
Automatic grayscale-image colorization is ill-posed, while existing methods may rely on common image themes or processed inputs. The paper uses a conditional DCGAN with modified objectives, architectural and training strategies, and evaluates it on CIFAR-10 and Places365. GAN colorizations were qualitatively better than U-Net on CIFAR-10, but Places365 produced mixed results with frequent mis-colorization in highly textured regions.
Problem
Automatic colorization has many possible color assignments, and prior methods often target common themes or require highly processed inputs.
Method
The paper uses a conditional DCGAN for grayscale-to-color image translation, combining adversarial training with L1 structure-preserving regularization and stabilization strategies.
Results
GAN images were consistently better-looking than U-Net on CIFAR-10, while Places365 produced mixed results with frequent mis-colorization in highly textured images.
Takeaways & Limitations
GAN-based colorization can produce acceptable visual results, but performance remains sensitive to dataset content and textured image regions.
Takeaways & Limitations
Mode collapse can cause the generator to emit the same output, and training data biases can produce frequent color errors such as red cars or green textured regions.
Abstract
from arXiv · showhide
Over the last decade, the process of automatic image colorization has been of significant interest for several application areas including restoration of aged or degraded images. This problem is highly ill-posed due to the large degrees of freedom during the assignment of color information. Many of the recent developments in automatic colorization involve images that contain a common theme or require highly processed data such as semantic maps as input. In our approach, we attempt to fully generalize the colorization procedure using a conditional Deep Convolutional Generative Adversarial Network (DCGAN), extend current methods to high-resolution images and suggest training strategies that speed up the process and greatly stabilize it. The network is trained over datasets that are publicly available such as CIFAR-10 and Places365. The results of the generative model and traditional deep neural networks are compared.
1 Introduction
Automatic grayscale-image colorization is valuable for restoration and animation but remains ill-posed because many color assignments are possible. The paper extends GAN-based colorization toward generalized, high-resolution processing with stabilized training.
- Automatic colorization supports applications including restoring aged images and coloring animations.
- Colorization is ill-posed because grayscale images allow many possible assignments of color information.
- The paper explores GAN-based colorization and compares its results with existing convolutional neural networks.
- The approach aims to generalize colorization to high-resolution images while introducing training strategies that speed and stabilize learning.
2 Generative Adversarial Network
The paper describes GANs as competing generator-discriminator networks and adapts their objectives for conditional image colorization. It uses a non-saturating generator cost and an L1 regularizer to improve training behavior and preserve image structure.
- GANs train a generator to produce realistic samples while a discriminator distinguishes generated data from original data.The generator and discriminator are trained simultaneously until the discriminator cannot consistently classify generated outputs.
- The GAN objective can be represented as separate generator and discriminator costs or as a single minimax value function.
- The original generator cost can produce near-zero gradients when the discriminator performs well, slowing convergence.The paper also states that this cost is unbounded below and may diverge during minimization.
- The proposed heuristic replaces minimizing discriminator correctness with maximizing the probability that the discriminator is mistaken.This is described as a non-saturating game suggested in Goodfellow’s NIPS 2016 tutorial.
- An L1 regularization term encourages generated colors to remain similar to ground-truth images and discourages arbitrary pixel colors.The regularization parameter is λ, and y denotes the ground-truth color labels.
- Conditional GANs replace random generator noise with a grayscale image prior and condition the discriminator on grayscale-color image pairs.
3 Method
The method treats colorization as a high-dimensional image-to-image translation task and combines convolutional encoder-decoder architectures with conditional adversarial training. It uses L*a*b* color space, skip-connected U-Net structure, and training strategies intended to stabilize GAN optimization.
- Problem formulation: Colorization maps grayscale images to color images, with L*a*b* separating brightness from color channels to reduce intensity-related color variation.The model predicts color information while preserving spatial structure from the grayscale input.
- Baseline Network: The baseline uses an encoder-decoder network with downsampling and upsampling, while U-Net skip connections preserve low-level information and prominent edge locations.Contracting features are concatenated with upsampled features in the expansive path.
- Conditional GAN: The GAN follows DCGAN-style convolutional generator and discriminator architectures, modified into a conditional GAN that pairs colored images with their grayscale inputs.The discriminator classifies whether each grayscale–color pair contains a real colored image.
- Training Strategies: The training setup uses Adam optimization, an initial learning rate of 2 × 10^-4, manual tenfold decay after plateaus, and λ = 100 for the regularization term.The λ setting encourages generated images to remain similar to ground truth.
- Training Strategies: The alternative generator cost function addresses near-zero gradients and unbounded decrease, while one-sided label smoothing encourages softer discriminator probabilities.These strategies target convergence and adversarial-training stability.
- Training Strategies: Batch normalization, strided convolutions, reduced Adam momentum, and LeakyReLU activations are used as additional GAN training strategies.Batch normalization is described as preventing the generator from collapsing samples to a single point.
4 Experimental Results
The experiments evaluate colorization with mean absolute error and threshold-based pixel accuracy on CIFAR-10 and Places365. GAN outputs were visually more vibrant than baseline CNN outputs, but dataset-frequency biases and mis-colorization remained.
- Evaluation: Mean absolute error measures average pixelwise color error, while accuracy counts pixels whose color channels fall within threshold distance ϵ of the source.Accuracy is computed over pixels and color channels using the threshold-based indicator definition.
- CIFAR-10: GAN images showed clear visual improvement over baseline CNN images on preliminary CIFAR-10 results, with more vibrant colors and occasional near-ground-truth reconstructions.The baseline CNN results were described as suffering from a light hue.
- Dataset effects: Many car images were colored red, likely reflecting the larger number of red-car examples in the training data.This illustrates a tendency to assign frequently observed colors.
- Places365: Places365 results included green mis-colorization in high-fluctuation regions, a sepia effect under U-Net, and unusual sky gradients.The authors associate the green bias with many grassland images and suspect insufficient training contributed to the sepia effect.
5 Conclusion and Future Work
The GAN colorized grayscale images to an acceptable visual degree, outperforming U-Net qualitatively on CIFAR-10 but producing mixed results on Places365.
- GAN colorization reached an acceptable visual degree on grayscale images.
- On CIFAR-10, GAN outputs consistently looked better than U-Net outputs.U-Net frequently produced a brownish Sepia effect associated with its L2 loss and blurring.
- Places365 produced mixed results, with frequent mis-colorization in highly textured images.The model appears to have identified textured regions as grass, likely reflecting leaves and grass in its training images.
- Places365 colorization was less well trained than CIFAR-10 because it used higher resolution and a much larger dataset.The resolutions were 256×256 versus 32×32, and the dataset sizes were 1.8 million versus 50,000.
- The authors expect further training to improve results and note the need for a better quantitative performance metric.
A CIFAR-10 Results
Figure 3 presents CIFAR-10 colorization by showing grayscale input, original image, U-Net output, and GAN output.
- Figure 3 compares grayscale input, original images, U-Net colorization, and GAN colorization on CIFAR-10.
B Places365 Results
Figure 4 presents Places365 colorization by showing grayscale input, original image, and GAN output.
- Figure 4 compares grayscale input, original images, and GAN colorization on Places365.