Source-linked AI summary
Towards Vivid and Diverse Image Colorization with Generative Color Prior
Yanze Wu, Xintao Wang, Yu Li, Honglun Zhang, Xun Zhao, Ying Shan
TL;DR
Colorization must infer missing colors from grayscale images, where multiple solutions can be plausible and automatic methods may produce artifacts or incoherent colors. GCP-Colorization uses pretrained GAN priors, retrieving matched features for modulation; the authors report more vivid and diverse results than previous works, with controllable latent-code transitions.
Problem
Colorization is uncertain because grayscale images omit color channels and allow multiple plausible color solutions, while automatic CNN methods can produce artifacts and incoherent colors.
Source: §1 · p. 2Method
GCP-Colorization uses a GAN encoder to retrieve matched features from a pretrained GAN prior and incorporates them into colorization through feature modulation.
Source: §1 · p. 2Results
The authors report more vivid and diverse colorization results than previous works.
Source: §1 · p. 2Takeaways & Limitations
Modifying GAN latent codes provides diverse outputs and controllable, smooth colorization transitions.
Source: §1 · p. 2When an input is outside the GAN distribution or inversion fails, color guidance may be unavailable and colorization may become unnatural or incoherent.
Source: §4.5 · p. 8
Abstract
from arXiv · showhide
Colorization has attracted increasing interest in recent years. Classic reference-based methods usually rely on external color images for plausible results. A large image database or online search engine is inevitably required for retrieving such exemplars. Recent deep-learning-based methods could automatically colorize images at a low cost. However, unsatisfactory artifacts and incoherent colors are always accompanied. In this work, we propose GCP-Colorization that leverages the rich and diverse color priors encapsulated in a pretrained Generative Adversarial Networks (GAN) for automatic colorization. Specifically, we first "retrieve" matched features (similar to exemplars) via a GAN encoder and then incorporate these features into the colorization process with feature modulations. Thanks to the powerful generative color prior (GCP) and delicate designs, our GCP-Colorization could produce vivid colors with a single forward pass. Moreover, it is highly convenient to obtain diverse results by modifying GAN latent codes. GCP-Colorization also inherits the merit of interpretable controls of GANs and could attain controllable and smooth transitions by walking through GAN latent space. Extensive experiments and user studies demonstrate that GCP-Colorization achieves superior performance than previous works. Codes are available at https://github.com/ToTheBeginning/GCP-Colorization.
1. Introduction
Colorization is ill-posed because grayscale input leaves colors uncertain and plausible solutions non-unique. GCP-Colorization uses pretrained GAN priors to combine vivid, diverse colorization with automatic processing and latent-code control.
1. Introduction
- Colorization is ill-posed: a grayscale value must stand in for missing color channels, and multiple colors can plausibly fit the same image.The resulting uncertainty and diversity make colorization challenging.Source: §1 · pp. 1–2
- Reference-based methods depend on suitable exemplar images, whose retrieval can require substantial user effort and a large image database or online search engine.Generated color quality, including realness and vividness, depends strongly on the references.Source: §1 · p. 2
- Automatic CNN-based methods reduce the cost of colorization but are accompanied by unsatisfactory artifacts and incoherent colors.The cited methods learn semantics and directly predict colorization results.Source: §1 · p. 2
- GCP-Colorization uses a pretrained GAN’s rich color prior as an exemplar, retrieving matched features with a GAN encoder and incorporating them through feature modulation.Different GAN samples or latent codes can produce diverse outputs, and latent-space walks support controllable, smooth transitions.Source: §1 · p. 2
- The authors report that GCP-Colorization generates more vivid and diverse colorizations than previous works while enabling latent-code control and smooth transitions.They present the framework as combining reference-based realism with automatic, low-cost processing.Source: §1 · p. 2
2. Related Work
Prior colorization approaches use user hints, reference images, learned predictions, or GAN inversion, each with distinct requirements or limitations. GCP-Colorization instead uses GAN features as spatially modulated priors in a feed-forward colorization process.
2. Related Work
- Interactive methods propagate user-drawn color strokes through constrained optimization, with related work exploring propagation efficiency, similarity metrics, long-range relations [53], and bleeding reduction.Careful interaction can produce appealing results, but these methods demand intensive user effort; sparse color points and global palettes are alternatives.Source: §2 · p. 2
- Reference-based methods transfer color statistics through low-level, semantic, or super-pixel correspondences, while deep methods improve matching but still depend on suitable references.Finding references remains time-consuming and challenging for automatic retrieval systems.Source: §2 · p. 2
- Automatic learning-based colorizers use semantic representations, dual-task structures, instance features, class rebalancing, or per-pixel color distributions.Their results can still contain unnatural or incoherent colors.Source: §2 · p. 2
- GAN-inversion colorization methods optimize latent codes to reconstruct images; mGANprior optimizes multiple codes and DGP jointly fine-tunes the generator and codes.These methods struggle to retain local details because low-dimensional latent codes lack spatial information for colorization guidance.Source: §2 · pp. 2–3
- GCP-Colorization derives GAN features and an inversion image from grayscale input, then uses aligned multi-resolution features for modulation rather than per-image iterative optimization.Adjusting latent code z can produce diverse colorizations and smooth, controllable transitions.Source: Figure 2 · p. 3, §2 · p. 3
3. Method
GCP-Colorization uses a pretrained GAN’s color prior to guide automatic colorization. It encodes grayscale inputs, aligns GAN features to them, and uses those features to modulate a colorization network.
3.1. Overview
- The framework encodes a grayscale image, generates a relevant inversion image and intermediate GAN features, aligns those features to the input, then uses SPADE modulation to produce colorization.GAN features, rather than the inversion image alone, guide the colorization network.Source: §3.1 · p. 3
3.2. Generative Color Prior and GAN Encoder
- GAN inversion finds a latent code for a target image; the encoder replaces costly iterative inference by mapping grayscale images to codes, while coarse inversion features suffice for refinement.The paper notes that optimization-based inversion can produce more accurate images, but argues that precise inversion images are unnecessary for this colorization setting.Source: §3.2 · pp. 3–4
3.3. Colorization Network
- The alignment module computes dense correspondence between grayscale-input and inversion-image features, warps multi-scale GAN features, and uses aligned features to modulate corresponding network layers.SPADE preserves and injects the prior color features after spatial alignment.Source: §3.3 · p. 4
3.4. Objectives
- Encoder training uses discriminator-feature discrepancy and latent-code regularization, with domain alignment and contextual loss also included in its objectives.The contextual loss encourages semantic similarity using deeper VGG19 features with larger weights.Source: §3.4 · pp. 4–5
- The colorization network and its discriminator have separate full objectives; colorization uses perceptual and adversarial losses, with adversarial training using LS-GAN losses.The generator and discriminator are trained alternately.Source: §3.4 · pp. 4–5
4. Experiments
On ImageNet, GCP-Colorization compares favorably with automatic colorization methods in quantitative metrics and user preferences. Ablations examine its GAN prior, feature guidance, and spatial alignment, while latent-code changes provide diverse, controllable outputs; the method remains vulnerable when GAN inversion fails or inputs fall outside the GAN distribution.
4.1. Implementation Details
- The experiments use ImageNet and a pretrained BigGAN; training first fits an encoder with the generator frozen, then freezes the encoder while training the remaining networks end-to-end.Images are resized to 256×256; both stages run for 10 epochs.Source: §4.1 · p. 5
4.2. Comparisons with Previous Methods
- GCP-Colorization achieved the lowest FID and better colorfulness on 50,000 ImageNet validation images; users preferred it 46.3% of the time, versus 12.4% for CIC, 11.7% for ChromaGAN, 16.6% for DeOldify, and 13.0% for InstColor.The user study included 30 participants and 70 images; the quantitative comparison also reports that GCP-Colorization’s colorfulness score was closest to ground truth.Source: §4.2 · pp. 5, 7, Table 1 · p. 6, Figure 4 · p. 7
4.3. Ablation Studies
- The GAN prior contributes brighter, more varied colors; multi-resolution feature guidance improves coherence over image guidance, while spatial alignment reduces artifacts from feature misalignment.Without the prior, the method degrades to common automatic colorization; alignment helps preserve coherent colors across objects.Source: §4.3 · p. 7, Figure 5 · p. 7
4.4. Controllable Diverse Colorization
- Perturbing GAN latent codes yields varied exemplars and corresponding colorizations, while walking through latent space supports smooth, controllable changes to colors.The paper describes transitions in background colors, object colors, and saturation.Source: §4.4 · p. 8, Figure 6 · p. 8
4.5. Limitations
- When an input lies outside the GAN distribution or inversion fails, missing GAN guidance can leave objects unnaturally colorized; the authors point to improved inversion and automatic colorization as possible mitigations.The examples include a person on a beach and a cactus missing from inversion results.Source: §4.5 · p. 8
1. GAN Inversion Results
GAN inversion produces colorful images with semantic content similar to the grayscale inputs.
1. GAN Inversion Results
- GAN inversion could generate colorful images.The paper reports that these images share similar semantic content to the grayscale inputs.Source: §1 · p. 11
- The inverted images share semantic content similar to the grayscale inputs.The paper presents these results in Fig. 9.Source: §1 · p. 11
2. Ablation Study of BigGAN Inversion
The BigGAN inversion study addresses optimization challenges with a latent-code L2 penalty and discriminator feature loss; adding more discriminator layers or pixel loss did not improve results and produced blurrier images.
2. Ablation Study of BigGAN Inversion
- An L2 penalty on the latent code and discriminator feature loss are used to address BigGAN inversion challenges.The authors report that the penalty alleviates the issue of large latent-code norms and low-quality inversions; discriminator features are more coherent with inversion images than classification-trained perceptual features.Source: §2 · p. 11
- The authors identify encoder-based BigGAN inversion as a remaining challenge.They conjecture that BigGAN's self-attention architecture makes optimization difficult; domain-guided encoding did not help in their practice.Source: §2 · p. 11
- Adding more discriminator layers or pixel loss did not improve inversion images and instead produced blurrier images with fewer details.These attempts were compared with the aforementioned discriminator-feature-loss and L2-norm method.Source: §2 · p. 11
3. Comparison with GAN Inversion
Compared with DGP, GCP-Colorization better preserves texture faithfulness by incorporating spatial features, and uses feed-forward inference rather than iterative optimization.
3. Comparison with GAN Inversion
- GCP-Colorization obtains better texture faithfulness than DGP by incorporating spatial features with SPADE modulation.The passage attributes DGP's difficulty retaining local details and object shape to low-dimensional latent codes that fail to encode spatial information.Source: §3 · p. 11
- GCP-Colorization uses straightforward feed-forward inference, avoiding DGP's iterative optimization process.Source: §3 · p. 11
4. More Diverse Colorization Results
Compared with PIC, GCP-Colorization avoids the spatially incoherent and unrealistic colorizations described for PIC, while supporting smooth, controllable outputs.
4. More Diverse Colorization Results
- PIC usually generates spatially incoherent and unrealistic colorizations, while GCP-Colorization does not suffer from these issues.PIC obtains diverse colorizations by sampling a PixelCNN model of the joint distribution of pixel colors.Source: §4 · p. 11
- GCP-Colorization can produce smooth and controllable colorizations.Source: §4 · p. 11
5. User study results
The user study reports a significant preference for GCP-Colorization over four competing methods, while comparisons with PIC describe more spatially coherent, natural colorizations with smooth control.
5. User study results
- Compared with PIC, GCP-Colorization produced spatially coherent, more natural results with smooth control.Source: Figure 12 · p. 14, Figure 13 · p. 15
- Users significantly preferred GCP-Colorization over CIC, ChromaGAN, DeOldify, and InstColor.Figure 14 reports user preferences of 10% (CIC), 3% (ChromaGAN), 13% (DeOldify), 3% (InstColor), and 70% (GCP-Colorization).Source: §5 · p. 11, Figure 14 · p. 16
- The user-preference figures compare CIC, ChromaGAN, DeOldify, InstColor, and GCP-Colorization.Source: Figure 14 · p. 16