Source-linked AI summary
Real-Time User-Guided Image Colorization with Learned Deep Priors
Richard Zhang, Jun-Yan Zhu, Phillip Isola, Xinyang Geng, Angela S. Lin, Tianhe Yu, Alexei A. Efros
TL;DR
User-guided colorization must balance the convenience of automatic methods with the control of interactive editing. This paper trains a CNN on simulated interactions to combine grayscale images with sparse hints, recommend colors, and produce real-time colorizations that novice users can create effectively, while retaining extensions to global hints.
Problem
Traditional edit propagation can require more than fifty strokes, while automatic methods may produce artifacts and cannot express users’ preferred plausible colors.
Method
The paper trains a CNN to map grayscale images and sparse user hints to colorizations, using synthetically generated interactions and learned image priors.
Results
Novice users can often produce colorizations that fool human judges after one minute, while global histograms and average saturation achieve PSNR 27.9 and 25.6, respectively, versus automatic colorization PSNR 24.4.
Takeaways & Limitations
The framework supports interactive creation of realistic, unusual, and globally controlled colorizations from sparse or statistical user guidance.
Takeaways & Limitations
The network can produce undesired non-local background changes, incompletely propagate points, and require dense boundary marking; high-precision work may be better suited to Photoshop.
Abstract
from arXiv · showhide
We propose a deep learning approach for user-guided image colorization. The system directly maps a grayscale image, along with sparse, local user "hints" to an output colorization with a Convolutional Neural Network (CNN). Rather than using hand-defined rules, the network propagates user edits by fusing low-level cues along with high-level semantic information, learned from large-scale data. We train on a million images, with simulated user inputs. To guide the user towards efficient input selection, the system recommends likely colors based on the input image and current user inputs. The colorization is performed in a single feed-forward pass, enabling real-time use. Even with randomly simulated user inputs, we show that the proposed system helps novice users quickly create realistic colorizations, and offers large improvements in colorization quality with just a minute of use. In addition, we demonstrate that the framework can incorporate other user "hints" to the desired colorization, showing an application to color histogram transfer. Our code and models are available at https://richzhang.github.io/ideepcolor.
1 INTRODUCTION
The paper combines learned image priors with sparse user control to address the limitations of both traditional edit propagation and fully automatic colorization. It trains a CNN with simulated interactions and adds a data-driven color palette to help users efficiently create realistic or unconventional results.
- Traditional edit propagation can require more than fifty strokes because users must explicitly indicate each differently colored region.These methods rely on user-provided colors and hand-defined priors such as piecewise smoothness.
- Fully automatic colorization is easy to apply but can produce incorrect colors and artifacts while preventing users from selecting plausible alternative colors.Object colors can be inherently ambiguous, such as whether a t-shirt should be blue, red, or green.
- The proposed CNN maps grayscale images and sparse user inputs directly to full color images, using randomly simulated interactions during training.This combines learned priors from large-scale data with user control without requiring collected interaction data.
- The learned interaction procedure may translate user edits more powerfully than hand-designed image filters or optimization constraints.The paper also demonstrates extensions to unusual colorizations and other output hints, including global color statistics.
- The system provides a data-driven color palette that recommends probable colors at each location to guide user decisions.The palette is intended to reduce the difficulty of selecting plausible colors, especially for novice users.
- After one minute of use, novice users can often produce colorizations that fool human judges in real-versus-fake tests.The study used minimal interface training and limited colorization time.
2 RELATED WORK
Related work spans local edit propagation, automatic colorization, global control, and deep semantic image editing. This paper positions its learned interactive network as combining expressive user guidance with semantic, data-driven image changes.
- User-guided colorization: Earlier interactive colorization methods propagate user strokes with low-level similarity metrics, so realistic results typically require numerous edits.These methods emphasize local control through strokes.
- User-guided colorization: Global color themes and palettes provide expressive controls beyond local strokes, and the paper integrates global hints through color distributions and average saturation.The related work identifies global control as a complementary direction to local interaction.
- Automatic colorization: Example-based automatic methods transfer color statistics from reference images, while newer methods learn grayscale-to-color mappings from large-scale image collections.The latter methods use CNNs to combine low- and high-level cues for colorization.
- Deep semantic image editing: Deep neural networks extract semantic information about materials, segmentation, objects, and scenes that can support semantic image editing.This motivates using learned semantic representations for higher-level visual changes.
- Deep semantic image editing: Unlike approaches focused on user-controllable generation, the paper uses minimal guidance to produce global image changes with a few clicks and real-time feedback.The network architecture includes shared colorization layers plus branches for local points and global inputs.
3 METHODS
The method trains deep networks to colorize grayscale images using local or global user hints, with synthetic interactions and data-driven color suggestions for interactive control.
- Learning to Colorize: The system maps a grayscale image and user tensor to predicted ab color channels with a CNN trained against desired colorizations.The grayscale input is the CIE Lab lightness channel, and the output estimates the two ab channels.
- Learning to Colorize: Two network variants use sparse local points or global statistics, with training hints generated by projecting ground-truth colors.The global variant accepts color histograms and average saturation, while the local variant uses point colors and a binary mask.
- Learning to Colorize: The network uses a smooth-ℓ1 loss instead of class rebalancing or a GAN term, favoring conservative initial colorizations that users can refine.The loss is evaluated per pixel and summed over the image.
- Local Hints Network: Synthetic user interactions avoid large-scale interaction-data collection by randomly sampling revealed color patches during training.The method samples patch locations and sizes, reveals average patch colors, and provides full ground-truth color in 1% of training instances to encourage copying.
- Local Hints Network: A data-driven palette predicts per-pixel color distributions and clusters them into K = 9 discrete suggestions for real-time user selection.Suggestions are intended to reflect plausible colors associated with object type, material, and texture.
- Global Hints Network: The end-to-end framework also supports global hints that alter the color distribution and average saturation of the output.This extends user control beyond sparse local points to global color statistics.
4 EXPERIMENTS
Experiments evaluate sparse local hints, user-guided realism, unusual colorizations, and global statistics. The results show strong performance with limited supervision, rapid novice use, and controllable colorization behavior, while identifying boundaries of the interaction setting.
- 4.1 How well does the system incorporate inputs?: 24.4 PSNR is achieved automatically, and the method maintains higher PSNR than competing methods through 500 random points.Edit-propagation methods approach the learned method as the number of revealed points grows, eventually surpassing it when densely supervised.
- 4.1 How well does the system incorporate inputs?: Intelligent max-error sampling typically achieves higher PSNR, while the learned method remains more accurate than the current state-of-the-art method.The strategy reveals 7 × 7 patches selected using maximum error averaged over a 25×25 window.
- User study: Novice users produced realistic colorizations after 2 minutes of training and 1 minute per image.The study used 28 subjects, each colorizing 20 randomly selected ImageNet test images.
- User study: Added user points typically increase saturation and restore higher-frequency detail missing from automatic colorization.Compared with Levin et al. (2004), the method was also more effective at finding segment boundaries from sparse inputs.
- 4.3–4.5: User inputs can impose unusual colors within objects, transfer reference-image color distributions, and create seams between multiple colors in one segment.Green face colorization, histogram transfer, and color-dependent seam behavior demonstrate control beyond typical natural-image color statistics.
- 4.4 Is the system able to incorporate global statistics?: Global histograms and average saturation raise PSNR to 27.9 and 25.6, respectively, compared with 24.4 for automatic colorization.The Global Hints Network also showed higher performance than Deshpande et al. (2015) and nearly matched Larsson et al. (2016) on SUN-6.
5 LIMITATIONS AND DISCUSSION
The system can propagate sparse inputs using learned semantic similarities, but difficult boundaries and non-local effects still require extra user input. Its current point-based training also leaves stroke inputs and combined local-global control as future directions.
- Foreground points can undesirably change the background, while some user points are not completely propagated.Additional control points can remedy non-local changes or incomplete propagation.
- Difficult segmentation boundaries sometimes require dense marking on both sides to prevent color bleeding.The interface can incorporate hundreds of points, but this exceeds its intended few-seconds-to-couple-minutes interaction regime.
- The system is trained on point inputs; incorporating stroke-based inputs and integrating local points with global statistics remain future directions.The authors report that random sampling covers the low-dimensional point workspace surprisingly well.
CHANGE LOG
The supplied change-log passage identifies the camera-ready initial release and provides the paper’s DOI.
- The version is labeled v1 Initial release.
- The release is identified as the SIGGRAPH camera-ready version.
- The passage provides the paper’s DOI link.