Source-linked AI summary
Deep Exemplar-based Colorization
Mingming He, Dongdong Chen, Jing Liao, Pedro V. Sander, Lu Yuan
TL;DR
Grayscale colorization is ambiguous, and existing approaches offer limited control or require manual guidance. This paper introduces a CNN that selects, propagates, and predicts colors from references, while preserving plausible colors without reliable correspondence. The method supports customizable outputs, automatic reference recommendation, and extensions to legacy photographs and movies, but remains limited for unusual colors and some visually ambiguous regions.
Problem
Image colorization is inherently ambiguous, while existing learning-based methods generally produce single outputs and exemplar methods require suitable references or manual guidance.
Method
A two-sub-network CNN uses semantic similarity and aligned reference chrominance to select, propagate, and predict target colors from a grayscale image and reference.
Results
The approach produces plausible, reference-faithful colors even with dissimilar references and outperforms other methods on image-recognition accuracy, despite a lower PSNR.
Takeaways & Limitations
Different references provide customizable colorizations, while reference recommendation enables fully automatic colorization and the approach extends to legacy photographs and movies.
Takeaways & Limitations
The method cannot colorize unusual or artistic colors reliably, and perceptual loss can miss incorrect colors in less semantic regions or under large luminance disparities.
Abstract
from arXiv · showhide
We propose the first deep learning approach for exemplar-based local colorization. Given a reference color image, our convolutional neural network directly maps a grayscale image to an output colorized image. Rather than using hand-crafted rules as in traditional exemplar-based methods, our end-to-end colorization network learns how to select, propagate, and predict colors from the large-scale data. The approach performs robustly and generalizes well even when using reference images that are unrelated to the input grayscale image. More importantly, as opposed to other learning-based colorization methods, our network allows the user to achieve customizable results by simply feeding different references. In order to further reduce manual effort in selecting the references, the system automatically recommends references with our proposed image retrieval algorithm, which considers both semantic and luminance information. The colorization can be performed fully automatically by simply picking the top reference suggestion. Our approach is validated through a user study and favorable quantitative comparisons to the-state-of-the-art methods. Furthermore, our approach can be naturally extended to video colorization. Our code and models will be freely available for public use.
1 Introduction
Image colorization is ambiguous and often requires user guidance, motivating a controllable deep-learning approach that uses reference images while predicting plausible colors when correspondence is unavailable.
- Motivation: Colorizing grayscale images is inherently ambiguous because multiple colors can produce perceptually meaningful results.Human intervention often helps resolve this ambiguity.
- Motivation: Scribble-based methods require careful placement and color selection, which can challenge untrained users.Reference-based methods instead depend on finding a similar color image.
- Motivation: Existing learning-based colorization methods automate inference but generally produce a single plausible result despite multimodal uncertainty.Their learned dominant colors limit user control over alternative colorizations.
- Approach: The paper proposes a deep-learning exemplar-based method whose output can be controlled by choosing different references, including dissimilar references that still yield plausible colors.The results are intended to remain visually faithful to references and perceptually meaningful.
- Approach: The CNN directly selects, propagates, and predicts colors from an aligned reference through two sub-networks.The Similarity sub-network provides semantic matching, while the Colorization sub-network learns color propagation and prediction.
- Approach: Reference retrieval combines semantic information and luminance statistics to recommend images and enable fully automatic colorization.The proposed system is also presented as robust to reference selection and extendable to video colorization.
2 Related work
Prior work includes manual propagation, exemplar transfer, and fully learned colorization, while hybrid methods combine learning with user hints; this paper replaces those hints with references and automated retrieval.
- Manual-guidance methods: Scribble-based colorization propagates user-provided colors using low-level similarity measures.Earlier methods used luminance, texture, or related local cues for propagation.
- Exemplar-based methods: Exemplar-based methods reduce user effort by transferring colors from a similar reference image, but global statistics can ignore spatial information.Later approaches improve transfer through local correspondence techniques.
- Learning-based methods: Learning-based methods use neural networks to predict colors automatically from large-scale data, with different approaches defined partly by their loss functions.These include reconstruction, classification, and adversarial objectives.
- Learning-based methods: Fully learned colorization methods generally produce one plausible result per input despite the intrinsic multimodal uncertainty of colorization.This limits their ability to provide user-directed alternatives.
- Hybrid methods: Hybrid methods combine learned robustness with controllability from color points or strokes, but this paper uses reference images instead of user-guided hints.The reference can be selected automatically with the proposed image retrieval system.
3 Exemplar-based Colorization Network
The network colorizes a grayscale target using an aligned color reference, learning semantic matching, color selection, propagation, and fallback prediction end to end. Its two-branch design balances fidelity to reliable reference colors with natural colors when correspondence is weak or unavailable.
- Similarity Sub-Network: A gray-VGG-19 similarity sub-network compares target and reference luminance features to obtain a robust semantic similarity metric.It uses five-level feature pyramids, upsamples them to the input resolution, and computes cosine similarity in both mapping directions.
- Colorization Sub-Network: The end-to-end colorization network learns to select and propagate reference colors while predicting dominant colors for unrelated or misaligned regions.This addresses the difficulty of replacing hand-crafted selection and propagation rules, especially when the reference does not contain related content.
- System pipeline: The system combines a grayscale target, aligned reference chrominance, and bidirectional similarity maps to predict the target’s ab channels.The predicted ab channels are combined with the target luminance channel to form the colorized Lab image.
- Design objective: The framework targets controllable colorization by preserving reliable reference colors while remaining natural when reference colors cannot be trusted.The paper describes this as a hybrid objective because direct pixelwise agreement with ground truth is not always appropriate when multiple plausible reference-driven colors are possible.
- Loss and training: Two shared-weight branches jointly optimize chrominance consistency and perceptual plausibility for reference-guided colorization.The chrominance branch encourages correct reference-color propagation, whereas the perceptual branch penalizes unnatural high-level appearance differences while tolerating plausible color alternatives.
4 Color Reference Recommendation
The system recommends color references by combining semantic similarity with luminance statistics, progressively ranking candidates from a large image database. This supports automatic colorization using the top-ranked reference.
- Reference recommendation: The retrieval algorithm recommends references using both high-level semantic information and low-level luminance statistics.The ideal reference should match the target in semantic content and photometric luminance.
- Feature extraction: Gray-VGG-19 extracts precomputed database features and corresponding query features for reference retrieval.The method uses relu5_4 and fc6 features from luminance-channel images.
- Global ranking: Candidates are first narrowed to approximately 1,000 images sharing the query’s Top-1 class, then globally ranked using fully connected features.The class restriction removes broad semantic mismatches before finer ranking.
- Local ranking: The top 200 globally ranked candidates undergo local reranking using semantic and luminance terms.Local ranking addresses the loss of spatial information in fully connected features.
- Local ranking: Local similarity matches spatial feature points by cosine similarity and compares corresponding local luminance statistics.The score combines these terms with β empirically set to 0.25.
- Efficiency: PCA compression reduces fc6 channels from 4,096 to 128 and relu5_4 channels from 512 to 64, enabling real-time reference retrieval.The dimensionality reductions incur practically negligible loss.
5 Discussion
The discussion shows how the double-branch network learns color selection, propagation, and prediction, combining reference fidelity with plausible colors when matches are unreliable. It remains robust to reference choice, matching algorithms, and unseen image types, while alignment cannot help objects absent from the reference.
- 5.1 What does the Colorization sub-net learn?: The Chrominance branch selects and propagates reference colors, producing consistent colors across well-matched image regions.Examples include blue sky, white plane, and green lawn.
- 5.1 What does the Colorization sub-net learn?: Similarity maps provide a learned hint for sampling, because matching errors correlate with color differences but do not identify every selected point.The network learns this mechanism from large-scale data rather than relying on intuitive heuristics.
- 5.1 What does the Colorization sub-net learn?: The Perceptual branch predicts dominant natural colors for regions that lack a good reference match, such as trunk objects.Its prediction is independent of the reference.
- 5.1 What does the Colorization sub-net learn?: Joint training adaptively fuses reference colors in well-matched regions with learned natural colors in mismatched or unrelated regions.The Colorization sub-net jointly learns sample selection, propagation, and dominant-color prediction.
- 5.2 Comparison with sequential processing: Sequentially selecting samples with thresholds or cross-check matching before applying Zhang et al. [2017] produces noticeable color artifacts.The authors attribute this to a mismatch between the pre-trained network’s training inputs and the selected samples.
- 5.3 Robustness: The method remains plausible with related or unrelated references, becoming more faithful as semantic similarity increases and more conservative otherwise.Different dense matching algorithms show the same pattern: aligned regions follow the reference, while misaligned regions use dominant colors.
- 5.3 Robustness: Better alignment cannot improve colorization for objects that do not exist in the reference.Alignment helps only when semantic correspondences can be found.
- 5.4 Transferability: The method transfers to paintings and cartoons unseen during natural-image training by using correlations between image pairs in the Chrominance branch.The paper contrasts this with prior learning-based methods whose colors degrade on such images.
6 Comparison and Results
The method is evaluated against exemplar-, learning-, and interactive-based approaches using quantitative metrics, qualitative comparisons, and a user study. It also demonstrates automatic reference selection, legacy-image colorization, and video extension.
- Comparisons: The evaluation compares the method with exemplar-, learning-, and interactive-based colorization techniques across quantitative and qualitative tests.The experiments include a 35-pair exemplar-based comparison, a 10,000-image ImageNet validation comparison, and user studies.
- Quantitative comparison: 22.9178dB PSNR is lower than Larsson et al. [2016] and Iizuka et al. [2016], while the method achieves the highest image-recognition accuracy.The authors explain that PSNR penalizes plausible colorizations that differ from ground truth, whereas recognition accuracy better reflects natural and recognizable outputs.
- Qualitative comparison: The method produces more saturated, colorful, and reference-faithful results than selected learning-based methods, while avoiding some overly aggressive color artifacts.Different references provide control over the desired colors, unlike class rebalancing that can produce excessive blue and orange coloration.
- User study: 38.08% fooling rate for the Top-1 reference ranked first, compared with 35.36% for Zhang et al. [2016] in the real-versus-fake user study.The study compared ground-truth photographs with outputs from the proposed method and learning-based baselines.
- Reference selection: A good reference is important for high-quality colorization, while random unrelated references can lead to conservative color prediction.The authors attribute this to the availability of reliable color samples in the reference.
- Applications: The method extends to legacy grayscale photographs and videos by independently colorizing frames and then applying temporal smoothing.The system was trained on synthetic grayscale images and tested on legacy images and movie frames.
7 Limitations and Conclusions
The paper concludes that reference-guided deep colorization produces plausible results even without clear target-reference correspondences and enables controllable, automatic recoloring. It also identifies limitations involving unusual colors, semantically weak regions, luminance disparities, and occasional local failures.
- Conclusion: The approach provides a general exemplar-based solution that yields plausible colors even when the target lacks clear correspondences in the reference.It combines a deep learning architecture with a reference color image rather than relying on strict low-level correspondence.
- Conclusion: Different references control the output colorization, while reference recommendation enables automatic recoloring of photographs and movies.This extends controllability beyond most deep-learning colorization frameworks and reduces manual reference selection.
- Limitations: The network cannot colorize objects with unusual or artistic colors because learning from the Perceptual branch constrains its predictions.This limitation is illustrated in the paper’s discussion of Fig. 21.
- Limitations: VGG-based perceptual loss may miss incorrect colors in semantically unimportant regions or confuse regions with similar local textures.The paper gives walls, sand, and grass as examples of affected regions.
- Limitations: Dramatic luminance disparities between target and reference reduce fidelity to the reference, while some local regions may remain uncolored.The recommendation algorithm enforces luminance similarity, but occasional local prediction failures remain.