Source-linked AI summary
Instance-aware Image Colorization
Jheng-Wei Su, Hung-Kuo Chu, Jia-Bin Huang
TL;DR
Image colorization is ill-posed and existing learning-based methods struggle with multiple objects because they lack effective object-level semantics. The paper detects and colorizes cropped instances alongside the full image, fusing their features to produce globally consistent colors, and reports favorable or state-of-the-art performance across benchmark datasets.
Problem
Existing colorization models struggle with multiple objects in cluttered backgrounds, where image-level learning does not sufficiently capture object-level semantics.
Method
An off-the-shelf object detector supplies cropped instances for an instance colorization network, whose features are fused with full-image features by a learned fusion module.
Results
The method compares favorably with existing colorization methods across three benchmark datasets and achieves state-of-the-art performance.
Takeaways & Limitations
Instance-aware processing improves visual quality for complex scenes, particularly those containing multiple instances.
Takeaways & Limitations
When object instances are not detected, the method reverts to full-image colorization and may produce washed-out colors or boundary bleeding.
Abstract
from arXiv · showhide
Image colorization is inherently an ill-posed problem with multi-modal uncertainty. Previous methods leverage the deep neural network to map input grayscale images to plausible color outputs directly. Although these learning-based methods have shown impressive performance, they usually fail on the input images that contain multiple objects. The leading cause is that existing models perform learning and colorization on the entire image. In the absence of a clear figure-ground separation, these models cannot effectively locate and learn meaningful object-level semantics. In this paper, we propose a method for achieving instance-aware colorization. Our network architecture leverages an off-the-shelf object detector to obtain cropped object images and uses an instance colorization network to extract object-level features. We use a similar network to extract the full-image features and apply a fusion module to full object-level and image-level features to predict the final colors. Both colorization networks and fusion modules are learned from a large-scale dataset. Experimental results show that our work outperforms existing methods on different quality metrics and achieves state-of-the-art performance on image colorization.
1. Introduction
Image colorization is ill-posed and multimodal, while existing learning-based methods struggle with multiple objects in cluttered backgrounds. The paper proposes instance-aware colorization using object-level and full-image features, with experiments across datasets and baselines.
- Motivation: Colorizing grayscale images is ill-posed because multiple plausible colors can correspond to the same object.Applications include legacy photo or video restoration and image compression.
- Limitations: Existing learning-based colorization methods perform poorly on images containing multiple objects in cluttered backgrounds.Figure 2 reports implausible colors for skiers and vehicles, including context confusion toward green.
- Proposed approach: Instance-aware colorization separates objects from backgrounds to learn object-level representations and reduce color confusion.The framework uses an object detector, separate instance and full-image networks, and a feature-fusion module.
- Evaluation: The method is evaluated on ImageNet, COCO-Stuff, and Places205 against existing colorization methods.The experiments report comparisons using quality metrics across all three datasets.
- Contributions: The authors present a fully automatic method and claim state-of-the-art performance in image colorization.The stated contributions include automatic instance-aware colorization, the fusion architecture, and comprehensive baseline comparisons.
2. Related Work
Prior colorization methods range from user-guided and reference-based techniques to deep learning systems modeling semantics and multimodality. This paper argues that object-level semantics are needed for appearance variation and extends instance-aware processing to multiple-instance scenes.
- Scribble-based colorization: Scribble-based methods propagate user-specified colors under low-level similarity constraints, but require manual guidance.They use color points or strokes to constrain colorization.
- Example-based colorization: Reference-based methods transfer color statistics from a user-specified or retrieved image, with performance depending on reference similarity.Finding a suitable reference remains non-trivial, and some methods still require manual region annotations.
- Learning-based colorization: Deep convolutional networks learn color prediction from large-scale datasets and address image semantics and color multimodality.Proposed strategies include local-global feature fusion, semantic interpretability, and per-pixel color distributions.
- Learning-based colorization: The paper observes that image-level or pixel-level semantics do not sufficiently model object appearance variations.Its approach learns semantics from cropped object images and fuses object-level with full-image features.
- Instance-aware image synthesis and manipulation: Instance-aware synthesis provides figure-ground separation for visual generation and manipulation, while this method targets complex scenes with multiple instances.The paper distinguishes its focus from methods handling only a single instance.
3. Overview
The model detects object instances in a grayscale image, colorizes cropped instances and the full image through parallel networks, then fuses their features layer by layer. Sequential training produces globally consistent color predictions.
- Input and output: The system accepts a grayscale image X and predicts its two missing color channels Y in CIE L*a*b* space.The prediction is performed end to end.
- Instance extraction: An off-the-shelf object detector produces bounding boxes, which are used to crop and resize grayscale instance images.The number of detected instances is denoted by N.
- Dual branches: Instance and full-image colorization networks process cropped instances and the original grayscale image using the same architecture with different weights.Their feature maps are extracted at corresponding network layers.
- Training: Training proceeds sequentially by training the full-image network, instance network, and fusion module in that order.The first two networks are frozen while the feature fusion module is trained.
- Feature fusion: A fusion module combines all instance feature maps with full-image features at each layer before forwarding the fused representation.The resulting representation is used to predict the color image Y and maintain global consistency.
4. Method
The method combines full-image and detected-instance colorization through a learned fusion module. Instance features and weight maps are resized, zero-padded, and combined with full-image features to produce colorization.
- Instance extraction: An off-the-shelf object detector supplies bounding boxes for cropping grayscale instance images, which are resized for instance colorization.The method uses detected object regions as inputs to its instance-processing branch.
- Dual colorization branches: Two same-depth colorization branches separately process each instance image and the full grayscale image to extract compatible features.The branches share architecture but use different weights, facilitating feature fusion across corresponding layers.
- Feature fusion: The fusion module predicts full-image and per-instance weight maps from their features using small three-layer convolutional networks.Bounding boxes determine how instance features and weight maps are aligned with the full-image representation.
- Feature fusion: Instance features and weight maps are resized and zero-padded to the full-image size before weighted summation produces the fused feature.The operation is applied at multiple layers of the colorization networks.
- Training: Training proceeds sequentially by learning the full-image network, initializing and training the instance network, then freezing both while training fusion.The method adopts the smooth-ℓ1 loss with δ = 1.
5. Experiments
Experiments evaluate instance-aware colorization across three datasets, quantitative benchmarks, user preferences, visual comparisons, ablations, runtime, legacy photographs, and failure cases. Results show improved quality, especially for complex multi-instance scenes, while missed detections can cause artifacts.
- Experimental setting: The evaluation uses ImageNet, COCO-Stuff, and Places205, with PSNR and SSIM reported under established experimental protocols.ImageNet and COCO-Stuff support training or evaluation, while Places205 assesses transferability.
- Quantitative comparisons: The method performs favorably against recent colorization methods across all three datasets at the full-image level.The comparison includes models trained on ImageNet and variants finetuned on COCO-Stuff.
- Quantitative comparisons: Instance-level evaluation on COCO-Stuff shows a significant performance boost in all reported metrics.The evaluation averages results over instances formed using ground-truth bounding boxes.
- User study: In a 2,400-vote user study, the method is preferred over Zhang et al. at 61% versus 39% and DeOldify at 72% versus 28%.The study uses 100 COCO-Stuff validation images and 24 participants.
- Ablation study: Ablations find that fusing instance and full-image features at both encoder and decoder levels performs best, while the default top-eight detected boxes slightly outperform ground-truth boxes.The fusion module is also reported to improve visual quality in complex scenarios.
- Runtime analysis: Inference averages 0.187s per 256 × 256 image, with instance processing ranging from 0.013s for one instance to 0.1s for eight instances.The two colorization stages each account for approximately half of total runtime, and step-one complexity grows with the number of instances.
- Failure modes: When objects are missed during detection, the model falls back to full-image colorization and may produce washed-out colors or bleeding across object boundaries.The fusion module may also become confused when many object bounding boxes overlap.
6. Conclusions
The paper presents instance-aware image colorization that combines object-level and full-image features through a fusion module. Extensive experiments compare favorably with existing methods on three benchmark datasets.
- The method uses an off-the-shelf object detector to crop instances before extracting instance-branch features.
- A full-image branch and instance branch are fused by a proposed module to produce feature maps for color prediction.
- The method compares favorably against existing methods on ImageNet, COCO-Stuff, and Places205.
A. Visualization of Fusion Module
The fusion module is visualized on images containing multiple detected instances. Its weighted masks adaptively blend instance-branch and full-image features to support coherent image-wide colorization.
- The visualization uses images in which the object detector identifies multiple instances.
- Weighted masks are shown at the 3rd, 7th, and 10th fusion-network layers.
- The fusion module adaptively blends instance-branch features to enforce coherent colorization across the entire image.
B. Extended quantitative evaluation
The extended evaluation examines additional quantitative comparisons, including backbone models with different weights on COCOStuff. Because the original backbone setting does not match this task, the model is retrained for fully automatic 256×256 colorization.
- The section reports two additional quantitative evaluations.
- The original backbone is trained for guidance colorization at 176×176, whereas this setting requires fully automatic colorization at 256×256.
- The backbone is retrained on ImageNet before quantitative evaluation on COCOStuff.
- Table 4 compares the colorization backbone using different model weights on COCOStuff.
- Figure 11 presents grayscale, original-weight, ImageNet-retrained, and COCOStuff-fine-tuned outputs from model [41].
C. User Study setup
The user study uses no-reference pairwise comparisons to assess color correctness and naturalness. Redundant comparisons are included as sanity checks, and votes are rejected under a stated subject-quality condition.
- Subjects first read webpage instructions and selection criteria based on color correctness and naturalness.
- Subjects then perform pairwise forced-choice comparisons without ground-truth or grayscale references.
- Two redundant comparisons are embedded for sanity checking, and votes are rejected when subjects fail the study condition.
D. Failure cases
The model improves colorization overall but still struggles with out-of-distribution images, particularly when predicting vibrant, bright colors from grayscale inputs.
- The model cannot predict vibrant, bright colors in some grayscale images.These examples are presented as failure cases, including out-of-distribution images for which the model still produces plausible colors.
- Single-image colorization remains a challenging problem despite the reported improvements.
- Figure 13 compares grayscale inputs, ground-truth color images, and the model’s results.