PaperlayerView exact revision on arXiv

Source-linked AI summary

Image Colorization: A Survey and Dataset

Saeed Anwar, Muhammad Tahir, Chongyi Li, Ajmal Mian, Fahad Shahbaz Khan, Abdul Wahab Muzaffar

arXiv:2008.10774v4cs.CVcs.AIcs.LGeess.IV

TL;DR

Colorization must infer missing RGB values from grayscale images, and progress in deep-learning methods calls for systematic synthesis and benchmarking. This paper surveys and categorizes recent methods, introduces a colorization-specific dataset, and evaluates existing methods. It reports broad trends in model diversity, complex-scene performance, and gains associated with higher model complexity.

  • Problem

    Deep colorization methods have advanced, but the paper identifies a lack of a comprehensive survey and notes that existing evaluation datasets were not designed specifically for colorization.

    Source: §1 · p. 2

  • Method

    The paper surveys and categorizes deep colorization methods, introduces the Natural-Color Dataset, and evaluates state-of-the-art models on it.

    Source: §1 · p. 2

  • Results

    The survey reports that GAN methods produce more visually diverse colorizations than CNN methods, complex scenes often remain suboptimal, and higher model complexity brings slight numerical improvement.

    Source: §6 · p. 25

  • Takeaways & Limitations

    The authors identify unsupervised learning as a future direction and suggest attention mechanisms and loss functions as techniques that could be incorporated for performance.

    Source: §6 · p. 25

    Existing evaluation datasets were collected for other tasks, and their image quality may be insufficient for colorization.

    Source: §5 · p. 24

Abstract

from arXiv · show

Image colorization estimates RGB colors for grayscale images or video frames to improve their aesthetic and perceptual quality. Over the last decade, deep learning techniques for image colorization have significantly progressed, necessitating a systematic survey and benchmarking of these techniques. This article presents a comprehensive survey of recent state-of-the-art deep learning-based image colorization techniques, describing their fundamental block architectures, inputs, optimizers, loss functions, training protocols, training data, etc. It categorizes the existing colorization techniques into seven classes and discusses important factors governing their performance, such as benchmark datasets and evaluation metrics. We highlight the limitations of existing datasets and introduce a new dataset specific to colorization. We perform an extensive experimental evaluation of existing image colorization methods using both existing datasets and our proposed one. Finally, we discuss the limitations of existing methods and recommend possible solutions and future research directions for this rapidly evolving topic of deep image colorization. The dataset and codes for evaluation are publicly available at https://github.com/saeed-anwar/ColorSurvey.

1. Introduction

Image colorization must infer missing colors across varying imaging conditions, while also contending with illumination changes, viewpoint variation, and occlusion. The paper surveys deep colorization, organizes its methods, and introduces a dataset intended specifically for colorization evaluation.

1. Introduction

  • Colorization is severely ill-posed because two of three image dimensions are missing, and a single algorithm must also handle varied imaging conditions, illumination, viewpoints, and occlusions.Scene semantics can help in cases such as grass, clouds, and sky, but the paper notes that such priors are uncommon for many manufactured and natural objects.Source: §1 · p. 1
  • The paper reviews settings, metrics, and datasets; proposes a network taxonomy; introduces the Natural-Color Dataset; and evaluates state-of-the-art models on it.It also summarizes network components and discusses challenges and future directions.Source: §1 · p. 2
  • The authors report that no survey covered recent colorization developments and provide a comprehensive overview of deep-learning methods.They aim to contextualize existing research, identify gaps, and point to future focus areas.Source: §1 · p. 2
  • Existing evaluation datasets were not built for colorization and may contain subjective or false colors, so evaluations may not accurately represent algorithms.The paper gives blue grass as an example of false color.Source: §1 · p. 2
  • Colorization commonly predicts only U and V in YUV space rather than all three RGB channels.The paper frames historical colorization as enhancement and also notes grayscale conversion as a way to save storage or communication bandwidth.Source: §1 · p. 2

2. Single-Image Deep Colorization

The paper groups single-image deep colorization networks by structural differences, input type, user assistance, and related factors. It notes that some methods fit multiple categories but assigns each to the most appropriate one.

2. Single-Image Deep Colorization

  • The taxonomy classifies colorization networks by structural differences, input type, user assistance, and other factors, assigning methods that fit multiple categories to the most appropriate one.The section also presents category strengths and weaknesses in Table 1.Source: §2 · p. 3

2.1. Plain Networks

The section reviews plain colorization networks, from early CNN approaches to depth colorization and U-Net methods. These approaches vary in architecture and inputs, and the reported comparisons include a depth-colorization model that outperforms several alternatives on three benchmarks.

2.1. Plain Networks

  • Early plain networks stacked convolutional layers with no or naive skip connections; they are straightforward to implement but learn slowly and require more training data.Deep Colorization added joint bilateral filtering as post-processing to remove CNN artifacts.Source: §2.1 · p. 3, §2.1.1 · p. 4

2.1.1. Deep Colorization

  • Deep Colorization uses five fully connected layers to predict U and V values from concatenated grayscale-patch and DAISY features, then applies joint bilateral filtering to remove artifacts.Inference also uses low-level gray values and high-level semantic labeling.Source: §2.1.1 · p. 4

2.1.2. Colorful Colorization

  • Colorful Colorization predicts 313 “ab” color pairs and reweights its loss by pixel rarity to reduce the influence of common low-“ab” values.The network stacks eight convolutional blocks and uses striding instead of pooling to reduce image size.Source: §2.1.2 · p. 4

2.1.3. Deep Depth Colorization

  • (DE)2CO outperforms CaffeNet, VGG16, GoogleNet, and ResNet50 under the same settings on Washington-RGBD, JHUIT50, and BigBIRD.For an unseen dataset, only the last convolutional layer is retrained while the other layers remain frozen.Source: §2.1.3 · p. 5

2.1.4. U-Net Grayscale Image Colorization

  • Hu et al. convert images to Lab, quantize grayscale levels with Heckbert’s median cut method, and use U-Net to colorize from the quantized L component.The method achieved 84.81% accuracy on ImageNet.Source: §2.1.4 · p. 5

2.2. User-guided networks

User-guided networks accept user-provided color input, which can help with ambiguous regions but increases effort and limits scalability. The section describes approaches using sketches and strokes, local and global hints, interactive inputs, and anime line-art hints.

2.2. User-guided networks

  • User-guided colorization can improve control over complex or ambiguous regions, but requires user effort and knowledge, can be time-consuming, and scales poorly to large datasets.Different users may produce inconsistent outputs for similar photos.Source: §2.2 · p. 5

2.2.1. Scribbler

  • Scribbler uses sketches and color strokes to guide structural information and color patterns in an end-to-end GAN, with adversarial loss intended to produce more realistic colorizations.Its performance was assessed qualitatively on faces, cars, and bedrooms.Source: §2.2.1 · pp. 5–6

2.2.2. Real-Time User-Guided Colorization

  • Real-Time User-Guided Colorization uses local hints to process user input into a color distribution and global hints to accept a histogram and average image saturation.The two variants share a main colorization branch and use Huber and regression losses.Source: §2.2.2 · p. 6

2.2.3. Interactive Deep Colorization

  • Interactive Deep Colorization combines a grayscale image, local input, and gradient map, then combines global input and reconstructs the colorized output.The model uses a modified Huber loss and was trained with ImageNet and Places images.Source: §2.2.3 · p. 7

2.2.4. Anime Line Art Colorization

  • Anime line-art colorization uses user hints and grayscale line art with a conditional GAN whose generator adopts a U-Net-based architecture.Its generator uses perceptual and adversarial losses, while the discriminator combines Wasserstein critic and penalty losses.Source: §2.2.4 · p. 7

2.3. Domain-Specific Colorization

Domain-specific colorization targets modalities such as infrared and radar, and its networks exploit domain-specific patterns while often generalizing poorly beyond their training domain. The section surveys infrared, SAR, radar, and sketch-colorization approaches.

2.3. Domain-Specific Colorization

  • Domain-specific models can outperform general-purpose models by learning domain-specific subtleties, but often generalize poorly beyond their trained domain and require high-quality domain-specific data.The survey notes that such data may only occasionally be available.Source: §2.3 · p. 7

2.3.1. Infrared Colorization

  • Infrared colorization uses multi-branch CNNs on normalized image-pyramid levels, merges branch outputs, then applies joint bilateral filtering and restores high-frequency input information.The described real-world dataset contains road scenes captured with a multi-CCD NIR/RGB camera; the model was trained on about 32 pairs and tested on 800 images.Source: §2.3.1 · pp. 7–8

2.3.2. SAR-GAN

  • SAR-GAN cascades despeckling and colorization subnetworks to turn speckle-contaminated SAR inputs into visible-color images.Its evaluation included 85 of 3,292 synthetic images and real SAR images.Source: §2.3.2 · pp. 8–9

2.3.3. Radar Image Colorization

  • For single-polarization radar images, Song et al. feed VGG16-derived hyper-column descriptors into a five-layer feature translator that predicts nine polarimetric groups.The feature extractor and translator are trained using full-polarimetric radar image patches.Source: §2.3.3 · p. 9

2.3.4. Sketch Image Colorization

  • Reference-based sketch colorization uses a colored reference to guide the more information-scarce sketch, transferring spatially corresponding features before decoding the colored output.Training references are generated through color perturbation and geometric distortion, with thin plate splines applied to the reference image.Source: §2.3.4 · p. 9

2.4. Text-based Colorization

Text-based colorization adds text to a grayscale image so users can specify color schemes and preferences, though complex instructions may yield inconsistent results. The section describes language-conditioned models and Text2Colors, which separates palette generation from image colorization.

2.4. Text-based Colorization

  • Text-based colorization combines text with a grayscale image to specify color schemes and preferences, but complex instructions can reduce consistency and reliability.More precise descriptions may be needed.Source: §2.4 · p. 9, Figure 5 · p. 10

2.4.1. Learning to Color from Language

  • Among FCNN, CONCAT, and FILM trained on MS-COCO, FILM achieved the highest automatic accuracy, and crowd-sourced assessments validated the models’ effectiveness.The network output is a 56×56 color image.Source: §2.4.1 · p. 10

2.4.2. Text2Colors

  • Text2Colors uses TPN to generate a palette from text and PCN to colorize the grayscale image using that palette.TPN learns text–palette pairings on the PAT dataset, which contains five palettes for each of 10,183 textual phrases.Source: §2.4.2 · p. 10
  • In Text2Colors, TPN’s generator learns text–palette pairings while its discriminator distinguishes real from generated palettes; PCN combines U-Net colorization with palette conditioning.The networks are trained in stages, with TPN trained on PAT for 500 epochs and then on ground-truth images for 100 epochs.Source: §2.4.2 · p. 10

2.5. Diverse Colorization

Diverse colorization targets multiple plausible outputs rather than exact restoration, using GANs and VAEs to explore color alternatives. The surveyed approaches vary in how they introduce diversity, semantic or exemplar information, and refinement.

2.5. Diverse Colorization

  • GANs and VAEs can generate varied colors, but diversity can create ambiguity, inconsistency, and added computational cost when an object's color is consistent.The survey notes that selecting the most appropriate output can be challenging.Source: §2.5 · p. 11

2.5.1. Unsupervised Diverse Colorization

  • Cao et al. condition a GAN on grayscale images and add noise to early generator layers; its LSUN bedroom outputs received a 62.6% convincing rate versus 70% for ground-truth images.A significance t-test reported p = 0.1359, indicating no significant difference between generated and real images.Source: §2.5.1 · p. 11

2.5.2. Tandem Adversarial Networks

  • Frans [23] uses tandem color-prediction and shading networks, with the first predicting a color scheme and the second rendering it with the outline.The color-prediction network uses ℓ2 loss, while the shading network uses adversarial loss.Source: §2.5.2 · p. 11

2.5.3. ICGAN

  • ICGAN feeds grayscale images to its generator and changes the generator cost to maximize the discriminator's probability of being incorrect; its visual results are favorable to traditional CNNs.The survey reports evaluation on CIFAR10 and Places356.Source: §2.5.3 · p. 12

2.5.4. Learning Diverse Image Colorization

  • Deshpande et al. combine a VAE embedding of a 64 × 64 × 2 color field with an MDN to generate multiple colorizations for one grayscale image.At test time, embeddings sampled from the MDN are decoded by the VAE; the design uses specificity, colorfulness, and gradient losses.Source: §2.5.4 · pp. 13–14

2.5.5. ChromaGAN

  • ChromaGAN combines chrominance and class-distribution branches to use geometric, perceptual, and semantic features for realistic rather than merely aesthetic colorization.It is trained adversarially on 1.3M ImageNet images for five epochs.Source: §2.5.5 · p. 14

2.5.6. MemoPainter: Coloring with Limited Data

  • MemoPainter uses external memory to retain rare examples and condition a GAN colorizer on retrieved color information, aiming to avoid dominant-color effects and preserve object color identity.Its memory stores spatial keys, color values, and entry ages; testing retrieves color information as a generator condition.Source: §2.5.6 · p. 14

2.5.7. CycleGAN based Image Colorization

  • SS-CycleGAN uses paired grayscale-to-color and color-to-grayscale generators, while Shafiq and Lee combine global VGG features, color encoding, and transformer-based feature fusion.SS-CycleGAN adds self-attention and detail loss; the transformer approach uses perceptual, adversarial, and color losses.Source: §2.5.7 · p. 15, §2.5.8 · p. 15

2.6. Multi-path Colorization Networks

Multi-path colorization networks combine features from branches or tasks to model visual context at different levels. The approaches include scene and semantic guidance, autoregressive color generation, multiple hypotheses, and separate shadow handling.

2.6.1. Let there be Color

  • Iizuka et al. jointly train colorization and scene-classification branches, combining low-, mid-, and global-level features to predict chrominance and classify the scene.The global subnet computes a 256D image vector, while the colorization network restores the original resolution.Source: §2.6.1 · pp. 15–16

2.6.2. Learning Representations for Automatic Colorization

  • Learning Representations for Automatic Colorization extracts VGG16 hypercolumns for each pixel and predicts hue and chroma distributions over color bins.Its 12k-channel descriptors feed a 1k-channel fully connected layer, and KL divergence is used as the loss.Source: §2.6.2 · p. 16

2.6.3. PixColor

  • PixColor uses conditional PixelCNN to produce low-resolution color, then a refinement CNN combines that image with the grayscale input to produce a high-resolution result.PixelCNN predicts each pixel's color based on previous pixels; the refinement network is trained on downsampled ground-truth chroma images.Source: §2.6.3 · p. 16

2.6.4. Colorize Capsule Network

  • ColorCapsNet [60] achieves performance comparable to other models despite its shallow architecture, after training on ILSVRC 2012 and fine-tuning on DIV2K.The model uses mean squared error loss and Adam with a learning rate of 0.001.Source: §2.6.4 · p. 18

2.6.5. Pixelated

  • Pixelated combines autoregressive pixel color distributions with semantic segmentation and color embedding to produce diverse, category-consistent colorizations.Its semantic segmentation branch uses multi-scale features, and the fused semantic generator is evaluated on Pascal VOC2012 and COCO-stuff.Source: §2.6.5 · p. 18

2.6.6. Multiple Hypothesis Colorization

  • Multiple Hypothesis Colorization uses a shared convolutional trunk and multiple branches to predict pixel colors, storing the best pixel-level hypothesis for low-cost colorization.The survey describes separate CIFAR100 and ImageNet architectures with residual connections.Source: §2.6.6 · pp. 18–19

2.6.7. Pixel-Level Semantics Guided Image Colorization

  • Pixel-Level Semantics Guided Colorization jointly optimizes semantic segmentation and colorization branches, then applies joint bilateral upsampling at test time.The branches share four initial convolutional layers; the segmentation and colorization tasks use different cross-entropy losses.Source: §2.6.7 · p. 19

2.6.8. Shadow-Aware Image Colorization

  • The shadow-aware network separates shadow and non-shadow colorization, using shadow-specific information to preserve saturation and diversity in non-shadow regions.Its two branches extract shadow-detection and shadow-aware color features, sharing features between their encoders.Source: §2.6.8 · p. 19

2.7. Exemplar-based Colorization

Exemplar-based methods use reference colors to guide grayscale colorization, but their effectiveness depends on finding suitable examples. The surveyed methods transfer exemplar colors, accelerate that transfer, or combine instance and full-image processing.

2.7.1. Deep Exemplar-based Colorization

  • Deep exemplar-based colorization transfers colors from a reference image using a similarity network and a colorization network, aiming for diverse rather than naturally accurate colors.The similarity network aligns target and reference luminance and computes bidirectional similarity maps for the colorization network.Source: §2.7.1 · p. 20

2.7.2. Fast Deep Exemplar Colorization

  • Fast deep exemplar colorization uses a transfer subnetwork to make a coarse chrominance map and a colorization subnetwork to refine it, targeting real-time results regardless of semantic relation between target and exemplar.The transfer network uses AdaIN for feature matching and blending; its initial map can contain artifacts that the second network refines.Source: §2.7.2 · p. 20

2.7.3. Instance-Aware Image Colorization

  • Instance-aware colorization combines cropped-object and full-image networks through feature fusion, but the Natural-Color Dataset comparisons show many methods fail to reproduce fruit and vegetable colors.The method detects instances, colorizes crops and the full image separately, then fuses their features.Source: §2.7.3 · pp. 20–21, Figure 10 · p. 21, Figure 11 · p. 22

3. Experiments

The experiments section describes commonly reused evaluation datasets and the metrics used to assess colorization. It notes limitations in both human judgments and mathematical measures.

3.1. Datasets

  • Colorization evaluations reuse datasets from other tasks; Places205 has 20,500 test images in 205 categories and tests transferability without a training partition; Table 2 compares PSNR, SSIM, and LPIPS on ImageNet, COCOStuff, and Places205.Source: §3.1 · p. 22, Table 2 · p. 23

3.2. Evaluation Metrics

  • Human judgments are considered the gold standard for many applications, but scaling them and determining accurate colors are difficult; general mathematical metrics may also misrepresent algorithm performance.The survey names PSNR and SSIM as commonly used mathematical metrics.Source: §3.2 · p. 23

4. Comparisons

On existing datasets, methods report similar SSIM and PSNR, while instance-aware colorization performs better in LPIPS. On NCD, different methods lead on different metrics, and qualitative comparisons still show inaccurate colorization.

Existing Datasets Comparisons

  • On Places205, ImageNet ILSVRC2012, and COCO-stuff, methods have broadly similar SSIM and PSNR, while instance-aware colorization performs better in LPIPS.The passage describes the metric values as more or less similar for SSIM and PSNR.Source: §4 · p. 23

NCD Comparisons

  • 21.93 dB PSNR and 0.881 SSIM are reported for Real-time colorization on NCD; instance-aware colorization leads in PCQI and colorful colorization in IQM.Table 3 compares methods across PSNR, SSIM, PCQI, and IQM; higher values indicate better performance.Source: §4 · p. 23, Table 3 · p. 24
  • Most methods fail to recover natural colors in the NCD fruit and vegetable comparisons, although Real-Time and instance-aware methods are more consistent and closer to original colors.The authors still describe these methods as far from accurate colorization performance.Source: §4 · p. 24

5. Limitations and Future Directions

The paper identifies mismatched evaluation metrics and datasets as limitations for colorization assessment, and describes NCD as a purpose-built alternative. It also points to a lack of competitions and limited availability of open-source code.

Lack of Appropriate Evaluation Metrics

  • Metrics inherited from image restoration may not suit colorization; the authors recommend comparing predicted U and V channels and using color-aware metrics such as PCQI and UIQM.PCQI considers patch mean intensity, contrast change, and structure distortion; UIQM uses image colorfulness, sharpness, and contrast.Source: §5 · p. 24

Lack of Benchmark Dataset

  • Datasets created for other tasks can contain objects with arbitrary colors, making them inappropriate for fair PSNR or SSIM comparisons.The passage notes that such datasets were not originally collected for colorization and that their image quality may be insufficient.Source: §5 · pp. 24–25
  • NCD comprises 723 internet-collected images across 20 categories, with one object and a white background per image to assess color bias and color spill.The collection is intended to use images true to their colors and provide a realistic evaluation scenario.Source: §5 · p. 25

Lacking of Competitions; Limited Availability of Open-Source Codes

  • The paper identifies the absence of colorization competitions and the rarity or obsolescence of open-source implementations as field-level challenges.It notes that code is not routinely recycled or reimplemented for newer frameworks and environments.Source: §5 · p. 25

6. Conclusion

The paper reviews deep-learning methods for image colorization and identifies performance gains alongside practical constraints. Its synthesis highlights model trends and future directions.

6. Conclusion

  • Deep-learning colorization performance has improved in recent years at the cost of increasing network complexity.The paper also says inadequate metrics and failure to handle real-life degradations restrict application to critical real-world scenarios.Source: §6 · p. 25
  • The review finds that GAN-based methods provide more visually diverse colorization than CNN-based methods, while complex scenes with many small objects remain challenging.It also reports only slight numerical improvement from higher-complexity deep models.Source: §6 · p. 25
  • The paper identifies unsupervised learning as a future direction and suggests attention mechanisms and loss functions as techniques that could be incorporated.The article presents itself as a thorough review of deep-learning methods for image colorization.Source: §6 · p. 25

Related papers

Loading 2008.10774v4…