PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Colorization

Zezhou Cheng, Qingxiong Yang, Bin Sheng

arXiv:1605.00075v1cs.CV

TL;DR

The paper addresses the limitations of manual scribbles, carefully selected references, and noisy large-scale patch matching in grayscale colorization. It proposes a fully automatic deep-learning method with semantic features, adaptive image clustering, and post-processing, producing visually similar results with almost no visible artifacts and fast inference.

  • Problem

    Grayscale colorization often requires substantial user scribbles or similar reference images, while large reference databases introduce excessive matching noise.

  • Method

    The method uses deep learning with informative feature descriptors, semantic awareness, adaptive image clustering, and joint bilateral filtering post-processing for fully automatic colorization.

  • Results

    The method produces color images with almost no visible artifacts, visually similar to ground truth, and colorizes a 256×256 image in 6.780 seconds in Matlab.

  • Takeaways & Limitations

    Deep learning provides a fully automatic colorization approach that improves quality while avoiding the massive-candidate matching required by slower state-of-the-art methods.

  • Takeaways & Limitations

    The model requires a huge reference database covering all possible objects, and the current model trained on real images is invalid for synthetic images.

Abstract

from arXiv · show

This paper investigates into the colorization problem which converts a grayscale image to a colorful version. This is a very difficult problem and normally requires manual adjustment to achieve artifact-free quality. For instance, it normally requires human-labelled color scribbles on the grayscale target image or a careful selection of colorful reference images (e.g., capturing the same scene in the grayscale target image). Unlike the previous methods, this paper aims at a high-quality fully-automatic colorization method. With the assumption of a perfect patch matching technique, the use of an extremely large-scale reference database (that contains sufficient color images) is the most reliable solution to the colorization problem. However, patch matching noise will increase with respect to the size of the reference database in practice. Inspired by the recent success in deep learning techniques which provide amazing modeling of large-scale data, this paper re-formulates the colorization problem so that deep learning techniques can be directly employed. To ensure artifact-free quality, a joint bilateral filtering based post-processing step is proposed. We further develop an adaptive image clustering technique to incorporate the global image information. Numerous experiments demonstrate that our method outperforms the state-of-art algorithms both in terms of quality and speed.

I. INTRODUCTION

Image colorization traditionally requires user scribbles or suitable reference images, while large databases introduce matching noise. The paper proposes a fully automatic deep-learning approach using semantic features, adaptive clustering, and extensive reference images.

  • Motivation: Scribble-based methods require substantial user effort, especially for grayscale images containing fine-scale structures.
  • Motivation: Example-based methods reduce manual scribbling but depend on finding suitable reference images, with no standard criterion for selecting them.
  • Motivation: Large reference databases offer a more reliable matching source in principle, but matching noise becomes too high in practice as database scale increases.
  • Proposed approach: The method uses deep learning to model large-scale colorization data and introduces semantic feature descriptors to incorporate image-level understanding.
  • Proposed approach: Adaptive image clustering incorporates global image information and reduces training ambiguities by training neural networks for image clusters.
  • Contribution: The proposed fully automatic method uses diverse reference images and is reported to provide artifact-free quality and high speed across different scenes.

III. OUR METRIC

The paper formulates colorization as learning a gray-to-color mapping from reference image pairs. Its pipeline clusters references, trains a neural-network ensemble, predicts chrominance from pixel descriptors, and refines the result before recombining it with luminance.

  • III. OUR METRIC: A separate deep neural network is constructed and trained for each image cluster, producing a trained neural-network ensemble.
  • III. OUR METRIC: Training uses grayscale–color reference pairs, adaptive global clustering, semantic histograms, sampled feature descriptors, and corresponding chrominance values.
  • III. OUR METRIC: For a target grayscale image, the nearest cluster and network are selected, pixel descriptors are processed, chrominance is refined, and luminance is combined with it to produce color.
  • III. OUR METRIC: Colorization is formulated as learning a function that maps per-pixel feature descriptors from grayscale images to corresponding chrominance values.
  • III. OUR METRIC: The mapping parameters are learned by minimizing a least-squares objective over sampled training pixels.

2) Architecture:

The architecture uses a fully connected deep neural network whose input dimension matches the pixel descriptor and whose two-neuron output predicts the U and V chrominance channels.

  • 2) Architecture:: The network contains one input layer, multiple hidden layers, and one output layer with fully connected successive layers.
  • 2) Architecture:: The input-layer width equals the feature-descriptor dimension extracted at each grayscale-image pixel.
  • 2) Architecture:: The output layer has two neurons that predict the U and V channels of the corresponding color value.
  • 2) Architecture:: Connections between neurons carry learned weights, while bias units provide a constant value and activation functions transform layer outputs.
  • 2) Architecture:: The model uses ReLU activation and classical error back-propagation to train its connection weights.

B. Feature Descriptor

The method combines low-, mid-, and high-level image features into a pixel descriptor because feature design strongly affects colorization quality. These descriptors are selected experimentally for practical impact.

  • B. Feature Descriptor: Feature design is identified as key to the success of the colorization method.
  • B. Feature Descriptor: The adopted features are organized into low-, mid-, and high-level descriptors extracted at each pixel location.
  • B. Feature Descriptor: The authors evaluate many candidate features and retain those with practical effects on colorization results.
  • B. Feature Descriptor: The pixel descriptor is constructed by concatenating descriptors from different feature levels.

1) Low-level Patch Feature:

The method combines a 7×7 grayscale patch descriptor with DAISY to distinguish pixels across simple and fine-structure regions. Patch features help low-texture areas, while DAISY improves detail in complex regions but can introduce low-texture artifacts.

  • 1) Low-level Patch Feature:: A 7×7 grayscale patch centered at each pixel provides the low-level patch feature descriptor.The patch captures neighboring grayscale values, making pixels more distinguishable than luminance alone.
  • 1) Low-level Patch Feature:: Patch features outperform traditional SIFT and DAISY features in low-texture regions such as sea.Without the patch feature, the model can assign one color across an entire semantic region despite intensity variation.
  • 2) Mid-level DAISY Feature:: DAISY is a fast local descriptor that more accurately describes local patches and improves colorization quality in complex scenarios.The paper uses DAISY as a mid-level feature for fine-structure regions.
  • 2) Mid-level DAISY Feature:: Adding DAISY produces more detailed and accurate colorization on fine-structure objects, but reduces performance around low-texture regions.The final result applies artifact removal after DAISY introduces artifacts in low-texture areas.

3) High-level Semantic Feature:

The method adds semantic descriptors and adaptive image clustering to reduce ambiguity from local matching and incorporate global image information. Joint bilateral filtering refines noisy chrominance, while clustering organizes reference images into progressively smaller training groups.

  • 3) High-level Semantic Feature:: A semantic feature expresses each pixel’s category, such as sky, sea, or animal, alongside patch and DAISY features.The method uses scene parsing to produce category probabilities for each pixel and smooths the semantic map with an edge-preserving filter.
  • 3) High-level Semantic Feature:: Semantic information significantly reduces matching and training ambiguities by restricting candidate colors to those associated with the detected category.For example, pixels detected as sky use sky color values from the reference database, producing visually better colorization.
  • 4) Chrominance Refinement:: Joint bilateral filtering smooths and refines neural-network chrominance using the noise-free grayscale image as guidance.The refinement is designed to reduce artifacts around low-texture regions.
  • 4) Chrominance Refinement:: Most visible artifacts can be successfully removed after chrominance refinement.Figure 5 compares the colorization result before and after refinement.
  • C. Adaptive Image Clustering: Adaptive image clustering improves colorization performance by organizing reference images according to global information.The technique is presented as an adaptive clustering procedure for improving the proposed method.
  • C. Adaptive Image Clustering: Reference images are clustered adaptively across layers, and images with sufficiently low training error are removed from lower-layer training sets.The training error is measured using negative PSNR; lower layers therefore contain fewer reference images.
  • C. Adaptive Image Clustering: The minimum reference-image count for training a DNN depends on the network’s weights and the samples contributed by each reference image.The paper denotes this minimum count as µ and defines α, Nw, and Ns in the associated equation.

1) Semantic Histogram:

Semantic histograms refine cluster selection by combining global similarity with object-composition similarity, avoiding semantically mismatched clusters that can produce unnatural colors.

  • Semantic Histogram: Global image features can select a cluster that is visually similar but semantically different from the target.For example, global-only searching selects a building scene for the target in Figure 7(a), producing unnatural colorization.
  • Semantic Histogram: Semantic histograms encode the percentage of pixels belonging to predefined object categories.The histogram has one bin per object category and represents each category's pixel proportion.
  • Semantic Histogram: The method first selects top-k clusters by global-descriptor distance, then chooses among them using cosine similarity of semantic histograms.This two-stage search targets clusters that are both globally and semantically similar.
  • Semantic Histogram: Incorporating the semantic histogram can significantly change colorization performance compared with global-feature searching alone.Figure 7 compares results without semantic histograms and after their incorporation.

2) The Evaluation of Image Clustering:

Adaptive image clustering improves colorization by incorporating global image information, while large reference databases and learned matching reduce dependence on carefully selected references and support fast inference.

  • The Evaluation of Image Clustering: Image clustering improves colorization accuracy and significantly reduces visible artifacts, especially for objects with large color variances.The evaluation covers the PSNR distribution of 1519 test images and qualitative comparisons.
  • The Evaluation of Image Clustering: A large reference image database makes the method less dependent on a single suitable reference image.The model finds similar pixels in the database and transfers their colors to target pixels.
  • The Evaluation of Image Clustering: Using multiple reference images can improve example-based colorization, but feature-matching noise can still create unnatural correspondences.Reported mismatches include a lake matched to a door and sky matched to a building.
  • The Evaluation of Image Clustering: The deep neural network combines varied pixel features to compute corresponding chrominance values for large reference databases.This learned approach avoids relying solely on massive candidate searches during colorization.
  • The Evaluation of Image Clustering: Colorizing a 256×256 grayscale image takes 6.780 seconds in Matlab after training, while training is slow for large databases.The paper contrasts this inference cost with the slow candidate-matching process of state-of-the-art methods.
  • The Evaluation of Image Clustering: Compared with Deshpande et al., the proposed model learns the mapping automatically and requires no spatial prior, while Deshpande et al. uses a scene histogram refinement step.The comparison also notes that Deshpande et al. requires careful objective-function and hyper-parameter design.

IV. EXPERIMENTAL RESULTS

The experiments train the colorization network on SIFT Flow images and evaluate it on Sun images, using segmented object categories and a semantic segmentation model assessment.

  • IV. EXPERIMENTAL RESULTS: The proposed colorization neural network is trained on 2344 SIFT Flow images and evaluated on 1519 Sun images.The evaluation images are segmented into object regions for the colorization experiments.
  • IV. EXPERIMENTAL RESULTS: Table I compares scene-parsing algorithms using pixel accuracy across different training and test-image versions.The semantic segmentation model is evaluated on 200 standard test images, and the cited experiment reports that outperforms the other listed algorithms.
  • IV. EXPERIMENTAL RESULTS: The dataset representation uses 33 object categories, including building, car, and sea.These categories support semantic information used by the neural network.
  • IV. EXPERIMENTAL RESULTS: The neural network contains an input layer, three hidden layers, and an output layer.Experiments report that adding more hidden layers does not further improve colorization results.
  • IV. EXPERIMENTAL RESULTS: The model uses a 49-dimension patch feature and a 32-dimension DAISY feature together with semantic features.The cited passage identifies the patch and DAISY feature dimensions and begins describing the semantic feature component.

B. Comparisons with State-of-the-Arts

The proposed method is compared with state-of-the-art colorization systems across reference-image dependence, artifacts, global styles, and runtime, with results favoring its robustness and speed.

  • B. Comparisons with State-of-the-Arts: Reference-based methods perform well with optimal references but can drop significantly when the reference is only similar to the target.The proposed method avoids this limitation through a large reference image database.
  • B. Comparisons with State-of-the-Arts: The paper notes an earlier-version error: 33 object categories, rather than 47, were used in.This correction concerns the category count reported for the earlier work.
  • B. Comparisons with State-of-the-Arts: The proposed method produces more natural colorizations with higher spatial coherency and fewer artifacts than Deshpande et al., without spatial priors.Deshpande et al. performs well when supplied a suitable scene histogram, but visible artifacts still frequently appear.
  • C. Colorization in Different Global Styles: The model can learn different global colorization styles, including daytime and dusk.The paper contrasts this flexibility with an earlier model that automatically colorized all images in one global daytime style.
  • B. Comparisons with State-of-the-Arts: The proposed model processes images at any resolution with runtime increasing nearly linearly with image resolution.Table II reports running times and compares the model with Deshpande et al.; the comparison focuses on fully automatic methods.

E. More Colorization Results

Figure 14 shows colorizations that are visually very similar to ground truth with almost no visible artifacts. The method nevertheless has documented limitations for synthetic images and information lost during grayscale conversion.

  • The proposed method produces color images with almost no visible artifacts and visual similarity to the ground truth.
  • The current model is invalid for synthetic images because it was trained on real images.
  • Color information lost during conversion from color to grayscale cannot be recovered by the method.Figure 15 illustrates this limitation with a green number that disappears from the corresponding grayscale image.
  • The approach is fully automatic and is designed to reduce user effort and dependence on example color images.
  • The method combines deep neural networks, adaptive image clustering, and joint bilateral filtering for artifact-free colorization.The network uses informative patch, DAISY, and semantic features, while clustering incorporates global image information.
Loading 1605.00075v1…