Source-linked AI summary
Infrared Colorization Using Deep Convolutional Neural Networks
Matthias Limmer, Hendrik P. A. Lensch
TL;DR
The paper addresses automatic colorization of grayscale NIR images, where missing color information and single-channel input limit conventional colorization and color-transfer methods. It uses a deep multi-scale CNN to estimate RGB values and transfers NIR details during postprocessing, producing natural-looking summer road-scene colorizations without user guidance or reference images. The method is reported to outperform existing approaches, while remaining unable to recover information absent from the NIR image.
Problem
Single-channel NIR images require estimating both luminance and chrominance, limiting conventional colorization and color-transfer methods.
Method
A deep multi-scale CNN directly estimates RGB values from NIR images, followed by filtering and transfer of input details during postprocessing.
Results
The approach produces automatic colorizations without user guidance or reference-image retrieval and is reported to achieve significantly better results than existing colorization and color-transfer methods.
Takeaways & Limitations
CNN-based integrated colorization can infer luminance and chrominance for most natural objects in summer road scenes without scene labels or hand-crafted features.
Takeaways & Limitations
The method cannot correctly colorize objects whose appearance does not correlate with a specific color, and cannot recover signals absent from the NIR image.
Abstract
from arXiv · showhide
This paper proposes a method for transferring the RGB color spectrum to near-infrared (NIR) images using deep multi-scale convolutional neural networks. A direct and integrated transfer between NIR and RGB pixels is trained. The trained model does not require any user guidance or a reference image database in the recall phase to produce images with a natural appearance. To preserve the rich details of the NIR image, its high frequency features are transferred to the estimated RGB image. The presented approach is trained and evaluated on a real-world dataset containing a large amount of road scene images in summer. The dataset was captured by a multi-CCD NIR/RGB camera, which ensures a perfect pixel to pixel registration.
I. Introduction
NIR cameras support low-light road-scene sensing but produce grayscale images whose appearance can reduce user acceptance. The paper proposes automatic CNN-based NIR-to-RGB colorization without user guidance or reference-image retrieval.
- Motivation: NIR cameras retain sensitivity and can illuminate scenes without blinding road users, but their grayscale output differs from filtered RGB imagery.These properties motivate colorizing NIR images for driver-assistance systems.
- Problem: NIR colorization must estimate both luminance and chrominance, unlike grayscale colorization, while single-channel input limits many color-transfer methods.Color-transfer methods often rely on inter-color distinctions that are unavailable in grayscale NIR images.
- Contribution: The proposed integrated deep-learning approach directly transfers the NIR spectrum to RGB images and adds input details during postprocessing.It uses a deep multi-scale CNN for low-frequency RGB estimation and transfers high-frequency details from the NIR input.
- Evaluation: The approach is evaluated on pixel-registered NIR/RGB summer road scenes and is reported to outperform existing colorization and color-transfer methods.The supplied introduction reports extensive numerical experiments and significantly better results, without providing numerical values here.
- Contribution: The method performs automatic colorization without user guidance, scene labels, hand-crafted features, or a reference-image database.This distinguishes it from scribble-based, reference-matching, and scene-label-dependent approaches.
III. Approach
The approach uses deep multi-scale CNNs to colorize infrared images, training on patches while performing image-based inference to preserve resolution.
- III. Approach: Deep multi-scale CNNs colorize infrared images, combining a CNN architecture with multi-scale processing of the input.The architecture is inspired by prior CNN designs and a multi-scale scheme.
- III. Approach: Patch-based training is paired with image-based inference to preserve the input image resolution.The inference procedure uses techniques described in prior work for resolution-preserving application.
A. Deep Multiscale CNNs
The method processes multiple scales of the same input concurrently and combines them into one output, using a three-stage preprocessing, inference, and postprocessing pipeline.
- A. Deep Multiscale CNNs: Multiple scales of the same input are processed concurrently and combined into one output to increase scale invariance.The multi-scale scheme processes corresponding input data at different resolutions before fusion.
- A. Deep Multiscale CNNs: The pipeline normalizes an image pyramid, infers color with a CNN, then filters the raw output and transfers input details to the final image.Figure 2 depicts data flow through these processing stages.
C. Preprocessing
Preprocessing constructs a locally normalized image pyramid and derives low- and high-frequency components, which feed separate network branches before fusion.
- C. Preprocessing: The input image pyramid contains n_l levels, each downsampled by 0.5 per image dimension and locally normalized to zero mean and unit variance.Local normalization enhances texture in the pyramid images.
- C. Preprocessing: Normalization produces a mean-filtered image I_μ as the low-frequency component and I′_1 ◦ I_σ as the high-frequency component I_h.The low- and high-frequency components are derived from the normalized image and local standard deviations.
- C. Preprocessing: Each pyramid level feeds its own structurally identical network branch, and the branches are fused in a final fully connected output layer.The supplied architecture description states that the branches do not share weights and use convolution and pooling layers.
- C. Preprocessing: The processing overview represents colored blocks as processing steps, arrows as data flow, and variable pyramid levels as a variable number of inference branches.Convolution and activation layers are included within the convolution blocks rather than drawn separately.
E. Postprocessing
Postprocessing removes checkerboard artifacts from the CNN output and restores image details by filtering the estimate and adding high-frequency input information.
- E. Postprocessing: The raw inference output contains checkerboard noise because pooling and convolution create interleaved coherent maps during pixel-wise evaluation.For three 2 × 2 pooling layers, the coherence gap is 8 pixels in both image dimensions.
- E. Postprocessing: A joint bilateral filter uses the input image as guidance to remove noise while preserving object contours and edges.The postprocessing first filters the raw CNN output before detail augmentation.
- E. Postprocessing: Adding the input detail component partially recovers surface textures after filtering, producing visible textures and distinct object boundaries.The postprocessing sequence is illustrated as raw output, filtered output, and detail-enhanced output in Fig. 3.
- E. Postprocessing: The dataset examples pair RGB images on the left with corresponding NIR images on the right.
IV. Experiments
The approach is trained and evaluated on real-world road-scene imagery captured with a two-CCD camera that synchronizes and registers NIR and RGB channels.
- IV. Experiments: A two-CCD camera separates NIR and RGB wavelengths onto dedicated sensors, ensuring pixel-to-pixel registration and temporal synchronization.The dataset was recorded during sunny summer days over approximately five hours of video.
- IV. Experiments: The dataset was assembled for the proposed application from real-world road-scene video recorded during several sunny summer days.
A. Color Transfer and Colorization
The comparison evaluates colorization and color-transfer methods on an example NIR image, showing that conventional approaches struggle with correct luminance or object-specific color transfer.
- A. Color Transfer and Colorization: User-guided and automatic colorization methods colorize the sky but fail for trees and grass because they do not estimate the correct luminance.
- A. Color Transfer and Colorization: Table I summarizes the evaluated network topologies and identifies configurations that bypass the mean image input to the fully connected layer.
- A. Color Transfer and Colorization: Global and local color-transfer methods cannot assign different colors to different objects and are therefore unsuitable for NIR colorization.The proposed method is shown for comparison without knowledge of the target image.
B. Network Topologies
The study evaluates multi-scale CNN topologies using RMSE and S-CIELAB, identifying topo-3-12-3-bp as the best-performing architecture. Increasing scales helps up to a threshold, while bypass paths consistently improve performance.
- Network evaluation: topo-3-12-3-bp achieves the best RMSE and S-CIELAB performance among the evaluated network architectures.The evaluation uses raw outputs from 800 images before postprocessing; Table II reports averages and standard deviations, while Fig. 6 visualizes RMSE distributions.
- Architecture factors: The best architecture uses the largest input patch, three scales, and a bypass path, while larger input patches generally improve performance.The trend has an exception for topo-1-12-3-bp.
- Scale analysis: Increasing the number of scales improves performance up to four scales, after which gains disappear or slightly reverse.With five scales, topo-5-9-2 shows almost no improvement and topo-5-9-2-bp shows a slight performance loss.
- Bypass path: The bypass path improves performance across all evaluated topologies by providing the mean filtered image as an additional prior.The mean filtered image contains information absent from the normalized input images.
C. Postprocessing
Postprocessing adds NIR high-frequency details after filtering the CNN output, and bilateral-filter parameters strongly affect the final appearance. For topo-3-12-3-bp, σg values up to 17 mainly reduce noise, whereas larger values cause color bleeding.
- Bilateral-filter parameters: The postprocessing filter controls spatial blur through σg and edge sensitivity through σf, shaping the appearance after details are added.
- Visual results: Fig. 10 compares NIR inputs, colorized outputs, and target images using topo-3-12-3-bp with σg = 17 and σf = 0.005.
- Bilateral-filter parameters: For topo-3-12-3-bp, σg values up to 17 mainly smooth coherence-gap noise, while larger values increasingly cause color bleeding.When smoothing becomes excessive, the error surpasses the error of the raw output after adding the detail layer.
V. Discussion
The method colorizes summer road-scene NIR images automatically, but single-channel NIR data cannot recover all missing scene information. Ambiguous artificial objects may be falsely colorized, and objects absent from NIR cannot be reconstructed.
- Scope and capability: The approach fully automatically colorizes NIR images of summer road scenes, but cannot reconstruct information absent from single-channel NIR images.
- Scope and capability: Artificial objects such as cars and buildings may be falsely colorized when their NIR appearance does not correlate with a specific color.
- Unrecoverable information: A signal absent from the NIR image, such as a green LED traffic light, is not recoverable even with additional scene-label information.The passage states that scene labels might help in some cases, but cannot restore missing signals.
VI. Conclusion and Future Work
The integrated NIR-to-RGB approach directly estimates RGB values with a deep multi-scale CNN, then filters and restores input details for naturally colorized outputs. Its current limitation is incorrect coloring when appearance and color are not correlated, motivating broader seasonal data and improved detail and multimodal modeling.
- The approach directly estimates RGB values with a deep multi-scale CNN, then jointly filters the output and adds input details for natural colorization.A mean-filtered input improves the final fully connected layer, while the input guides joint-bilateral filtering.
- Its main failure occurs when object appearance and color do not correlate, causing incorrect object colorization.
- Future work targets other seasons, more accurate detail-layer estimation, and multimodal loss with semantic segmentation to improve realism and difficult object colors.
- The paper identifies situations in which the approach shows limitations.