Source-linked AI summary
Color Constancy Using CNNs
Simone Bianco, Claudio Cusano, Raimondo Schettini
TL;DR
Color constancy must estimate scene illumination from image data despite an under-determined relationship between observed colors and physical factors. This paper uses patch-based CNN regression, achieving state-of-the-art performance on RAW images and showing preliminary local-illuminant estimation.
Problem
Color constancy is under-determined from sensor responses because observed colors depend on surface reflectance, illumination, and sensor properties.
Method
The method applies a CNN to contrast-normalized image patches, integrating feature learning and regression to estimate and pool patch illuminants.
Results
The approach achieves state-of-the-art performance on a standard RAW-image dataset and outperforms 21 statistic-based and learning-based algorithms.
Takeaways & Limitations
Preliminary experiments indicate that the CNN can be adapted to estimate local illuminants, including under spatially varying illumination.
Takeaways & Limitations
The local-illuminant extension remains preliminary and requires broader experiments on larger datasets and comparisons with more state-of-the-art algorithms.
Abstract
from arXiv · showhide
In this work we describe a Convolutional Neural Network (CNN) to accurately predict the scene illumination. Taking image patches as input, the CNN works in the spatial domain without using hand-crafted features that are employed by most previous methods. The network consists of one convolutional layer with max pooling, one fully connected layer and three output nodes. Within the network structure, feature learning and regression are integrated into one optimization process, which leads to a more effective model for estimating scene illumination. This approach achieves state-of-the-art performance on a standard dataset of RAW images. Preliminary experiments on images with spatially varying illumination demonstrate the stability of the local illuminant estimation ability of our CNN.
1. Introduction
Color constancy seeks reliable color descriptions despite changing illumination, and this work applies CNNs to learn discriminant features for scene-illuminant estimation, including local regions.
- Color constancy preprocessing helps keep recorded object colors stable under different illumination conditions.Observed color depends on surface reflectance, illumination, and their relative positions.
- Computational color constancy estimates the scene illuminant and then corrects the image toward a reference light source.Color invariance instead represents images with features unchanged under specified imaging conditions.
- The proposed work uses a CNN to learn discriminant features for illuminant estimation from image data.
- CNNs can accept raw images, integrate feature learning into training, and learn complicated mappings with minimal domain knowledge.
- The method estimates illuminants from small patches and reports state-of-the-art performance on RAW images alongside local-estimation results.The authors describe this as the first investigation of CNNs for illuminant estimation.
2. Problem formulation and related works
Color constancy estimates illuminant chromaticity from sensor responses, but the problem is under-determined and existing methods rely on statistical or learned assumptions.
- 2. Problem formulation and related works: Lambertian pixel values depend mainly on illuminant spectrum, surface reflectance, and sensor spectral sensitivities.
- 2. Problem formulation and related works: The illuminant color is the projection of the illuminant spectrum onto the sensor sensitivities, estimated from image responses.
- 2. Problem formulation and related works: Illuminant color is usually estimated up to scale because chromaticity matters more than overall intensity, making color constancy under-determined.
- 2. Problem formulation and related works: Many computational methods assume a uniform light-source color across the scene.
- 2. Problem formulation and related works: Statistic-based approaches infer illumination from single-image content using statistical or physical assumptions, whereas learning-based approaches require training data.
- 2.1. Statistic-based algorithms: A unified statistic-based formulation varies derivative order, Minkowski norm, and Gaussian scale to generate different illuminant estimators.Gray World, White Point, and Gray Edge correspond to different parameter settings and assumptions.
- 2.1. Statistic-based algorithms: Gamut-based methods estimate illumination by mapping an input gamut inside a canonical gamut while producing the most colorful scene.
- 2.2. Learning-based algorithms: Learning-based methods include probabilistic models and fusion or selection methods, with Bayesian approaches conditioning illuminant estimates on image intensity data.Other approaches model surfaces using texture and color features and compare nearest-neighbor surfaces.
3. The proposed approach
The proposed framework normalizes image patches, predicts patch illuminants with a CNN, and pools those estimates for an image; it also evaluates pretrained CNN features with SVR.
- Non-overlapping color-image patches are contrast-normalized, processed by a CNN, and combined into an image-level illuminant estimate.
- 3.1. Network architecture: The network maps contrast-normalized 32x32x3 patches through convolution, max pooling, a 40-node fully connected layer, and three-output linear regression.It uses 240 1x1x3 filters, 8x8 pooling kernels, and a 3840-dimensional reshaped representation.
- Global histogram stretching preserves relative color-channel contributions while normalizing patches for lighting robustness and scale-invariant estimation.
- Large pooling kernels are used because illuminants are locally homogeneous across 32x32 patches, including under spatially varying illumination.
- ReLUs replace sigmoid or tanh units in the fully connected layer.The cited motivation is faster training with nearly identical performance compared with tanh units.
- 3.5. CNN features: A pretrained AlexNet produces 4096-dimensional features that a linear SVR uses to estimate each image's illuminant color.The pretrained network was trained on ILSVRC 2012 for 1000-class image classification.
4. Experimental Setup
The experiments evaluate CNN-based illuminant estimation on a standard RAW-image dataset using angular error and comparisons with established algorithms. Training uses random RAW patches with three-fold cross-validation, then pools patch estimates into one image-level estimate.
- Dataset and evaluation: The study targets illuminant estimation on a standard dataset of 568 RAW images captured with Canon 5D and Canon 1D DSLR cameras.The dataset contains known color targets and linear 14-bit images without color correction.
- Dataset and evaluation: Performance is measured by the angular error between estimated and measured RGB illuminant vectors.The metric compares the direction of the two RGB triplets rather than their overall scale.
- Benchmark algorithms: The benchmark includes statistic-based, learning-based, pixel-based gamut-mapping, and other established color-constancy algorithms.The compared methods include SVR, Bayesian, NIS, HLVI, spatio-spectral, and related variants.
- Benchmark algorithms: Table 1 reports the (n, p, σ) settings used for benchmark algorithms derived from the unified color-constancy formulation.These parameter combinations instantiate different statistic-based illuminant estimators.
- Training procedure: The CNN is trained on random 32x32 RAW patches using three-fold cross-validation and image-level pooling at test time.Each patch receives its source image’s illuminant ground truth, and predicted patch illuminants are pooled into one estimate per image.
5. Results and Discussion
The CNN and its pooling variants outperform many established methods on RAW-image illuminant estimation, while parameter studies identify favorable architectural settings. Preliminary synthetic experiments extend the evaluation to spatially varying illumination using local patch estimates and spatial filtering.
- RAW-dataset results: The fine-tuned CNN with median pooling achieves median, average, and maximum angular errors better than all considered state-of-the-art algorithms.The reported improvements are 1.5%, 5.1%, and 0.2%, respectively.
- RAW-dataset results: Both average and median pooling of patch illuminant estimates outperform most state-of-the-art algorithms in median angular error.Average pooling has a maximum error only 0.3% worse than the best state-of-the-art algorithm.
- RAW-dataset results: A pre-trained deep CNN coupled with SVR already outperforms most statistic-based algorithms and some learning-based algorithms.This result uses features from a CNN pre-trained on ILSVRC 2012.
- Error analysis: Figure 3 compares difficult examples across the original image, ground-truth correction, CNN correction, and the best state-of-the-art correction.The figure focuses on images where the fine-tuned CNN makes its largest estimation errors.
- Parameter effects: Reducing convolution-kernel size lowers estimation error, consistent with evidence that spatial information adds no information beyond color distributions.The parameter study varies kernel width while searching over other parameter combinations.
- Parameter effects: The CNN prefers an intermediate number of convolution kernels and an intermediate pooling size.These trends are reported in the kernel-count and pooling-size parameter studies.
- Parameter effects: About 40 fully connected units and larger patches produce better performance in the reported parameter study.Patch performance is evaluated while estimating the illuminant from sampled patches.
- Local illuminant estimation: The local-estimation test uses synthetic images formed by assigning two different illuminants to separate halves of RAW-dataset images.Table 3 reports patch errors and spatially filtered estimates using 3x3 Gaussian or median filters.
6. Conclusions
The CNN achieves state-of-the-art performance on a standard RAW-image dataset and can be adapted to estimate local illuminants. The authors identify further work on combining patch estimates and validating local estimation more thoroughly.
- The CNN achieves state-of-the-art performance on a standard dataset of RAW images, outperforming 21 statistic-based and learning-based algorithms.
- Preliminary experiments show that the algorithm can be adapted to estimate local illuminants.
- Future work includes investigating pooling strategies for combining patch-based estimates into a global illuminant and conducting broader local-estimation studies on larger datasets.
- Figure 3 compares original RAW images with corrections using the groundtruth illuminant, the CNN estimate, and the best state-of-the-art estimate for images with the largest CNN errors.
- Table 3 reports angular error statistics for the synthetic RAW dataset with spatially varying illumination.
- Figure 4 evaluates angular error while varying convolution kernel width, convolutional-kernel count, pooling size, fully connected units, and input patch size.