Source-linked AI summary
Approaching the Computational Color Constancy as a Classification Problem through Deep Learning
Seoung Wug Oh, Seon Joo Kim
TL;DR
Estimating illumination from an image is ill-posed because different illuminant and surface colors can produce the same image value. This paper classifies illumination with a CNN, then computes chromaticity from classification results; it reports outperforming previous color-constancy methods on multiple test datasets.
Problem
Illumination estimation is ill-posed because infinitely many combinations of illuminant and surface color can produce the same image value.
Source: §1 · p. 1Method
The method casts illumination estimation as classification, clustering similar training illuminants to make classes easier for the CNN to distinguish.
Source: §1 · p. 1, §3.2 · p. 3Results
The CNN extracts useful illumination-estimation features and outperforms previous color-constancy methods on multiple test datasets.
Source: §1 · p. 1Takeaways & Limitations
The authors conclude that CNNs can be powerful for low-level color or appearance problems such as estimating illuminant color.
Source: §6 · p. 12The method assumes a single illuminant across the scene and fails when that assumption is violated.
Source: §5.4 · p. 12
Abstract
from arXiv · showhide
Computational color constancy refers to the problem of computing the illuminant color so that the images of a scene under varying illumination can be normalized to an image under the canonical illumination. In this paper, we adopt a deep learning framework for the illumination estimation problem. The proposed method works under the assumption of uniform illumination over the scene and aims for the accurate illuminant color computation. Specifically, we trained the convolutional neural network to solve the problem by casting the color constancy problem as an illumination classification problem. We designed the deep learning architecture so that the output of the network can be directly used for computing the color of the illumination. Experimental results show that our deep network is able to extract useful features for the illumination estimation and our method outperforms all previous color constancy methods on multiple test datasets.
1. Introduction
Computational color constancy seeks to estimate illumination so images can be normalized despite varying lighting, but the image-formation problem is ill-posed. This paper casts illumination estimation as classification and reports that its CNN outperforms prior color-constancy methods on multiple test datasets.
1. Introduction
- Estimating illumination is ill-posed because infinitely many illuminant and surface-color combinations can produce the same image value.The computational objective is to estimate the illumination chromaticity from the observed image.Source: §1 · p. 1
- CNNs learn task-specific features from training samples rather than relying on manually designed features.The paper motivates applying this learned-feature approach to illumination estimation.Source: §1 · p. 1
- The method transforms illumination estimation into classification, then computes illumination chromaticity from the classification results.The authors designed the network output to be directly useful for computing illumination color.Source: §1 · p. 1
- The CNN extracts useful illumination-estimation features and outperforms previous color-constancy methods on multiple test datasets.The authors present this as evidence that CNNs can also be powerful for low-level color or appearance problems.Source: §1 · p. 1
2. Related Work
Prior illumination-estimation work is divided into statistics-based and learning-based methods. This section reviews their assumptions and feature strategies, then distinguishes this paper's classification of real illuminants from earlier learning-based regression approaches.
2. Related Work
- The field is organized into statistics-based and learning-based approaches to computing illumination from an image.The learning-based category is further divided into combinatorial and direct methods.Source: §2 · p. 2, §2.2 · p. 2
2.1. Statistics-based Illumination Estimation
- Some statistics-based approaches mimic human visual-system mechanisms, but the authors note that those mechanisms remain incompletely understood.The paper proposes training a CNN on large data without needing to specify how that visual processing works.Source: §2.1 · p. 2
- Statistics-based methods rely on scene-statistics or physical-property cues, while the authors note that they are computationally efficient but usually underperform learning-based methods.Examples include assumptions about achromatic reflectance statistics and analysis of bright or dark pixels, specular highlights, or grey pixels.Source: §2.1 · p. 2
2.2. Learning-based Illumination Estimation
- Combinatorial methods select among statistics-based estimators using scene characteristics, whereas direct methods learn an illumination-estimation model from training data.Direct approaches include gamut mapping and models using pixel intensity, chromaticity, derivatives, or spatial distributions.Source: §2.2 · p. 2
- Prior CNN approaches learn features and regress images to illuminant chromaticity, while this paper classifies real illuminants and reports outperforming previous learning-based algorithms.Earlier CNN approaches are described as minimizing Euclidean loss on illuminant color.Source: §2.2 · p. 2
3. Convolutional Neural Network for Illumination Estimation
The system clusters similar training illuminants, trains a CNN to predict illumination-class probabilities, and combines those probabilities to estimate illuminant chromaticity. Its design also addresses limited color-constancy training data through transfer learning and data augmentation.
3.1. System Overview
- The system clusters training images by illuminant, trains the CNN to predict cluster probabilities, then combines its outputs to estimate illumination.This is the three-stage overview described for Figure 1.Source: §3.1 · p. 2
3.2. Clustering the Illuminants
- Similar training illuminants are grouped into more separable classes with K-means and angular distance before CNN training.The authors chose K = 25 for Gehler-Shi, K = 20 for Gray-Ball, and K = 50 for NUS 8-camera.Source: §3.2 · p. 3
3.3. Convolutional Neural Network Architecture
- The CNN uses five convolutional and max-pooling feature-extraction layers, followed by three fully connected layers and a softmax output over K classes.ReLU is used in hidden layers, and dropout regularizes the sixth and seventh fully connected layers.Source: §3.3 · pp. 3–4
3.4. Training strategy
- Because the available color-constancy datasets are limited, the network is initialized with ImageNet-pretrained weights before optimization for illumination classification.The authors state that transfer learning speeds convergence and prevents overfitting to the small training datasets.Source: §3.4 · p. 4
- Training augmentation uses color-preserving rotations, translations, scaling, and flips; 200 patches are sampled per image, and test-image predictions are averaged across 200 patches.Color-preserving transformations are used because changing RGB values may remove illuminant-classification cues.Source: §3.4 · p. 4
- The network is optimized with stochastic gradient descent using batches of 100, momentum 0.9, and weight decay 0.0005.The first convolutional layer and final two fully connected layers use learning rates 10 times larger than the others because they are tuned more for the target task after pretraining.Source: §3.4 · pp. 4–5
3.5. Illumination Estimation
- The CNN estimates a test image’s illuminant by taking a probability-weighted average of cluster centers, allowing chromaticities beyond the discrete cluster centers.The authors reject choosing only the most-probable cluster center because that would restrict possible illuminants to K values.Source: §3.5 · p. 5
4. Experiments
The CNN-based method leads prior methods on Gehler-Shi and Gray-ball, while performing similarly to method on NUS 8-camera.
4. Experiments
- Evaluation used 3-fold cross-validation on Gehler-Shi and 15-fold splits by video clip on Gray-ball, with the color targets masked before training and testing.Gray-ball images were linearized and evaluated using recomputed ground truth; values for other methods were taken from prior work.Source: §4 · p. 5
- On Gehler-Shi, the method outperforms previous works in every test category; on Gray-ball, it is most accurate in all categories.The comparisons use Table 2 and Table 3, respectively.Source: §4 · p. 5, Table 2 · p. 6, Table 3 · p. 6
- A t-SNE view of the CNN’s seventh-layer features shows input patches well separated according to scene illuminant.The visualization places the patches in a two-dimensional feature space.Source: Figure 7 · p. 8
- On NUS 8-camera, the method performs similarly to, evaluated across all eight cameras without knowing the test sample’s camera model.The dataset contains roughly 200 scenes.Source: §4 · pp. 5, 8, Table 4 · p. 7
- Qualitative examples show visually pleasing corrected images for indoor and outdoor scenes, including some extreme cases.The examples span different datasets and compare outputs from different methods.Source: §4 · p. 8
5. Discussion
The discussion examines what the CNN learns, tests its color features with synthetic images, and evaluates how cluster count affects estimation. It also identifies failure cases involving unfamiliar scenes and violations of the single-illuminant assumption.
5.1. Network Visualization
- The CNN develops chromatic filters, replaces some achromatic edge filters with chromatic edge filters, and separates input patches by scene illuminant in its feature space.Hidden units also respond to receptive fields with different color impressions.Source: §5.1 · p. 8
5.2. Experiments on Synthetic Images
- Synthetic images with identical scene contents but different simulated illuminations provide a test of whether the CNN extracts useful color features.The authors report accurate illuminant estimates for these experiments.Source: §5.2 · p. 8, Figure 8 · p. 10
5.3. Determination and validation of K
- Choosing K trades easier classification at smaller K against more accurate illuminant estimation for correctly classified samples at larger K.The authors choose K based on each dataset’s illumination distribution and camera count; validation suggests values from 10 to 50 perform reasonably across datasets, while color-processed SFU Gray-ball images require more clusters than expected, possibly because nonlinear camera processing widens the illumination-color space.Source: §5.3 · pp. 8, 11
5.4. Failure Cases
- The method fails on unrealistic scenes rarely seen in training and when the single-illuminant assumption is violated.Source: §5.4 · pp. 11–12
6. Conclusion
The paper presents a CNN-based approach to estimating illuminant color, applying deep learning to a low-level scene-understanding task. It reports outstanding performance against numerous previous methods and identifies future work on architectures, mixed illumination, and broader datasets.
6. Conclusion
- The paper presents a CNN approach to estimating illuminant color as an application of deep learning to low-level scene understanding.Source: §6 · p. 12
- The authors report outstanding performance compared with numerous previous methods on publicly available datasets.Source: §6 · p. 12
- Future work includes other deep architectures, pixel- or patch-level estimation for mixed illumination, and a more comprehensive color-constancy database.The authors expect a more comprehensive database to improve deep-learning performance.Source: §6 · p. 12