Source-linked AI summary
Convolutional Color Constancy
Jonathan T. Barron
TL;DR
Color constancy is underconstrained because image colors combine reflectance and illumination. CCC reformulates it as discriminative two-dimensional localization in log-chrominance space, using convolutional and structured-prediction techniques, and reports substantial benchmark error reductions.
Problem
Color constancy must infer illumination color from image colors even though reflectance and illumination are fundamentally confounded.
Method
CCC constructs a chrominance histogram, convolves it with a learned filter, and selects the highest-scoring tint as the estimated illumination.
Results
30% and 39% error reductions on Color Checker mean and median errors, respectively, and a 22% reduction in average error on the Cheng et al. dataset.
Takeaways & Limitations
Discriminative training outperforms generative training by 30−40% improvement, while extended spatial features improve results by 10−20%.
Takeaways & Limitations
The authors found their available datasets insufficient to prevent overfitting when learning a deep set of filters.
Abstract
from arXiv · showhide
Color constancy is the problem of inferring the color of the light that illuminated a scene, usually so that the illumination color can be removed. Because this problem is underconstrained, it is often solved by modeling the statistical regularities of the colors of natural objects and illumination. In contrast, in this paper we reformulate the problem of color constancy as a 2D spatial localization task in a log-chrominance space, thereby allowing us to apply techniques from object detection and structured prediction to the color constancy problem. By directly learning how to discriminate between correctly white-balanced images and poorly white-balanced images, our model is able to improve performance on standard benchmarks by nearly 40%.
1. Intro
Color constancy infers illumination color from image colors, but the task is underconstrained because pixel color combines reflectance and illumination. CCC reframes white-balancing as discriminative localization in a two-dimensional log-chrominance space.
- Problem: Pixel color combines surface reflectance with illumination, making it difficult to determine whether a yellow pixel reflects a yellow object or colored lighting.Color constancy estimates and corrects illumination color, commonly called white balance.
- Prior approaches: Most prior statistical and learning-based approaches generatively model natural colors from white-balanced images before correcting new images.These methods assume regularities in the colors of natural objects viewed under white light.
- CCC: CCC instead explicitly trains a model to distinguish white-balanced images from non-white-balanced images.The paper describes discriminative learning as largely unexplored in color constancy.
- CCC: Per-channel image scaling translates the image’s log-chromaticity histogram, allowing color constancy to be formulated as localizing a template in two-dimensional space.This formulation borrows techniques from convolutional neural networks, structured prediction, and object detection.
- Outcome: CCC reports state-of-the-art results and nearly 40% lower error rates on standard benchmarks.The introduction presents the method as a discriminative reformulation of color constancy rather than a generative model of natural colors.
2. Image Formation
The image-formation model represents each RGB pixel as white-balanced reflectance multiplied by shared illumination. Under channel-wise correction and scale ambiguity, estimating illumination reduces to recovering two log-chrominance quantities.
- Image formation: Each RGB pixel is modeled as the product of its true white-balanced value W and a shared RGB illumination L.The model assumes black-level correction and no saturated pixel values.
- Assumptions: The formulation ignores shading, reflectance properties, and spatially varying illumination, while assuming channel-wise gain correction under the Von Kries law.Given I, the goal is to estimate L and produce W = I/L.
- Chrominance: Because absolute scaling of W is irrelevant, illumination estimation simplifies to recovering its chrominance as two numbers.The two chrominance measures are defined from RGB values of I and W.
- Log-chrominance: In log-chrominance space, the image-formation relationship becomes a reformulation based on logarithmic properties.The paper introduces chrominance and luminance measures before rewriting the problem in this space.
- Recovery: The remaining unknowns are (Lu, Lv); assuming unit-norm illumination then permits recovery of the RGB illumination components.The inverse RGB-to-UV mapping is otherwise undefined because of absolute scale ambiguity.
- Why this representation: The log-chrominance formulation has two unknowns and a simple linear constraint, properties required for a two-dimensional spatial-localization formulation.The RGB formulation instead has three unknowns and a multiplicative constraint.
3. Learning
CCC learns color constancy discriminatively by treating tint-induced translations of log-chrominance histograms as a two-dimensional localization problem. It constructs histogram features, convolves them with a learned filter, and trains the filter to score the true illuminant highly while distinguishing well- from poorly white-balanced images.
- Histogram features: The algorithm constructs a luminance-weighted chrominance histogram from the input image and applies a square-root transform to normalized counts.The histogram records pixel chrominances near each (u, v) bin; experiments use bin-width ϵ = 0.025 and 256 bins.
- Log-chrominance representation: Tinting scales red and blue channels, translating an image’s log-chrominance histogram without changing its shape.This translation property motivates using convolution to localize histograms in two-dimensional log-chrominance space.
- Inference: CCC scores all possible image tints by convolving the histogram with a filter, then selects the highest-scoring illumination and computes the white-balanced image as Ŵ = I/L̂.Convolution replaces brute-force scoring over all tints when the scoring function is a linear combination of histogram bins.
- Discriminative learning: The filter is learned with a convolutional model resembling multinomial logistic regression and structured prediction, using a softmax over candidate illumination bins and a regularized loss.The training objective encourages the convolution response to be larger at the ground-truth illumination than at alternative locations.
- Optimization: The loss measures angular error between proposed and ground-truth illuminations, and training combines stochastic gradient descent with batch L-BFGS until convergence.Using both optimizers yields lower losses and test-set error rates than SGD alone, while being faster than using only batch L-BFGS.
- Model scope: Deeper filter sets forming a convolutional neural network overfit because the available datasets lack sufficient training data.The proposed single-filter approach performed more accurately on the test set than the experimented alternatives.
- Discriminative versus generative training: Discriminative training learns richer filters than generative training because it distinguishes well-white-balanced images from improperly white-balanced images.The discriminative filter can upweight sky-like blues and downweight pale greens associated with badly white-balanced images, unlike the generative gray-world-like filter.
4. Efficient Filtering
CCC uses pyramid filtering to evaluate chroma-histogram filters efficiently, preserving fine central detail while modeling coarse context and simplifying optimization.
- Accurate filters tend to use a log-polar or retinotopic structure, with high-frequency variation near the filter center.
- Compared with naive convolution, pyramid filtering simplifies regularization during training and improves testing speed.
- Pyramid filtering builds a multiscale representation, applies small filters at each scale, and collapses the results into a filtered histogram.The implementation uses a 7-level Gaussian pyramid, 5 × 5 filters, bilinear sampling, and [1, 2, 1] blur before upsampling.
- The pyramid approximation corresponds to densely computing log-polar features and classifying them with a linear classifier.This makes dense computation and classification tractable, supporting precise illuminant estimation.
5. Generalization
CCC generalizes beyond individual pixel chroma by filtering multiple histograms from scale-preserving augmented images that encode spatial and edge-related information.
- 5. Generalization: The extended model filters a set of histograms from augmented images rather than classifying one histogram from the original image alone.Filtered responses are summed before softmax probabilities are computed.
- 5. Generalization: Augmented images must preserve per-channel scaling so illumination changes remain translations in log-chrominance histogram space.Their outputs must also be non-negative because the model computes their logarithms.
- 5. Generalization: The model uses four channels: the input, sharpened and rectified, soft max-filtered, and standard-deviation-filtered images.These channels capture information such as edges, texture, highlights, and local spatial statistics.
- 5. Generalization: Each augmented channel receives a learned pyramid filter, and the resulting filter responses are summed before softmax normalization.Using a small number of simple channels supports speed and helps prevent overfitting.
- 5. Generalization: Inference transforms the input into augmented images and chroma histograms, scores histogram bins, selects the highest-scoring illuminant, and divides the image by it.
6. Results
CCC was evaluated with three-fold cross-validation on two color-constancy datasets using angular illumination error and multiple summary statistics. It improved substantially over state-of-the-art baselines, with discriminative training and extended features both contributing to performance.
- Datasets and metrics: The evaluation uses the Color Checker Dataset with 568 images and the Cheng et al. dataset with 1736 images from 8 cameras.Performance is measured by the angular error in degrees between estimated and true illuminations.
- Benchmark results: 30% mean-error and 39% median-error reductions were achieved on Color Checker versus Corrected-Moment.
- Benchmark results: 22% average-error reduction was achieved on the Cheng et al. dataset versus its state-of-the-art baseline.The improvement was fairly consistent across different error metrics.
- Model comparisons: Discriminative training outperformed generative training by 30−40% improvement, while the extended model improved over the simple model by 10 −20%.Generatively trained models performed similarly to some earlier generative techniques.
- Evaluation procedure: The authors reproduced the Corrected-Moment baseline to support a thorough and fair evaluation across both datasets.Their reproduced results differed slightly from prior reports, apparently because of parameter settings or inconsistencies in the provided implementation.
- Runtime: The unoptimized Matlab implementation requires about 1.2 seconds per megapixel for feature construction and histogram extraction, plus about 20 milliseconds for pyramid filtering.
7. Conclusion
CCC reframes color constancy as discriminative localization in log-chrominance space, using pyramid filtering and augmented inputs to achieve state-of-the-art results on two large datasets.
- CCC uses illumination-induced log-chrominance translation to apply object-detection and structured-prediction ideas to white balancing.
- Pyramid-based filtering improves efficiency, while augmented input variants capture different kinds of spatial information and improve accuracy.
- CCC beats the best-performing techniques by 20%−40% on various error metrics across two large color-constancy datasets.The experiments suggest that color constancy may benefit from much larger datasets.
Supplemental Material
The supplied supplemental-material passage identifies the paper as an arXiv version dated 18 September 2015.
- The paper is identified as arXiv:1507.00410v2.
- The version is associated with the computer-vision subject classification.
- The displayed version date is 18 Sep 2015.
1. Exponential Decay SGD
The paper introduces exponential decay SGD, which adaptively tracks gradient, gradient-squared, and loss averages before conventional batch L-BFGS finalization.
- Exponential decay SGD is used to accelerate optimization before finalizing training with conventional batch L-BFGS.The authors developed this stochastic-gradient variant for their task.
- The method divides gradients by the square root of a moving average of gradient-squared values before taking descent steps.It also maintains moving averages of the gradient and loss, extending the RMSProp and AdaDelta pattern.
- The algorithm adjusts step size by comparing each sampled datapoint’s loss with a moving average of loss over the entire dataset.Step size increases slightly below the average and decreases otherwise.
- The technique parameterizes exponential decay using half-life rather than decay multipliers.The reported settings use loss half-life near dataset size, gradient half-life near mini-batch size, and an intermediate gradient-squared half-life.
- Running averages are represented as ratios to remain correct at optimization’s beginning, while experiments use 50 epochs with randomized datapoint order.Each epoch samples every datapoint once in a random order to improve training-data coverage.
2. Additional Images
Additional Color Checker examples compare camera inputs and ground-truth corrections with the proposed CCC output and three leading baseline methods.
- Each example presents the camera-produced input image alongside the ground-truth illumination and white-balanced image.Images are shown as cropped squares for easier visualization.
- The examples include the output of the proposed model, labeled “CCC.”
- The CCC output is compared with three best-performing baseline techniques, including two Corrected Moment variants.