PaperlayerView exact revision on arXiv

Source-linked AI summary

Single and Multiple Illuminant Estimation Using Convolutional Neural Networks

Simone Bianco, Claudio Cusano, Raimondo Schettini

arXiv:1508.00998v2cs.CV

TL;DR

Illuminant estimation from RAW images is under-determined and commonly assumes uniform scene illumination, despite real-world violations. The paper combines local CNN estimates with adaptive aggregation and multiple-illuminant detection, reporting improved state-of-the-art performance on single- and multiple-illuminant data while retaining a three-step pipeline.

  • Problem

    Color constancy must estimate illumination from sensor responses alone, making the problem under-determined; most methods also assume uniform illumination despite real-world nonuniform scenes.

  • Method

    A patch-based CNN produces local illuminant estimates, while KDE detects illumination multiplicity and an RBF-kernel regressor aggregates local estimates when a global estimate is appropriate.

  • Results

    The method improves state-of-the-art performance on images with single illuminants and outperforms general-purpose methods on synthetically relighted multiple-illuminant data, except for a face-based method.

  • Takeaways & Limitations

    The method provides one framework for estimating illuminants under both single and multiple illumination conditions in RAW images.

  • Takeaways & Limitations

    The current method consists of three separate steps rather than one end-to-end estimation model.

Abstract

from arXiv · show

In this paper we present a method for the estimation of the color of the illuminant in RAW images. The method includes a Convolutional Neural Network that has been specially designed to produce multiple local estimates. A multiple illuminant detector determines whether or not the local outputs of the network must be aggregated into a single estimate. We evaluated our method on standard datasets with single and multiple illuminants, obtaining lower estimation errors with respect to those obtained by other general purpose methods in the state of the art.

1 INTRODUCTION

The paper presents a supervised CNN-based illuminant-estimation method that produces local estimates and adaptively handles uniform and nonuniform illumination. It introduces aggregation and detection components and evaluates the method on single- and multiple-illuminant datasets.

  • Color constancy estimates scene illumination and corrects images toward a reference illuminant, while color invariance uses illumination-insensitive features.
  • The method uses a CNN for spatially varying local illuminant estimates, an SVR-based local-to-global regressor, and a KDE-based multiple-illuminant detector.
  • The extended method distinguishes uniform from nonuniform illumination and processes the two cases differently.
  • For uniform illumination, a nonlinear RBF-kernel regression replaces per-channel median pooling to aggregate local estimates globally.
  • The evaluation includes a synthetic multiple-illuminant dataset and RAW images containing at least two known color targets.
  • Experiments report state-of-the-art advances on standard RAW-image datasets for both single and multiple illuminants.

2 PROBLEM FORMULATION AND RELATED WORKS

Illuminant estimation is under-determined because images provide sensor responses rather than direct illumination measurements, motivating assumptions, statistical models, and learned approaches. The section reviews single- and multiple-illuminant methods and the difficulty of extending global estimators to local settings.

  • 2 PROBLEM FORMULATION AND RELATED WORKS: For Lambertian surfaces, image values depend on illuminant spectrum, surface reflectance, and sensor spectral sensitivities; color constancy estimates the illuminant projection sensed by the camera.
  • 2 PROBLEM FORMULATION AND RELATED WORKS: Illuminant color is usually estimated up to scale, with angular error measuring the angle between estimated and measured RGB illuminants.
  • 2 PROBLEM FORMULATION AND RELATED WORKS: Color constancy is under-determined from sensor responses alone, so algorithms require additional assumptions or knowledge; uniform scene illumination is the most common assumption.
  • 2.1 Single illuminant estimation: Single-illuminant methods include statistic approaches based on image assumptions and learning-based approaches trained to build statistical image models.
  • 2.1 Single illuminant estimation: Statistic methods unify derivative order, Minkowski norm, and Gaussian scale choices, with normalization enforcing unit illuminant length.
  • 2.1 Single illuminant estimation: Gray World estimates illumination from average channel shifts under a gray-average assumption, while White Patch uses a different parameter setting within the same family.
  • 2.1 Single illuminant estimation: Learning-based work spans probabilistic models, semantic or image-property-driven algorithm selection, decision-tree fusion, neural networks, Bayesian inference, and luminance-to-chromaticity likelihoods.
  • 2.2 Multiple illuminant estimation: Most state-of-the-art methods assume uniform illumination, although real-world scenes often violate this assumption and local extension is difficult because reduced spatial support can bias estimates.

3 THE PROPOSED APPROACH

The proposed approach combines patch-based CNN illuminant prediction with KDE-based detection of single versus multiple illuminants. It uses local estimates directly for multiple-illuminant scenes and a learned local-to-global regressor for single-illuminant scenes.

  • Overview: Limited annotated data motivates training a CNN on image patches, which produces multiple local illuminant estimates across each input image.Patch extraction provides many training samples even from relatively few annotated images.
  • Multiple-illuminant detection: When multiple illuminants are detected, the method uses the local estimates directly; otherwise, it aggregates them into one global prediction.The aggregation branch is intended for scenes with a single illuminant or similarly colored light sources, where local estimates may be noisy.
  • CNN local estimation: The CNN processes contrast-normalized non-overlapping patches through convolution, pooling, fully connected, and ReLU layers to output a three-channel RGB illuminant estimate.The described network contains 154,723 learned parameters and is trained using Euclidean squared error.
  • Multiple-illuminant detection: KDE analyzes normalized local estimates in chromaticity space to identify dominant modes for deciding whether a scene contains multiple illuminants.Modes exceeding a density threshold are retained, and their pairwise angular differences are evaluated.
  • Local-to-global aggregation: The local-to-global regressor applies an RBF-kernel SVR to pooled statistics of smoothed local-estimate maps and minimizes median angular error.It combines regional average and standard-deviation pooling with whole-image median pooling.

4 EXPERIMENTAL SETUP

The evaluation tests global and multiple-illuminant estimation on standard and synthetic RAW-image datasets, using known color targets or generated ground truth. CNN training uses Gehler-Shi patches, while the local-to-global regressor is retrained for other camera and sensor types.

  • Evaluation scope: The experiments aim to compare the proposed algorithm with state-of-the-art methods for single- and multiple-illuminant estimation on standard RAW-image datasets.The setup includes both global illuminant estimation and synthetic nonuniform-illumination evaluation.
  • Image datasets: The Gehler-Shi and NUS datasets contain RAW DSLR images with Macbeth ColorChecker targets that provide known illuminant references.The Gehler-Shi dataset contains 568 images, while NUS uses nine different cameras.
  • Training procedure: The CNN is trained on 32 × 32 RAW patches from Gehler-Shi, and the local-to-global regressor is retrained for datasets with different cameras and sensors.Three-fold cross-validation is used for training, validation, and testing.
  • Image datasets: A relighted Gehler-Shi benchmark contains 1,704 synthetic images with two, three, or four randomly positioned illuminants and smoothed ground truth.Average maximum angular distances are 8.6°, 12.2°, and 14.8° for the two-, three-, and four-illuminant subsets.
  • Benchmark algorithms: Benchmark comparisons use global color-constancy algorithms because each standard-dataset image contains only one color target.The evaluated baselines include six parameterized statistic-based algorithms and a Do Nothing method.

5 RESULTS AND DISCUSSION

The proposed method is evaluated for global and local illuminant estimation on single- and multiple-illuminant datasets. It achieves strong angular-error performance, with aggregation improving global estimates and the method outperforming comparison algorithms across several settings.

  • Experimental scope: The evaluation covers both single and multiple illuminant estimation on standard datasets.Experiments include global estimation, synthetic spatially varying illumination, mixed single/multiple scenes, and real-world local estimation.
  • Global illuminant estimation: 1.44 degrees is the proposed method’s median angular error on the Gehler-Shi dataset, the lowest among the compared methods.The comparison includes statistic-based, learning-based, and proposed-method variants.
  • Global illuminant estimation: More than one degree is added to the median error without local-to-global aggregation, increasing it to 2.69 degrees.The aggregation module also improves on average pooling, median pooling, and fine tuning, whose median errors are 2.44, 2.32, and 1.98 degrees, respectively.
  • Global illuminant estimation: 15.8% is the average improvement over the best competing algorithm on the NUS dataset, corresponding to 0.35 degrees across cameras.The proposed method outperforms the other algorithms on every camera.
  • Local illuminant estimation: The method outperforms general-purpose single- and multiple-illuminant algorithms on the Milan portrait dataset, while a face-specific method performs better.The proposed detector decides whether local-to-global regression should be applied.
  • Local illuminant estimation: The proposed method reduces median error by almost 14% on the Beigpour et al. multiple-illuminant dataset in both laboratory and real-world settings.Results are reported with the multiple-illuminant detector deciding whether to apply the local-to-global regressor.

6 NETWORK ARCHITECTURE

The network uses a compact CNN designed for local illuminant estimation, combining color projections, pooled spatial information, and learned regression. Its architecture favors 1 × 1 convolutions, large pooling fields, intermediate fully connected width, and large input patches.

  • Architecture: The final model is a simplified CNN with one convolutional layer, max pooling, and two fully connected layers.It was obtained by starting from a deeper LeNet-like CNN and removing layers until further improvement was no longer observed.
  • Parameter sensitivity: Performance is relatively robust to individual changes in patch size, convolutional width and count, pooling size, and fully connected width.Figure 10 varies one parameter at a time while keeping the others at their optimal configuration.
  • Parameter choices: The network uses 240 convolutional kernels, 8 × 8 pooling receptive fields, 40 fully connected units, and 32 × 32 input patches.Smaller pooling fields reduced performance, while the optimal fully connected width was intermediate and larger patches were preferred.
  • Regression output: The last layer maps 40 learned features to the three illuminant components through an affine transformation with 40 × 3 coefficients and three biases.The features combine pooled responses over a 4 × 4 spatial grid, retaining some spatial information for illuminant estimation.
  • Model interpretation: Different fully connected neurons respond to distinct patch types, including dominant colors, highlights, edges, textures, sky, and skin.The activation maps suggest that the network performs a rough content analysis to select scene elements useful for illuminant estimation.
  • Convolutional layer: 1 × 1 convolutional units project RGB values onto directions in color space while preserving spatial information for later layers.The 240 units form a soft quantization of the color space, and subsequent pooling combines these responses spatially.

7 CONCLUSION

The paper presents a CNN-based color constancy method that produces local illuminant estimates and aggregates them when a single illuminant is detected. It improves state-of-the-art results across single- and multiple-illuminant evaluations, while remaining a three-step pipeline targeted for future unification.

  • Conclusion: The method combines feature learning and regression to estimate local illuminant colors from image patches and predict a global illuminant when appropriate.A trained local-to-global regressor aggregates local estimates when the image contains a single illuminant.
  • Conclusion: The algorithm improves state-of-the-art performance on images with a single illuminant and outperforms general-purpose local methods on synthetically relighted multiple-illuminant data.Results are also confirmed on two real-world multiple-illuminant datasets.
  • Conclusion: On the real-world multiple-illuminant datasets, the method is outperformed only by an illuminant estimator that exploits detected faces.This comparison motivates adding semantic information such as scene categories or detected objects.
  • Future work: The current method consists of three separate steps, and future work aims to merge them into a single end-to-end estimation model.The authors are collecting a larger RAW-image dataset containing both single and multiple illuminants for this purpose.
Loading 1508.00998v2…