PaperlayerView exact revision on arXiv

Source-linked AI summary

Unsupervised Learning for Color Constancy

Nikola Banić, Karlo Koščević, Sven Lončarić

arXiv:1712.00436v4cs.CV

TL;DR

Accurate supervised color constancy depends on calibrated training images, whose acquisition is time-consuming. The paper learns parameters from non-calibrated images and extends this approach across camera sensors; it reports outperforming all statistics-based and many learning-based methods.

  • Problem

    Supervised color constancy requires calibrated images, preferably from the same sensor, and their careful acquisition and manual calibration are a bottleneck.

    Source: §I · p. 2

  • Method

    The method learns illumination-estimation parameters from non-calibrated images using approximated ground-truth illuminations, with an extension that estimates and neutralizes sensor gains for cross-camera use.

    Source: §II · p. 3, §IV · p. 6, §VII · p. 14

  • Results

    The method outperforms all statistics-based and many learning-based methods in accuracy.

    Source: §VII · p. 14

  • Takeaways & Limitations

    The approach demonstrates highly accurate color constancy without calibrating training images and can use parameters learned with one sensor on images from another.

    Source: F. Discussion · p. 13, §VII · p. 14

    The inter-camera transformation is inappropriate when G^-1e differs significantly from white, where camera differences may resolve metamers or enable hyperspectral imaging.

    Source: §IV · p. 6

Abstract

from arXiv · show

Most digital camera pipelines use color constancy methods to reduce the influence of illumination and camera sensor on the colors of scene objects. The highest accuracy of color correction is obtained with learning-based color constancy methods, but they require a significant amount of calibrated training images with known ground-truth illumination. Such calibration is time consuming, preferably done for each sensor individually, and therefore a major bottleneck in acquiring high color constancy accuracy. Statistics-based methods do not require calibrated training images, but they are less accurate. In this paper an unsupervised learning-based method is proposed that learns its parameter values after approximating the unknown ground-truth illumination of the training images, thus avoiding calibration. In terms of accuracy the proposed method outperforms all statistics-based and many learning-based methods. An extension of the method is also proposed, which learns the needed parameters from non-calibrated images taken with one sensor and which can then be successfully applied to images taken with another sensor. This effectively enables inter-camera unsupervised learning for color constancy. Additionally, a new high quality color constancy benchmark dataset with 1707 calibrated images is created, used for testing, and made publicly available. The results are presented and discussed. The source code and the dataset are available at http://www.fer.unizg.hr/ipg/resources/color_constancy/.

I. INTRODUCTION

Learning-based color constancy can be accurate, but supervised methods require calibrated images, making acquisition and manual calibration a bottleneck. This paper proposes unsupervised learning from non-calibrated images, extends it across camera sensors, and introduces a calibrated benchmark dataset.

I. INTRODUCTION

  • Calibration of training images for a given sensor requires careful acquisition and manual work, creating a bottleneck for accurate supervised color constancy.Supervised methods are preferably trained on calibrated images from the same sensor.Source: §I · pp. 1–2
  • The proposed method learns illumination-estimation parameters from non-calibrated images by clustering approximations of their unknown ground-truth illuminations.The paper describes the method as fast and hardware-friendly, and reports accuracy exceeding many state-of-the-art methods.Source: §I · p. 2
  • An inter-camera extension learns parameters from non-calibrated images taken with one sensor and applies them to images from another.The paper characterizes this extension as independent of camera model and ground-truth illumination.Source: §I · p. 2
  • The authors created and publicly released a color constancy benchmark containing 1707 calibrated high-quality images.The dataset includes images from varied illumination conditions, indoor and outdoor scenes, several countries, and day and night.Source: §I · p. 2

II. MOTIVATION

Calibration makes supervised color-constancy learning time-consuming, while statistical methods can be less accurate. The motivation is to learn from approximate illuminations by using their distribution across images, rather than relying only on per-image accuracy.

II. MOTIVATION

  • Statistical estimations can roughly occupy the same chromaticity regions as ground-truth illuminations, even when individual estimates are less accurate.The paper notes that such estimations can still provide information about where future illuminations are likely to appear.Source: §II · pp. 2–3
  • Across 1,000 random Sony subsets, SAE decreased as subset size grew, but empirical results suggest method choice influenced it more than set size.SAE matches each ground-truth illumination to an estimation from another image to minimize mean angular error; a low SAE alone does not establish per-image accuracy.Source: §II · pp. 2–3
  • Well-chosen statistical estimations across enough images can achieve low SAE, supporting the feasibility of approximating the set of unknown ground-truth illuminations.The estimations need not be accurate for each corresponding image: SAE measures the best overall assignment between the two sets.Source: §II · pp. 2–3
  • Color Dog uses learned cluster centers as its only possible estimates, selecting a center by combining cosine-based votes from Gray-world and White-patch estimates.Its centers are learned with k-means using angular distance; learning them from approximate ground truths gives poor results, motivating a method that can learn from approximations.Source: §II · p. 3

III. THE PROPOSED METHOD

Color Tiger learns two illumination centers from trimmed Shades-of-Gray estimates computed across several parameter values. It uses a warm-versus-cool illumination assumption and trims distant estimates to reduce noise.

III. THE PROPOSED METHOD

  • The two-illuminations assumption classifies most images as having either warmer, reddish or cooler, bluish illumination, simplifying the Planckian locus.The paper reports that this assumption is generally valid in its experiments and discusses what to do when it does not hold for a training set.Source: §III · p. 4
  • Combining SoG estimates across p values broadens chromaticity coverage around ground-truth illuminations; Fig. 4 compares nearest-estimation angles for n = 1, 4, and 8.The first row shows ground-truth-to-estimation angles and the second row the reverse direction.Source: §III · p. 4, Figure 4 · p. 5
  • After initial k-means clustering, Color Tiger removes the furthest 100 · t% of estimates in each cluster and reclusters the remaining estimates.The paper reports that trimming leaves the remaining illuminations much closer to ground truth; Fig. 5 illustrates estimates before and after trimming at t = 0.3.Source: §III · p. 4, Figure 5 · p. 5
  • Color Tiger learns two centers from trimmed Shades-of-Gray estimates using p values 1 through 8; its default trimming fraction is t = 0.3.The method applies the learned centers to estimate illumination for new images, in a procedure resembling Color Dog.Source: §III · p. 4

IV. INTER-CAMERA LEARNING

Color Bengal Tiger adapts illumination estimates across camera sensors by transforming training estimates into a common color space without calibration. The transformation relies on illumination colors being near white, and the method learns separate gain matrices for the training and target sensors.

IV. INTER-CAMERA LEARNING

  • Different sensors shift ground-truth illumination regions in chromaticity space, so centers learned on one sensor may not align with another’s modes.The shift is attributed to differences in camera spectral sensitivities.Source: §IV · p. 6
  • Color Bengal Tiger transforms illumination estimates between camera RGB spaces while preserving illumination influence by omitting the white-balancing component.The sensor transformation maps between RGB color spaces; the method assumes it has little impact on gain-normalized colors near white.Source: §IV · p. 6
  • The approximation in Eq. (10) is inappropriate when the gain-normalized illumination color differs significantly from white.In that case, camera differences may be large enough to resolve camera metamers or improvise hyperspectral imaging.Source: §IV · p. 6
  • Sensor gains are estimated without calibration by assuming mean ground-truth illumination is white and using the median of illumination-estimation approximations.The median is used to handle outliers in the combined estimates.Source: §IV · p. 7
  • Training estimates are gain-normalized and clustered into two centers, with separate gain matrices learned for the training and target sensors.The target-sensor matrix is learned from images taken with that sensor; the extended method is named Color Bengal Tiger.Source: §IV · p. 7

V. GENERAL APPLICABILITY

The methods restrict illumination estimates to two colors, limiting flexibility but also preventing estimates of highly unlikely illuminations. This restriction is presented as useful for illumination conditions that differ considerably from dataset averages.

V. GENERAL APPLICABILITY

  • The proposed methods’ flexibility is limited for illuminations that differ considerably from the tested datasets’ average illumination.This is the stated boundary of using only two colors to cover possible illuminations.Source: §V · pp. 7–8
  • Fig. 9 compares real-world illumination locations with the camera’s built-in restriction on illumination estimates in the rb-chromaticity plane.The described camera restriction covers only a relatively small part of the plane and excludes many relatively common real-world illuminations.Source: Figure 9 · p. 8, §V · p. 8
  • Restricting estimates can prevent highly unlikely choices when image evidence admits multiple explanations, such as a purple scene under white or purple illumination.The paper notes that Gray-world may favor the purple-illumination estimate in this example.Source: §V · p. 8
  • Full correction of highly saturated illumination may leave reflectances of other colors insufficiently visible and make results look washed out.The paper presents this as another reason full correction is not always desirable.Source: §V · p. 8
  • Two illumination centers restrict flexibility but can prevent estimates of highly unlikely illuminations produced by Gray-world and White-patch.The paper describes the restriction as a feature often used in industry, rather than a bug.Source: §V · p. 8

A. Experimental setup

The evaluation uses linear-image benchmarks with calibration-derived ground-truth illuminations, masks calibration objects during testing, and summarizes accuracy primarily with median angular error. Cross-validation follows established folds for two datasets and uses three equal folds for the newly described datasets.

A. Experimental setup

  • The benchmark comparison uses GreyBall, its approximated linear version, the eight linear NUS datasets, and a newly created dataset.ColorChecker is excluded to avoid confusion from differing and erroneous uses in prior publications; linear images are preferred because camera illumination estimation usually operates on them.Source: A. Experimental setup · p. 8
  • Calibration objects used to obtain ground-truth illuminations are masked out of each test image to avoid bias.Examples include a color checker or gray ball.Source: A. Experimental setup · p. 8
  • Median angular error is the most important accuracy statistic because angular-error distributions are nonsymmetric.Angular error is the angle between estimated and ground-truth illumination vectors.Source: A. Experimental setup · p. 8
  • Cross-validation uses the same folds as prior publications for GreyBall and NUS, and three equal folds for Cube and Cube+.The source code for recreating results from a following subsection is publicly available.Source: A. Experimental setup · p. 8

B. The Cube dataset

The Cube dataset provides calibrated outdoor images captured with one camera across parts of three countries. Its two gray calibration faces support ground-truth illumination extraction, while its outdoor-only illumination distribution limits the two-center method’s fit to some scenes.

B. The Cube dataset

  • The Cube dataset contains 1365 outdoor images captured with a Canon EOS 550D in Croatia, Slovenia, and Austria across various seasons.The images are publicly available, and their capture-time ordering was shuffled.Source: B. The Cube dataset · p. 8
  • Two neutral SpyderCube faces provide ground-truth illuminations, allowing both illuminations to be recovered when an image contains two distinct sources.The authors manually checked adapted versions to identify the dominant illumination and exclude scenes that violate the uniform-illumination assumption.Source: B. The Cube dataset · p. 9, Figure 11 · p. 9
  • The dataset uses a black level of 2048, discards pixels at or above m − 2, and masks the calibration object before testing.The suggested mask is the lower-right rectangle beginning at row 1050 and column 2050.Source: B. The Cube dataset · p. 9
  • All images use one camera and cover richer scene content than NUS images, which often repeat scenes across cameras.The same-camera collection is noted as useful for methods requiring more training data because it keeps camera spectral sensitivity constant.Source: B. The Cube dataset · p. 10
  • Because Cube contains only outdoor images, its illumination distribution lacks warmer indoor illuminations and can undermine Color Tiger’s two-center clustering.For scenes with a single illumination type, the paper identifies a single-illumination method such as Color Mule as more appropriate.Source: Figure 12 · p. 9, B. The Cube dataset · p. 10

C. The Cube+ dataset

Cube+ combines the original Cube images with 342 additional images captured under warmer, reddish illuminations. The resulting 1707-image dataset includes indoor and outdoor images, including night scenes.

C. The Cube+ dataset

  • The authors added 342 images under warmer, reddish illuminations using the same camera and ground-truth extraction procedure as Cube.These images were added to address Cube’s illumination-distribution problem.Source: C. The Cube+ dataset · p. 10
  • The combined dataset has 1707 images, with the new images making up about one fifth of the total.The expanded illumination distribution is described as similar to that of the NUS datasets.Source: C. The Cube+ dataset · p. 10
  • The expanded collection is named Cube+ and includes indoor and outdoor images captured at night.The new images were kept distinct from the earlier Cube dataset because that version had already been used by other researchers.Source: C. The Cube+ dataset · p. 10

D. Accuracy

Color Tiger outperforms all statistics-based methods and many learning-based methods on the reported datasets, though some methods do better by small margins. The section also discusses its error profile, training-data needs, and limits on Cube comparisons.

D. Accuracy

  • Color Tiger’s median angular error is lower than Spatio-spectral Statistics’ 2.58°, while its worst 25% error is higher at 7.50° versus 6.17°.Across datasets, its median angular error is below 3° except on GreyBall; methods that outperform it do so on average by a small, mostly imperceptible margin.Source: D. Accuracy · p. 11
  • Color Tiger’s higher worst 25% error is attributed to using only two centers: the ideal two-center variant still has a 5.95° worst 25% error.With ideal selection, using 3, 4, or 5 centers yields worst 25% errors of 4.29°, 3.76°, and 3.39°, respectively; more centers make classification harder.Source: D. Accuracy · p. 12
  • Color Tiger demonstrates that learning-based illumination estimation can succeed without ground-truth illumination, as does the inter-camera Color Bengal Tiger proof of concept.The paper characterizes Color Tiger’s accuracy as below state of the art despite its proof-of-concept contribution.Source: D. Accuracy · p. 12
  • On Sony, Color Tiger’s median angular error stabilizes relatively quickly and stays below 2° for training sets larger than 20 images.The figure studies how limiting training-fold size affects median angular error.Source: D. Accuracy · p. 12, Figure 15 · p. 12
  • Cube comparisons include statistics-based methods and proposed-method precursors, but omit other state-of-the-art methods when their results could not be reproduced reliably.The authors cite incomplete descriptions and latent parameters as reasons those implementations might be suboptimal.Source: D. Accuracy · p. 12

E. Inter-camera accuracy

Color Bengal Tiger learns parameters across cameras using non-calibrated images, with successful results across practically all tested NUS camera pairs. Its performance depends on sensor-gain neutralization and the amount of data available for learning.

E. Inter-camera accuracy

  • Color Bengal Tiger’s median angular error is almost always below 2° across the tested NUS camera pairs.Eight NUS datasets were evaluated in ordered pairs, using one for training the illumination centers and another for testing and learning matrix G′.Source: E. Inter-camera accuracy · p. 12, Table VI · p. 12
  • Without neutralizing camera sensor gains, the Table VII medians are significantly higher than the Table VI results.The authors state that estimating sensor gains is important for bringing images into the same neutral RGB colorspace.Source: E. Inter-camera accuracy · p. 12, Table VI · p. 12, Table VII · p. 13
  • Smaller training sets affect learning cluster centers more than learning G′, although center-learning accuracy converges faster as dataset size grows.With few images from a new camera, the paper suggests applying Color Tiger training to them rather than using clusters learned on another camera; with full datasets, Color Bengal Tiger can outperform Color Tiger.Source: E. Inter-camera accuracy · pp. 12–13, Figure 16 · p. 13
  • The authors report accuracy on par with the approach in, while stating their method does not require ground-truth illuminations or camera spectral sensitivities.They note that reports no explicit numerical results and leaves its average-error definition unclear.Source: E. Inter-camera accuracy · p. 13

F. Discussion

The proposed method achieves strong color-constancy accuracy without ground-truth illumination data, and its discussion highlights time savings, cross-camera use, and hardware-friendly operation. The paper also reports comparative computational costs, introduces the Cube+ benchmark, and identifies directions for future research.

F. Discussion

  • Without ground-truth illumination data, the method outperforms all statistics-based and many learning-based methods, while enabling inter-camera learning and avoiding calibration time.The discussion also notes that avoiding calibration can reduce problems from inaccurate ground-truth data.Source: F. Discussion · p. 13

G. Computational cost

  • At test time, the method runs Gray-world and White-patch once; training runs Shades-of-Gray eight times per image, while clustering takes a small share of runtime.The table compares training and testing times, including approximated values for the proposed methods; the discussion says they compare relatively well to other methods.Source: G. Computational cost · pp. 13–14, Table VIII · p. 14

VII. CONCLUSIONS AND FUTURE RESEARCH

  • The authors created Cube+, a publicly available benchmark of 1707 calibrated images, and demonstrated applying parameters learned with one camera sensor to images from another.They describe Cube+ as one of the largest datasets of calibrated linear images made with a single camera.Source: §VII · p. 14
  • Future research will seek more useful information from statistics-based training-image estimates and alternative outlier-removal methods.Source: §VII · p. 14
Loading 1712.00436v4…