PaperlayerView exact revision on arXiv

Source-linked AI summary

Fast Fourier Color Constancy

Jonathan T. Barron, Yun-Ta Tsai

arXiv:1611.07596v3cs.CV

TL;DR

Color constancy needs accurate illuminant estimation that is also fast and temporally coherent enough for practical white balance. FFCC reduces the task to localization on a torus and learns it with frequency-domain operations, addressing aliasing and uncertainty. It reports 13−20% lower error and 250−3000× speedups relative to prior work, including 1.44 milliseconds per image on a consumer mobile platform.

  • Problem

    Practical white balance requires both accuracy and real-time speed, while video use also requires temporal coherence.

  • Method

    FFCC reduces color constancy to localization on a small torus, using FFT convolution, de-aliasing, and a posterior distribution over illuminants.

  • Results

    13−20% lower error and 250−3000× greater speed than prior work are reported, with 1.44 milliseconds per image on a consumer mobile platform.

  • Takeaways & Limitations

    FFCC’s speed, accuracy, posterior uncertainty, and temporal smoothing support real-time automatic white balance on consumer cameras.

  • Takeaways & Limitations

    Temporal smoothing is not used in the benchmarks and is demonstrated only in supplemental video.

Abstract

from arXiv · show

We present Fast Fourier Color Constancy (FFCC), a color constancy algorithm which solves illuminant estimation by reducing it to a spatial localization task on a torus. By operating in the frequency domain, FFCC produces lower error rates than the previous state-of-the-art by 13-20% while being 250-3000 times faster. This unconventional approach introduces challenges regarding aliasing, directional statistics, and preconditioning, which we address. By producing a complete posterior distribution over illuminants instead of a single illuminant estimate, FFCC enables better training techniques, an effective temporal smoothing technique, and richer methods for error analysis. Our implementation of FFCC runs at ~700 frames per second on a mobile device, allowing it to be used as an accurate, real-time, temporally-coherent automatic white balance algorithm.

1. Intro

FFCC targets practical color constancy by reducing illuminant estimation to localization on a small torus and implementing the computation in the frequency domain. It reports improved accuracy, large speedups, posterior uncertainty, and temporal smoothing for white balance.

  • Color constancy estimates scene illumination and object colors under white light, while automatic white balance seeks a natural-looking image without user intervention.
  • 13−20% lower error and 250−3000× greater speed than prior work make FFCC suitable as a prospective real-time white balance algorithm.The paper also reports 1.44 milliseconds per frame on a standard consumer mobile platform.
  • FFCC outputs a complete posterior distribution over illuminants, supporting uncertainty reasoning and effective temporal smoothing.
  • FFCC reduces color constancy to localization on a torus, repeatedly wrapping a large localization problem onto a smaller representation.The toroidal formulation creates detection, wraparound, and aliasing challenges.
  • FFCC uses FFTs for convolution because their periodic structure handles detections wrapping around toroidal boundaries and provides a significant speedup.

2. Convolutional Color Constancy

CCC and FFCC model illumination through log-chroma, turning multiplicative image formation into an additive localization problem. FFCC retains this convolutional formulation while using compact histograms and frequency-domain computation.

  • The model assumes each pixel’s RGB value equals its true white-balanced RGB value multiplied by a global RGB illumination shared across the image.Saturated pixels are identified and removed from downstream computation.
  • Color constancy estimates the illumination L from input image I and recovers white-balanced pixels as W(k) = I(k)/L.
  • The absolute scale of illumination is unrecoverable, so estimating its log-chroma is sufficient before reconstructing RGB illuminant values with unit magnitude.
  • CCC uses log-chroma coordinates because they reduce the unknowns from three to two and convert the multiplicative image constraint into an additive one.
  • The log-chroma histogram N supports a 2D spatial localization formulation in which a sliding-window classifier filters the histogram to estimate the illuminant.

3. Illuminant Aliasing

FFCC makes log-chroma histograms small and toroidal, improving convolutional efficiency but causing illuminant aliasing. The method therefore combines periodic processing with explicit de-aliasing strategies.

  • FFCC builds a periodic log-chroma histogram with n = 64 bins and h = 1/32 bin size, using modular arithmetic so pixels wrap around histogram boundaries.
  • Wrapping means one histogram coordinate represents an infinite family of colors, so a filtered centroid becomes an infinite set of possible illuminants called illuminant aliasing.
  • De-aliasing uses external information to select one illuminant from the aliased estimate; gray world de-aliasing favors the estimate that makes average image color most neutral.
  • Gray light de-aliasing assumes the illuminant is near the histogram center by choosing histogram bounds from training-data ground-truth illuminants.It systematically fails when the histogram is too small to contain all illuminants.
  • The FFCC pipeline uses FFT convolution on a small aliased histogram, estimates its centroid, and de-aliases the result within a differentiable end-to-end model.

4. Differentiable Bivariate von Mises

FFCC fits toroidal illuminant PDFs with a differentiable bivariate von Mises model, avoiding Gaussian failures at wrap-around boundaries while producing calibrated uncertainty.

  • Differentiable Bivariate von Mises: A bivariate von Mises distribution parameterizes toroidal PDFs, unlike a Gaussian that assumes a planar domain.
  • Differentiable Bivariate von Mises: The proposed BVM estimator directly predicts mean µ and covariance Σ in a closed-form differentiable parameterization.Existing BVM parameterizations lack closed-form maximum-likelihood estimators and convenient backpropagation; the proposed estimator is approximate but accurate for well-concentrated distributions.
  • Differentiable Bivariate von Mises: Gaussian fitting fails when a toroidal PDF wraps around the boundary, whereas both fits behave similarly near the unwrapped plane’s center.
  • Differentiable Bivariate von Mises: The mean is estimated using circular means, while covariance is computed after unwrapping coordinates relative to the estimated mean.
  • Differentiable Bivariate von Mises: Training with Gaussian negative log-likelihood produces a calibrated complete posterior over illuminants rather than only a point estimate.The posterior also supports confidence estimation through the entropy of Σ and temporal video processing.

5. Model Extensions

FFCC extends its toroidal baseline with augmented image channels and illuminant-dependent gain and bias maps to incorporate spatial and absolute-color information.

  • Model Extensions: The baseline is limited because it reasons about pixels independently, ignores spatial information, and does not model the illuminant’s absolute color.
  • Model Extensions: Augmented images create a stack of filtered log-chroma histograms, with learned filters applied across channels and summed after convolution.
  • Model Extensions: The two selected augmented features perform similarly to four features used in prior work while costing less to compute.
  • Model Extensions: The learned filters, gain map, and bias map are visualized in centered log-chroma space, where gain and bias represent camera and black-body structure as wrap-around segments.
  • Model Extensions: An illuminant gain map G(i, j) and bias map B(i, j) apply a per-illuminant affine transformation to the convolution output.These maps allow the model to reason directly about illumination despite toroidal translation invariance.

6. Fourier Regularization and Preconditioning

FFCC regularizes periodic model weights with smoothness penalties and reparameterizes them in the Fourier domain, improving optimization speed and the losses reached.

  • Fourier Regularization and Preconditioning: The regularizer combines squared periodic convolutions, including adjacent-value differences and squared parameter values, to favor small and smooth weights.
  • Fourier Regularization and Preconditioning: The regularization strengths λ1 and λ0 control the smoothness terms, with λ0 > 0 required to avoid divide-by-zero during preconditioning.
  • Fourier Regularization and Preconditioning: Reparameterizing model components as rescaled FFT vectors converts the complicated image-space regularization into simple L2 regularization with a preconditioning effect.
  • Fourier Regularization and Preconditioning: Preconditioned frequency-domain optimization reaches lower-loss minima faster than non-preconditioned frequency-domain and naive time-domain optimization.Figure 5 compares the methods across three cross-validation folds on the Gehler-Shi dataset using LBFGS.
  • Fourier Regularization and Preconditioning: Training uses LBFGS for 16 convex pre-training iterations followed by 64 iterations optimizing the non-convex BVM loss.

7. Temporal Smoothing

FFCC uses its per-frame illuminant posterior to smooth estimates over video while allowing responsiveness to changing illumination and multiple valid hypotheses.

  • Temporal Smoothing: Video white balance requires temporal smoothing that avoids both erratic viewfinder changes and excessive lag during rapid illumination changes.
  • Temporal Smoothing: The model updates an ongoing illuminant Gaussian by convolving it with an isotropic temporal prior and multiplying by the observed Gaussian.
  • Temporal Smoothing: Parameter α sets the expected illuminant variance over time, and the update resembles a Kalman filter with a simplified transition model and variable observation noise.
  • Temporal Smoothing: Temporal smoothing is not used in the reported benchmarks; its effect is shown only in the supplemental video.

8. Results

FFCC is evaluated on two standard color constancy datasets, including ablations and thumbnail-image variants, with accuracy, speed, and training behavior compared across models. Its best-performing model reduces error substantially, while its fastest model reaches millisecond-scale inference and the phone implementation meets real-time constraints.

  • The evaluation uses three-fold cross-validation on Gehler-Shi and Cheng et al. datasets, reporting angular RGB error and posterior-entropy measures for selected images.
  • 20% reduction in error on Gehler-Shi is achieved by Model P versus the previously best-performing published technique.
  • 1.1 milliseconds per image is achieved by Model Q, whose accuracy is comparable to prior techniques while processing images much faster.Most other FFCC models take approximately 30 ms/image, compared with 520 ms for CCC and 3 seconds for Shi et al.; Model Q’s Matlab implementation takes 2.37 ms/image.
  • 1.44 ms per image on a 2016 Google Pixel XL is equivalent to 30 frames per second using less than 5% of the total compute budget.The implementation uses thumbnail images produced by the camera stack and satisfies the stated speed requirements for automatic white balance.

9. Conclusion

The paper concludes that FFCC combines improved color-constancy accuracy with large speedups by using toroidal convolutional color constancy and frequency-domain optimization. Its posterior illuminant distribution supports confidence estimation and temporal smoothing, enabling real-time white balance on consumer cameras.

  • 13−20% reduction in error and 250−3000× speedup relative to prior work summarize FFCC’s reported performance.
  • FFCC introduces toroidal convolutional color constancy, illuminant de-aliasing, differentiable bivariate von Mises fitting, and fast Fourier-domain optimization.
  • A complete posterior distribution over illuminants enables confidence assessment and a Kalman filter-like temporal smoothing model.
  • 1.44 milliseconds per frame on a standard consumer mobile platform demonstrates the speed needed for real-time white balance.
  • The data-term loss fits a bivariate von Mises distribution to a toroidal PDF and computes the true white point’s negative log-likelihood.The loss is non-convex, while the alternative logistic loss behaves similarly to CCC’s loss and is convex.

C. Deep Models

The deep models map image-specific external features to FFCC parameters through a small neural network, while metadata and semantic inputs provide additional information. Frequency-domain vectorization and preconditioning support optimization and regularization of the model weights.

  • Deep Models: Models N, O, and P incorporate external features during training and testing, allowing the model to use information beyond pixel and edge log-chroma histograms.
  • Deep Models: A 2-layer ReLU network maps each training instance’s feature vector x_i to FFCC weights w_i instead of learning one shared model.
  • Deep Models: EXIF metadata encodes camera exposure settings and sensor identity; Model O’s improved performance shows that this information can significantly reduce error.The exposure features include log shutter speed, log F number, and a constant, while the sensor identity distinguishes the two Canon cameras in Gehler-Shi.
  • Deep Models: Semantic features are CNN outputs computed from pre-whitebalance images, and they have a modest positive effect.
  • Frequency-Domain Optimization: The modified FFT maps an n×n real image to an n^2-dimensional real vector without redundant coefficients, enabling frequency-domain preconditioning.The inverse mapping restores Hermitian symmetry before applying the inverse FFT, and the scaling preserves the magnitude of the image.

E. Results

The results examine posterior uncertainty alongside angular error across benchmark images and visualize white-balanced camera images. Entropy-based ordering provides a way to assess whether predicted confidence tracks error.

  • Uncertainty and error: Posterior covariance supplies an entropy-based confidence measure for each illuminant estimate.The entropy is derived from the determinant of the covariance matrix, with lower entropy indicating a tighter output distribution.
  • Uncertainty and error: Entropy-ordered error is computed by sorting all test images by ascending entropy and measuring twice the normalized area under the cumulative-error curve.The metric uses the union of test-set images across cross-validation folds.
  • Uncertainty and error: Entropy-ordered error is substantially lower than mean error for both datasets, indicating that low entropy is suggestive of low error.The paper notes that no existing color constancy technique was known to explicitly predict a confidence measure for comparison.
  • Visualized results: The supplemental visualizations include model predictions for Gehler-Shi and Cheng images selected across increasing angular error.Each dataset uses three-fold cross-validation, with images sampled from lowest to highest error.
  • Visualized results: Additional examples show HDR+ images from a Nexus 6 after white balancing with the thumbnail-oriented Model Q.These images are presented as a separate visualization of the model's white-balancing output.

F. Color Rendering

Color rendering visualizations apply estimated illuminant gains, camera-specific color correction, and sRGB gamma correction. The displayed examples span benchmark images with reported errors and entropies, plus Nexus 6 HDR+ outputs.

  • Rendering pipeline: Rendered images apply RGB gains from the estimated illuminant, a color correction matrix, and an sRGB gamma-correction function.The color correction matrices are estimated separately for each camera and affect visualization rather than illuminant estimation or reported results.
  • Benchmark examples: The remaining Gehler-Shi examples using Model J report errors of 1.61°, 2.35°, 3.84°, and 21.64°.These figures extend the displayed range toward the highest-error examples.
  • Benchmark examples: The Cheng examples using Model J report errors from 0.12° to 17.85° and entropies from −6.82 to −3.04.The displayed figures provide benchmark examples from the Cheng dataset with varying error and entropy.
  • Mobile-camera examples: Figure 22 samples unedited HDR+ images from a Nexus 6 after processing with Model Q.Model Q is the version designed to run on thumbnail images.
Loading 1611.07596v3…