Source-linked AI summary
Learning Data-driven Reflectance Priors for Intrinsic Image Decomposition
Tinghui Zhou, Philipp Krähenbühl, Alexei A. Efros
TL;DR
Intrinsic image decomposition must separate reflectance from shading despite an ill-posed image formation model and weaknesses in hand-crafted priors. The paper learns pairwise relative reflectance from human annotations, integrates it into energy minimization, and reports improved decomposition performance, while still relying on hand-crafted unary terms for state-of-the-art results.
Problem
Intrinsic image decomposition is ill-posed, and existing hand-crafted priors can fail under strong shadows, sharp surface-orientation changes, and smoothly varying textures.
Method
An end-to-end convolutional network predicts whether two pixels are brighter, darker, or equal in reflectance, and its pairwise prior is integrated into existing energy-minimization frameworks.
Results
The best variant achieves WHDR = 19.9 and error rate = 27.3, improving on Bell et al.’s decomposition quantitatively and qualitatively.
Takeaways & Limitations
Relative reflectance priors provide decomposition benefits, especially for scenes under challenging illumination conditions.
Takeaways & Limitations
State-of-the-art results still use hand-crafted unaries on chromaticity and shading alongside the learned reflectance prior.
Abstract
from arXiv · showhide
We propose a data-driven approach for intrinsic image decomposition, which is the process of inferring the confounding factors of reflectance and shading in an image. We pose this as a two-stage learning problem. First, we train a model to predict relative reflectance ordering between image patches (`brighter', `darker', `same') from large-scale human annotations, producing a data-driven reflectance prior. Second, we show how to naturally integrate this learned prior into existing energy minimization frameworks for intrinsic image decomposition. We compare our method to the state-of-the-art approach of Bell et al. on both decomposition and image relighting tasks, demonstrating the benefits of the simple relative reflectance prior, especially for scenes under challenging lighting conditions.
1. Introduction
Intrinsic image decomposition separates reflectance from illumination, but the problem is ill-posed because observed intensity can be explained by either factor. This paper learns relative reflectance priors from human pairwise judgments and integrates them into decomposition.
- 1. Introduction: The motivating example shows that regions can differ in brightness while still sharing the same surface reflectance.The mini-sofa illustrates this distinction between illumination effects and material identity.
- 1. Introduction: Intrinsic image decomposition models an observed luminance image as reflectance multiplied by shading, making inverse recovery under-constrained.Any pixel intensity can otherwise be attributed to either reflectance or shading.
- 1. Introduction: Hand-crafted priors are often too weak, failing under strong shadows, sharp surface-orientation changes, and smoothly varying planar textures.The Retinex-style assumption that large gradients indicate reflectance edges is unreliable in these settings.
- 1. Introduction: The paper learns a relative reflectance prior from pairwise human judgments rather than training a classifier to distinguish reflectance from shading on individual patches.The prior targets whether two regions are brighter, darker, or equal in reflectance despite differing intensities.
- 1. Introduction: The learned prior is integrated into existing energy-minimization frameworks for intrinsic image decomposition, with benefits demonstrated especially under challenging illumination.The training data come from the Intrinsic Images in the Wild database, which contains human relative-reflectance judgments across real-world scenes.
2. Learning a model of reflectance
The method predicts pairwise relative reflectance with a multi-stream convolutional network, then converts noisy pairwise outputs into a globally consistent reflectance ordering for decomposition.
- 2. Learning a model of reflectance: Relative reflectance is defined as whether two scene points are equal, darker, or brighter in reflectance.The classifier estimates ordering rather than absolute reflectance values.
- 2.1. Relative reflectance classifier: The multi-stream network combines local features around both pixels, global scene features, and their spatial coordinates.Local feature-extraction streams share weights; concatenated features pass through three fully connected layers.
- 2.1. Relative reflectance classifier: The network learns from pairwise human judgments augmented through symmetry and transitivity, using end-to-end softmax training.Its predictions can still be globally inconsistent because the annotations and pairwise outputs are noisy.
- 2.2. Globally consistent reflectance estimate: A constrained optimization reconciles classifier scores into a globally consistent reflectance ordering using nonnegative pairwise weights and slack variables.The resulting formulation becomes a global energy minimization whose penalties reflect disagreement with the classifier.
- 2.2. Globally consistent reflectance estimate: The global ordering supplies a reflectance prior that can be incorporated into a standard intrinsic-image pipeline to recover absolute reflectance.The ordering alone lacks absolute reflectance information, so it is combined with decomposition terms.
3. Intrinsic image decomposition
The decomposition framework combines unary reconstruction and pairwise smoothness terms, then replaces an arbitrary color-based reflectance regularizer with a learned pairwise prior. A low-rank Nyström approximation makes the classifier-guided dense filtering computationally feasible for integration into a fully connected CRF.
- The Bell et al. framework models intrinsic decomposition as energy minimization in a fully connected CRF, recovering reflectance and shading from an input image.
- Unary terms impose lightweight shading and reflectance priors while enforcing reconstruction, whereas pairwise terms enforce reflectance and lighting smoothness.
- The original reflectance term encourages similar reflectance for pixels with similar colors, but the paper identifies this color-based regularizer as arbitrary and replaces it with a data-driven pairwise prior.
- Dense evaluation of the classifier-guided pairwise term is computationally intractable for more than a few thousand pixels, motivating an efficient approximation.
- The classifier output is low rank; sampled experiments with K = 500 points never produced rank above 100, supporting a low-rank approximation.
- Nyström approximation samples K = 64 grid points, reducing message passing to a few hundred milliseconds instead of multiple days for naive evaluation.
4. Experiments
Experiments evaluate the learned reflectance prior on IIW and ILC data, comparing network predictions, decomposition variants, relighting, and learned feature structure against Bell et al. The full system performs best among the tested decomposition variants and generally improves boundary separation and relighting under challenging illumination, while some cases favor stronger smoothness constraints.
- Experimental setup: The evaluation uses IIW and ILC datasets, with Bell et al. as the main state-of-the-art intrinsic decomposition baseline.The models are trained and evaluated using the dataset split of Narihira et al.
- Network performance: 22,903,366 augmented comparisons are generated, including 18,621,626 for training and 4,281,740 for testing.Augmentation uses symmetry and transitivity to create additional, potentially long-range pixel pairs.
- Network performance: Error rate = 24.6 is obtained on augmented annotations, while the original-annotation model performs poorly on these longer-range comparisons.The authors attribute this contrast to a spatial-proximity bias in the original IIW annotations that is reduced by augmentation.
- Intrinsic image decomposition: The final decomposition generally separates reflectance and shading boundaries better than Bell et al., especially under challenging lighting, while stronger smoothness constraints win in some cases.Relighting experiments also significantly outperform Bell et al. quantitatively and perceptually, and learned features appear predictive of reflectance and surface properties.
Discussion
The learned features capture surface color while also grouping patches by material or local appearance. However, state-of-the-art decomposition still relies partly on hand-crafted chromaticity and shading unaries.
- Limitation: The method still uses hand-crafted unaries on chromaticity and shading to achieve state-of-the-art decomposition results.The authors identify replacing these unaries with learned ones as beyond the paper’s scope.
- Feature representation: The learned features are predictive of surface color and coherently group bathroom tiles, wallpaper, and cloth by material properties or local appearance.These groupings indicate that the representation captures properties beyond reflectance.