Source-linked AI summary
Reflectance Adaptive Filtering Improves Intrinsic Image Estimation
Thomas Nestmeyer, Peter V. Gehler
TL;DR
Intrinsic image estimation lacks large-scale, precise, realistic decomposition ground truth, motivating learning from sparse IIW reflectance judgments. The paper proposes end-to-end dense CNN prediction and filtering with reflectance priors; filtered predictions reach the lowest reported WHDR for a dense decomposition on the stated test set. The authors conclude that explicit prior knowledge remains important alongside learning.
Problem
Intrinsic image learning lacks large-scale, precise, realistic ground-truth decompositions, while IIW provides sparse relative reflectance judgments.
Source: §1 · p. 1Method
The paper trains a CNN to predict dense reflectance from pairwise judgments and applies joint bilateral or guided filtering to incorporate reflectance priors.
Source: §1 · p. 2, §5.1 · p. 6, §5.2 · p. 6Results
15.78% WHDR on its testset is reported as the lowest performance for a dense decomposition, while filtered CNN output is on par with the best published learning-based methods.
Source: §6 · p. 8Takeaways & Limitations
The authors conclude that explicit priors through regularization, CRFs, or filtering still drive performance, and available annotation may be insufficient for learning alone.
Source: §1 · p. 2WHDR evaluates reflectance only at labeled locations, not across every point in an image.
Source: §3.1 · p. 3
Abstract
from arXiv · showhide
Separating an image into reflectance and shading layers poses a challenge for learning approaches because no large corpus of precise and realistic ground truth decompositions exists. The Intrinsic Images in the Wild~(IIW) dataset provides a sparse set of relative human reflectance judgments, which serves as a standard benchmark for intrinsic images. A number of methods use IIW to learn statistical dependencies between the images and their reflectance layer. Although learning plays an important role for high performance, we show that a standard signal processing technique achieves performance on par with current state-of-the-art. We propose a loss function for CNN learning of dense reflectance predictions. Our results show a simple pixel-wise decision, without any context or prior knowledge, is sufficient to provide a strong baseline on IIW. This sets a competitive baseline which only two other approaches surpass. We then develop a joint bilateral filtering method that implements strong prior knowledge about reflectance constancy. This filtering operation can be applied to any intrinsic image algorithm and we improve several previous results achieving a new state-of-the-art on IIW. Our findings suggest that the effect of learning-based approaches may have been over-estimated so far. Explicit prior knowledge is still at least as important to obtain high performance in intrinsic image decompositions.
1. Introduction
Intrinsic image estimation must separate reflectance and shading despite an ill-posed problem, diverse appearances, and limited realistic ground truth. The paper tests learning from sparse reflectance judgments and combines dense CNN predictions with filtering that encodes reflectance priors.
1. Introduction
- The problem remains open because models must resolve ill-posedness while handling varied image appearances.Source: §1 · p. 1
- The paper asks whether CNNs can perform well on intrinsic decomposition and whether prior terms remain necessary when human weak labels are available.Source: §1 · p. 1
- Accurate training data is scarce: existing intrinsic-image datasets lack realism, diversity, detail, or extent, while MPI-Sintel scenes differ from real-world images.IIW provides 5,230 mostly indoor photos with sparse relative reflectance judgments, enabling empirical validation and training.Source: §1 · pp. 1–2
- A CNN trained end-to-end from pairwise judgments gives competitive performance even when it predicts reflectance from pixels alone, while filtering adds strong reflectance priors and improves IIW results.The authors conclude that prior knowledge through regularization, CRFs, or filtering still drives performance, and available annotation may be insufficient for learning alone.Source: §1 · p. 2
2. Related Work
Prior-based methods, filtering, and learning-based approaches address intrinsic image estimation under different data and output constraints. The related work contrasts their scope, use of priors, and ability to produce dense decompositions.
2. Related Work
- A richly prior-based method achieved impressive MIT intrinsic-dataset results but was limited to single masked objects and struggled with complex illumination.Its priors cover reflectance, shading, shape, and illumination.Source: §2 · p. 2
- Filtering followed by clustering groups pixels into same-reflectance regions and achieved the best IIW results, but required several minutes per image.The paper builds on this approach with filtering applicable to other intrinsic-image estimates.Source: §2 · p. 2
- IIW introduced relative-reflectance human annotations as training and test data and enabled comparison of algorithms across real-world scenes.Source: §2 · p. 2
- Methods trained on IIW relative judgments often predict pairwise labels rather than dense reflectance, requiring separate post-processing to produce dense decompositions.These methods achieve good IIW results and take several seconds per image.Source: §2 · p. 2
- A dense CNN trained against rendered MPI-Sintel decompositions did not generalize well to real-world IIW images, while another approach requires dense ground truth and reports no IIW results [25] [24].Source: §2 · pp. 2–3
- Relative-depth learning also uses CNNs and pairwise judgments [12], but accurate ground-truth depth is capturable, unlike reflectance and shading.Source: §2 · p. 3
3. Preliminaries
The formulation assumes linear RGB, Lambertian reflectance, and achromatic light, reducing decomposition to estimating one scalar per pixel. WHDR evaluates predictions against sparse human judgments, and the section introduces a margin-based loss for learning dense reflectance.
3. Preliminaries
- Under linear RGB and Lambertian reflectance, the image is modeled as the pixel- and channel-wise product I = RS; achromatic light reduces estimation to one scalar per pixel.The achromatic-light assumption is often violated in IIW, especially with multiple light sources, but WHDR is invariant to it because it compares relative lightness rather than color.Source: §3 · p. 3
3.1. A quantitative measure for intrinsic images
- WHDR scores weighted disagreements between predictions and human judgments of darker, lighter, or equal reflectance at annotated location pairs.The annotations have varying confidence and are sparse; the number per image ranges from 1 to 1,181, with a median of 113.Source: §3.1 · p. 3
- WHDR evaluates only labeled locations, so it does not assess reflectance predictions at every image point.Source: §3.1 · p. 3
- The WHDR-Hinge loss uses δ to set the darker/lighter or equal-class decision boundary and ξ to encourage a margin between neighboring classes.When ξ > δ, the equal class always has non-zero loss.Source: Figure 1 · p. 4
- Unlike pairwise classifiers that need separate post-processing for dense output, the proposed approach directly predicts a dense reflectance map.Source: §3.1 · pp. 3–4
4. Direct Reflectance Prediction with a CNN
The section trains a CNN to produce dense reflectance from sparse pairwise IIW judgments using a WHDR-oriented loss. Its pixel-wise model performs competitively without spatial context.
4.1. WHDR-Hinge loss
- The WHDR-hinge objective trains dense reflectance predictions against human pairwise judgments by comparing predicted reflectance ratios and weighting losses over annotated pairs.The margin ξ separates neighboring judgment classes, while δ sets the WHDR threshold.Source: §4 · p. 4, §4.1 · p. 4
4.2. Train and test data set
- The IIW split uses 70% of images for training, 10% for validation, and 20% for testing, retaining the test images from.Every tenth-image series contributes its seventh image to validation; the test set follows the split proposed in.Source: §4.2 · p. 4
4.3. Network architecture of the CNN
- The CNN applies convolutional layers to linearized RGB input and fuses skipped-layer features to predict a bounded, single-channel reflectance intensity.A final layer recovers RGB reflectance and shading; the sigmoid bounds intensity because reflectance and shading are identifiable only up to a constant.Source: §4.3 · p. 4
4.4. Experiments
- The pixel-wise lookup achieves better WHDR than most methods on IIW, despite using no image context.The network learns a pixel-wise mapping rather than relying on spatial context.Source: §4.4 · p. 5, §4.5 · p. 5
- Kernel size has little effect, and the selected network uses n = 5 layers, f = 25 filters, and k = 1, yielding a pixel-wise lookup table.Layer count and filter count have little influence above n ≥ 2 and f ≥ 24; Figure 2(a) plots WHDR across depths and filter counts.Source: §4.4 · p. 5, Figure 2 · p. 5
- Alternative layouts, PReLU nonlinearities, and dilated convolutions did not improve results; simpler networks performed better, which the authors attribute to weakly labeled training data.The tested alternatives included removing skip connections and using a U-net-like architecture [30].Source: §4.4 · p. 5
4.5. Discussion of the results
- The CNN maps RGB colors to reflectance intensity, but bright saturated colors may receive misleading human judgments; piecewise-constant reflectance is proposed to improve the result.The lookup table shows many colors judged similar to white, while blue is mostly judged darker than green.Source: §4.5 · p. 5, Figure 3 · p. 5
- 25.7 WHDR results from rescaling input intensities to [0.55, 1], which favors the two-thirds of IIW judgments labeled equal.The CNN output averages [0.48, 0.96], and its nonlinearity makes more unequal judgments correct through small color variations.Source: §4.5 · p. 5
5. Reflectance Filtering
The section uses image-aware filtering to encourage reflectance constancy, with a guidance image determining how the input is smoothed. Filtering improves intrinsic-image estimates, and repeated guided filtering with flattened guidance reaches a new state of the art on IIW.
5. Reflectance Filtering
- The method applies a filter to encode the prior that scenes contain a sparse set of reflectances, and it can be integrated into existing intrinsic-image techniques.The authors report that filtering reflectance estimates always improves performance.Source: §5 · pp. 5–6
5.1. Image-aware filtering
- Image-aware filtering smooths input p under guidance image I; joint bilateral weights favor pixels close in space and guidance-image intensity.The guided filter is edge-preserving and avoids the bilateral filter’s quadratic dependency on kernel size, especially at larger spatial scales.Source: §5.1 · p. 6
5.2. Filtering for piecewise constancy
- The L1-flattening optimization supplies a piecewise-constant guidance image by grouping pixels into regions of similar reflectance.Its objective combines local flattening, global sparsity, and image-approximation terms.Source: §5.2 · p. 6
- 18.1 WHDR with BF(CNN,flat) and 17.7 with GF(CNN,flat) approach the 17.67 WHDR state of the art.Using the L1-flattened result directly as reflectance gives 20.9 WHDR, indicating complementary information in the CNN output and guidance image.Source: Table 3 · p. 7, §5.2 · p. 7
- 15.78 WHDR after three guided-filter applications improves ’s 17.85 WHDR result and represents a new state of the art.A single application yields 16.38 WHDR.Source: §5.2 · p. 7
- Filtering the CNN prediction with the input image lowers WHDR from 19.5 to 18.9 with bilateral filtering and to 19.2 with guided filtering.The guided-filter settings were r = 7 and ε = 52.Source: §5.2 · p. 7
5.3. Discussion
- Guided filtering generally improves performance and is magnitudes faster than bilateral filtering, adding less than 0.1 s per image; computing the flattened guidance is the bottleneck.Some cases favor the joint bilateral filter.Source: §5.3 · p. 7
6. Conclusion
The paper pairs a context-free CNN reflectance predictor with filtering that encodes reflectance priors. The filtered results set a strong IIW benchmark, and the authors conclude that strong prior knowledge remains important for empirical performance.
6. Conclusion
- Figure 6 shows example decompositions by [38] before and after filtering with a flat guidance image.Source: Figure 6 · p. 8
- A context-free per-pixel reflectance judgment is sufficient for competitive results, setting a lower bar for learning methods on IIW.The CNN uses a hinge loss and predicts a dense result without post-processing.Source: §6 · p. 8
- Filtering the result of reaches 15.78% WHDR, the lowest reported for a dense decomposition, and filtered CNN output matches the best published learning-based methods.The paper attributes the result to filtering that implements piecewise-constant and sparse reflectance assumptions.Source: §6 · p. 8
- The authors conclude that strong prior knowledge drives empirical performance and identify combining priors with learning from sparse pairwise judgments as future work.They also propose replacing ’s expensive optimization with CNN inference as a direction toward fast, high-quality intrinsic video decomposition.Source: §6 · p. 8
A.1. Data Augmentation
Transitive closure greatly increased the training comparisons but did not improve WHDR on the original comparisons.
A.1. Data Augmentation
- Despite adding more than 20 million comparisons (23.6 times as many), transitive-closure augmentation did not improve WHDR computed on the original comparisons.Generated comparison weights used the minimum confidence of their source relations; contradictory relations were removed by a consistency check.Source: §A.1 · p. 10
A.2. Weak label analysis
The appendix examines how training-label quantity, network hyperparameters, and reflectance scaling affect WHDR.
A.2. Weak label analysis
- About 50% of annotation pairs per image can be removed before WHDR loss starts to decrease; removing roughly 400 more-densely annotated images degrades performance.The removed images contain 303–1181 evaluated pairs each (median 916), compared with 1–216 pairs (median 108) for other images.Source: §A.2 · p. 10, Figure 8 · p. 10
A.3. Network Hyper-Parameters
- Larger kernel sizes overfit more heavily and yield higher validation mean WHDR, while performance differs little between kernel sizes k = 1 and k = 3.The sweep covers network depth and filter count across kernel sizes k = 1, 3, 5, 7, and 9; some settings are missing because of graphics-card memory limits.Source: §A.3 · p. 10, Figure 9 · p. 11
A.4. Rescaling of lower bound
- Scaling reflectance values from [0, 1] to [a, 1] changes reflectance ratios and thus WHDR; a = 0.55 achieves 25.7 WHDR on the test set and outperforms Retinex.The paper links small reflectance variations below the δ threshold for the “equal” class to the use of filtering to enforce piecewise constancy.Source: §A.4 · p. 10
B.1. Hyper-Parameters of Filters
The filtering experiments compare bilateral, guided, and flat filtering for direct CNN predictions, and also apply filtering to.
B.1. Hyper-Parameters of Filters
- Bilateral filtering improves direct CNN test performance from 19.5 to 18.9 using σs = 22 and σr = 20.The spatial and color scales were selected for the lowest mean WHDR on training and validation data.Source: §B.1 · p. 11
- Guided filtering improves test performance to 19.2 with r = 7 and ε = 52.Source: §B.1 · p. 11
- 17.7 test performance is achieved by flat filtering of direct CNN reflectance predictions with r = 45 and ε = 3.These parameters were selected using validation performance.Source: §B.1 · p. 11
- 15.8 mean WHDR outperforms state of the art by 1.9 percentage points when the method of Zoran et al. 2015 is filtered with the CNN-selected parameters.The parameters were reused because only the method’s test set was available, preventing optimization on training or validation data.Source: §B.1 · p. 11
B.2. Extended Qualitative Results
The qualitative comparisons show that flat-guided reflectance filtering removes staircase effects in Zoran et al.’s outputs, improving both visual and quantitative results. The figures also compare filtering scales and show reflectance and shading outputs alongside input-image judgments.
B.2. Extended Qualitative Results
- Filtering with the flat image removes staircase effects from Zoran et al.’s reflectance and improves qualitative and quantitative results.The displayed images are randomly sampled from the intersection of the Narihira and Zoran test splits.Source: §B.2 · p. 11
- Figure 10 compares training- and validation-set mean WHDR across BF(CNN, CNN), GF(CNN, CNN), and GF(CNN, flat) at varying color and spatial scales.The caption identifies the methods and scale variation but does not report specific values or a winning setting.Source: Figure 10 · p. 12
- Figure 11 presents input images, evaluated comparisons, and flat guidance above color reflectance, grayscale reflectance, and shading outputs.Comparison-line colors indicate confidence; the narrow line end marks the point judged darker, while straight lines indicate “about the same.”Source: Figure 11 · p. 12
- The extended qualitative examples in Figures 12 and 13 use Photo IDs 78671, 9499, 60820, and 34647.Both figures extend the comparisons in Figure 11.Source: Figure 12 · p. 13, Figure 13 · p. 14