Source-linked AI summary
Learning Intrinsic Image Decomposition from Watching the World
Zhengqi Li, Noah Snavely
TL;DR
Single-view intrinsic decomposition is ill-posed, while scalable ground-truth collection is difficult. The paper learns from fixed-viewpoint sequences with changing illumination, using sequence-consistency losses to train a model that predicts decompositions from single images. It reports competitive or superior performance across several benchmarks, including results on MIT that improve over fully supervised learning methods.
Problem
Single-view intrinsic decomposition is highly ill-posed, and direct ground truth training data is difficult to collect at scale.
Method
The method trains a feed-forward network on unlabeled, fixed-viewpoint image sequences with varying illumination, using losses that evaluate predictions across whole sequences.
Results
The method achieves competitive or superior performance on IIW and SAW without their annotations and improves MIT results over learning methods using full supervised ground truth.
Takeaways & Limitations
Training from changing-illumination videos can produce competitive single-image intrinsic decompositions across diverse benchmarks.
Takeaways & Limitations
The shading-smoothness weight based on image derivatives does not always capture desired smoothness for textured surfaces because of noise in dark regions.
Abstract
from arXiv · showhide
Single-view intrinsic image decomposition is a highly ill-posed problem, and so a promising approach is to learn from large amounts of data. However, it is difficult to collect ground truth training data at scale for intrinsic images. In this paper, we explore a different approach to learning intrinsic images: observing image sequences over time depicting the same scene under changing illumination, and learning single-view decompositions that are consistent with these changes. This approach allows us to learn without ground truth decompositions, and to instead exploit information available from multiple images when training. Our trained model can then be applied at test time to single views. We describe a new learning framework based on this idea, including new loss functions that can be efficiently evaluated over entire sequences. While prior learning-based methods achieve good performance on specific benchmarks, we show that our approach generalizes well to several diverse datasets, including MIT intrinsic images, Intrinsic Images in the Wild and Shading Annotations in the Wild.
1. Introduction
The paper addresses ill-posed intrinsic image decomposition by learning from unlabeled, fixed-viewpoint videos with changing illumination rather than collecting direct ground truth. It trains on sequences but predicts single-view decompositions, using sequence-level consistency losses and achieving competitive or superior benchmark performance.
- Motivation: Intrinsic image decomposition factorizes an image I into reflectance R and shading S, but the problem is highly ill-posed and direct ground truth is difficult to collect.Prior ground truth sources include painted objects, synthetic renderings, and manual annotations, each with significant limitations.
- Training data: BIGTIME uses Internet image sequences with fixed viewpoints and varying illumination as an alternative source of training data.Although these sequences lack ground truth, they provide constraints requiring predictions to remain consistent across the sequence.
- Learning framework: The model trains on image sequences but learns to infer decompositions from single views in a feed-forward framework.Its losses incorporate priors similar to optimization-based methods while retaining machine-learning inference.
- Learning framework: Two sequence-level losses evaluate all image pairs and all pixels jointly, with efficient implementations inside a deep network.The losses are an all-pairs weighted least-squares loss and a dense spatio-temporal smoothness loss.
- Results: The method achieves competitive or superior results on IIW and SAW without their annotations and improves results on MIT relative to methods trained with full supervision.These evaluations indicate generalization across diverse intrinsic-image benchmarks.
2. Related work
Prior intrinsic-image work uses optimization priors, multiple observations, or supervised learning, but each approach has important scope or data limitations. The paper positions sequence-based training as an alternative that provides full-image constraints while enabling single-image inference.
- Optimization methods: Optimization-based intrinsic-image methods address ill-posedness by imposing priors such as Retinex gradient assumptions and dense-CRF inference.Many different priors and optimization tools have been proposed to guide decomposition.
- Multiple observations: Methods using multiple observations assume constant reflectance and varying shading, but often require many images captured outdoors or under controlled lighting.Unlike these methods, the proposed model can run on a single image after training.
- Supervised learning: Supervised learning methods commonly train on synthetic or object-centric datasets, while some real-world methods rely on sparse IIW or SAW annotations.These datasets and annotation strategies do not provide broad, full-image supervision at scale.
- Proposed alternative: The proposed alternative uses image sequences that are easier to capture and provide full-image constraints during training.The approach avoids requiring direct decomposition annotations for sequence-based learning.
3. Overview and network architecture
The method introduces BIGTIME and a network that processes sequence images independently while coupling their predictions through a joint loss during training. Consequently, multiple images are unnecessary at test time.
- Contributions: The paper contributes BIGTIME, a dataset of image sequences, and a method for learning single-view intrinsic images from that data.The dataset and learning approach are presented as the two main contributions.
- Training architecture: During training, the CNN takes an image sequence and produces reflectance and shading for each image in a feed-forward fashion.The sequence images interact through a loss that evaluates predicted decompositions jointly over the entire sequence.
- Inference: Because the network processes each image independently, inference requires only a single image rather than multiple views.This separates the multi-image training regime from the single-view deployment setting.
- Network architecture: The network is a U-Net variant with one encoder and separate decoders for log-reflectance and log-shading.Skip connections support both decoders, and an additional side output predicts illumination color.
4. Dataset
BIGTIME combines indoor and outdoor sequences with varying illumination, including challenging conditions that expose the limits of prior sequence-based intrinsic-image methods. Despite these difficulties, the authors report that their approach remains robust to strong illumination.
- Dataset construction: BIGTIME includes indoor and outdoor scenes with varying illumination, expanding beyond time-lapse datasets focused primarily on outdoor scenes.Indoor sequences came from online and prior sources, while outdoor sequences came from AMOS and Time Hallucination.
- Dataset challenges: Many relevant clips span short periods and show little change in light direction.This limits the illumination variation available for learning from individual sequences.
- Dataset challenges: Some room-light clips contain fewer than 8 valid images with different lighting conditions.The limited number of valid frames reduces the sequence information available under these conditions.
- Dataset challenges: Strong sunlight or shadows can cause indoor-scene saturation or clipping that breaks intrinsic-image algorithms.The dataset therefore includes high-dynamic-range examples that are difficult for decomposition methods.
- Dataset challenges: Prior sequence-based intrinsic-image methods often fail on these indoor videos because their assumptions mainly hold for outdoor or laboratory sequences.Figure 4 illustrates such failure cases, while Figure 3 shows examples with colorful illumination and strong sunlight or shadows.
- Observed robustness: The proposed approach is reported to remain robust under strong illumination conditions and to generalize well to IIW and SAW.This result contrasts with the failures observed for prior sequence-based methods on challenging indoor videos.
5. Approach
The framework trains a feed-forward single-view intrinsic-image predictor from fixed-view sequences with varying illumination, using sequence-level priors encoded in differentiable losses. It combines image formation, consistency, smoothness, and efficient all-pairs computations to make training tractable.
- Framework: The model predicts log-reflectance and log-shading from each image while training on sequences captured from a fixed viewpoint under varying illumination.Each image has predicted reflectance and shading, with masks excluding invalid regions such as saturated pixels, sky, and dynamic objects.
- Framework: Training formulates the sequence decomposition as a dense CRF energy and uses that energy directly as the neural network’s differentiable loss.The objective maximizes p(R, S|I), equivalently minimizing an energy over predicted reflectance and shading.
- Sequence-level losses: All-pairs reconstruction pairs each image’s shading with every sequence image’s reflectance, enforcing reconstruction under the assumption that reflectance remains constant over time.The reconstruction is weighted by input luminance because dark pixels are noisier and log-space differences are magnified there.
- Sequence-level losses: Including all-pairs reconstruction improves prediction results, while a closed-form computational trick reduces its direct O(m^2n) cost to O(mn).The linear-time implementation is described as key to making training tractable; the same reduction is also applied to reflectance consistency.
- Reflectance smoothness: Dense spatio-temporal reflectance smoothness compares predicted reflectance across all sequence pixels using chromaticity and intensity affinities, with bilateral embedding reducing computation from O(m^2n^2) to O((d + 1)mn).The affinity features include spatial position, intensity, and two chromaticity coordinates; the resulting term is differentiable and efficiently computed through sparse matrices.
- Shading smoothness: Shading smoothness uses neighboring-pixel weights derived from median log-image derivatives across the sequence, while an additional median-normalized weight addresses textured-surface failures caused by dark-region noise.The original weight discourages smoothness at gradients differing from the sequence median, such as shadow boundaries; the extra weight targets textures such as checkerboard floors.
6. Evaluation
The method is evaluated on IIW, SAW, and MIT using sequence-based training without target-dataset annotations or, for MIT, ground-truth training decompositions. It achieves competitive or superior results while exposing qualitative and modeling limitations.
- IIW and SAW evaluation: Training solely on BIGTIME generalizes competitively to IIW and SAW despite cross-dataset domain shift.Networks trained on Sintel, MIT, or ShapeNet do not generalize well to these datasets.
- IIW and SAW evaluation: Our method is competitive with optimization-based and learning-based methods on IIW, using full decompositions rather than pairwise CNN classifier outputs.The best WHDR uses pairwise classifier outputs, whereas this method evaluates full image decompositions.
- IIW and SAW evaluation: The method achieves the best tested SAW performance, and its full loss outperforms variants with terms removed.All methods, including ours, are trained without SAW data.
- Qualitative results: Qualitative IIW/SAW predictions are visually comparable to Bell et al. and Zhou et al., although dark shading predictions can reduce reflectance contrast.The reduced contrast does not hurt numerical performance.
- Qualitative results: At test time, two decoders predict reflectance and shading simultaneously, so outputs may not strictly satisfy I = R·S.The authors suggest using predictions as priors for optimization in future work.
- MIT intrinsic images: On MIT, the method numerically outperforms prior CNN-based supervised methods averaged over reflectance and shading while using only image sequences for training.Reflectance estimates are significantly better, while shading estimates are comparable to prior methods.
7. Conclusion
The paper learns intrinsic decompositions from image sequences with changing illumination instead of ground-truth decompositions. The resulting model runs on single images and produces competitive benchmark results, motivating future combinations with annotations and optimization.
- Conclusion: The method learns decompositions from varying-illumination image sequences rather than ground-truth decompositions.Training encourages predictions consistent across the observed sequences.
- Conclusion: The trained model runs on single images and produces competitive results on several benchmarks.The conclusion emphasizes learning decompositions by watching large amounts of video.
- Conclusion: Future work will combine the approach with IIW and SAW annotations and use its outputs as inputs to optimization-based methods.