Source-linked AI summary
CGIntrinsics: Better Intrinsic Image Decomposition through Physically-Based Rendering
Zhengqi Li, Noah Snavely
TL;DR
Intrinsic image decomposition requires separating reflectance from shading, yet accurate ground truth for real scenes is difficult to obtain. The paper introduces CGINTRINSICS, a physically based rendered scene dataset, and a partially supervised CNN training method combining full synthetic supervision with sparse real-image annotations. CGI-only models achieve state-of-the-art performance on IIW and SAW, with further gains from adding IIW and SAW data.
Problem
Intrinsic decomposition remains ill-posed, while collecting full ground truth for real-world scenes is extremely difficult.
Method
The paper builds CGINTRINSICS with over 20,000 high-quality rendered indoor scenes and trains a CNN using full CGI decompositions plus optional sparse IIW and SAW annotations.
Results
CGI-only decomposition networks achieve state-of-the-art performance on both IIW and SAW, and combined CGI, IIW, and SAW training performs better still.
Takeaways & Limitations
Carefully rendered synthetic data can generalize effectively to real-world intrinsic-image benchmarks, and rendering details materially affect that transfer.
Takeaways & Limitations
The approach depends on rendering choices because existing SUNCG-derived renderings were not ideal for training networks intended to generalize to real images.
Abstract
from arXiv · showhide
Intrinsic image decomposition is a challenging, long-standing computer vision problem for which ground truth data is very difficult to acquire. We explore the use of synthetic data for training CNN-based intrinsic image decomposition models, then applying these learned models to real-world images. To that end, we present \ICG, a new, large-scale dataset of physically-based rendered images of scenes with full ground truth decompositions. The rendering process we use is carefully designed to yield high-quality, realistic images, which we find to be crucial for this problem domain. We also propose a new end-to-end training method that learns better decompositions by leveraging \ICG, and optionally IIW and SAW, two recent datasets of sparse annotations on real-world images. Surprisingly, we find that a decomposition network trained solely on our synthetic data outperforms the state-of-the-art on both IIW and SAW, and performance improves even further when IIW and SAW data is added during training. Our work demonstrates the suprising effectiveness of carefully-rendered synthetic data for the intrinsic images task.
1 Introduction
Intrinsic image decomposition separates an image into reflectance and shading, but learning is hindered by scarce real-world ground truth. CGINTRINSICS addresses this with realistic scene renderings and a partially supervised approach that achieves strong transfer to real-image benchmarks.
- Intrinsic decomposition factors an input image I into reflectance R and shading S, but the problem remains ill-posed and real-scene ground truth is difficult to collect.
- Existing synthetic datasets are limited by object-centric scenes or simplified illumination, while IIW and SAW provide sparse, difficult-to-scale annotations.
- CGINTRINSICS is a large-scale scene-level dataset containing over 20,000 physically based rendered indoor-scene images with full decompositions.
- Rendering details are critical: high-quality physically based rendering can substantially improve how synthetic-trained CNNs generalize to real images.
- The proposed partially supervised method combines CGI ground truth with sparse IIW and SAW annotations to train direct reflectance and shading predictions.
- Networks trained solely on CGI achieve state-of-the-art performance on both IIW and SAW, while adding IIW and SAW data improves performance further.
2 Related work
Prior intrinsic-image methods use hand-crafted optimization priors, learned models, or sparse real-image annotations, but these approaches face generalization and supervision limitations. Synthetic scene datasets offer scale, yet rendering quality and domain mismatch remain central concerns.
- Optimization-based methods combine hand-crafted priors, but those assumptions are difficult to design and often violated in real-world scenes.
- Some optimization methods incorporate surface normals or depth, but requiring depth during optimization limits their applicability.
- Learning-based approaches include sophisticated prior models and CNN regressors trained on datasets such as Sintel, MIT Intrinsics, and ShapeNet.
- IIW and SAW provide sparse crowd-sourced reflectance and shading annotations for real indoor images, but those annotations alone are insufficient for direct regression.
- Synthetic data supports real-world prediction across vision tasks, but SUNCG-derived renderings can have low signal-to-noise ratios and unrealistic sensor properties.
3 CGINTRINSICS Dataset
CGINTRINSICS is built from physically based renderings but substantially reworks rendering quality and tone mapping to produce realistic indoor-scene images with full decompositions.
- Rendering quality: BDPT with 8,192 samples per pixel produced acceptable quality, increasing rendering time from 31 seconds to approximately 30 minutes per image.The higher sample count addresses challenging indirect illumination in SUNCG scenes.
- Tone mapping: CGI uses a simpler tone-mapping method than PBRS because Reinhard tone mapping can differ substantially from consumer-camera images.The authors report that their method better captures natural color, illumination, and shadows.
- Dataset quality: CGI renderings have higher SNR and greater realism than the original SUNCG/PBRS images, according to the visual comparison.The dataset masks light sources when producing ground-truth reflectance and shading.
- Tone mapping: The tone-mapping pipeline scales linear HDR radiance using r90, applies gamma correction, and clips the result to [0, 1].α maps r90 to 0.8, γ = 1/2.2, and at most 10% of pixels are saturated.
- Dataset construction: CGI re-rendered approximately 20,000 PBRS images and integrated 152 realistic renderings, focusing on indoor scenes with sophisticated structure and illumination.Unlike object-centered datasets, CGI includes effects such as cast shadows and spatially varying lighting; unlike IIW and SAW, it provides full ground truth collected at scale.
4 Learning Cross-Dataset Intrinsics
The method combines full CGI supervision with sparse IIW/SAW annotations and additional smoothness, ordinal, and reconstruction constraints to train intrinsic decompositions end-to-end.
- Cross-dataset supervision: The full training loss combines CGI, IIW, and SAW dataset losses, with CGI providing full ground truth and IIW/SAW providing sparse annotations.The network is trained jointly across the three datasets rather than treating sparse annotations as the only supervision.
- CGI supervision: For CGI images, the supervised loss combines scale-invariant mean-squared error with multi-scale gradient matching.The scale factors for reflectance and shading are computed by least squares, and gradient matching encourages piecewise smooth decompositions with sharp discontinuities.
- Cross-dataset effects: Adding IIW or SAW data qualitatively improves decomposition predictions, including more uniform quilt and floor reflectance and better texture assignment for paintings and sofas.The reported visual improvements concern both reflectance and shading predictions.
- IIW supervision: IIW supervision adds ordinal reflectance constraints, while CGI superpixels provide additional ordinal comparisons that can improve reflectance predictions.IIW relations indicate which of two points is darker or whether their reflectances are equal, weighted by annotation confidence.
- SAW supervision: SAW supervision constrains constant-shading regions, shadow-boundary regions, and shading discontinuities using annotated smoothness and non-smoothness information.Shadow-boundary points are dilated before training, and depth or normal discontinuities are masked from the shading smoothness term.
- Regularization: Real-image training data also receives reflectance and shading smoothness losses plus a reconstruction constraint linking the predicted components to the input image.Reflectance smoothness encourages piecewise-constant predictions, while densely connected shading smoothness can be evaluated in linear time using bilateral embeddings.
5 Evaluation
The method is evaluated on IIW and SAW real-scene benchmarks and on MIT Intrinsic Images for cross-dataset generalization. CGI-trained models achieve strong performance, while combining CGI with sparse real-image annotations improves results further.
- Evaluation setup: Evaluation uses unseen IIW and SAW test data, with additional generalization testing on the MIT Intrinsic Images benchmark.The study compares against several state-of-the-art intrinsic-image algorithms.
- Evaluation on IIW: CGI-trained models outperform prior synthetic-data models on IIW, indicating that rendering decisions substantially affect real-image generalization.Models trained on SUNCG, Sintel, MIT Intrinsics, or ShapeNet generalize poorly to IIW.
- Evaluation on IIW: Adding IIW and SAW training data to CGI further improves performance, whereas sparse IIW/SAW-only training produces qualitatively degenerate decompositions.The sparsity particularly harms predicted shading images.
- Evaluation on SAW: The gradient-weighted SAW metric downweights smooth constant-shading regions so textured regions contribute more strongly to evaluation error.This produces more distinguishable performance differences between methods.
- Evaluation on SAW: CGI-only networks achieve state-of-the-art SAW performance under both unweighted and gradient-weighted metrics, with further gains from adding IIW and SAW data.The gradient-weighted metric emphasizes errors in richly textured regions more than the unweighted metric.
- Evaluation on MIT intrinsic images: On MIT Intrinsic Images, CGI-trained models outperform ShapeNet-trained networks qualitatively and quantitatively, while SIRFS remains best for this object-focused benchmark.The results support improved generalization of CGI compared with existing datasets, although SIRFS is designed specifically for single objects.
6 Conclusion
The paper presents CGI and an end-to-end learning approach that combines datasets with different label types for intrinsic-image decomposition. Its evaluations show effectiveness on Internet photos of real-world scenes, while rendering details are identified as important.
- 6 Conclusion: CGI is a synthetic dataset for learning intrinsic images, paired with an end-to-end approach that leverages datasets containing different label types.The approach combines supervision from the proposed dataset with other annotation types.
- 6 Conclusion: The evaluations demonstrate the effectiveness of CGI on Internet photos of real-world scenes.This conclusion concerns transfer from synthetic training data to real-world imagery.
- 6 Conclusion: Rendering details matter for synthetic intrinsic-image training, and the authors hypothesize that improved physically-based rendering may benefit other vision tasks.The suggested tasks include normal prediction and semantic segmentation.