Self-Supervised Intrinsic Image Decomposition

1 Introduction

• Intrinsic decomposition is underconstrained: different shape and lighting assumptions can render the same observed image, so a valid reconstruction need not be the most probable explanation.The vase examples include both a flat-shape interpretation and an interpretation with unnatural red lighting.Source: §1 · p. 1, Figure 1 · p. 2
• Existing approaches rely on hand-tuned natural-image priors or complete ground-truth intrinsic images, which are difficult to generalize or expensive to annotate.The paper notes that constructing real-world intrinsic-image training data requires substantial human effort and millions of crowd-sourced annotations.Source: §1 · pp. 1–2
• RIN combines intrinsic-image prediction with learned shading and image reconstruction, using the reconstructed input to provide an unsupervised training signal.A shared encoder predicts reflectance, shape, and lighting; shape and lighting drive a differentiable shading function that is combined with reflectance.Source: §1 · p. 2, Figure 2 · p. 3
• Unlabeled-data transfer tests show adaptation across primitives to common objects, missing lighting conditions, and a dissimilar ShapeNet category.The approach also uses supervision on intermediate representations alongside reconstruction error.Source: §1 · p. 2

2 Related Work

• Intrinsic-image models represent an image as reflectance multiplied by shading, with shading further modeled from shape and ambient lighting.The interpretation of the shading function varies by implementation.Source: §2 · p. 3
• Retinex-inspired methods used gradient assumptions to separate color and shading, while later methods optimized intrinsic proposals under natural-image or physical priors [Land and McCann, 1971; Barron and Malik, 2015; Lombardi and Nishino, 2012, 2016; Oxholm and Nishino, 2016].SIRFS combines shape and lighting estimates with a spherical-harmonics renderer [Barron and Malik, 2015].Source: §2 · p. 3
• Deep-learning studies estimated reflectance, illumination, or fuller decompositions, including models incorporating specularity, occlusion, and skip connections [Tang et al., 2012; Narihira et al., 2015a, 2015b; Rematas et al., 2016; Hold-Geoffroy et al., 2017; Innamorati et al., 2017; Shi et al., 2017].Earlier work also used deep networks to learn reflectance priors or estimate illumination parameters.Source: §2 · p. 3
• RIN models the image-combination process to relax the need for complete ground truth, and its learned shader can handle shadows cast between objects.Other autoencoder work explored disentangled representations through transforming autoencoders, selective gradient updates, or mutual-information objectives [Hinton et al., 2011; Kulkarni et al., 2015; Chen et al., 2016].Source: §2 · p. 4, Figure 3 · p. 4

3.1 Use of Reconstruction

• RIN uses reconstruction error as a learning signal, unlike methods that force exact image matching or rely only on ground-truth labels.Exact matching can deprive a model of reconstruction error, while label-driven methods do not explicitly model reconstruction.Source: §3.1 · p. 4
• Low reconstruction error does not guarantee accurate intrinsic images, and all-ones shading permits a degenerate zero-error solution.The predictions therefore need further constraints to avoid such explanations.Source: §3.1 · p. 4

3.2 Shading Engine

• A learned shading function converts predicted shape and lighting into shading, avoiding a fixed Lambertian dot product.The authors note that a linear Lambertian function cannot model lighting that changes drastically across an image or ray-traced cast shadows.Source: §3.2 · pp. 4–5
• After training only on synthetic ShapeNet cars, the shader also produces outputs for real shapes, including a Beethoven bust, without further training.The figure varies horizontal light panning and light intensity.Source: Figure 4 · p. 5

3.3 Architecture

• RIN uses two encoder-decoder networks: one predicts reflectance, shape, and lighting, and the other approximates shading for recombination with reflectance.The intrinsic-image network has one encoder and separate decoders; the final recombination is componentwise multiplication of shading and reflectance.Source: §3.3 · p. 5
• On an airplane outside the motorbike training category, poorer intrinsic predictions are reflected in RIN's rendered reconstruction.The figure also shows SIRFS predictions, whose reflectance definition has no analogous reconstruction.Source: Figure 5 · p. 6
• Table 1 reports MSE for motorbikes and held-out airplanes; it does not directly compare lighting because RIN and SIRFS use different lighting representations.The caption describes a 27-component lighting vector for SIRFS.Source: Table 1 · p. 6

4 Experiments

• The reconstruction error is backpropagated through the differentiable shading model to improve intrinsic-image predictions, with decoders updated separately while others are held fixed.RIN uses a shared intrinsic-image encoder and three separate decoders.Source: §4 · p. 6
• RIN is first trained with ground-truth intrinsic-image labels, then further trained on unlabeled data using reconstruction loss alone.The supervised phase uses mean squared error on intrinsic-image predictions; both phases use Adam.Source: §4 · p. 6
• Each transfer minibatch combines unlabeled transfer examples with labeled data to keep representations from shifting too far from their supervised initialization.Half of each minibatch comes from the unlabeled transfer data and half from labeled data.Source: §4 · p. 6

4.1 Supervised training

• Intrinsic-image predictions that were accurate on ShapeNet motorbikes degraded on other classes, especially for shape, while shading generalized beyond training shapes.The shader also produced plausible predictions for real-world objects and was frozen during self-supervised training to prevent nonsensical shading; supervised experiments used 40,000 images.Source: §4.1 · pp. 7–8

4.2 Shape transfer

• 29% average improvement in held-out-shape predictions followed self-supervised transfer, with shading predictions improving alongside shape.Only the shape decoder was updated during transfer across the three held-out shapes.Source: §4.2 · p. 8

4.3 Lighting transfer

• 18% lower lighting mean-squared error followed self-supervised learning on right-lit images, and predictions better reflected the new lighting distribution.The model initially mirrored the left-lit training distribution and tended to predict centered lighting on right-lit test images.Source: §4.3 · p. 8, Figure 7 · p. 8

4.4 Category transfer

• 32% shading-prediction improvement and 21% reflectance improvement followed category transfer, while lighting predictions remained largely unchanged.Shape estimates did not always improve with shading because multiple shapes can produce the same shading under a given light; lighting was already adequate because training covered all lighting regions.Source: Table 3 · p. 9, §4.4 · p. 9

• RIN uses reconstruction loss to improve intrinsic-image representations and adapt predictions when labeled and unlabeled data differ in their underlying intrinsic images.Source: §5 · p. 9