PaperlayerView exact revision on arXiv

Source-linked AI summary

Direct Intrinsics: Learning Albedo-Shading Decomposition by Convolutional Regression

Takuya Narihira, Michael Maire, Stella X. Yu

arXiv:1512.02311v1cs.CV

TL;DR

Intrinsic image decomposition seeks to recover albedo and shading from a single color image, but real-image ground truth is difficult to obtain. The paper learns a multiscale CNN that directly predicts both components from RGB and trains it largely on synthetic decompositions. On Sintel, the RGB-only approach outperforms methods using RGB and depth, while the paper reports reasonable transfer to real MIT images.

  • Problem

    Real-image albedo-shading ground truth is difficult to collect, while IIW supplies sparse human reflectance judgements rather than physical decompositions.

  • Method

    Direct intrinsics learns a multiscale fully convolutional CNN that maps color images directly to albedo and shading outputs using synthetic training data.

  • Results

    On Sintel, MSCR+dropout+GL significantly outperforms competing methods on MSE and LMSE, despite using RGB alone while some competitors use depth.

  • Takeaways & Limitations

    The results indicate that combining CNNs with synthetic training data can address intrinsic-image decomposition without hand-designed physical priors or graph-based inference.

  • Takeaways & Limitations

    The authors speculate that predicting human reflectance judgements and physically correct decompositions may be discrepant, with possible domain shift involving IIW.

Abstract

from arXiv · show

We introduce a new approach to intrinsic image decomposition, the task of decomposing a single image into albedo and shading components. Our strategy, which we term direct intrinsics, is to learn a convolutional neural network (CNN) that directly predicts output albedo and shading channels from an input RGB image patch. Direct intrinsics is a departure from classical techniques for intrinsic image decomposition, which typically rely on physically-motivated priors and graph-based inference algorithms. The large-scale synthetic ground-truth of the MPI Sintel dataset plays a key role in training direct intrinsics. We demonstrate results on both the synthetic images of Sintel and the real images of the classic MIT intrinsic image dataset. On Sintel, direct intrinsics, using only RGB input, outperforms all prior work, including methods that rely on RGB+Depth input. Direct intrinsics also generalizes across modalities; it produces quite reasonable decompositions on the real images of the MIT dataset. Our results indicate that the marriage of CNNs with synthetic training data may be a powerful new technique for tackling classic problems in computer vision.

1. Introduction

Direct intrinsics replaces physics-based priors and graph inference with a data-driven CNN that predicts albedo and shading from RGB input. Synthetic Sintel decompositions provide the training data needed for this approach.

  • Motivation: The intrinsic image model represents color image I as the point-wise product of albedo A and shading S.Albedo denotes the physical reflectivity of scene surfaces.
  • Contribution: The paper reports that its RGB-only model outperforms modern approaches using RGB and depth input.The introduction frames this as a departure from classical intrinsic-image methods.
  • Direct intrinsics: Direct intrinsics learns a multiscale fully convolutional regression mapping color images directly to albedo and shading outputs.The approach forgoes physical modeling constraints and graph-based inference methods.
  • Training data: Large-scale ground-truth decompositions are difficult to collect for real images because acquisition requires control over lighting environments.Laboratory collection is possible, but realistic scenes are harder to capture with physical ground truth.
  • Training data: The IIW dataset provides sparse human relative-reflectance judgements rather than actual physical albedo-shading decompositions.The paper states that these judgements are insufficient for learning intrinsic decompositions from scratch.
  • Evaluation: The method addresses data scarcity by training on synthetic images and testing on both synthetic Sintel and real MIT images.MPI Sintel supplies photorealistic renders with albedo-shading ground truth derived from 3D models and art assets.

2. Direct Intrinsics

The direct intrinsics system is organized around a CNN architecture, training data, and loss-function design.

  • System components: The system description covers the CNN architecture, training data, and loss function used during learning.These components structure the paper’s account of direct intrinsics.

2.1. Model

The model uses a multiscale CNN to combine global scene context with fine image details while simultaneously predicting albedo and shading.

  • MSCR architecture: The multiscale architecture extracts global context at a coarse scale and feeds its output to a finer-scale network.This design is intended to provide both precise patch details and the overall scene gist.
  • MSCR architecture: A 1×1 convolution after upsampling replaces fully connected layers in scale 1, enabling arbitrary-sized fully convolutional inputs.The change preserves the multiscale design while removing a fixed image-size constraint.
  • Activations: PReLUs use a learnable negative slope for each activation-map channel, and the experiments report better convergence than with ReLUs.The slope parameter is associated with each channel’s activation map.
  • Outputs: The network simultaneously predicts two outputs: albedo and shading.These are the model’s paired intrinsic-image outputs.
  • Upsampling: Learned deconvolution optionally upsamples a richer 64-channel representation to three-channel output resolution.Without deconvolution, the model upsamples a three-channel output using fixed bilinear interpolation.
  • Architecture details: Scale 1 uses five AlexNet-style convolutional layers, while scale 2 uses four feature-extraction layers followed by albedo and shading prediction.The first scale-2 layer uses 9×9 filters and 96 output maps.

2.2. MPI Sintel Dataset

Training uses MPI Sintel clean-pass renders with synthetic albedo-shading ground truth, while defective pixels are excluded from evaluation and masked during training.

  • Training images: The training data uses MPI Sintel clean-pass images rather than final images containing additional graphics effects.This removes depth of field, motion blur, and fog from the training inputs.
  • Ground truth: Ground-truth shading is generated by rendering scenes with every element assigned a constant grey albedo.This produces shading targets from the underlying graphics scene.
  • Defect handling: Defective Sintel images are excluded from evaluation, but defective pixels are masked during training so those pixels do not contribute to training error.The masking strategy allows otherwise defective images to remain in the training set.

2.3. MIT Intrinsic Image Dataset

The MIT intrinsic image dataset provides real-image reflectance and shading ground truth, but its limited scale requires generated training examples.

  • The MIT dataset contains real-world object images acquired with specialized apparatus that provides reflectance and shading components as ground truth.In this paper, reflectance is synonymous with albedo.
  • Only one of each object’s 11 differently lit images has shading ground truth, limiting the available supervision.
  • The authors generate 10 additional shading images per object by element-wise division of the original image by scaled reflectance.The scale α minimizes the squared error between I and αA·S.
  • Including generated shading examples improves overall performance despite some MIT images not exactly satisfying I = αA · S.

2.4. Data Synthesis: Matching Sintel to MIT

Because Sintel’s rendering process differs from real-image formation, the authors resynthesize inputs from ground-truth albedo and shading to better match MIT.

  • Sintel ground-truth renderings do not satisfy I = αA · S, unlike the intended real-image formation model.
  • The ResynthSintel variant reconstructs I from ground-truth A and S and improves performance when testing on MIT.

2.5. Data Augmentation

The training pipeline augments images by cropping and mirroring, with optional scaling and rotation under the DA setting.

  • Training images are cropped and mirrored to generate additional examples throughout the experiments.
  • Optional data augmentation, denoted DA, additionally scales and rotates images.

2.6. Learning

Learning uses a fully convolutional network that predicts albedo and shading with scale-invariant losses, optionally adding a gradient loss for albedo. Dropout regularizes training because Sintel remains small for deep learning.

  • Learning objective: Given image I, the CNN predicts dense albedo A and shading S maps through learned parameters Θ.
  • Scale Invariant L2 Loss: Scale-invariant L2 loss is used because ground-truth albedo and shading intensities are not absolute.
  • Scale Invariant L2 Loss: λ = 0 gives least-squares loss, λ = 1 gives scale-invariant loss, and λ = 0.5 averages both; training uses λ = 0.5 separately and λ = 1 jointly.
  • Gradient L2 Loss: Gradient L2 loss compares prediction and ground-truth image gradients and is added to the albedo objective to favor piecewise-constant outputs.
  • Gradient L2 Loss: Gradient loss is not applied to shading because shading cannot be assumed piecewise constant.
  • Regularization: Dropout with probability 0.5 regularizes nearly all convolutional layers because Sintel’s 890 examples remain small for deep networks.

2.7. Implementation Details

The implementation trains convolutional networks with stochastic gradient descent and data augmentation, using cropped and mirrored image patches. The architecture accepts images whose dimensions are multiples of 32 pixels and restores the original output size after padded testing.

  • Training uses stochastic gradient descent with random initialization, momentum 0.9, hand-tuned layer learning rates, and batches of 32.Networks train for 8000 to 50000 mini-batch iterations depending on convergence speed and dataset.
  • Images are randomly cropped to 416 × 416 pixels and mirrored horizontally during training.
  • The scale 1 architecture accepts images whose width and height are multiples of 32 pixels.At test time, images are padded to meet this requirement, then outputs are cropped back to the original input size.

3. Empirical Evaluation

The evaluation benchmarks direct intrinsics on synthetic Sintel and real MIT images using standard intrinsic-decomposition metrics and cross-validation splits. RGB-only predictions outperform competing RGB+D methods on Sintel, adapt reasonably to MIT with learned deconvolution and resynthesized training data, but perform less satisfactorily on IIW.

  • Sintel evaluation: The Sintel evaluation uses 890 images from 18 scenes with two-fold scene-split cross-validation, keeping entire scenes in either training or testing.
  • Evaluation metrics: MSE, LMSE, and DSSIM are reported, with lower values indicating better performance; MSE and LMSE use scale-invariant error measures.LMSE averages scale-invariant MSE over overlapping windows spanning 10% of the image’s larger dimension, while DSSIM derives from structural similarity.
  • Sintel results: On Chen and Koltun’s test set, MSCR+dropout+GL significantly outperforms competing methods on MSE and LMSE.
  • Sintel results: 0.0145 lower shading DSSIM gives the RGB-only model an overall DSSIM advantage over Chen and Koltun’s depth-using method, despite 0.0054 higher albedo DSSIM.The RGB-only model is not trained directly to optimize DSSIM, whereas Chen and Koltun’s method uses depth and is trained on DSSIM.
  • Generalization and variants: Scene-split errors are generally larger than image-split errors, while dropout, gradient loss, learned deconvolution, and data augmentation each improve performance.The scene split is more indicative of out-of-sample generalization because test scenes are unseen during training; joint Sintel–MIT training gives a small Sintel improvement.
  • MIT results: On MIT, the system is competitive with Barron et al. on MSE but lags behind on LMSE, while learned deconvolution and resynthesized Sintel training improve results.Naively adding Sintel data hurts performance, whereas resynthesized Sintel data improves albedo prediction but can leak a Sintel-specific bluish shading tint.
  • IIW results: On IIW, testing without IIW-specific training produces less satisfactory performance, with WHDR=27.2, compared with prior and state-of-the-art methods trained specifically for the task.The authors speculate that human reflectance judgments and physically correct decompositions may represent discrepant tasks, with additional domain shift between datasets.

4. Conclusion

Direct intrinsics learns albedo–shading decompositions directly from training data with a multiscale CNN, avoiding physics-based constraints and graph-based inference. The approach is presented as flexible across synthetic and real images.

  • Direct intrinsics learns dual associations between an image and its albedo-plus-shading components directly from training data.
  • The method uses a two-level feed-forward CNN in which coarse predictions provide global context for finer-resolution outputs.
  • Direct intrinsics outperforms state-of-the-art methods using more complex priors, graph-based inference, and additional scene-depth input.
  • The data-driven approach requires training data rather than hand-designed features or representations and can adapt to complex illumination, albedo, shape, and lighting patterns.
  • The model works with synthetic and real images, with further improvement on real images when training is augmented with synthetic examples.
Loading 1512.02311v1…