PaperlayerView exact revision on arXiv

Source-linked AI summary

CNN based Learning using Reflection and Retinex Models for Intrinsic Image Decomposition

Anil S. Baslamisli, Hoang-An Le, Theo Gevers

arXiv:1712.01056v2cs.CV

TL;DR

Intrinsic-image decomposition has a gap between traditional methods based on scene priors and deep models that do not use image formation to guide learning. This paper combines reflection-model constraints, CNNs, and Retinex-inspired gradient learning, and reports that its models outperform existing methods across synthetic, real-world, and in-the-wild evaluations.

  • Problem

    Traditional approaches derive scene priors, while deep-learning methods often treat decomposition as a black box without using the established image-formation process to guide learning.

    Source: §1 · p. 1

  • Method

    The paper proposes CNN models guided by a physics-based reflection model and Retinex-inspired learning of reflectance and shading gradients, alongside a large-scale synthetic dataset.

    Source: §1 · p. 2

  • Results

    The evaluation across synthetic, real-world, and in-the-wild images demonstrates that the proposed model outperforms existing methods.

    Source: §6 · p. 8

  • Takeaways & Limitations

    Visual inspection shows that image-formation loss augments color reproduction and gradient information produces sharper edges.

    Source: §6 · p. 8

    The image-formation model assumes body (diffuse) reflection when approximating the observed image as the product of reflectance and shading.

    Source: §3.1 · p. 3

Abstract

from arXiv · show

Most of the traditional work on intrinsic image decomposition rely on deriving priors about scene characteristics. On the other hand, recent research use deep learning models as in-and-out black box and do not consider the well-established, traditional image formation process as the basis of their intrinsic learning process. As a consequence, although current deep learning approaches show superior performance when considering quantitative benchmark results, traditional approaches are still dominant in achieving high qualitative results. In this paper, the aim is to exploit the best of the two worlds. A method is proposed that (1) is empowered by deep learning capabilities, (2) considers a physics-based reflection model to steer the learning process, and (3) exploits the traditional approach to obtain intrinsic images by exploiting reflectance and shading gradient information. The proposed model is fast to compute and allows for the integration of all intrinsic components. To train the new model, an object centered large-scale datasets with intrinsic ground-truth images are created. The evaluation results demonstrate that the new model outperforms existing methods. Visual inspection shows that the image formation loss function augments color reproduction and the use of gradient information produces sharper edges. Datasets, models and higher resolution images are available at https://ivi.fnwi.uva.nl/cv/retinet.

1. Introduction

Intrinsic decomposition separates reflectance from shading, while traditional priors and black-box CNNs differ in their use of physical image formation and their qualitative and quantitative strengths. This paper combines CNN learning with reflection-model constraints and Retinex gradient information.

1. Introduction

  • Reflectance represents object color, whereas shading captures illumination effects; the two components can support different vision and computational-photography tasks.Shading provides cues for shape-from-shading, while reflectance can benefit segmentation and detection by reducing illumination effects.Source: §1 · p. 1
  • Traditional methods derive scene priors, while deep models often treat decomposition as a black box without using the image-formation process to guide learning.The paper frames its approach as combining deep learning with physics-based reflection and traditional gradient information.Source: §1 · pp. 1–2
  • IntrinsicNet uses a physics-based image-formation loss, while RetiNet learns reflectance and shading gradients before decomposing the full image pixel by pixel.The image-formation loss accounts for the reconstruction from predicted reflectance and shading.Source: §1 · p. 2
  • The synthetic dataset contains around 20,000 man-made-object images rendered across environment maps and viewpoints for indoor and outdoor scenes.The paper presents it as a response to the limited availability of large-scale intrinsic ground-truth data.Source: §1 · p. 2
  • The proposed contributions include gradient (re)integration using simple convolutions instead of more complex computations such as Poisson methods.The paper also introduces an object-centered synthetic dataset with intrinsic ground truth.Source: §1 · p. 2

2. Related Work

Prior work constrains an ill-posed decomposition with scene priors or uses supervised learning, whose dense intrinsic-image training data are difficult to obtain. The paper identifies physics-guided learning and Retinex-driven gradient separation as gaps in prior deep-learning methods.

2. Related Work

  • Retinex assumes piece-wise constant reflectance and smoothly varying shading, assigning larger image derivatives to reflectance changes and smaller ones to illumination changes.Color extensions can exploit chromaticity information described as invariant to shading cues.Source: §2 · p. 2
  • Traditional methods constrain decomposition with texture, reflectance sparsity, user input, depth, or multiple images, but their applicability is limited by reliance on priors.Multiple-image approaches treat reflectance as constant and illumination as changing.Source: §2 · p. 2
  • Dense intrinsic-image ground truth is difficult to collect in real settings, so supervised learning relies on synthetic datasets, including scene-level and object datasets.The cited scene-level dataset has around a thousand images, while another cited non-Lambertian object dataset has around 3 million images and is not publicly available.Source: §2 · p. 2
  • Crowdsourced indoor-scene annotations provide relative reflectance comparisons rather than ground-truth intrinsic images, while supporting data-driven learning of priors and relationships.These annotations are collected over point pairs in real-world indoor scenes.Source: §2 · p. 3
  • Prior deep-learning methods do not combine consistent image-formation constraints during training with Retinex-driven gradient separation.The paper positions its approach as combining supervised learning with reflection and Retinex models.Source: §2 · p. 3

3. Approach

The approach uses a reflection model to constrain CNN-based intrinsic decomposition, then adds a Retinex-inspired network that learns from image gradients. Its components cover image formation, reflectance and shading prediction, and gradient-based decomposition.

3.1. Image Formation Model

  • The dichromatic reflection model represents surface reflection as diffuse body and specular interface components, while the pixel value is expressed over the visible spectrum.Under body reflection, the observed image can be approximated as the element-wise product of reflectance and shading.Source: §3.1 · p. 3

3.2. IntrinsicNet: CNN driven by Reflection Models

  • IntrinsicNet uses a shared encoder for shading and reflectance cues, separate decoders for their predictions, and an image-formation loss that evaluates the reconstructed image from predicted intrinsics.The loss supplements reconstruction losses for reflectance and shading.Source: §3.2 · p. 4
  • The image-formation loss is intended to improve color reproduction, while RetiNet combines it with gradient information to target edge sharpness.The paper presents these as the respective aims of the loss and the two-stage gradient-based network.Source: §3.2 · p. 4

3.3. RetiNet

  • RetiNet learns intrinsic gradients directly from data rather than using hard-coded thresholds, then reintegrates them with simple convolutions instead of a pseudo-inverse or Poisson solve.The paper describes RetiNet as a two-stage Retinex-inspired CNN that combines gradient information with image-formation loss.Source: §3.3 · p. 5
  • RetiNet’s first stage receives RGB images with per-channel gradients as a six-channel input and predicts intrinsic gradients; its second stage combines the image with those predictions for pixel-wise decomposition.The second stage uses a fully convolutional sub-network.Source: §3.3 · p. 5, Figure 2 · p. 5
  • Compared with [11], RetiNet uses gradient rather than edge information, learns gradients for multiple components, and uses simple convolutions for reintegration.The compared method uses an encoder-decoder network with deconvolutions for reintegration.Source: §3.3 · p. 5

4. Experiments

The experiments introduce an object-centered synthetic dataset with intrinsic ground truth, describe the evaluation metrics, and specify the network and training setup.

4.1. New Synthetic Dataset of Man-made Objects

  • The synthetic dataset contains around 20,000 object-centered images rendered from ShapeNet models with randomized component colors, roughness, environment maps, and viewpoints.The rendering pipeline produces the image, reflectance, and shading maps in high dynamic range without gamma correction.Source: §4.1 · pp. 5–6

4.2. Error Metrics

  • Evaluation uses brightness-adjusted MSE, LMSE over local regions with k = 20, and DSSIM to assess perceptual visual quality.LMSE aggregates MSE over overlapping k × k regions, with steps of k/2, and is normalized to a maximum possible error of 1.Source: §4.2 · p. 6

4.3. Implementation Details

  • The encoder uses VGG16 without fully connected layers, stride-2 convolutions for downsampling, mirrored deconvolutional decoding, and skip connections for image details.Batch normalization follows convolutional layers except the decoders’ final layers and RetiNet’s inference net.Source: §4.3 · p. 6
  • Experiments use 120 × 160 inputs, batches of 16, SGD with learning rate 1e-5 and momentum 0.9, plus random flips and shifts for augmentation.Figure 4 contrasts IntrinsicNet with and without image-formation loss; its caption says the loss suppresses color artifacts and halo effects.Source: §4.3 · p. 6, Figure 4 · p. 6

5. Evaluation

The proposed methods outperform comparison models on the synthetic and MIT benchmarks, with RetiNet leading by a large margin on ShapeNet. Visual results show improved color reproduction and less blurriness, while strong cast shadows remain difficult to remove from reflectance.

5.1. Image Formation Loss

  • On the MIT benchmark, the image formation loss improves DSSIM and improves average MSE and LMSE results.On ShapeNet, the loss gives similar MSE and LMSE performance and proper albedo prediction on DSSIM; on unseen real-world data, it achieves the best performance across all metrics.Source: §5.1 · pp. 6–7, Table 1 · p. 7

5.2. ShapeNet Dataset

  • RetiNet outperforms all compared models by a large margin on the ShapeNet test set.The same test set contains 7,000 images, with one randomly selected image per test model.Source: Table 2 · p. 7, §5.2 · p. 7

5.3. MIT Intrinsic Dataset

  • On the MIT benchmark, proposed methods outperform ShapeNet and DirectIntrinsics; RetiNet shows more vivid reflectance colors and less artifacting and blurriness than IntrinsicNet.All models struggle to eliminate strong cast shadows from reflectance.Source: §5.3 · p. 7, Figure 6 · p. 8

5.4. Real and In-the-wild Images

  • On real and in-the-wild images, RetiNet produces reflectance images free of geometry-caused shading, and reconstructing the input from albedo and shading predictions shows decomposition consistency.The MIT benchmark comparison reports better results for the proposed methods, and an experiment with intrinsic-gradient ground truth shows the benefit of exploiting those gradients.Source: §5.4 · pp. 7–8, Table 3 · p. 8

6. Conclusion

The paper combines a physics-based reflection model with learned intrinsic gradients in two CNN models. The reported evaluations show better results than existing methods, with visual gains in color reproduction and edge sharpness.

6. Conclusion

  • IntrinsicNet uses a physics-based image-formation loss, while RetiNet learns albedo and shading gradients and reintegrates them using simple convolutions.The RetiNet design is an end-to-end, Retinex-inspired approach based on derivatives.Source: §6 · p. 8
  • The proposed models outperform existing methods, while visual inspection shows improved color reproduction and sharper edges.The authors identify incorporating all intrinsic components into the learning model as future work.Source: §6 · p. 8
  • Training uses a newly created object-centered synthetic dataset with intrinsic ground-truth images.The paper evaluates its models on synthetic, real-world, and in-the-wild images.Source: §6 · p. 8

Related papers

Loading 1712.01056v2…