PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Non-Lambertian Object Intrinsics across ShapeNet Categories

Jian Shi, Yue Dong, Hao Su, Stella X. Yu

arXiv:1612.08510v1cs.CV

TL;DR

The paper tackles single-image decomposition of non-Lambertian objects into diffuse albedo, shading, and specular highlights, where existing methods and datasets are inadequate. It constructs a ShapeNet rendering dataset and trains an encoder-decoder CNN, which consistently outperforms the state of the art, generalizes across categories, and produces robust real-image results.

  • Problem

    Single-image intrinsic decomposition must recover albedo, shading, and specular effects, but existing methods assume Lambertian reflection and available datasets lack suitable non-Lambertian ground truth.

  • Method

    The paper uses physically based ShapeNet rendering to create supervised non-Lambertian data and trains an encoder-decoder CNN to predict albedo, shading, and specular components.

  • Results

    The model consistently outperforms the state of the art across error metrics and visual comparisons, with a reported 40-50% gain on DSSIM on the synthetic dataset.

  • Takeaways & Limitations

    The approach supports cross-category generalization and robust application to real images and videos, including image-based albedo and specular editing.

  • Takeaways & Limitations

    The rendering pipeline does not assume a specific BRDF model, although simpler BRDFs trade photo-realism for rendering efficiency while complex models provide higher photo-realism.

Abstract

from arXiv · show

We consider the non-Lambertian object intrinsic problem of recovering diffuse albedo, shading, and specular highlights from a single image of an object. We build a large-scale object intrinsics database based on existing 3D models in the ShapeNet database. Rendered with realistic environment maps, millions of synthetic images of objects and their corresponding albedo, shading, and specular ground-truth images are used to train an encoder-decoder CNN. Once trained, the network can decompose an image into the product of albedo and shading components, along with an additive specular component. Our CNN delivers accurate and sharp results in this classical inverse problem of computer vision, sharp details attributed to skip layer connections at corresponding resolutions from the encoder to the decoder. Benchmarked on our ShapeNet and MIT intrinsics datasets, our model consistently outperforms the state-of-the-art by a large margin. We train and test our CNN on different object categories. Perhaps surprising especially from the CNN classification perspective, our intrinsics CNN generalizes very well across categories. Our analysis shows that feature learning at the encoder stage is more crucial for developing a universal representation across categories. We apply our synthetic data trained model to images and videos downloaded from the internet, and observe robust and realistic intrinsics results. Quality non-Lambertian intrinsics could open up many interesting applications such as image-based albedo and specular editing.

1. Introduction

The paper addresses single-image non-Lambertian intrinsic decomposition by recovering albedo, shading, and specular highlights, overcoming Lambertian assumptions and limited ground-truth data. It introduces a ShapeNet-based dataset and encoder-decoder CNN that produces sharper results, generalizes across categories, and performs robustly on real images and videos.

  • Motivation: Existing intrinsic algorithms assume Lambertian or diffuse reflection, causing large decomposition errors when real images contain specular components.The paper identifies specularity as a mismatch between traditional model assumptions and everyday object appearance.
  • Problem formulation: The task decomposes a single object image into albedo, shading, and an additive specular component.The non-Lambertian formulation extends the Lambertian image model I = A × S with specular residue R.
  • Motivation: A major obstacle is the lack of non-Lambertian ground truth, since human annotations are impractical and existing datasets are limited, artificial, or incomplete.The cited datasets provide either few objects, cartoonlike scenes, or sparse albedo-only judgments rather than full decomposition ground truth.
  • Approach: Skip connections between matching encoder and decoder resolutions preserve visual details, addressing coarse and blurry outputs from prior CNN designs.The network is designed to produce sharper and more detailed intrinsic components.
  • Approach: The authors build a ShapeNet-based rendering dataset and train an encoder-decoder CNN to learn image-to-intrinsics associations simultaneously.Millions of synthetic images with specular materials and environment maps provide training supervision.
  • Results: The model consistently outperforms prior methods, generalizes across object categories, and delivers solid results on real images and videos.The reported cross-category behavior includes models trained on chairs obtaining reasonable performance on categories such as cars.

2. Related Work

Prior intrinsic-image methods largely target Lambertian or constrained reflectance settings, while this work formulates single-image decomposition for general non-Lambertian objects using physics-based rendering and separate diffuse and specular components.

  • Intrinsic Image Decomposition: Existing intrinsic-image methods address an ill-posed decomposition of one image into reflectance and shading using priors or additional inputs.Related approaches use reflectance sparsity, texture, shape, illumination, image sequences, depth, or user strokes.
  • Deep Learning: Deep-learning intrinsics methods have used sparse human judgments or synthetic data, but Direct Intrinsics produces blurry results and relies on artificial scenes.The paper attributes blur to downsampling and deconvolutions in the adapted depth-regression architecture.
  • Reflectance Estimation: Unlike methods constrained to stochastic textures, known lighting, homogeneous objects, or distant light sources, this work targets spatially varying albedo under general lighting conditions.The stated goal is general intrinsic decomposition from a single image without constraints on material or lighting distributions.
  • Intrinsic Image with Specular Reflectance: The rendering equation models observed radiance as incident lighting integrated with surface reflectance, whose BRDF contains diffuse and specular terms.The formulation uses viewing direction, lighting direction, and surface normals in the rendering process.
  • Intrinsic Image with Specular Reflectance: Because diffuse and specular reflectance differ spatially, the proposed image model decomposes I into diffuse albedo A, diffuse shading S, and residual specular reflectance R: I = A × S + R.The residual representation avoids separately estimating high-frequency specular albedo and shading.
  • Intrinsic Image with Specular Reflectance: The model is based on physical diffuse and specular properties without assuming a specific BRDF, allowing simple or complex BRDFs for different rendering requirements.Phong can support efficiency, while Cook-Torrance can support higher photo-realism.

4. Learning Intrinsics

The learning system combines a shared encoder, separate intrinsic-component decoders, resolution-matched mirror links, and rendering-based synthetic supervision. Training data uses ShapeNet objects under varied realistic environments and specular settings, with object-level train/test separation.

  • Mirror-Link CNN: The mirror-link CNN uses one encoder and separate decoders for albedo, shading, and specular components.Early encoder features are forwarded to decoder layers at matching spatial resolutions to preserve local sharp details.
  • Mirror-Link CNN: The architecture differs from related skip-connection systems because it performs image-wise intrinsic regression rather than geometry interpolation or image segmentation.The comparison specifically distinguishes the proposed goal from DRM and UNet.
  • Losses: Gradient-weighted pixel errors form an edge-sensitive loss intended to improve precision and sharpness around image edges.The weighting addresses the equal treatment of pixel errors in standard MSE.
  • Losses: Scale ambiguity between albedo and shading is handled by combining MSE and scale-invariant MSE with edge-sensitive weighting.Only the product of albedo and shading is directly constrained by the decomposition.
  • ShapeNet-Intrinsics Dataset: The dataset contains 31,072 ShapeNet models from common object categories, rendered with 98 indoor and outdoor HDR environment maps.The environments provide varied illumination and surrounding information relevant to specular effects.
  • Rendering: Objects are rendered with random upper-hemisphere viewpoints and modified Phong reflectance, using ks ∈ (0, 0.3) and Ns ∈ (0, 300) to span diffuse-to-high-specular appearances.The renderer produces albedo, shading, and specular layers for synthetic supervision.
  • Training: Object-level splitting uses 24,932 models for training and 6,240 for testing, with 2,443,336 training images and one test image per testing model.The split prevents images of the same object from appearing in both sets.

5. Evaluation

The model is evaluated against established intrinsic-decomposition methods on synthetic, MIT, and real-world images, consistently producing accurate, detailed non-Lambertian components.

  • Evaluation: The method is compared with SIRFS, IIW, DI, and DI* using MSE, LMSE, and DSSIM metrics.DI* denotes Direct Intrinsics trained on the ShapeNet intrinsics dataset.
  • Synthetic dataset: 40–50% DSSIM performance gains are reported over off-the-shelf solutions on the synthetic testing set.DI* achieves the second-best results across almost all error metrics, highlighting the dataset’s value.
  • Synthetic dataset: Visual comparisons show that the model removes specularity from albedo and shading while preserving sharper boundaries and extracting large car highlights.Earlier methods produce artifacts when specular regions are assigned to albedo or shading.
  • MIT dataset: On MIT, the fine-tuned model achieves results comparable to SIRFS and better than fine-tuned DI while preserving more details.The MIT lighting setup differs from the synthetic color-environment-light model, creating domain shift.
  • Real-world images: On real images, the synthetic-data-trained model produces better results than other algorithms and handles previously unseen objects and specular highlights.The authors attribute this to realistic rendering and task generalization.

6. Cross-category generalization

Cross-category experiments show strong transfer across object classes, while category-specific training remains best on its matching category. Encoder representations are more important than decoder fine-tuning for this generalization.

  • Cross-category results: Category-specific training gives the best decomposition on nearly every matching category, while all-category training increases DSSIM error by less than 0.02.The cross-category evaluation covers cars, chairs, airplanes, and sofas.
  • Cross-category results: Models tested on unseen categories still produce reasonable results, with DSSIM on-par with or better than existing general intrinsic methods.The result supports cross-category generalization beyond category-specific training.
  • Cross-category results: Similar categories such as chairs and sofas transfer well because they share textures, albedo features, and shapes.These shared properties support transfer across all three output channels.
  • Cross-category results: Car-trained models generalize better to airplanes than airplane-trained models generalize to cars, which the authors relate to cars’ richer shape and texture variation.The asymmetry is also observed in benchmarks using all categories.
  • Encoder-decoder analysis: Decoder fine-tuning yields limited improvement when the encoder was trained on another category, indicating that encoder features are crucial for decomposition.The all-category model’s similar errors suggest that its encoder captures category-dependent and category-independent features.

7. Application

Intrinsic layers enable appearance editing that is difficult to perform directly on the original image while preserving natural reconstructed scene appearance.

  • Appearance editing: Recoloring diffuse albedo changes a car’s paint while preserving shading and specular highlights.Blurring and rescaling specular components can also change the car from highly specular to matte without visible artifacts.

8. Conclusion

The paper introduces a non-Lambertian intrinsic-decomposition framework built from realistic ShapeNet renderings and reports strong numerical, visual, and cross-category performance.

  • Conclusion: The method adds an explicit specular term to intrinsic decomposition and learns albedo, shading, and specular components with deep learning.Training data is generated through physically based rendering on ShapeNet.
  • Conclusion: The model consistently outperforms the state of the art visually and numerically while generalizing across object categories.The conclusion also identifies albedo and specular editing as supported applications.

1. Network

The Mirror-Link CNN uses a symmetric encoder-decoder with shared encoding, separate intrinsic-component decoders, and skip and cross links to preserve detail and exploit component correlations.

  • Architecture: The Mirror-Link CNN links same-resolution encoder and decoder features to preserve local details and produce sharper intrinsic outputs.Its encoder and decoder have mirrored feature-map sizes, with three-channel RGB input and intrinsic-component outputs.
  • Architecture: The network uses a shared encoder and separate decoders for albedo, shading, and specular components.Cross-decoder links further strengthen correlations among the three outputs.
  • Architecture: Compared with U-Net and Deep Reflectance Map, the design targets a different regression task, enforces symmetric feature maps, and has three decoders.The comparison identifies architectural differences rather than claiming exclusive use of skip links.
  • Ablations: A shared encoder achieves comparable albedo and shading performance and better specular performance than independent networks despite fewer parameters.The authors therefore regard one shared encoder as sufficient for all three outputs.
  • Ablations: Cross links between decoders significantly improve performance by strengthening correlations among the three intrinsic components.The network variations include removing cross links and changing the number of encoder-decoder skip links.
  • Ablations: Removing all skip links or three outer links tests their contribution, and the experiments observe improved performance from skip links.Skip Link-0 removes all links, whereas Skip Link-3 retains three middle links.

2. Loss Functions

The training objective combines scale-invariant and standard MSE losses to address albedo-shading scale ambiguity, while specular prediction uses MSE under the additive model.

  • Loss design: Scale-invariant MSE rescales predicted outputs before comparing them with ground truth, and it is combined with standard MSE.This combination follows the loss design used for intrinsic networks.
  • Loss design: The formulation I = A × S + R retains scale ambiguity between albedo and shading, so scale-invariant loss is applied to those components.The specular component is handled separately because scale- or shift-invariant treatments affect the inferred A × S pattern differently.
  • Loss design: Specular output uses standard MSE rather than an invariant loss, and object masks restrict loss computation and gradient back-propagation to foreground pixels.Ground truth is available only for objects.

3. Rendering Pipeline

The rendering pipeline creates intrinsic ground truth from ShapeNet objects using Mitsuba, realistic environment maps, randomized viewpoints, and compositing of rendered components.

  • Dataset rendering: The dataset renders 31,072 ShapeNet models with Mitsuba, extracting albedo, shading, and specular ground truth through material changes.Albedo is rendered directly; shading uses pure diffuse white, while specular rendering removes diffuse reflectance.
  • Dataset rendering: Models are rendered under 98 environment maps with randomly assigned upper-hemisphere viewpoints generated by a low-discrepancy Halton sequence.The sequence keeps viewpoints approximately uniform while randomizing their distribution.
  • Compositing: Path tracing renders the intrinsic components and object mask, then synthesizes images with I = A × S + R.Rendering components separately saves 30% of rendering time because albedo and mask rendering require fewer samples.
  • Evaluation: Figure 2 compares real-image results from the proposed method, SIRFS, Direct Intrinsics, and Direct Intrinsics trained on the authors’ dataset.The figure separates input/specular visualization from albedo and shading comparisons.

5. Another synthetic evaluation

Additional synthetic evaluations test generalization to peppers and a kettle with specular effects, while video experiments assess framewise stability without temporal constraints.

  • Scene context: Figure 3 presents a photo-realistic rendered kitchen scene, extending evaluation beyond isolated object renders.The scene contains nearby objects that can produce interreflections.
  • 5.1. Pepper: Rendered peppers contain sharp specular highlights and interreflections, yet the method produces acceptable results despite no similar object in training.The evaluation reports especially acceptable shading and specular estimates.
  • 5.2. Kettle: The rendered kettle has broader, less sharp specular reflection, and the method produces reasonable specular and shading estimates.Its albedo prediction is better than other methods but fails to fill the correct hole colors in the specular region.
  • Video evaluation: The model produces stable and reasonable results when applied independently to each video frame without temporal consistency constraints.This experiment evaluates framewise application rather than a temporally coupled model.
Loading 1612.08510v1…