PaperlayerView exact revision on arXiv

Source-linked AI summary

Inverse Rendering for Complex Indoor Scenes: Shape, Spatially-Varying Lighting and SVBRDF from a Single Image

Zhengqin Li, Mohammad Shafiei, Ravi Ramamoorthi, Kalyan Sunkavalli, Manmohan Chandraker

arXiv:1905.02722v1cs.CV

TL;DR

The paper addresses holistic reconstruction of shape, reflectance, and lighting from a single indoor RGB image, an ill-posed problem complicated by interacting scene factors. It combines realistic synthetic data with a physically motivated cascaded inverse-rendering network and differentiable rendering. The framework achieves competitive results across constituent tasks and supports photorealistic object insertion and material editing, although geometric reconstruction is not competitive with a specialized method.

  • Problem

    Single-image reconstruction of indoor scene shape, lighting, and material properties is ill-posed because multiple factor combinations can produce the same image.

  • Method

    The framework augments SUNCG with photorealistic SVBRDF materials, then uses a cascaded network with SVSG lighting, differentiable rendering, and bilateral refinement to jointly predict scene factors.

  • Results

    The method achieves competitive results against methods focused on constituent sub-problems and enables photorealistic object insertion and material editing on real images.

  • Takeaways & Limitations

    Holistic physically based decomposition supports augmented-reality and interior-design applications involving global illumination effects.

  • Takeaways & Limitations

    The method does not achieve competitive performance on geometric reconstruction, which the authors identify as outside their main focus.

Abstract

from arXiv · show

We propose a deep inverse rendering framework for indoor scenes. From a single RGB image of an arbitrary indoor scene, we create a complete scene reconstruction, estimating shape, spatially-varying lighting, and spatially-varying, non-Lambertian surface reflectance. To train this network, we augment the SUNCG indoor scene dataset with real-world materials and render them with a fast, high-quality, physically-based GPU renderer to create a large-scale, photorealistic indoor dataset. Our inverse rendering network incorporates physical insights -- including a spatially-varying spherical Gaussian lighting representation, a differentiable rendering layer to model scene appearance, a cascade structure to iteratively refine the predictions and a bilateral solver for refinement -- allowing us to jointly reason about shape, lighting, and reflectance. Experiments show that our framework outperforms previous methods for estimating individual scene components, which also enables various novel applications for augmented reality, such as photorealistic object insertion and material editing. Code and data will be made publicly available.

1. Introduction

The paper targets holistic inverse rendering of arbitrary indoor scenes from one RGB image, jointly recovering geometry, reflectance, and spatially varying lighting despite severe ambiguity and complex light transport. It combines realistic synthetic training data with a physically motivated network to support photorealistic editing applications.

  • Single-image indoor inverse rendering is ill-posed because geometry, lighting, and materials interact, while prior methods often estimate subsets or restricted scene settings.
  • The goal is to reconstruct geometry, spatially varying surface reflectance, and spatially varying lighting from an arbitrary indoor RGB image captured under uncontrolled conditions.
  • The model uses a deep convolutional network to regress scene parameters jointly, incorporating learned priors and interactions among scene factors.
  • The training dataset replaces unrealistic SUNCG materials with photorealistic SVBRDFs and renders them using a GPU-based physically based renderer.
  • Spatially varying spherical Gaussians encode local lighting and light-transport effects, enabling a fast differentiable rendering layer for predicted geometry, SVBRDFs, and lighting.
  • Holistic reconstruction supports photorealistic object insertion and material editing, including realistic shading, occlusion shadows, reflections, and spatially varying specular highlights.

2. Related Work

Prior inverse-rendering work commonly addressed single objects, individual scene factors, or simplified lighting and geometry settings. This paper improves the SUNCG-based training data and introduces a specialized differentiable renderer for complex indoor scenes.

  • Single objects: Single-object methods often assume known lighting or geometry, or target specific classes such as faces, making arbitrary-scene reconstruction difficult.
  • Large-scale scenes: Indoor-scene methods have separately estimated materials, reflectance and shading, geometry, or lighting rather than jointly modeling scene appearance.
  • Large-scale scenes: Some full-scene approaches require RGBD input or extensive user annotations, whereas the proposed setting uses a single image.
  • Datasets: Existing datasets contain missing or unrealistic material and lighting specifications, motivating a higher-quality SUNCG-based dataset.
  • Differentiable rendering: Prior differentiable rendering layers were customized for simple settings, while general-purpose global-illumination renderers were too expensive for neural-network training.

3. Dataset for Complex Indoor Scenes

The dataset augments SUNCG indoor scenes with photorealistic microfacet materials, spatially varying lighting, and physically based GPU rendering. Its mapping and rendering pipeline addresses material realism, semantic assignment, global illumination, and scalable data generation.

  • Material mapping: The dataset replaces unrealistic SUNCG materials with high-quality, photorealistic SVBRDFs while preserving scene semantics through material mapping.The materials use a physically motivated microfacet BRDF and high-resolution 4096 × 4096 SVBRDF textures.
  • Material mapping: Tileable texture synthesis preserves sharp material structures such as brick edges and wood-floor patterns when replacing SUNCG textures.The method selects crops by minimizing boundary gradients and finds seams with a graph-cut procedure.
  • Material mapping: Spatially varying and homogeneous SUNCG materials are mapped to the new materials using diffuse appearance, semantic categories, and sampled microfacet specular parameters.Homogeneous mappings retain diffuse albedo and convert Phong specular parameters through a learned conditional distribution.
  • Material mapping: Images rendered with the mapped materials show more realistic specular highlights than SUNCG Lambertian and Phong renderings, with materials tiled and assigned to correct objects.Lambertian renderings lack specularity, while SUNCG Phong renderings produce strong but flat highlights.
  • Spatially Varying Lighting: Per-pixel 16 × 32 environment maps encode spatially varying lighting, with global illumination and occlusion baked into the lighting representation.Integrating the lighting and BRDF produces realistic images with correctly rendered high-frequency specular highlights.
  • Rendering and dataset scale: The GPU renderer produces 480 × 640 images with 16,384 samples per pixel in 3–6 minutes, while Mitsuba requires about 1 hour on 16 CPU cores.The rendered dataset contains 78,794 HDR images, including 72,220 training and 6,574 testing images.

4. Network Design

The network jointly predicts material, geometry, and spatially varying lighting from a single image through cascaded refinement and physically based rendering. Spherical Gaussians provide a compact lighting representation that preserves high-frequency effects, while scale-aware losses and bilateral refinement improve prediction stability and smoothness.

  • Architecture: The architecture cascades material-and-geometry prediction with spatially varying lighting, differentiable rendering, and bilateral-solver refinement.Predictions and rendered images are passed to later stages for iterative refinement.
  • Lighting representation: Spherical Gaussian lighting uses 72 parameters versus 75 for fourth-order spherical harmonics while performing better, especially near specular regions.Each spherical Gaussian lobe is represented by six parameters, and the comparison includes lighting-approximation and rendered-image errors.
  • Lighting representation: The lighting predictor receives the image, segmentation mask, and predicted material and geometry, then estimates spherical Gaussian directions, bandwidths, and intensities.Low-dynamic-range parameters are transformed into high-dynamic-range lighting; HDR output supports relighting and material editing.
  • Differentiable rendering: The differentiable rendering layer numerically integrates SVBRDF and spatially varying lighting over sampled hemisphere directions to model image formation.The model samples 16 × 8 lighting directions and finds this sufficient to recover most high-frequency lighting effects.
  • Training objectives: Scale-invariant losses address color–intensity and single-image depth-scale ambiguities, while supervising environment maps and spherical Gaussian parameters preserves high-frequency details.The objective also includes a scale-invariant rendering loss based on diffuse and specular rendered components.
  • Refinement: A trainable bilateral solver uses predictions, diffuse albedo guidance, and confidence maps to impose a smoothness prior during refinement.The solver refines albedo, roughness, and depth predictions.

5. Experiments

Experiments evaluate the framework’s component estimates, design choices, and applications. The method improves shape, BRDF, lighting, and intrinsic decomposition while supporting photorealistic scene editing and object insertion.

  • Overall evaluation: The framework achieves high accuracy on synthetic data and competitive performance on real images across shape, material, and lighting estimation.The experiments also study pipeline components and demonstrate object insertion and material editing.
  • Network and training choices: Cascade estimation produces clear gains for shape, BRDF, and lighting by iteratively improving and upsampling predictions.It removes noise while preserving high-frequency details in both materials and lighting, across real and synthetic data.
  • Network and training choices: Jointly supplying predicted BRDF and geometry improves recovery of spatially varying lighting, while direct SVSG supervision produces sharper predictions.Without predicted material and geometry, lighting—especially ambient color—does not adapt sufficiently to the scene.
  • Network and training choices: The bilateral solver yields modest quantitative gains after the second cascade stage but noticeable qualitative improvements on real images.Because the network already learns smoothness priors, the solver is used for all real experiments primarily for visible refinement.
  • Comparisons with previous works: Intrinsic decomposition benefits from the cascade structure and achieves lower error than prior work on IIW, supporting the value of the photorealistic training data and network design.The comparison includes models trained on the proposed data alone and fine-tuned on IIW.
  • Comparisons with previous works: 0.87, 0.86, and 0.83 scale-invariant shading errors on RGB channels outperform Barron et al.’s 2.33, 2.10, and 1.90.The predicted lighting captures complex variations and high-frequency components that the comparison method cannot represent physically.
  • Comparisons with previous works: 2.43 log L2 error outperforms Gardner et al.’s 3.34 on lighting estimation.Gardner et al. predict one environment light for the whole scene, so their results cannot show spatially varying illumination effects.
  • Novel applications: The decomposition enables photorealistic object insertion and material editing that preserve or reproduce global illumination, reflections, and specular highlights.Material replacement and roughness editing are rendered using predicted geometry, diffuse albedo, and spatially varying lighting.

6. Conclusion

The paper presents a holistic single-image inverse-rendering framework for indoor scenes, combining realistic data, physical representations, and network design choices. It reports competitive constituent-task performance and demonstrates augmented-reality and interior-design applications.

  • 6. Conclusion: The framework estimates disentangled shape, SVBRDF, and spatially varying lighting from a single indoor image.It addresses the joint inverse-rendering problem using insights from computer vision, graphics, and deep convolutional networks.
  • 6. Conclusion: A GPU-accelerated renderer creates a large-scale realistic dataset containing complex materials and global illumination.The dataset supports training for the holistic reconstruction task.
  • 6. Conclusion: Per-pixel SVSG lighting captures high-frequency effects, while differentiable rendering, cascades, and bilateral refinement provide further benefits.These representation and design choices are presented as important to generalization and prediction quality.
  • 6. Conclusion: Despite solving the joint problem, the method achieves competitive results against prior work focused on constituent sub-problems.The conclusion attributes this outcome to the dataset, representation choices, and network design.
  • 6. Conclusion: Object insertion and material editing on real images demonstrate applications that preserve or reproduce global illumination effects.The paper motivates these capabilities for augmented reality and interior design.

B. Generalization to Outdoor Scenes

The model, trained only on synthetic indoor scenes, is tested qualitatively on outdoor scenes. Its geometry, lighting, and diffuse-albedo estimates appear reasonable, and the predictions are also used for object insertion.

  • B. Generalization to Outdoor Scenes: Synthetic-only training generalizes qualitatively to outdoor scenes, although the network interprets the outdoor environment as a room surrounded by walls.The overall geometry, lighting, and diffuse-albedo estimates look reasonable in the reported examples.
  • B. Generalization to Outdoor Scenes: The outdoor predictions are used to insert a new object following the paper’s scene-editing pipeline.The passage describes this as an additional qualitative test of the predictions.

C. A Failure Case

The method generally produces successful object insertions, but ambiguity between albedo and lighting causes occasional failures.

  • Some object-insertion failures occur despite largely successful results across experiments.
  • The ambiguity between albedo and lighting remains difficult to disentangle in the reconstruction.
  • Over-bright albedo estimates can be offset by correspondingly lower-intensity lighting estimates, producing darker-than-expected inserted objects.

D. BRDF Model and Material Categories

The paper uses a physically motivated microfacet BRDF with spatially varying diffuse albedo, normals, and roughness, and organizes materials into ten semantic categories for SUNCG mapping.

  • BRDF Model: The microfacet BRDF represents appearance as the sum of diffuse and specular components.The model is written as f(l, v; A, N, R) = fd(l, v; A, N) + fs(l, v; N, R).
  • BRDF Model: Diffuse albedo, surface normal, and roughness are modeled as spatially varying quantities.The paper denotes these quantities by A, N, and R, respectively.
  • BRDF Model: The specular model uses distribution, Fresnel, and geometric terms, with view and lighting directions explicitly defined.The prose identifies v and l as view and lighting directions and D, F, and G as the corresponding microfacet terms.
  • BRDF Model: The dataset fixes the normal-incidence reflectance parameter at F0 = 0.05.This value follows the cited recommendation.
  • Material Categories: The authors manually classify materials from their dataset and SUNCG into ten categories to map them consistently onto scene geometry.The categories include fabric, ground, leather, stone floor, plastic, stone specular, stone wall, wall paint, wood floor, and wood.
  • Applications: The method generalizes reasonably well to outdoor scenes despite training only on synthetic indoor scenes, supporting reasonable object insertion compared with a state-of-the-art method.The comparison is shown against [24].
  • Applications: A failure case shows darker-than-expected inserted objects caused by an over-bright albedo estimate offset by lower-intensity lighting.The figure identifies this as a trade-off between albedo and lighting estimates.

E. Tileable Texture Synthesis

The paper synthesizes tileable textures from cropped SVBRDF patches using graph cuts, boundary-aware energies, and constraints that preserve seamlessness in both directions.

  • Patch Selection: The synthesis pipeline crops SVBRDF patches and searches globally for patches with low gradients perpendicular to their boundaries.The objective is computed efficiently with an integral graph, giving O(K) complexity for K image pixels.
  • Results: The results figure shows textures formed by tiling 3 × 3 patches together.The construction demonstrates the output of the tileable texture synthesis process.
  • Patch Selection: Minimizing boundary gradients reduces strong seams and artifacts in the synthesized tileable textures.The optimization specifically avoids strong gradients near patch boundaries.
  • Seam Optimization: Graph cuts stitch overlapping right and left surrounding regions to make the selected patch tileable in the x direction.The seam is chosen by minimizing a customized energy function across the overlap.
  • Seam Optimization: The final seam energy combines losses from different texture maps with weighting.The paper defines the final loss as a weighted combination of texture-map losses.
  • Two-Direction Tiling: The y-direction process repeats graph-cut seam selection while enforcing consistency at the x boundaries.Infinite smoothness terms keep corresponding boundary pixels in the same region, preserving x-direction tiling.

F. Ground Truth Spherical Gaussian Lobes

The paper supervises spatially varying lighting with twelve ordered spherical Gaussian lobes whose parameters are fitted to environmental lighting and constrained over the hemisphere.

  • Lobe Fitting: Ground-truth spherical Gaussian lobe parameters are computed by approximating environmental lighting with LBFGS.These parameters supervise spatially varying lighting prediction.
  • Lobe Representation: The lighting representation uses 12 lobes per pixel to approximate spatially varying illumination.The lobes are ordered during training by constraining each to a prescribed hemisphere region.
  • Lobe Representation: The hemisphere is roughly divided into 2 × 6 regions to assign ordered lobe locations.The ordering facilitates the training process.
  • Parameterization: Each spherical Gaussian lobe direction is parameterized from elevation and azimuth angles as ξk = (sin θk cos φk, sin θk sin φk, cos θk).The paper defines θk, φk, λk, and Fk as the spherical Gaussian parameters.
  • Parameterization: The parameters are reparameterized to impose the lobe constraints during prediction.The constrained variables are denoted by hatted versions of the angular, sharpness, and intensity parameters.
  • Comparison: Spherical Gaussians recover high-frequency lighting effects better than spherical harmonics with a similar number of parameters.The comparison reports reconstructed spatially varying lighting closer to ground truth for spherical Gaussians.

G. Network Structures and Training Details

The network uses cascade-level training and differentiable rendering losses, with architecture choices that support arbitrary image sizes. Real-data fine-tuning uses mixed synthetic and real batches, dataset-specific losses, and augmentation.

  • Network Structures: Group normalization and dynamically assigned padding let the network train with smaller batches and process arbitrarily sized images without scaling or cropping.Dynamic padding aligns up-sampled feature maps with skip-link features.
  • Training Strategy: Each cascade level trains MGNet_i and LightNet_i separately before jointly fine-tuning them with Adam.The full pipeline and BSNet are trained with their respective defined loss functions.
  • Loss Functions: Rendering loss L_ren is added when training LightNet_i, while the complete pipeline is fine-tuned using a separate loss function.The rendering loss is introduced specifically for lighting-network training.
  • Optimization: Learning rates are halved every 10 epochs, with remaining hyperparameters documented in Tables 7 and 8.The passage also identifies initial learning rate, training epochs, and coefficient settings as tabulated hyperparameters.
  • Real-Dataset Fine-Tuning: Real-data fine-tuning proceeds sequentially across cascade levels using mixed synthetic and real batches, IIW ordinal reflectance loss, and NYU-specific geometric augmentation.For NYU, images are randomly flipped, cropped, and scaled by a factor uniformly sampled from 0.8 to 1.2.
Loading 1905.02722v1…