IRON: Inverse Rendering by Optimizing Neural SDFs and Materials from Photometric Images

Kai Zhang; Fujun Luan; Zhengqi Li; Noah Snavely

Source-linked AI summary

IRON: Inverse Rendering by Optimizing Neural SDFs and Materials from Photometric Images

Kai Zhang, Fujun Luan, Zhengqi Li, Noah Snavely

arXiv:2204.02232v1Apr 5, 2022cs.CV

TL;DR

Inverse rendering seeks to recover shape and appearance from 2D images while producing content usable in standard graphics pipelines. IRON optimizes neural SDFs and materials through a hybrid volumetric and edge-aware surface-rendering scheme, then exports meshes and material textures. The paper reports superior inverse-rendering quality over baseline photometric-image methods and identifies capture, lighting, and material-model boundaries.

Problem
Inverse rendering must recover shape and appearance from photometric images while supporting deployable 3D representations for standard graphics applications.
Method
IRON optimizes neural SDFs and materials first with volumetric radiance rendering and then with edge-aware physics-based surface rendering, while enabling mesh and material-texture export.
Results
IRON reports superior inverse-rendering quality over baseline methods targeting photometric images and improved reconstruction quality around edges using its neural-SDF edge sampling algorithm.
Takeaways & Limitations
The pipeline combines neural optimization flexibility with mesh and material-texture compatibility for downstream graphics renderers and AR/VR applications.
Takeaways & Limitations
IRON requires the more involved capture of photometric images, omits multiple light bounces, and assumes opaque surfaces, limiting performance for interreflective, transparent, or translucent objects.

Abstract

from arXiv · show

We propose a neural inverse rendering pipeline called IRON that operates on photometric images and outputs high-quality 3D content in the format of triangle meshes and material textures readily deployable in existing graphics pipelines. Our method adopts neural representations for geometry as signed distance fields (SDFs) and materials during optimization to enjoy their flexibility and compactness, and features a hybrid optimization scheme for neural SDFs: first, optimize using a volumetric radiance field approach to recover correct topology, then optimize further using edgeaware physics-based surface rendering for geometry refinement and disentanglement of materials and lighting. In the second stage, we also draw inspiration from mesh-based differentiable rendering, and design a novel edge sampling algorithm for neural SDFs to further improve performance. We show that our IRON achieves significantly better inverse rendering quality compared to prior works. Our project page is here: https://kai-46.github.io/IRON-website/

1. Introduction

IRON is a neural inverse-rendering pipeline that optimizes SDF-based geometry and materials from photometric images, then exports meshes and material textures for standard graphics applications. Its hybrid optimization and edge sampling address topology, geometric refinement, material–lighting disentanglement, and occlusion-boundary gradients.

1. Introduction: Existing mesh optimization can struggle with topology changes and regularity, while prior neural material-decoupling methods may support only simple shapes or incompatible neural components.These limitations motivate a representation that combines neural flexibility with deployable mesh and texture outputs.
1. Introduction: IRON combines neural SDFs and materials with mesh and material-texture export, targeting high-quality photometric inverse rendering for downstream graphics applications.The pipeline uses compact neural scene components during optimization while preserving compatibility with traditional renderers and AR/VR applications.
1. Introduction: IRON first uses volumetric radiance-field optimization for topology, then edge-aware physics-based surface rendering to refine geometry and disentangle materials from lighting.This hybrid scheme combines the flexibility of neural representations with the downstream convenience of meshes and material textures.
1. Introduction: IRON reports superior inverse-rendering quality over baseline methods targeting photometric images.The introduction states this comparison as an overall contribution of the pipeline.
1. Introduction: IRON introduces neural-SDF edge sampling that produces unbiased gradient estimates and significantly improves reconstruction quality around occlusion boundaries.The method adapts edge-aware optimization ideas from mesh-based differentiable rendering to neural SDF representations.

2. Related work

Related work spans neural shape and appearance representations, mesh-based differentiable rendering, and inverse rendering under several capture and illumination settings. IRON builds on these directions while using physics-based surface shading for inverse rendering.

Neural reconstruction and view synthesis: Neural representations such as NeRF, DVR, and IDR support view synthesis or 3D reconstruction, but volume-based and surface-based methods exhibit different geometry or scene-complexity trade-offs.UNISURF, VolSDF, and NeuS seek to combine volume- and surface-based rendering near surfaces.
Neural reconstruction and view synthesis: Unlike surface light fields that entangle lighting and materials, IRON uses a physics-based surface shading model within an inverse-rendering framework.This representation is described as part of IRON’s neural shape-and-appearance approach.
Mesh-based differentiable rendering: Mesh-based differentiable rendering jointly optimizes shape, material, and camera parameters but must handle derivatives at depth discontinuities called edge derivatives.Meshes remain widely used in graphics pipelines and game engines.
Inverse rendering from multiple images: Prior inverse-rendering work considers static or rotating scenes under environmental lighting as well as flashlight illumination co-located with a moving camera.The co-located flashlight and moving-camera setting is the photometric-image scenario used by IRON.

3. Method

IRON optimizes neural SDF and material representations from photometric images under collocated flashlight assumptions, using a two-stage scheme that combines volumetric initialization with edge-aware physics-based surface rendering. Its neural components are later connected to standard graphics workflows through mesh and texture export.

Assumptions: IRON assumes opaque objects captured with collocated flashlight illumination, no ambient light, and negligible shadows or global illumination effects.
Neural representations: The system represents shape and materials with four compact MLPs: an SDF, diffuse albedo, specular albedo, and roughness.The SDF also outputs a 256D local geometric feature descriptor used by the material networks.
Two-stage optimization: IRON first optimizes the SDF and diffuse albedo as a volumetric radiance field to recover correct topology, including the number and location of holes.The second stage is initialized from this volumetric solution because directly optimizing materials and geometry for mesh-compatible rendering is inconsistent with the first-stage representation.
Physics-based rendering: The second stage jointly optimizes neural shape and materials using differentiable physics-based shading and edge-aware surface rendering.With collocated light and camera, the rendering equation simplifies because the light direction aligns with the view direction, while a point-light model uses inverse-square falloff and GGX material parameters.
Edge-aware rendering: IRON addresses biased gradients at geometric discontinuities by localizing subpixel edges, reparameterizing edge points for SDF-weight differentiation, and computing edge-pixel shading.The method begins from sphere-traced ray-surface intersections and performs surface walks selectively at depth-discontinuity pixels to reduce SDF evaluations.

4. Evaluation

IRON is evaluated on single-image silhouette fitting and multi-view photometric inverse rendering, with comparisons against neural and mesh-based baselines. The experiments show improved reconstruction, relighting, and edge quality, while ablations validate edge sampling and loss design.

4.2. Inverse rendering from photometric images: IRON matches ground truth better in synthetic specular-highlight regions and appears more perceptually convincing than PSDR under real environmental lighting.The evaluation renders reconstructions under novel natural environmental illumination.
4.1. Optimizing neural SDFs to fit single image: IRON’s edge-aware rendering converges to a valid silhouette-fitting solution, whereas IDR fails and mesh optimization degrades without intermediate remeshing.The controlled experiment uses a single image of a constant-color object and optimizes a neural SDF through image loss.
4.2. Inverse rendering from photometric images: IRON outperforms state-of-the-art baselines by a large margin in synthetic geometric accuracy and novel co-located-lighting generalization, while producing fewer blurry-texture artifacts on real data.The comparison uses synthetic and real datasets with DRV and PSDR baselines.
4.2. Inverse rendering from photometric images: Removing edge sampling produces poorer reconstruction around the dragon horn and pony nose because existing neural surface rendering lacks edge derivatives.The ablation examines error maps and surface normals on synthetic dragon and real pony data.
4.2. Inverse rendering from photometric images: The proposed loss functions produce the sharpest and most plausible material reconstructions; SSIM improves sharpness, while VGG introduces spurious texture details.The loss ablation is conducted on real dragon data.

5. Conclusion

IRON presents a neural-SDF inverse-rendering pipeline that combines flexible optimization with mesh and material-texture export. The authors identify capture, illumination, and surface-transparency limitations that motivate future work.

5. Conclusion: IRON uses neural SDFs and materials with volumetric radiance and edge-aware physics-based rendering while preserving conversion to meshes and material textures.The output format is intended for downstream graphics applications.
5. Conclusion: Photometric-image capture requires a more involved data-collection process, despite simplifying inverse rendering through known single-point lighting and minimal shadows.Future work considers combining flashlight and ambient illumination.
5. Conclusion: Without modeling multiple light bounces, IRON can estimate materials inaccurately in concave regions with significant interreflection.The authors propose efficient global-illumination rendering for neural SDFs as future work.
5. Conclusion: The opaque-surface BRDF assumption limits expected performance on transparent and translucent objects with significant refraction or subsurface scattering.This is identified as a limitation of the current model.

A. Neural network structures

IRON represents geometry and material components with compact multilayer perceptrons. The networks map spatial and directional inputs, together with geometric features, to SDF, albedo, specular, and roughness quantities.

A. Neural network structures: The neural SDF network is an 8-layer, width-256 MLP with a fourth-layer skip connection and six-frequency positional encoding for 3D locations.It outputs an SDF value and local geometric feature descriptor.
A. Neural network structures: The diffuse-albedo network is an 8-layer, width-256 MLP whose inputs include positional-encoded location and surface-direction features.The location uses 10 frequencies, while the second normal uses four.
A. Neural network structures: The specular-albedo network is a four-layer, width-256 MLP with six-frequency positional encoding of the 3D location.Its inputs include location, surface normal, and geometric feature information.
A. Neural network structures: The roughness network is a four-layer, width-256 MLP with six-frequency positional encoding of the 3D location.It predicts roughness from location, surface normal, and geometric features.

B. Implementation details

IRON uses a Mitsuba-compatible roughplastic BRDF and trains in two optimization stages with regularization and edge-localization parameters. The implementation fixes BRDF settings and specifies numerical thresholds for surface rendering.

B. Implementation details: IRON follows Mitsuba’s roughplastic BRDF, using the GGX distribution and default intIOR, extIOR, and nonlinear parameters.These settings define the BRDF implementation used during rendering.
B. Implementation details: The volumetric stage trains for 100k iterations with 512 randomly sampled pixels per iteration, using L1 image loss and eikonal regularization weighted by λ1 = 0.1.The eikonal term regularizes the neural SDF during volumetric radiance-field optimization.
B. Implementation details: The edge-aware surface stage uses eikonal and roughness-range loss weights of 0.1, with τ = 1e-2, K = 16, ϵ = 1e-3, and δ = 5e-2 for edge processing.These parameters control depth-gradient thresholding, surface walks, step size, and edge-point localization.

C. Proof of edge point re-parametrization

The proof establishes an edge-point re-parametrization by moving the point along the surface normal while preserving the SDF zero level set, then derives its first-order parameter dependence.

C. Proof of edge point re-parametrization: The derivation uses three steps: re-parametrizing the edge point, matching its function value, and matching its first derivative under the current parameters.This proof strategy is described as analogous to differentiable ray-surface intersection.
C. Proof of edge point re-parametrization: Implicit differentiation relates changes in the SDF parameters to the edge-point displacement through the surface gradient and the derivative of the normal-direction offset.Substitution yields the first-order expression for the offset derivative and then the final re-parametrization.
C. Proof of edge point re-parametrization: The proof derives the edge-point re-parametrization x_Θs = x − n/(n^Tn)·S_Θs(x) from the zero-level-set constraint.The construction begins with normal-direction motion and enforces S_Θs(x_Θs)=0 during deformation.
C. Proof of edge point re-parametrization: The re-parametrized point provides an unbiased first-order gradient with respect to Θs for gradient-based optimization.The claim is explicitly limited to first-order gradients suitable for gradient-based optimizers.

D. Reconstructed meshes and materials

IRON reconstructs meshes and materials for the five real-world scenes evaluated in the work.

D. Reconstructed meshes and materials: IRON reconstructs meshes and materials for 5 real-world scenes.The results are presented in Figure 11.

E. Comparison with PhySG

The comparison with PhySG highlights IRON’s handling of scene and material settings and its more accurate geometry reconstruction for the horse object.

E. Comparison with PhySG: Unlike PhySG, IRON does not require input object segmentation masks and can represent spatially-varying specular roughness.PhySG assumes a constant and uniform specular lobe shape.
E. Comparison with PhySG: Our IRON achieves a lower horse-object chamfer L1 distance than PhySG: 5.35e-4 versus 18.67e-4.The figure caption states that IRON recovers much more accurate geometry details for the horse object.