Source-linked AI summary
TensoIR: Tensorial Inverse Rendering
Haian Jin, Isabella Liu, Peijia Xu, Xiaoshuai Zhang, Songfang Han, Sai Bi, Xiaowei Zhou, Zexiang Xu, Hao Su
TL;DR
Inverse rendering must recover scene properties from images captured under unknown illumination, while purely MLP-based neural fields are limited by capacity and computation. TensoIR extends TensoRF with a tensor-factorized representation that jointly estimates geometry, materials, illumination, and radiance fields, including secondary lighting and multiple capture conditions. The authors report state-of-the-art reconstruction and rendering quality with faster reconstruction across challenging synthetic and real scenes.
Problem
Inverse rendering seeks scene geometry and materials from images under unknown illumination, but purely MLP-based neural fields have limited capacity and high computational costs.
Method
TensoIR extends TensoRF with a tensor-factorized representation that jointly reconstructs radiance fields and physically based scene properties, including geometry, materials, illumination, and secondary lighting.
Results
TensoIR achieves state-of-the-art scene reconstruction and rendering quality, supports single or multiple unknown lighting conditions, and reconstructs faster than previous neural-field methods.
Takeaways & Limitations
The shared tensorial representation enables photo-realistic novel-view synthesis, relighting, material editing, and efficient multi-light reconstruction.
Takeaways & Limitations
The lighting model assumes the unknown natural environment is distant and represents global illumination as an environment map parameterized by spherical Gaussians.
Abstract
from arXiv · showhide
We propose TensoIR, a novel inverse rendering approach based on tensor factorization and neural fields. Unlike previous works that use purely MLP-based neural fields, thus suffering from low capacity and high computation costs, we extend TensoRF, a state-of-the-art approach for radiance field modeling, to estimate scene geometry, surface reflectance, and environment illumination from multi-view images captured under unknown lighting conditions. Our approach jointly achieves radiance field reconstruction and physically-based model estimation, leading to photo-realistic novel view synthesis and relighting results. Benefiting from the efficiency and extensibility of the TensoRF-based representation, our method can accurately model secondary shading effects (like shadows and indirect lighting) and generally support input images captured under single or multiple unknown lighting conditions. The low-rank tensor representation allows us to not only achieve fast and compact reconstruction but also better exploit shared information under an arbitrary number of capturing lighting conditions. We demonstrate the superiority of our method to baseline methods qualitatively and quantitatively on various challenging synthetic and real-world scenes.
1. Introduction
TensoIR addresses inverse rendering under unknown illumination with a tensor-factorized framework that jointly reconstructs geometry, materials, lighting, and radiance fields. Its design targets efficient secondary-light computation and multi-light capture while enabling high-quality rendering and relighting.
- Inverse rendering seeks 3D scene attributes such as shape and materials from images, but unknown illumination makes the problem challenging and ill-posed.
- Pure MLP-based neural fields have limited capacity and high computational costs, restricting inverse-rendering accuracy and efficiency.
- TensoIR uses a low-rank tensor-factorized representation with shared features and small MLPs to jointly estimate geometry, materials, illumination, and radiance fields.The representation regresses density, view-dependent color, normals, and material properties, supporting both radiance-field and physically based rendering.
- Online low-cost second-bounce ray marching explicitly computes visibility and indirect lighting instead of ignoring or distilling secondary effects.This design supports more accurate modeling of shadows and indirect illumination.
- A fifth tensor dimension represents lighting conditions, allowing joint multi-light reconstruction that exploits additional photometric cues while limiting the cost of separate radiance fields.The added vector factor varies appearance across lights, and multi-light capture can improve reconstruction quality over a single-light setting.
- Experiments report state-of-the-art reconstruction and rendering quality, faster reconstruction, and support for novel-view synthesis, relighting, and material editing.The reported evaluation covers challenging synthetic and real scenes under single or multiple unknown lighting conditions.
2. Related Works
Related work spans neural scene representations and inverse-rendering methods, with prior neural approaches often limited by computation-intensive MLPs. These costs become especially problematic when modeling secondary shading effects.
- Neural scene representations offer flexible alternatives to meshes, volumes, and point clouds, supporting high-fidelity geometry and appearance modeling.NeRF and related neural fields have been applied to novel-view synthesis and other 3D-content applications.
- TensoIR builds on TensoRF to combine neural radiance-field rendering with physically based light transport in a tensor-factorized representation.
- Traditional inverse-rendering methods commonly infer geometry and materials using triangle meshes obtained or initialized from depth sensors or multi-view stereo.
- Neural inverse-rendering methods are constrained by computation-intensive MLPs, particularly when secondary rays must be extensively sampled for shadowing.
- Prior methods often ignore secondary effects, use collocated flash lighting, or distill effects into additional MLP networks at extra computational cost.
3. Method
TensoIR jointly represents radiance fields and physically based scene properties with a TensoRF-based tensor factorization, supporting efficient reconstruction under unknown single or multiple lighting conditions.
- Framework: The framework jointly reconstructs scene geometry, materials, and illumination from multi-view images under unknown lighting using neural radiance-field and physically based rendering.It determines surface points from volume-rendering weights and applies a parametric BRDF at those points.
- Radiance Field Rendering: Radiance-field rendering samples points along camera rays and combines view-dependent radiance using density-based transmittance and rendering weights.Volume density, step size, radiance color, and transmittance determine each sample’s contribution.
- Physically Based Rendering: Physically based rendering integrates incident illumination, BRDF response, and the surface-normal cosine term over the upper hemisphere.The BRDF uses spatially varying albedo and roughness together with the predicted surface normal.
- TensoRF-Based Representation: The representation uses a low-rank factorized density tensor and a 4D appearance tensor, decoding radiance, normals, and material properties with small MLPs.Density is obtained by tensor interpolation, while appearance features are decoded into view-dependent color, shading normal, and BRDF parameters.
- Joint Representation: Shared tensor features support both rendering paradigms, enabling efficient density and transmittance computation while radiance-field rendering facilitates physically based reconstruction.The joint representation is described as a radiance field and physically based rendering model sharing neural features.
- Multi-Light Representation: A 5D appearance tensor adds a lighting dimension for multi-light inputs, sharing physical scene properties while decoding view-dependent colors per lighting condition.This extension efficiently queries radiance under different illumination and supports indirect-illumination computation; multi-light inputs provide photometric cues for more accurate geometry and material estimation.
4. Experiments
Experiments on synthetic and real scenes show that TensoIR improves inverse-rendering quality, secondary shading accuracy, and reconstruction efficiency across single- and multi-light settings.
- Datasets: Experiments cover four complex synthetic scenes, the NeRF-synthetic dataset, and four captured real scenes.Synthetic scenes include Blender and Stanford 3D scanning data with rendered ground-truth images, BRDF parameters, and normal maps.
- Comparisons with previous methods: TensoIR significantly outperforms NeRFactor and InvRender on albedo, normal, and relighting metrics averaged across four scenes.The evaluation uses PSNR, SSIM, and LPIPS under a single unknown lighting condition, with additional multi-light results.
- Comparisons with previous methods: Explicit second-bounce ray marching produces more accurate shadows and indirect lighting than approaches using visibility approximations or limited shading computation.NeRFactor ignores indirect illumination, while InvRender uses spherical-Gaussian shading with limited accuracy.
- Multi-light results: Multi-light inputs improve inverse-rendering accuracy without additional computation costs and help resolve ambiguity between lighting and materials.The tensor representation supports additional lighting conditions efficiently, unlike purely MLP-based baselines.
- Comparisons with previous methods: 25 minutes of optimization yields high-quality geometry reconstruction and higher-frequency details than methods trained for 15 and over 100 hours.Full training takes 5 hours for best performance, but useful quality is reached substantially earlier.
- Indirect illumination and visibility: Removing visibility and indirect-illumination modeling causes secondary shading to be baked into albedo or normals, while the full method achieves more accurate reconstruction.The ablation demonstrates the role of physically accurate light transport in inverse rendering.
5. Conclusion
TensoIR jointly reconstructs radiance fields and physically based scene models from multi-view images under unknown illumination. Its tensor-factorized representation supports efficient, high-quality reconstruction, accurate secondary lighting, and flexible multi-light capture.
- Conclusion: TensoIR jointly estimates density, normals, lighting, materials, and radiance fields for efficient scene reconstruction under unknown illumination.The joint representation supports photo-realistic rendering and relighting.
- Conclusion: The framework computes shadowing and indirect lighting accurately while supporting an arbitrary number of lighting conditions.These capabilities follow from the efficiency and extensibility of the tensor-factorized representation.
- Conclusion: TensoIR achieves state-of-the-art inverse-rendering results and outperforms previous neural methods in reconstruction quality and efficiency.
Supplementary Material
The supplementary material provides additional scene results, implementation details, design-choice analyses, and expanded dataset and multi-light setup information.
- Supplementary Material: Additional results include per-scene reconstruction for four synthetic scenes and reconstruction results for four complex real scenes.
- Supplementary Material: The supplementary discusses implementation details, design choices, loss-weight effects, synthetic dataset generation, and multi-light capture setups.
B. Per-Scene Results on the Synthetic Dataset
Across four synthetic scenes, TensoIR outperforms both baseline methods and recovers normals, albedo, roughness, and relighting results under single- and multi-light settings.
- TensoIR outperforms both baselines on all four evaluated synthetic scenes.
- The method recovers normals, albedo, roughness, and relighting results across the four synthetic scenes.
C. Results on Real-World Captures
The authors evaluate TensoIR on four real objects captured under natural illumination in the wild, using fixed camera parameters and approximately uniform views around each object.
- Four real objects are captured under natural illumination in the wild for real-data evaluation.
- Camera parameters are fixed, photos are taken roughly uniformly around each object, and COLMAP estimates the camera poses after background removal.
- Figure 8 presents reconstructed geometry, BRDF, and lighting for the real captures.
D. Implementation Details
The implementation uses factorized tensor fields, multi-light feature averaging, coarse-to-fine training, specialized ray sampling, and regularization losses with tuned weights.
- Representation details: The TensoRF-based representation uses factorized 3D density and 4D appearance tensors, with higher resolutions and more components generally improving reconstruction quality.
- Multi-light representation: Multi-light decoding uses a mean appearance feature computed by averaging lighting vectors along the lighting dimension.
- Training details: Training uses Adam, initial learning rates of 0.02 for tensor factors and 0.001 for MLPs, and coarse-to-fine spatial upsampling.
- Training details: Training runs for 80k iterations and averages 5 hours, while alphaMask generation enables 4096-ray batches instead of memory-limited 1024-ray batches.
- Ray marching details: During training, visibility and indirect lighting use 512 secondary rays per surface point with 96 points per ray, filtering half by surface orientation.
- Ray marching details: Relighting uses lighting-intensity importance sampling to draw more rays toward directions with stronger illumination.
- Loss details: BRDF smoothness regularization encourages spatial coherence through scale-invariant relative differences in predicted roughness and albedo.
- Loss details: Larger normal-difference loss weights reduce overfitting but damage high-frequency detail, with reasonable weights between 4 × 10−4 and 6 × 10−3.
E. More Details and Analysis on Our Synthetic
Synthetic and real multi-light experiments test rotated, general, and image-count-controlled settings, showing that multi-light inputs improve reconstruction and lighting estimation while favoring novel-light rendering.
- Dataset and settings: The synthetic dataset contains four complex scenes rendered with high-resolution environment maps, using 100 training views and 200 test views.
- Real multi-light capture: Rotating a real object three times under the same environment lighting provides a practical multi-light capture that produces higher-quality reconstruction than single-light capture.
- Dataset and settings: Rotated multi-light data uses three azimuth rotations of the same environment map, while general multi-light data uses three unrelated environment maps optimized separately.
- Controlled multi-light evaluation: The limited general multi-light setting matches single-light image counts yet achieves better BRDF estimation and geometry reconstruction.
- Lighting reconstruction: Multi-light input enables more accurate environment-map reconstruction than single-light input.
- Results and analysis: Multi-light reconstruction recovers more normal-map details and better shading- and artifact-free albedo maps than single-light reconstruction.
- Results and analysis: Single-light novel-view synthesis is slightly better under the same single lighting, whereas multi-light reconstruction renders much better under novel lighting.
F. Limitations
TensoIR’s surface-based rendering model is limited for complex light transport and non-dielectric materials. Its fixed Fresnel assumption restricts modeling of metals, while the method otherwise reconstructs normals, albedo, roughness, and relighting across synthetic and real scenes.
- Rendering limitations: Surface-based rendering cannot model translucent water with strong reflection and refraction or transparent glass.These cases are identified as limitations in the Ship and Drums scenes.
- Material assumptions: Fixing the Disney BRDF Fresnel F0 term to 0.04 assumes dielectric materials and limits modeling of metals.The paper suggests a learned neural BRDF as a possible future direction.
- Synthetic evaluations: Synthetic evaluations report normal, albedo, roughness, and relighting results, but ground-truth roughness is unavailable because the data does not use Disney BSDF rendering.
- Real-scene results: The method decomposes normal, albedo, roughness, and lighting on four real objects.