Source-linked AI summary
ReLi3D: Relightable Multi-view 3D Reconstruction with Disentangled Illumination
Jan-Niklas Dihlmann, Mark Boss, Simon Donne, Andreas Engelhardt, Hendrik P. A. Lensch, Varun Jampani
TL;DR
Single-view image-to-3D reconstruction remains ill-posed for separating materials from illumination, motivating a unified multi-view solution. ReLi3D uses transformer fusion, two prediction paths, differentiable Monte Carlo rendering, and mixed-domain training to reconstruct relightable assets in under one second. The paper reports improved reconstruction, relighting fidelity, and material realism, while noting resolution and out-of-domain lighting limitations.
Problem
Single-view reconstruction cannot reliably disentangle surface materials from illumination, limiting spatially varying material prediction and relighting fidelity.
Method
ReLi3D fuses arbitrary posed views with a cross-conditioning transformer, predicts geometry and PBR materials alongside HDR illumination through two paths, and trains them with differentiable Monte Carlo rendering and mixed-domain data.
Results
ReLi3D jointly predicts spatially varying PBR materials and coherent HDR environments from sparse images in under one second, with improved reconstruction, relighting fidelity, and material realism over recent pipelines.
Takeaways & Limitations
Multi-view constraints can support material-lighting separation in a unified feed-forward pipeline that produces complete relightable 3D assets at interactive speeds.
Takeaways & Limitations
Limited triplane resolution constrains practical texture and geometry resolution, while strong out-of-domain lighting can cause baked-in illumination in material maps.
Abstract
from arXiv · showhide
Reconstructing 3D assets from images has long required separate pipelines for geometry reconstruction, material estimation, and illumination recovery, each with distinct limitations and computational overhead. We present ReLi3D, the first unified end-to-end pipeline that simultaneously reconstructs complete 3D geometry, spatially-varying physically-based materials, and environment illumination from sparse multi-view images in under one second. Our key insight is that multi-view constraints can dramatically improve material and illumination disentanglement, a problem that remains fundamentally ill-posed for single-image methods. Key to our approach is the fusion of the multi-view input via a transformer cross-conditioning architecture, followed by a novel unified two-path prediction strategy. The first path predicts the object's structure and appearance, while the second path predicts the environment illumination from image background or object reflections. This, combined with a differentiable Monte Carlo multiple importance sampling renderer, creates an optimal illumination disentanglement training pipeline. In addition, with our mixed domain training protocol, which combines synthetic PBR datasets with real-world RGB captures, we establish generalizable results in geometry, material accuracy, and illumination quality. By unifying previously separate reconstruction tasks into a single feed-forward pass, we enable near-instantaneous generation of complete, relightable 3D assets. Project Page: https://reli3d.jdihlmann.com/
1 INTRODUCTION
ReLi3D addresses the ill-posedness and limitations of single-view 3D reconstruction by using multi-view constraints to disentangle materials from illumination. It unifies geometry, spatially varying PBR materials, and HDR environment prediction in a feed-forward system operating in under one second.
- Motivation: Single-view reconstruction remains ambiguous because identical 2D appearances can result from different combinations of surface reflectance and illumination.This ambiguity particularly affects unobserved regions, spatially varying materials, normals, and relighting fidelity.
- Motivation: Multi-view geometric consistency narrows the feasible material-lighting solutions by observing the same surface points under common illumination.ReLi3D makes multi-view fusion the primary mechanism for material-lighting disentanglement rather than an auxiliary robustness feature.
- Contribution: ReLi3D converts a variable number of posed images into a textured mesh with spatially varying PBR materials and a coherent HDR environment in less than one second.The system is presented as a unified feed-forward pipeline for jointly reconstructing these components.
- Contribution: A shared cross-conditioning transformer, two-path prediction, differentiable Monte Carlo multiple-importance-sampling rendering, and mixed-domain training jointly support the reconstruction pipeline.The geometry-and-appearance path predicts mesh and materials, while the illumination path predicts environment lighting; training combines synthetic PBR and real-world RGB data.
- Results: The resulting pipeline jointly reconstructs geometry, spatially varying materials, and HDR illumination at interactive speed, with experiments reporting improved reconstruction, relighting fidelity, and material realism.The comparison is made against recent generative and reconstruction pipelines.
2 RELATED WORK
Related work spans optimization-based inverse rendering, image-to-3D generation, direct 3D diffusion, and feed-forward reconstruction. ReLi3D is positioned as a multi-view, feed-forward approach that adds spatially varying materials and environment estimation while combining synthetic and real-world training data.
- Inverse Rendering: Optimization-based inverse-rendering methods estimate shape, appearance, and lighting from dense imagery but require lengthy per-instance optimization.Differentiable rendering and regularization reduce ambiguity without eliminating the many plausible material-lighting explanations.
- Image-to-3D Generation: Diffusion-based image-to-3D methods can produce geometric fidelity but suffer from artifacts, slow inference, view inconsistencies, or inherited inverse-rendering ambiguities.These limitations arise across score-distillation, multi-view generation, and related image-to-3D pipelines.
- Image-to-3D Reconstruction: Large Reconstruction Models perform direct feed-forward image-to-3D inference using transformer architectures and large datasets.Earlier regression approaches were constrained by smaller datasets such as ShapeNet, whereas LRMs operate at larger scale.
- Image-to-3D Reconstruction: Existing fast reconstruction methods commonly use single-view input, global material values, and no environment estimation, leaving material-lighting disentanglement ill-posed.The same appearance may still correspond to countless material-illumination combinations.
- ReLi3D: ReLi3D uses multi-view constraints as the primary disentanglement mechanism and combines environment estimation with mixed-domain training for synthetic and real-world applicability.The related work discussion distinguishes it from LIRM, which lacks illumination prediction and uses purely synthetic supervision.
3 PRELIMINARIES
The preliminaries define spatially varying physically based materials, environment illumination, and triplane representations used for feed-forward 3D reconstruction. ReLi3D combines these representations with a two-path architecture that predicts object structure and appearance alongside illumination.
- Physically Based Materials: Physically based rendering represents surface appearance through BRDFs, with spatially varying BRDFs allowing material properties to change across the surface.The BRDF describes reflected light as a function of incoming and outgoing directions.
- Physically Based Materials: ReLi3D parameterizes materials with RGB albedo, scalar roughness, scalar metallic values, and normal bump maps for fine geometric detail.The local tangent space is defined without predefined UV mappings using the surface normal and world coordinate system.
- ReLi3D Architecture: ReLi3D fuses arbitrary input views and jointly predicts spatially varying materials and environment illumination through a unified two-path architecture.Its overview uses a geometry-and-appearance path and an illumination path connected through differentiable rendering.
- Environment Illumination: Environment illumination is represented as a directional environment map, while RENI++ provides a compact latent representation suited to fast feed-forward reconstruction.Traditional spherical harmonics and spherical Gaussians are described as limited for high-frequency lighting such as sharp shadows and bright sources.
- Triplane Representations: A triplane stores 3D information in three orthogonal 2D feature planes, whose projections at a 3D point are concatenated and decoded into geometry and appearance properties.This representation underlies direct image-to-3D systems such as LRM and TripoSR.
4 METHOD
ReLi3D fuses arbitrary multi-view inputs into shared representations that jointly predict geometry, spatially varying materials, and HDR illumination. Its two-path architecture and differentiable MC+MIS training support physically consistent reconstruction from mixed-domain data.
- Cross-view Feature Fusion: A shared cross-conditioning transformer fuses arbitrary views into consistent triplane features for both reconstruction paths.A randomly selected hero view drives the query stream, while latent mixing incorporates non-hero views into the memory attended by the transformer.
- Spatially Varying Material Prediction: The geometry+appearance path predicts mesh structure and spatially varying BRDF parameters from unified triplane features.Task-specific MLP heads derive density, albedo, roughness, metallic, and normal perturbations from a single shared triplane embedding; Flexicubes extracts the mesh.
- Multi-view Environment Estimation: The illumination path estimates coherent HDR environments from multi-view background observations or indirect material-reflection cues.Mask-aware image tokens are concatenated with object-transformer outputs, and a dedicated transformer maps the environment context to a RENI++ latent and global rotation.
- Multi-view Environment Estimation: Stochastic background masking trains the system to infer illumination directly from visible backgrounds or indirectly from object reflections and shading.This dual-mode setup targets real-world scenes where backgrounds may be partially unavailable or occluded.
- Disentangled Training via MC+MIS: A differentiable physically based MC+MIS renderer couples reconstruction paths, supports mixed supervision, and bridges synthetic and real-world domains.The renderer enforces joint material-light transport consistency, uses direct PBR supervision when available, and enables image-reconstruction supervision otherwise.
- Disentangled Training via MC+MIS: ReLi3D learns spatially varying material reconstruction from mixed-domain data without supervision collapse while maintaining physical plausibility.The reported system is designed to preserve robust real-world performance across synthetic PBR, synthetic RGB-only, and real captures.
5 EXPERIMENTS
ReLi3D is evaluated on geometry, material, relighting, illumination, and real-world generalization, with multi-view input improving material-lighting disentanglement and reconstruction quality. The method combines competitive reconstruction with fast inference, while remaining limited by geometric resolution and some out-of-domain lighting cases.
- Material Prediction: ReLi3D ranks first across all material metrics, including 25.00 dB albedo PSNR versus SF3D’s 18.42 dB, 22.69 dB roughness PSNR, and 32.73 dB metallic prediction.Additional views further improve spatially varying material prediction by resolving material-lighting ambiguities.
- Relighting Performance: ReLi3D ranks first across all relighting metrics, and its relit reconstructions closely resemble ground truth under novel out-of-distribution HDR environments.The reported results attribute this to accurate material estimation that generalizes to novel lighting conditions.
- Environment Estimation: Single-view predictions recover sky color and sun direction, while background information identifies light sources and multiple views improve light-direction recovery in dark environments.Compared with ReLi3D, SPAR3D often produces over-smoothed, low-contrast environment maps without clear light sources.
- Overall Reconstruction Quality: Four views improve Chamfer Distance by 27% and yield an F-score@0.5 of 0.993, while multi-view reconstruction runs in an average 0.31s.Performance saturates beyond 4–8 views because additional random views often provide redundant coverage.
- Limitations: ReLi3D delivers complete relightable assets in under a second and runs 100× faster than generative approaches like Hunyuan3D, but specialized diffusion methods may achieve superior geometric detail.The method’s largest remaining weakness is limited triplane resolution, which constrains texture and geometry resolution in practice.
6 CONCLUSION
ReLi3D addresses illumination disentanglement by jointly predicting spatially varying PBR materials and coherent HDR environments from sparse images. The resulting feed-forward system produces relightable assets at interactive speed and is positioned for rapid asset digitization.
- ReLi3D jointly predicts spatially varying PBR materials and coherent HDR environments from sparse image inputs.
- The two-path architecture and differentiable Monte Carlo training support material-lighting separation at interactive speeds.
- The framework could enable rapid generation of physically accurate 3D assets from casual captures for real-time asset digitization.
- The authors release code, pretrained weights, and dataset-generation scripts to support adoption and reproducibility.
A.1 COMPARISON
ReLi3D produces detailed, spatially varying material decompositions and robust reconstructions across synthetic and challenging real-world inputs. Its illumination predictions more closely match ground truth, while multi-view inputs reduce uncertainty in unseen regions.
- Material prediction: ReLi3D predicts detailed spatially varying albedo, roughness, and metallic maps, including mixed-material objects.Unlike approaches predicting global properties or failing at material-lighting separation, it preserves realistic spatial variation.
- Reconstruction: ReLi3D generalizes across synthetic objects and real-world captures tested with imperfect masks, camera errors, and difficult lighting.The evaluation covers both controlled synthetic data and challenging real-world conditions.
- Multi-view input: Additional views improve real-world reconstruction by resolving uncertainties in previously unseen object regions.The multi-view setting also benefits material prediction when observed lighting aligns more closely with ground truth.
- Material prediction: In challenging real-world captures with strong reflections and blur, ReLi3D separates metallic and non-metallic materials.This result is reported for UCO3D examples with motion blur and cluttered backgrounds.
- Material prediction: ReLi3D produces PBR maps and relit renderings for complex multi-material objects across diverse geometries and real materials.The Blender Shiny evaluation includes basecolor, roughness, metallic, and normal predictions in novel environments.
- Illumination: ReLi3D more accurately matches environment-map shape and color than DiffusionLight and SPAR3D in qualitative illumination comparisons.DiffusionLight predicts substantially different environments, while SPAR3D fails to recover meaningful illumination.
A.2 ABLATION
The ablations identify physically based Monte Carlo rendering as central to material-lighting disentanglement, while intermediate Gaussian stages primarily support geometry coverage. Illumination comparisons further show ReLi3D’s qualitative and quantitative advantages over competing approaches.
- Material results: On complex objects, ReLi3D predicts spatially varying PBR maps and produces relit renderings across novel environments.The Blender Shiny results include basecolor, roughness, metallic, and normal maps.
- Renderer ablation: Removing the Monte Carlo renderer reduces reconstruction PSNR from 19.92 to 17.54 dB.The ablation attributes the degradation to removing the differentiable Monte Carlo renderer with Multiple Importance Sampling.
- Renderer ablation: Monte Carlo rendering is presented as a fundamental component for physically accurate material-lighting disentanglement, not merely an optimization detail.This conclusion follows from the substantial reconstruction-quality drop in the ablation.
- Training-stage ablation: Gaussian stages contribute 70–80% of 3D coverage gains while progressively preparing training for full Monte Carlo integration.Their larger batch sizes mainly stabilize geometry before the more expensive rendering stage.
- Illumination comparison: Qualitative results show ReLi3D matching ground-truth environment shape and color more closely than DiffusionLight and SPAR3D.The comparison reports hallucinated environments for DiffusionLight and failed illumination recovery for SPAR3D.
- Illumination comparison: ReLi3D achieves 20.88 dB relighting PSNR versus 20.93 dB for DiffusionLight, while inference takes 0.34 seconds versus 21.46 seconds.SPAR3D reaches 17.10 dB PSNR at similar speed, indicating substantially lower relighting quality.
B IMPLEMENTATION DETAILS
ReLi3D uses mixed-domain supervision, progressive rendering stages, and multi-view transformer conditioning to train a unified geometry, material, and illumination reconstruction pipeline. Its design balances physical rendering fidelity, representation efficiency, and computational constraints.
- Training objectives: The training objective combines image reconstruction with material, geometry, and environment supervision across mixed-domain data.The formulation adapts supervision to the varying ground-truth availability of synthetic and real-world examples.
- Training objectives: Material supervision uses MSE for basecolor, roughness, and metallic values, cosine similarity for normals, and flatness regularization for bump maps.Environment supervision uses RENI++ latents when available and neutral-white demodulation regularization otherwise.
- Rendering pipeline: Training progresses from volumetric implicit-field rendering to spherical-Gaussian mesh rendering and finally full Monte Carlo integration.The final stage uses VNDF sampling, spherical caps, and antithetic sampling for physically accurate shading.
- Rendering pipeline: All stages use image, material, geometry, and environment losses, while every network component remains trainable throughout the progressive schedule.Stages 1–3 use spherical-Gaussian lighting and stage 4 uses full Monte Carlo integration.
- Training configuration: The pipeline uses 512 × 512 inputs, randomly samples 1–4 conditioning views, and trains end-to-end with a learning rate of 5 × 10^-5.Batch sizes are 64 for volumetric rendering, 192 for spherical-Gaussian stages, and 32 for Monte Carlo integration.
- Multi-view conditioning: The hero view feeds the cross-conditioning transformer, and random selection yields metrics largely comparable to fixed frontal-view selection.The authors report only marginal differences, with slight perceptual gains for random views possibly reflecting side-view parallax.
- Illumination representation: RENI++ is selected because its compact, high-frequency representation supports photorealistic relighting under memory constraints.Low-frequency Gaussian representations miss sharp highlights and directional suns, while larger environment-map HDR prediction is not feasible under the stated constraints.
- Dataset and rendering protocol: Mixed-domain training combines synthetic PBR data with real-world RGB captures to provide material supervision, geometric diversity, and domain adaptation.The synthetic corpus extends the TripoSR protocol with ABO and ARIA datasets, while renderings use diverse HDRI environments and asymmetric view counts by supervision level.
C.2 REAL-WORLD DATA PREPARATION
Real-world UCO3D data is heavily filtered and preprocessed before integration with synthetic training data. RGB-only real-world supervision is combined with synthetic material supervision to support generalization while preserving physical material properties.
- UCO3D requires extensive preprocessing to become compatible with the synthetic data pipeline.
- Quality filtering removes samples with motion blur, inaccurate masks, or poor camera estimates, retaining only objects with scores ≥1.0.This substantially reduces dataset size but improves training stability and supervision quality.
- The preprocessing pipeline crops and centers objects, adjusts camera intrinsics, tracks valid regions, estimates surface normals, and normalizes scale.
- The processed data preserves challenging real-world characteristics while enabling seamless integration with synthetic training data.
- RGB-only real-world supervision is used with image-space reconstruction losses, while synthetic data supplies direct material supervision.This mixed-domain protocol supports real-world generalization while maintaining physically based material properties learned from synthetic supervision.
D LIMITATIONS AND FAILURE CASES
ReLi3D’s decomposition can fail under unusual illumination, strong self-shadowing, or dark scenes, and its reconstruction quality is constrained by representation resolution. Transparent surfaces and inaccurate camera poses remain additional scope boundaries.
- Rare decomposition failures bake lighting into material maps under out-of-prior lighting, strong self-shadowing, or dark scenes.The authors note that ReLi3D still outperforms strong baselines in these challenging cases.
- Limited triplane resolution constrains practical texture and geometry resolution, with blur also linked to a DINOv2 fine-tuning bottleneck.
- Explicit mesh reconstruction of transparent surfaces remains outside the current scope despite density-based NeRF pre-training handling transparency.
- The method assumes known camera poses and physically plausible materials, while severe pose errors can produce blur artifacts.Generated multi-view images may also degrade performance when their poses and appearances are inconsistent.