PaperlayerView exact revision on arXiv

Source-linked AI summary

RelitLRM: Generative Relightable Radiance for Large Reconstruction Models

Tianyuan Zhang, Zhengfei Kuang, Haian Jin, Zexiang Xu, Sai Bi, Hao Tan, He Zhang, Yiwei Hu, Milos Hasan, William T. Freeman, Kai Zhang, Fujun Luan

arXiv:2410.06231v2cs.CVcs.GRcs.LG

TL;DR

RelitLRM addresses relightable 3D reconstruction from sparse posed images under unknown static lighting, where prior methods often require dense captures and slow optimization. It uses a feed-forward transformer with deterministic geometry reconstruction and diffusion-based probabilistic appearance generation. The model matches or often exceeds state-of-the-art baselines while using fewer input images and taking 2 to 3 seconds instead of hours for per-scene optimization.

  • Problem

    Relightable 3D reconstruction from sparse images under unknown lighting is difficult because prior approaches often require dense controlled captures, slow optimization, and struggle with material-lighting ambiguity and complex light transport.

    Source: §1 · p. 1, §3.2 · p. 4

  • Method

    RelitLRM combines a transformer-based deterministic geometry reconstructor with a diffusion-based probabilistic relit-appearance generator trained end-to-end on relighting data.

    Source: §1 · pp. 1–2, §3.3 · p. 5, §6 · p. 10

  • Results

    RelitLRM matches or often exceeds state-of-the-art baselines while requiring significantly fewer input images and completing reconstruction in 2 to 3 seconds.

    Source: §1 · p. 2, §6 · pp. 10–11

  • Takeaways & Limitations

    The approach provides an efficient sparse-view route to relightable 3D reconstruction that captures complex effects including shadows and specularity.

    Source: §6 · pp. 10–11

    The method requires camera parameters for input views and represents target lighting with environment maps that cannot accurately represent near-field lighting.

    Source: §5.5 · p. 10

Abstract

from arXiv · show

We propose RelitLRM, a Large Reconstruction Model (LRM) for generating high-quality Gaussian splatting representations of 3D objects under novel illuminations from sparse (4-8) posed images captured under unknown static lighting. Unlike prior inverse rendering methods requiring dense captures and slow optimization, often causing artifacts like incorrect highlights or shadow baking, RelitLRM adopts a feed-forward transformer-based model with a novel combination of a geometry reconstructor and a relightable appearance generator based on diffusion. The model is trained end-to-end on synthetic multi-view renderings of objects under varying known illuminations. This architecture design enables to effectively decompose geometry and appearance, resolve the ambiguity between material and lighting, and capture the multi-modal distribution of shadows and specularity in the relit appearance. We show our sparse-view feed-forward RelitLRM offers competitive relighting results to state-of-the-art dense-view optimization-based baselines while being significantly faster. Our project page is available at: https://relit-lrm.github.io/.

1 INTRODUCTION

RelitLRM reconstructs relightable 3D objects from sparse posed images under unknown lighting using a transformer-based generative architecture. It matches leading inverse-rendering performance while reducing input-view and processing requirements.

Input and output

  • RelitLRM reconstructs high-quality, relightable 3D objects from as few as 4–8 posed images captured under unknown lighting.The method targets novel viewpoints and novel lighting conditions while representing objects as 3D Gaussian primitives.Source: §1 · pp. 1–2

Relighting capability

  • Its generative appearance model captures multi-modal relit effects such as small strong specular highlights that regression-based approaches tend to ignore.This is especially relevant for glossy objects and high-frequency novel lighting.Source: §1 · p. 1

Applications

  • RelitLRM supports object insertion into virtual scenes by relighting objects according to surrounding illumination and producing shadows and glossy highlights under challenging lighting.The demonstrations include indoor scene insertion and strong directional sunlight.Source: Figure 1 · p. 2, §1 · p. 2

Results

  • Experiments on synthetic and real-world datasets show performance matching state-of-the-art inverse-rendering methods with significantly fewer input images and much less processing time.The reported processing gap is seconds versus hours, and the method avoids fixed analytic reflectance models.Source: §1 · p. 2

Architecture

  • RelitLRM combines a regression-based geometry reconstructor with a diffusion-based appearance synthesizer to disentangle geometry from appearance and model relighting uncertainty.Both transformer components are trained end-to-end on a large-scale 3D object dataset.Source: §1 · pp. 1–2

2 RELATED WORK

Prior work advances neural reconstruction and image-based relighting but remains limited by dense optimization, difficult relighting integration, or single-view inputs. RelitLRM combines sparse-view feed-forward reconstruction with generative relighting in one transformer-based architecture.

2.1 INVERSE RENDERING

  • Traditional and learning-based inverse-rendering methods commonly require dense inputs and computationally intensive optimization, limiting scalability with complex lighting and materials.Traditional systems typically use 100–200 controlled-lighting views, while later neural methods retain dense-input or optimization burdens.Source: §2.1 · pp. 2–3
  • RelitLRM addresses these limitations by reconstructing relightable 3D objects from 4–8 sparse images without per-scene optimization.Its transformer architecture uses neural implicit radiance representations within the 3DGS framework to disentangle geometry, appearance, and illumination.Source: §2.1 · p. 3

2.2 LARGE RECONSTRUCTION MODEL

  • Large Reconstruction Models provide scalable transformer-based feed-forward reconstruction from sparse inputs, but integrating relighting capabilities remains challenging.RelitLRM extends this model family with relightable reconstruction using neural implicit radiance representations within 3DGS.Source: §2.2 · p. 3

2.3 IMAGE-BASED RELIGHTING AND DIFFUSION MODELS

  • Diffusion-based image-relighting methods improve lighting control, but the cited approaches are constrained to single-view inputs.Earlier relighting methods are also often domain-specific or rely on explicit scene decomposition.Source: §2.3 · p. 3
  • RelitLRM integrates generative relighting and 3D reconstruction end-to-end to render objects from novel viewpoints under arbitrary lighting given sparse inputs.This directly combines the multi-view reconstruction and controllable relighting directions described in the related work.Source: §2.3 · p. 3

3 METHOD

RelitLRM combines deterministic geometry reconstruction with diffusion-based relit appearance generation to produce relightable 3D Gaussian representations from sparse views. Its denoising process predicts and renders appearance under target lighting while jointly training geometry and appearance.

3 METHOD

  • RelitLRM replaces GS-LRM’s deterministic appearance component with a generative relighting module for relightable output assets.The architecture retains transformer-based geometry reconstruction while adding relit-view diffusion for appearance synthesis.Source: §3 · p. 3

3.1 REVIEW OF GS-LRM MODEL

  • GS-LRM concatenates posed images with Plücker rays, tokenizes them, and predicts one 3DGS per input pixel through transformer blocks trained with novel-view rendering loss.Source: §3.1 · p. 4

3.2 GENERATING RELIT 3DGS APPEARANCE VIA RELIT VIEW DIFFUSION

  • Sparse unknown-lighting observations make relighting ambiguous, while glossy objects under high-frequency illumination require modeling many possible small specular regions.These limitations motivate a generative model rather than regression dominated by diffuse or weakly specular areas.Source: §3.2 · p. 4
  • The relit-view diffusion model conditions on target illumination and predicted geometry, then predicts and renders relit 3D Gaussian appearance at every denoising step.The final denoising step supplies the relit 3DGS output, with the 3D representation serving as a bottleneck.Source: §3.2 · p. 5
  • Geometry parameters and geometry features are first predicted from sparse inputs, then combined with noisy relit views, lighting tokens, and timestep tokens in a second transformer stack.The stack decodes spherical-harmonic coefficients for the 3D Gaussians while non-appearance tokens are discarded.Source: §3.2 · p. 5

3.3 TRAINING SUPERVISION

  • Training samples sparse source views and differently lit target views, applies ℓ2 and perceptual losses to rendered views, and jointly trains geometry and relit-view denoising end-to-end.Source: §3.3 · p. 5

3.4 SAMPLING RELIT RADIANCE

  • At inference, DDIM sampling and classifier-free guidance generate a chain of relit radiances, retaining only the final denoising step as the predicted appearance.Source: §3.4 · p. 5

4 DATASET AND IMPLEMENTATION

RelitLRM is trained and implemented with large-scale object, lighting, and multi-view data. Its training schedule uses staged resolution and view-count increases, while sampling is tuned for efficient generation.

DATASET

  • The training corpus combines 800K Objaverse objects with 210K Zeroverse objects, including added concave, metallic, and rough materials for more complex shadowing and specularity.Source: §4 · p. 6
  • Over 8,000 HDR environment maps are augmented 16-fold, and objects are rendered from 10–32 viewpoints under 2–12 lighting conditions.Source: §4 · p. 6

IMPLEMENTATION

  • The model contains 24 geometry-transformer layers, 8 appearance-diffusion layers, and approximately 0.4 billion trainable parameters.Source: §4 · p. 6
  • Images and environment maps use 8 × 8 patches, while denoising views use 16×16 patches to improve computational efficiency.Source: §4 · p. 6
  • Training begins with four input and four target views at 256 × 256, then fine-tunes with six input and six target views at 512 × 512.Pretraining uses 80K iterations and the higher-resolution stage expands the context window to up to 31K tokens.Source: §4 · p. 6
  • Five sampling steps produce realistic and diverse results, while CFG 3.0 gives the best result for the res-256 model.Source: §4 · p. 6

5 EXPERIMENTS

RelitLRM achieves competitive relighting with sparse inputs and seconds-scale processing, while its probabilistic design improves complex specular appearance modeling. Experiments also show practical insertion capabilities, configurable relighting behavior, and limitations in camera requirements and scalability.

5.1 COMPARISON WITH OPTIMIZATION-BASED METHODS

  • 6–8 views and 2–3 seconds yield comparable relighting to optimization-based methods using 42–100 views and hours of processing.Qualitative comparisons show fewer baked highlights and better specular reflections than optimization baselines.Source: §5.1 · p. 7, Figure 3 · p. 7, Table 1 · p. 8

5.2 COMPARISON WITH DATA-DRIVEN BASED RELIGHTING METHODS

  • Better image quality metrics result when RelitLRM jointly processes four views, unlike image-based baselines that relight each view independently.The held-out evaluation uses per-channel RGB scaling and compares PSNR, SSIM, and LPIPS.Source: §5.2 · p. 8, Figure 4 · p. 8, Table 3 · p. 9

5.3 OBJECT SCENE INSERTION

  • RelitLRM inserts objects into indoor and strong-sunlight scenes with realistic shadows and accurate specular highlights.The demonstrations include two objects in a living-room scene and eight objects under directional sunlight.Source: §5.3 · pp. 8–9

5.4 ABLATION STUDY

  • The probabilistic diffusion design improves visual quality for specular objects over a matched deterministic counterpart, despite similar quantitative metrics.Diffusion better models highly multi-modal radiance with sharp highlights, whereas deterministic relighting produces overly smooth results.Source: §5.4 · p. 9, Figure 5 · p. 9
  • Relighting quality increases with more input views, while classifier-free guidance makes outputs more specular and zero guidance resembles average dataset lighting.Performance saturates around 16 views, and denoising-view count is independent of input-view count.Source: §5.4 · pp. 9–10, Figure 6 · p. 10

5.5 LIMITATIONS

  • The method requires posed input images and uses environment maps that cannot accurately represent near-field lighting.Improving scalability to more views and higher resolutions remains future work.Source: §5.5 · p. 10

6 CONCLUSION

RelitLRM reconstructs relightable 3D objects from sparse images using deterministic geometry reconstruction and probabilistic diffusion-based appearance generation. It matches or exceeds strong baselines in 2–3 seconds rather than hours, supporting practical relightable reconstruction.

6 CONCLUSION

  • RelitLRM combines transformer-based geometry reconstruction with probabilistic relit-view diffusion to model complex lighting effects.The model disentangles geometry and appearance while capturing shadows and specularity.Source: §6 · p. 10
  • 2–3 seconds of processing and significantly fewer input images match or often exceed state-of-the-art baselines that require hours of per-scene optimization.The conclusion frames this efficiency as relevant to real-world relightable 3D reconstruction.Source: §6 · pp. 10–11
  • The model may be used to reconstruct 3D assets from images containing humans or commercial intellectual property.This possibility is noted in the ethics statement because the model attempts to preserve object identity.Source: §6 · p. 11

A.1 PSEUDO CODE

The pseudo code describes RelitLRM’s training pipeline from ray-augmented input images and environment maps to Gaussian geometry, radiance parameters, rendering, and loss computation.

A.1 PSEUDO CODE

  • Ray-augmented input images are patchified and encoded by a geometry transformer into geometry tokens.Environment maps are separately transformed into light tokens using dark- and bright-emphasizing representations plus ray directions.Source: §A.1 · p. 16
  • Noisy target relit views, their ray directions, input geometry tokens, light tokens, and timestep tokens are jointly processed by a denoising transformer.The transformer splits its output to obtain input radiance features.Source: §A.1 · p. 16
  • Decoded geometry and spherical-harmonic appearance parameters form 3D Gaussians that are rendered for target and supervision views to compute training loss.The Gaussian parameterization includes position, scaling, rotation, opacity, and spherical-harmonic weights.Source: §A.1 · p. 16

A.2 MODEL DETAILS

HDR environment maps are converted into complementary dark- and bright-region features before tokenization. Ray directions are appended to preserve lighting-direction information in the resulting light tokens.

A.2 MODEL DETAILS

  • Each HDR environment map is tonemapped into one dark-emphasizing feature map and one bright-emphasizing feature map.The two representations help the network ingest HDR lighting with different intensity ranges.Source: §A.2 · p. 17
  • The tonemapped features are concatenated with ray directions, producing nine-channel pixels that are patchified and passed through a two-layer MLP into light tokens.The bright representation uses the hybrid log-gamma transfer function.Source: §A.2 · p. 17

A.3 IMAGE AND LIGHTING PREPROCESSING

The preprocessing normalizes rendered images and environment-map lighting to address relighting scale ambiguity. It computes sine-weighted environment-map energy, rescales both lighting and images, then tonemaps the rendered images.

A.3 IMAGE AND LIGHTING PREPROCESSING

  • The pipeline normalizes environment maps and rendered images with a shared scale factor so their lighting energy matches a predefined constant.The constant is precomputed as the dataset median total energy.Source: §A.3 · p. 17
  • Environment-map energy is computed by weighting each pixel by the sine of its elevation angle before summation.The weighting accounts for the rectangular environment map’s elevation sampling.Source: §A.3 · p. 17
  • Rendered images are scaled, percentile-normalized using the 1st and 99th percentiles, clipped to [0, 1], and tonemapped with an exponent of 1/2.2.The scaling factor is computed as constant_energy / weighted_energy.Source: §A.3 · p. 17

A.4 TRAINING HYPERPARAMETERS

Training uses a two-stage schedule that increases resolution and input-token count during finetuning. Optimization uses AdamW with warmup, perceptual-loss scheduling, gradient clipping, and skipped steps for excessively large gradients.

A.4 TRAINING HYPERPARAMETERS

  • The model is first trained at 256×256 for 80K iterations with four input views, four denoising views, and two target-lighting supervision views.This stage uses batch size 512, AdamW, a peak learning rate of 4e −4, and weight decay 0.05.Source: §A.4 · p. 17
  • Finetuning raises resolution to 512 × 512, uses six input and six denoising views plus two supervision views, and increases the input-token count to 31K.The finetuning stage runs for 20K iterations with batch size 128 and a peak learning rate of 4e −5.Source: §A.4 · p. 17
  • Training introduces perceptual loss after 5000 iterations and applies warmup, gradient clipping at 1.0, and step skipping when gradient norms exceed 20.0.Warmup lasts 2000 iterations initially and 1000 steps during finetuning; β1 and β2 are 0.9 and 0.95.Source: §A.4 · p. 17

Related papers

Loading 2410.06231v2…