A Diffusion Approach to Radiance Field Relighting using Multi-Illumination Synthesis

• Single-lighting captures make relighting ill-posed because lighting and material properties are entangled.Source: §1 · p. 1
• Existing alternatives require heavy multi-illumination capture, synthetic priors, restricted objects, simplifying lighting assumptions, or accurate geometry and normals.Source: §1 · pp. 1–2
• Diffusion models provide rich image-distribution priors and complex lighting synthesis, but lighting-specific control for full 3D scenes had not been demonstrated.Source: §1 · p. 2
• The method fine-tunes a direction-conditioned diffusion model, augments single-lighting multi-view images, and trains a 3D Gaussian relightable field with an MLP and auxiliary feature vectors.Source: §1 · p. 2
• The resulting system provides direct lighting control and corrects generated-lighting inconsistencies for interactive relighting.Source: §1 · p. 2
• On synthetic and real indoor scenes, the method produces realistic single-capture relighting in real time.Source: §1 · p. 2

2.1. Radiance Fields

• Radiance fields synthesize novel views from multi-view images, while 3D Gaussian Splatting enables detailed real-time rendering; single-lighting captures lack direct lighting control.Source: §2.1 · p. 2

2.2. Single Image Relighting

• Single-image relighting has mainly addressed faces and restricted effects, whereas generic-scene relighting is substantially harder because geometry is less constrained.Source: §2.2 · p. 2
• Indoor-scene relighting differs from approaches focused on hard cast shadows, outdoor environments, or isolated objects with environment maps.Source: §2.2 · p. 3

2.3. Multi-view Relighting

• Multi-view methods improve geometry but often depend on MVS quality, complex texture-space pipelines, or rough proxies and neural textures.Source: §2.3 · p. 3
• Relightable radiance-field methods commonly target isolated objects under distant lighting and struggle with complex scene-scale geometry and near-field illumination.Source: §2.3 · p. 3
• The paper simulates multi-illumination data with diffusion models to reduce the capture constraints of directly collecting varied lighting.Source: §2.3 · p. 3

2.4. Diffusion Models

• Diffusion models learn denoising-based image generation, while latent-space and accelerated sampling methods make high-dimensional synthesis more practical.Source: §2.4 · p. 3
• ControlNet enables efficient conditioning of Stable Diffusion, but realistic fine-tuned relighting of full cluttered scenes had not been established.Source: §2.4 · p. 3

3. Method

• The pipeline combines 2D relighting, synthetic multi-illumination augmentation, and a consistency-aware 3D Gaussian radiance field for interactive scene relighting.The radiance field uses direct lighting control and auxiliary per-image vectors to address inaccuracies in synthesized relit views.Source: §3 · p. 4, §3.3 · p. 7

3.1. Single-View Relighting with 2D Diffusion Priors

• Single-light captures are underconstrained because lighting and material properties are entangled, motivating priors for predicting appearance under new illumination.Existing approaches often require multi-illumination capture or restrictive assumptions, while lighting-specific diffusion priors for full 3D scenes had not been demonstrated.Source: §3.1 · p. 4

3.1.2. Controlling Relighting Diffusion

• The method fine-tunes Stable Diffusion with ControlNet on 1,015 scenes and 18 non-front-facing light directions, conditioning relighting on an explicitly specified direction.Training uses paired images of the same scene under different lighting directions, with depth and image conditioning but no text conditioning.Source: §3.1 · p. 4, §3.1.2 · pp. 4–5

3.1.1. Lighting Direction

• Lighting directions are estimated by fitting a gray sphere with Phong shading to averaged diffuse-sphere targets while minimizing L1 error.The optimization jointly estimates ambient light, albedo, specular intensity and hardness, and a Fresnel coefficient.Source: §3.1.1 · p. 4, Figure 3 · p. 4

3.1.3. Improving the Diffusion Quality

• Color matching and a v-parameterized objective correct naive ControlNet relighting errors, while decoder fine-tuning restores sharper edges and legible text.Naive adaptation produces inaccurate colors, poor contrast, and distorted edges that degrade multi-view consistency.Source: §3.1.3 · p. 5, Figure 4 · p. 5, Figure 5 · p. 5

3.2. Augmenting Multi-View/Single-Lighting Datasets

• Each single-illumination multi-view image is relit into 18 target directions, producing an approximate multi-view, multi-illumination dataset for radiance-field training.Known camera poses register the synthesized views, and forward-facing flash directions are excluded.Source: §3.2 · p. 5, Figure 7 · p. 6

3.3. Training a Lighting-Consistent Radiance Field

• The 3D Gaussian radiance field replaces spherical-harmonic appearance with a light- and view-conditioned MLP, then adds per-image auxiliary vectors to absorb inconsistent synthesized lighting.Globally registered light directions provide direct control, while auxiliary vectors help fit each view and reduce artifacts such as floaters and holes.Source: §3.3 · pp. 5–6

4.1. Test Datasets

• Synthetic evaluation uses four cluttered Blender scenes, simpler diffuse-only variants, multiview training and testing sweeps, and approximate ground-truth lighting.The evaluation renders 200 novel views under 18 lighting directions and computes standard image-quality metrics.Source: §4.1 · p. 7, §4.3 · p. 10
• Real-scene experiments cover five captured indoor scenes plus COUNTER and ROOM from the MipNeRF360 dataset.The captured scenes include KETTLE, HOT PLATES, PAINT GUN, CHEST OF DRAWERS, and GARAGE WALL.Source: §4.1 · p. 7, §4.2 · p. 7

4.2. 3D Relighting Results

• The relightable radiance field produces realistic and temporally consistent results while supporting interactive novel-view synthesis and relighting at 30fps on an A6000 GPU.The implementation adds a small MLP over the core 3DGS architecture, with memory usage comparable to original 3DGS.Source: §4.2 · p. 7

4.3. Evaluation

• On the real KETTLE scene, our method shows fewer geometry and shading artifacts than Philip et al. [PMGD21], whose imprecise MVS preprocessing causes missing geometry.Real-scene qualitative results also show changes in highlights and shadows as the light source moves.Source: Figure 10 · p. 8, Figure 9 · p. 8
• Our method outperforms the compared methods in nearly all synthetic scenarios across PSNR, SSIM, and LPIPS, remaining competitive in the exceptions.Table 1 reports per-scene quantitative results against OutCast, Relightable 3D Gaussians, and TensoIR.Source: Table 1 · p. 8, §4.3 · p. 10
• Qualitative synthetic comparisons show results closer to Blender-rendered ground truth than OutCast, Relightable 3D Gaussians, and TensoIR.The baselines exhibit incorrect shadows, geometry-reconstruction difficulty, or splotchy appearance associated with inaccurate normals.Source: Figure 11 · p. 9, §4.3 · p. 10
• The method is trained for indoor scenes and generalizes to some unseen scenes and lighting conditions, but with lower realism than on in-distribution samples.This establishes an explicit scope boundary for the reported out-of-distribution behavior.Source: §4.3 · p. 10

• It leverages a general-purpose single-view, multi-illumination dataset and fine-tunes a large pretrained generative model instead of relying on accurate geometry, material, or lighting estimation.Source: §5 · p. 10
• The method synthesizes realistic relighting for captured scenes while enabling interactive novel-view synthesis and surpassing state of the art for cluttered indoor scenes.Source: §5 · p. 10
• The method can hallucinate shadows from unseen geometry and misplace highlights because its target light direction is noisy and not fully physically accurate.Source: §5 · p. 10, Figure 12 · p. 10
• Future work should enforce multi-view consistency and use radiance-field geometry more explicitly to guide accurate relighting and support complex lighting changes.Source: §5 · p. 10