Source-linked AI summary
Neural Light Transport for Relighting and View Synthesis
Xiuming Zhang, Sean Fanello, Yun-Ta Tsai, Tiancheng Sun, Tianfan Xue, Rohit Pandey, Sergio Orts-Escolano, Philip Davidson, Christoph Rhemann, Paul Debevec, Jonathan T. Barron, Ravi Ramamoorthi, William T. Freeman
TL;DR
Recovering light transport from images is difficult because scene effects are intertwined. NLT embeds a neural model in a geometric texture atlas to fuse sparse observations, and its view-synthesis results show more realistic specularity and finer hair detail than NeRF in the reported comparison.
Problem
Recovering light transport from images requires disentangling intertwined effects such as occlusion, shading, interreflections, and scattering, and supports ground-truth generation for several learning tasks.
Source: §1 · p. 2Method
Results
NLT synthesizes more realistic facial and eye specularity and higher-frequency hair detail than NeRF in the reported view-synthesis comparison.
Source: §4.2 · p. 11Takeaways & Limitations
The framework supports photorealistic free-viewpoint rendering under controllable lighting for mixed reality, special effects, and machine-learning ground-truth generation.
Source: §1 · p. 2The fixed 1024×1024 texture-space resolution limits high-frequency synthesis, especially in close-up views or where a patch receives too few texels.
Source: §5 · p. 15
Abstract
from arXiv · showhide
The light transport (LT) of a scene describes how it appears under different lighting and viewing directions, and complete knowledge of a scene's LT enables the synthesis of novel views under arbitrary lighting. In this paper, we focus on image-based LT acquisition, primarily for human bodies within a light stage setup. We propose a semi-parametric approach to learn a neural representation of LT that is embedded in the space of a texture atlas of known geometric properties, and model all non-diffuse and global LT as residuals added to a physically-accurate diffuse base rendering. In particular, we show how to fuse previously seen observations of illuminants and views to synthesize a new image of the same scene under a desired lighting condition from a chosen viewpoint. This strategy allows the network to learn complex material effects (such as subsurface scattering) and global illumination, while guaranteeing the physical correctness of the diffuse LT (such as hard shadows). With this learned LT, one can relight the scene photorealistically with a directional light or an HDRI map, synthesize novel views with view-dependent effects, or do both simultaneously, all in a unified framework using a set of sparse, previously seen observations. Qualitative and quantitative experiments demonstrate that our neural LT (NLT) outperforms state-of-the-art solutions for relighting and view synthesis, without separate treatment for both problems that prior work requires.
1 INTRODUCTION
Recovering a scene’s light transport from images is difficult because image formation intertwines many effects, while existing capture systems sample only sparse camera and lighting directions. NLT uses a geometric texture-atlas model and neural residuals to learn from sparse observations and support simultaneous relighting and view synthesis.
1 INTRODUCTION
- Light-stage systems provide only sparse samples—about 300 LEDs and 50–100 cameras—so conventional image-based rendering cannot reliably synthesize unsupported lighting and novel views.Traditional approaches are usually designed for fixed viewpoints and cannot synthesize novel views under desired illumination.Source: §1 · p. 2
- NLT embeds a neural network in a classical mesh and diffuse texture-atlas model, using graphics-domain knowledge to learn complex transport from sparse observations.The neural model accounts for light-transport effects not captured by the classical model.Source: §1 · p. 2
- NLT combines relighting and view synthesis in one framework, learning the 6D light-transport function per subject from real observations.The approach uses convolutional neural networks and OLAT images in a parameterized texture atlas.Source: §1 · p. 2
- Augmented texture-space inputs and residual learning over a physically accurate diffuse base let the network model specular highlights, subsurface scattering, and global illumination.The diffuse base retains physically correct effects such as hard shadows.Source: §1 · p. 2
- The method targets photorealistic free-viewpoint rendering under controllable lighting, with applications in mixed reality, special effects, and machine-learning ground-truth generation.The paper also identifies high-quality relit images from multiple viewpoints as useful ground truth for several learning tasks.Source: §1 · p. 2
2 RELATED WORK
Prior approaches address parts of light-transport recovery, but sparse inputs, fixed viewpoints, restricted lighting, or simplified appearance limit their scope. NLT is positioned as a per-subject texture-space approach for jointly interpolating views and lighting.
Single observation
- Single-image methods infer scene properties from one observation, but do not show complex effects such as specular highlights or subsurface scattering and usually address only relighting or viewpoint change.Some also support only a limited range of viewpoint changes.Source: §2 · p. 2
Multiple views
- Multiview geometry and image-based rendering support view synthesis, but baked-in appearance prevents relighting, while reprojection cannot synthesize pixels absent from input views.Learned neural renderers often preserve view-dependent effects under fixed illumination but do not explicitly model light transport or enable relighting.Source: §2 · p. 3
Multiple illuminants
- Relighting methods from multiple illuminants generally do not support view synthesis; Nestmeyer et al. [2020] model shaded albedo and residuals rather than cast shadows in a physically accurate diffuse base.Xu et al. [2019] support free-viewpoint relighting but require a separate model as a second stage.Source: §2 · p. 3
Multiple views and illuminants
- Dense OLAT light-stage capture supports relighting by combining observations, but lacks an explicit geometric model for novel viewpoints; other systems either focus on faces or assume predefined BRDFs.Guo et al. [2019] support relighting and view synthesis but cannot synthesize more complex light-transport effects present in real images.Source: §2 · p. 3
Neural rendering
- Neural-rendering approaches are typically trained per subject and do not yet generalize across subjects, while NLT’s texture-space formulation supports simultaneous light and view interpolation.The formulation can also generate subject renderings for machine-learning tasks that require images under varied lighting and viewpoints.Source: §2 · p. 3
3 METHOD
NLT embeds a semi-parametric model in texture space to fuse observations and synthesize images under desired lighting and viewpoints. It adds learned residuals to a diffuse rendering, preserving its physical structure while modeling higher-order light transport.
3 METHOD
- NLT fuses recorded observations in texture space to close the gap between a geometry proxy’s diffuse rendering and the real image.It maps images into a shared UV atlas and warps the synthesized texture back to the requested camera view.Source: §3 · pp. 3–4
3.1 Hardware Setup and Data Acquisition
- 331 OLAT images were captured for 64 cameras, with approximately 130 usable images per camera because some were glare-polluted or overly dark.The dataset contains 70 fixed-pose subjects, each with about 18,000 frames across 331 lighting conditions and 55 viewpoints.Source: §3.1 · pp. 4–5
3.2 Texture-Space Inputs
- The texture-space inputs combine light and view cosine maps, a diffuse base, and residual maps computed from observed OLAT images.Visibility-masked light cosine maps encode rough cast shadows; the diffuse base averages views, while residual maps retain non-diffuse and non-local content.Source: §3.2 · p. 5
3.3 Query and Observation Networks
- Observation paths encode nearby residual maps and pool their features, which are concatenated with query-path features to predict the requested rendering.Training samples K=1 or 3 nearby observations; pooling makes the observation representation independent of their order and number.Source: Figure 4 · p. 6, §3.3 · p. 6
3.4 Residual Learning of High-Order Effects
- The network adds an unconstrained predicted residual to a diffuse base, letting it represent higher-order effects while retaining a physically accurate diffuse component.Positive residuals can add specularities, while negative residuals can represent shading or shadowing; the network can fall back to the base by emitting zeros.Source: §3.4 · p. 7, Figure 5 · p. 7
- NLT closely matches ground truth for unseen-light specular highlights and subsurface scattering, and predicts the green indirect glow cast by a wall.Subsurface scattering is an effect that a BRDF cannot capture; the global-illumination example uses a matte green wall.Source: §3.4 · pp. 7–8
3.5 Simultaneous Relighting and View Synthesis
- Because its design interpolates the 6D light transport function, the same framework performs relighting, view synthesis, or both simultaneously.Varying both lights and views in the training data supports rendering under a desired illumination from a desired camera position.Source: §3.5 · p. 8
3.6 Network Architecture, Losses, and Other Details
- Training minimizes image-space robust photometric and perceptual losses after resampling predictions from UV space to camera space.The losses are summed with equal weights; training observations are sampled from task-specific neighborhoods, and fixed test observations reduce flickering from neighborhood switching.Source: §3.6 · pp. 8–9
4 RESULTS
NLT performs relighting and view synthesis from sparse observations, with strong benchmark and visual results across both tasks. Its performance persists with degraded geometry, while ablations show the value of observation paths, diffuse-base residual learning, and perceptual loss.
4 RESULTS
- NLT outperforms relighting baselines and ablations in PSNR and LPIPS, and view-synthesis baselines and ablations in LPIPS.For view synthesis, UV-space barycentric blending and DNR are competitive in PSNR and SSIM, but DNR does not support relighting.Source: Table 3 · p. 9, Table 4 · p. 9, §4.1 · p. 9
4.1 Relighting
- NLT renders relighting shadows, specular highlights, and self-occlusions more precisely than simple blending and neural or geometric baselines.The visual comparison reports sharper, ghosting-free results that differ from nearest neighbors.Source: §4.1 · p. 9, Figure 7 · p. 10
4.2 View Synthesis
- NLT captures view-dependent specularities and fine details, outperforming methods that produce blurry results or omit non-Lambertian effects.Against NeRF, it renders more realistic facial and eye specularity and higher-frequency hair detail; NeRF + Light struggles with hard shadows and specular highlights.Source: §4.2 · pp. 10–11, Figure 9 · p. 12
- Embedding the network in the subject’s UV atlas enables simultaneous relighting and view synthesis with accurate shadows, global illumination, and specular detail.The paper reports that this requires interleaved training data for both tasks and a single network instance.Source: §4.2 · p. 10, Figure 10 · p. 13
4.3 Performance Analysis
- At 500 proxy vertices, NLT still produces reasonable views without missing pixels and outperforms Floating Textures in LPIPS across mesh resolutions.Floating Textures leaves pixels unfilled and struggles with high-frequency patterns, even at the original mesh resolution.Source: §4.3 · p. 13, Figure 12 · p. 14
- Training with 60% of the lights still yields reasonable relighting, but the results show ghosting shadows.The reduced setup corresponds to around 75 lights per camera.Source: §4.3 · p. 13, Figure 13 · p. 14
4.4 Ablation Studies
- Removing observation paths oversmooths results, while removing the diffuse base lowers quality and leaves the network to infer shadows and albedo.With residual learning, the model can focus on higher-order light transport effects.Source: §4.4 · p. 14
- Perceptual loss helps preserve high-frequency details such as hard shadow boundaries, and full NLT achieves the best perceptual scores in both tasks.Removing this loss blurs the shadow boundary.Source: §4.4 · p. 14, Figure 14 · p. 14
5 LIMITATIONS
NLT is trained per scene, and its fixed texture-space resolution limits synthesis of high-frequency content. Complex light-transport effects can also be difficult to synthesize, while dynamic performers remain a prospective application.
5 LIMITATIONS
- NLT must be trained individually per scene; generalization to unseen scenes remains an important future step.Source: §5 · p. 14
- Dynamic-scene rendering is desirable, and a fixed texture atlas may directly enable NLT to work for dynamic performers.Source: §5 · p. 15
- The fixed 1024×1024 texture-space resolution limits high-frequency content, especially in close-ups or patches allocated too few texels.Higher-resolution training could address this, but would increase memory requirements and likely require significant engineering effort.Source: §5 · p. 15
- Complex light-transport effects can be hard to synthesize, with occasional failures lacking high-frequency detail, as on the glittery chain.Source: §5 · p. 15, Figure 15 · p. 15
6 CONCLUSION
NLT learns light transport for full-body human scans from light-stage OLAT images using a model embedded in mesh-and-atlas UV space. The authors report improvement over specialized relighting or view-synthesis solutions while also enabling both tasks together.
6 CONCLUSION
- NLT is a semi-parametric framework for simultaneous relighting and view synthesis of full-body human scans using light-stage OLAT images.Source: §6 · p. 15
- A UV-space dual-path network aggregates observations and synthesizes texture-space RGB images, with augmented inputs and residual learning focused on higher-order transport effects.Source: §6 · p. 15
- Experiments show clear improvement over previous specialized relighting or view-synthesis solutions, while enabling simultaneous relighting and view synthesis.Source: §6 · p. 15