Source-linked AI summary
Single Image Portrait Relighting
Tiancheng Sun, Jonathan T. Barron, Yun-Ta Tsai, Zexiang Xu, Xueming Yu, Graham Fyffe, Christoph Rhemann, Jay Busch, Paul Debevec, Ravi Ramamoorthi
TL;DR
Single-image portrait relighting is difficult because prior methods require controlled captures, specialized equipment, or explicit geometry and reflectance models. The paper uses a neural network trained with real light-stage data to relight one unconstrained cellphone portrait under an arbitrary environment map. It reports superior validation results, convincing results on hundreds of real-world portraits, and 160-millisecond processing for a 640 × 640 image.
Problem
Existing portrait-relighting methods are limited by requirements for specialized hardware, multiple controlled images, or accurate geometry and reflectance models, restricting unconstrained cellphone use.
Method
A single deep neural network uses real light-stage captures to predict the input illumination and generate a portrait relit under a user-specified environment map.
Results
The technique outperforms state-of-the-art relighting methods on its validation set and performs well on hundreds of real-world cellphone portraits.
Takeaways & Limitations
The method's quality, generality, and 160-millisecond runtime for a 640 × 640 image may enable consumer-facing photographic relighting applications.
Takeaways & Limitations
The model generalizes poorly to hard shadows, sharp specularities, or saturated pixels because these phenomena are underrepresented in its training data.
Abstract
from arXiv · showhide
Lighting plays a central role in conveying the essence and depth of the subject in a portrait photograph. Professional photographers will carefully control the lighting in their studio to manipulate the appearance of their subject, while consumer photographers are usually constrained to the illumination of their environment. Though prior works have explored techniques for relighting an image, their utility is usually limited due to requirements of specialized hardware, multiple images of the subject under controlled or known illuminations, or accurate models of geometry and reflectance. To this end, we present a system for portrait relighting: a neural network that takes as input a single RGB image of a portrait taken with a standard cellphone camera in an unconstrained environment, and from that image produces a relit image of that subject as though it were illuminated according to any provided environment map. Our method is trained on a small database of 18 individuals captured under different directional light sources in a controlled light stage setup consisting of a densely sampled sphere of lights. Our proposed technique produces quantitatively superior results on our dataset's validation set compared to prior works, and produces convincing qualitative relighting results on a dataset of hundreds of real-world cellphone portraits. Because our technique can produce a 640 $\times$ 640 image in only 160 milliseconds, it may enable interactive user-facing photographic applications in the future.
1 INTRODUCTION
The paper presents single-image portrait relighting for unconstrained cellphone photographs, using a neural network to produce portraits under user-specified illumination. It targets consumer photography through learned appearance modeling, real light-stage training data, and potential interactive speed.
- Approach: The system uses an encoder-decoder architecture that injects target illumination at the bottleneck and can recover, modify, and decode the input illumination.This supports operations such as rotating the original illumination.
- Motivation: Consumer photographers often cannot control portrait lighting and may lack the expertise or equipment needed for professional-quality illumination.Casual cellphone photography is commonly constrained by available environmental light or a built-in flash.
- Approach: The model takes one casually captured RGB portrait and produces a relit version under an arbitrary user-specified environment map.It also predicts the environment map corresponding to the input image.
- Approach: Unlike inverse-rendering approaches, the method directly predicts the final relit image without explicitly estimating geometry and reflectance.The representation is learned through internal network activations rather than constrained by an explicit physical scene model.
- Evaluation: Real light-stage data from 22 individuals, including 18 for training and 4 for validation, supports the proposed architecture and preserves non-Lambertian facial reflectance.The setup captures subjects across 7 viewpoints and 304 lighting directions.
2 RELATED WORK
Prior portrait-relighting approaches rely on multiple controlled images, inferred scene models, reference subjects, or constrained face models. The paper positions its method against these limitations while using real light-stage data for training and evaluation.
- Image-Based Relighting: Image-based relighting combines many images captured from one viewpoint under known illuminations, but requires hundreds of images, a motionless subject, and precise illumination control.These requirements prevent it from handling a single RGB image of an unknown subject in an unconstrained environment.
- Intrinsic Images and Shape from Shading: Intrinsic-image and shape-from-shading methods separate shading and reflectance or recover geometry, but do not provide general-purpose relighting under novel illuminations.Shape-from-shading additionally assumes materials and illumination are known or fixed.
- Monocular Scene Understanding: Monocular scene-understanding methods can recover geometry for adding lights, but do not provide a means to remove existing lights and relight the image.Thus, geometry recovery alone does not solve the relighting problem.
- Reflectance and Style Transfer: Reference-face transfer methods can move high-frequency lighting changes between portraits but require manually selected similar subjects and reference lighting data.Transferred effects include changes in hair and specular reflections.
- Face Reconstruction and Relighting: Morphable-face models support relighting in some settings but perform poorly on non-generic faces, expressions, and portrait content such as glasses, shoulders, or hair.A related single-image lighting-transfer method does not allow general-purpose relighting with any environment map.
- Photographic Post-Processing and Deep Learning: The paper relates relighting to photographic post-processing and deep learning, including brightness, exposure, tone, local tone mapping, and convolutional neural networks.These connections frame relighting as an extension of existing photometric manipulation and learned image processing.
3 LEARNING RELIGHTING FOR PORTRAITS
The paper formulates portrait relighting as learning a function that maps a single source portrait and target illumination to a relit portrait while also estimating the source lighting. It trains this model on real light-stage captures synthesized under many environment illuminations, using an encoder-decoder network and multiple losses.
- Problem formulation: The task assumes a roughly front-facing portrait within ±20 degrees and distant lights, while seeking both target relighting and source-illumination prediction.Given source image Is and target lighting Lt, the learned function predicts source illumination and a target portrait.
- Data: The training data use real light-stage captures because synthetic Lambertian faces poorly reproduce real skin reflectance and fine facial geometry.The setup captures one-light-at-a-time images from 304 LEDs with synchronized cameras, using optical flow from tracking frames to compensate for subject motion.
- Network architecture: PR-Net uses an encoder-decoder with skip connections: the encoder estimates source illumination, while target lighting is encoded at the bottleneck and decoded into the target portrait.The decoder progressively upsamples to the input resolution and concatenates activations from corresponding encoder layers.
- Loss and training: Training minimizes a weighted combination of target-image error, source-light error, and self-supervision loss.The self-supervision variant feeds a rotated predicted source illumination back as the decoder’s target lighting.
4 EVALUATION
The evaluation measures relit-image quality, estimated illumination quality, and runtime against prior methods and model ablations. The proposed model outperforms prior work on target relighting metrics, produces realistic qualitative results, and benefits from illumination supervision, self-supervision, and confidence learning.
- Evaluation setup: The evaluation uses relit-image and illumination-prediction tasks, with RMSE, scale-invariant RMSE, and DSSIM for relighting quality.Source metrics assess reconstruction of the input image, while target metrics assess prediction of desired relit images.
- Relit Images: Our model was ∼4.6× faster than SfSNet and ∼4800× faster than SIRFS while producing realistic relighting results across challenging scenes.Figure 5 includes back-lit-to-front-lit relighting, cast shadows, and subsurface-scattering or translucency from backlighting.
- Relit Images: The model outperformed two lighting-transfer baselines on all error metrics with randomized references and on two of three metrics with oracle references.Randomized references depict different subjects under the target illumination; oracle references provide the ground-truth target image.
- Ablation study: Removing illumination supervision or self-supervision reduced target-metric performance, showing that both provide useful cues for relighting.Self-supervision also improves source reconstruction and helps disentangle skin color from illumination color.
- Estimated Lighting: The complete model outperformed prior lighting-estimation methods and the no-confidence-learning ablation by a large amount.The authors attribute confidence learning’s benefit to reasoning about parts of the lighting within image patches and reducing speckling behind the subject.
5 REAL-WORLD RESULTS
On real-world cellphone portraits, the model supports complete relighting, illumination retargeting, and lighting transfer across varied subjects and viewing conditions, while showing specific failures on inputs outside its training distribution.
- Complete relighting: The model generalizes well to varied “in the wild” portraits despite the small number of training subjects.Evaluation uses Portrait-mode and HDR+ cellphone images, including variation in skin tones, ages, and viewing conditions.
- Complete relighting: The model produces plausible relighting for portraits containing previously unseen objects and accessories, including food, hats, and sunglasses.These objects are relit and recolored similarly to skin or clothing.
- Illumination retargeting: Illumination retargeting recovers the input lighting, allows manual rotation, and renders the portrait under the modified illumination.The relit foreground can be composited back onto the input background using the subject mask.
- Light transfer: The model also supports lighting transfer by predicting illumination from one portrait and using it to relight another, with reasonable results despite differing hair colors or expressions.This capability was not the model’s original design target.
- Complete relighting: Using an RGB image and portrait mask, the model produces relit images under arbitrary environment illuminations for diverse subjects and viewing angles.Complete relighting replaces the scene illumination and composites the relit foreground over a synthetic target-illumination background.
- Limitations: Inputs with hard shadows, sharp specularities, saturated pixels, or saturated-color accessories can produce artifacts because these cases are underrepresented in training.The limitation is attributed to poor generalization beyond the training distribution.
6 CONCLUSION
The paper presents a learning-based system for relighting a portrait from one unconstrained RGB image and demonstrates strong practical performance, generality, and speed alongside known robustness limits.
- 6 CONCLUSION: The proposed technique relights a single RGB portrait captured in an unconstrained environment as though illuminated by a different environment.Training combines a small directional-light light-stage database with natural environment maps to generate realistic training data.
- 6 CONCLUSION: The model predicts the original illumination and a relit output under a new target environment, using a neural network architecture with self-supervision.The conclusion identifies self-supervision as valuable to the approach.
- 6 CONCLUSION: The technique outperforms state-of-the-art relighting methods on the validation set and performs well on hundreds of real-world cellphone portraits.The reported real-world evaluation uses unconstrained portrait photographs.
- 6 CONCLUSION: A 640 × 640 image is produced in 160 milliseconds, suggesting potential for consumer-facing photographic relighting applications.The paper also suggests possible use for synthesizing training data for facial recognition or 3D reconstruction.
- 6 CONCLUSION: Additional training data may improve robustness to current limitations such as very hard shadows and sharp specularities.These phenomena are identified as current limitations of the model.