Source-linked AI summary
UniRelight: Learning Joint Decomposition and Synthesis for Video Relighting
Kai He, Ruofan Liang, Jacob Munkberg, Jon Hasselgren, Nandita Vijaykumar, Alexander Keller, Sanja Fidler, Igor Gilitschenski, Zan Gojcic, Zian Wang
TL;DR
Relighting requires scene understanding and light-transport synthesis, but scarce multi-illumination data and decoupled pipelines limit existing methods. UniRelight jointly predicts albedo and relit outputs with a video diffusion model, trained on synthetic and automatically labeled real-world data. It achieves realistic, temporally consistent relighting across diverse domains while modeling effects such as shadows, reflections, and transparency.
Problem
Relighting lacks affordable multi-illumination data, while two-stage pipelines can accumulate errors and miss complex material effects.
Method
UniRelight jointly denoises relighting and albedo representations in a single video-diffusion pass, conditioning relit generation on encoded lighting features.
Results
UniRelight achieves state-of-the-art, high-quality, realistic relighting with temporally consistent shadows, reflections, and transparency across diverse domains.
Takeaways & Limitations
Joint intrinsic decomposition and relighting supports generalization while reducing reliance on explicit G-buffers and accommodating complex visual effects.
Takeaways & Limitations
The environmental-lighting design cannot handle emitting objects or toggle lights within scenes, and it does not implement text-based relighting.
Abstract
from arXiv · showhide
We address the challenge of relighting a single image or video, a task that demands precise scene intrinsic understanding and high-quality light transport synthesis. Existing end-to-end relighting models are often limited by the scarcity of paired multi-illumination data, restricting their ability to generalize across diverse scenes. Conversely, two-stage pipelines that combine inverse and forward rendering can mitigate data requirements but are susceptible to error accumulation and often fail to produce realistic outputs under complex lighting conditions or with sophisticated materials. In this work, we introduce a general-purpose approach that jointly estimates albedo and synthesizes relit outputs in a single pass, harnessing the generative capabilities of video diffusion models. This joint formulation enhances implicit scene comprehension and facilitates the creation of realistic lighting effects and intricate material interactions, such as shadows, reflections, and transparency. Trained on synthetic multi-illumination data and extensive automatically labeled real-world videos, our model demonstrates strong generalization across diverse domains and surpasses previous methods in both visual fidelity and temporal consistency.
1 Introduction
UniRelight addresses scarce multi-illumination data and the limitations of decoupled relighting pipelines by jointly learning intrinsic decomposition and relighting in one generative model. Synthetic supervision plus automatically labeled real-world videos supports generalization and realistic effects across diverse scenes.
- Motivation: Multi-illumination data is costly to capture, while synthetic-only and narrow-domain approaches limit generalization to real-world scenes.Real-world acquisition requires controlled environments, calibrated equipment, and repeated captures.
- Motivation: Two-stage inverse-then-forward rendering avoids multi-illumination supervision but accumulates errors and misses complex properties such as transparency and subsurface scattering.The stages are inherently coupled, while G-buffers do not encode all material behavior.
- Approach: UniRelight jointly denoises relighting and albedo latents in one pass, implicitly learning scene structure instead of relying on explicit G-buffers.Albedo demodulation is intended to provide a strong relighting prior, including shadow removal.
- Training and outcome: The model combines fully supervised synthetic data with automatically labeled real-world single-illumination videos to improve generalization across unseen domains.The hybrid strategy is reported to support high-quality relighting and intrinsic decomposition from a single image or video.
- Training and outcome: The joint formulation produces temporally consistent shadows, reflections, and transparency while outperforming state-of-the-art methods.These outcomes are reported for the model trained with real-world auto-labeled data.
2 Related Work
Related work spans inverse rendering, scene relighting, diffusion-based methods, and joint generative modeling. Existing approaches often depend on domain-specific data, per-scene optimization, or intrinsic buffers, motivating UniRelight’s joint prediction of relit appearance and albedo.
- Inverse Rendering: Inverse-rendering methods estimate geometry, materials, and lighting but typically require large domain-specific datasets and struggle outside their training domains.Earlier optimization methods use hand-crafted priors, while newer supervised and self-supervised methods depend on learned data priors.
- Diffusion Relighting: RGB↔X and DiffusionRenderer [39] use diffusion models for intrinsic decomposition and neural rendering, with DiffusionRenderer extending the approach to video relighting.These methods are among the closest prior approaches to UniRelight.
- Relighting: Relighting methods modify image or video illumination, but many require per-scene optimization and remain limited to static, object-centric scenes under single illumination.Multi-scene methods explore latent features and neural rendering with intrinsic buffers as priors.
- Diffusion Relighting: Diffusion-based relighting methods show promise but are often specialized for portraits, single objects, or outdoor scenes and still require difficult multi-illumination datasets.The cited methods span recent diffusion approaches for relighting [53] [31] [26] [39] [72].
- Joint Generative Modeling: Joint generative models predict multiple modalities with one DiT, and UniRelight applies this paradigm to jointly predict relit images and albedo.Matrix3D jointly predicts pose, depth, and novel views, while VideoJAM [12] jointly predicts pixels and motion.
3 Preliminaries: Video Diffusion Models
Video diffusion models learn to generate videos by iteratively denoising Gaussian-corrupted latent representations. UniRelight fine-tunes a DiT video model with a pretrained VAE that compresses videos into a lower-dimensional latent space.
- Latent Video Diffusion: Video diffusion models encode an RGB video into a lower-dimensional latent tensor with a pretrained VAE and decode the final latent back into a video.The input contains L frames at resolution H × W, while the latent has reduced temporal and spatial dimensions.
- Model Backbone: UniRelight fine-tunes the Cosmos-Predict1 DiT video model and uses the Cosmos-1.0-Tokenizer-CV8x8x8 VAE for latent encoding and decoding.The tokenizer compresses videos by a factor of eight along spatial and temporal dimensions, with C = 16.
- Diffusion Training: Training constructs noisy latents as zτ = ατz0 + στϵ and optimizes the denoiser with a denoising score-matching objective.The noise schedule follows EDM, and iterative denoising from Gaussian noise produces samples from pdata(I).
4 Method
UniRelight jointly denoises relit video and albedo representations in one DiT-based video diffusion pass, using cross-modal attention and varied conditioning. It combines synthetic, multi-illumination, and auto-labeled real-world data to support relighting with complex lighting and materials.
- The model jointly predicts albedo and relit appearance from an input image or video under target lighting using a DiT-based video diffusion model.
- Concatenating albedo and relit latents enables single-pass denoising and self-attention across modalities, capturing shared scene structure for improved generalization and temporal consistency.
- The architecture encodes input, albedo, and relit videos with a VAE, concatenates their latents temporally, and uses type embeddings and condition masks to distinguish modalities and targets.
- Lighting is represented with LDR, normalized log-intensity, and directional buffers so HDR environment-map information can be encoded by the latent-diffusion VAE.
- Training combines fully supervised synthetic data, MIT multi-illumination samples, and automatically albedo-labeled real-world videos to improve coverage and generalization.
- The method targets complex lighting effects including inter-reflections and shadows, with qualitative comparisons spanning synthetic and MIT multi-illumination scenes.
5 Results
The evaluation shows that the proposed method produces higher-fidelity relighting across synthetic, benchmark, and in-the-wild scenes, while joint modeling and real-world auto-labeled data improve generalization and shadow handling.
- Qualitative comparison: The method yields more accurate and high-fidelity relighting across diverse scenes, especially for intricate shadows, inter-reflections, and complex materials.Baselines struggle with anisotropic surfaces, glass, transparency, fur, skin, car windows, and plastic materials represented through simple G-buffers.
- Quantitative comparison: On SyntheticScenes and the MIT multi-illumination benchmark, the method is compared quantitatively with DiLightNet, NeuralGaffer, and DiffusionRenderer variants using image metrics and user preference.The evaluation uses PSNR, SSIM, LPIPS, and a perceptual user study; Table 1 reports relighting metrics and preference results.
- Albedo estimation: The method performs competitively for albedo estimation, matching or exceeding prior work across metrics while the Cosmos DiffusionRenderer variant has a slight edge.Albedo quality is evaluated quantitatively on SyntheticScenes in Table 2.
- Ablation on joint modeling: Joint modeling slightly improves relighting quality over pure relighting and reduces false shadowing on out-of-distribution urban street scenes.Providing ground-truth albedo further enhances quality, while the ablated model often bakes input shadows into the relit result.
- Ablation on real-world auto-labeled data: Real-world auto-labeled data significantly improves generalization to out-of-domain images despite sparse labels.Without it, models show artifacts in out-of-focus regions and outdoor scenes because the synthetic and MIT datasets have limited depth of field and weak sky-like coverage.
- Application for data augmentation: The model can generate diverse nighttime and dusk relightings from one input scene, supporting data augmentation for driving and robotics scenarios.Five random illuminations are sampled without environment maps under different random seeds.
6 Discussion
UNIRELIGHT jointly models scene intrinsics and illumination to address data scarcity and limitations of decoupled relighting pipelines. The framework achieves realistic relighting while remaining limited to environmental lighting and excluding emitting objects.
- 6 Discussion: UNIRELIGHT jointly models scene intrinsics and illumination, improving generalization, reducing error accumulation, and capturing complex visual effects.The model implicitly reasons about scene properties rather than relying on explicit G-buffers.
- 6 Discussion: When trained on real-world auto-labeled data, UNIRELIGHT achieves state-of-the-art, high-quality, realistic relighting results.
- 6 Discussion: The method cannot handle emitting objects or toggling lights within scenes because it focuses on environmental lighting.Extending the design to emittance effects remains future work.
A Broader Impact
UniRelight jointly estimates albedo and synthesizes relit videos from a single input across synthetic and real-world scenes. Its applications include visual content creation and vision-data augmentation, while training-data bias and misuse remain risks.
- A Broader Impact: UniRelight jointly estimates albedo and synthesizes relit videos from a single input for diverse lighting manipulation across synthetic and real-world scenes.
- A Broader Impact: The capability can support creative content generation, visual effects, virtual production, and potentially data augmentation for robotics and autonomous driving.
- A Broader Impact: Training-data biases may limit generalization to underrepresented scenes or lighting conditions, while lighting manipulation may enable visual misrepresentation.The paper identifies surveillance and media as sensitive misuse contexts.
- A Broader Impact: The authors discourage uses that may contribute to misinformation, misattribution, or privacy violations and recommend fair dataset curation.They specifically mention representation across skin tones, races, and gender identities.
B.1 Implementation Details
The implementation fine-tunes a pretrained video diffusion model with concatenated latent representations and mixed synthetic, real-world, and multi-illumination training data. Evaluation uses standardized resizing, masking, and comparisons with IC-Light.
- B.1 Implementation Details: The model fine-tunes Cosmos-Predict1-7B-Video2World [47], a pretrained DiT video diffusion model.
- B.1 Implementation Details: Concatenated latents have 71 channels, combining latent features, binary condition masks, and type embeddings for lighting conditions.Each latent has a condition mask, and lighting features indicate whether they are provided.
- B.1 Implementation Details: Training mixes synthetic images and videos first, then combines synthetic, auto-labeled real-world, and MIT multi-illumination data.The first stage runs for 15,000 iterations and the second for 12,000 iterations.
- B.1 Implementation Details: Inference uses 35 denoising steps without classifier-free guidance because the authors found this more accurate and visually consistent.
B.2 User Study Details
The user studies measure perceptual relighting quality through pairwise comparisons against baselines or ablated variants. Participants judge similarity to ground truth or realism of shadows and reflections across MIT and StreetScenes scenes.
- B.2 User Study Details: On MIT multi-illumination, participants compare a ground-truth relit image with two results and select the one closer to ground truth.Judgments consider transparency, shadows, and reflections.
- B.2 User Study Details: The MIT study evaluates 30 scenes against four baselines, with three runs of 11 participants, totaling 3960 individual comparisons.
- B.2 User Study Details: The StreetScenes study uses 19 urban scenes without ground-truth relighting and asks participants to choose the video with more realistic shadows and reflections.Each comparison presents two relit videos from ablated method versions.
- B.2 User Study Details: The ablation study evaluates 19 scenes across three runs with 11 unique participants, producing 1254 individual comparisons.
C Additional Results
Additional evaluations show that UniRelight improves relighting quality across synthetic complex-material scenes and real scenes, while using a single joint pass that reduces runtime relative to two-stage baselines.
- 445.5 seconds are required for 35 denoising steps on a 57-frame 480 × 848 video using one A100 GPU.DiffusionRenderer requires five inverse-rendering passes plus one forward-rendering pass per video, whereas UniRelight performs joint relighting and albedo estimation in a single pass.
- Sharper specular highlights, more realistic shadows, and improved visual fidelity produce better relighting than IC-Light under novel lighting conditions.The comparison differs in conditioning: IC-Light uses background context, while UniRelight uses full HDR illumination.
- More accurate transparency, specular highlights, and shadows across MIT scenes outperform all baselines in visual quality.The comparison includes a re-implemented DiffusionRenderer with the Cosmos backbone, which has higher visual fidelity than the original implementation.
- High-quality albedo and relighting results with realistic specular highlights and shadows are achieved on real scenes under target lighting.