Source-linked AI summary
Relit-LiVE: Relight Video by Jointly Learning Environment Video
Weiqing Xiao, Hong Li, Xiuyu Yang, Houyuan Chen, Wenyi Li, Tianqi Liu, Shaocong Xu, Chongjie Ye, Hao Zhao, Beibei Wang
TL;DR
Video relighting remains difficult because imperfect intrinsic decomposition can produce physically inconsistent and temporally unstable results, especially without known camera pose. Relit-LiVE fuses raw RGB references with intrinsic representations and jointly generates relit and environment videos, outperforming state-of-the-art methods in physically consistent, temporally stable relighting.
Problem
Video relighting lacks reliable physical and temporal consistency, while intrinsic-decomposition pipelines remain limited by decomposition accuracy in complex lighting.
Method
Relit-LiVE fuses raw RGB references with intrinsic representations and jointly generates relit videos and viewpoint-aligned environment videos without camera-pose priors.
Results
Relit-LiVE outperforms state-of-the-art methods in physically consistent, temporally stable video relighting across complex lighting and viewpoint changes.
Takeaways & Limitations
The framework supports camera-free relighting with spatially consistent results and extensions to scene-level rendering, editing, and streaming video relighting.
Takeaways & Limitations
Relit-LiVE trades off resolution and frame rate, reaching at most 57 frames at 832 × 480 during training, with 57-frame generation taking about 10 minutes on an A800 GPU.
Abstract
from arXiv · showhide
Recent advances have shown that large-scale video diffusion models can be repurposed as neural renderers by first decomposing videos into intrinsic scene representations and then performing forward rendering under novel illumination. While promising, this paradigm fundamentally relies on accurate intrinsic decomposition, which remains highly unreliable for real-world videos and often leads to distorted appearances, broken materials, and accumulated temporal artifacts during relighting. In this work, we present Relit-LiVE, a novel video relighting framework that produces physically consistent, temporally stable results without requiring prior knowledge of camera pose. Our key insight is to explicitly introduce raw reference images into the rendering process, enabling the model to recover critical scene cues that are inevitably lost or corrupted in intrinsic representations. Furthermore, we propose a novel environment video prediction formulation that simultaneously generates relit videos and per-frame environment maps aligned with each camera viewpoint in a single diffusion process. This joint prediction enforces strong geometric-illumination alignment and naturally supports dynamic lighting and camera motion, significantly improving physical consistency in video relighting while easing the requirement of known per-frame camera pose. Extensive experiments demonstrate that Relit-LiVE consistently outperforms state-of-the-art video relighting and neural rendering methods across synthetic and real-world benchmarks. Beyond relighting, our framework naturally supports a wide range of downstream applications, including scene-level rendering, material editing, object insertion, and streaming video relighting. The Project is available at https://github.com/zhuxing0/Relit-LiVE.
1 Introduction
Relit-LiVE addresses physically consistent, temporally stable video relighting without requiring prior camera pose. It combines RGB and intrinsic information with jointly generated relit and environment videos, and supports robust training and downstream extensions.
- Motivation: Video relighting must modify illumination while preserving intrinsic scene properties, yet realistic reflections, stable shadows, material variation, and temporal accuracy remain challenging.The introduction identifies physical consistency and temporal accuracy under complex light transport as central difficulties.
- Limitations of prior work: Existing diffusion-based relighting methods improve visual quality but provide imprecise lighting control, retain original-illumination artifacts, or require known camera poses.Pose requirements constrain environment-map placement and method flexibility.
- Relit-LiVE: Relit-LiVE produces physically consistent, temporally stable relit videos without prior camera pose by addressing scene-content preservation and flexible novel-light injection.The framework is designed around two core challenges: preserving content integrity under complex light transport and injecting novel lighting without known pose.
- Training enhancements: Latent-space interpolation synthesizes diverse multi-illumination data, and cycle-consistent self-supervised illumination learning promotes temporal lighting coherence without additional annotations.These training enhancements improve robustness in complex scenarios.
- Results and applications: Experiments show realistic material reflections and viewpoint-change modeling, enabling physically plausible spatio-temporally accurate relighting without camera-pose priors and extensions to scene rendering, editing, and streaming.The paper also states that task extensions can be enabled by modifying generation conditions.
- Key contributions: An RGB-intrinsic fusion renderer combines real-world lighting effects from RGB space with physical constraints from intrinsic space, while joint relit-video and environment-video generation aligns geometry and illumination.These are presented as key contributions of Relit-LiVE.
2 Related work
Related work follows two main paradigms: direct video relighting, which increasingly targets precise lighting control and physical realism, and diffusion-based intrinsic decomposition and synthesis, which offers flexible scene manipulation. Early direct methods emphasize temporal consistency, while Relit-LiVE contrasts its decoupled design with joint estimation to improve intrinsic-content comprehensiveness and architectural flexibility.
- Direct video relighting: Direct video relighting adjusts illumination while preserving scene content through end-to-end modeling, with research shifting toward precise lighting control and physical realism.This development is enabled by controllable video diffusion technology.
- Direct video relighting: Early methods such as Light-A-Video and TC-Light extend image relighting effects across videos using text prompts or reference backgrounds and temporal-consistency enhancement schemes.These approaches build on the image re-illumination technique IC-Light.
- Intrinsic decomposition and synthesis: PBR-inspired diffusion pipelines decompose and synthesize intrinsic scene components, offering greater flexibility than end-to-end generation for functions such as relighting and related editing.Their flexibility comes from adjusting intrinsic components.
- Intrinsic decomposition and synthesis: Intrinsic decomposition methods fine-tune pretrained diffusion models and use conditional generation to estimate specific intrinsic properties.Representative methods include IntrinsiX, NormalCrafter, and GeometryCrafter.
- Intrinsic decomposition and synthesis: Other approaches jointly perform intrinsic decomposition and synthesis to form closed-loop capabilities, while Relit-LiVE argues that decoupling improves intrinsic-content comprehensiveness and architectural flexibility.The cited RGBX example supports bidirectional estimation and rendering between images and G-buffers.
3 Our method
Relit-LiVE jointly generates relit videos and camera-aligned environment videos without requiring known camera poses. It combines RGB references with intrinsic scene properties to improve rendering under imperfect decomposition and diverse lighting conditions.
- RGB-Intrinsic fusion renderer: Relit-LiVE uses an RGB-Intrinsic fusion renderer to combine observable RGB cues with G-buffer constraints, bypassing imperfect intrinsic decomposition.The renderer uses raw reference images alongside base color, surface normal, relative depth, roughness, and metallic properties.
- RGB-Intrinsic fusion renderer: Randomly sampled raw reference images guide generation while breaking fixed correspondences that could propagate source lighting at the pixel level.Different reference frames can be sampled during diffusion denoising, helping suppress residual source illumination.
- Joint environment video generation: The method learns warped environment maps as environment videos, replacing the common assumption that camera poses are known or accurately estimated.Environment maps are aligned with each camera viewpoint through joint prediction rather than direct pose-based warping.
- Joint environment video generation: A single diffusion process jointly generates the relit video and corresponding normalized log-intensity and LDR environment maps.Both outputs are encoded into latent space, independently noised, and denoised together with reference, intrinsic, and lighting conditions.
- Training strategy: Training proceeds in three stages, including an intrinsic perception enhancement strategy that creates multi-illumination references balancing realistic appearance against lighting plausibility.Interpolated outputs are decoded and reused as raw references to train under diverse lighting conditions on real-world scenes.
4 Results
Relit-LiVE outperforms competing relighting methods across synthetic, MIT, and in-the-wild evaluations, producing stronger material fidelity, physical consistency, and temporal stability. Ablations and additional applications show that raw references and joint environment-video generation are central to these gains, while long-video processing remains computationally costly.
- Qualitative comparison: Compared with existing and text prompt-based methods, Relit-LiVE better preserves materials and avoids distorted or physically unreasonable lighting effects.NeuralGaffer retains scene shadows and highlights, Diffusion Renderer distorts materials, and text-prompt methods struggle especially under neon lighting.
- Long-video relighting: Relit-LiVE maintains temporally consistent lighting in long videos by perceiving camera-viewpoint changes and warping environment maps.The method processes long videos clip by clip, using the generated environment video to initialize lighting for the next clip.
- Quantitative comparison: Relit-LiVE surpasses baselines across all relighting metrics on the synthetic and MIT multi-illumination datasets.Table 1 reports quantitative comparisons using visual-fidelity and temporal-consistency measures.
- Ablation studies: Raw reference images significantly improve relighting and restore realistic transmission effects that intrinsic decomposition alone fails to reproduce.The ablation particularly shows improvements for transparent objects such as plastic bags and glass bottles.
- Ablation studies: Joint environment-video generation significantly improves camera-free relighting under substantial camera motion or dynamic lighting.The joint model aligns relighting with generated environment videos, improving performance over the ablated model on synthetic videos.
- Limitations: Training reaches a maximum of 57 frames at 832 × 480 resolution, while generating a 57-frame video takes approximately 10 minutes on an A800 GPU.The frame-concatenation control method imposes substantial training and inference costs, requiring a resolution–frame-rate trade-off.
5 Conclusion
Relit-LiVE is a video relighting framework that produces physically consistent, temporally stable results without prior camera-pose knowledge. It combines RGB-intrinsic fusion rendering with joint relit-video and environment-video generation to incorporate real-world lighting while respecting estimated physical constraints.
- Core framework: Relit-LiVE produces physically consistent and temporally stable video relighting without requiring prior camera-pose knowledge.This improves the framework’s practical flexibility by eliminating dependence on explicit pose estimation.
- Core framework: An RGB-intrinsic fusion renderer forms the framework’s core rendering component.The renderer is paired with joint generation of the relit video and environment video.
- Joint generation: Jointly generating relit and environment videos incorporates real-world lighting effects while adhering to estimated physical constraints.The formulation results in realistic relighting outcomes.
- Training strategies: Two complementary training strategies mitigate the scarcity of existing multi-light datasets.The supplied passage introduces these strategies as a response to limited training data.
A Appendix · A.1 Data generation strategy
The appendix describes a training-data strategy combining substantial synthetic data with auto-labeled real-world datasets. Synthetic videos are rendered with comprehensive intrinsic scene annotations and draw on public PBR material and HDR environment resources.
- A.1 Data generation strategy: Training data combines a substantial synthetic dataset with auto-labeled real-world datasets.
- A.1 Data generation strategy: The synthetic-data workflow produces a large number of rendered videos.
- A.1 Data generation strategy: Each rendered video includes base colors, roughness, metallicness, normal maps, depth maps, environment maps, and camera trajectories.
- A.1 Data generation strategy: The workflow begins by collecting 5700 high-quality PBR material maps from public resources.
- A.1 Data generation strategy: It also collects 2241 HDR environment maps from public resources.
- A.1 Data generation strategy: The resulting synthetic dataset is designed to provide both rendered video observations and scene-level factors for training.
A.2 Initial training
Initial training uses a staged schedule that begins with synthetic data for foundational rendering, then freezes cross-attention while training on the full dataset to improve generalization across lighting conditions. The model also simulates pure rendering by zeroing zI with probability 0.3 and supports real-world scenes with single lighting conditions.
- Training schedule: Training begins exclusively on synthetic data to learn foundational rendering, then freezes cross-attention and uses the full dataset to improve generalization across lighting conditions.The schedule addresses illumination-distribution differences between synthetic and real-world datasets while preserving adaptability to varying lighting.
- Training schedule: The latent zI is set to zero with probability 0.3 to simulate a pure rendering task.This procedure follows UniRelight [He et al. 2025b].
- Real-world training: For real-world datasets with and without environment maps, the denoising targets enable training on scenes with single lighting conditions.The targets are ˆzs(𝜃) and ˆzElog (𝜃) = f𝜃([zI, zs𝜏, z{a,d,m}, z{n,r}+0]; 0,𝜏), respectively.
A.3 Additional Details on Intrinsic Perception Enhancement
Intrinsic Perception Enhancement generates multi-illumination data to support Relit-LiVE’s relighting and rendering flexibility. The strategy uses generated data as pseudo-original references because existing open-source relighting models do not reliably produce physically realistic lighting effects.
- Multi-illumination data generation: The model supports both relighting with zI and rendering without zI, differing only in whether the raw reference image is provided during inference.This distinction motivates multi-illumination data generation for Intrinsic Perception Enhancement.
- Motivation: Generated multi-illumination data serves as pseudo-original reference images and therefore must exhibit accurate, physically realistic lighting effects.The data is intended to provide reliable reference information for the enhancement strategy.
- Motivation: Existing open-source relighting models are unsuitable for generating this data because they fail to produce realistic relit results that adhere to material properties.Section 4.1 reportedly demonstrates this limitation even for advanced open-source methods.
A.4 Experimental details
Relit-LiVE is fine-tuned from Wan2.1-T2V-1.3B through a three-stage training procedure using synthetic, full, and pseudo-realistic data, followed by SIC training. Experiments use diverse real-world videos and assess motion preservation, material consistency, realism, physical consistency, and lighting alignment.
- Training Details: The model is fine-tuned from Wan2.1-T2V-1.3B in three stages: synthetic pretraining, full-dataset training, pseudo-realistic image training, and final SIC training.The stages use 10,000, 20,000, 5,000, and 5,000 iterations, respectively.
- Training Details: Training uses 8 A800 GPUs, batch size 16, 832 × 480 resolution, AdamW with learning rate 1e-5, and takes about 7 days.SIC is applied with probability 0.1.
- Evaluation Data: The evaluation set includes 277 high-quality Pexels and Sora videos spanning humans, animals, objects, camera movements, and object motions.It also includes 100 videos each from PandaSet and Bridgev2 for embodied and autonomous-driving evaluation, unrelated to the training data.
- Evaluation Metrics: Real-world relighting evaluation measures motion preservation with RAFT optical-flow differences and material consistency with existing pretrained models.Physical consistency of relit effects is assessed through user studies.
- Evaluation Metrics: The user study evaluates visual realism (VR), physical consistency (PC), and lighting alignment (LA) for results requiring high realism.Participants compare the input video, target lighting, and randomly shuffled outputs from four methods.
A.5 User Study
The user study used a dedicated interface to evaluate relighting results on realism, preservation of original lighting cues, and consistency with the target lighting condition. Participants assessed anomalous artifacts, unchanged shadows or metallic highlights, and target-condition mismatch.
- Study Interface: Participants used the interface illustrated in Figure A16 to evaluate relighting results.The study interface supported three evaluation questions.
- Evaluation Criteria: The first question asked which result was inconsistent with the input video’s realism, including anomalous glowing or artifacts.This criterion targeted visible realism failures relative to the input video.
- Evaluation Criteria: The second question asked which result failed to modify the original shadows or metallic highlights.This criterion evaluated whether relighting changed the input video’s original lighting cues.
- Evaluation Criteria: The third question asked which result showed lighting inconsistent with the target condition defined by a text description or environment map.The target condition was displayed in the bottom-left of the interface.
A.6 Evaluation of forward rendering … A.10 More visualizations of our methods
The supplementary evaluation shows that Relit-LiVE renders stably under changing viewpoints and lighting, while ablations, editing workflows, failure analyses, and additional visualizations characterize its behavior and applications.
- A.6 Evaluation of forward rendering: Average per-frame PSNR across 200 synthetic videos shows high temporal stability, comparable to methods conditioned on per-frame environment maps.The error distribution is visualized across frames for the video rendering task.
- A.6 Evaluation of forward rendering: Relit-LiVE robustly propagates lighting from the initial viewpoint across changing camera viewpoints, despite dynamic cameras and lighting introducing distinct challenges.The synthetic dataset includes dynamic cameras, dynamic lighting, and combinations of both.
- A.7 Supplement to ablation studies: The architecture ablations train for 10,000 synthetic-data iterations followed by 20,000 full-dataset iterations without the proposed training strategies.Training-strategy ablations then apply each strategy sequentially for 5,000 iterations.
- Experimental settings.: Group-wise addition performs comparably to frame-concat while consuming 25% fewer resources.The operation is introduced to preserve semantic separability within each group and is also used in UniRelight.
- Why use G-buffer latent group-wise addition?: Fusing environment-light features into scene-intrinsic latents improves overall detail over using cross-attention alone, but can replicate content in natural scenes.Relying solely on cross-attention fails to convey environment-map texture details accurately, whereas duplicate light control is adopted in the final architecture.
- A.8 Scene editing workflow: The scene-editing workflow modifies materials through intermediate intrinsics and inserts objects into the original image as reference input.Ground-SAM provides object masks for material adjustments on specific objects.
- A.9 Analysis of failure cases: Relighting failures include color shifts and abnormal illumination, attributed to pseudo-label errors in real-world G-buffers, environment maps, and base colors.Inaccurate base-color labels may cause color shifts, while inaccurate environment-map pseudo-labels impair relighting.