PaperlayerView exact revision on arXiv

Source-linked AI summary

Relit-LiVE: Relight Video by Jointly Learning Environment Video

Weiqing Xiao, Hong Li, Xiuyu Yang, Houyuan Chen, Wenyi Li, Tianqi Liu, Shaocong Xu, Chongjie Ye, Hao Zhao, Beibei Wang

arXiv:2605.06658v1cs.CV

TL;DR

Video relighting remains difficult because physically consistent, temporally accurate illumination requires modeling materials and controllable lighting. Relit-LiVE fuses raw RGB references with intrinsic representations and jointly generates relit and viewpoint-aligned environment videos; experiments report state-of-the-art results in physically consistent, temporally stable relighting.

  • Problem

    Existing decomposition-and-render relighting is limited by imperfect intrinsic decomposition, while physically consistent, temporally accurate lighting remains difficult across materials and lighting conditions.

    Source: §1 · p. 2, §3.2 · p. 3

  • Method

    Relit-LiVE fuses raw RGB reference frames with intrinsic representations and jointly generates relit video and viewpoint-aligned environment video without per-frame camera poses.

    Source: §1 · p. 2

  • Results

    Experiments show Relit-LiVE outperforms state-of-the-art methods in physically consistent, temporally stable video relighting with strong generalization.

    Source: §1 · p. 2, §5 · p. 10

  • Takeaways & Limitations

    The framework also supports scene-level rendering, editing, and streaming video relighting through modified generation conditions and intermediate outputs.

    Source: §1 · p. 2

    Training remains costly and trades off resolution and frame rate, with training capped at 57 frames at 832 × 480 and 57-frame generation taking approximately 10 minutes on an A800 GPU.

    Source: §4.5 · p. 10

Abstract

from arXiv · show

Recent advances have shown that large-scale video diffusion models can be repurposed as neural renderers by first decomposing videos into intrinsic scene representations and then performing forward rendering under novel illumination. While promising, this paradigm fundamentally relies on accurate intrinsic decomposition, which remains highly unreliable for real-world videos and often leads to distorted appearances, broken materials, and accumulated temporal artifacts during relighting. In this work, we present Relit-LiVE, a novel video relighting framework that produces physically consistent, temporally stable results without requiring prior knowledge of camera pose. Our key insight is to explicitly introduce raw reference images into the rendering process, enabling the model to recover critical scene cues that are inevitably lost or corrupted in intrinsic representations. Furthermore, we propose a novel environment video prediction formulation that simultaneously generates relit videos and per-frame environment maps aligned with each camera viewpoint in a single diffusion process. This joint prediction enforces strong geometric-illumination alignment and naturally supports dynamic lighting and camera motion, significantly improving physical consistency in video relighting while easing the requirement of known per-frame camera pose. Extensive experiments demonstrate that Relit-LiVE consistently outperforms state-of-the-art video relighting and neural rendering methods across synthetic and real-world benchmarks. Beyond relighting, our framework naturally supports a wide range of downstream applications, including scene-level rendering, material editing, object insertion, and streaming video relighting. The Project is available at https://github.com/zhuxing0/Relit-LiVE.

1 Introduction

Relit-LiVE targets physically consistent, temporally stable video relighting without camera-pose priors. It combines raw RGB references with intrinsic representations and jointly predicts relit video and viewpoint-aligned environment video.

1 Introduction

  • Direct diffusion relighting using prompts or backgrounds can lack precise lighting control and retain original-illumination artifacts [Liu et al. 2025a; Zhou et al. 2025].The alternative two-stage intrinsic-decomposition approach offers clearer separation and control, but depends heavily on representation fidelity [Fang et al. 2025b; Liang et al. 2025].Source: §1 · p. 2
  • Prior methods require precise camera-pose knowledge to position environment maps in the viewport, limiting flexibility; intrinsic rendering can also fail on transparent objects or subsurface scattering.The introduction identifies these as challenges involving complex light transport and pose-dependent lighting placement.Source: §1 · p. 2
  • Its RGB-intrinsic fusion renderer uses raw RGB frames to restore visual and semantic cues while combining real-world lighting effects with physical constraints from intrinsic representations.The raw references guide and correct rendering when decomposed attributes fail to capture complex global illumination.Source: §1 · p. 2
  • Relit-LiVE jointly generates relit video and per-frame warped environment maps in one inference pass, implicitly learning lighting transformations without per-frame camera poses.The resulting environment video is aligned with the scene geometry and illumination, supporting flexible relighting under camera motion.Source: §1 · p. 2
  • Training robustness comes from latent-space interpolation of relighting and rendering outputs to synthesize multi-illumination data, plus cycle-consistent self-supervision for temporal lighting coherence without extra annotations.Both strategies are presented as ways to handle complex scenarios.Source: §1 · p. 2
  • Relit-LiVE outperforms existing state-of-the-art methods, producing realistic material reflections and modeling viewpoint changes for physically plausible, spatio-temporally accurate relighting without camera-pose priors.The framework also supports scene-level rendering, editing, and streaming relighting by modifying generation conditions and intermediate outputs.Source: §1 · p. 2

2 Related work

Prior work in direct video relighting has shifted toward precise lighting control and physical realism, while intrinsic-aware diffusion offers flexible editing but faces real-world decomposition and compositing challenges.

2.1 Direct video relighting

  • Direct relighting has shifted from temporal consistency using prompts or reference backgrounds toward precise lighting control and physical realism.The section describes early relighting approaches and the broader direction of recent work.Source: §2.1 · p. 3
  • RelightMaster uses multi-plane light images, UniLumos structured prompts and geometric feedback, and UniRelight jointly generates relit videos and albedo while implicitly decoupling ambient lighting.UniLumos uses depth and normal feedback supervision to support plausible shadows.Source: §2.1 · p. 3
  • The paper contrasts parallel joint estimation, which is limited by model capacity, with a decoupled approach designed for comprehensive, expandable intrinsic content and flexible support for relighting and neural rendering.Source: §2.1 · p. 3

2.2 Intrinsic-aware diffusion model

  • PBR-inspired intrinsic decomposition and synthesis provide flexible image and video editing, including light modification and material editing.The section frames this paradigm as more flexible than end-to-end generation.Source: §2.2 · p. 3
  • Fine-tuned diffusion models target specific intrinsic decomposition properties, while RGBX, Diffusion Renderer, and V-RGBX extend bidirectional decomposition-synthesis from images to video.Examples include IntrinsiX, NormalCrafter, and GeometryCrafter for decomposition, and RGBX for estimating G-buffers from images and rendering images from G-buffers.Source: §2.2 · p. 3
  • Real-world decomposition challenges can make closed-loop methods domain-limited and prone to cumulative errors, while compositing typically requires precise per-frame lighting information, motivating the paper’s two-design framework.Source: §2.2 · p. 3

3 Our method

Relit-LiVE combines raw RGB references with intrinsic cues and jointly predicts relit frames and viewpoint-aligned environment videos, avoiding dependence on explicit camera-pose estimates. Its staged training adds intrinsic-perception and illumination-consistency strategies.

3.1 Problem statement

  • The task is to synthesize a scene-faithful target video under novel illumination from a static or dynamic source video; target lighting may also be static or dynamic.For static target lighting, one environment map is applied to every frame.Source: §3.1 · p. 3

3.2 RGB-Intrinsic fusion renderer

  • The RGB-Intrinsic fusion renderer uses observable RGB cues alongside intrinsic properties, retaining visual scene evidence and physical constraints despite imperfect decomposition.The intrinsic properties are predicted as G-buffers by a pretrained inverse renderer.Source: §3.2 · pp. 3–4
  • Partial latent summation separates correlated or similar-valued G-buffer modalities, while randomly sampled raw frames guide generation without fixed pixel correspondences to suppress source-light propagation.The method sums base color, depth, and metallic latents, and separately sums normal and roughness latents; different reference frames can be sampled at each denoising step.Source: §3.2 · p. 4

3.3 Joint generation of relighting and environment video

  • Relit-LiVE jointly generates relit frames and viewpoint-aligned environment maps, learning lighting transformations implicitly to avoid explicit camera-pose estimation.The environment video is predicted alongside the relit video to support spatio-temporal lighting accuracy.Source: §3.3 · pp. 4–5
  • HDR lighting conditions are encoded as tonemapped LDR, normalized log-intensity, and directional images, then supplied through latent conditioning and cross-attention.The directional encoding uses camera-coordinate ray directions, opposite to standard panoramas.Source: §3.3 · p. 5

3.4 Training strategies

  • Intrinsic-perception enhancement interpolates outputs with and without raw-reference latents, balancing realism against residual source lighting and detail distortion to create pseudo-realistic training references.Using the raw-reference latent produces more realistic appearances but may retain source lighting; setting it to zero avoids residual lighting but can distort details.Source: §3.4 · p. 5
  • Final-stage illumination consistency training uses randomly relit videos and reverse-order inference to form frame-correspondence self-supervised pairs, promoting alignment between predicted lighting and relit results.The strategy is designed to improve generalization across diverse scenes and lighting conditions.Source: §3.4 · pp. 5–6

4 Results

Relit-LiVE delivers physically consistent relighting and stable environment-video predictions, while supporting editing and delighting applications. Ablations attribute improvements to raw reference images and joint environment-video generation; the method trades resolution and frame rate for video length.

4.1 Evaluation of video relighting

  • Relit-LiVE outperforms comparison methods across reported relighting metrics, with stronger material consistency and physically accurate reflections and refractions.Text-prompt methods can produce unreasonable luminous effects and preserve original shadows or highlights; Diffusion Renderer shows material distortion. Table 2 also reports user-study preference metrics for visual realism, physical consistency, and lighting alignment.Source: §4.1 · p. 6, Table 1 · p. 7, Table 2 · p. 7
  • For long videos, Relit-LiVE processes clips sequentially, using the preceding clip’s predicted environment video to relight the next clip with temporally consistent lighting.It tracks camera-viewpoint changes and warps the environment map accordingly.Source: §4.1 · p. 6, Figure 9 · p. 9

4.2 Evaluation of environment video generation

  • Relit-LiVE’s generated environment maps closely match references, with stable directional-error predictions over time that support spatially accurate lighting as camera viewpoints change.The task estimates per-frame lighting from a single frame’s environment map; Table 3 compares against frame-by-frame StyleLight and DiffusionLight estimates.Source: §4.2 · p. 6, Table 3 · p. 7, Figure 10 · p. 9

4.3 Other applications

  • Scene-level rendering supports inserting objects and editing materials with realistic reflections and shadows, while video delighting removes specular highlights and restores original material properties.The delighting results are relevant to perception tasks sensitive to specular artifacts, including 3D reconstruction and depth estimation.Source: §4.3 · pp. 6–7

4.4 Ablation Study

  • Raw reference images improve relighting and help recover accurate transmission effects that intrinsic-decomposition errors would otherwise compromise.The ablation describes improvements on complex materials such as plastic bags and glass bottles, and says the references guide realistic physical effects.Source: Table 4 · p. 7, §4.4 · pp. 8–9, Figure 13 · p. 10
  • Jointly generating environment video and relit frames significantly improves results for synthetic videos with substantial camera motion or dynamic lighting by aligning lighting to each frame’s viewpoint.This alignment supports spatially consistent camera-free relighting.Source: Table 4 · p. 7, §4.4 · p. 9
  • After supervised training, the initial model reaches PSNR 21 on MIT multi-illumination, while intrinsic-perception enhancement and self-supervised training improve lighting decoupling and relighting quality.The enhancement strategy creates pseudo-raw references for real-world scenes with special lighting; self-supervision enables closed-loop training under arbitrary lighting and scenes.Source: §4.4 · pp. 9–10

4.5 Limitations

  • Relit-LiVE trades resolution and frame rate for video length, training up to 57 frames at 832 × 480 and taking about 10 minutes to generate a 57-frame video on an A800 GPU.The frame-dimensional concatenation-based control method still incurs substantial training costs despite merging intrinsic latents to reduce overhead.Source: §4.5 · p. 10

5 Conclusion

Relit-LiVE combines RGB-intrinsic fusion with joint relit-video and environment-video generation to produce physically consistent, temporally stable relighting without explicit camera-pose estimation. The framework generalizes to complex scenes and supports downstream video-editing tasks.

5 Conclusion

  • Relit-LiVE combines an RGB-intrinsic fusion renderer with joint generation of relit and environment videos, enabling realistic relighting under physical constraints without explicit pose estimation.The joint formulation incorporates real-world lighting effects.Source: §5 · p. 10
  • Two complementary training strategies mitigate the scarcity of multi-light datasets and improve generalization in complex scenes.Source: §5 · p. 10
  • Experiments show that Relit-LiVE outperforms state-of-the-art methods in physically consistent, temporally stable relighting, including shadows and reflections.The conclusion also reports strong generalization.Source: §5 · p. 10
  • The framework supports downstream scene rendering and video illumination estimation, demonstrating potential as a universal video-editing engine.Source: §5 · p. 10

A Appendix · A.1 Data generation strategy

The data-generation strategy combines rendered synthetic videos with filtered real-world clips and images, adding scene representations and environment-map annotations to support training across synthetic and realistic scenarios.

A.1 Data generation strategy

  • The synthetic dataset contains 8,000 videos, each with 120 frames at 512 × 512 resolution.Each video includes base colors, roughness, metallicness, normal maps, depth maps, environment maps, and camera trajectories.Source: §A.1 · p. 12
  • The synthesis workflow draws on 5,700 PBR material maps and 2,241 HDR environment maps from public resources [Li et al. 2025; Vecchio and Deschaintre 2024].Source: §A.1 · p. 12
  • Scenes contain a plane and up to 12 non-intersecting primitives, textured with randomly selected PBR maps.The workflow uses collision detection to avoid object intersections.Source: §A.1 · p. 12
  • Synthetic videos cover camera rotation with fixed lighting, lighting rotation with a fixed camera, and simultaneous camera-and-lighting rotation, each scene rendered under at least two random lighting conditions per motion pattern.Source: §A.1 · p. 12
  • Real-world samples come from DL3DV [Ling et al. 2024], SpatialVID-HQ [Wang et al. 2025], MIT multi-illumination, RemovalBench [Wei et al. 2025], and SOBAv2 [Liu et al. 2024].A VLM [Yang et al. 2025a] filtered datasets other than MIT multi-illumination, removing blurred frames and samples with significant shadows from objects outside the frame; this yielded 13,809 57-frame clips and 792 images.Source: §A.1 · p. 12
  • The team generated pseudo ground-truth G-buffers with Diffusion Renderer’s inverse renderer and obtained MIT multi-illumination environment maps from reflective chrome sphere screenshots.Source: §A.1 · p. 12
  • The training data combines synthetic samples with real-world datasets of varying completeness, including environment maps for 8,278 videos and 206 images aligned frame by frame to camera trajectories.For other datasets, DiffusionLight [Phongthawee et al. 2024] annotated only images with horizontal perspectives; non-horizontal samples were excluded because of significant estimation errors, and obvious errors were manually filtered.Source: §A.1 · p. 12

A.2 Initial training

Relit-LiVE uses a staged training schedule: it first learns foundational rendering on synthetic data, then trains on the full dataset with cross-attention frozen to improve generalization while retaining adaptability to varied lighting.

A.2 Initial training

  • Training begins exclusively on synthetic data to learn foundational rendering.Source: §A.2 · p. 12
  • The model then trains on the full dataset with cross-attention frozen, enhancing generalization while preserving adaptability to varying lighting conditions.Source: §A.2 · p. 12
  • Setting latent zI to zero with probability 0.3 simulates a pure rendering task; real-world denoising targets support datasets with or without environment maps and single-lighting scenes.The real-world target formulation follows UniRelight [He et al. 2025b].Source: §A.2 · p. 12

A.3 Additional Details on Intrinsic Perception Enhancement

Intrinsic Perception Enhancement fuses relighting and rendering outputs to create more stable, high-quality multi-illumination training data. Repeated generation with improving models helps the data better support decoupling original illumination.

A.3 Additional Details on Intrinsic Perception Enhancement

  • Relighting preserves visual detail but can retain original lighting, while rendering gives reasonable lighting but lacks realism, motivating fusion of both outputs for consistent multi-illumination data.The approach regenerates data using the optimal model at each stage and adjusts parameter w; later generated data improves substantially and supports better relighting.Source: §A.3 · p. 12
  • Existing open-source relighting models are unsuitable for generating pseudo-reference images because they fail to produce realistic lighting that respects material properties.The passage points to advanced open-source methods discussed in Section 4.1 as examples of this limitation.Source: §A.3 · p. 12

A.4 Experimental details

The appendix specifies staged fine-tuning, comparison baselines, task-specific test sets, and evaluation procedures spanning synthetic and real-world videos.

A.4 Experimental details

  • Relit-LiVE is fine-tuned from Wan2.1-T2V-1.3B [Wan et al. 2025] in three stages, including synthetic and full-data training, pseudo-realistic image training, and SIC training.The stages use 10,000 and 20,000 iterations, then 5,000 iterations to train on pseudo-realistic images and 5,000 final iterations with SIC at probability 0.1.Source: §A.4 · pp. 12–13
  • Training uses 8 A800 GPUs, batch size 16, 832 × 480 resolution, AdamW at 1e-5, and takes about 7 days.Video length is fixed at 17 frames for synthetic training, then cyclically increased from 1 to 57 using the 8n+1 pattern.Source: §A.4 · p. 13
  • Comparisons cover five video relighting methods and NeuralGaffer, RGBX and Cosmos-Diffusion Renderer for scene rendering, and DiffusionLight and StyleLight for environment-light estimation.UniRelight [He et al. 2025b] results are replicated from its paper because it is not open-sourced.Source: §A.4 · p. 13
  • Tests include 1,000 synthetic 120-frame videos, 30 MIT scenes across 25 lighting configurations, and real-world collections from Pexels, Sora, PandaSet, and Bridgev2.MIT lighting pairs match condition i with i+12; real-world data include 277 Pexels and Sora videos and 100 each from PandaSet and Bridgev2, unrelated to training data.Source: §A.4 · p. 13
  • Synthetic and MIT tests assess relighting and rendering with PSNR, SSIM, and LPIPS, and estimate illumination using angular error in degrees for concentrated sunlight.These test sets evaluate relighting, rendering, and lighting estimation.Source: §A.4 · p. 13
  • Real-world evaluation measures motion preservation, material consistency, and physical plausibility through optical-flow differences, CLIP-MC and DINO-MC scores, and user studies.User studies assess visual realism, physical consistency, and lighting alignment; CLIP and DINOv3 similarities are invariant to lighting changes (≥0.94) and sensitive to material differences (≤0.89).Source: §A.4 · pp. 13–14

A.5 User Study

The user study compared Relit-LiVE with baselines through participant judgments of realism, shadow and highlight modification, and consistency with target lighting. The final metric was the ratio of comparisons in which Relit-LiVE outperformed each baseline.

A.5 User Study

  • Participants judged whether results preserved video realism, modified original shadows or metallic highlights, and matched the target lighting condition.The target condition was shown through a text description or environment map.Source: §A.5 · p. 14
  • The study collected responses from 37 participants, each completing 10 sets of comparisons.Participants could select 0 to 2 options for each question.Source: §A.5 · p. 14
  • Ties prompted repeated surveys until a clear judgment was reached, and the final metric was the ratio of comparisons favoring Relit-LiVE over each baseline.Source: §A.5 · p. 14

A.6 Evaluation of forward rendering … A.10 More visualizations of our methods

The supplement evaluates forward rendering, ablations, lighting control, scene editing, failure cases, and additional visual results. It reports robust rendering across dynamic conditions while identifying limits tied to inaccurate real-world pseudo-labels.

A.6 Evaluation of forward rendering

  • Relit-LiVE achieves optimal neural-rendering performance across static and dynamic camera and lighting patterns, with stable frame-wise results comparable to methods using per-frame environment maps.Despite receiving only the initial-viewpoint environment map, it robustly perceives camera changes and propagates lighting across viewpoints.Source: §A.6 · p. 14, Figure A17 · p. 14, Table A1 · p. 15

A.7 Supplement to ablation studies

  • Architecture ablations use 10,000 synthetic-data iterations followed by 20,000 full-dataset iterations; training-strategy ablations apply each strategy sequentially for 5,000 iterations.All ablation runs use Wan2.1-T2V-1.3B initialization, 17-frame videos, batch size 4, and one A800 GPU; these models are unrelated to the final model.Source: Experimental settings · p. 14

Experimental settings.

Why use G-buffer latent group-wise addition?

  • Fusing environment-light features into scene-intrinsic latents improves overall natural-scene detail over cross-attention-only lighting, but can copy raw-reference content and degrade relighting.Cross-attention-only control produces color temperatures closer to target lighting but lacks reflective detail; the authors attribute this partly to text prompts' sparsity in fine-grained spatial structure.Source: Why choose dual-input light conditions? · pp. 14–15

Why choose dual-input light conditions?

A.8 Scene editing workflow

  • The scene-editing workflow modifies materials in intermediate intrinsics and inserts objects into the reference image supplied to the model.Ground-SAM [Ren et al. 2024] provides object masks for material adjustments, and latent-space interpolation generates edited images when the processed reference still contains elements to modify.Source: §A.8 · p. 15, Figure A20 · p. 16

A.9 Analysis of failure cases

  • Relighting failures include color shifts and abnormal illumination, which the authors attribute to inaccurate real-world base-color and environment-map pseudo-labels.Incorrect environment maps can lead the model to misclassify background pixels as strong light sources.Source: §A.9 · p. 15, Figure A19 · p. 15

A.10 More visualizations of our methods

  • Additional visualizations cover training-strategy ablations, in-the-wild relighting comparisons, portraits, dynamic-lighting scenes, and the same scenes under different environment lights.The supplied captions describe qualitative video comparisons as showing superior relighting quality, temporal consistency, and photorealistic results versus baselines.Source: §A.10 · p. 15, Figure A21 · p. 16, Figure A22 · p. 17

Related papers

Loading 2605.06658v1…