PaperlayerView exact revision on arXiv

Source-linked AI summary

Light-A-Video: Training-free Video Relighting via Progressive Light Fusion

Yujie Zhou, Jiazi Bu, Pengyang Ling, Pan Zhang, Tong Wu, Qidong Huang, Jinsong Li, Xiaoyi Dong, Yuhang Zang, Yuhang Cao, Anyi Rao, Jiaqi Wang, Li Niu

arXiv:2502.08590v2cs.CV

TL;DR

Video relighting is constrained by scarce diverse datasets and high training costs, and frame-by-frame image relighting can yield inconsistent lighting and flicker. Light-A-Video uses training-free cross-frame attention and progressive light fusion; it improves temporal coherence while preserving high-quality relighting of individual frames.

  • Problem

    Video relighting faces scarce diverse datasets and high training costs, while frame-by-frame image relighting can produce lighting and appearance inconsistencies that cause flicker.

    Source: §1 · p. 2, §4.2 · p. 4

  • Method

    Light-A-Video is a training-free video-relighting approach that combines cross-frame Consistent Light Attention with progressive linear fusion of relighted appearances into video denoising targets.

    Source: §1 · p. 2, §3.2 · p. 4

  • Results

    Light-A-Video enhances temporal coherence of relighted videos while preserving high-quality relighting of individual frames.

    Source: §6 · p. 8

  • Takeaways & Limitations

    The method supports smooth video relighting without additional training or optimization.

    Source: §1 · p. 2, §6 · p. 8

    Performance is constrained by the underlying image-relighting model and video diffusion model, and CLA has limitations modeling dynamic lighting changes.

    Source: §5.6 · p. 8

Abstract

from arXiv · show

Recent advancements in image relighting models, driven by large-scale datasets and pre-trained diffusion models, have enabled the imposition of consistent lighting. However, video relighting still lags, primarily due to the excessive training costs and the scarcity of diverse, high-quality video relighting datasets. A simple application of image relighting models on a frame-by-frame basis leads to several issues: lighting source inconsistency and relighted appearance inconsistency, resulting in flickers in the generated videos. In this work, we propose Light-A-Video, a training-free approach to achieve temporally smooth video relighting. Adapted from image relighting models, Light-A-Video introduces two key techniques to enhance lighting consistency. First, we design a Consistent Light Attention (CLA) module, which enhances cross-frame interactions within the self-attention layers of the image relight model to stabilize the generation of the background lighting source. Second, leveraging the physical principle of light transport independence, we apply linear blending between the source video's appearance and the relighted appearance, using a Progressive Light Fusion (PLF) strategy to ensure smooth temporal transitions in illumination. Experiments show that Light-A-Video improves the temporal consistency of relighted video while maintaining the relighted image quality, ensuring coherent lighting transitions across frames. Project page: https://bujiazi.github.io/light-a-video.github.io/.

1. Introduction

Video relighting lacks abundant, diverse data and affordable training, while frame-wise image relighting can produce inconsistent lighting and flicker. Light-A-Video addresses this with training-free cross-frame attention and progressive fusion, aiming for temporally smooth relighting.

1. Introduction

  • Video relighting faces scarce datasets and high training costs, while frame-wise image relighting can produce inconsistent lighting and flicker.Traditional physical illumination methods also struggle to generalize beyond controlled settings.Source: §1 · p. 2
  • Light-A-Video is a training-free approach that relights videos in a zero-shot manner using image-relighting models and video-diffusion motion priors.It requires no additional training or optimization.Source: §1 · pp. 2–3
  • CLA adds cross-frame interactions in image-relighting self-attention to stabilize generated lighting sources and improve temporal consistency.The introduction describes temporally averaged features in the attention computation.Source: §1 · pp. 2–3, Figure 2 · p. 2
  • PLF progressively linearly blends relighted appearances into original denoising targets, guiding video denoising toward the desired lighting while smoothing appearance transitions.The strategy uses the physical principle of light transport independence.Source: §1 · pp. 2–3
  • Experiments demonstrate effectiveness and versatility, including relighting full video sequences and provided foreground sequences.The framework is described as compatible with UNet- and DiT-based video-generation backbones.Source: §1 · pp. 2–3

2. Related Work

Prior work develops video diffusion models with temporal priors and image-lighting control, but video lighting methods do not yet simultaneously ensure precise control and high visual quality. Light-A-Video applies pretrained image lighting control during video denoising through progressive guidance.

2. Related Work

  • Video diffusion models use text or image conditions and learned video priors to synthesize temporally consistent frames, while lighting-control research includes neural and diffusion-based methods.Related work also surveys controllable generation and image-animation approaches.Source: §2 · p. 3
  • Existing video-lighting approaches are described as unable to ensure both precise lighting control and exceptional visual quality.The paper also notes a lack of approaches specifically designed to control video lighting.Source: §2 · p. 3
  • Light-A-Video inserts a pretrained image-lighting model into a text-to-video denoising process with progressive guidance, using video priors to transfer lighting control smoothly.The related-work passage frames this as enabling accurate, harmonized control of video lighting.Source: §2 · p. 3

3. Preliminary

The preliminaries describe latent diffusion denoising and light transport as foundations for video relighting. The pipeline uses denoising targets for lighting guidance, while light transport motivates blending image appearances.

3.1. Diffusion Model

  • The diffusion setup encodes an input image into latent space, then introduces Gaussian noise through a forward diffusion process.The passage describes the forward process as a T-step Markov chain.Source: §3.1 · p. 3
  • With v-prediction, the model outputs a velocity representing the denoising direction, from which the noise-free component is recovered as the denoising target.The recovered target is denoted ˆz0←t.Source: §3.1 · pp. 3–4
  • At each denoising step, CLA transforms the consistent target into a relight target, and PLF fuses both into a refined fusion target.The consistent target represents the video diffusion model’s denoising direction.Source: Figure 3 · p. 4

3.2. Light Transport

  • Light transport represents image appearance IL as transport T multiplied by illumination L; because T is linear, blending illumination corresponds to fusing image appearances.The passage identifies T as a single linear light-transform matrix and L as variable environment illumination.Source: §3.2 · p. 4

4. Light-A-Video

Light-A-Video frames video relighting as preserving source motion while matching a lighting condition, with temporal consistency and motion preservation as added challenges. It combines Consistent Light Attention (CLA) with Progressive Light Fusion (PLF) to stabilize lighting and guide coherent relighting through denoising.

4.1. Problem Formulation

  • Video relighting must preserve source-video motion while aligning illumination, unlike image relighting, which focuses on appearance.Frame-by-frame image relighting can produce flickering light sources and inconsistent temporal illumination.Source: §4.1 · p. 4, §4.2 · p. 4

4.2. Consistent Light Attention

  • CLA captures global video context across frames and generates a more stable lighting source than frame-by-frame relighting.Its dual-stream design combines frame-by-frame attention with temporally averaged features.Source: §4.2 · pp. 4–5

4.3. Progressive Light Fusion

  • PLF progressively injects relighting guidance into VDM denoising, reducing the gap between targets and refining the denoising direction toward relighting.The fusion weight decreases as denoising progresses; other direction-modeling schedulers, including Euler Scheduler and Rectified Flow, are also applicable.Source: §4.3 · pp. 5–6
  • VDM-predicted targets are treated as temporally consistent, while detail compensation addresses detail loss before the relighted latent is obtained through CLA.The process encodes and noises the source video, predicts denoising targets, and incorporates detail compensation into the consistent target.Source: §4.3 · p. 5

5. Experiments

The experiments compare Light-A-Video with video-relighting baselines, demonstrate background generation, and examine its components and limitations. The reported results highlight relighting quality and temporal stability.

5.1. Experimental Details

  • The evaluation uses 73 videos, three quantitative metrics, and volunteer rankings; IC-Light and AnimateDiff are the default relighting and video-diffusion models.The 52 volunteers rank five methods across prompt alignment, smoothness, and identity preservation.Source: §5.1 · p. 6

5.2. Qualitative Results

  • Light-A-Video shows high temporal consistency and fidelity to the light condition, while comparison methods exhibit flickering, jitter, or identity shifts.IC-Light + SDEdit-0.6 is smoother but shifts identity; AnyV2V can produce unreasonable illumination changes.Source: §5.2 · p. 6, Figure 6 · p. 7

5.3. Video Relighting with Background Generation

  • Given a foreground sequence and illumination prompt, the pipeline generates a background and harmonious illumination through progressive denoising, CLA, and PLF.The background is entirely noised to initialize the video diffusion model, then progressively denoised before consistent relighting.Source: §5.3 · p. 7, Figure 7 · p. 8

5.4. Quantitative Evaluation

  • Light-A-Video achieves low FID while maintaining high temporal consistency, supporting relighted-image quality and temporal stability.The comparison is reported against several baselines; the supplied table caption identifies relighted-image quality and temporal stability as its comparison dimensions.Source: Table 1 · p. 7, §5.4 · p. 8

5.5. Ablation Study

  • CLA stabilizes background-light generation, while PLF progressively fuses relight targets; video-diffusion motion priors also improve overall video quality.The ablation compares video relighting with CLA or PLF removed.Source: §5.5 · p. 8, Figure 8 · p. 8

5.6. Limitation and Future Work

  • Performance is constrained by the underlying image-relighting model and video diffusion model, and CLA has difficulty modeling dynamic lighting changes.The paper identifies handling dynamic lighting as future work.Source: §5.6 · p. 8

6. Conclusion

Light-A-Video adapts image-relighting models for training-free video relighting using CLA and PLF. The paper concludes that it improves temporal coherence while preserving high-quality frame relighting.

6. Conclusion

  • Light-A-Video is a training-free method that uses image-relighting models for temporally consistent video relighting.The conclusion presents it as a way to extend image relighting to video.Source: §6 · p. 8
  • CLA stabilizes lighting-source generation, while PLF supports smooth transitions in relighted appearance.The two designs address lighting stability and appearance transitions, respectively.Source: §6 · p. 8
  • The method enhances temporal coherence while preserving the high-quality relighting of individual frames.Source: §6 · p. 8

Supplementary Material

Figure 9 tracks how different PLF strategies vary the relight-target proportion over time.

Supplementary Material

  • Figure 9 plots λt over time steps t for different PLF strategies.Source: Figure 9 · p. 12
  • λt determines the proportion of the relight target mixed into the fusion target.Source: Figure 9 · p. 12

A. Comprehensive Ablation Studies

The comprehensive ablations examine how CLA’s γ and PLF’s λt schedules affect relighting quality and temporal consistency.

A. Comprehensive Ablation Studies

  • An excessively large γ degrades relighting quality, while values from 0.2 to 0.5 keep FID stable or can improve it, especially with PLF k = 1 or k = 0.5.γ controls the share of cross-frame averaged features in CLA.Source: A. Comprehensive Ablation Studies · p. 12
  • Temporal Clip score increases as γ rises, while γ values from 0.2 to 0.5 support high motion consistency with the source video.Source: A. Comprehensive Ablation Studies · p. 12
  • A constant λt ≡1 underperforms progressively decreasing λt in relight quality and temporal consistency; it can also disrupt motion preservation and degrade image quality.The paper attributes these effects to perturbing video-diffusion motion priors and repeatedly injecting relight appearance outside the image model's training distribution.Source: A. Comprehensive Ablation Studies · p. 12

B. Additional Results

Additional qualitative results cover foreground-sequence relighting with background generation and direct video relighting. The examples span AnimateDiff and CogVideoX.

Related papers

Loading 2502.08590v2…