Source-linked AI summary
Light4D: Training-Free Extreme Viewpoint 4D Video Relighting
Zhenghuang Wu, Kang Chen, Zeyu Zhang, Hao Tang
TL;DR
4D relighting remains limited by scarce paired training data and temporal-consistency challenges under extreme viewpoints. Light4D combines pretrained geometric and illumination priors with time-aware guidance and temporal regularization, achieving competitive consistency and lighting fidelity across camera rotations from -90 to 90.
Problem
4D relighting lacks paired training data and must maintain temporal consistency across extreme viewpoints.
Method
Light4D is training-free and combines EX-4D and IC-Light through Disentangled Flow Guidance, Temporal Consistent Attention, and deterministic coherence regularization.
Results
Experiments show competitive performance in temporal consistency, geometric consistency, and lighting fidelity compared with baselines under extreme viewpoint changes.
Takeaways & Limitations
The modular framework offers a scalable approach to controllable 4D content creation under extreme camera trajectories and target illumination.
Takeaways & Limitations
Performance is bounded by EX-4D’s geometric fidelity and IC-Light’s single-frame prior, making global lighting consistency challenging during extreme viewpoint traversals.
Abstract
from arXiv · showhide
Recent advances in diffusion-based generative models have established a new paradigm for image and video relighting. However, extending these capabilities to 4D relighting remains challenging, due primarily to the scarcity of paired 4D relighting training data and the difficulty of maintaining temporal consistency across extreme viewpoints. In this work, we propose Light4D, a novel training-free framework designed to synthesize consistent 4D videos under target illumination, even under extreme viewpoint changes. First, we introduce Disentangled Flow Guidance, a time-aware strategy that effectively injects lighting control into the latent space while preserving geometric integrity. Second, to reinforce temporal consistency, we develop Temporal Consistent Attention within the IC-Light architecture and further incorporate deterministic regularization to eliminate appearance flickering. Extensive experiments demonstrate that our method achieves competitive performance in temporal consistency and lighting fidelity, robustly handling camera rotations from -90 to 90. Code: https://github.com/AIGeeksGroup/Light4D. Website: https://aigeeksgroup.github.io/Light4D.
1. Introduction
Light4D addresses the underexplored problem of jointly controlling illumination and extreme camera trajectories in 4D video. It uses pretrained generative priors and training-free guidance to preserve geometry, temporal coherence, and lighting fidelity.
- 4D content generation requires simultaneous control over camera trajectory and illumination for high-fidelity dynamic scenes.
- Existing relighting methods operate mainly in 2D, while camera-controlled 4D methods generally provide geometry without controllable illumination.
- Supervised 4D relighting methods require expensive paired multi-view, multi-illumination datasets and struggle to generalize to extreme viewpoints.
- Light4D is a training-free framework that combines EX-4D’s geometric prior with IC-Light’s illumination prior for extreme-viewpoint 4D relighting.
- Disentangled Flow Guidance, Temporal Consistent Attention, and deterministic regularization coordinate lighting injection with geometric preservation and reduce temporal flicker.
- Experiments show competitive temporal consistency and lighting fidelity against video relighting baselines, particularly under extreme viewpoint changes.
2. Related Work
Prior work separately advances video relighting, camera-controllable 4D generation, and dynamic-scene relighting. These lines differ in their use of diffusion, geometric structure, supervision, and explicit scene representations.
- Learning-based Illumination Editing: Learning-based relighting evolved from paired CNN supervision toward diffusion-based formulations, with video methods adding cross-frame consistency constraints to reduce flicker.
- Learning-based Illumination Editing: IC-Light achieves strong relighting results through scalable diffusion training coupled with light-transport consistency.
- 4D Video Generation: Camera-controllable 4D generation enforces structure through explicit geometric priors, intermediate reconstructions, camera-conditioned diffusion, or pseudo-4D guidance.
- 4D Video Generation: EX-4D uses watertight mesh reasoning to support extreme viewpoint variation in dynamic video generation.
- 4D Relighting: Dynamic-scene relighting remains relatively underexplored, with existing systems commonly relying on training and explicit scene representations.
3. The Proposed Method
Light4D rerenders dynamic 4D scenes under user-specified camera trajectories and illumination while preserving geometry and motion. Its pipeline combines time-aware illumination guidance, temporally consistent attention, and deterministic stabilization.
- Temporal Consistent Attention: Temporal Consistent Attention combines standard self-attention for frame-specific structure with Gaussian-smoothed appearance context to suppress temporal variance while retaining frame distinctiveness.Queries remain frame-specific, whereas keys and values are aggregated over a local temporal neighborhood; residual interpolation balances sharpness and stability.
- Disentangled Flow Guidance: Premature high-frequency illumination injection can disrupt early geometric formation and cause geometric collapse.Standard flow matching models prioritize early denoising stages for coherent 3D structure, motivating time-aware illumination control.
- Disentangled Flow Guidance: Disentangled Flow Guidance progressively integrates illumination into the latent space by guiding EX-4D’s geometric flow with an IC-Light-based relighting correction.At each timestep, the method estimates a clean geometric state, decodes it to pixels, predicts a relit image, and fuses geometry with illumination using λ(t).
- Disentangled Flow Guidance: A four-phase adaptive λ(t) schedule isolates geometry, ramps up lighting, maintains a stable illumination plateau, and then linearly adjusts the guidance weight.The geometric isolation phase sets λ(t)=0, while later phases introduce and reconcile appearance cues with the established 3D structure.
- Deterministic Coherence: Deterministic coherence regularization applies Canonical Noise Initialization, Global Moment Matching, and Frequency-Decoupled Illuminance Regularization before IC-Light fusion.These procedures are used to suppress temporal stochasticity in the hybrid target.
4. Experiment
Light4D is evaluated against training-based and cascaded training-free baselines on synthetic and real-world videos across camera-motion ranges, with ablations examining its modules. It delivers strong relighting and content-preservation results under extreme viewpoints, while component removals degrade temporal stability, detail retention, or relighting quality.
- Experimental Setup: Light4D is compared with Light-X, two cascaded EX-4D–LAV pipelines, and a frame-wise EX-4D+IC-Light baseline.The comparisons cover both training-based and training-free alternatives.
- Experimental Setup: The benchmark contains 100 videos spanning humans, animals, objects, and landscapes, evaluated at 30°, 90°, and 180° camera movement ranges, plus OpenScene driving sequences.
- Main Results: Across 30°, 90°, and 180°, Light4D achieves the strongest overall CLIP-Frame, HFPR, and Aesthetic Score performance.These metrics reflect temporal coherence, detail preservation, and perceptual quality.
- Main Results: Cascaded pipelines improve temporal behavior over frame-wise relighting but misalign lighting with evolving geometry under large rotations, while Light-X degrades at larger viewpoints.The frame-wise baseline produces visibly unstable illumination, and Light-X can show static, baked-in lighting when new geometry appears.
- Main Results: Light4D consistently leads Frame PSNR and SSIM across viewpoint ranges and achieves the lowest LPIPS at 90° and 180°.These results indicate content preservation despite substantial illumination edits and increasing viewpoint changes.
- Ablation Studies: Removing DGA, CLA, or the regularization modules degrades motion preservation, temporal coherence, detail retention, or perceptual quality.The largest regularization degradation occurs without FDI, followed by CNI and GMM; removing smoothing increases temporal instability.
5. Limitation and Future Work
Light4D’s performance is bounded by the geometric fidelity of EX-4D and the single-frame nature of IC-Light. Global lighting consistency remains challenging during extreme viewpoint traversals, motivating stronger video-native and 4D priors.
- Light4D depends on the geometric fidelity of EX-4D and the single-frame illumination prior of IC-Light.
- Global lighting consistency remains challenging when adapting 2D image priors to temporal relighting, particularly across −90° to 90° viewpoint traversals.
- Future work targets advanced video-native illumination models, stronger 4D backbones, photometric inconsistency mitigation, and complex effects such as cast shadows and inter-reflections.
6. Conclusion
Light4D resolves the conflict between geometric reconstruction and illumination synthesis through time-aware guidance, temporal attention, and deterministic coherence regularization. Experiments report competitive geometric consistency and lighting fidelity, while its modular design supports future generative-model improvements.
- Disentangled Flow Guidance harmonizes geometric reconstruction and illumination synthesis through a time-aware schedule.
- Temporal Consistent Attention and Deterministic Coherence Regularization maintain geometric consistency and reduce visible flicker.
- Experiments show competitive performance in geometric consistency and lighting fidelity compared with baselines.
- The modular framework is extensible to future advances in generative models for controllable 4D content creation.
A.1. Ablation on Geometric Isolation Phase (τg).
The ablation shows that illumination timing must balance geometric formation against sufficient denoising for relighting. Deterministic coherence mechanisms further stabilize appearance across frames.
- Geometric Isolation Phase: Premature lighting injection can disrupt coherent 4D geometry, reducing temporal consistency and detail preservation while increasing motion errors.The study identifies a trade-off between early and delayed illumination fusion.
- Geometric Isolation Phase: Delayed illumination injection can reduce the number of denoising steps available for relighting refinement.
- Deterministic Coherence: Rectified IC-Light predictions are fused with geometric projections to construct the hybrid flow target used for flow matching.The signal is rectified before stochastic appearance output is integrated.
- Deterministic Coherence: Canonical noise broadcasts one shared noise map across frames to create a topologically consistent generation path for identical semantic regions.This reduces stochastic texture jitter across views.
- Deterministic Coherence: Global moment matching aligns each frame’s intensity mean and standard deviation with a temporal-average reference for consistent brightness.The aligned frame uses the reference moments µref and σref.
- Deterministic Coherence: Frequency-decoupled regularization smooths the low-frequency illumination layer temporally while preserving high-frequency texture details.The method uses spectral decomposition and temporal smoothing of the base illumination layer.
C. More Implementation Details
The implementation evaluates Light4D under matched extreme camera trajectories and documents its ablation, user-study, and temporal-fusion settings. Viewpoint changes span 30°, 90°, and 180° in the reported evaluation setup.
- Evaluation Setup: Evaluation compares training-based and training-free baselines under identical 30°, 90°, and 180° viewpoint changes.Camera trajectories are aligned across methods for fair comparison.
- Ablation: Table 4 summarizes relighting and video-quality metrics for a 30° viewpoint change, marking best and second-best results.
- Ablation: Figure 6 reports HFPR, Temporal CLIP, and Motion Flow L1 across geometric-isolation thresholds τg.
- Fusion Schedule: The geometric-isolation schedule sets illumination fusion to zero for the first 60% of denoising steps, then increases it linearly to 0.5.The transition occurs from Steps 0–14 to Steps 15–24.
- Temporal Processing: The Consistent Light Attention module uses γ = 0.7, followed by adaptive temporal smoothing with a nine-frame window and Gaussian σ = 25.The smoothing is intended to suppress high-frequency flicker while preserving texture details.
- User Study: The user study surveys 30 participants across three viewpoint settings using four 1–5 Likert dimensions for relit-video quality.The dimensions include prompt match, lighting consistency, geometric consistency, and relighting realism.
D. Limitation and Future Work
Light4D’s performance is bounded by the geometric fidelity of EX-4D and the single-frame nature of the IC-Light prior, especially during extreme viewpoint traversals.
- Limitation and Future Work: The framework remains constrained by its foundation models: EX-4D’s geometric fidelity and IC-Light’s single-frame prior.Global lighting consistency is particularly challenging when adapting the image prior to temporal sequences under −90°∼90° traversals.
E. More Experimental Results
Additional experiments report stable temporal transitions, preserved 3D structure, coherent shadows, and realistic vehicle relighting across viewpoints and environments.
- Extended Results: Qualitative results cover challenging viewpoints, diverse lighting prompts, and complex real-world environments.
- Normal Viewpoints: Under normal viewpoints, Light4D preserves high-frequency details, produces natural light–material interactions, and yields smooth temporal transitions.The results span diverse object categories and moving light sources.
- Extreme Viewpoints: Under large camera shifts, the method preserves underlying 3D structure without noticeable ghosting artifacts.The approach also keeps relit shadows and highlights spatially coherent.
- Autonomous Driving: In autonomous-driving sequences, Light4D handles outdoor lighting changes and produces realistic shadows on moving vehicles.These results indicate practical utility for high-fidelity relighting and driving-data augmentation.
- User Study: Figure 5 presents the user-study survey interface for side-by-side comparison of anonymized relighting results.