PaperlayerView exact revision on arXiv

Source-linked AI summary

Light-X: Generative 4D Video Rendering with Camera and Illumination Control

Tianqi Liu, Zhaoxi Chen, Zihao Huang, Shaocong Xu, Saining Zhang, Chongjie Ye, Bohan Li, Zhiguo Cao, Wei Li, Hao Zhao, Ziwei Liu

arXiv:2512.05115v2cs.CV

TL;DR

Monocular videos show only 2D projections of scenes shaped by geometry, motion, and illumination, and paired multi-view, multi-illumination training videos are unavailable in real-world settings. Light-X combines camera and illumination control using disentangled geometry/motion and lighting cues, while Light-Syn synthesizes training pairs from in-the-wild footage. The paper reports that Light-X surpasses baselines in joint control and video relighting across text- and background-conditioned settings.

  • Problem

    Monocular videos capture only 2D projections, and real-world paired multi-view, multi-illumination training videos needed to disentangle geometry and lighting are unavailable.

    Source: §1 · p. 2

  • Method

    Light-X separates geometry and motion cues from illumination cues, while Light-Syn uses degradation and inverse mapping to synthesize paired training data from in-the-wild footage.

    Source: §1 · p. 2, §3.3 · p. 6

  • Results

    Light-X consistently surpasses existing baselines in joint camera–illumination control and video relighting, including text- and background-conditioned settings.

    Source: §1 · p. 2, §5 · p. 10

  • Takeaways & Limitations

    The framework supports joint camera–illumination control as well as video relighting and novel-view synthesis within a single model.

    Source: §1 · p. 2

    The method depends on single-image relighting priors and point clouds; inaccurate depth can degrade generation, and very wide camera motions such as 360° remain challenging.

    Source: §5 · p. 10

Abstract

from arXiv · show

Recent advances in illumination control extend image-based methods to video, yet still facing a trade-off between lighting fidelity and temporal consistency. Moving beyond relighting, a key step toward generative modeling of real-world scenes is the joint control of camera trajectory and illumination, since visual dynamics are inherently shaped by both geometry and lighting. To this end, we present Light-X, a video generation framework that enables controllable rendering from monocular videos with both viewpoint and illumination control. 1) We propose a disentangled design that decouples geometry and lighting signals: geometry and motion are captured via dynamic point clouds projected along user-defined camera trajectories, while illumination cues are provided by a relit frame consistently projected into the same geometry. These explicit, fine-grained cues enable effective disentanglement and guide high-quality illumination. 2) To address the lack of paired multi-view and multi-illumination videos, we introduce Light-Syn, a degradation-based pipeline with inverse-mapping that synthesizes training pairs from in-the-wild monocular footage. This strategy yields a dataset covering static, dynamic, and AI-generated scenes, ensuring robust training. Extensive experiments show that Light-X outperforms baseline methods in joint camera-illumination control and surpasses prior video relighting methods under both text- and background-conditioned settings.

1 INTRODUCTION

Monocular videos capture only 2D projections of scenes shaped by geometry, motion, and illumination, while existing relighting and camera-control methods address these factors separately. Light-X combines camera-trajectory and illumination control, using disentangled conditioning and Light-Syn training pairs; experiments report strong results across joint control and video relighting.

1 INTRODUCTION

  • Existing video relighting methods face a trade-off between lighting fidelity and temporal coherence and lack camera control, while camera-controlled generation lacks illumination editing.The paper identifies joint camera–illumination control as an open challenge and notes that required paired multi-view, multi-illumination training videos are unavailable in real-world settings.Source: §1 · p. 2
  • Light-X separates geometry and motion cues from illumination cues, using dynamic point-cloud renderings along user-defined camera trajectories and a relit frame projected into the same geometry.A light-DiT layer is also introduced to enforce global illumination consistency.Source: §1 · p. 2
  • Light-Syn uses inverse mapping in a degradation-based pipeline to synthesize paired training data from in-the-wild footage, including AI-generated, static, and dynamic scenes.Degraded variants serve as inputs, while original videos provide supervision.Source: §1 · p. 2
  • Light-X outperforms baselines in joint camera–illumination control and video relighting under text- and background-conditioned settings.The introduction reports the joint-control comparison and relighting results in text- and background-conditioned settings.Source: §1 · p. 2
  • The paper calls method outputs “degraded” because they are typically lower in fidelity than natural footage.This terminology is used when describing Light-Syn's training inputs.Source: §1 · p. 2

2 RELATED WORK

Prior video-generation work progressed through GANs, VAEs, autoregressive transformers, and diffusion models, while recent research developed video relighting and camera-controlled synthesis along largely separate tracks. The paper positions Light-X as combining illumination and camera control.

Video Generative Models

  • Video-generation research moved from GANs and VAEs to autoregressive transformers and diffusion models, with recent scalable video synthesis using Diffusion Transformers.The passage cites VDM, Make-A-Video [Singer et al., 2023], and Sora among these developments.Source: §2 · p. 3

Learning-Based Illumination Control

  • Video relighting methods extend image-based approaches, including Light-A-Video [Zhou et al., 2025] and RelightVid [Fang et al., 2025], but still face a trade-off between lighting fidelity and temporal consistency.Light-A-Video uses cross-frame light attention and progressive fusion; RelightVid extends IC-Light's 2D U-Net to a 3D backbone with temporal attention.Source: §2 · p. 3

Camera-Controlled Video Generation

  • Camera-controlled video methods use camera parameters, pose conditioning, or explicit geometric cues for viewpoint control, but remain focused solely on camera trajectories.The paper instead pursues joint camera-motion and illumination control.Source: §2 · p. 3

3 METHOD

Light-X synthesizes a dynamic scene from a monocular video under a user-specified camera trajectory and illumination, separating geometry and motion cues from lighting cues. It combines aligned point-cloud projections, conditional video diffusion, paired training data from Light-Syn, and flexible illumination inputs.

3 METHOD

  • The target video is intended to preserve the source scene’s appearance and dynamics while following user-specified camera and illumination conditions.Illumination may be supplied as a text prompt, HDR environment map, or reference image.Source: §3 · p. 3

3.1 FORMULATION: CAMERA–ILLUMINATION DECOUPLING

  • Source-frame depths form dynamic point clouds that are projected along the target trajectory, providing geometry-aligned views and masks as priors for coherent viewpoint changes.The projections are intended to guide the diffusion model along the specified trajectory.Source: §3.1 · p. 4
  • A relit frame is lifted using the original video’s depths and projected into the same geometry, keeping its lighting cues aligned with the source content.Reusing the original depths is specified to ensure geometric alignment between the relit and original content.Source: §3.1 · p. 4

3.2 ARCHITECTURE: CAMERA–ILLUMINATION CONDITIONED VIDEO DIFFUSION

  • Projected source views encode scene content, geometry, and motion, while projected relit views provide illumination cues; a Q-Former extracts global illumination tokens for DiT denoising.The framework feeds these cues into conditional denoising and decodes the result into a target video.Source: Figure 2 · p. 4, §3.2 · p. 5

3.3 DATA CURATION

  • Light-Syn degrades in-the-wild target videos and applies inverse transformations to create aligned conditioning cues, drawing on static, dynamic, and AI-generated scenes.The dataset includes 8k static, 8k dynamic, and 2k AI-generated videos.Source: §3.3 · pp. 5–6, Figure 3 · p. 5

3.4 FRAMEWORK FLEXIBILITY

  • Decoupled conditioning supports camera-only and relighting modes, while HDR-map and reference-image cues use soft domain-indicator masks in a shared model.The framework also supports foreground video relighting conditioned on background images.Source: §3.4 · p. 6, Figure 4 · p. 6

4 EXPERIMENTS

The experiments evaluate joint camera–illumination control and video relighting against constructed and existing baselines, using quantitative metrics and user studies. Light-X is reported to outperform baselines across these tasks, while ablations examine the contributions of its data and design choices.

4.1 EXPERIMENTAL SETTINGS

  • The evaluation compares constructed joint-control baselines and video-relighting methods, using FID, aesthetic preference, temporal-consistency measures, real-video perceptual and temporal metrics, and user studies.The evaluation set includes 200 videos; background-conditioned relighting uses 300 foreground–background combinations. Training uses 49-frame videos at 384 × 672 for 16,000 iterations.Source: §4.1 · pp. 7–8

4.2 CAMERA-ILLUMINATION CONTROL RESULTS

  • Light-X achieves the best PSNR, SSIM, LPIPS, and FVD scores in joint camera–illumination control evaluated against real in-the-wild videos.The comparison is reported in Table 2, and user studies further validate the improvements.Source: §4.2 · p. 8, Table 2 · p. 8

4.3 VIDEO RELIGHTING RESULTS

  • In text-conditioned relighting, Light-X improves relighting fidelity and temporal smoothness and achieves the best performance across metrics against real in-the-wild videos.The reported comparisons include quantitative results and user studies.Source: Table 3 · p. 8, §4.3 · p. 9, Table 4 · p. 9
  • For background-conditioned foreground relighting, Light-X surpasses all baselines in image quality and video smoothness.The quantitative comparison is reported in Table 5.Source: §4.3 · p. 9, Table 5 · p. 9

4.4 ABLATION STUDIES

  • Ablations show that static, dynamic, and AI-generated data, fine-grained lighting cues, global control, supervision choices, and soft masks each affect reported output quality or consistency.Relighting all frames instead of one increases cost and reduces temporal coherence despite better FID.Source: §4.4 · p. 10, Table 6 · p. 10

5 CONCLUSION

Light-X jointly controls camera trajectory and illumination from monocular videos, using disentangled geometry and lighting cues and Light-Syn paired training data. The paper reports that it surpasses baselines in joint control and video relighting, while identifying limitations in its lighting priors, point-cloud geometry, and video generation.

5 CONCLUSION

  • Light-X jointly controls camera trajectory and illumination from monocular videos, using dynamic point clouds for geometry and motion cues and reprojected relit frames for illumination.Light-Syn constructs paired training videos through degradation and inverse geometric mapping, without requiring multi-view, multi-illumination captures.Source: §5 · p. 10
  • The method relies on single-image relighting priors and point clouds, and may degrade with suboptimal lighting priors or inaccurate depth; very wide camera motion and fine details remain challenging.The authors also identify multi-step denoising as computationally expensive.Source: §5 · p. 10

A.1 TRAINING-FREE BASELINE: TL-FREE

  • The training-free TL-Free baseline adapts light fusion and attention for projected views, disabling detail compensation because it propagates black borders and holes.IC-Light fusion is delayed until later denoising, after missing content has been filled in.Source: §A.1 · p. 18

A.2 BACKGROUND-CONDITIONED CONTROL

  • For background-conditioned relighting, foreground and background videos are composited with masks, then paired with an IC-Light relit video as illumination cues.The model uses both the composite source video and sparse relit video to produce consistent illumination and motion.Source: §A.2 · p. 18

A.3 BASELINE

  • Baseline evaluation uses official Light-A-Video and AnyV2V implementations, while RelightVid is evaluated only for background-conditioned relighting because no text-conditioned variant or training code is provided.Baseline outputs are resized to match the evaluation resolution.Source: §A.3 · p. 18

A.4 EVALUATION PROTOCOL AND USER STUDY

The evaluation protocol samples lighting prompts, directions, and camera trajectories, applying the same conditions across methods. An online study compares paired video outputs across relighting quality, smoothness, identity preservation, and 4D consistency.

A.4 EVALUATION PROTOCOL AND USER STUDY

  • The protocol randomly selects lighting prompts and directions for relighting and one of four camera trajectories for novel-view generation, applying each selected condition consistently across methods.For joint control, the protocol uses TrajectoryCrafter novel-view sequences in motion-preservation comparisons and an IC-Light-relit sequence as the FID reference.Source: §A.4 · p. 18
  • An online user study compares two relit results side by side across relighting quality, video smoothness, identity preservation, and 4D consistency, with responses from 57 participants.Participants could choose “Hard to judge” or skip an example; the study collected no personally identifiable data.Source: §A.4 · pp. 18–19

B DETAILED DATA CURATION

Light-Syn constructs training pairs by degrading in-the-wild target videos and inverse-mapping geometry and illumination information into conditioning cues. Its data sources contribute static-scene viewpoints, dynamic motion, and diverse lighting; the section also outlines video diffusion’s forward and reverse processes.

B DETAILED DATA CURATION

  • Training pairs require a high-quality, temporally consistent target; an input consistent with it in overlapping regions; and geometrically aligned projected source and relit views.The sparse relit video supplies an illumination prior, while projected relit views align illumination cues with projected geometry.Source: B DETAILED DATA CURATION · p. 19
  • Light-Syn degrades an in-the-wild target video into an input and applies inverse transformations to produce paired geometry and illumination conditioning cues.The three data sources are described as complementary, and the pipeline figure depicts the degraded input, aligned projections, relit counterparts, and original-video ground truth.Source: B DETAILED DATA CURATION · pp. 19–20, Figure C · p. 21
  • Static-scene curation uses reconstructed depth and camera poses to pair views and relighting variations, yielding 8k training samples from DL3DV.The two pairing strategies use repeated frames from one clip or clips with overlapping content.Source: B DETAILED DATA CURATION · p. 19
  • Dynamic-scene curation creates degraded views through combinations of relighting and novel-view synthesis, then uses forward and reverse flows for geometric alignment, yielding 8k VDW samples.Depth is estimated once from the original video rather than re-estimated from intermediate results.Source: B DETAILED DATA CURATION · p. 20
  • AI-generated training videos enrich lighting diversity; the authors filter for small motion to limit content inconsistency, retaining 2k OpenVid-1M samples.A commercial video-generation model produces the relit video from a relit first frame and a novel-view sequence.Source: B DETAILED DATA CURATION · p. 20

C PRELIMINARY: VIDEO DIFFUSION MODELS

  • Video diffusion progressively adds noise in its forward process and removes noise in reverse; iterative latent denoising followed by decoding produces the video.The described model compresses videos into latents with a pretrained 3D VAE before denoising.Source: C PRELIMINARY: VIDEO DIFFUSION MODELS · p. 20

D.1 ADDITIONAL RESULTS ON CAMERA–ILLUMINATION CONTROL

Light-X is evaluated across joint camera–illumination control, several relighting conditions, and novel-view synthesis. The reported comparisons favor Light-X on the iPhone benchmark and describe strong relighting and novel-view results across the tested settings.

D.1 ADDITIONAL RESULTS ON CAMERA–ILLUMINATION CONTROL

  • On the iPhone multi-view benchmark, Light-X achieves the best overall performance, the lowest FID, and significantly improves motion preservation while balancing metrics better than baselines.IC-Light applied to target-view videos attains a slightly higher aesthetic score, but does not require novel-view generation.Source: §D.1 · pp. 22–23
  • Light-X maintains high-quality relighting and temporal consistency across diverse lighting prompts, while the compared baselines show fidelity or consistency trade-offs.The comparisons describe LAV as fragile under large camera motion and frame-wise IC-Light as flickering; applying LAV after TrajectoryCrafter can undermine geometric reconstruction.Source: §D.1 · pp. 21–23

D.2 ADDITIONAL RESULTS ON TEXT-CONDITIONED RELIGHTING

  • Under text-conditioned relighting, Light-X produces high-quality, temporally consistent videos; frame-wise IC-Light flickers, while LAV can compromise lighting fidelity and detail.Source: §D.2 · p. 23

D.3 ADDITIONAL RESULTS ON BACKGROUND-CONDITIONED RELIGHTING

  • For background-conditioned relighting, Light-X integrates subjects with target-background illumination and maintains natural, temporally stable outputs; it also adapts to diverse backgrounds while preserving identity and detail.Source: §D.3 · p. 23, §D.4 · p. 24

D.4 ADDITIONAL RESULTS ON HDR MAP-CONDITIONED RELIGHTING

  • With HDR-map conditioning, Light-X produces plausible relighting and scores better on reported PSNR, SSIM, and LPIPS metrics, while VBench visual quality is comparable to DiffusionRenderer.The authors note that the metric setting inherently favors their training paradigm.Source: §D.1 · p. 22, §D.4 · p. 23

D.5 ADDITIONAL RESULTS ON REFERENCE IMAGE-CONDITIONED RELIGHTING

  • A single reference image can specify the illumination style transferred to an input video, and Light-X also supports simultaneous relighting and novel-view synthesis.Source: §D.5 · p. 24

D.6 ADDITIONAL RESULTS ON NOVEL VIEW SYNTHESIS

  • On the iPhone dataset, Light-X achieves higher PSNR and lower LPIPS than TrajectoryCrafter while maintaining comparable SSIM in novel-view video synthesis.The authors describe its additional flexibility as joint camera–illumination control.Source: §D.6 · p. 24

D.7 GEOMETRY CONSISTENCY EVALUATION

The supplementary evaluations examine geometry preservation, non-Lambertian surfaces, wide camera motions, recent baselines, temporal distance, and occluded relighting references. Reported results include lower reconstruction distances, robustness under substantial viewpoint changes, and continued illumination propagation in the tested occlusion cases.

D.7 GEOMETRY CONSISTENCY EVALUATION

  • Light-X achieves the lowest mean and median Chamfer Distance between point clouds reconstructed from input and relighted videos, indicating better preservation of geometric structure.Source: §D.7 · p. 24, Table D · p. 25

D.9 LARGE CAMERA TRAJECTORIES

  • Point-cloud cues can become sparse or unavailable under extremely wide camera motions, negatively affecting synthesis quality, although Light-X remains robust to viewpoint deviations of roughly 60°.Source: §D.9 · pp. 25–26, §D.11 · p. 27

D.8 ANALYSIS ON NON-LAMBERTIAN SURFACES

D.10 COMPARISON WITH MORE RECENT METHODS

  • Across comparisons with recent camera- and lighting-control systems, Light-X maintains superior image fidelity, aesthetic quality, and motion consistency; Free4D is evaluated on 10 project-page scenes because its per-scene optimization typically takes over an hour.Source: §D.10 · p. 26, §D.11 · pp. 27–28

D.11 FID DEGRADATION WITH TEMPORAL DISTANCE

  • Over 49 frames, FID rises from 38 to 82 for video relighting and from 56 to 100 for joint camera–illumination control, with the last frame still outperforming LAV (FID: 112.45) and TL-Free (FID: 122.73).The analysis measures distance from an IC-Light-relit reference frame; later frames rely on Light-X to propagate illumination cues.Source: §D.11 · pp. 26–27

D.12 PERFORMANCE WITH OCCLUDED RELIT FRAMES

  • With a partially occluded relit reference, Light-X propagates illumination coherently and produces reasonable, consistent relighting in newly visible regions during zoom-outs.Source: §D.12 · p. 28

D.13 ROBUSTNESS TO DEPTH NOISE

Light-X remains coherent under moderate Gaussian depth-map noise, and its performance is robust to which video frame is selected as the relighting reference.

D.13 ROBUSTNESS TO DEPTH NOISE

  • Light-X degrades gracefully as Gaussian depth noise increases, maintaining coherent illumination and motion consistency under moderate noise and consistently outperforming baselines.The controlled experiment used 12 randomly selected scenes and DepthCrafter as the default depth estimator; the perturbation strength is controlled by rate.Source: §D.13 · p. 28

D.14 CHOICE OF THE RELIGHTING REFERENCE FRAME

  • Across 200 videos, selecting the first, middle, last, or a random frame as the relighting reference produces similar performance.During training the relit frame is randomly sampled, and at inference any frame may be selected.Source: §D.14 · p. 28

D.15 USING LIGHT-X FOR DATA GENERATION

The supplementary analysis tests Light-X-generated relighting as added training data, synthetic relighting from graphics-style procedures, and the composition of Light-Syn data. The reported effects differ: Light-X-generated data brings marginal gains, synthetic relighting degrades performance, and the three Light-Syn sources provide complementary benefits.

D.16 USE OF SYNTHETIC RELIGHTING DATA

  • Adding 2k DiffusionRenderer-style synthetic samples consistently degrades performance; the authors point to a domain gap from real illumination and call for more realistic, diverse synthetic pipelines.The paper uses in-the-wild videos as ground truth in Light-Syn because graphics-engine relighting may not capture real-world illumination complexity and variability.Source: §D.16 · p. 30

D.15 USING LIGHT-X FOR DATA GENERATION

  • Fine-tuning with 2k Light-X-relit training samples yields only marginal improvements, suggesting the original Light-Syn data already provides sufficiently strong supervision.The authors note that real in-the-wild videos remain the primary source of learning signals and that synthetic relighting serves only as input, limiting its impact.Source: §D.15 · pp. 28–29, §D.16 · p. 31

E ADDITIONAL ABLATION ANALYSES

  • The additional ablations examine training-data composition and the global illumination control module; the reported data-source findings show complementary contributions from static, dynamic, and AI-generated data.Source: E ADDITIONAL ABLATION ANALYSES · p. 31, §E.1 · p. 31

E.1 TRAINING DATA

  • Removing static data weakens unseen-view synthesis, removing dynamic data introduces motion artifacts and reduces temporal reliability, and omitting AI-generated data lowers robustness to rare lighting conditions.The full mixture helps maintain fidelity, consistency, and stability under diverse lighting.Source: §E.1 · p. 31

E.2 GLOBAL ILLUMINATION CONTROL MODULE

The global illumination control module helps maintain stable lighting under complex changes, avoiding brightness drift and improving temporal consistency.

E.2 GLOBAL ILLUMINATION CONTROL MODULE

  • Disabling global illumination control causes fading or abrupt brightness shifts, especially with strong directional or spatially varying lighting; enabling it preserves coherent lighting trends and improves temporal consistency.Figure P presents the qualitative ablation of this module.Source: §E.2 · pp. 31, 33
  • Table K compares performance with and without synthetic relighting data on video relighting and joint camera–illumination control.Source: §E.2 · p. 31
  • Figure Q qualitatively compares joint camera–illumination control, with Light-X reported to achieve superior relighting quality, temporal consistency, and realistic novel-view content generation versus baselines.Source: §E.2 · p. 34

Related papers

Loading 2512.05115v2…