Source-linked AI summary
Lighting-grounded Video Generation with Renderer-based Agent Reasoning
Ziqi Cai, Taoyu Yang, Zheng Chang, Si Li, Han Jiang, Shuchen Weng, Boxin Shi
TL;DR
Video-generation models often provide weak or entangled control over layout, lighting, and camera trajectory, limiting explicit scene control. LiVER addresses this with renderer-derived 3D scene conditions, a lighting-aware dataset, lightweight conditioning, progressive training, and an agent that translates text into scene controls. The framework reports improved video quality, realism, control fidelity, and spatial-structure preservation, while its coarse reconstruction makes geometric consistency sensitive to user prompts.
Problem
Existing video-generation methods struggle to model complex physical interactions and physically accurate lighting while controlling scene factors such as layout and camera.
Method
LiVER uses a renderer-based agent, a unified stack of physically meaningful render passes, lightweight conditioning modules, and staged training to control lighting, layout, and camera.
Results
LiVER achieves the lowest FVD and FID, highest CLIP, control fidelity, and mIoU scores on the LiVER-Real test split.
Takeaways & Limitations
LiVER provides explicit, fine-grained control over spatiotemporal composition by jointly modeling layout, lighting, and camera in video generation.
Takeaways & Limitations
Coarse initial 3D reconstruction makes output quality, including geometric consistency, sensitive to user-provided prompts for fine-grained geometry and material details.
Abstract
from arXiv · showhide
Diffusion models have achieved remarkable progress in video generation, but their controllability remains a major limitation. Key scene factors such as layout, lighting, and camera trajectory are often entangled or only weakly modeled, restricting their applicability in domains like filmmaking and virtual production where explicit scene control is essential. We present LiVER, a diffusion-based framework for scene-controllable video generation. To achieve this, we introduce a novel framework that conditions video synthesis on explicit 3D scene properties, supported by a new large-scale dataset with dense annotations of object layout, lighting, and camera parameters. Our method disentangles these properties by rendering control signals from a unified 3D representation. We propose a lightweight conditioning module and a progressive training strategy to integrate these signals into a foundational video diffusion model, ensuring stable convergence and high fidelity. Our framework enables a wide range of applications, including image-to-video and video-to-video synthesis where the underlying 3D scene is fully editable. To further enhance usability, we develop a scene agent that automatically translates high-level user instructions into the required 3D control signals. Experiments show that LiVER achieves state-of-the-art photorealism and temporal consistency while enabling precise, disentangled control over scene factors, setting a new standard for controllable video generation.
1. Introduction
LiVER addresses the limited physical modeling and entangled control of existing video-generation methods by grounding layout, lighting, and camera trajectory in renderer-based scene representations. It combines a lighting-aware dataset, renderer-based agent, lightweight alignment modules, and staged training for controllable, physically realistic video generation.
- Existing data-driven video methods still struggle to model complex physical interactions such as dynamic object occlusions.
- Existing 3D-aware controls provide geometric foundations but largely overlook physically accurate lighting, causing mismatched shadows, reflections, and ambient occlusion.
- LiVER uses renderer-generated diffuse, glossy GGX, and rough GGX image sequences as a scene proxy that preserves lighting cues and layout information.
- LiVERSet contains over 11K videos with 81 frames at 720 × 1280 resolution, split into 10K training and 1K evaluation videos across real and synthetic subsets.
- The framework combines a renderer-based agent, lightweight encoder and adapter, and three-stage training to align scene controls with video latents while improving lighting diversity and preserving visual quality.
2. Related Work
Prior 3D-grounded and relighting methods improve geometric or illumination control but often leave physically accurate lighting entangled with camera and layout. LiVER treats rendered lighting as a unified physical condition while retaining layout and camera controllability.
- Explicit 3D spatial information is a prominent route toward improving physical realism in video generation.
- 2D proxies and trajectory conditions control object motion, camera pose, and view synchronization, strengthening geometric consistency in generated videos.
- Existing 3D-aware methods largely ignore physically accurate lighting, leading to unrealistic artifacts in generated videos.
- Relighting methods modify illumination after capture, but video approaches often entangle lighting with camera and scene-layout properties.
- LiVER renders lighting from a 3D scene proxy as the primary condition while preserving controllability over scene layout and camera.
3. Dataset
LiVERSet combines real and synthetic videos with physical annotations and a unified renderer-based scene proxy. The proxy decomposes illumination into render passes that provide lighting cues and scene-layout information for controllable video generation.
- LiVERSet combines real videos with complex natural lighting and synthetic videos offering broader, dynamically varied physically based illumination.
- Real-video annotation reconstructs coarse dynamic 3D scenes from estimated camera poses, first-frame depth, and object segmentation.
- A physically based renderer generates the scene proxy from each scene mesh, lighting representation, and camera trajectory.
- The proxy stacks diffuse, rough GGX, and glossy GGX passes to encode ambient lighting, broad reflections, and specular highlights.
- The dataset contains approximately 11K clips, split into 10K training and 1K evaluation videos, with 81 frames at 720 × 1280 resolution.
4. Methodology
LiVER translates high-level descriptions into structured scene proxies containing geometry, lighting, and camera information, then injects those proxies into a pretrained video diffusion model. A three-stage training scheme progressively learns proxy translation, lighting control, and lighting diversity while preserving generative priors.
- Renderer-based Agent Reasoning: The renderer-based agent converts text into a scene proxy through scene building, lighting setup, and camera planning.It constructs a scene graph, selects or generates illumination, and produces a temporally smooth camera trajectory before rendering the proxy.
- Renderer-based Agent Reasoning: Scene building organizes objects and spatial relationships into a scene graph, retrieves mesh assets, and optimizes their poses to satisfy relational constraints.Each graph node represents an object with semantic and material properties, while edges encode relations such as “in front of.”
- Lighting-grounded Video Generation: The model uses a lightweight proxy encoder and adapter to align 2D render-pass features with the latent space of a pretrained video model.The proxy encoder maps stacked render passes to compact features, which are injected into video latents as spatial conditioning.
- Lighting-grounded Video Generation: A learnable scalar initialized to zero lets proxy features gradually guide video latents without initially disrupting the pretrained generation process.The conditioning pathway therefore begins with no impact and progressively enables lighting-grounded control during training.
- Training Strategy: Training proceeds through proxy translation, joint LoRA fine-tuning, and mixed real-synthetic training to improve semantic alignment, lighting control, and lighting diversity.The first stage freezes the backbone for 10 epochs, the second jointly fine-tunes LoRA and conditioning modules for 10 epochs, and the final stage mixes real and synthetic videos at a 1 : 1 ratio.
5. Experiments
Experiments compare LiVER with several state-of-the-art methods for video generation from 3D-aware conditions using the LiVER-Real test split and a curated training setup.
- Experimental Setup: Training uses approximately 100K steps on 8 NVIDIA H100 GPUs with total batch size 16 and learning rate 1 × 10−5.Videos are generated at 704 ×1280 resolution using AdamW with a constant learning rate.
- Baselines: The comparison includes CameraCtrl for camera-pose control and MotionCtrl for camera and object-motion control through explicit conditions.These methods represent publicly available baselines capable of generating video from 3D-aware conditions.
5.3. Evaluation Metrics
The evaluation measures video quality, realism, semantic consistency, and fidelity to camera, lighting, object-shape, and spatial conditions.
- Quality & Realism: FVD measures distributional similarity between generated and real videos, while per-frame FID evaluates image quality.Both metrics target visual quality and realism from complementary perspectives.
- Quality & Realism: CLIP image–text similarity evaluates semantic consistency between generated videos and their text prompts.The score is computed with a pretrained CLIP model.
- Control Fidelity: Camera control is evaluated with ATE, RPEt, and RPEr after estimating generated-video poses and aligning trajectories via global Sim(3).ATE and RPEt are scaled by ×100, while RPEr is reported in degrees.
- Control Fidelity: Lighting fidelity is measured by recovering HDR environment maps from generated frames with an existing lighting estimator.This metric targets adherence to the input lighting condition.
5.4. Quantitative Comparison
On the LiVER-Real test split, LiVER outperforms the compared baselines across video quality, realism, control fidelity, and preservation of object structure.
- Quantitative Comparison: LiVER attains the lowest FVD and FID scores and the highest CLIP score, indicating stronger video quality, realism, and text semantics.The comparison uses original videos in the LiVER-Real test split sourced from Ling et al.’s public dataset.
- Quantitative Comparison: LiVER achieves the highest control fidelity with reduced camera-pose and lighting errors.The reported gains cover both camera consistency and lighting adherence.
- Quantitative Comparison: LiVER obtains the highest mIoU, demonstrating more accurate preservation of object shapes and spatial structure throughout the sequence.This result complements the camera and lighting control measurements.
- Qualitative Evaluation: Visual comparisons show more realistic lighting and more precise layout and camera control, with outputs most closely matching the reference video.The qualitative comparison is presented in Fig. 4.
5.6. Ablation Study
The ablations show that diverse synthetic lighting data and progressive training are important for learning controllable, physically realistic video generation. Removing either component substantially degrades lighting fidelity or scene-condition adherence.
- Impact of Synthetic Data: Training only on real-world videos produces incorrect, mostly even illumination and fails to reproduce correct lighting effects.The authors attribute this to less varied real-world lighting patterns.
- Impact of Synthetic Data: LiVER-Syn’s diverse and dynamic illumination is crucial for enabling lighting control and preventing overfitting to limited real-world lighting patterns.
- Importance of the Staged Training Scheme: End-to-end training from scratch leads to nearly still output and degraded performance, weakening the model’s ability to follow specified scene conditions.
- Importance of the Staged Training Scheme: Progressively introducing conditioning signals is essential for stable convergence and for integrating the control module without corrupting pretrained generative priors.
5.7. User Study
A 25-participant user study compared four methods across video quality, scene control, camera trajectory control, and lighting control. LiVER was preferred across all four evaluation dimensions.
- Study Design: The study involved 25 participants evaluating 20 video sets, with each set containing results from four competing methods.
- Study Design: Participants selected their preferred method separately for video quality, scene control, camera trajectory control, and lighting control.
- Results: LiVER outperforms competing methods across all four user-preference dimensions.Preferences were measured as the percentage of samples in which each method was selected.
5.8. Controllability
LiVER provides disentangled control over lighting, layout, and camera motion through editable scene representations. Its controls support continuous illumination changes while preserving scene geometry and materials.
- Lighting Control: Manipulating the HDR environment map produces dynamic, continuous, and physically consistent lighting changes.Changes include effects such as rotating illumination.
- Lighting Control: Lighting variations alter shading and reflections while keeping scene geometry and material properties consistent.
- Layout and Camera Control: An explicit 3D scene proxy enables precise object placement and camera-motion control compared with methods using 2D-based proxies.
- Flexible Editing Workflow: The renderer-based agent creates an editable 3D proxy that users can modify in standard 3D software by changing geometry, lighting, or camera trajectories.
6. Conclusion
LiVER jointly models layout, lighting, and camera for fine-grained control over video composition. However, coarse initial geometry makes fine geometric and material details sensitive to user-provided text prompts.
- Conclusion: LiVER jointly models layout, lighting, and camera to provide explicit, fine-grained control over spatiotemporal video composition.
- Conclusion: The framework is intended to bridge generative quality and scene-level controllability for creative media, virtual cinematography, and immersive content production.
- Limitations: Because the initial 3D scene reconstruction is coarse, final geometric consistency and material detail depend on the user’s text prompt.