Source-linked AI summary
DimensionX: Create Any 3D and 4D Scenes from a Single Image with Controllable Video Diffusion
Wenqiang Sun, Shuo Chen, Fangfu Liu, Zilong Chen, Yueqi Duan, Jun Zhang, Yikai Wang
TL;DR
DimensionX addresses the difficulty of recovering controllable 3D and 4D scenes from a single image with video diffusion. It decouples spatial and temporal generation using dimension-aware LoRAs and combines specialized scene-generation strategies, reporting superior results across video, 3D, and 4D generation.
Problem
Video diffusion models produce realistic visuals but have limited spatial and temporal controllability for directly recovering 3D and 4D scenes from one image.
Method
DimensionX uses ST-Director with dimension-aware LoRAs and training-free composition to control spatial and temporal factors, plus trajectory-aware 3D and identity-preserving 4D generation.
Results
DimensionX reports superior performance to baseline methods in controllable video, 3D, and 4D generation across real-world and synthetic data.
Takeaways & Limitations
Controllable video diffusion can support recovery of accurate 3D structures and 4D dynamics from sequentially generated frames.
Abstract
from arXiv · showhide
In this paper, we introduce \textbf{DimensionX}, a framework designed to generate photorealistic 3D and 4D scenes from just a single image with video diffusion. Our approach begins with the insight that both the spatial structure of a 3D scene and the temporal evolution of a 4D scene can be effectively represented through sequences of video frames. While recent video diffusion models have shown remarkable success in producing vivid visuals, they face limitations in directly recovering 3D/4D scenes due to limited spatial and temporal controllability during generation. To overcome this, we propose ST-Director, which decouples spatial and temporal factors in video diffusion by learning dimension-aware LoRAs from dimension-variant data. This controllable video diffusion approach enables precise manipulation of spatial structure and temporal dynamics, allowing us to reconstruct both 3D and 4D representations from sequential frames with the combination of spatial and temporal dimensions. Additionally, to bridge the gap between generated videos and real-world scenes, we introduce a trajectory-aware mechanism for 3D generation and an identity-preserving denoising strategy for 4D generation. Extensive experiments on various real-world and synthetic datasets demonstrate that DimensionX achieves superior results in controllable video generation, as well as in 3D and 4D scene generation, compared with previous methods.
1. Introduction
DimensionX addresses the challenge of generating photorealistic 3D and 4D scenes from one image by controlling spatial and temporal factors in video diffusion. Its framework combines dimension-aware diffusion control with specialized strategies for realistic 3D and 4D synthesis, outperforming prior methods in visual quality and generalization.
- Motivation: The work responds to limited large-scale 3D and 4D video data, which constrains single-image generation of photorealistic interactive environments.This data scarcity is presented as a central challenge for high-quality scene generation.
- Core approach: DimensionX decouples spatial and temporal priors in video diffusion through dimension-aware LoRAs trained on spatially and temporally varying data.The framework also uses training-free composition to combine spatial and temporal controls.
- Core approach: The framework uses controllable video sequences to recover 3D appearances and 4D dynamic motions from a single image.It targets both spatial structure and temporal evolution through generated frame sequences.
- Results: Extensive experiments report superior visual quality and generalization over previous methods for 3D and 4D scene generation.The reported conclusion also identifies video diffusion as a promising direction for realistic dynamic environments.
- Scene generation: Trajectory-aware 3D generation and identity-preserving 4D denoising bridge controllable video generation with realistic scene reconstruction.These mechanisms are designed to improve realism and controllability in real-world scenes.
2. Related work
Prior work has used diffusion priors and controllable video generation for 3D and 4D synthesis, but approaches differ in their control signals, optimization requirements, and scene scope. DimensionX’s pipeline combines ST-Director-based video control with separate 3D and 4D generation procedures.
- Controllable video diffusion: Controllable video methods guide diffusion with camera trajectories, coordinate embeddings, reference videos, or object-motion trajectories.These approaches inject additional conditions to control generated video content.
- DimensionX pipeline: DimensionX organizes controllable generation, 3D reconstruction, and 4D optimization into a pipeline using ST-Director and S-Director.Its 4D branch generates temporal and spatial variants, combines them into multi-view videos, and refines consistency before scene optimization.
- 3D generation: Diffusion-prior 3D methods use SDS or multi-view camera conditioning, while video diffusion extends novel-view generation from single images.Prior work spans both object-level and scene-level generation.
- 4D generation: Early 4D methods optimize representations per scene with SDS, but they can require hours and produce obvious inconsistency.Later approaches improve efficiency and consistency using dynamic-mesh data and multi-view video diffusion.
3. Methodology
DimensionX builds dimension-variant data and orthogonal S- and T-Directors to control spatial and temporal video variation separately, then composes them through denoising for 3D and 4D generation.
- 3.1. Building Dimension-variant Dataset: DimensionX constructs dimension-variant datasets and trains separate LoRA-based S-Director and T-Director modules for spatial and temporal control.Spatial data uses planned camera trajectories, while temporal data uses flow guidance and stationary-camera settings.
- 3.1. Building Dimension-variant Dataset: The dataset pipeline estimates camera distributions and scene occupancy, filters camera placements, reconstructs 3D scenes, and renders views for spatial control.Camera centers and principal axes are computed with PCA/SVD, while TSDF extracts meshes from reconstructed RGB-D data.
- 3.2. ST-Director for Controllable Video Generation: Each frame is modeled as a projection of a 4D scene containing static background and time-varying objects, motivating separate spatial and temporal directors.The projection depends on camera parameters, so changing scene structure and camera motion represents spatial variation while changing objects represents temporal variation.
- 3.2. ST-Director for Controllable Video Generation: S-Director captures camera-motion sequences, whereas T-Director captures object-motion sequences with a stationary camera, enabling independent control along orthogonal dimensions.The directors are trained separately on spatial-variant and temporal-variant datasets using parameter-efficient LoRA fine-tuning.
- 3.2. ST-Director for Controllable Video Generation: Attention maps show that spatial structure emerges before temporal information, while each director changes primarily its designated dimension during denoising.Early denoising establishes foundational outlines; spatial structure appears earlier than temporal object motion.
- 3.2.2 Tuning-free Dimension-aware Composition: Switch-Once starts denoising with S-Director and switches to T-Director at the 4th or 5th step to combine camera motion with object motion.This training-free composition uses early spatial formation followed by temporal refinement to produce hybrid-dimension videos.
4. Experiment
DimensionX is evaluated for controllable video generation and single-image 3D/4D scene generation on real-world and synthetic data. It outperforms baselines qualitatively and quantitatively while its ablations identify mechanisms supporting reconstruction and cross-frame consistency.
- Experimental Setup: Experiments use real-world and synthetic datasets, including dimension-variant data and 100 temporal-variant OpenVid videos for T-Director training.
- Controllable Video Generation: DimensionX outperforms baseline models in Consistency, Dynamic, and Aesthetic scores for controllable video generation.
- 3D Scene Generation: DimensionX outperforms all baselines across PSNR, SSIM, and LPIPS in both single-view and sparse-view 3D reconstruction.
- 4D Scene Generation: Given a single image, DimensionX generates highly consistent dynamic videos from large-angle novel views on real-world Neu3D scenes.
- Ablation Study: Removing S-Director in large-angle sparse-view generation causes the Janus problem, producing multiple heads and lower reconstruction quality.
- Ablation Study: Reference video latent sharing improves global background and subject consistency across frames, while appearance refinement further enhances appearance-detail coherence.
5. Conclusion
DimensionX generates photorealistic 3D and 4D scenes from a single image by controlling spatial and temporal video-diffusion factors. Experiments report state-of-the-art performance, while the framework remains limited by its diffusion backbone.
- DimensionX uses ST-Director and dimension-aware LoRAs to decouple spatial and temporal priors, then combines them for hybrid-dimension control.
- Trajectory-aware 3D generation and identity-aware 4D generation strategies improve the framework’s generalization to real-world scenes.
- Extensive experiments demonstrate state-of-the-art performance in controllable video, 3D scene, and 4D scene generation.
- DimensionX remains limited by the diffusion backbone, whose models still struggle with understanding and generating realistic scenes.