Source-linked AI summary
One4D: Unified 4D Generation and Reconstruction via Decoupled LoRA Control
Zhenxing Mi, Yuxin Wang, Dan Xu
TL;DR
Video foundation models provide visual fidelity and dynamic understanding but lack explicit geometry, while coupled RGB-geometry finetuning can interfere with pretrained priors. One4D combines modality-specific LoRA branches and masked conditioning to unify 4D generation and reconstruction, reporting high-quality results across both tasks.
Problem
Video foundation models lack explicit geometry, and coupling RGB with geometry can cause cross-modal interference and rapid degradation during low-resource fine-tuning.
Source: §1 · p. 2Method
One4D uses decoupled RGB and geometry LoRA branches with zero-initialized control links, plus masked conditioning for single-image, sparse-frame, or full-video inputs.
Source: §1 · p. 2Results
Takeaways & Limitations
One4D takes a step toward geometry-aware world simulation with video foundation models.
Source: §5 · p. 8
Abstract
from arXiv · showhide
We present One4D, a unified framework for 4D generation and reconstruction that produces dynamic 4D content as synchronized RGB frames and pointmaps. By consistently handling varying sparsities of conditioning frames through a Unified Masked Conditioning (UMC) mechanism, One4D can seamlessly transition between 4D generation from a single image, 4D reconstruction from a full video, and mixed generation and reconstruction from sparse frames. Our framework adapts a powerful video generation model for joint RGB and pointmap generation, with carefully designed network architectures. The commonly used diffusion finetuning strategies for depthmap or pointmap reconstruction often fail on joint RGB and pointmap generation, quickly degrading the base video model. To address this challenge, we introduce Decoupled LoRA Control (DLC), which employs two modality-specific LoRA adapters to form decoupled computation branches for RGB frames and pointmaps, connected by lightweight, zero-initialized control links that gradually learn mutual pixel-level consistency. Trained on a mixture of synthetic and real 4D datasets under modest computational budgets, One4D produces high-quality RGB frames and accurate pointmaps across both generation and reconstruction tasks. This work represents a step toward general, high-quality geometry-based 4D world modeling using video diffusion models. Project page: https://mizhenxing.github.io/One4D
1. Introduction
Video foundation models offer strong visual fidelity and dynamic understanding but lack explicit geometry, while existing joint RGB-geometry methods can interfere with pretrained video priors. One4D combines decoupled modality-specific branches with masked conditioning to support 4D generation and reconstruction in one framework.
Problem
- Video foundation models lack explicit geometry, and existing RGB-geometry coupling can cause cross-modal interference and rapid degradation during low-resource fine-tuning.Accurate geometry is identified as a step toward downstream tasks such as spatial reasoning.Source: §1 · p. 2
Decoupled LoRA Control
- DLC uses separate RGB and geometry LoRA branches with zero-initialized control links that gradually learn pixel-level alignment.The branches share frozen base parameters but not the base forward computation.Source: §1 · p. 2
- DLC is reported to prevent mode collapse and produce accurate geometry without sacrificing RGB fidelity.This is the stated outcome of using modality-specific branches and control links.Source: §1 · p. 2
Unified Masked Conditioning
- UMC encodes single-image, sparse-frame, and full-video inputs as conditioning videos with unobserved frames zero-filled, supporting generation, mixed generation–reconstruction, and reconstruction without architectural changes.The model generates missing RGB frames and full pointmaps from the conditioning video.Source: §1 · p. 2
One4D
- One4D unifies 4D generation and reconstruction in a video diffusion model and reports generalizable, high-quality results across dynamic scenarios.The framework is trained on a curated mixture of synthetic and real 4D datasets.Source: §1 · p. 2
2. Related Work
Related work spans video foundation models, scalable pointmap-based geometry, and diffusion methods for 3D and 4D generation or reconstruction. One4D distinguishes itself by unifying dynamic 4D generation and reconstruction while changing how RGB and geometry are coupled.
Video generation models
- Recent video generation systems show strong world-modeling abilities, motivating investigation of geometry generation from video foundation models.Image-to-video systems also unify image-to-video, frame interpolation, and video continuation within one architecture.Source: §2 · p. 2
3D and 4D generation
- Diffusion approaches span depth or normal-map reconstruction, camera-conditioned multiview generation, and multiview-video extensions for 4D generation.Other cited lines of work generate 3D objects from images or text.Source: §2 · p. 3
- One4D unifies dynamic 4D generation and reconstruction, unlike approaches that specialize in one task, and reports higher quality and geometry accuracy than channel-wise or spatial-wise concatenation.WVD uses channel-wise concatenation, while 4DNeX uses spatial-wise concatenation.Source: §2 · p. 3, Figure 2 · p. 3
3. Method
One4D adapts a video diffusion model to generate synchronized RGB frames and pointmaps, using decoupled modality-specific computation, controlled cross-modal links, and masked conditioning to unify 4D generation and reconstruction. A post-optimization stage recovers camera parameters and depth maps from generated pointmaps.
3.1. Overview
- One4D generates synchronized RGB frames and pointmaps with a video diffusion model, encoding both modalities into latents and training their predicted velocities with a mean-squared error loss.After generation, camera poses and depth maps are recovered from pointmaps through lightweight global optimization.Source: §3.1 · p. 3
3.2. Decoupled LoRA Control
- DLC shares frozen DiT parameters while keeping separate RGB and XYZ LoRA branches computationally decoupled, reducing memory demands and limiting cross-modal degradation.This design avoids duplicating the large backbone while preserving pretrained video priors in the RGB branch and adapting the XYZ branch for pointmaps.Source: §3.2 · p. 4
- Zero-initialized control links connect selected layers so the initially independent branches gradually learn pixel-level alignment; DLC yields cleaner RGB and sharper, more consistent XYZ and depth than concatenation baselines.The links provide controlled cross-modal communication while avoiding spatial-wise concatenation’s token-doubling cost.Source: Figure 3 · p. 3, §3.2 · pp. 4–5
3.3. Unified Masked Conditioning
- UMC represents single-image, sparse-frame, and full-video conditions as a masked conditioning video, generating missing RGB frames and all pointmaps without changing the architecture.Conditioning latents and the observation mask are concatenated only with RGB latents; geometry receives conditioning indirectly through DLC links.Source: Figure 4 · p. 4, §3.3 · p. 5
3.4. Post-Optimization
- Global post-optimization aligns predicted pointmaps with parameterized 3D points and smooths camera trajectories to recover globally consistent camera parameters and depth maps.The objective combines point-map alignment and trajectory smoothness losses.Source: §3.4 · p. 5
4. Experiments
One4D is evaluated for single-image 4D generation, full-video reconstruction, and sparse-frame reconstruction. The reported comparisons show advantages over 4DNeX in generation and competitive geometry and camera reconstruction against reconstruction-only methods.
4.1. Implementation Details
- Training uses about 17k synthetic and 17k real-world clips, and the model has 935.7M trainable parameters trained for 5,500 steps on eight H800 GPUs.The dataset contains roughly 2M frames; task sampling covers single-image, sparse-frame, and full-video inputs.Source: §4.1 · pp. 5–6
4.2. 4D Generation
- Users prefer One4D over 4DNeX across video consistency, motion, aesthetics, depth quality, and overall 4D coherence; VBench also shows stronger motion and aesthetics with comparable I2V consistency.The user-study caption reports the clearest advantage on geometry-related criteria.Source: §4.2 · p. 6, Table 1 · p. 6, Table 2 · p. 6
- Compared with 4DNeX, One4D produces finer geometry, more accurate depth, richer motion, and more coherent 4D scenes across diverse indoor, outdoor, static, and dynamic scenarios.The qualitative comparison also describes stable backgrounds and naturally moving foreground objects.Source: §4.2 · p. 6, Figure 5 · p. 6, Figure 6 · p. 7
4.3. 4D Reconstruction
- On Sintel, One4D reaches 70.4% on δ < 1.25 versus 58.5 / 62.0 for MonST3R and CUT3R, while remaining close to reconstruction-only Geo4D-ref.One4D is trained once for both generation and reconstruction.Source: §4.3 · p. 7, Table 3 · p. 7
- At 50% or 25% of frames, depth accuracy is almost full-video level on Sintel and Bonn; even at sparsity 0.05 or 0.03, reasonable geometry persists from 2–3 frames.Boundary frames can already capture sufficient background information.Source: Table 5 · p. 7, §4.3 · p. 8
4.4. Additional Ablation Study
- CFG scales 4 and 5 yield very similar depth accuracy to the default CFG = 6, indicating robustness to guidance-scale choice.The caption for Table 6 summarizes robustness to CFG choice and improvement with longer training.Source: §4.4 · p. 8, Table 6 · p. 8
- With 1K optimization steps One4D reaches reasonable accuracy, and at 3K it is close to the full model; performance generally improves with more steps.The paper contrasts this with WVD [59], which requires around 1M steps over two weeks on 64 A100 GPUs.Source: §4.4 · p. 8
5. Conclusion
One4D unifies 4D generation and reconstruction in a single video diffusion model. Its DLC and UMC designs support joint RGB-geometry modeling across these tasks, with results trained on mixed synthetic and real 4D data.
5. Conclusion
- One4D bridges 4D generation and 4D reconstruction within a single video diffusion model.The framework is presented as a unified 4D approach.Source: §5 · p. 8
- DLC supports joint RGB and geometry modeling, while UMC handles pure generation, mixed generation–reconstruction, and pure reconstruction without modifying the architecture.The conditioning scheme varies with the sparsity of observed frames.Source: §5 · p. 8
- Trained on a curated mixture of synthetic and real 4D data, One4D achieves generalizable, high-quality 4D results and takes a step toward geometry-aware world simulation.Source: §5 · p. 8