Source-linked AI summary
Beyond Pixels: From Video Priors to 4D Worlds
Zihao Liu, Xiaolong Shen, Zhenglin Zhou, Ruijie Quan, Yi Yang
TL;DR
Existing 4D generation either inserts RGB reconstruction between video priors and geometry or binds prediction to particular generators. Latent-to-4D instead maps shared video VAE latents directly to 4D tokens, surpassing matched Wan+4RC cascades on Text4D-200 and I4D-200 while receiving stronger human preferences.
Problem
Current approaches trade generator modularity against direct video-prior transfer, motivating a reusable interface that avoids RGB intermediates and generator-specific retraining.
Method
Latent-to-4D maps final denoised video VAE latents to a pretrained 4D decoder through alignment and frame-wise plus global spatiotemporal refinement.
Results
By 2.88–3.45 DINO-F1 points on Text4D-200 and 5.81 points on I4D-200, Latent-to-4D surpasses matched Wan+4RC cascades and wins human preferences for geometry and temporal stability.
Takeaways & Limitations
Shared video latents provide an effective, reusable interface for 4D generation within the evaluated common-VAE setting.
Takeaways & Limitations
The approach requires compatible video models sharing the VAE checkpoint, normalization, layout, compression convention, and supported latent shape.
Abstract
from arXiv · showhide
4D generation synthesizes dynamic 3D scenes from conditions such as text or images. Existing methods either reconstruct generated RGB videos with a separate 4D model or adapt a particular video generator to predict geometry directly. The former suffers from distribution mismatch and error propagation, whereas the latter ties 4D prediction to a specific generator and may require retraining when the generator or conditioning regime changes. We ask whether the final denoised latents of video models that share a variational autoencoder (VAE) can instead provide a reusable interface to explicit 4D prediction. Building on this insight, we introduce direct latent-to-4D generation and instantiate it as Latent-to-4D, which bypasses RGB by aligning a video latent with the token grid of a pretrained 4D decoder and refining it through frame-wise and global spatiotemporal attention. Trained on roughly 1K existing reconstruction clips, a single checkpoint transfers unchanged across multiple video diffusion transformers within the same VAE family. On Text4D-200 and I4D-200, Latent-to-4D surpasses matched same-latent Wan+4RC cascades in projection-based DINO-F1 by 2.88--3.45 and 5.81 points, respectively, while also being preferred by human raters for geometry, temporal stability, and overall quality.
Introduction
The introduction identifies a trade-off between RGB-mediated reconstruction and generator-specific integrated 4D prediction, then proposes shared final VAE latents as a reusable interface. Latent-to-4D implements this interface directly, achieving cross-generator transfer and higher projection-based DINO-F1 than matched Wan+4RC cascades.
- Motivation: Existing methods either generate RGB observations before separate 4D reconstruction or integrate geometry into a specific video generator.The two paradigms combine video generation and feed-forward 4D reconstruction differently.
- Problem: Generate-then-reconstruct preserves generator modularity but can propagate open-domain content and temporal artifacts into unstable geometry, while integrated methods transfer priors more directly.The trade-off arises from the RGB interface and generator dependence of the two paradigms.
- Motivation: Compatible video generators sharing a VAE produce final denoised latents in a common representation space upstream of RGB decoding.This shared latent space is proposed as a reusable interface for geometry-supervised 4D prediction.
- Results: 2.88–3.45 points: Latent-to-4D surpasses matched Wan+4RC cascades in projection-based DINO-F1 on Text4D-200; 5.81 points on I4D-200.A single checkpoint operates unchanged across two text-to-video DiTs and one image-to-video DiT sharing the same VAE.
Related Work
Related work spans video generators that learn controllable appearance and motion, feed-forward 4D reconstructors that infer dynamic geometry from RGB videos, and 4D generation methods that combine generative priors with dynamic scene representations.
- Video Generation: Video generators learn rich appearance and motion distributions under text, image, camera, trajectory, and animation controls, with representations reused for depth, point maps, and dynamic geometry.These models provide generative priors and transferable representations for geometric prediction.
- 4D Reconstruction: Feed-forward 4D reconstructors predict dynamic point maps, scene flow, trajectories, or dense motion from RGB videos, while query-based models recover cameras and geometry across viewpoint and time.Related image-set methods estimate cameras and dense point maps.
- 4D Generation: 4D generation methods combine generative priors with dynamic scene representations through object-centric optimization and generate-then-reconstruct routes.Earlier text- and image-guided methods separately optimize dynamic NeRF or Gaussian representations for each output; Diffusion4D, 4Diffusion, and CAT4D are generate-then-reconstruct examples.
Method
Latent-to-4D directly maps video VAE latents into a pretrained 4D token space, bypassing RGB decoding and reencoding. It aligns and refines latents with spatiotemporal modules before decoding cameras and dynamic world-space geometry, enabling transfer across compatible video DiTs.
- Latent interfaces: A frozen VAE supplies observed-video latents during training, while a compatible frozen DiT supplies final denoised latents during generation through the same downstream pathway.At inference, generated latents replace observed latents without changing the alignment, refinement, or decoding pathway.
- Direct latent-to-4D formulation: Latent-to-4D replaces RGB reconstruction with a direct mapping from video VAE latents to the structured token space of a pretrained 4D reconstructor.The objective is to bypass RGB decoding and subsequent reencoding, avoiding an RGB representation boundary through which artifacts can propagate.
- Cross-DiT compatibility: Compatibility requires shared VAE checkpoint, normalization, tensor layout, compression convention, and supported latent shape, while DiT architectures and conditioning regimes may differ.A single checkpoint is evaluated across two text-to-video DiTs and one image-to-video DiT.
- Alignment Module: The Alignment Module resamples VAE latents and uses a learned 3D convolution to aggregate local spatiotemporal neighborhoods and project channels onto the 4D hierarchy’s token grid.This conversion addresses mismatches in temporal resolution, spatial grid, and feature dimension.
- Spatiotemporal Refinement Module: The Refinement Module combines frame-wise and global attention to capture spatial detail, long-range correspondence, viewpoint changes, and cross-frame motion before 4D decoding.Multi-level features from selected hierarchy depths provide complementary intra-frame geometry and sequence-level context.
Experiments
Across held-out text- and image-conditioned benchmarks, direct latent lifting outperforms matched RGB reconstruction cascades on structure-sensitive metrics and transfers across compatible video diffusion transformers without retraining. Human judgments, residual sensitivity tests, and control examples further support improved plausibility, stability, and interface compatibility, while the reported DINO measures remain appearance-dependent proxies rather than metric geometry accuracy.
- Experimental setup: The final training stage uses 1,143 clips from six reconstruction datasets, while benchmarks remain held out with no test-time adaptation.The video models and pretrained base weights are frozen; only the Alignment Module, prediction heads, and rank-16 LoRA-adapted refinement hierarchy are trained.
- Experimental setup: One checkpoint serves three Wan video diffusion transformers sharing the frozen Wan VAE, including text- and image-conditioned models.Evaluation renders predictions from two off-axis cameras and reports CLIP, CLIP-I, DINO similarity, DINO matching, and DINO set F1 metrics.
- Text-to-4D: 2.88–3.45 points: DINO-F1 gains over matched Wan+4RC cascades on Text4D-200, with nearly identical performance across both text DiTs.CogVideoX-5B+4RC retains the highest RGB-reference CLIP-I, so the method does not win uniformly on every metric.
- Image-to-4D: 5.81-point DINO-F1 gain: the method ranks first on all I4D-200 metrics against the matched Wan2.2+4RC cascade and other reconstructors.The consistent off-axis DINO gains indicate more complete visible geometry but are not metric geometry measurements.
- Human evaluation: Every human-preference interval exceeds 50%, with strongest preferences for geometry and completeness over baselines.These judgments support more plausible and stable 4D outputs but do not measure metric geometry.
- Sensitivity analysis: 0.0053/0.0047 versus 0.3827/0.3160: at ρ = 0.6, point-map drift is far lower for Ours than the baseline on 7-Scenes/NRGBD.All 30 residual-sensitivity comparisons favor Ours, isolating a near-terminal residual component rather than arbitrary DiT errors.
Conclusion
The conclusion presents direct latent-to-4D generation as a reusable interface from final denoised video VAE latents to a pretrained 4D decoding hierarchy. Latent-to-4D transfers across compatible text- and image-conditioned DiTs and outperforms RGB-decoding cascades in benchmarks while receiving favorable human evaluation.
- Method: Direct latent-to-4D generation uses a video model’s final denoised VAE latent as a reusable interface to a pretrained 4D decoding hierarchy.The hierarchy is initialized from a pretrained reconstructor.
- Transferability: Trained on roughly 1K reconstruction clips, Latent-to-4D transfers one checkpoint unchanged across three compatible text- and image-conditioned DiTs.
- Evaluation: Latent-to-4D achieves higher projection-based DINO-F1 than RGB-decoding cascades, while human evaluation favors its geometric plausibility and completeness.