Source-linked AI summary
Text-To-4D Dynamic Scene Generation
Uriel Singer, Shelly Sheynin, Adam Polyak, Oron Ashual, Iurii Makarov, Filippos Kokkinos, Naman Goyal, Andrea Vedaldi, Devi Parikh, Justin Johnson, Yaniv Taigman
TL;DR
MAV3D addresses the lack of text-paired or otherwise readily available 4D data for generating dynamic 3D scenes from language. It optimizes a dynamic NeRF with a pretrained T2V model through temporal SDS and staged refinement, producing viewpoint-renderable scenes whose quality improves with super-resolution and static pre-training, while remaining bounded by T2V quality and facing mesh-conversion and texture limitations.
Problem
Generating dynamic 3D scenes from text is difficult because large-scale datasets of text-4D pairs and readily available 4D models are absent.
Method
MAV3D optimizes a 4D dynamic NeRF using a pretrained T2V scene prior, temporal SDS, static-to-dynamic optimization, and super-resolution fine-tuning.
Results
Static pre-training and super-resolution fine-tuning improve scene quality, convergence, rendered-video detail, and human preferences across quality, text alignment, and motion.
Takeaways & Limitations
MAV3D extends diffusion-based generation to dynamic scenes that can be rendered from a variety of viewpoints using only text descriptions.
Takeaways & Limitations
The method depends on T2V generation quality across views, converts dynamic NeRFs to disjoint meshes inefficiently, and still needs higher-detail textures.
Abstract
from arXiv · showhide
We present MAV3D (Make-A-Video3D), a method for generating three-dimensional dynamic scenes from text descriptions. Our approach uses a 4D dynamic Neural Radiance Field (NeRF), which is optimized for scene appearance, density, and motion consistency by querying a Text-to-Video (T2V) diffusion-based model. The dynamic video output generated from the provided text can be viewed from any camera location and angle, and can be composited into any 3D environment. MAV3D does not require any 3D or 4D data and the T2V model is trained only on Text-Image pairs and unlabeled videos. We demonstrate the effectiveness of our approach using comprehensive quantitative and qualitative experiments and show an improvement over previously established internal baselines. To the best of our knowledge, our method is the first to generate 3D dynamic scenes given a text description.
1. Introduction
MAV3D combines video and 3D generative models to generate dynamic 3D scenes from text, addressing the scarcity of paired 4D training data through a dynamic NeRF and staged optimization.
- Text-to-4D generation: MAV3D generates dynamic 3D scene representations from natural-language descriptions that can be rendered from arbitrary viewpoints.The system targets text-to-4D generation and supports applications including animated assets for games, visual effects, and augmented or virtual reality.
- Motivation: The method addresses the absence of readily available collections of 4D models, including models with textual annotations.Unlike image and video generation, 4D generation lacks large collections of paired or unpaired training models.
- Core approach: MAV3D optimizes a dynamic Neural Radiance Field jointly with text-conditioned video generation while sampling random viewpoints around the object.The video generator acts as a statistical multi-camera setup for reconstructing deformable-object geometry and appearance.
- Representation: The 4D representation uses six multiresolution feature planes to model dynamic scenes efficiently.This design builds on advances in efficient static NeRFs and dynamic NeRFs.
- Optimization: A multi-stage static-to-dynamic optimization pipeline incorporates gradient information from static, temporal, and super-resolution models.The pipeline first fits a static 3D scene with a text-to-image model, then adds dynamics and uses super-resolution fine-tuning for higher-resolution outputs.
2. Related work
Related work spans neural rendering for static and dynamic scenes, text-to-3D generation, and diffusion-based image and video synthesis; MAV3D builds on these lines to generate dynamic scenes from text.
- Neural rendering: Neural radiance fields represent 3D scenes with neural networks that map scene coordinates to images through volume rendering.Efficiency improvements include voxel grids that may be sparse or multiresolution.
- Dynamic neural rendering: Dynamic neural rendering conditions NeRFs on space and time to synthesize novel views of moving scenes.Related methods may also use depth or scene-flow supervision.
- Text to 3D: Text-to-3D methods range from parsing geometric relations and assembling known objects to neural generation from paired text-shape datasets.The paired-data approach is difficult to scale because such data are scarce.
- MAV3D samples: MAV3D samples generated content across time and viewpoint, with the final column displaying the depth image of the adjacent viewpoint.In Figure 2, rows vary time and columns vary viewpoint.
- Text-to-4D distinction: MAV3D is distinguished from prior approaches by targeting 3D dynamic content rather than only 3D content using a text-to-video model.This contrast identifies the method’s focus within text-conditioned generative modeling.
- Diffusion-based generative models: Diffusion models have advanced image synthesis and expanded to other media, including video.MAV3D uses Make-A-Video as its video-generation foundation.
3. Method
MAV3D represents text-conditioned dynamic scenes with a 4D NeRF and supervises rendered videos using a pretrained T2V diffusion model. Its multi-stage pipeline combines a flexible scene representation, static-to-dynamic optimization, temporal SDS, and super-resolution fine-tuning.
- Overview: MAV3D fits a 4D scene representation to a natural-language prompt and renders the resulting dynamic scene from arbitrary viewpoints.Rendered images are formed along camera trajectories and stacked into videos for supervision.
- 4D Scene Representation: HexPlane represents spacetime with six feature planes spanning every pair of spatial and temporal axes, then predicts density and color with an MLP.The representation uses multiresolution grids, a background model, and an occupancy grid to support efficient dynamic rendering.
- Dynamic Scene Optimization: The frozen T2V diffusion model scores rendered videos conditioned on the prompt, and temporal SDS uses its denoising estimates to update scene parameters.The rendered video is noised, denoised, and used to compute an averaged gradient update over noise levels and noise samples.
- Static to Dynamic: MAV3D first optimizes a static 3D scene with a T2I model, then adds temporal planes initialized to zero before optimizing the full dynamic scene with T2V supervision.This schedule addresses artifacts and sub-optimal convergence observed when directly optimizing a dynamic HexPlane with T2V SDS.
- Camera Sampling: Dynamic camera training produces more pronounced and realistic object motion while reducing the multi-face problem associated with static-camera training.The authors hypothesize that static cameras encourage the model to explain both object and camera motion.
- Super-Resolution Fine-Tuning: Temporal-aware super-resolution fine-tuning uses high-resolution gradients from the T2V super-resolution module to improve visual fidelity and enable higher-resolution inference.The super-resolution stage renders high-resolution video while conditioning on the low-resolution output.
4. Experiments
MAV3D is evaluated for text-to-4D generation against adapted baselines, across objective and human metrics, viewing angles, ablations, and image-to-4D extension. The experiments also assess reduced text-to-3D and text-to-video variants and dynamic-mesh conversion.
- Text-to-4D comparison: MAV3D surpasses adapted naive baselines in R-Precision and is highly preferred by human raters across video quality, text faithfulness, motion amount, and motion realism.The baselines combine Make-A-Video with Point-E, pixelNeRF, or D-NeRF; Table 1 reports R-Precision and human preference.
- Viewing-angle evaluation: MAV3D maintains consistent R-Precision across camera viewing angles, while MAV+D-NeRF and MAV+pixelNeRF deteriorate as pan angle increases.MAV+Point-E also maintains a consistent R-Precision score across the evaluated viewing angles.
- Sub-task evaluations: The method is evaluated through reduced text-to-3D and text-to-video variants, with comparisons to Stable-DreamFusion, Point-E, and Make-A-Video.MAV3D is reduced by sampling a single time step for text-to-3D and by sampling fixed viewing directions for text-to-video; its T2V performance is bounded by Make-A-Video’s performance.
- Training-stage ablations: Static pre-training improves convergence and quality: the static-pretrained model is preferred for video quality in 73% and realistic motion in 65% of comparisons.Removing super-resolution fine-tuning also reduces quality, text alignment, and motion preference while increasing noise and reducing fine details.
- Motion-regularizer ablations: Dynamic-camera training produces more motion and avoids the multi-face artifacts observed when camera position is fixed across frames.Gaussian annealing extends the model’s spatial bias toward the larger blob and yields larger, more realistic motion.
- Extensions and applications: An image-to-4D extension conditions MAV3D directly on an input image’s CLIP embedding and generates an asset sharing its semantics, including depth and motion.The experiments also describe conversion of dynamic NeRFs into animated meshes using marching cubes, mesh decimation, and texture mapping.
5. Discussion
MAV3D addresses text-to-dynamic-3D generation by combining diffusion models with dynamic NeRFs, producing viewpoint-flexible scenes despite scarce 4D training data. Its current limitations include inefficient mesh conversion, insufficient texture detail, and dependence on T2V view generation.
- 4D models remain scarce, unlike the large captioned datasets available for images and videos.
- MAV3D combines several diffusion models with dynamic NeRFs to generate text-described dynamic scenes from multiple viewpoints.
- Converting dynamic NeRFs into disjoint meshes is inefficient for real-time applications.
- Higher-detail textures still need improvement, despite quality gains from super-resolution information.
- Representation quality depends on the T2V model generating videos from varied views, while view-dependent prompts only mitigate the multi-face problem.
A.1. Ablation on static scene pretrainig steps.
Static-scene pretraining strongly affects dynamic-scene optimization: direct optimization converges sub-optimally, while 2000 pretraining iterations achieve high quality without further gains from additional steps.
- 63.5% R-Precision results from directly optimizing the dynamic scene, indicating sub-optimal convergence.
- 83.7% R-Precision is achieved with 2000 static pretraining iterations.
- Increasing static pretraining beyond 2000 iterations does not improve scene-representation quality.
A.2. Implementation details
The implementation uses multiresolution neural representations and evaluates ablations of dynamic-camera training and static-scene pretraining. These components improve motion complexity, rendering quality, and convergence.
- Architecture details: The architecture uses seven resolution levels spanning 16 × 16 to 2048 × 2048, with a five-layer 128-unit ReLU MLP.
- Dynamic camera ablation: Dynamic-camera training produces larger, more complicated motion and mitigates the multi-face problem.
- Static scene pretraining ablation: Static-scene pretraining leads to higher-quality renderings, whereas direct dynamic-scene training may lack convergence.
A.3. Training details
Training combines randomized camera trajectories, Gaussian annealing, regularization, and staged optimization to supervise and refine dynamic 4D scenes. Dynamic cameras improve motion realism and reduce temporal artifacts, while static pretraining supports convergence.
- Camera positions are randomly sampled on a spherical range with radius [1, 1.5], and each scene is bounded by radius 1.
- Dynamic Camera: Dynamic camera trajectories bridge generated-video and NeRF-rendered-video distributions by simulating moving-camera motion.
- Dynamic Camera: Dynamic cameras give more pronounced, realistic object motion and reduce multi-face artifacts by exposing objects from multiple directions.
- Gaussian annealing: Gaussian annealing reduces the static-scene center bias as moving objects occupy surrounding regions, increasing σ from 0.2 to 2.0 over 5000 steps.
- Optimization: Optimization uses static 64 × 64 images, dynamic 64 × 64 × 16 videos, and super-resolution 256 × 256 × 16 videos in successive stages.
- Training objective: The training objective adds soft binary cross-entropy regularization to encourage harder object-versus-background predictions.
A.4. Additional results for Image-to-4D
Figure 10 presents additional results for the Image-to-4D application.
- Figure 10 contains additional Image-to-4D application results.
- The figure is presented as supplementary evidence for Image-to-4D results.
- The results are organized under the Image-to-4D application.