PaperlayerView exact revision on arXiv

Source-linked AI summary

Align Your Gaussians: Text-to-4D with Dynamic 3D Gaussians and Composed Diffusion Models

Huan Ling, Seung Wook Kim, Antonio Torralba, Sanja Fidler, Karsten Kreis

arXiv:2312.13763v2cs.CVcs.LG

TL;DR

Existing score-distillation work mainly synthesizes static 3D scenes, leaving text-to-4D generation with an added temporal dimension underexplored. AYG combines dynamic 3D Gaussians and deformation fields with composed image, video, and multiview diffusion guidance, achieving state-of-the-art text-to-4D performance while supporting temporal extension and scene composition.

  • Problem

    Score-distillation methods generally synthesize static 3D scenes, while extending 3D synthesis to dynamic 4D generation remains underexplored.

  • Method

    AYG uses dynamic 3D Gaussians with deformation fields and combines text-to-image, text-to-video, and 3D-aware multiview diffusion models within score-distillation synthesis.

  • Results

    AYG achieves state-of-the-art text-to-4D performance and generates diverse, vivid, detailed, 3D-consistent dynamic scenes that can be extended and composed.

  • Takeaways & Limitations

    AYG supports temporally extended and compositional 4D content creation, with potential applications in digital content creation and synthetic data generation.

Abstract

from arXiv · show

Text-guided diffusion models have revolutionized image and video generation and have also been successfully used for optimization-based 3D object synthesis. Here, we instead focus on the underexplored text-to-4D setting and synthesize dynamic, animated 3D objects using score distillation methods with an additional temporal dimension. Compared to previous work, we pursue a novel compositional generation-based approach, and combine text-to-image, text-to-video, and 3D-aware multiview diffusion models to provide feedback during 4D object optimization, thereby simultaneously enforcing temporal consistency, high-quality visual appearance and realistic geometry. Our method, called Align Your Gaussians (AYG), leverages dynamic 3D Gaussian Splatting with deformation fields as 4D representation. Crucial to AYG is a novel method to regularize the distribution of the moving 3D Gaussians and thereby stabilize the optimization and induce motion. We also propose a motion amplification mechanism as well as a new autoregressive synthesis scheme to generate and combine multiple 4D sequences for longer generation. These techniques allow us to synthesize vivid dynamic scenes, outperform previous work qualitatively and quantitatively and achieve state-of-the-art text-to-4D performance. Due to the Gaussian 4D representation, different 4D animations can be seamlessly combined, as we demonstrate. AYG opens up promising avenues for animation, simulation and digital content creation as well as synthetic data generation.

1. Introduction

Text-to-4D extends score-distillation-based 3D synthesis with temporal dynamics, a setting with limited prior research. AYG combines dynamic Gaussian representations, composed diffusion guidance, and stabilization techniques to generate and extend dynamic scenes.

  • Text-to-4D remains underexplored because prior score-distillation methods typically synthesize static 3D scenes rather than dynamic worlds.
  • AYG represents 4D scenes with dynamic 3D Gaussians and deformation fields that transform the Gaussians to model object motion.
  • AYG composes text-to-image, text-to-video, and 3D-aware multiview diffusion gradients to optimize temporal dynamics, frame-wise visual quality, and initial 3D shape.
  • AYG stabilizes motion learning with Jensen-Shannon-divergence regularization that approximately preserves the mean and variance of moving Gaussian distributions.
  • Motion amplification increases frame differences, while autoregressive generation extends sequences and combines dynamic scenes with changing text guidance.
  • AYG achieves state-of-the-art text-to-4D performance and supports long sequences, compositional scene assembly, and diverse dynamic content.

2. Background

The paper builds on 3D Gaussian Splatting, diffusion-based score distillation, and related dynamic-scene generation methods. AYG differs from MAV3D through composed diffusion guidance and a representation that supports dynamic-scene composition.

  • 3D Gaussian Splatting: 3D Gaussian Splatting represents scenes with positioned, colored, opaque 3D Gaussians and renders pixels by projecting them onto camera image planes.
  • Diffusion Models and Score Distillation Sampling: Diffusion models progressively perturb images or videos with noise and learn denoisers, whose scores can guide optimization of rendered 3D representations.
  • Related 4D generation: AYG combines image, video, and 3D-aware diffusion models with score distillation, Gaussian Splatting, and deformation fields for 4D synthesis.
  • Related 4D generation: MAV3D uses NeRFs with HexPlane features and does not disentangle static geometry from a deformation field, preventing composition of multiple 4D objects into large dynamic scenes.
  • Related 4D generation: AYG outperforms MAV3D qualitatively and quantitatively, achieving higher-quality 4D scenes through its compositional generation approach.

3. Align Your Gaussians

AYG represents dynamic 4D scenes with deformable 3D Gaussians and optimizes them through compositional score distillation. It adds regularization, motion amplification, view guidance, and autoregressive extension to stabilize motion, improve consistency, and lengthen or combine sequences.

  • 3.1. AYG’s 4D Representation: AYG combines 3D Gaussian Splatting with deformation fields, separating the scene’s 3D structure from its temporal dynamics.An MLP predicts time-dependent displacements for Gaussian locations, allowing the scene to move and deform.
  • 3.1. AYG’s 4D Representation: Different dynamic scenes can be combined because each animation has an explicit set of Gaussians and its own deformation field.The explicit representation also supports efficient rendering during score-distillation optimization.
  • 3.2. Text-to-4D as Compositional Generation: AYG first synthesizes a static 3D Gaussian object, then optimizes only its deformation field to add temporal dynamics.The initial 3D stage uses MVDream’s multiview prior together with Stable Diffusion and directional text guidance for canonical views.
  • 3.3. AYG’s Score Distillation in Practice: The 4D stage composes text-to-image and text-to-video diffusion models, while view guidance and concurrent score distillation support consistent synthesis.The framework uses rendered frames from varying times and camera angles as diffusion-model inputs.
  • 3.4. Scaling Align Your Gaussians: A modified Jensen-Shannon regularizer preserves the Gaussian distribution’s mean and diagonal covariance across time, encouraging complex motion over global translation or scaling.The regularizer compares distributions estimated from Gaussian locations in the initial and later frames.
  • 3.4. Scaling Align Your Gaussians: Motion amplification enlarges differences between per-frame video scores and their average, while autoregressive deformation-field interpolation extends, changes, and loops 4D sequences.The extension uses overlapping sequences and interpolates their deformation fields across the overlap interval.

4. Experiments

Experiments show that AYG generates realistic, expressive, detailed, and vivid dynamic scenes, including zero-shot creative prompts. User studies report advantages over MAV3D, while ablations show that image feedback and two-stage decomposition matter.

  • Text-to-4D: AYG generates realistic, expressive, detailed, and vivid dynamic 4D scenes and generalizes zero-shot to creative prompts.The generated scenes can also be rendered at varying speeds and frame rates.
  • Comparison to MAV3D: AYG outperforms MAV3D on overall quality, 3D appearance, text alignment, motion amount, motion text alignment, and motion realism.The comparison used corresponding generations from 28 text prompts in a user study.
  • Ablation Studies: Removing image diffusion feedback produces much worse 3D and overall quality, and omitting the initial 3D stage performs poorly.These findings come from user preferences across ablated AYG variants generated with 30 text prompts.
  • Temporally Extended 4D Synthesis and Large Scene Composition: Autoregressive extension connects sequences with changing text guidance, expressive behavioral transitions, looping animations, and large-scene composition.Loops are formed by matching a later sequence’s final frame to an earlier sequence’s first frame and suppressing deformation there.

5. Conclusions

AYG enables expressive text-to-4D synthesis using dynamic Gaussian representations, composed diffusion guidance, and techniques for extending and combining animated scenes. The paper reports state-of-the-art dynamic scene generation while identifying topological change as an unresolved limitation.

  • 5. Conclusions: AYG represents 4D scenes with dynamic 3D Gaussian Splatting and deformation fields that model object motion.The deformation field transforms the Gaussian collection over time.
  • 5. Conclusions: The method combines multiple diffusion models with score distillation to guide text-to-4D synthesis.Its framework uses text-to-image, text-to-video, and 3D-aware multiview diffusion guidance.
  • 5. Conclusions: AYG introduces regularization and guidance techniques for dynamic scene generation, including autoregressive temporal extension and composition of multiple 4D objects.The representation supports temporally extended synthesis and combining dynamic objects in larger scenes.
  • 5. Conclusions: The paper reports state-of-the-art dynamic scene generation and high-quality, diverse, and dynamic 4D scenes.It also reports qualitative and quantitative improvements over MAV3D.
  • 5. Conclusions: AYG currently cannot easily produce topological changes in dynamic objects.The authors identify scaling beyond object-centric generation and personalized 4D synthesis as future directions.

C.5. JSD-based Regularization of the Evolving Distribution of the Dynamic 3D Gaussians

AYG regularizes the evolving distribution of dynamic 3D Gaussians with a modified Jensen–Shannon divergence. The regularizer keeps the distribution’s spatial mean and diagonal covariance near their initial values during deformation optimization, although it is not an exact JSD.

  • C.5. JSD-based Regularization: The regularizer models the Gaussian set at each time with a Gaussian distribution whose mean and diagonal covariance describe the evolving 3D Gaussian locations.The statistics are computed from Gaussian means along the 4D sequence.
  • C.5. JSD-based Regularization: AYG replaces the generally unavailable Gaussian-mixture JSD with a Gaussian constructed by averaging the two distributions’ means and covariances.This modification yields the practical JSD-based regularization term.
  • C.5. JSD-based Regularization: The deformation field is regularized so the evolving distribution’s mean and xyz-direction variances remain close to their initial values.The initial Gaussians at τ = 0 are fixed and provide the reference distribution.
  • C.5. JSD-based Regularization: The regularization term has a unique minimum when the evolving means and variances match the initial values in each spatial direction.This supports its use for constraining the dynamic Gaussian distribution.
  • C.5. JSD-based Regularization: The modified term is not necessarily an accurate approximation of the true JSD, and alternative distribution regularizers remain unexplored.The authors report that the approach worked well in practice and improved optimization behavior and 4D results.

D.2. AYG’s Score Distillation Scheme

AYG derives its 4D score-distillation update by matching rendered sequences to the product of video and image diffusion distributions. The resulting gradients separately provide temporal and frame-wise guidance, with classifier score distillation used in practice.

  • D.2. AYG’s Score Distillation Scheme: AYG minimizes reverse KL divergence from the deformation-field rendering distribution to the product of video and image teacher distributions.This formalizes compositional generation for the 4D stage.
  • D.2. AYG’s Score Distillation Scheme: The video model receives consecutive rendered frames, while the image model receives independent frames, producing separate score-distillation gradient terms.In practice, the image-model frames are a subset of those given to the video model.
  • D.2. AYG’s Score Distillation Scheme: Diffused renderings follow z = α_t x + σ_tϵ, with w(t) weighting diffusion times and α_t, σ_t defining the shared noise schedule.The signal-to-noise ratio α_t/σ_t decreases with diffusion time.
  • D.2. AYG’s Score Distillation Scheme: The image-model gradient stabilizes optimization, while the video-model gradient supplies temporal guidance during dynamic scene synthesis.Both gradients are backpropagated through differentiable rendering into the deformation field.
  • D.2. AYG’s Score Distillation Scheme: AYG uses classifier score distillation by setting the noise-based terms to classifier-style scores, avoiding ProlificDreamer’s separate fine-tuning procedure.The authors report improved performance over standard score distillation for this scheme.

E.1. Video Diffusion Model Training

AYG trains a latent text-to-video diffusion model based on VideoLDM, adding fps conditioning, 16-frame inputs, more training data, and filtering of slow-motion videos. A larger video model produced better videos but did not improve 4D distillation.

  • E.1. Video Diffusion Model Training: The video model follows VideoLDM by extending a Stable Diffusion backbone with temporal layers for text-video generation.AYG adds fps conditioning, scales inputs from 8 to 16 frames, and trains on a larger dataset.
  • E.1. Video Diffusion Model Training: Training begins with WebVid-10M, containing 10 million text-video pairs, while filtering videos with low average optical-flow magnitude.The filtering targets slow-motion videos.
  • E.1. Video Diffusion Model Training: The model samples fps values with a distribution favoring lower frame rates because they indicate more motion and are harder to learn.Fps conditioning is implemented through cross-attention to a sinusoidal fps embedding.
  • E.1. Video Diffusion Model Training: A larger video diffusion model generated noticeably higher-quality videos but did not improve 4D score-distillation performance.The authors therefore retained the more memory-efficient VideoLDM-style model.
  • E.1. Video Diffusion Model Training: The dynamic 4D optimization stage renders 16 consecutive images per batch to match the video diffusion model’s 16-frame input.Four of those frames are additionally supplied to the text-to-image model.

E.3. Evaluation Prompts

The evaluation uses fixed prompt sets for baseline and ablation studies, plus randomized human comparisons of generated 4D animations. User studies compare AYG with MAV3D and full AYG with ablated variants under specified viewing and response procedures.

  • E.3. Evaluation Prompts: The MAV3D baseline comparison uses all 28 prompts from MAV3D’s project page.
  • E.3. Evaluation Prompts: The 28-prompt set includes varied animated subjects and actions, such as animals, robots, vehicles, musical performances, and object interactions.
  • E.3. Evaluation Prompts: The ablation study uses 30 text prompts spanning animated subjects, actions, and motion descriptions.
  • E.4. User Study Details: Human evaluations assess generated 4D-scene quality by comparing AYG with MAV3D and conducting ablation studies through Amazon Mechanical Turk.
  • E.4. User Study Details: For MAV3D comparisons, participants view videos rendered from similar camera perspectives and judge six categories, with an option to select equal quality.
  • E.4. User Study Details: Ablation evaluations compare full AYG animations with modified models for the 30 prompts, allowing participants to select the preferred character or no preference.
  • E.4. User Study Details: Video-pair order is randomized, participants focus on moving foreground characters, and each pair is evaluated by seven participants.

F.1. Comparisons to MAV3D and R-Precision Evaluation

The appendix compares AYG with MAV3D using available project-page outputs and discusses R-Precision as a limited measure for dynamic 4D scenes. The reported user studies favor AYG across all categories, while R-Precision is approximately similar between methods.

  • Comparison setup: Because MAV3D released neither code nor models, the comparison uses its project-page results and identical text prompts rendered as comparable videos.
  • R-Precision: R-Precision measures top-1 retrieval accuracy of the generated scene’s text prompt from CLIP image embeddings of 2D renderings.
  • R-Precision: R-Precision evaluates text-aligned 3D appearance but does not measure temporal dynamics or object motion, limiting its suitability for dynamic 4D scenes.
  • R-Precision: The evaluation uses 300 prompts originally collected for Make-A-Video video-diffusion evaluation, rather than specifically for synthesized 4D dynamic scenes.
  • R-Precision: AYG and MAV3D perform approximately similarly on R-Precision, although unknown MAV3D rendering details may affect the metric non-negligibly.
  • User-study comparison: User studies are presented as more meaningful for dynamic 4D evaluation, and AYG outperforms MAV3D across all reported categories.

F.2. Extended Discussion of Ablation Studies

The ablations support AYG’s main design: its initialization, regularization, diffusion-model composition, and motion controls each contribute to dynamic 4D quality. User preferences and visual comparisons also identify settings that reduce motion, distort geometry, or destabilize optimization.

  • 3D initialization: Removing the initial 3D stage destabilizes optimization and produces no coherent dynamic 4D scenes.The full model with the initial 3D stage is strongly preferred.
  • Regularization: Without JSD-based regularization, sequences show little motion and global translations or object-size changes instead of complex local motion.The authors hypothesize that regularization prevents a local minimum associated with slow global motion.
  • Diffusion-model composition: Removing the image diffusion-model score degrades 3D appearance and text alignment, while the full model is preferred except for “Motion Realism”.The text-to-image model is used to maintain visual and 3D quality while the video model handles temporal dynamics.
  • MVDream placement: MVDream is beneficial in the initial 3D stage but not during 4D optimization, where users prefer the regular AYG model and observe odd or reduced motion.The authors hypothesize that conflicting MVDream and video gradients suppress smooth 4D dynamics.
  • Motion controls: Sampling varied fps conditionings improves motion relative to using only fps = 4 or fps = 12, while dynamic cameras and the motion amplifier also improve motion quality or amount.The full model is preferred over both fixed-fps variants; static cameras produce less motion, and omitting the amplifier reduces motion.
  • Overall findings: The full AYG model wins on “Overall Quality” across all ablations, with large margins.The authors conclude that all AYG components are important for high-quality dynamic 4D scenes.

F.3. View-guidance Ablation Study

The view-guidance ablation found no clear user-rated advantage for including view guidance during AYG’s initial 3D stage, although the authors retained it based on subjective early observations.

  • View guidance produced approximately similar user ratings to omitting it across overall quality, 3D appearance, and 3D text alignment.The study compared static 3D scenes generated with and without view guidance using the same prompts and protocol.
  • The authors nevertheless retained view guidance because early experiments sometimes improved overall 3D appearance and slightly reduced the Janus-face problem.They describe these benefits as subjective and conclude that view guidance is not a crucial component determining synthesis success.

G. Additional Qualitative Results—More AYG Samples

Additional results show diverse AYG-generated 4D samples, autoregressively extended sequences with changing prompts and looping behavior, comparisons with MAV3D, and fps-controlled video diffusion outputs.

  • Additional Samples: The supplementary material includes additional dynamic 4D scene samples and videos from AYG’s newly trained latent video diffusion model.These materials extend the qualitative examples beyond the main figures.
  • More AYG Samples: Additional figures present varied AYG text-to-4D samples, typically shown from two views with deformation-field dynamics indicated by dotted lines.Figures 10, 11, and 13 provide additional samples similar to the main-paper qualitative results.
  • Temporal Extension and Composition: AYG autoregressively extends 4D sequences, combines sequences under changing text guidance, and creates looping animations that return to the initial pose.Figure 12 illustrates these capabilities, while the accompanying text specifies changing prompts and looping behavior.
  • Comparison with MAV3D: Additional visual comparisons show AYG and MAV3D across four 4D frames, different times, and camera angles.The comparison is presented in Figure 14 and described as supplementary to the main-paper evaluation.
  • fps-Conditioned Video Diffusion: With the same prompt and four displayed frames, fps = 4 produces more motion than fps = 12, while fps = 12 maintains good temporal consistency.The latent video diffusion model synthesizes 16 frames and displays frames 1, 6, 11, and 16.
Loading 2312.13763v2…