PaperlayerView exact revision on arXiv

Source-linked AI summary

Animate124: Animating One Image to 4D Dynamic Scene

Yuyang Zhao, Zhiwen Yan, Enze Xie, Lanqing Hong, Zhenguo Li, Gim Hee Lee

arXiv:2311.14603v2cs.CV

TL;DR

Dynamic 3D video generation remains relatively unexplored, and animating one image with text-directed motion requires a representation that handles spatio-temporal structure and reference drift. Animate124 optimizes a 4D-grid dynamic NeRF through staged diffusion guidance, and its reported comparison finds more realistic videos that maintain reference appearance than MAV3D.

  • Problem

    Dynamic 3D video generation remains relatively unexplored, and single-image animation must represent spatio-temporal structure while retaining the reference appearance.

    Source: §1 · pp. 1–2

  • Method

    Animate124 optimizes a 4D-grid dynamic NeRF using 2D and 3D priors for a static initialization, video diffusion for motion, and personalized diffusion for semantic refinement.

    Source: §3.1 · p. 3, §3.4 · p. 5, §5 · pp. 8–9

  • Results

    Animate124 produces more realistic videos than MAV3D in the reported comparison and consistently maintains the reference image’s appearance.

    Source: §4.2 · pp. 7–8

  • Takeaways & Limitations

    The three-stage framework produces high-quality 4D dynamic scenes from a reference image and textual descriptions.

    Source: §5 · pp. 8–9

Abstract

from arXiv · show

We introduce Animate124 (Animate-one-image-to-4D), the first work to animate a single in-the-wild image into 3D video through textual motion descriptions, an underexplored problem with significant applications. Our 4D generation leverages an advanced 4D grid dynamic Neural Radiance Field (NeRF) model, optimized in three distinct stages using multiple diffusion priors. Initially, a static model is optimized using the reference image, guided by 2D and 3D diffusion priors, which serves as the initialization for the dynamic NeRF. Subsequently, a video diffusion model is employed to learn the motion specific to the subject. However, the object in the 3D videos tends to drift away from the reference image over time. This drift is mainly due to the misalignment between the text prompt and the reference image in the video diffusion model. In the final stage, a personalized diffusion prior is therefore utilized to address the semantic drift. As the pioneering image-text-to-4D generation framework, our method demonstrates significant advancements over existing baselines, evidenced by comprehensive quantitative and qualitative assessments.

1. Introduction

Although 3D generation has advanced, research has largely focused on static scenes while dynamic 3D video generation remains relatively unexplored. Animate124 aims to animate one image with text-directed motion, addressing representation and reference-drift challenges through a staged diffusion-guided NeRF approach.

1. Introduction

  • Most existing 3D-generation research concentrates on static scenes, while dynamic 3D video generation remains relatively unexplored partly because video-generation models are underdeveloped.Dynamic scenes can represent the 3D world and have applications in video games, augmented reality, and virtual reality.Source: §1 · p. 1
  • Animate124 lifts and animates a single input image into a 3D video guided by textual motion, using diffusion priors to supervise a dynamic NeRF.The approach uses a 4D grid feature encoding to represent spatio-temporal information, addressing the Janus problem associated with plane-based HexPlane features under single-image supervision.Source: §1 · p. 2
  • A static model guided by 2D image and 3D diffusion priors initializes a dynamic model, which video diffusion guides across timesteps and camera perspectives.The reference image helps align the first frame, but later frames rely primarily on the video diffusion model, and the object can drift from the reference over time.Source: §1 · p. 2
  • Frame-level personalized modeling with ControlNet-Tile and Textual Inversion refines appearance while preserving the video’s structure and motion to counter semantic drift.The refinement also enhances resolution and compensates for reference information.Source: §1 · p. 2

2. Related Work

Related work covers dynamic NeRF representations, diffusion-based 3D generation, image-to-3D lifting, and image-to-video animation. The paper positions its work at their intersection: animating an in-the-wild image into a 3D video.

Dynamic Neural Rendering

  • Canonical/deformation-field methods face challenges with scene-topology changes; plane-based methods tend to cause the Janus problem when generating 3D video from a single reference image, so this paper uses a 4D grid for dynamic scenes.Source: §2 · pp. 2–3

Text-to-3D Generation

  • Text-to-3D methods progressed from CLIP-based prompt alignment to diffusion distillation, while MAV3D applies a static-to-dynamic strategy to optimize a dynamic NeRF.Later diffusion-based approaches include mesh representations and distribution optimization.Source: §2 · p. 3

Image-to-3D Generation

  • Image-to-3D work has moved from domain-specific lifting toward in-the-wild generation using fine-tuned multi-view models or diffusion priors, but primarily targets static scenes.Source: §2 · p. 3

Animating Single Image to Video

  • Image-to-video methods use a reference image as an initial frame or semantic condition; the paper proposes animating an established static 3D scene with text.Source: §2 · p. 3

3. Our Method

Animate124 builds a static scene from the reference image, then animates it through coarse dynamic optimization and fine semantic refinement. Its stages combine diffusion priors with a 4D grid NeRF to address drift from the reference image.

3.2. Static Scene Optimization

  • Animate124 initializes a 4D grid dynamic NeRF from a static NeRF learned with 2D and 3D diffusion priors and reference-view supervision.The static-stage objective combines diffusion SDS with reference-image reconstruction.Source: §3 · p. 3, §3.2 · p. 4

3.1. 4D Grid Encoding

  • The 4D grid divides time into T grids, each containing a 3D multi-scale grid, and linearly interpolates spatio-temporal features between the nearest time grids.Projection MLPs use those features to calculate color and density.Source: §3.1 · p. 3

3.4. Semantic Refinement

  • The fine stage uses personalized ControlNet conditioning from the frozen coarse model to refine details and correct semantic drift while reducing error accumulation.Video and 3D diffusion priors remain active in the fine-stage loss.Source: §3 · p. 3, Figure 2 · p. 4, §3.4 · p. 5

3.3. Coarse Dynamic Scene Optimization

  • The coarse dynamic NeRF starts from the static scene and uses video diffusion to align its motion with the text prompt.Each 3D multi-scale grid and the projection layers are initialized from the static model.Source: §3 · p. 3, §3.3 · p. 4
  • Temporal balanced sampling raises the probability of sampling clips beginning at t=0 or ending at t=1, addressing under-sampling of endpoint time grids.The method also limits sampled FPS to [16, 256] for ModelScope, which cannot condition on frame rate.Source: §3.3 · p. 5
  • The dynamic model receives first-frame reconstruction and 3D-diffusion SDS supervision in addition to video-diffusion guidance.The first frame corresponds to the reference image.Source: §3.3 · pp. 4–5

3.4. Semantic Refinement

  • The coarse stage can drift because only its first frame uses reference-image information, while later frames rely on video diffusion guided by text.The paper illustrates this with a panda whose later representation may differ from the reference image.Source: §3.4 · p. 5
  • Frame-wise personalized text-to-image refinement uses ControlNet-Tile to refine textures and details while avoiding unexpected motion changes from a model unaware of other frames.The conditioning image comes from the fixed coarse-stage NeRF to prevent error accumulation.Source: §3.4 · p. 5

4. Experiment

Animate124 is evaluated against two baselines and MAV3D, with component ablations examining semantic refinement, 3D diffusion supervision, and temporal balanced sampling. The experiments also define a 24-pair benchmark and assess text-video alignment, image-video alignment, and video quality.

4.1. Experimental Setup

  • Evaluation uses 24 image-text pairs, measuring text-video alignment, image-video alignment, and frame consistency, alongside five human-rated video qualities.Rendered videos are assessed from 10 views; human ratings include reference similarity, prompt faithfulness, quality, motion realism, and amount of motion.Source: §4.1 · p. 6

4.2. Comparison with Other Methods

  • Animate124 outperforms Zero-1-to-3-V and RealFusion-V quantitatively on CLIP and human evaluations, while preserving reference appearance and consistent 3D geometry.Zero-1-to-3-V shifts colors or omits objects, whereas RealFusion-V shows geometric inconsistencies.Source: §4.2 · p. 6, Table 1 · p. 7
  • Against MAV3D, Animate124 aligns generated motion with the protagonist and better preserves reference appearance, including the flamingo’s body color.The comparison covers text-to-4D and image-to-4D settings.Source: §4.2 · pp. 7–8, Figure 5 · p. 8

4.3. Ablation Studies and Analysis

  • Semantic refinement improves text alignment, image alignment, and video consistency, with the largest improvement in image alignment.The refinement uses personalized ControlNet to alleviate semantic drift.Source: §4.3 · p. 8
  • Removing the 3D prior impairs geometry and all three video aspects; removing temporal balanced sampling weakens reference similarity and 3D consistency despite good text alignment.Without temporal balanced sampling, first-frame supervision is nearly ignored during dynamic stages.Source: §4.3 · p. 8

5. Conclusion

Animate124 animates a single in-the-wild image into a 3D video using textual motion descriptions and diffusion priors. Its semantic refinement uses personalized diffusion supervision, and the framework is described as producing high-quality dynamic scenes from the image and text.

5. Conclusion

  • Animate124 animates a single in-the-wild image into a 3D video through textual motion descriptions, using three diffusion priors with a 4D grid dynamic NeRF.The framework optimizes the NeRF in a static-to-dynamic and coarse-to-fine manner.Source: §5 · pp. 8–9
  • HexPlane struggles with limited motion and a conspicuous Janus problem in the six-frame comparisons.Source: Figure 6 · p. 9
  • Personalized diffusion supervision in semantic refinement addresses semantic drift associated with the video diffusion prior.The paper describes the resulting framework as capable of producing high-quality 4D dynamic scenes from the reference image and textual descriptions.Source: §5 · p. 9

A. Outline

The appendix outlines comparisons of the 4D grid encoding, semantic refinement, and temporal balanced sampling.

A. Outline

  • The appendix compares the 4D grid encoding with the HexPlane dynamic NeRF backbone.Source: A. Outline · p. 9
  • The appendix compares semantic refinement using a ControlNet prior with refinement based on a super-resolution prior.Source: A. Outline · p. 9
  • The appendix provides a qualitative demonstration of temporal balanced sampling.Source: A. Outline · p. 9

B. Backbone

The section describes the 4D grid encoding and compares it with HexPlane for dynamic scene representation. The comparison reports more motion and less susceptibility to the Janus problem with 4D grid encoding.

B. Backbone

  • The method divides time evenly into T grids, establishes a 3D multi-scale grid for each, and interpolates features from the two nearest time grids.Source: B. Backbone · p. 9
  • 4D grid encoding typically exhibits more motion than HexPlane, and the 4D scene is less prone to the Janus problem in the back view.The comparison sets HexPlane's azimuth angle to 45° to avoid a reference camera pose perpendicular to one of its planes.Source: B. Backbone · p. 9
  • HexPlane maps the X, Y, Z, and time axes onto six 2D planes and fuses their features to calculate density and color.Source: B. Backbone · p. 9

C. Refinement

The section presents personalized semantic refinement as a way to mitigate video-diffusion semantic drift while enhancing resolution. Compared with super-resolution refinement, it also corrects semantic inaccuracies from the coarse stage.

C. Refinement

  • Semantic refinement with a personalized ControlNet diffusion prior is introduced to mitigate video-diffusion semantic drift and enhance video resolution.MAV3D's refinement instead uses a super-resolution diffusion prior to enhance results.Source: C. Refinement · p. 9
  • The figure compares coarse-stage output with the results of semantic refinement and super-resolution refinement.Source: Figure 7 · p. 10
  • Semantic refinement enhances video quality and rectifies coarse-stage semantic inaccuracies, whereas super-resolution can amplify errors when it lacks reference-image context.Super-resolution can improve resolution in some cases, but the comparison notes that it can also amplify errors in the coarse stage.Source: C. Refinement · p. 10, Figure 7 · p. 10

D. Temporal Balanced Sampling

Temporal balanced sampling is introduced to strengthen reference-image supervision and improve learning of the initial and final time grids. Compared with random sampling, it gathers more information from the first frame and yields more accurate appearance in early stages.

D. Temporal Balanced Sampling

Loading 2311.14603v2…