PaperlayerView exact revision on arXiv

Source-linked AI summary

Consistent4D: Consistent 360{\deg} Dynamic Object Generation from Monocular Video

Yanqin Jiang, Li Zhang, Jin Gao, Weimin Hu, Yao Yao

arXiv:2311.02848v1cs.CV

TL;DR

Reconstructing dynamic objects from monocular video is difficult, while existing methods often depend on multi-view data and calibration. Consistent4D uses a diffusion-supervised Cascade DyNeRF with interpolation-driven consistency and video enhancement; experiments on synthetic and in-the-wild data demonstrate its effectiveness.

  • Problem

    Recovering dynamic 3D information from a single monocular video is difficult, while existing approaches often need synchronized multi-view data or training with multi-view cues.

    Source: §1 · p. 1

  • Method

    Consistent4D uses a diffusion-supervised Cascade DyNeRF, an interpolation-driven consistency loss, and a lightweight video enhancer for monocular video-to-4D generation.

    Source: §1 · p. 2, §4.1 · p. 5, §4.3 · p. 6

  • Results

    Experiments on synthetic and in-the-wild data demonstrate effectiveness, and the ablation reports that results with ICL always outperform those without it.

    Source: §5.3 · p. 9, §6 · p. 10

  • Takeaways & Limitations

    The approach demonstrates video-to-4D generation from statically captured monocular video, with ICL also reported to mitigate multi-face issues in text-to-3D.

    Source: §1 · p. 2

    Few methods address 4D generation from static-camera video, so comparisons are limited to D-NeRF and K-Planes.

    Source: §5.2 · p. 6

Abstract

from arXiv · show

In this paper, we present Consistent4D, a novel approach for generating 4D dynamic objects from uncalibrated monocular videos. Uniquely, we cast the 360-degree dynamic object reconstruction as a 4D generation problem, eliminating the need for tedious multi-view data collection and camera calibration. This is achieved by leveraging the object-level 3D-aware image diffusion model as the primary supervision signal for training Dynamic Neural Radiance Fields (DyNeRF). Specifically, we propose a Cascade DyNeRF to facilitate stable convergence and temporal continuity under the supervision signal which is discrete along the time axis. To achieve spatial and temporal consistency, we further introduce an Interpolation-driven Consistency Loss. It is optimized by minimizing the discrepancy between rendered frames from DyNeRF and interpolated frames from a pre-trained video interpolation model. Extensive experiments show that our Consistent4D can perform competitively to prior art alternatives, opening up new possibilities for 4D dynamic object generation from monocular videos, whilst also demonstrating advantage for conventional text-to-3D generation tasks. Our project page is https://consistent4d.github.io/.

1 INTRODUCTION

Dynamic 3D reconstruction from a single monocular video is difficult, while existing approaches often rely on multi-view data and calibration. Consistent4D instead uses a static monocular video for 4D generation, combining diffusion-supervised DyNeRF, an interpolation-driven consistency loss, and a video enhancer.

Motivation

  • Existing dynamic-object reconstruction often requires synchronized multi-view videos or training data with multi-view cues, yet may fail on regions unobserved in the input.Multi-view capture also requires synchronized camera rigs and meticulous calibration, limiting potential real-world applications.Source: §1 · p. 1
  • Static monocular video is a practical input because it is easy to collect with handheld cameras, reduces camera-motion blur risk, and avoids camera-parameter estimation.Because it lacks effective multi-view information, the authors turn to generation approaches that rely less on multi-view data.Source: §1 · p. 2

Contributions

  • Consistent4D represents dynamic objects with Cascade DyNeRF, uses a pre-trained diffusion model for optimization, and adds ICL and a lightweight video enhancer.ICL uses a pre-trained video interpolation model for spatiotemporally coherent supervision and is also reported to mitigate multi-face issues in text-to-3D.Source: §1 · p. 2
  • Evaluations on synthetic videos rendered from animated 3D models and in-the-wild Internet videos show promising results for video-to-4D generation.The listed contributions describe the method as targeting dynamic-object generation from a statically captured monocular video.Source: §1 · p. 2

2 RELATED WORK

Prior work spans 3D generation, dynamic-scene reconstruction, and 4D generation, with different representations, inputs, and data requirements. The paper situates its task among these approaches and notes limitations in reconstruction coverage and video-generation quality.

3D Generation

  • Text-to-3D methods use image diffusion to optimize NeRFs, while later methods improve visual quality; the multi-face Janus problem remains challenging in the cited works.Image-conditioned 3D generation must also generate invisible regions; Zero123 uses multi-view-trained, 3D-aware image diffusion and alleviates the multi-face problem to some extent.Source: §2 · p. 2

4D Reconstruction

  • Dynamic-scene reconstruction uses canonical-space deformation or time-varying world-space NeRFs, while monocular methods may require teleporting cameras or quasi-static scenes.Some approaches require synchronized multi-view data, whose collection and calibration are inconvenient.Source: §2 · p. 3

4D Generation

  • Earlier 4D-generation methods use category-specific parametric models or templates, whereas text-to-4D offers zero-shot category-agnostic generation but is limited by low-quality video diffusion.The described category-specific approaches commonly condition on images or videos and require templates or per-category training data.Source: §2 · p. 3

3 PRELIMINARIES

The preliminaries describe diffusion-based supervision for optimizing NeRFs and the K-Planes representation of dynamic scenes. Together, these provide context for the paper’s use of image diffusion and dynamic neural fields.

3.1 SCORE DISTILLATION SAMPLING FOR IMAGE-TO-3D

  • SDS uses a 2D diffusion model as a prior for NeRF optimization, perturbing the rendered image with timestep-dependent noise and estimating an update toward a higher-density region.For image-to-3D, Zero123 supplies viewpoint-conditioned target views from an input image and relative camera extrinsics, without a text prompt.Source: §3.1 · p. 3

3.2 K-PLANES

  • K-Planes factorizes a dynamic 3D volume into six spatial and spatiotemporal feature planes, projects each 4D point onto them, and combines their features to predict color and density.The planes are bilinearly sampled, their features are combined by element-wise multiplication, and M × M × F describes each plane’s memory layout.Source: §3.2 · pp. 3–4
  • The Consistent4D schematic shows diffusion-supervised Cascade DyNeRF with ICL, followed by a GAN-trained cross-frame enhancer for rendered video.The caption describes ICL as compensating for spatiotemporal inconsistency from image diffusion.Source: Figure 2 · p. 4

4 METHOD

Consistent4D generates 360° dynamic objects from uncalibrated monocular video using DyNeRF supervised by a pre-trained 2D diffusion model, with a GAN-trained video enhancer for post-processing. Cascade DyNeRF and interpolation-driven regularization address temporal continuity and spatiotemporal consistency.

4 METHOD

  • The pipeline renders 360° views of a dynamic object from uncalibrated monocular video using DyNeRF supervised by Zero123 and a GAN-trained video enhancer.The enhancer is a post-processing step.Source: §4 · p. 4

4.1 CASCADE DYNERF

  • Cascade DyNeRF uses low-resolution planes for a temporally coherent coarse object, then high-resolution planes learn residuals to balance continuity and image detail.Losses apply to each scale’s renderings, improving temporal consistency without sacrificing much object quality.Source: §4.1 · pp. 4–5

4.2 INTERPOLATION-DRIVEN CONSISTENCY LOSS

  • Interpolation-driven Consistency Loss matches rendered frames to a pre-trained interpolator’s predictions, enhancing frame continuity and spatiotemporal consistency.Its spatial version also alleviates the multi-face problem in 3D generation.Source: §4.2 · p. 5

4.3 CROSS-FRAME VIDEO ENHANCER

  • A lightweight GAN-trained video enhancer targets blurry edges, floaters, and insufficient smoothness, using cross-frame attention to query information from adjacent frames.Its real images come from an image-to-image technique using a super-resolution diffusion model, while rendered images serve as fake images.Source: §4.3 · pp. 5–6

5 EXPERIMENT

The experiments evaluate Consistent4D on synthetic and in-the-wild videos, compare it with dynamic NeRF methods, and ablate its components.

5 EXPERIMENT

  • Evaluation spans synthetic and in-the-wild videos; each input is foreground-segmented and sampled uniformly at 32 frames.Most input videos are about two seconds long, with some around one second or longer than five seconds.Source: §5 · p. 6, §5.1 · p. 6

5.1 IMPLEMENTATION DETAILS

  • Training uses coarse and fine DyNeRF levels, then shifts from batch size 4 at resolution 64 to batch size 1 at resolution 256; ICL runs in the first 5,000 iterations with 25% probability.The coarse and fine spatial and temporal resolutions are 50 and 8, and 100 and 16, respectively.Source: §5.1 · p. 6

5.2 COMPARISONS WITH OTHER METHODS

  • Because few methods target 4D generation from static-camera video, comparisons are limited to D-NeRF and K-planes.Source: §5.2 · p. 6
  • Consistent4D achieves the best per-frame LPIPS score and CLIP similarity among the compared methods; Figure 3 shows novel-view renderings at two timestamps.The quantitative evaluation averages scores over four testing views for seven animated models.Source: §5.2 · pp. 6–7, Figure 3 · p. 7

5.3 ABLATIONS

  • Replacing Cascade DyNeRF with the original K-planes architecture destabilizes training and can yield incomplete or blurry objects or unclear moving parts.Cascade DyNeRF instead exhibits stable training and relatively satisfactory results.Source: §5.3 · p. 7, Figure 4 · p. 8
  • Using both spatial and temporal ICL yields superior results, and users preferred ICL in 75% of 20-object cases after cases where both methods failed were filtered out.Source: §5.3 · p. 8
  • For DreamFusion text-to-3D, users preferred ICL across success criteria, and qualitative comparisons show reduced multi-face artifacts.The stated criteria include prompt alignment, absence of multi-face issues, and geometry and texture clarity.Source: §5.3 · pp. 8–9
  • The cross-frame video enhancer improves uneven color distribution, smooths rough edges, and removes some floaters.Source: §5.3 · p. 9

6 CONCLUSION

Experiments on synthetic and in-the-wild data demonstrate Consistent4D’s effectiveness. The framework generates 360° dynamic objects from stationary monocular video using a Cascade DyNeRF, an interpolation-driven consistency loss, and a post-processing video enhancer.

6 CONCLUSION

  • Consistent4D targets 360° dynamic-object generation from uncalibrated monocular videos captured by a stationary camera.Source: §6 · p. 9
  • Cascade DyNeRF supports stable training under discrete diffusion supervision, while Interpolation-driven Consistency Loss is introduced to enhance spatial and temporal consistency.Source: §6 · p. 9
  • A lightweight post-processing video enhancer rectifies scattered color discrepancies and eliminates minor floating artifacts.Source: §6 · p. 9
  • Experiments on both synthetic and in-the-wild data demonstrate the effectiveness of Consistent4D.Source: §6 · p. 10

A.1 ADDITIONAL VISUALIZATION RESULTS

Figure 6 presents results on four in-the-wild videos: robot dancing, squirrel feeding, toy-Spiderman dancing, and toy-rabbit deforming.

A.1 ADDITIONAL VISUALIZATION RESULTS

  • Figure 6 shows results from four in-the-wild videos.Source: §A.1 · p. 13
  • The examples depict robot dancing and squirrel feeding, alongside toy-Spiderman dancing and toy-rabbit deforming.Source: §A.1 · p. 13
  • The authors recommend watching the attached videos to see the visualizations.Source: §A.1 · p. 13

A.2 DATA USED IN VIDEO-TO-4D QUANTITATIVE EVALUATION

The appendix presents additional dynamic-object visualizations and a video-enhancer demonstration. It notes that the additional objects show observations similar to the main-paper results.

A.2 DATA USED IN VIDEO-TO-4D QUANTITATIVE EVALUATION

  • Figure 7 visualizes four additional dynamic objects, with observations described as similar to the main-paper results.Source: §A.2 · p. 13
  • Figure 11 demonstrates the effectiveness of the video enhancer.Source: §A.2 · p. 13
  • The enhancer analysis is in the main paper.Source: §A.2 · p. 13

A.3 THE NUMBER OF FRAMES

Experiments sample each input video to 32 frames, although using the unsampled videos sometimes gives slightly better results.

A.3 THE NUMBER OF FRAMES

  • All experiments sample each input video to 32 frames.Source: §A.3 · p. 13
  • Using input videos without sampling sometimes gives slightly better results, as shown in Figure 8.Source: §A.3 · p. 13

A.4 IMPLEMENTATION DETAILS

The appendix specifies implementation choices for the video-to-4D losses, enhancer, and text-to-3D experiments. It also documents failure boundaries, including motion-related video-to-4D failures and multi-face text-to-3D cases that ICL does not resolve.

A.4 IMPLEMENTATION DETAILS

  • The video-to-4D objective combines SDS, ICL, reconstruction, mask, normal-smoothness, and orientation losses; reconstruction and SDS are applied alternately.The appendix gives weights λ1=0.1, λ2=2500, λ3=500, λ4=50, λ5=2.0, and increases λ6 from 1 to 20 over 5000 iterations.Source: §A.4 · p. 13
  • The video enhancer modifies pix2pix’s unet256 into a lightweight network with three up/down layers and one cross-frame attention layer; for real images, DeepFloyd-IF stage II resizes rendered 64 × 64 inputs to 256 × 256.Source: §A.4 · p. 13
  • Text-to-3D experiments use Threestudio and train for 5000 iterations, applying ICL during the first 1000 iterations with 30% probability and weight 2000.The batch size and resolution change after the first 1000 iterations from 8 and 64 to 2 and 256.Source: §A.4 · p. 13

A.5 FAILURE CASES

  • Complex or abrupt motion can defeat video-to-4D generation: an occluded dog tail disappears, and a rapidly jumping frog appears blurry.Source: §A.5 · p. 13
  • Figure 6 visualizes the method on four in-the-wild input videos at novel views 22.5° and 112.5° from the input view, showing RGB, normal-map, and depth-map results left to right.Source: Figure 6 · p. 14
  • Figure 8 compares sampled 32-frame videos with unsampled inputs of 72 frames for Aurorus and 39 for minions, showing RGB and normal maps.Source: Figure 8 · p. 15
  • Some text-to-3D multi-face cases remain unalleviated by ICL, despite the reported improvement in DreamFusion success results with ICL.Source: §A.5 · p. 16
Loading 2311.02848v1…