PaperlayerView exact revision on arXiv

Source-linked AI summary

Unsupervised Volumetric Animation

Aliaksandr Siarohin, Willi Menapace, Ivan Skorokhodov, Kyle Olszewski, Jian Ren, Hsin-Ying Lee, Menglei Chai, Sergey Tulyakov

arXiv:2301.11326v1cs.CV

TL;DR

Prior unsupervised animation methods use 2D representations, limiting their ability to model 3D effects such as occlusions and viewpoint changes. This work learns canonical volumetric object structure and parts from single-view data, then evaluates geometry and animation across image and video datasets.

  • Problem

    Prior unsupervised animation methods use 2D motion representations, which are limited in modeling 3D effects such as occlusions, viewpoint changes, and extreme rotations.

    Source: §1 · p. 2

  • Method

    The framework learns canonical voxel-based object volumes and parts, using predicted 2D keypoints with differentiable PnP to estimate part poses.

    Source: §3 · p. 3

  • Results

    The method learns high-quality geometry, parts, segmentation, and normals from real monocular videos, and supports animation and novel-view synthesis; evaluation includes VoxCeleb and TEDXPeople videos and Cats still images.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    The learned geometry and structure open possibilities for future fine-grained shape control or relighting objects in new environments.

    Source: §6 · p. 9

    Volumetric rendering has a resolution limitation relative to state-of-the-art 2D animation methods.

    Source: §5 · p. 9

Abstract

from arXiv · show

We propose a novel approach for unsupervised 3D animation of non-rigid deformable objects. Our method learns the 3D structure and dynamics of objects solely from single-view RGB videos, and can decompose them into semantically meaningful parts that can be tracked and animated. Using a 3D autodecoder framework, paired with a keypoint estimator via a differentiable PnP algorithm, our model learns the underlying object geometry and parts decomposition in an entirely unsupervised manner. This allows it to perform 3D segmentation, 3D keypoint estimation, novel view synthesis, and animation. We primarily evaluate the framework on two video datasets: VoxCeleb $256^2$ and TEDXPeople $256^2$. In addition, on the Cats $256^2$ image dataset, we show it even learns compelling 3D geometry from still images. Finally, we show our model can obtain animatable 3D objects from a single or few images. Code and visual results available on our project website, see https://snap-research.github.io/unsupervised-volumetric-animation .

1. Introduction

Prior unsupervised animation methods model motion in 2D, limiting their ability to represent 3D effects; this work learns object structure and motion from single-view data to animate in 3D. Its framework combines a canonical volumetric representation, learned parts, and keypoint-based pose estimation, and is evaluated on images and videos.

1. Introduction

  • Prior unsupervised animation methods use 2D motion representations, limiting their ability to model occlusions, viewpoint changes, and extreme rotations.These effects require accounting for the observed objects' 3D nature.Source: §1 · p. 2
  • Learning 3D parts and controlling them from only 2D videos is challenging, as is modeling the camera distribution without relying on existing pose predictors.The authors also identify the lack of a tractable 3D counterpart to the inductive bias of 2D CNNs used in unsupervised keypoint detection.Source: §1 · p. 2
  • The framework maps object embeddings to canonical voxel volumes, assigns points to softly defined rigid parts, and uses differentiable PnP to estimate part poses from predicted 2D keypoints.Linear blend skinning deforms the volume according to the part poses, while photometric reconstruction drives learning.Source: §1 · p. 2
  • The learned parts correspond to recognizable constituents, including facial features and body regions such as the torso, head, and hands.The parts are learned without supervision.Source: §1 · p. 2
  • A two-stage strategy first learns overall geometry with one part, then discovers additional parts for non-rigid animation and novel-view synthesis.The single-part stage supports 3D reconstruction and novel-view synthesis.Source: §1 · p. 2
  • The method is evaluated on VoxCeleb and TEDXPeople videos, and its geometry learning is also demonstrated on still images of cat faces.The authors introduce metrics for novel-view synthesis and animation quality because the task uses single-view data.Source: §1 · p. 2

2. Related work

Related work spans 3D-aware synthesis and reconstruction, supervised animation, and unsupervised 2D animation. The paper contrasts its unsupervised learning of geometry and parts with methods that depend on camera information, annotations, or category-specific models.

2. Related work

  • Many 3D-aware synthesis methods require ground-truth camera distributions or poses, whereas this work learns geometry and parts without supervision in a non-adversarial framework.The camera information provides a strong inductive bias for recovering 3D geometry.Source: §2 · p. 2
  • Prior unsupervised 3D reconstruction methods use annotations such as template shapes, camera poses, keypoints, optical flow, or masks; this method obtains even object masks from video without supervision.Earlier deformable-object approaches also faced limits in scaling to large video datasets or rendering quality.Source: §2 · p. 3
  • Supervised 3D animation can require fitting a body model to every frame and often does not support multiple identities in one framework, unlike this method's canonical animation-ready object space.Supervised animation also relies on fitted predictors or morphable models and is typically designed for a single object category.Source: §2 · p. 3
  • Unsupervised image-animation methods share photometric reconstruction and category flexibility, but focus on motion representations such as kinematic trees, thin-plate splines, or latent motion spaces.A latent-space direction reported for novel views cannot be used reliably to synthesize them.Source: §2 · p. 3
  • Mixed supervision schemes that supervise pose while learning expression work for faces but have not generalized to other categories.Source: §2 · p. 3

3. Method

The method reconstructs each frame by mapping an identity embedding to a canonical volume, estimating part poses from image keypoints, deforming the volume, and rendering it. Training first learns geometry with one part, then adds parts for animation; unseen identities are fitted by optimizing their embeddings and appearance.

3.1. Canonical Voxel Generator

  • The voxel generator maps an optimized identity embedding to canonical density, RGB radiance, and soft assignments of volume points to rigid parts.The method assumes a known identity for each video, assigned consistently across its frames.Source: §3.1 · p. 3

3.2. Unsupervised Pose Estimation

  • A CNN predicts 2D keypoints corresponding to learned canonical 3D keypoints, and differentiable PnP recovers each part’s pose.The 3D keypoints are shared across dataset identities, enabling poses estimated from one identity to animate another.Source: §3.2 · p. 4

3.3. Volumetric Skinning

  • Linear blend skinning maps canonical points to the deformed volume using soft part weights and each part’s rigid transform.The inverse-weight approximation is exact for strict single-part assignments when the deformed object has no self-penetration.Source: §3.3 · p. 5

3.4. Volumetric Rendering

  • Differentiable volumetric rendering integrates density and radiance along camera rays, using fixed camera extrinsics and intrinsics during training.The background is modeled as a stationary, fixed high-density plate.Source: §3.4 · p. 5

3.5. Training

  • A two-stage schedule first learns object geometry with one part, then introduces 10 parts for pose learning and animation.The second stage copies first-stage weights and initializes added skinning weights to zero.Source: §3.5 · p. 5
  • Training combines perceptual reconstruction, background, equivariance, and keypoint reprojection losses with equal weights; the second stage omits the background loss.The coarse background-mask loss is reduced over epochs and is needed only in early iterations.Source: §3.5 · pp. 5–6

3.6. Inference

  • For an unseen identity, optimizing its embedding against reconstruction loss yields geometry; generator fine-tuning improves texture while regularizing density and part assignments.The keypoint predictor remains fixed so motion can transfer from other objects.Source: §3.6 · p. 6

4. Experiments

The experiments assess geometry, animation, and novel-view synthesis on image and video datasets, including tests designed for settings without ground-truth animated images. They compare UVA with prior methods and evaluate key design choices and geometry on synthetic images.

4. Experiments

  • The evaluation covers 9,993 cat-face images, 19,522 VoxCeleb training videos, and 40,896 TEDXPeople training videos.The reported test sets contain 200 cat images and 100 videos from each video dataset.Source: §4 · p. 6

4.2. Animation Evaluation

  • AYD measures yaw accuracy against the rendered camera angle, while ASC and APC assess identity and pose preservation across frontal and rotated views.Lower ASC and APC indicate better shape and pose preservation, respectively.Source: §4.2 · pp. 6–7

4.1. Geometry from Image Data

  • 0.63 vs 0.53 Pearson depth correlation: UVA exceeds EpiGRAF without camera supervision during UVA training.EpiGRAF often fails to render accurate depth for non-frontal cameras.Source: §4.1 · p. 6, Figure 3 · p. 7

4.2. Animation Evaluation

  • Novel-view tests adapt 2D baselines using depth-lifted keypoints, while LIA’s VoxCeleb yaw is generated from a correlated latent component.The adaptation estimates new affine transformations from reprojected keypoints; LIA’s latent component also correlates with other motion.Source: §4.2 · p. 7
  • Standard reconstruction metrics favor 2D methods and do not reflect complex 3D transformations; multiple source frames can instead be pooled at inference.The evaluation also notes that subjects in individual VoxCeleb and TEDXPeople videos typically have limited 3D rotations.Source: §4.2 · pp. 7–8
  • UVA offers the most meaningful rotations by AYD and best preserves shape and expressions by ASC and APC; its five-frame reconstruction is on par with baselines.When source and driving poses differ substantially, MRAA is less consistent and produces occlusion artifacts.Source: Table 1 · p. 8, §4.2 · p. 8

4.3. Ablation Studies

  • The full model scores best; removing the geometry phase corrupts results and parts, while removing background loss prevents foreground-background separation.The direct part-pose variant learns essentially flat geometry and cannot proceed to the multi-part phase.Source: §4.3 · p. 8, Table 2 · p. 8

4.4. Geometry Evaluation for Synthetic Objects

  • 0.793 vs 0.602 foreground-depth Pearson correlation on synthetic faces and 0.568 vs 0.470 on bodies for UVA and Omnidata, respectively, despite real-only training.Models pretrained on real VoxCeleb or TEDXPeople images were evaluated on synthetic images, with background depth masked out.Source: §4.4 · pp. 8–9

5. Limitations

The method has limits in representation, inference cost, rendering resolution, and the geometry it learns from training views.

5. Limitations

  • A 64^3 voxel cube can produce seam-like artifacts in novel views with large camera displacements.The authors attribute these artifacts to the cube’s small size and errors in predicting part distances, which may slightly displace parts.Source: §5 · p. 9
  • Test-time embedding optimization and generator fine-tuning increase inference cost, though the cost is amortized when generating many frames for an identity.Source: §5 · p. 9
  • Frames are rendered at 256×256, below resolutions typically supported by state-of-the-art 2D animation methods.Source: §5 · p. 9
  • Geometry is learned only from training views, leaving the back of VoxCeleb faces and TEDXPeople bodies without precise geometry.Source: §5 · p. 9

6. Conclusion

The method learns object geometry and structure from real-world monocular 2D videos without explicit labels, and uses that representation for animation and novel-view synthesis.

6. Conclusion

  • Trained exclusively on real-world monocular 2D videos, the method obtains geometry, object parts, 3D segmentation, and normals.Source: §6 · p. 9
  • Without explicit labels or other cumbersome supervision, the same approach applies to a variety of object categories.Source: §6 · p. 9
  • The learned geometry and structure support animation and novel-view synthesis at the same time.Source: §6 · p. 9
  • The authors identify fine-grained shape control and relighting for composition into new environments as directions for future exploration.Source: §6 · p. 9

A. Implementation details

The implementation combines voxel-based object generation, fixed-camera volumetric rendering, differentiable keypoint-based pose estimation, and staged training and inference.

A. Implementation details

  • The generator maps 64-dimensional identity embeddings to 64^3 voxel volumes with density, RGB, and linear-blend-skinning features.It uses a U-Net keypoint predictor, a fully connected voxel-generation layer, four upsampling 3D residual blocks, and a final projection layer.Source: A. Implementation details · p. 14
  • Rendering uses fixed camera extrinsics and intrinsics, samples 128 points per ray, and reduces sampling to 48 points during the single-part generator phase.The rendering cube is scaled by 1.2 for Cats and 1.075 for other datasets; training also perturbs ray positions and sampled densities.Source: A. Implementation details · p. 14
  • Differentiable EPnP estimates part poses from 125 keypoints per part, while a density-based loss pulls parts with no density toward the pose of the largest part.The method uses 10 parts, for 1,250 keypoints total; during single-part training, the identity matrix replaces the largest-part pose.Source: A. Implementation details · p. 14
  • Training proceeds in two phases: first learning overall geometry with one part, then discovering additional parts for animation.Video datasets use 200 epochs followed by 1,000; Cats uses 1,000 epochs in the first phase, and TEDXPeople training adds unsupervised MRAA co-part segmentation loss in phase two.Source: A. Implementation details · p. 14
  • Test-image inference first optimizes an identity embedding, then fine-tunes the generator with geometry regularization to preserve the learned geometry prior.The second stage adds random Euclidean transformations; it runs for 500 steps with one image and 3,000 steps with five input frames.Source: A. Implementation details · pp. 14–15
  • Inference-time filtering smooths abrupt part-depth changes, which become more noticeable at extreme camera angles.The method rescales each part’s camera-to-rendering-cube vector to have the same length across frames.Source: A. Implementation details · p. 15

B. Dataset details

The training data comprise VoxCeleb, TEDXPeople, and Cats, with TEDXPeople samples filtered and segmented into video clips.

B. Dataset details

  • The method uses VoxCeleb, TEDXPeople, and Cats for training, applying established preprocessing to VoxCeleb and Cats.Source: B. Dataset details · p. 15
  • TEDXPeople videos are divided into clips of up to 512 frames, with human detections and bounding boxes used to crop upper-body subjects.Samples are stopped or restarted when detections fail or bounding boxes shift substantially.Source: B. Dataset details · p. 15
  • TEDXPeople filtering excludes short, small, static, background-moving, or unsuitable-aspect-ratio clips; the final dataset contains 40,896 samples from 17,451 YouTube videos.Only front views are selected, with at most three samples taken from each video.Source: B. Dataset details · p. 15

C. Metric details

The evaluation introduces metrics for novel-view quality when multi-view observations are unavailable. It measures camera yaw, identity preservation, and pose preservation across rendered views.

C. Metric details

  • Each metric uses a triplet of source, driving, and target-camera rendered frames to evaluate novel-view quality without multi-view observations.For pure novel-view evaluation, the first frame serves as both source and driving frame; 256 camera angles are sampled from −π/2 to +π/2.Source: C. Metric details · pp. 15–16
  • AYD measures whether the rendered object matches the target camera perspective, while ASC and APC measure identity and pose preservation, respectively.Lower ASC indicates better identity preservation during rotation; lower APC indicates better pose preservation.Source: C. Metric details · p. 15
  • For faces, yaw is estimated with 6DRepNet, while DECA supplies FLAME identity parameters and expression-plus-jaw pose parameters.The face identity code has 100 shape parameters; the pose code has 53 elements.Source: C. Metric details · pp. 15–16
  • For bodies, 3DCrowdNet fits SMPL using detected keypoints, and evaluation uses 10 shape parameters and 13 joints above the belly button.The selected joints exclude those typically absent from TEDXPeople frames; yaw is extracted from the root-joint rotation.Source: C. Metric details · p. 16
  • Animation evaluation uses each video's first frame as source and five evenly spaced driving frames, rendered at predefined camera rotations.The angle set includes 0 and positive and negative rotations through π/4; the resulting triplets are used to compute AYD, ASC, and APC.Source: C. Metric details · p. 16

D. Baseline details

The baselines adapt 2D motion representations for novel views, while LIA maps yaw changes into its learned motion space. Table 3 reports novel-view results across camera angles.

D. Baseline details

  • MRAA and FOMM estimate new affine transformations for novel views by lifting region keypoints with depth and projecting them to the target camera.Four nearby keypoints are sampled around each region center; absolute depth is used for animation to preserve the driving-frame pose.Source: D. Baseline details · p. 16
  • Because the off-the-shelf depth estimator provides relative depth, the baseline aligns it to the method's depth map with a linear mapping.The mapping uses dscale and dshift parameters; the passage describes a closed-form solution and identifies the sample mean, variance, and covariance in its equation.Source: D. Baseline details · p. 16
  • For VoxCeleb, LIA's second learned motion direction correlates with y-axis head rotation, but is entangled with other motion components.The baseline fits a2 = 7.453Θ using test-set yaw changes, then uses that magnitude along the second direction to generate novel views; AYD may therefore be biased low.Source: D. Baseline details · pp. 16–17
  • Table 3 reports novel-view results for first video frames rendered at camera angles from −90° to +90°.The supplied caption gives the table's scope but not its cell values.Source: Table 3 · p. 17

E. Novel view synthesis

In novel-view synthesis without animation, the methods trade off camera control against preservation of identity and pose. The proposed model has the best AYD.

E. Novel view synthesis

  • The proposed model achieves the best AYD, suggesting the most accurate camera manipulation in novel-view synthesis without animation.MRAA has favorable ASC and APC but high AYD, while LIA has low AYD but high ASC and APC.Source: E. Novel view synthesis · p. 17
  • MRAA's favorable identity and pose consistency comes from translating the subject rather than rotating it to match the requested yaw.This preserves pose and identity at the cost of poor novel-view synthesis.Source: E. Novel view synthesis · p. 17
  • LIA's low AYD accompanies high ASC and APC errors, consistent with latent motion directions that entangle viewpoint changes with pose and identity.The passage says these entangled directions prevent novel-view synthesis without significantly altering pose and identity.Source: E. Novel view synthesis · p. 17

F. Canonical visualization

The canonical-space visualization presents objects in a shared pose. The camera is selected to make the objects look reasonable because their placement in the rendering cube is not constrained.

F. Canonical visualization

  • The visualization selects a camera that makes objects look reasonable, since the model does not constrain their placement in the rendering cube.The canonical pose sets Tp to the identity matrix for every part.Source: F. Canonical visualization · p. 17
  • Figure 7 shows objects sharing the same pose, which the paper identifies as crucial for animation.The same Tp is used for all parts and objects.Source: F. Canonical visualization · p. 17, Figure 7 · p. 17

G. Failed Experiments

Failed experiments identify trade-offs in representation, pose estimation, and training choices; the ethical discussion connects synthesis behavior and deployment concerns to dataset coverage and possible misuse.

G. Failed Experiments

  • Triplanar and VM representations favored flat geometry, MLP was slow, and upsampling was faster and lighter but reduced geometry detail and view consistency.Source: G. Failed Experiments · pp. 17–18
  • Direct pose architectures failed to yield meaningful geometry, while per-frame pose optimization worked for one video but became infeasible at millions of frames.Source: G. Failed Experiments · p. 17
  • Declarative PnP layers were too slow, Kornia DLT diverged, and EPnP worked in PyTorch 10.1 but not 11 because pinverse gradients were unstable.Source: G. Failed Experiments · p. 17
  • Depth and normal supervision improved one-phase training but was unnecessary with two-phase training; normal losses also required a custom CUDA second-derivative kernel.Source: G. Failed Experiments · pp. 17–18
  • Alternative multipart radiance and density volumes produced reasonable results but tended to use only one or two parts when discovering many parts.Source: G. Failed Experiments · p. 18
  • Few-shot NeRF regularizers—including entropy, MiP NeRF weights, surface normals, and warping losses—were unnecessary with two-phase training.Source: G. Failed Experiments · p. 18
  • The novel-view discriminator reduced geometry quality because suitable object-pose rotation ranges were difficult to find.Source: G. Failed Experiments · p. 18

H. Ethical considerations

  • Demographic variation in synthesis depends on dataset coverage, while realistic performance-driven animation also carries misuse risk, including deceptive videos of public figures.Source: H. Ethical considerations · p. 18
Loading 2301.11326v1…