Source-linked AI summary
Deformable Shape Completion with Graph Convolutional Autoencoders
Or Litany, Alex Bronstein, Michael Bronstein, Ameesh Makadia
TL;DR
The paper addresses completion of partial 3D shapes undergoing non-rigid deformation, where rigid-oriented representations struggle. It trains a graph-convolutional VAE for complete shapes and optimizes its latent representation against partial input, obtaining promising synthetic and real-scan completions while retaining multiple plausible solutions.
Problem
Non-rigid 3D shape completion remains difficult because deformations violate the rigid self-similarity assumptions of common deep-learning representations.
Method
A graph-convolutional variational autoencoder learns a latent space of complete shapes, and inference optimizes that latent space to fit a partially observed input.
Results
The method shows promising completion results on synthetic and real human body and face scans with varied articulation and partiality.
Takeaways & Limitations
The approach supports arbitrary partiality, multiple plausible completions, and shape classes beyond a single manually modeled category.
Takeaways & Limitations
The method assumes that the desired shape topology is known when decoding shapes and leaves unknown-topology completion for future work.
Abstract
from arXiv · showhide
The availability of affordable and portable depth sensors has made scanning objects and people simpler than ever. However, dealing with occlusions and missing parts is still a significant challenge. The problem of reconstructing a (possibly non-rigidly moving) 3D object from a single or multiple partial scans has received increasing attention in recent years. In this work, we propose a novel learning-based method for the completion of partial shapes. Unlike the majority of existing approaches, our method focuses on objects that can undergo non-rigid deformations. The core of our method is a variational autoencoder with graph convolutional operations that learns a latent space for complete realistic shapes. At inference, we optimize to find the representation in this latent space that best fits the generated shape to the known partial input. The completed shape exhibits a realistic appearance on the unknown part. We show promising results towards the completion of synthetic and real scans of human body and face meshes exhibiting different styles of articulation and partiality.
1. Introduction
The paper targets 3D shape completion for articulated and non-rigid objects, where conventional representations struggle because deformations violate rigid self-similarity assumptions. It separates completion from generative-model training and uses graph convolutions to produce plausible, varied mesh completions.
- Motivation: Non-rigid deformations make each pose effectively a novel object for Euclidean CNNs, increasing parameter and training requirements.Rigid self-similarity assumptions, such as a chair seat remaining parallel to the floor, do not hold under articulation.
- Approach: Graph convolutions directly exploit 3D mesh structure to learn non-rigid shape representations without an explicit model.
- Approach: Separating completion from training allows generalization to any style of partiality without training on partial shapes.This design also avoids substantial preprocessing and augmentation of training data.
- Advantages: The mesh-convolutional network produces complete and plausible surfaces by design, avoiding triangulation artifacts from point clouds or volumetric grids.
- Contributions: The method decouples partial completion from generative-model learning, supports multiple plausible solutions, and is not restricted to a specific shape class.The approach is intended to handle arbitrary partiality and 3D data, including ambiguities in deformable completion.
- Pipeline: Figure 1 shows training on complete corresponding shapes followed by latent-space optimization of the decoder against a partial input.
2. Related work
Related work spans geometric priors, model-based completion, partial correspondence, image inpainting, generative modeling, and geometric deep learning. The paper positions intrinsic graph operations as a response to deformation difficulties in Euclidean shape representations.
- 3D shape completion: Geometric priors handle small missing regions, whereas larger occlusions require stronger priors such as parametric morphable models.
- 3D shape completion: Non-rigid completion differs from rigid completion because inference may involve deformations unseen during training, especially when occlusions are large.
- Generative methods for non-rigid shapes: VAE, GAN, and VAEGAN methods have been applied to dynamic surface and image-to-shape generation, while this work uses graph convolutions rather than fully connected operations.
- Geometric deep learning: Euclidean views, voxels, and point clouds are difficult to use for deformations; voxel representations also face memory and resolution drawbacks.
- Geometric deep learning: Intrinsic CNNs define convolution on manifolds and can be invariant to isometric deformations, but graph methods must carefully define neighborhoods and filter correspondence.
- Geometric deep learning: The method builds on dynamic graph filters whose filter assignments depend on local feature values and operate directly on embedding features.
- Partial shape correspondences: Partial non-rigid correspondence supports high-level tasks and is central to this method, which matches inputs to a canonical shape.
- Inpainting: Unlike image inpainting, the proposed optimization searches directly for a complete shape without constraints on internal feature representations.
3. Method
The method trains a graph-convolutional VAE as a complete-shape generator, then optimizes its latent code and rigid alignment to fit a partial corresponding input. Random initializations expose multiple plausible completions, while the decoder imposes a mesh-based shape prior.
- 3D shape generator: A graph-convolutional VAE learns a latent representation whose decoder generates complete 3D shapes with fixed vertex count and reference topology.
- 3D shape generator: The VAE combines pointwise reconstruction loss with KL-divergence regularization toward a centered unit-variance Gaussian prior.The total loss is L = Lr + λLp.
- Design choice: The method omits discriminator losses to avoid additional model complexity, leaving alternative reconstruction losses for future work.
- Graph convolutions: Intrinsic mesh convolutions use graph neighborhoods and learned, translation-invariant edge-weight mappings based on feature differences.
- Partial shape completion: At inference, the encoder is discarded and latent optimization searches for a decoder output aligned with the partial input using dense correspondence and extrinsic dissimilarity.
- Partial shape completion: Rigid registration and non-rigid deformation alternate during inference; the former has an SVD closed form under ℓ2 dissimilarity, while the latter uses stochastic gradient descent.
- Partial shape completion: Random initializations can produce multiple plausible completions because shape completion is inherently ill-posed.
4. Experiments
Experiments evaluate the learned representation, completion variability, synthetic range-scan completion, latent-space fusion, and real-scan completion. Results show plausible latent-space behavior, generalization to unseen poses, view-independent reconstruction, and improved multi-view accuracy with latent fusion.
- Experimental setup: The DFAUST VAE uses 10 human subjects performing 14 activities, with meshes subsampled to N = 3446 vertices.The final DFAUST design uses M = 8 and a latent dimensionality of 128.
- Representation quality: With λ = 10−8, the VAE prioritizes representing a wide variety of shapes accurately over realistic prior samples.Random samples from the relaxed Gaussian prior may therefore be slightly unrealistic.
- Representation quality: Latent interpolation produces nonlinear 3D shape transitions, while latent arithmetic learns pose symmetry without explicit modeling.The reported example transfers a knee-motion difference between subjects and changes the corresponding knee.
- Completion variability: Random latent initializations produce different plausible completions when large regions such as limbs are missing.This reflects the non-uniqueness of completion for ambiguous partial inputs.
- Synthetic range scans completion: The method generalizes well to partial shapes in poses unseen during training, while noisy correspondences create a performance gap relative to oracle correspondences.Recalculating correspondences after SGD plateaus improves completions when optimization becomes poorly aligned.
- Dynamic Fusion: Latent-space fusion consistently improves reconstruction accuracy for three partial views and produces fused shapes for dynamic multi-view inputs.For dynamic inputs, averaging completed-shape latent variables yields an interpolated pose that may differ from every input pose.
- Real range scan completion: The method is also applied to Kinect point clouds and real DFAUST scans, with completed-shape examples shown for both data sources.The real-scan evaluation includes depth images, extracted point clouds, and completed shapes.
5. Conclusions and future work
The paper presents graph-convolutional shape completion as a promising approach for real-world scans, while identifying correspondence quality and known topology as important boundaries.
- Conclusions: The method is a promising first step toward completing real-world scans with a graph-convolutional approach.The evaluations cover synthetic and real scans, including human body and face meshes.
- Future work: Improving initialization for noisy real-world data remains future work because the method requires correspondences between partial and canonical shapes.The paper reports resilience to poor correspondences but identifies improved initialization as beneficial.
- Future work: Completion currently assumes that the desired shape topology, including vertex connectivity, is known during decoding.The paper leaves completion with unknown topology to future work.