Source-linked AI summary
3D-CODED : 3D Correspondences by Deep Deformation
Thibault Groueix, Matthew Fisher, Vladimir G. Kim, Bryan C. Russell, Mathieu Aubry
TL;DR
Reliable correspondence estimation for noisy, non-rigid 3D shapes is difficult. The paper introduces Shape Deformation Networks that learn template-to-shape transformations through global features, with local Chamfer optimization improving alignment; it achieves 2.878 cm on FAUST-inter and reports robustness to perturbations and non-human shapes.
Problem
Reliable correspondences are needed for noisy, low-resolution 3D scans with articulated shapes and substantial non-rigid deformation.
Method
An encoder predicts a global feature and a decoder deforms a template into each target, with test-time Chamfer-distance optimization refining the feature.
Results
2.878 cm average correspondence error on FAUST-inter, an 8% improvement over state of the art, with robustness to studied perturbations except strong sampling variation.
Takeaways & Limitations
The method learns deformation parameters without a manually designed deformable template and can train without explicit correspondence supervision.
Takeaways & Limitations
When reconstruction initialization is too far from the input, the regression step fails.
Abstract
from arXiv · showhide
We present a new deep learning approach for matching deformable shapes by introducing {\it Shape Deformation Networks} which jointly encode 3D shapes and correspondences. This is achieved by factoring the surface representation into (i) a template, that parameterizes the surface, and (ii) a learnt global feature vector that parameterizes the transformation of the template into the input surface. By predicting this feature for a new shape, we implicitly predict correspondences between this shape and the template. We show that these correspondences can be improved by an additional step which improves the shape feature by minimizing the Chamfer distance between the input and transformed template. We demonstrate that our simple approach improves on state-of-the-art results on the difficult FAUST-inter challenge, with an average correspondence error of 2.88cm. We show, on the TOSCA dataset, that our method is robust to many types of perturbations, and generalizes to non-human shapes. This robustness allows it to perform well on real unclean, meshes from the the SCAPE dataset.
1 Introduction
The paper targets reliable correspondences for noisy, deforming 3D scans and introduces Shape Deformation Networks to align inputs through a learned template. The approach avoids manually designed deformable templates, can learn without correspondence supervision, and achieves state-of-the-art results with simple losses.
- Motivation: Noisy, low-resolution scans and substantial non-rigid deformation make reliable correspondences between 3D shapes difficult.The challenge is especially pronounced for articulated humans and animals with varied poses and shapes.
- Approach: Shape Deformation Networks align a shared template to each input shape, then derive correspondences between inputs through the template.The network learns template-to-target deformations rather than directly predicting point associations.
- Approach: An encoder-decoder architecture predicts a global feature for each target, while a decoder deforms the template into that target.At test time, the feature is locally optimized using Chamfer distance between the target and generated shape.
- Novelty: The method implicitly learns deformation parameters and degrees of freedom, eliminating the need for a manually designed deformable template.It can also exploit correspondence supervision when available, but is trainable without it.
- Results: Training on sufficient data enables state-of-the-art performance while using simple reconstruction and correspondence losses instead of complex multiterm objectives.The authors report that this simple approach outperforms techniques requiring more complex objective functions.
2 Related work
Prior work addresses non-rigid shape registration through intrinsic geometry, explicit templates, learned descriptors, and broader 3D representations. These approaches trade generic applicability against category-specific priors, template-design effort, optimization complexity, or reliance on handcrafted descriptors.
- Overview: Non-rigid registration studies correspondences between geometries that vary in pose and shape, with extensive prior work spanning generic and category-specific methods.The related work also covers deep learning methods for shape matching and general 3D processing.
- Generic shape matching: Generic methods commonly assume relatively consistent intrinsic structure and optimize correspondences under metric-distortion constraints.They use local descriptors and hierarchical strategies because the underlying optimization is non-convex and sensitive to initialization.
- Generic shape matching: Category-specific priors can improve matching for classes such as humans because substantial class data is available.This motivates leveraging knowledge beyond generic intrinsic geometry.
- Template-based shape matching: Template-based methods can provide strong correspondences but require careful template parameterization and typically a non-convex, multiterm fitting objective.Developing mature deformable human templates has required more than a decade of research.
- Deep learning for shape matching: Deep shape-matching methods learn point-wise descriptors from models with ground-truth correspondence, sometimes adding global context through functional maps.These methods still rely on handcrafted point-wise descriptors as inputs.
- Deep Learning for 3D data: 3D deep-learning systems operate on representations including volumetric grids, point clouds, geometry images, parameterized surfaces, folded surfaces, and chart representations.The paper situates its method within this broader range of learned 3D representations.
3 Method
The method learns to deform a template into input shapes, refines the latent representation using Chamfer distance, and transfers correspondences through the shared template. It supports supervised or unsupervised training and recovers correspondences between pairs of shapes.
- Template-driven correspondence: The method learns a transformation from a template to each input shape, then derives correspondences between inputs through the shared template.The transformation is predicted rather than learning associations directly between variable-sized point sets.
- Learning shape reconstruction: An encoder produces a global shape feature, while a decoder deforms the template into the target shape.The encoder-decoder architecture is trained end-to-end.
- Training objectives: Training supports both known-correspondence supervision and an unsupervised objective combining Chamfer reconstruction with Laplacian and edge-length regularization.The regularization terms encourage deformations toward isometries; their weights are set to 5.10^-3 in the experiments.
- Latent refinement: At test time, the predicted global feature is optimized by minimizing Chamfer distance between the reconstructed and input shapes.The refinement starts from the encoder prediction and uses gradient-based optimization.
- Pairwise matching: For two shapes, the algorithm deforms the template to each shape and matches points by nearest-neighbor searches through the corresponding template locations.The procedure first estimates parameters for both shapes, then maps a reference point through the template to the target.
4 Results
The method is evaluated across human, animal, synthetic, incomplete, and perturbed shapes, with performance depending strongly on training coverage and reconstruction initialization. It achieves strong correspondence results on FAUST, SCAPE, and TOSCA, including real partial scans and multiple perturbations.
- FAUST results: 2.878 cm average error on FAUST-inter, an 8% improvement over the state of the art, while FAUST-intra reaches 1.99 cm.
- Qualitative results: The method handles real incomplete scans, strong synthetic perturbations, and non-human shapes, as illustrated by reconstructions with correspondence colors.
- SCAPE results: 3.14 cm average Euclidean error is obtained on full SCAPE meshes, while real partial scans achieve 4.04 cm, similar to full meshes.
- TOSCA robustness: The method is robust on TOSCA to isometry, noise, shotnoise, holes, micro-holes, topology changes, and sampling, but scale requires normalization.
- Ablations: Optimizing the latent feature improves reconstruction and correspondence quality, while regular training sampling and a high-resolution template provide additional boosts.
- Training data: Synthetic training data is important: adding it reduces FAUST-inter error by a factor of 3, while missing test-like poses causes failures that augmentation alleviates.
- Unsupervised correspondences: Without correspondence supervision, regularization yields 4.88 cm error on FAUST-inter, close to Deep Functional Map’s 4.83 cm.
- Rotation invariance: Rotation-invariant training gives 3.10 cm on FAUST-inter, slightly worse than the main approach but with improved robustness and slightly better SCAPE results.
5 Conclusion
The paper concludes that an encoder-decoder architecture can generate competitive human correspondences using simple losses. Its global descriptor and template deformation are refined by gradient descent to improve correspondence quality.
- The encoder produces a global shape descriptor, while the decoder Shape Deformation Network maps template points back to the original geometry.
- A straightforward gradient-descent regression step significantly improves final correspondence quality.
6 Supplementary
Supplementary experiments examine template choice, perturbation robustness, regularization, Chamfer-loss variants, and failure cases. The method is broadly robust, but sampling changes and reconstruction or nearest-neighbor errors remain important boundaries.
- 6.1 Template study: The neutral “FAUST” template performs best among the three tested template shapes.The comparison evaluates FAUST, zero, and separated templates.
- 6.2 Robustness to perturbations: The method is robust to all studied TOSCA perturbation categories except strong sampling variation.Evaluated perturbations include noise, shotnoise, sampling, scale, local scale, topology, holes, microholes, and isometry.
- 6.3 Non-human shapes: SMAL’s cross-category correspondences allow the approach to extend to animals from different species.The paper provides qualitative evidence for this extension in Figure 10.
- 6.4 Regularization for the unsupervised case: The unsupervised Chamfer-only objective can reach a distorted local minimum, while Laplacian and edge regularization constrain generated surfaces.Chamfer-only training produced consistent template distortions; Laplacian regularization uses λlaplace = 0.005, and edge loss uses Ledges = 0.005.
- 6.6 Failure cases: The main failure sources are Euclidean nearest-neighbor matches that are geodesically distant and regression failures when reconstruction starts too far from the input.The error visualization also highlights large errors where Euclidean distance is small but geodesic distance is high.
- 6.5 Asymmetric Chamfer distance: Symmetric Chamfer optimization performs better on average, although asymmetric Chamfer can be qualitatively better for scans with holes.The comparison is reported for latent-feature searches using a low-resolution template without regular surface sampling.