Source-linked AI summary
Learning elementary structures for 3D shape generation and matching
Theo Deprelle, Thibault Groueix, Matthew Fisher, Vladimir G. Kim, Bryan C. Russell, Mathieu Aubry
TL;DR
Existing surface-parametric methods rely on manually selected primitives, motivating a way to learn shared structures for diverse 3D shapes. The paper learns elementary structures through patch deformation or point translation, then adjusts and combines them for reconstruction and matching. It reports a 16% improvement over AtlasNet for generic object reconstruction and a 6% improvement over 3D-CODED for FAUST human-shape matching.
Problem
Surface-parametric approaches rely on hand-chosen primitives tuned to the target shape collection and task.
Method
The method learns shared elementary structures using patch deformation or point translation, then deforms and combines them through shape-dependent adjustment networks.
Results
16% relative improvement over AtlasNet is reported for generic object shape reconstruction, alongside a 6% improvement over 3D-CODED for FAUST human-shape matching.
Takeaways & Limitations
Learned structures can be reused across shapes, often correspond to interpretable parts, and support dense correspondence through points aligned with the structures.
Abstract
from arXiv · showhide
We propose to represent shapes as the deformation and combination of learnable elementary 3D structures, which are primitives resulting from training over a collection of shape. We demonstrate that the learned elementary 3D structures lead to clear improvements in 3D shape generation and matching. More precisely, we present two complementary approaches for learning elementary structures: (i) patch deformation learning and (ii) point translation learning. Both approaches can be extended to abstract structures of higher dimensions for improved results. We evaluate our method on two tasks: reconstructing ShapeNet objects and estimating dense correspondences between human scans (FAUST inter challenge). We show 16% improvement over surface deformation approaches for shape reconstruction and outperform FAUST inter challenge state of the art by 6%.
1 Introduction
The paper learns shared elementary 3D structures from shape collections instead of relying on manually chosen primitives, then deforms and combines them for reconstruction and matching. It explores patch deformation and point translation learning, reporting improvements on both tasks.
- Motivation: Surface-parametric methods use hand-chosen primitives tuned to the target shape collection and task.The paper asks what primitive set best represents diverse shapes.
- Contribution: Learnable elementary structures are automatically learned from shape collections and reused for shape reconstruction and matching.They often correspond to recurrent object parts, such as airplane tails, wings, and reactors.
- Approach: Patch deformation learning deforms a surface element, such as a 2D square, into continuous structures shared across shapes.Unlike AtlasNet, the learned deformations and elementary structures are common to all shapes.
- Approach: At inference, learned structures are positioned by adjustment transformations, separating structure shape from pose.Linear transformations favor interpretability, while multilayer perceptrons generalize prior reconstruction methods using learned templates.
- Results: 16% relative improvement over AtlasNet is reported for generic object shape reconstruction.The paper also reports a 6% improvement over 3D-CODED for human shape matching on FAUST.
2 Related Work
Prior work spans independently fitted geometric primitives, consistent primitive fitting, deformable meshes, and neural template fitting. These approaches face trade-offs involving consistency, optimization, template sensitivity, and model complexity.
- Primitive fitting: Classical primitive fitting targets parsimonious approximations using forms such as generalized cylinders and geons.Independent fitting does not consistently reuse primitives across objects, limiting common-structure discovery and correspondence estimation.
- Primitive fitting: Collection-level primitive and part methods seek consistency across shapes but often involve non-convex optimization, heuristics, slow solutions, and local optima.These limitations complicate consistent structure discovery.
- Learning-based fitting: Learning-based methods replace some hand-crafted heuristics by predicting primitive parameters from reconstruction loss or using supervised fitting systems.Recent extensions use anisotropic 3D Gaussians and category-specific morphable models.
- Learning-based fitting: Diverse primitive sets and constructive solid geometry graphs can improve reconstruction but usually do not fit shapes consistently across a collection.Consequently, they cannot support common shape-structure discovery or inter-shape relationships.
- Deformable models: Deformable mesh models capture complex structures but are prone to local optima because they have many degrees of freedom.Neural mesh fitting also depends on the chosen template, motivating learned reference structures.
3 Approach
The approach learns shared elementary structures across a shape collection, then deforms and positions them with shape-dependent adjustment modules for reconstruction and matching. It supports correspondence-supervised and correspondence-free training, with alternative structure-learning, adjustment, and dimensionality choices.
- Shared elementary structures: The model learns K shared elementary structures over the training set and adjusts them at test time to produce each output shape.The structures are independent of the test input before adjustment, while the adjustment modules are shape-dependent.
- Elementary-structure learning: Two learning modules are considered: patch deformation learns continuous surfaces, while point translation independently translates sampled points.Patch deformation applies a learned mapping to sampled points; point translation adds a learned vector to each point.
- Structure dimensionality: The framework supports 2D, 3D, and 10D elementary structures, with higher-dimensional structures yielding better reconstruction but lower interpretability.The paper reports this trade-off across dimensionality choices.
- Structure adjustment: Adjustment modules position each elementary structure, using either affine transformations or more complex MLP mappings predicted from an encoded input shape.The shape encoder is a simplified PointNet that produces a global feature for the adjustment modules.
- Training with correspondences: With correspondence supervision, a single elementary structure is trained against ordered points; at inference, order-invariant encoding enables correspondence prediction between reconstructed test shapes.Reconstructed points remain aligned with the elementary structure and therefore with one another.
- Training without correspondences: Without correspondence supervision, the method starts from K unit 2D square patches and minimizes symmetric Chamfer distance between the output and target point clouds.For these experiments, K = 10.
4 Experiments
Experiments evaluate learned elementary structures for generic objects and human shapes across reconstruction, category generalization, and correspondence tasks. The results show benefits from point translation learning, flexible MLP adjustments, higher-dimensional structures, and deformation learning for correspondence estimation.
- Experiments evaluate shape reconstruction and shape matching on generic 3D objects and human shapes.Generic-object experiments use ShapeNet; human experiments use SURREAL and FAUST.
- Generic object shape reconstruction: The ShapeNet evaluation reports symmetric Chamfer distance and compares against AtlasNet with ten unit-square primitives.Single-category, multi-category, and category-generalization settings are included.
- Generic object shape reconstruction: Multi-class reconstructions achieve higher fidelity than AtlasNet, which incorrectly synthesizes geometry in hollow regions between chair backs and seats.The quantitative evaluation confirms less accurate AtlasNet reconstructions.
- Generic object shape reconstruction: MLP adjustment significantly improves quantitative results over linear adjustment, while linear transformations provide more interpretability but less reconstruction flexibility.Linear adjustment limits adaptation of elementary structures; MLP adjustment enables more faithful reconstruction.
- Generic object shape reconstruction: Point translation learning outperforms patch deformation learning because patch deformations do not allow topological changes and discontinuities in mapping.Patch deformation learning instead estimates an entire deformation field, enabling arbitrary point warping or tessellation.
- Generic object shape reconstruction: Higher-dimensional elementary structures improve quantitative results, although gains diminish in higher-dimensional spaces that are harder to visualize and interpret.Learned structures can capture symmetric, topological, and meaningful-part patterns across ShapeNet categories.
- Generic object shape reconstruction: Learned structures generalize across categories: networks trained on chairs reconstruct tables, with the same structures reconstructing chair and table legs.Point translation learning outperforms patch deformation learning and AtlasNet in the reported category-generalization experiment.
- Generic object shape reconstruction: The method learns elementary-structure parameters with less than 1% additional parameters, while inference has the same complexity as AtlasNet.Additional parameters are 2.0 × 10^6 for patch deformation and 2.5 × 10^3 for point translation, versus 1.8 × 10^8 for the full network.
5 Conclusion
The paper learns common elementary structures from shape collections and deforms and composes them to reconstruct arbitrary shapes. These structures support reconstruction and correspondence estimation without explicit point supervision, while deformation learning additionally produces continuously sampleable and meshable surfaces.
- The method learns common elementary structures from training shapes and deforms and composes them to reconstruct arbitrary shapes.
- The structures are learned consistently without explicit point supervision between shapes and improve reconstruction and correspondence tasks quantitatively.
- When trained on shape categories, learned structures are often interpretable as recurring parts.
- Patch deformation learning produces continuous surfaces that can be densely sampled and meshed at test time.
- The approach opens possibilities for shape morphing and scan completion.