Source-linked AI summary
Learning Shape Templates with Structured Implicit Functions
Kyle Genova, Forrester Cole, Daniel Vlasic, Aaron Sarna, William T. Freeman, Thomas Funkhouser
TL;DR
The paper addresses the need for general shape templates that avoid libraries of hand-made templates for varied geometry and topology. It learns structured implicit functions as compositions of local shape elements, and reports consistent template fitting across diverse shapes with applications including correspondence, interpolation, and semantic segmentation.
Problem
Existing methods generally use libraries of hand-made templates, motivating a learned general template for shapes with varied geometry and topology.
Method
The method learns a template whose implicit surface is a level set of a volumetric function composed of locally influential shape elements.
Results
The encoder-decoder network learns template elements that map detailed surface geometry consistently across related shapes with large variations.
Takeaways & Limitations
The learned template supports shape clustering, exploration, abstraction, correspondence, interpolation, and image segmentation.
Takeaways & Limitations
The fixed set of axis-aligned elements has limited ability to represent sharp or angled structures, razor-thin structures, and one-to-one semantic components.
Abstract
from arXiv · showhide
Template 3D shapes are useful for many tasks in graphics and vision, including fitting observation data, analyzing shape collections, and transferring shape attributes. Because of the variety of geometry and topology of real-world shapes, previous methods generally use a library of hand-made templates. In this paper, we investigate learning a general shape template from data. To allow for widely varying geometry and topology, we choose an implicit surface representation based on composition of local shape elements. While long known to computer graphics, this representation has not yet been explored in the context of machine learning for vision. We show that structured implicit functions are suitable for learning and allow a network to smoothly and simultaneously fit multiple classes of shapes. The learned shape template supports applications such as shape exploration, correspondence, abstraction, interpolation, and semantic segmentation from an RGB image.
1. Introduction
The paper learns a general shape template from data using structured implicit functions, enabling consistent fitting across varied geometry and topology. The learned element structure supports correspondence, interpolation, abstraction, exploration, and image segmentation.
- Existing template methods usually require multiple hand-made templates and a procedure for selecting the appropriate one across varied shapes.
- Structured implicit functions represent a shape as a level set of a volumetric function approximated by locally influential shape elements.The template is defined by the number and formulas of its elements, while its parameters concatenate the parameters of each element.
- A network can fit one template to shapes with widely varying geometry and topology.
- Similar shapes receive similar template parameters that vary gradually through shape-space, while corresponding elements associate with similar structures across shapes.Figure 2 illustrates templates trained with 10, 25, and 100 elements and consistent structure between shapes.
- Consistent shape elements support shape interpolation, vertex correspondence, influence-region prediction in images, and semantic segmentation.
- Compared with volumetric primitives, the structured implicit formulation permits an order-of-magnitude increase in shape elements, helping capture fine details.The method is described as producing more detailed and consistent templates than volumetric primitives in the comparison.
2. Related Work
Prior shape-template methods rely on primitives, semantic parts, or predefined structures, limiting consistency across shapes and requiring human input or initial decomposition. Structured implicit functions offer interpretable, sparse, topology-flexible representations that support detailed template learning and downstream shape applications.
- Template learning: Primitive-fitting methods explain individual observations but do not ensure consistency across input shapes for correspondence, transfer, or exploration.
- Template learning: Semantic-part decomposition methods depend on labeled examples and class-specific part arrangements, whereas this work learns structural templates without human input.
- Template learning: Previous template-fitting methods generally assume an initial set of primitives or part structure before learning.
- Neural template learning: The method extends box-primitive placement by greatly increasing the number and detail of shape elements for precise shape associations in correspondence and semantic segmentation.
- Implicit shape representations: Structured implicit functions represent shapes with local functions, supporting compact storage, interior queries, arbitrary topology, and smooth blending between related shapes.
- Representation comparison: Structured implicit functions are rated suitable for human interpretability, concise storage, surface and volume representation, topological changes, and smooth deformation.
3. Structured Implicit Shape Representation
The paper represents watertight shapes as level sets of structured implicit functions composed of local shape elements. A fixed collection of anisotropic Gaussian elements yields a concise, renderable template whose parameters define the surface.
- Each input shape is modeled as a watertight surface bounding an interior volume, represented as an ℓ-level set of F(x, Θ).
- F sums the local contributions of a fixed number N of shape elements, with Θ formed by concatenating their parameter vectors.
- The adopted shape elements are scaled, axis-aligned anisotropic 3D Gaussians with a scale constant, geometric center, and per-axis radii.
- The representation can be viewed as stretched or compressed 3D blobs, while more sophisticated elements are suggested but not evaluated.
- Because all Gaussian constants are negative, the surface uses a negative isolevel crossing while preserving the inside/outside sign convention.
4. Template Learning
The learning framework encodes meshes through rendered depth views and predicts structured implicit parameters using reconstruction losses. Sampling near surfaces, uniformly in space, and at element centers supports accurate fitting while keeping elements active.
- 4.1. Architecture: A feed-forward CNN maps multiple rendered depth images of a mesh to the complete template parameter vector Θ in one prediction.
- 4. Template Learning: The network does not directly enforce similar parameters for similar shapes, yet learns a smooth fitting function without intervention.
- 4.1. Architecture: The encoder uses 20 depth images rendered from uniformly sampled directions, avoiding voxel encoders' computational cost and point encoders' loss of surface information.
- 4.1. Architecture: The linear final layer usually predicts a 700-dimensional code, while an LSTM performed better in some cases but trained longer and scaled poorly to many elements.
- 4.2. Data Preparation: Input meshes must be made watertight because the loss requires ground-truth inside/outside labels; training uses depth maps paired with labeled samples.
- 4.3. Loss: The total loss combines uniform-sample and near-surface classification losses with a center classification loss that encourages every shape element to contribute.
- 4.3.2 Near Surface Sample Loss LS: Near-surface sampling emphasizes thin structures and balances both sides of the boundary, while uniform sampling keeps surrounding free space clear.
- 4.3.3 Shape Element Center Losses LC: The center loss places predicted element centers inside the predicted shape and ground-truth bounding box, preventing inactive elements from dying during training.
5. Experiments
Experiments show that learned template parameters organize similar shapes coherently and support RGB-based labeling, correspondence, interpolation, and reconstruction, while revealing limitations for symmetry, thin, sharp, and angled structures.
- 5.1. Clustering by Template Parameters: Template parameters cluster ShapeNet classes cleanly, with intuitive mixing among visually similar categories such as tables, benches, sofas, and chairs.Cabinets, speakers, and displays also share parameters because they are essentially box-shaped.
- 5.2. Comparison to Volumetric Primitives: Compared with volumetric primitives, templates using 10, 25, and 100 elements provide increasing element counts for representing shape structure and detail.Figure 5 compares training and test examples, reconstructed surfaces, and ground-truth meshes.
- 5.3. Single-View RGB Prediction and Labeling: RGB prediction fits template parameters and labels image pixels by the highest-weight shape element, yielding corresponding 3D regions without additional part labels.The method performs better for naturally oriented shapes and worse for shapes with rotational symmetries.
- 5.4. Shape Correspondence: Template-space coordinates enable nearest-neighbor vertex correspondence, while consistent elements associate equivalent parts across shapes.Coordinates normalize offsets by element radii, encode influence through vector length, and compare vertices using cosine distance.
- 5.5–5.6. Interpolation and Reconstruction: Blending multiple template parameters produces interpolated shapes, and reconstruction accuracy is similar to competing approaches despite substantially fewer degrees of freedom.The representation has 700 degrees of freedom versus 32768 for 3D-R2N2, 3072 for PSG, and 7398 for Pix2Mesh; Table 2 reports F-score (%) at τ = 10^-4.
- 5.7. Limitations: The method struggles with detailed, sharp, angled, and razor-thin structures, and its fixed element count does not provide one-to-one semantic component mappings.The authors suggest higher-order or non-axis-aligned functions, distance-based losses, supervision, or architecture search as possible remedies.
6. Conclusion
The paper learns structured implicit templates for diverse 3D shapes, supporting consistent geometry mapping and applications including clustering, exploration, abstraction, correspondence, interpolation, and image segmentation.
- An encoder-decoder network learns a template that maps detailed surface geometry consistently across related shapes with large shape variations.
- The learned template supports shape clustering, exploration, abstraction, correspondence, interpolation, and image segmentation.
- Future work includes higher-order or learned shape elements, supervised semantic elements, and structured implicit functions for 3D reconstruction.
A. Additional Algorithmic Details
The additional algorithmic details describe convolutional and fully connected networks for encoding 3D shapes or RGB images into the template representation, with RGB training using distillation.
- The 3D-to-template network uses convolutional layers followed by fully connected layers and is trained with Adam.The architecture was designed to encode 3D shape robustly and generally.
- The RGB-to-template network uses ResNet V2 50 followed by two fully connected layers mapping images to the 3D representation.
- The early-fusion depth network consists of 5 convolutional layers followed by 4 fully connected layers.
- The RGB network is trained with supervised L2 regression to templates generated by the depth network rather than the main-paper loss functions.
B. Additional Dataset Details
Experiments use ShapeNet data with an 80%-20% train-test split and an additional train-validation split; a larger rendered dataset provides more diverse RGB training data.
- The experiments use the ShapeNet 80%-20% train-test split, with the training portion further divided into 75% train and 5% validation.Validation selects hyperparameters, while networks are trained only on the train subset.
- A larger dataset of 2 million RGB ShapeNet renders uses 256x256 resolution and random brightness, contrast, hue, and saturation augmentation.The authors qualitatively found these changes improved transfer from synthetic renders to real images.
C. Runtime Analysis
Inference for depth-to-template prediction is fast, but preprocessing and mesh extraction dominate runtime; marching-cubes extraction trades interactive speed for quick rasterization and mesh comparison.
- 1.11ms is the mean inference time for the depth-to-template network on a GTX 1080 with batch size 1.
- Preprocessing, especially rendering input depth images, becomes the performance bottleneck because its cost depends strongly on input mesh complexity.
- Structured implicit functions enable direct isosurface reconstruction after inference, using methods with different accuracy-versus-time trade-offs.
- Ray marching accurately reflects predicted isosurfaces and normals but makes interactive frame rates difficult.
- Marching cubes enables quick rasterization and easier ground-truth mesh comparisons, but sampling F(x, Θ) is its primary computational bottleneck.
- 5.96 seconds are required for CPU mesh extraction at 256^3 resolution with one thread.GPU acceleration could further reduce runtime.
- Removing tiny connected components helps suppress artifacts from shape elements that nearly switch off but still cross the isosurface level.