Source-linked AI summary
Learning Local Shape Descriptors from Part Correspondences With Multi-view Convolutional Networks
Haibin Huang, Evangelos Kalogerakis, Siddhartha Chaudhuri, Duygu Ceylan, Vladimir G. Kim, Ersin Yumer
TL;DR
Semantic similarity between 3D surface points is hard to specify with hand-crafted geometric features, despite its importance for shape analysis. The paper learns point descriptors from multi-scale rendered views and synthetic correspondences; the descriptors outperform traditional hand-crafted and voxel-based alternatives on the reported benchmarks and work across several applications.
Problem
Hand-crafted geometric features do not readily capture semantic point similarity, which depends on local geometry, structure, and context.
Source: §1 · p. 1Method
A CNN processes multi-scale rendered neighborhoods and learns point descriptors from corresponding-point examples, using synthetic training data generated through part-aware non-rigid alignment.
Source: §1 · p. 2Results
Takeaways & Limitations
The descriptors can be used for keypoint detection, affordance labeling, segmentation, and partial shape matching to depth scans without fine-tuning.
Source: §1 · p. 2The method currently relies on a heuristic viewing configuration and rendering procedure.
Source: §8 · p. 12
Abstract
from arXiv · showhide
We present a new local descriptor for 3D shapes, directly applicable to a wide range of shape analysis problems such as point correspondences, semantic segmentation, affordance prediction, and shape-to-scan matching. The descriptor is produced by a convolutional network that is trained to embed geometrically and semantically similar points close to one another in descriptor space. The network processes surface neighborhoods around points on a shape that are captured at multiple scales by a succession of progressively zoomed out views, taken from carefully selected camera positions. We leverage two extremely large sources of data to train our network. First, since our network processes rendered views in the form of 2D images, we repurpose architectures pre-trained on massive image datasets. Second, we automatically generate a synthetic dense point correspondence dataset by non-rigid alignment of corresponding shape parts in a large collection of segmented 3D models. As a result of these design choices, our network effectively encodes multi-scale local context and fine-grained surface detail. Our network can be trained to produce either category-specific descriptors or more generic descriptors by learning from multiple shape categories. Once trained, at test time, the network extracts local descriptors for shapes without requiring any part segmentation as input. Our method can produce effective local descriptors even for shapes whose category is unknown or different from the ones used while training. We demonstrate through several experiments that our learned local descriptors are more discriminative compared to state of the art alternatives, and are effective in a variety of shape analysis applications.
1 INTRODUCTION
Local descriptors support many 3D shape-analysis tasks, but semantic similarity is difficult to capture with hand-crafted geometric features. The paper learns multi-scale point descriptors from rendered views and large synthetic correspondence data, then evaluates them across several applications.
1 INTRODUCTION
- Semantic point similarity is difficult to hand-code because functional properties depend on local geometry, structure, and context, while shapes and scans may vary or be incomplete.Dense alignment used to compensate for descriptor limitations is typically computationally expensive, difficult to develop, and task-specific.Source: §1 · pp. 1–2
- A multi-scale view-based CNN learns descriptors for any point on a mesh or point cloud, using adaptively selected projections to capture local and broader context.The network processes views with shared image-based layers, then pools their features into one point descriptor and trains on corresponding point pairs.Source: §1 · p. 2
- A part-aware non-rigid alignment method turns consistently segmented and labeled shapes into about 977 million synthetic corresponding point pairs for training.The authors describe this repository as several orders of magnitude larger than prior such datasets.Source: §1 · p. 2
- The descriptor is designed to combine fine-grained surface information with context, while reusing CNN architectures pretrained on large image datasets.The paper identifies higher spatial resolution than voxel representations and pretrained image filters as advantages of rendered views.Source: §1 · p. 2
- Learned descriptors outperform traditional hand-crafted and voxel-based descriptors on sparse correspondence and shape-segmentation benchmarks, and support several other shape-analysis applications.The paper also demonstrates keypoint detection, affordance labeling, and partial matching to depth scans without fine-tuning.Source: §1 · p. 2
2 RELATED WORK
Prior work spans hand-crafted geometric descriptors, learned global representations, and learned surface descriptors. This paper instead learns compact per-point representations by combining multi-view local projections with correspondence-based training.
2 RELATED WORK
- Hand-crafted local descriptors encode low-level geometry, which may not reliably capture functional or semantic properties and can be brittle to noise, partial data, and structural variation.They also often use hand-tuned parameters tailored to particular datasets or processing scenarios.Source: §2 · p. 3
- Learned global 3D representations target recognition, classification, and retrieval, whereas this work focuses on point descriptors for local analysis tasks.Prior global approaches include voxel-based networks, hand-crafted features processed by networks, and geometry-image embeddings.Source: §2 · p. 3
- Unlike multi-view networks designed for whole-shape classification or part-label confidence, this architecture learns surface-point descriptors using locally selected multi-scale views and point correspondences.The related architectures use fixed views or views selected for surface coverage rather than views adaptively selected around each point.Source: §2 · p. 3
- Intrinsic learned descriptors can be invariant to near-isometric deformation, but rigid part rotations may alter functionality and semantic correspondence in man-made shapes.Unlike a concurrent synchronization approach that assumes pre-existing extrinsic alignments, this method assumes no consistent shape alignment or orientation.Source: §2 · p. 4
- Compared with depth-scan pixel descriptors, hand-crafted-feature classifiers, and local volumetric patches, this method produces a single multi-view descriptor for each 3D point.The paper identifies limited training data and voxel resolution as continuing challenges for local volumetric patches.Source: §2 · p. 4
3 OVERVIEW
The method maps each surface point to a compact descriptor intended to bring geometrically and semantically similar points together, without requiring fixed shape alignment. It learns this mapping from corresponding-point examples and demonstrates it in several geometry-processing applications.
3 OVERVIEW
- The descriptor function maps any mesh or point-cloud surface point to a vector, with similar points across differently structured shapes close under Euclidean distance.The method favors rotational invariance and assumes no restrictions on mesh or point-cloud connectivity or topology.Source: §3 · p. 4
- A CNN learns the descriptor from training data, either for one category or across categories, by processing rendered local views in a Siamese network.Corresponding points are encouraged to be nearby, while non-corresponding points are separated by a margin.Source: §3 · pp. 4–5
- The descriptors are demonstrated for part labeling, cross-category human-centric affordance labeling, and matching shapes to depth data.The overview points to these as geometry-processing applications of the learned descriptors.Source: §3 · p. 5
4 ARCHITECTURE
The architecture selects visible views around each point at multiple scales, processes them with shared CNN layers, and pools them into a compact descriptor. A learned linear projection reduces the pooled feature dimensionality.
4 ARCHITECTURE
- At training time, the implementation samples 1,024 surface points per shape, while test-time descriptors can be computed for arbitrary points.Each point is represented using rendered local neighborhoods from selected viewpoints.Source: §4 · p. 5
- Per point, visible directions among 150 uniformly sampled views are clustered to select three representatives, each rendered at three distances from the shape center.The selected distances are 0.25, 0.5, and 0.75 times the bounding-sphere radius; the authors report no significant gains from increasing the view or distance counts.Source: §4 · p. 5
- Four in-plane rotations of each selected view produce 36 rendered images per point, with images rendered at 227 × 227 resolution.The rotations are spaced at 90-degree intervals to promote rotational invariance.Source: §4 · p. 5
- Each rendered view yields a 4,096-dimensional descriptor; element-wise maximum pooling across each point’s 36 unordered views produces one 4,096-dimensional descriptor, avoiding 1,296 pairwise comparisons between two points’ descriptors.Source: §4 · p. 5
- Average view pooling produced worse correspondence accuracy than max pooling, while concatenating view descriptors would depend on consistent local coordinate frames.The paper notes that consistent frames are not trivial to achieve.Source: §4 · pp. 5–6
- A learned linear projection reduces the pooled 4,096-dimensional feature to a 128-dimensional descriptor, selected using hold-out validation.The projection is also intended to down-weight dimensions that contain no useful information.Source: §4 · p. 6
5 LEARNING
The learning pipeline generates dense point correspondences by non-rigidly registering same-label parts in segmented shapes, then uses those correspondences to train a local descriptor network. It combines contrastive-loss training with ImageNet-pretrained convolutional filters.
Training correspondences
- Training correspondences are generated automatically by non-rigidly registering pairs of same-label parts from consistently segmented ShapeNetCore, which contains 17K man-made shapes across 16 categories.Source: §5 · p. 6
Part-based registration
- Registration samples 10K points per shape and optimizes offsets to align compatible part surfaces while keeping neighboring offsets smooth.The optimized offsets provide dense correspondences between closest compatible points.Source: §5 · p. 6
Training dataset
- 100K part pairs, each with 10K points, were aligned in 12 hours on 3 CPUs with 14 hyperthreaded cores each.Table 1 reports statistics for the training dataset across 16 ShapeNetCore classes.Source: §5 · p. 7, Table 1 · p. 7
Network training
- The contrastive objective penalizes descriptor differences for corresponding points and small distances for non-corresponding points, with weight decay and a margin of 1.The correspondence set comes from part-based registration, and descriptor distance is Euclidean.Source: §5 · p. 7
- Convolutional filters are initialized from AlexNet pretrained on ImageNet1K, which contains 1.2M images.The authors expect these image-trained filters to partially capture shape information.Source: §5 · p. 7
- Training uses batches of 64 point pairs and Adam updates; the implementation uses Caffe, with source code, results, and datasets publicly available.Each batch contains 32 corresponding pairs and 32 non-corresponding pairs.Source: §5 · p. 7
6 EVALUATION
The evaluation compares learned descriptors with state-of-the-art alternatives on the BHCP benchmark, using correspondence and retrieval measures. The learned descriptors outperform the alternatives in single-category training and remain effective when trained across categories.
6 EVALUATION
- The BHCP benchmark contains 404 diverse man-made shapes with 6–12 corresponding feature points per shape; evaluation uses CMC and correspondence accuracy under single- and cross-category training.Shapes are randomly rotated to test rotational invariance; learning-method comparisons use training and test shapes separated to avoid overlap.Source: §6 · pp. 7, 9, §6.1 · p. 9
6.1 Results: single-category training.
- In both symmetric and non-symmetric cases, LMVCNN outperforms 3DMatch and hand-engineered local descriptors on BHCP correspondence measures.Predicted correspondences appear visually plausible, though bike pedals show inconsistencies attributed to less accurate training correspondences for parts with varying geometry and topology.Source: Figure 6 · p. 8, Figure 7 · p. 8, §6.1 · p. 9
6.2 Results: cross-category training.
- Cross-category training across 16 ShapeNetCore categories slightly reduces performance, but still significantly outperforms the best alternative, shape contexts.The comparison averages results over all BHCP classes for symmetric and non-symmetric cases.Source: Figure 8 · p. 9, §6.2 · p. 10
- Training on 13 ShapeNetCore categories that exclude BHCP-overlapping classes yields performance comparable to shape contexts on unseen categories.The descriptor is higher in CMC and slightly lower in correspondence accuracy than shape contexts in this setting.Source: §6.2 · p. 10
6.3 Results: alternative algorithmic choices.
- View-pooling after AlexNet fc6 performs best among the tested AlexNet layers; VGG16 is marginally better but slower, and the proposed viewpoint configuration performs best.Fine-tuning AlexNet, using dimensionality reduction, and max rather than average view-pooling each contribute to higher performance in the reported comparisons.Source: §6.3 · p. 10, Figure 9 · p. 10
- Performance is similar with arbitrary and upright orientations, drops slightly with 18 or 24 views, and remains stable beyond 30 views.Per-point execution time scales approximately linearly with the number of views.Source: §6.3 · pp. 10–11, Figure 10 · p. 10, Table 3 · p. 11
7 APPLICATIONS
The paper applies its descriptors to shape segmentation, matching partial scans to complete models, and predicting affordance regions. The reported results include improved average labeling accuracy and dense scan-to-model matching despite scan noise and missing parts.
Shape segmentation
- Replacing hand-crafted features with learned descriptors improves average mesh-labeling accuracy, with significant gains for chairs and bikes.The descriptors provide features for the unary terms in a graph-cuts segmentation pipeline.Source: §7 · p. 11, Table 4 · p. 11
Matching shapes with 3D scans
- Without fine-tuning on scans or point clouds, the method densely matches partial scans to complete shapes despite noise, outliers, missing parts, and noisy normals.The examples show corresponding scan and model points in matching colors.Source: §7 · pp. 11–12, Figure 12 · p. 12
Predicting affordance regions
- After fine-tuning on marked contact regions, descriptor-space retrieval matches affordance regions across shapes from different categories and with different global structures.The benchmark includes contact regions such as pelvis and palms, and the fine-tuning split uses half of its shapes.Source: §7 · p. 12
8 CONCLUSION
The paper presents compact local descriptors from multiscale rendered views and a learned convolutional network, alongside a synthetic-data strategy for training. It notes limits in view selection and in the accuracy of generated correspondences.
8 CONCLUSION
- Multiscale rendered views processed by a learned deep convolutional network, with view pooling and dimensionality reduction, produce compact local descriptors for shape-analysis applications.Figure 13 shows corresponding affordance regions for pelvis and palms.Source: §8 · p. 12, Figure 13 · p. 12
- The authors also present a strategy to generate synthetic training data to automate learning.Source: §8 · p. 12
- The viewing configuration and rendering procedure rely on heuristics, while perspective projections may capture less surface information than alternative parameterizations.The authors suggest investigating optimized views and other local surface parameterizations.Source: §8 · p. 12
- Descriptors are less discriminative when training correspondences are erroneous, particularly for parts whose geometry and topology vary significantly.The authors suggest exploring human-annotated correspondences and iterating between data generation, learning, and alignment.Source: Figure 14 · p. 12, §8 · p. 13