Source-linked AI summary
3D Shape Segmentation with Projective Convolutional Networks
Evangelos Kalogerakis, Melinos Averkiou, Subhransu Maji, Siddhartha Chaudhuri
TL;DR
3D shape segmentation must handle ambiguous parts, subtle boundaries, occlusions, noise, and undersampling. The paper combines multi-view FCNs, a surface projection layer, and a surface CRF in an end-to-end architecture, reporting improved benchmark accuracy over prior methods. The approach remains fully supervised and currently supports single-level, non-hierarchical segmentation.
Problem
3D shape segmentation is challenging because semantic parts and boundaries can be ambiguous, while local and global cues must remain robust to noise, undersampling, and occlusion.
Method
The method combines multi-view image-based FCNs, a differentiable projection layer that aggregates outputs onto the surface, and a surface-based CRF trained end-to-end.
Results
The method significantly outperforms prior work, including 5.4% better category-average accuracy and 6.3% better dataset-average accuracy on ShapeNet.
Takeaways & Limitations
The architecture provides a broadly applicable approach to 3D shape segmentation without hand-engineered geometric descriptors or processing stages.
Takeaways & Limitations
The method is fully supervised and currently handles only single-level, non-hierarchical segmentations.
Abstract
from arXiv · showhide
This paper introduces a deep architecture for segmenting 3D objects into their labeled semantic parts. Our architecture combines image-based Fully Convolutional Networks (FCNs) and surface-based Conditional Random Fields (CRFs) to yield coherent segmentations of 3D shapes. The image-based FCNs are used for efficient view-based reasoning about 3D object parts. Through a special projection layer, FCN outputs are effectively aggregated across multiple views and scales, then are projected onto the 3D object surfaces. Finally, a surface-based CRF combines the projected outputs with geometric consistency cues to yield coherent segmentations. The whole architecture (multi-view FCNs and CRF) is trained end-to-end. Our approach significantly outperforms the existing state-of-the-art methods in the currently largest segmentation benchmark (ShapeNet). Finally, we demonstrate promising segmentation results on noisy 3D shapes acquired from consumer-grade depth cameras.
1. Introduction
3D shape part segmentation is difficult because semantic ambiguity, subtle boundaries, the need for local and global reasoning, and noise challenge existing geometric methods. The paper proposes a multi-view deep architecture that projects image-based predictions onto surfaces and enforces coherent labeling.
- 3D shape segmentation matters for vision, robotics, and virtual-reality applications involving semantic part reasoning.
- Semantic part segmentation is challenging because parts may be ambiguous, boundaries subtle, features local and global, and inputs noisy or undersampled.
- The proposed architecture repurposes image-based deep networks for view-based 3D reasoning and aggregates predictions onto the shape surface without hand-tuned descriptors.
- Multiple automatically selected views provide surface coverage, while FCNs produce per-part confidence maps that are fused and projected onto the 3D surface.
- A surface-based CRF promotes consistent labeling across the entire surface, and the network including the CRF is trained end-to-end.
- ∼8% higher part-labeling accuracy is reported for complex objects than the state of the art on the largest 3D shape segmentation dataset.
2. Related work
Prior 3D shape segmentation methods commonly depend on geometric descriptors, alignment, correspondence, or constrained representations. This paper instead learns view-based representations jointly with a surface CRF, avoiding those assumptions and processing stages.
- Image-based segmentation: Image segmentation methods commonly repurpose classification networks as FCNs for dense labeling and combine them with multi-scale, recurrent, deconvolutional, or CRF-based strategies.
- Image-to-shape learning: Unlike volumetric or fixed-topology image-to-shape methods, this architecture learns view-based representations with a surface CRF and uses a differentiable, parameter-free, sparse 3D-2D projection.
- Geometric shape segmentation: Existing geometric approaches often extract hand-engineered descriptors before clustering or classification and may add separate graph-based processing stages.
- Alignment and correspondence: Other methods rely on non-rigid alignment, deformable templates, surface correspondences, functional maps, or matching projected binary images.
- This work: The proposed approach transfers image-network features, avoids hand-engineered geometry processing, adaptively selects viewpoints, and trains all image and surface stages end-to-end.
3. Method
The projective convolutional network combines multi-view image FCNs, an order-invariant projection layer, and a surface CRF to produce coherent 3D part segmentations end-to-end.
- FCN and projection: The network processes rendered multi-view inputs with identical FCN modules that output one confidence map per part label.The inputs include shaded and depth images, while the FCNs use image-processing layers initialized from pre-trained networks.
- FCN and projection: The projection layer aggregates FCN confidences across unordered views and maps them to polygon-level label confidences using surface-reference images.For each polygon and label, view-pooling selects the maximum confidence among pixels and images mapped to that polygon.
- Surface CRF: The CRF addresses unobserved or highly occluded polygons and reduces label bleeding across surface convexities or concavities.Unobserved polygons receive zero initial label confidence, while neighboring surface regions propagate confidence through the CRF.
- Surface CRF: The surface CRF assigns polygon labels using projected unary confidences plus pairwise cues based on adjacency, curvature, and geodesic proximity.Geodesic factors apply to polygon pairs within 10% of the bounding sphere radius, making the CRF relatively dense and sensitive to long-range interactions.
- Inference and training: Mean-field inference approximates the CRF marginals for training and testing, and the alternative CRF versions show degraded segmentation when consistency terms are removed.The reported inference procedure often converges after 20 iterations.
- Inference and training: The FCN parameters and CRF weights are learned jointly by maximizing regularized training log-likelihood through backpropagation.The objective uses ground-truth surface labels and weight decay λ = 10^-3 in the experiments.
4. Evaluation
The evaluation measures labeling accuracy on ShapeNetCore and additional datasets, compares against prior methods and degraded variants, and examines generalization to noisy RGB-D reconstructions.
- Datasets: 17,773 expert-verified ShapeNetCore segmentations across 16 categories provide the main evaluation dataset.PSB and COSEG provide additional manually labeled shape datasets, with smaller sizes and largely clean mesh preprocessing.
- ShapeNetCore results: 5.4% higher category-average accuracy than ShapeBoost is achieved on ShapeNetCore.Weighting categories by test-set size increases the reported improvement to 6.3%.
- ShapeNetCore results: 7.8% higher unweighted accuracy and 7.9% higher dataset-average accuracy are reported for ShapeNetCore categories with more than three part labels.The largest gains occur for complex objects such as motor vehicles, aircraft, and furniture.
- Additional datasets: 92.6% accuracy is obtained across PSB and COSEG, compared with 90.6% for ShapeBoost and 86.3% for Guo et al.The method was evaluated without excluding categories considered unsuitable for some geometric approaches.
- Ablation analysis: The CRF produces a major performance gain, while pretraining, viewpoint adaptation, and joint training provide additional improvements in degraded-variant analysis.Viewpoint adaptation and joint training are described as smaller but still useful gains.
- RGB-D generalization: The method generalizes from complete, noise-free ShapeNetCore training shapes to noisy and potentially incomplete RGB-D reconstructions.The authors trained separately on chair, table, and motorbike categories before applying the method to reconstructed objects.
5. Conclusion
The paper combines image-based FCNs, a surface-based projection layer, and a surface-based CRF to segment and label 3D shape parts. It reports improved labeling accuracy, while identifying noisy boundaries, orientation assumptions, and supervision as limitations or future directions.
- The architecture combines image-based FCNs for view-based reasoning, a surface-based projection layer for multi-view aggregation, and a surface-based CRF for coherent segmentations.
- The method significantly outperforms prior work on 3D shape segmentation and labeling.
- Table 4 reports improved labeling accuracy for most classes and on average when using consistent upright orientation and an additional height channel.
- The simple pairwise term can produce noisy segmentations that do not align with strong mesh boundaries.
- The current architecture handles single-level, non-hierarchical segmentations and assumes consistent upright orientation for some additional input encodings.
- Training is currently fully supervised, motivating future extensions to semi-supervised or unsupervised settings.
A.1. Evaluation in PSB/COSEG
The PSB/COSEG evaluation measures labeling accuracy as the percentage of surface area assigned the correct ground-truth face label. Results are reported per category for ShapePFCN, ShapeBoost, and Guo et al., with aggregate performance also shown.
- Labeling accuracy is measured as the percentage of surface area labeled correctly against ground-truth face labels.
- Table 5 presents per-category labeling accuracy for ShapePFCN, ShapeBoost, and Guo et al.
- Table 6 reports aggregate labeling accuracy for the evaluated methods.
A.2. ShapeBoost results on RGB-D sensor data
ShapeBoost was evaluated on noisy objects reconstructed from RGB-D sensor data and failed to produce compelling segmentation results compared with the paper’s method.
- ShapeBoost failed to produce compelling results on noisy objects reconstructed from RGB-D sensor data.
- Noise, holes, and mesh degeneracies may distort hand-engineered geometric descriptors used by ShapeBoost and related methods.
- Shallow classifiers may underfit shape datasets with significant variability.
A.3. Additional data
The additional-data section provides dataset statistics and labeling-accuracy tables for PSB and COSEG, including aggregate performance across the datasets.
- Table 5 contains dataset statistics and per-category labeling accuracy for test shapes in PSB and COSEG.
- Table 6 reports aggregate labeling accuracy on PSB and COSEG.