Source-linked AI summary
PartNet: A Recursive Part Decomposition Network for Fine-grained and Hierarchical Shape Segmentation
Fenggen Yu, Kun Liu, Yan Zhang, Chenyang Zhu, Kai Xu
TL;DR
Deep-learning shape segmentation typically assigns labels from a fixed set, limiting flexibility across shapes with different part structures. PartNet recursively decomposes point clouds into hierarchical parts, combining contextual features with local part geometry. It reports state-of-the-art performance on public and fine-grained benchmarks and demonstrates use in image-to-shape refinement.
Problem
Deep-learning segmentation models are trained for fixed label sets, limiting their flexibility for shapes with different part counts; simultaneous primitive labeling also misses hierarchical structure cues.
Source: §1 · p. 1Method
PartNet recursively applies shared-weight binary decomposition to point clouds, classifying each node's split type and stopping decision while combining propagated context with local part features.
Source: §1 · pp. 1–2Results
Takeaways & Limitations
The model segments shapes into a number of parts that varies with structural complexity rather than a fixed label set.
Source: §6 · p. 8The model must be trained separately for each shape category and relies on reasonable ground-truth hierarchies; random training hierarchies degrade performance.
Source: §6 · p. 8
Abstract
from arXiv · showhide
Deep learning approaches to 3D shape segmentation are typically formulated as a multi-class labeling problem. Existing models are trained for a fixed set of labels, which greatly limits their flexibility and adaptivity. We opt for top-down recursive decomposition and develop the first deep learning model for hierarchical segmentation of 3D shapes, based on recursive neural networks. Starting from a full shape represented as a point cloud, our model performs recursive binary decomposition, where the decomposition network at all nodes in the hierarchy share weights. At each node, a node classifier is trained to determine the type (adjacency or symmetry) and stopping criteria of its decomposition. The features extracted in higher level nodes are recursively propagated to lower level ones. Thus, the meaningful decompositions in higher levels provide strong contextual cues constraining the segmentations in lower levels. Meanwhile, to increase the segmentation accuracy at each node, we enhance the recursive contextual feature with the shape feature extracted for the corresponding part. Our method segments a 3D shape in point cloud into an unfixed number of parts, depending on the shape complexity, showing strong generality and flexibility. It achieves the state-of-the-art performance, both for fine-grained and semantic segmentation, on the public benchmark and a new benchmark of fine-grained segmentation proposed in this work. We also demonstrate its application for fine-grained part refinements in image-to-shape reconstruction.
1. Introduction
Existing deep-learning segmentation models target fixed label sets, limiting flexibility across shapes with different part counts, while hierarchical decomposition can use upper-level structure to guide finer segmentation. PartNet uses shared-weight recursive binary decomposition to segment shapes into variable numbers of parts and reports state-of-the-art performance on public and new fine-grained benchmarks.
1. Introduction
- The same Chair model can produce different numbers of fine-grained parts for different shapes, depending on structural complexity.The model's output is not restricted to a fixed part count.Source: Figure 1 · p. 1
- Fixed-label models limit flexibility across shapes with different part counts, whereas hierarchical segmentation uses higher-level structure to constrain finer labels.The paper notes that training separate models for different label sets is neither general nor efficient.Source: §1 · p. 1
- PartNet performs top-down recursive binary decomposition of point-cloud shapes, using a shared-weight network and node classifiers to select adjacency or symmetry splits and stopping points.The node classifier also determines whether a node is a leaf.Source: §1 · pp. 1–2
- Recursive context from higher-level nodes is combined with each part's shape feature to inform segmentation at that node.The paper describes the higher-level decompositions as cues that constrain lower-level segmentation.Source: §1 · p. 2
- Training uses ShapeNet point clouds with hierarchies built by an existing rule-based method, and computes node-classification and binary point-labeling losses at each node.The paper says these principled training hierarchies help training converge faster.Source: §1 · p. 2
- PartNet reports state-of-the-art performance on a public benchmark and a new fine-grained segmentation benchmark, and demonstrates fine-grained part refinement for image-to-shape reconstruction.The refinement application segments a reconstructed point cloud hierarchically, then refines the point cloud at each leaf part.Source: §1 · p. 2
2. Related work
Prior learned 3D segmentation methods classify shape primitives across representations, but are trained for fixed semantic label sets. Earlier hierarchical approaches use grouping, topology, heuristics, or prescribed tag hierarchies; PartNet positions recursive neural networks as a data-driven, end-to-end alternative.
2. Related work
- Learned 3D segmentation methods classify geometric primitives across voxel, point-cloud, multi-view, and mesh representations, but generally target fixed semantic labels.Changing the target labels requires retraining with newly annotated data.Source: §2 · p. 2
- Earlier hierarchical segmentation uses bottom-up grouping, top-down topology, or proximity-and-symmetry heuristics, while Yi et al. [39] use fixed semantic tags in a canonical hierarchy.PartNet does not require a prescribed canonical hierarchy and learns decomposition hierarchies through recursive node classification.Source: §2 · p. 2
- Prior recursive neural networks support shape and scene generation or reconstruction, but the authors report no previous use for hierarchical 3D shape segmentation.PartNet presents its model as end-to-end trainable for that segmentation task.Source: §2 · p. 2
3. Method
PartNet recursively decomposes a shape point cloud through shared node-wise modules for context propagation, hierarchy construction, and point labeling. Node features and supervision are built across the decomposition hierarchy.
3.1. Recursive part decomposition network
- Each node combines parent-propagated recursive context with a PointNet feature of its partial shape; the shared decoder passes context to children.The root feature is formed from the full-shape PointNet feature, while non-root nodes combine context with a feature of their partial point cloud.Source: §3.1 · p. 3, Figure 2 · p. 3
- A node classifier chooses adjacency, symmetry, or leaf; symmetry nodes encode one generator and parameters, then transfer labels to counterparts.The left child is the symmetry generator and the right child holds symmetry parameters; only the generator needs segmentation.Source: §3.1 · pp. 3–4
- Segmentation combines node context and shape features with per-point PointNet features to predict binary point labels.The point classifier uses the last five layers of a PointNet and does not share weights with the two feature-extraction PointNets.Source: Figure 4 · p. 3, §3.1 · p. 4
3.2. Loss function
- The training objective averages node-classification and node-segmentation cross-entropy losses over the hierarchy.The classification loss is computed at all hierarchy nodes, while segmentation loss is computed at non-leaf nodes.Source: §3.2 · p. 4
3.3. Training details
- Training uses 2048-point clouds with normals and Gaussian-noise augmentation, optimized with Adam at batch size 10 and learning rate 0.001.Inputs are 2048 × 6; the networks use batch normalization and 20% random feature dropout between specified layers.Source: §3.3 · p. 4
4. Results and evaluations
The evaluations cover a new fine-grained benchmark, visual and quantitative tests of PartNet, and comparisons on semantic and instance segmentation. Results include flexible part counts, strong semantic-segmentation performance, and improvement over SGPN on FineSeg.
4.1. Benchmark
- FineSeg provides about 3,000 shapes across six categories with point-level fine-grained annotations and hierarchies for training and AP-based evaluation.The benchmark includes chair, table, airplane, sofa, helicopter, and bike shapes.Source: §4.1 · p. 4
4.2. Segmentation results and evaluation
- The same PartNet model produced fine-grained segmentations with varying part counts for unseen shapes, with the count depending on structural complexity.Figure 5 compares segmentations with ground truth; Figure 6 illustrates different counts for chairs.Source: §4.2 · p. 4, Figure 5 · p. 5, Figure 6 · p. 5
- The full model consistently surpassed versions without recursive context features or part shape features, and both features supported faster node-classification and segmentation training.A ShapeNet challenge user study reported average ratings above 4.0 for all tested categories and higher ratings than the w/o RCF baseline.Source: §4.2 · p. 6, Figure 7 · p. 6
4.3. Comparison of semantic segmentation
- PartNet with a part-label module achieved the highest mean accuracy across ShapeNet part-dataset categories and performed especially well on structurally complex categories.The comparison uses part-wise IoU; similar results were observed on the Princeton Segmentation Benchmark.Source: §4.3 · p. 7, Table 2 · p. 7
4.4. Comparison of instance segmentation
- PartNet consistently outperformed SGPN on FineSeg, with the authors attributing the improvement to recursive partial-shape decomposition and features combining local part shape with global context.The comparison uses AP at IoU thresholds 0.25 and 0.5.Source: §4.4 · p. 7, Table 3 · p. 7
5. Applications
PartNet guides single-view reconstruction refinement by recursively segmenting the holistic point cloud, then refining each part at a leaf node. The examples show detailed point clouds with plausible part structure, though part fidelity to the input image may sometimes be lost.
5. Applications
- The segment-and-refine pipeline recursively decomposes a holistic reconstruction, refines each leaf-part point cloud, and combines the refined parts into a full shape.The refiner uses separate PointNet channels for the part and full-shape point clouds, then concatenates their features; hierarchical reconstruction loss is computed at each node.Source: §5 · p. 8
- The part refiner uses two PointNet channels to encode the part and full-shape point clouds before generating a refined part point cloud.The concatenated features feed a four-layer fully connected network; training uses Chamfer distance and earth mover’s distance, with reconstruction loss composed bottom-up across the hierarchy.Source: §5 · p. 8
- Figure 10 presents examples in which fine-grained segmentation guides refinement, producing detailed point clouds with plausible part structure but sometimes losing fidelity to the input image.Each row shows the input image, holistic reconstruction, its fine-grained segmentation, and the final refinement.Source: §5 · p. 8
6. Conclusion
The conclusion presents PartNet as a recursive method for fine-grained segmentation whose number of parts can vary with shape complexity. It also reports limitations in hierarchy meaningfulness, category-specific training, and dependence on reasonable ground-truth hierarchies.
6. Conclusion
- PartNet segments complex 3D shapes into accurate fine-grained parts, with the number of parts depending on structural complexity rather than a fixed label set.The conclusion also reports superior semantic-segmentation performance from divide-and-conquer learning.Source: §6 · p. 8
- Figure 10 illustrates single-view point-cloud refinement examples, ordered from input image to holistic reconstruction, segmentation, and final refinement.Source: Figure 10 · p. 8
- The resulting hierarchy may be less meaningful than hierarchies learned purposively for shapes or scenes, and the model still requires separate training for each shape category.Training with totally random hierarchies degrades performance; the method instead uses reasonable ground-truth hierarchies.Source: §6 · p. 8