Source-linked AI summary
Shape2Motion: Joint Analysis of Motion Parts and Attributes from 3D Shapes
Xiaogang Wang, Bin Zhou, Yahao Shi, Xiaowu Chen, Qinping Zhao, Kai Xu
TL;DR
Mobility analysis from a single static 3D shape is difficult because prior approaches rely on pre-existing segmentation or multiple motion states. Shape2Motion jointly segments mobility-oriented parts and estimates their motions using proposal and optimization networks with motion-driven training. On a new benchmark, it achieves state-of-the-art performance for both segmentation and motion attribute estimation.
Problem
Mobility analysis needs simultaneous motion-part segmentation and motion-attribute estimation from a single 3D shape, whereas existing approaches commonly require pre-segmented parts or multiple motion states.
Method
Shape2Motion uses a mobility proposal network and mobility optimization network with motion-driven features and losses that exploit preserved shape structure during functional-part movement.
Results
Shape2Motion achieves state-of-the-art performance for both motion part segmentation and motion attribute estimation on a newly proposed benchmark.
Takeaways & Limitations
The paper establishes a joint single-shape framework and benchmark for mobility-oriented segmentation and motion attribute estimation.
Takeaways & Limitations
The method is limited by point-cloud representation, is not end-to-end trainable, and does not support hierarchical mobility extraction.
Abstract
from arXiv · showhide
For the task of mobility analysis of 3D shapes, we propose joint analysis for simultaneous motion part segmentation and motion attribute estimation, taking a single 3D model as input. The problem is significantly different from those tackled in the existing works which assume the availability of either a pre-existing shape segmentation or multiple 3D models in different motion states. To that end, we develop Shape2Motion which takes a single 3D point cloud as input, and jointly computes a mobility-oriented segmentation and the associated motion attributes. Shape2Motion is comprised of two deep neural networks designed for mobility proposal generation and mobility optimization, respectively. The key contribution of these networks is the novel motion-driven features and losses used in both motion part segmentation and motion attribute estimation. This is based on the observation that the movement of a functional part preserves the shape structure. We evaluate Shape2Motion with a newly proposed benchmark for mobility analysis of 3D shapes. Results demonstrate that our method achieves the state-of-the-art performance both in terms of motion part segmentation and motion attribute estimation.
1. Introduction
Shape2Motion addresses mobility analysis from a single 3D point cloud by jointly segmenting motion parts and estimating their motion attributes. It combines mobility proposals, optimization, motion-driven modeling, and evaluation on a new benchmark.
- Problem: Existing methods typically require pre-segmented parts or multiple motion states, limiting mobility analysis for shapes lacking mobility-oriented segmentation or suitable observations.They may also miss multiple mobilities within one part and motions difficult to capture through state comparison.
- Approach: Shape2Motion takes a single 3D point cloud and jointly computes mobility-oriented segmentation with associated motion attributes.The estimated attributes include motion type and parameters such as translation direction or rotation axis.
- Approach: Its proposal-and-optimize pipeline uses a mobility proposal network followed by a mobility optimization network.The first generates and selects mobility proposals; the second jointly optimizes segmentation and motion attributes before merging the results.
- Approach: Motion-driven features and losses exploit the observation that functional-part movement preserves shape structure.The method moves a proposed part under its motion and measures how well the resulting structure is preserved.
- Evaluation and contributions: The authors evaluate Shape2Motion on a newly proposed benchmark and report state-of-the-art performance for both motion part segmentation and motion attribute estimation.The contributions include a benchmark covering both tasks and a deep architecture with separate proposal and optimization networks.
2. Related Work
Prior work studies mobility, semantic, and functional segmentation, but commonly relies on pre-existing segmentations or motion sequences. Shape2Motion instead jointly learns mobility-oriented segmentation and motion attributes from a single static 3D model.
- Mobility analysis of 3D shapes: Mobility analysis characterizes articulated-object functionality through the motion of one or more constituent parts.Prior studies infer part mobility from input objects or RGBD sequences of articulated motion.
- Mobility analysis of 3D shapes: Existing mobility methods commonly assume pre-existing part segmentation, which may not conform to the object’s mobilities.This mismatch limits their use on shape repositories without mobility-oriented segmentation.
- Semantics- / function- / mobility-induced segmentation of 3D shapes: Semantic segmentation decomposes shapes into parts meaningful for assembly or functional purposes, supported by deep-learning feature learning.Related functional-segmentation work labels regions using class-specific functionality such as graspable, liftable, or pushable.
- Semantics- / function- / mobility-induced segmentation of 3D shapes: Mobility-induced segmentation remains less studied, with prior approaches including kinematic-surface segmentation and motion-state comparison for man-made objects.These approaches differ from Shape2Motion, which infers functional part mobilities from a single static model through joint segmentation and motion-attribute regression.
3. Mobility Proposal Network (MPN)
The Mobility Proposal Network generates candidate motion parts and motion attributes separately, then matches them using motion-driven similarity. Its attribute module parameterizes axes with anchors and predicts orientation through discretization plus residual regression.
- MPN contains motion-part proposal, motion-attribute proposal, and proposal-matching modules that generate, pair, and filter mobility candidates.
- Motion Part Proposal Module: Motion-part proposals use a learned similarity matrix whose rows represent candidate motion-part segmentations, with confidence scores estimating proposal quality.The similarity matrix encodes whether point pairs belong to the same motion part; thresholding produces binary segmentations during testing.
- Motion Attribute Proposal Module: Motion attributes are proposed independently as a motion type and axis line, avoiding combinatorial growth when one part has multiple motions.An axis line is parameterized by an anchor point, displacement vector, and orientation vector.
- Motion Attribute Proposal Module: The attribute module predicts anchors, displacement, orientation, and motion type; orientation uses 14 candidates with classification probabilities and residual corrections.Anchor selection and displacement regression make axis prediction invariant to shape pose.
- Proposal Matching Module: Proposal matching scores candidate part-attribute pairs with a motion-driven metric that compares spatial alignment after prescribed motion.Points are moved according to each mobility before alignment is measured; translation and rotation use prescribed movement amounts.
4. Mobility Optimization Network
The Mobility Optimization Network jointly refines a proposed mobility’s segmentation and motion axis. It combines static shape geometry with dynamic point clouds generated by moving the candidate part.
- MON jointly optimizes motion-part segmentation and motion attributes because separately generated proposals may be inaccurate or mutually inconsistent.
- The network outputs a refined binary motion-part segmentation and residual vectors correcting the axis displacement and orientation.
- MON encodes static PointNet++ shape features and dynamic features from the part moved by specified translation or rotation amounts.Dynamic inputs use three movement magnitudes for translation and three angles for rotation, with moving vectors also encoded.
- MON training combines point-wise labeling loss with L2 residual losses for motion-axis correction.
5. Final Mobility Extraction
Final mobility extraction merges optimized proposals while allowing one motion part to retain multiple distinct motions. It first suppresses overlapping parts, then selects compatible motion attributes.
- Overlapping motion-part proposals are reduced to the proposal with the highest matching score.
- Translation keeps only the highest-scoring direction, while rotation uses greedy selection of high-scoring axes.This extraction procedure supports multiple possible motions for a single part, unlike standard suppression that selects one output.
- For each selected part, attributes are chosen from proposals whose parts have sufficiently large IoU with it.
6. Details, Results and Evaluations
Shape2Motion is evaluated on a new mobility benchmark and against existing and ablated methods, with analyses of proposal quality, optimization, and computational cost.
- 6. Details, Results and Evaluations: The method is trained and tested with an 8:2 split; data enhancement yields 11766 training shapes containing 32298 mobility parts.
- 6. Details, Results and Evaluations: Shape2Motion outperforms the compared motion-state method because it detects mobilities that are difficult to capture through motion-state comparison.The comparison evaluates motion-part IoU and 3D motion-flow EPE after converting Shape2Motion predictions into motion flows.
- 6. Details, Results and Evaluations: Motion optimization improves both motion-part segmentation and motion-attribute estimation over an ablated network without MON.
- 6. Details, Results and Evaluations: The total computational cost is approximately 10 seconds per shape during testing.MPN and MON training take 31 and 35 hours per 50 epochs, respectively, on an NVIDIA TITAN X GPU.
7. Conclusion
Shape2Motion jointly segments motion parts and estimates their attributes from a single 3D shape using proposal generation, optimization, and motion-driven training. The paper identifies representation, trainability, and hierarchy as directions for future work.
- 7. Conclusion: Shape2Motion uses a proposal-and-optimize strategy with mobility proposal and mobility optimization networks for simultaneous segmentation and attribute estimation.
- 7. Conclusion: Motion-driven training exploits the coupling between a motion part and its mobility function, a concept the authors suggest may extend to other scenarios.
- 7. Conclusion: The point-cloud representation may not capture shapes with highly detailed structure, such as a knife with scabbard.
- 7. Conclusion: The overall proposal-and-optimize framework is not end-to-end trainable and does not support hierarchical mobility extraction.The authors identify hierarchical mobility and scanned real-world objects as future directions.