Deep Part Induction from Articulated Object Pairs

• Articulation offers a mobility-based clue to functional parts, but induction must handle differences in geometry and pose, ambiguous boundaries, noise, and missing scan data.For example, observations of scissors at different opening angles can reveal two blades rotating around a pivot; parsing moving parts also supports robotics, graphics, and vision applications.Source: §1 · pp. 1–2
• The method dramatically outperforms existing state-of-the-art approaches, which largely fail on even single-DoF objects, while parsing objects with one or several DoFs.The paper reports extensive qualitative and quantitative evaluation on synthetic and real datasets, alongside ablation studies.Source: §1 · p. 2
• The paper introduces mobility-based part segmentation from articulated object pairs that generalizes to novel categories without prior class, part-label, or component knowledge.The method takes static shape snapshots and uses articulation as evidence of part structure and motion.Source: §1 · p. 2
• Learned local features propose dense correspondences, and PairNet integrates global shape information to infer deformation flow between shapes with geometric and articulation differences.A correspondence mask supports partial matching when shapes differ structurally or contain missing data.Source: §1 · p. 2
• A neural segmentation module proposes rigid-motion hypotheses from deformation flow and sequentially extracts parts whose motion is consistent with them.The module implements a differential, neural-based RANSAC procedure for structure discovery from noisy observations.Source: §1 · p. 2
• The correspondence, flow, and part-estimation modules can iteratively reinforce one another in an ICP-like procedure, enabling induction across instances with limited overlap and corrupted inputs.Unlike iterative fitting approaches for clean, same-instance pairs, the learning-based method avoids sensitive hyperparameter tuning.Source: §1 · p. 2

Rigid part extraction

• Unlike rigid-part extraction methods based on continuous pose sequences, this method matches geometrically different shapes or partial scans, broadening the setting those methods address.Earlier approaches estimate motion from RGBD sequences and group points or segments by flows or trajectories [Pillai et al. 2014], or use correspondences, joint fitting, skeletons, or mesh rotation clustering [Kim et al. 2016; Yuan et al. 2016a; Li et al. 2016; James and Twigg 2005]. Skeleton-fitting methods are largely specific to classes such as human bodies [Bogo et al. 2016; Mehta et al. 2017; Newell et al. 2016].Source: §2 · pp. 2–3
• The paper contrasts its joint deep-learned correspondence and segmentation architecture with prior methods that separate these steps and often require hand-tuned parameters.The architecture is trained on a massive dataset of shapes.Source: §2 · p. 3

Shape segmentation

• Supervised shape-segmentation methods label parts observed in training data, so they cannot discover new part labels.Approaches use multiple views [Kalogerakis et al. 2017], volumetric grids, spatial structures [Klokov and Lempitsky 2017; Riegler et al. 2017], point sets [Qi et al. 2017a,b; Su et al. 2018], surface embeddings, or graphs.Source: §2 · p. 3
• Co-segmentation methods seek common parts through geometric clustering or shape correspondences, but can miss functional parts or fail under large rigid motions.This method instead learns correspondence and parts jointly, assuming articulated functional parts predominantly move rigidly; the paper reports better results than co-segmentation approaches [Hu et al. 2012; Sidi et al. 2011; van Kaick et al. 2013; Golovinskiy and Funkhouser 2009; Huang et al. 2014, 2008; Kim et al. 2013].Source: §2 · p. 3

Shape correspondences

• Learned correspondence methods can accommodate structurally and geometrically different shapes, but separating correspondence from part extraction performs significantly worse than the paper’s joint architecture.Intrinsic methods for deformable human bodies are limited by spectral instability on shapes with different topology or structure [Masci et al. 2015; Monti et al. 2017], while volumetric, view-based, and point-based networks learn descriptors across different shapes [Qi et al. 2017b].Source: §2 · p. 3

3 OVERVIEW

• The architecture co-segments shape pairs by using correspondences and deformation flows to infer rigidly moving parts, which then improve those estimates.Its modular design uses these mutually reinforcing cues to extract piecewise rigid motions.Source: §3 · p. 3

Neural network design

• A learned correspondence module predicts probabilistic point-wise matches and a mask, handling geometry and articulation differences as well as missing data and scan noise.It uses PointNet++ point features followed by a matching sub-module.Source: §3 · p. 3, Figure 2 · p. 4
• A PairNet flow module converts ambiguous correspondences into deformation fields, which a recurrent segmentation module aggregates to discover rigidly moving parts.The segmentation module extracts motion hypotheses and their point support, then sequentially discovers parts until no others remain; neither part count nor motion is known in advance.Source: §3 · p. 4

Iterative execution

• The three modules run in a closed loop until deformation-flow magnitude is minimized, yielding better segmentations than a single pipeline execution.Correspondences guide flow prediction, flow supports rigid-part extraction, and the extracted parts improve correspondences and flow.Source: §3 · p. 4

4 NETWORK ARCHITECTURE

• The input pair contains functionally related objects with rigidly moving parts in different articulation states, with no assumptions about orientation, point order, part count, or degrees of freedom.Meshes are uniformly sampled to 512 surface points in the implementation.Source: §4 · p. 4

4.1 Correspondence Proposal Module

• A correspondence module uses shared PointNet++ branches and pairwise point features to predict a soft matching matrix, while a correspondence mask handles points without matches.The pairwise features are processed by an MLP and softmax; the mask refines matching probabilities for missing data or structural differences.Source: §4.1 · pp. 4–5

4.2 Flow Module

• The flow module combines pairwise displacement and matching probabilities, then uses PairNet to aggregate across points and predict a point-wise 3D deformation field.Flow remains imperfect when shapes have geometric or structural differences, so learned segmentation is needed to extract parts reliably.Source: §4.2 · p. 5

4.3 Segmentation Module

• The segmentation module acts as a differentiable sequential RANSAC, generating rigid-motion hypotheses, predicting their point support, and recurrently extracting parts from strongest support regions.Its Recurrent Part Extraction Network outputs soft segments and decides when to stop; residual motion parameterizations were found easier to train and best-performing for motion and segmentation estimation.Source: §4.3 · pp. 5–6, Figure 4 · p. 6

4.4 Iterative Segmentation and Motion Estimation

• The iterative method significantly improves deformation flow and segmentation by alternating rigid-part deformation, residual-flow estimation, and segmentation updates.It can repeat until convergence or a maximum of 10 iterations; when orientations differ substantially, it initializes by testing 48 rotations and choosing the one with the smallest predicted flow magnitude.Source: §4.4 · p. 7, Figure 5 · p. 7

5.1 Training Dataset

• From 16,881 ShapeNetCore shapes across 16 categories, the authors create 16,881 articulated pairs with 2–6 moving parts, plus five synthetic scan pairs per pair, each sampled to 512 points.The pairs include ground-truth part correspondences, rigid motions, flow fields, point-wise correspondences, and flags for points without matches. The source annotations are from Yi et al. [2016], and the dataset is ShapeNetCore [Chang et al. 2015].Source: §5.1 · p. 7

5.2 Multi-task objective function

• The objective trains correspondence predictions with softmax and binary cross-entropy losses, and directly penalizes flow-field differences with an L2 loss.The correspondence losses address matching point pairs and whether points have any match; all three loss weights are set to 1 through hold-out validation.Source: §5.2 · pp. 7–8
• For rigid motion, the loss compares transformed points with corresponding points on the other shape, avoiding direct rotation and translation differences that performed poorly due to weight-balancing difficulties.It evaluates corresponding points belonging to the same underlying rigid part; its weight is set to 1 through hold-out validation.Source: §5.2 · p. 8
• Segmentation supervision covers the support matrix, recurrent segment assignments, and continuation scores, using Hungarian matching and relaxed IoU to handle unordered segment outputs.Training executes the recurrent net for up to K + 2 steps; the segmentation loss weights are 0.5, 1, and 1 for support, assignments, and continuation, respectively.Source: §5.2 · p. 8

5.3 Training Procedure and Implementation Details

• Training uses Adam at learning rate 0.0001 in three 100-epoch stages: correspondence and flow, then motion and support with ground-truth flow, then end-to-end optimization with 0.001 learning-rate decay.Layer hyperparameters are selected by hold-out validation for segmentation IoU.Source: §5.3 · p. 9

6.1 Test Dataset

• Synthetic tests cover 875 pairs per dataset across 23 categories, while real tests include 231 same-object scan pairs and 150 CAD-to-scan pairs across 10 categories each.Test categories differ from training categories; only laptop overlaps between the synthetic test and training data, and real testing may involve a shift in motion realism.Source: §6.1 · p. 9

6.2 Deformation Flow

• Our method achieves the best deformation-flow performance on SF2F and SF2P, with more accurate local and global correspondences and greater robustness to missing data.Compared with baselines, it models piecewise rigidity and missing correspondences, avoiding unsupported deformations in regions absent from the paired point cloud.Source: §6.2 · pp. 9–10, Table 1 · p. 10

6.3 Segmentation

• Our approach outperforms all baselines by a large margin on synthetic and real segmentation datasets, especially in per-part IoU, better detecting part counts and small parts.The real-data results also show generalization from synthetic training to scans, including segmenting small pedals and producing cleaner motion boundaries on geometrically different object pairs.Source: Table 2 · p. 11, §6.3 · pp. 11–12

6.4 Ablation Study

• Ablations show that the full iterative pipeline, correspondence-mask prediction and supervision, and the flow module improve flow estimation and handle missing regions.On SF2F, the RPEN segmentation design scores 0.838/0.773 RI/IoU, versus 0.740/0.584 for LSTM units; iterative flow becomes increasingly piecewise rigid.Source: §6.4 · pp. 12–13

6.5 Comparison with Other Learning-Based Motion Segmentation Approaches

• On SP2P, our point-cloud-only method outperforms Shao et al. [2018] by a large margin in flow and segmentation, including on complex, texture-less structures.Their approach uses RGB-D images and point clouds, but struggles with complex structures and nearby parts that move little.Source: §6.5 · p. 13, Table 5 · p. 13

6.6 Computational cost

• The method has O(N^2) training and testing complexity and processes a shape pair in 0.4 seconds on average on a NVIDIA TITAN Xp GPU.Using 1024 points raises SF2F segmentation Rand Index from 83.8% to 84.2%.Source: §6.6 · p. 13

6.7 Applications

• Predicted rigid parts and motions support shape animation by interpolating between articulation states, including revolute and prismatic joints.The interpolated sequence can reveal object functionality and add interactivity in virtual environments.Source: §6.7 · p. 13, Figure 11 · p. 13
• For a shape paired with a related product image, the framework lifts the image to 3D and propagates motion-based segmentation back to the image.This produces joint co-segmentation of the 3D shape and 2D image using discovered motion cues.Source: §6.7 · pp. 13–14, Figure 12 · p. 14

7 LIMITATIONS AND FUTURE WORK

• Point clouds may miss small parts or fail on rotationally symmetric parts, while sliding interior parts are difficult to distinguish; higher resolution or adaptive attention may help.The paper gives bottle caps and sliding knives as examples.Source: §7 · p. 14
• Large topology variation makes point-wise deformation flows harder to define, while large geometric differences require segmentation that handles both local rigid and non-rigid flows.Source: §7 · p. 14
• The method currently infers parts and motion from shape pairs; future work could use a single shape or larger object sets to discover shared articulation patterns.Source: §7 · p. 14

Review of PointNet, PointNet++ and Recurrent Net

• The supplement briefly reviews PointNet, PointNet++, and Recurrent Net, three building blocks used in the framework.Source: SUPPLEMENTARY MATERIAL · p. 16

Training Dataset

• It visualizes the training dataset with examples of shape pairs.Source: SUPPLEMENTARY MATERIAL · p. 16

Test Datasets and Per-Category Motion Segmentation

• It reports per-category motion-segmentation evaluation and statistics for synthetic and real test datasets.Source: SUPPLEMENTARY MATERIAL · p. 16

Network Architecture

• It explains the structural details of the network blocks used.Source: SUPPLEMENTARY MATERIAL · p. 16

Other Implementation Details

• It provides additional implementation details and discusses design choices.Source: SUPPLEMENTARY MATERIAL · p. 16

7.1.1 PointNet.

• PointNet [Qi et al. 2017a] aggregates point features with max pooling to produce an order-invariant global vector and can approximate any function on the input set.Its learned transformations are MLPs.Source: §7.1.1 · p. 16

7.1.2 PointNet++.

• PointNet++ [Qi et al. 2017c] aggregates features within local neighborhoods, then stacks sampling and grouping layers to capture increasingly broad context hierarchically.Each local neighborhood is an Euclidean ball around a sampled point.Source: §7.1.2 · p. 16

7.1.3 Recurrent nets.

• Recurrent nets update a hidden state from its previous value and the current input, then compute the output from the updated state.The initial hidden state is zero; both update and output functions have learnable weights.Source: §7.1.3 · p. 16

7.2 Training Dataset

• Training uses shapes from 16 man-made object categories, with generated motions that may be unrealistic, requiring generalization from fake training motions to real test motions.The categories include airplane, earphone, cap, motorbike, bag, mug, laptop, table, guitar, knife, rocket, lamp, chair, pistol, car, and skateboard.Source: §7.2 · p. 16

7.3 Test Datasets and Per-Category Motion Segmentation

• The approach outperforms all baseline methods by a large margin, though segmentation degrades for categories with parts that intrude into one another.The evaluation covers synthetic datasets SF2F and SF2P and real datasets RP2P and RF2P.Source: §7.2 · p. 16

7.4.1 PointNet Building Blocks.

• PointNetC aggregates MLP features by max pooling, PointNetS combines global and local features, and PointNet++S builds per-point features through multiscale downsampling and upsampling.PointNet++S uses sample-group and interpolation operations, each with an MLP.Source: §7.4.1 · pp. 16–17
• Correspondence uses a shared PointNet++S, flow uses PairNet for global then hierarchical aggregation, and segmentation combines PointNet++S hypothesis generation with PointNetS support features and recurrent PairNet selection.The correspondence network's PointNet++S uses sample-group and interpolation operations; PairNet first aggregates across the second point set, then encodes local dependencies in the first.Source: §7.4.1 · p. 18

7.4.2 Single Image Based 3D Reconstruction Network.

• The image-to-3D network maps foreground pixels to XYZ coordinates using a pretrained ResNet-34 encoder and three decoders for depth, x-y, and mask prediction.Training first uses textured ShapeNetCore CAD models, then fine-tunes on articulated objects without texture.Source: §7.4.2 · p. 18

7.5.1 Rigid Motion Parameterization.

• Rigid motion uses residual rotation and translation parameters, which the authors report optimize better than directly optimizing Rp and tp.The rotation is projected to the nearest orthogonal matrix by SVD; translation is computed after applying the inferred rotation.Source: §7.4.2 · p. 18

7.5.2 Graph Cuts.

• Soft segmentation is hardened by graph cuts on a K=10 Euclidean KNN graph, using graph-edge pairwise terms and negative-log-probability unary terms.The experiments use σ = 0.05 for the edge weights.Source: §7.5.2 · p. 18