Source-linked AI summary
Category-Level Multi-Part Multi-Joint 3D Shape Assembly
Yichen Li, Kaichun Mo, Yueqi Duan, He Wang, Jiequan Zhang, Lin Shao, Wojciech Matusik, Leonidas Guibas
TL;DR
Multi-part assembly must account for physical joint matching, not only geometric arrangement, while satisfying both shape structure and joint alignment. The paper addresses this with hierarchical graph learning that combines part- and joint-level reasoning, and reports better shape structure and joint matching than prior methods. Its scope is currently limited to peg-hole joints, with more complex joint types and sequential planning left for future work.
Problem
Method
A hierarchical graph network combines a part graph for shape structure with a joint-level graph for matching and aligning joints, using iterative pose refinement.
Source: §1 · pp. 1–2Results
The method achieves higher joint matching accuracy and more reliable shape structure than prior works.
Source: §1 · p. 2, §4.4 · p. 7, §5 · p. 8Takeaways & Limitations
Joint-aware hierarchical reasoning supports category-level multi-part assembly that addresses both holistic structure and physical joint alignment.
Source: §1 · p. 2, §5 · p. 8The paper focuses on simple peg-hole joints and leaves more complicated joint types and programmatic or sequential alignment planning for future work.
Source: §5 · p. 8
Abstract
from arXiv · showhide
Shape assembly composes complex shapes geometries by arranging simple part geometries and has wide applications in autonomous robotic assembly and CAD modeling. Existing works focus on geometry reasoning and neglect the actual physical assembly process of matching and fitting joints, which are the contact surfaces connecting different parts. In this paper, we consider contacting joints for the task of multi-part assembly. A successful joint-optimized assembly needs to satisfy the bilateral objectives of shape structure and joint alignment. We propose a hierarchical graph learning approach composed of two levels of graph representation learning. The part graph takes part geometries as input to build the desired shape structure. The joint-level graph uses part joints information and focuses on matching and aligning joints. The two kinds of information are combined to achieve the bilateral objectives. Extensive experiments demonstrate that our method outperforms previous methods, achieving better shape structure and higher joint alignment accuracy.
1. Introduction
The paper frames multi-part, multi-joint assembly as coordinating desired shape structure with correct peg-hole joint connections. It proposes hierarchical graph learning to address large matching spaces, mixed discrete-continuous optimization, and error compounding.
1. Introduction
- Multi-part assembly must connect all peg joints with matching hole joints while arranging parts into a desired shape.The setting uses joint-annotated part point clouds and peg-hole connections.Source: §1 · p. 1
- Prior geometry-focused methods can produce flipped, upside-down, or rotated poses that cause joint mismatches and unreliable functional assemblies.The task also involves a large matching search space, non-continuous optimization, and compounding errors.Source: §1 · pp. 1–2
- The proposed approach uses joint-level and part-level graphs to jointly optimize joint matching and holistic shape structure.Joint-centric features are aggregated into part features and combined with part-graph representations for pose prediction.Source: §1 · p. 2
- Iterative coarse-to-fine graph refinement reduces error compounding by correcting part poses across multiple assembly iterations.Each iteration refines predictions from the previous iteration toward both task objectives.Source: §1 · p. 2
- The paper introduces a joint-annotated dataset, evaluation metrics, and a hierarchical graph network evaluated on both shape structure and joint alignment accuracy.Experiments are designed to compare the approach with prior methods on both objectives.Source: §1 · p. 2
2. Related Work
Prior assembly methods model part geometry and relationships but generally neglect contact-joint alignment. This work instead uses hierarchical graph learning to reason simultaneously about joint relationships and part relationships.
Assembly-based 3D Modeling
- Earlier assembly-based synthesis methods generate shapes or estimate part transformations, whereas this task solves poses for one desired shape while matching all joints.The task also relaxes watertight-geometry assumptions by using point-cloud parts.Source: §2 · p. 2
- Willis et al. [47] model joints for two-part assembly with one joint pair, while this work addresses multiple parts with multiple contact joints.The earlier setting assumes explicit pairing and watertight part geometry.Source: §2 · p. 2
Graph Learning for Part Relationship
- Prior graph-learning methods reason about part-level or object-level relationships, whereas this method models a hierarchy spanning joints and parts.The hierarchical representation supports simultaneous shape-structure and joint-alignment objectives.Source: §2 · p. 2
- The architecture combines a part graph for desired shape structure with a joint graph that corrects poses to connect matched joints.Figure 2 depicts the two GNN modules and their respective roles.Source: Figure 2 · p. 3
3. Method
The method uses hierarchical graph learning to alternate between part-level shape construction and joint-level matching, combining both representations for joint-aware pose prediction. It optimizes shape structure and joint alignment through dedicated graph reasoning and losses.
Problem Setup
- The task requires one-to-one peg-hole matching in a large search space and simultaneous pose optimization for connecting joints and forming a valid shape.A part-graph rough structure reduces the matching space, after which alternating graph representations progressively refine poses.Source: §3 · pp. 3, 5, 6
Overview
- Hierarchical graph learning alternates part-level shape construction and joint-level matching, then combines both representations to predict poses satisfying both objectives.The part graph proposes and refines structure, while the joint graph discovers joint relationships; hierarchical aggregation combines their messages.Source: §3 · pp. 3, 6
Part Graph Pose Proposal
- The part graph encodes part geometries and uses message passing to propose poses that form the desired shape structure.Part geometry features initialize graph nodes, and iterative edge-node updates model relationships among parts.Source: §3 · p. 3
Joint Graph Relationship Reasoning
- The joint graph initializes all allowed peg-hole connections, updates edge and node features, and learns connectivity weights from joint geometry and part positions.The connectivity matrix weights edge features before joint-node aggregation, supporting joint relationship reasoning.Source: §3 · pp. 3–4
Joint-Aware Pose Prediction
- Joint-aware pose prediction pools relevant joint features into each part and concatenates them with part-graph features before iterative pose refinement.The resulting part features contain both joint and part information, and the pose operator refines the previous prediction.Source: §3 · p. 4
Loss Functions
- The network uses separate shape and joint losses, with shape losses supervising translation, rotation, and holistic structure and joint losses correcting flips and refining matched-joint alignment.The joint objective is coarse-to-fine, while Hungarian matching and graph traversal preserve order invariance for geometrically congruent parts.Source: §3 · p. 5
4. Experiments
The experiments evaluate a joint-augmented multi-part assembly benchmark using six adapted baselines, shape and joint metrics, and ablations of the hierarchical graph design. The proposed method improves shape structure, avoids collapsed predictions, and benefits from interleaved coarse-to-fine graph reasoning and joint-aware losses.
4.1. Dataset
- The dataset augments PartNet with peg-hole joint annotations across chairs, tables, and cabinets, retaining 3,736 chairs, 5,053 tables, and 719 cabinets after filtering.Parts are sampled with 1,000 points and evaluated using the official train/validation/test split.Source: §4.1 · p. 6
4.2. Baseline Methods
- The experiments compare six repurposed baselines, including sequential, global-context, graph, transformer, and joint-centric assembly methods under two fairness settings.Setting II retrains baselines with the proposed model’s joint inputs and losses, while B-Joinable is evaluated there because it explicitly models joints.Source: §4.2 · pp. 6–7, §4.4 · p. 7
4.3. Evaluation Metric
- Evaluation separates shape structure from joint alignment using Part Acc. and Shape CD, plus Joint Acc. and Joint CD, with joint accuracy threshold τj=0.01.Joint Acc. measures the percentage of matched joint pairs under order-invariant matching.Source: §4.3 · p. 7
4.4. Results and Analysis
- The proposed method consistently outperforms baseline methods on shape-structure metrics, producing structurally similar assemblies while avoiding symmetry-induced flips and incorrect poses.Joint information helps distinguish geometrically similar parts, such as cabinet boards, by encoding their functional connections.Source: §4.4 · p. 7
- Baseline performance under shared joint inputs and losses falls to almost one-third of the original-setting performance, demonstrating the difficulty of jointly optimizing structure and joint alignment.The resulting predictions can collapse because reducing joint distances forms a local minimum for joint losses.Source: §4.4 · pp. 7–8
- Interleaved part-graph and joint-graph iterations progressively form rough structure, correct joint matching, and refine poses into structurally sound assemblies.The empirically best graph combination is part-joint-part-joint-joint.Source: Figure 4 · p. 7, §4.4 · p. 8
- Removing the initial part-graph iteration significantly reduces performance, supporting its role as a shape-structure heuristic that simplifies joint matching.Removing joint embedding also significantly decreases Shape CD and Part Accuracy by eliminating an explicit joint-alignment pose-editing signal.Source: §4.4 · p. 8, Table 3 · p. 8
- Matching in the loss is necessary because it preserves permutation invariance and synchronizes shape and joint objectives; removing it significantly worsens all metrics.Without consistent matching, the two losses optimize in different directions and produce problematic poses.Source: §4.4 · p. 8
5. Conclusion and Future Work
The paper introduces peg-hole joints into category-level multi-part 3D shape assembly and proposes hierarchical relationship reasoning with joint and part embeddings. It establishes a joint-augmented dataset and metrics, while identifying more complex joints and sequential planning as future directions.
5. Conclusion and Future Work
- The work formulates category-level multi-part assembly around peg-hole joints and uses joint and part embeddings for hierarchical relationship reasoning.It also introduces a joint-augmented dataset and evaluation metrics for the task.Source: §5 · p. 8
- The current scope is limited to simple peg-hole joints; future work targets more complicated joint types and general formulations covering broader joint scenarios.Programmatic or sequential joint-alignment planning is another proposed direction for autonomous-system deployment.Source: §5 · p. 8
6. Appendix
The appendix details order-invariant joint matching, task constraints, graph-based implementation, qualitative results, and additional baseline comparisons.
6.1. Order Invariant Joint Matching Algorithm
- The order-invariant matching algorithm resolves ambiguous peg-hole assignments by graph traversal followed by conflict handling.It assigns joint types part-wise, caches conflicts, and removes or resolves conflicting edges.Source: §6.1 · p. 12
6.2. Additional Explanation of the Task
- Joint matching constrains valid assemblies beyond geometric plausibility because upside-down or rotated poses may prevent parts from connecting.The joint-matching objective can eliminate most geometric variations, leaving exchanges of congruent parts as the main residual variation.Source: §6.2 · pp. 12–13
6.3. Implementation Detail
- The part graph extracts PointNet features from part point clouds, performs message passing, and predicts each part’s 7D pose.Each pose contains a quaternion rotation and a 3D translation.Source: §6.3 · p. 13
- The joint graph encodes masked peg, hole, and non-joint points into joint embeddings before message passing.Mask values −1, +1, and 0 denote peg points, hole points, and non-joint points, respectively.Source: §6.3 · p. 13
- The method combines shape and joint losses, while additional joint-axis supervision was found to hurt performance when joint pairing is incorrect.The reported loss weights combine shape structure, matched-joint distances, Chamfer distance, and rotation terms.Source: §6.3 · p. 13
6.5. Additional Baseline Comparisons
- Additional qualitative examples compare the method with baselines and ground truth, while joint-aware inputs with original baseline losses produce the worst baseline performance.The appendix reports that baseline methods do not benefit from joint input without explicitly joint-centric loss design.Source: §6.4 · p. 13, §6.5 · p. 14, Table 4 · p. 15