Source-linked AI summary
MonoArt: Progressive Structural Reasoning for Monocular Articulated 3D Reconstruction
Haitian Li, Haozhe Xie, Junxiang Xu, Beichen Wen, Fangzhou Hong, Ziwei Liu
TL;DR
Monocular articulated reconstruction requires jointly inferring geometry, part structure, and motion from limited evidence. MonoArt progressively builds canonical geometry, structured part representations, and motion-aware embeddings, achieving state-of-the-art reconstruction accuracy and inference speed on PartNet-Mobility.
Problem
Single-image articulated reconstruction must infer geometry, part structure, and motion despite entangled visual cues that make direct articulation regression unstable.
Method
MonoArt progressively transforms image observations into canonical geometry, structured part representations, and motion-aware embeddings within one end-to-end architecture.
Results
MonoArt achieves state-of-the-art performance in reconstruction accuracy and inference speed on PartNet-Mobility.
Takeaways & Limitations
The framework supports accurate and efficient articulation inference and generalizes to robotic manipulation and indoor 3D reconstruction tasks.
Takeaways & Limitations
Very small parts and objects with novel topologies or uncommon articulation patterns can be difficult to segment and parameterize reliably.
Abstract
from arXiv · showhide
Reconstructing articulated 3D objects from a single image requires jointly inferring object geometry, part structure, and motion parameters from limited visual evidence. A key difficulty lies in the entanglement between motion cues and object structure, which makes direct articulation regression unstable. Existing methods address this challenge through multi-view supervision, retrieval-based assembly, or auxiliary video generation, often sacrificing scalability or efficiency. We present MonoArt, a unified framework grounded in progressive structural reasoning. Rather than predicting articulation directly from image features, MonoArt progressively transforms visual observations into canonical geometry, structured part representations, and motion-aware embeddings within a single architecture. This structured reasoning process enables stable and interpretable articulation inference without external motion templates or multi-stage pipelines. Extensive experiments on PartNet-Mobility demonstrate that OM achieves state-of-the-art performance in both reconstruction accuracy and inference speed. The framework further generalizes to robotic manipulation and articulated scene reconstruction.
1 Introduction
MonoArt addresses monocular articulated reconstruction by progressively reasoning from geometry to parts and motion. It reports state-of-the-art accuracy and substantially faster inference without external motion pipelines.
- Articulated reconstruction must jointly recover object geometry, part structure, and kinematic relationships, unlike rigid reconstruction.
- Existing methods use multiple motion states, retrieval, or other auxiliary resources, limiting practical availability, scalability, or efficiency.
- MonoArt progressively constructs structured part representations and motion-aware embeddings instead of directly regressing articulation from image features.
- MonoArt achieves higher F-score and substantially lower inference time than existing approaches.
- Explicit structural priors support part decomposition and articulation reasoning without video generation, handcrafted motion templates, or vision–language priors.
- MonoArt achieves state-of-the-art geometric and articulation performance on PartNet-Mobility while significantly reducing inference time and generalizing to robotics and indoor reconstruction.
2 Related Work
Related work progresses from multi-view articulated modeling toward explicit part decomposition, motion reasoning, and scalable 3D part segmentation.
- Early articulated modeling methods use multi-view observations and canonical deformations but lack explicit part or kinematic modeling.
- Later methods introduce explicit part decomposition and motion reasoning to estimate rigid parts and transformations.
- Learning-based 3D part segmentation methods include supervised point- or mesh-level classification, image–language transfer, feature distillation, and scaling-based feed-forward models.
- Supervised segmentation approaches are limited by the scale and diversity of available part annotations.
3 Our Approach
MonoArt uses a four-stage architecture that progressively converts an image into canonical geometry, part-aware features, motion-aware queries, and a kinematic tree with articulation parameters.
- The framework comprises a TRELLIS-based 3D Generator, Part-Aware Semantic Reasoner, Dual-Query Motion Decoder, and Kinematic Estimator.
- Kinematic Estimator: The Kinematic Estimator predicts masks, motion types, origins, axes, limits, and the kinematic hierarchy from refined queries.
- TRELLIS-based 3D Generator: TRELLIS reconstructs canonical geometry and produces geometry-aligned latent features for downstream part and motion reasoning.
- Part-Aware Semantic Reasoner: Tri-linear interpolation converts sparse voxel features into continuous surface-aligned point features.
- Part-Aware Semantic Reasoner: Triplane projection and transformer refinement incorporate global spatial context while preserving geometry-aligned structure.
- Dual-Query Motion Decoder: Dual queries separately encode part semantics and spatial motion anchors, then are iteratively refined using self- and cross-attention.
- Dual-Query Motion Decoder: Confidence estimation discards unreliable queries, allowing the model to determine the number of parts during inference.
4 Experiments
MonoArt is evaluated on PartNet-Mobility for reconstruction, kinematic prediction, ablations, real-world generalization, downstream manipulation, scene reconstruction, runtime, and limitations.
- Evaluation Protocol: PartNet-Mobility contains approximately 2K articulated objects across 46 categories, with fixed, prismatic, revolute, and continuous joints.Evaluation uses both a 7-category split and a full 46-category split.
- Evaluation Protocol: The evaluation averages CD, F-Score, PSNR, and CLIP similarity across six sampled articulation states after mesh alignment.Kinematic evaluation also includes joint type, axis, and pivot measures in normalized object coordinates.
- Main Results: MonoArt achieves the best overall geometry reconstruction and kinematic prediction on both the 7-category and 46-category benchmarks.On the full benchmark, pivot error is reduced by more than 40%.
- Ablation Study: Ablations show that the Part-Aware Semantic Reasoner, triplet supervision, dual-query refinement, and point embeddings improve geometry and kinematic prediction.Performance improves through L = 6 refinement layers, while L = 9 degrades performance.
- Applications, Runtime, and Limitations: MonoArt supports robot manipulation and articulated scene reconstruction, while requiring 20.5 seconds per instance on a single NVIDIA A6000 GPU.The framework converts inferred articulation parameters into simulation-ready assets and coherent articulated scenes; limitations remain for tiny parts and novel topologies.
5 Conclusion
MonoArt frames monocular articulated reconstruction as progressive structural reasoning, explicitly modeling geometry, part structure, and motion in one framework. It achieves state-of-the-art reconstruction accuracy and inference speed on PartNet-Mobility and generalizes to robotic manipulation and articulated scene reconstruction.
- MonoArt progressively reasons over geometry, part structure, and motion rather than depending on multi-view supervision, retrieval libraries, or auxiliary video synthesis.
- The unified framework performs articulation inference without handcrafted motion priors or external pipelines.
- MonoArt achieves state-of-the-art performance in reconstruction accuracy and inference speed on PartNet-Mobility.
- The framework generalizes to robotic manipulation and articulated scene reconstruction.
A.1 Loss Functions
The loss design supervises semantic part embeddings, part masks and confidence, articulation parameters, and kinematic structure. It combines contrastive, segmentation, regression, and parent-prediction objectives, with confidence filtering used to retain valid part hypotheses.
- Five objectives supervise part embeddings, motion-decoder masks and confidence, articulation parameters, and kinematic-tree prediction.The objectives are triplet, mask, confidence, motion, and structure losses.
- Hungarian bipartite matching assigns predicted queries to ground-truth articulated parts using weighted mask similarity costs.Unmatched queries are treated as null predictions.
- Triplet contrastive supervision trains discriminative motion-aware embeddings by grouping features from the same articulated part against features from different parts.Pairwise similarity uses cosine similarity scaled by a learnable temperature parameter.
- Mask supervision combines focal and Dice losses, while confidence supervision targets mask IoU and filters predictions below 0.5 during inference.The confidence score indicates the reliability of each predicted part mask.
- Motion loss regresses joint type, axis direction, motion origin, and motion limits, while structure loss predicts each matched part’s parent.Bounded motion ranges use a center–span parameterization.
A.2 Training Details
Training uses a four-stage procedure that progressively warms up semantic reasoning, initializes motion queries, jointly optimizes core modules, and finally trains the kinematic tree predictor. The implementation uses multi-GPU AdamW training with mixed precision and scheduled learning rates.
- Training proceeds through four stages: semantic-reasoner warm-up, dual-query initialization, joint optimization, and kinematic-tree training.The stages use 100, 20, 100, and 30 epochs respectively.
- Training uses four NVIDIA A100 GPUs with an effective batch size of 4 and requires approximately six days.
- Stage III jointly optimizes semantic, motion-decoder, and articulation-regression modules with triplet, mask, confidence, and motion losses.The motion-loss weight increases linearly during the first 40 epochs.
- All stages use AdamW with weight decay 0.01, FP16 mixed precision, and gradient clipping at maximum norm 0.1.
- The learning rate warms up for 10 epochs from 5 × 10^-5 and then follows cosine annealing to 10^-6.
B Additional Ablation Study Results
The additional ablation study visualizes PASR point features under different availability and supervision settings. PASR improves structural organization, with triplet supervision producing the most compact and separated part features.
- Fig. 7 visualizes PASR point features using PCA projections under variants with or without the reasoner and with different supervision strategies.Cross-entropy and triplet supervision are compared.
- Adding PASR improves structural organization compared with features without PASR.
- Triplet supervision produces more compact and well-separated part features than cross-entropy supervision.Cross-entropy yields partially separated clusters.
C More Discussion on Limitations
MonoArt faces failures when components are extremely small or when articulated categories are unseen during training. These cases expose limits in part identification, placement, and articulation recovery.
- Extremely small components, such as a printer button, are difficult to capture with uniform sampling and limited spatial resolution.These conditions result in inaccurate part identification and placement.
- Uniform sampling and limited spatial resolution can produce inaccurate identification of small parts.
- Unseen articulated categories, such as bicycles, can lead to incorrect articulation recovery.The passage attributes this to a large domain gap from the training data.
D More Qualitative Results
Additional qualitative comparisons evaluate MonoArt across diverse objects and motion ranges. The visualizations compare predicted articulations, including part masks and motion parameters, against prior methods.
- MonoArt reconstructs more consistent geometry and more plausible articulated motion than prior methods across diverse objects and motion ranges.
- The qualitative results include input motion states and corresponding articulations.
- The comparisons cover multiple motion states and visualize predicted articulations through part masks and motion parameters.