PaperlayerView exact revision on arXiv

Source-linked AI summary

ArtGS: Building Interactable Replicas of Complex Articulated Objects via Gaussian Splatting

Yu Liu, Baoxiong Jia, Ruijie Lu, Junfeng Ni, Song-Chun Zhu, Siyuan Huang

arXiv:2502.19459v2cs.CVcs.GRcs.LGcs.RO

TL;DR

Articulated-object reconstruction is limited by the difficulty of integrating partial observations across states, especially for complex multi-part objects. ArtGS uses canonical 3D Gaussians with coarse-to-fine alignment and skinning-inspired part dynamics modeling, achieving improved joint estimation and part-mesh reconstruction across synthetic and real-world datasets. The method is efficient and robust, while assuming Gaussian scale and opacity remain consistent under transformation.

  • Problem

    Existing methods have limited ability to integrate information across object states, reducing part-mesh reconstruction and dynamics modeling accuracy for complex multi-part objects.

  • Method

    ArtGS uses canonical 3D Gaussians with coarse-to-fine initialization and skinning-inspired, center-based part dynamics modeling to align states and learn articulation.

  • Results

    ArtGS achieves state-of-the-art performance in joint parameter estimation and part mesh reconstruction, with improvements particularly evident for complex multi-part objects.

  • Takeaways & Limitations

    ArtGS provides an efficient and robust approach for reconstructing interactable articulated-object replicas from synthetic and real-world data.

  • Takeaways & Limitations

    The method assumes that each Gaussian’s scale and opacity remain consistent under transformation.

Abstract

from arXiv · show

Building articulated objects is a key challenge in computer vision. Existing methods often fail to effectively integrate information across different object states, limiting the accuracy of part-mesh reconstruction and part dynamics modeling, particularly for complex multi-part articulated objects. We introduce ArtGS, a novel approach that leverages 3D Gaussians as a flexible and efficient representation to address these issues. Our method incorporates canonical Gaussians with coarse-to-fine initialization and updates for aligning articulated part information across different object states, and employs a skinning-inspired part dynamics modeling module to improve both part-mesh reconstruction and articulation learning. Extensive experiments on both synthetic and real-world datasets, including a new benchmark for complex multi-part objects, demonstrate that ArtGS achieves state-of-the-art performance in joint parameter estimation and part mesh reconstruction. Our approach significantly improves reconstruction quality and efficiency, especially for multi-part articulated objects. Additionally, we provide comprehensive analyses of our design choices, validating the effectiveness of each component to highlight potential areas for future improvement. Our work is made publicly available at: https://articulate-gs.github.io.

1 INTRODUCTION

Articulated-object reconstruction must integrate partial observations across multiple states to recover fine-grained parts. ArtGS addresses this with canonical Gaussian alignment and motion-aware part modeling for complex multi-part objects.

  • High-fidelity articulated-object replicas can facilitate sim-to-real transfer for robotic systems.
  • Partial views at individual states make articulated-object geometry difficult to reconstruct accurately.
  • Accurate reconstruction requires integrating observations across object states while aligning fine-grained parts.
  • Existing approaches rely heavily on pretrained segmentation or correspondence priors, often neglecting motion information and struggling with multi-part matching.
  • ArtGS uses canonical Gaussians with coarse-to-fine initialization and updates to bridge object states for deformation modeling.
  • ArtGS introduces skinning-inspired part dynamics modeling with self-guided motion priors and reports state-of-the-art performance, especially on complex multi-part objects.

2 RELATED WORK

Related work covers Gaussian deformation for dynamic reconstruction, articulation-parameter estimation, and articulated-object reconstruction with both part geometry and joint parameters.

  • Dynamic Gaussian Modeling: Dynamic Gaussian methods capture motion but often learn transformations implicitly, limiting controllable dynamics modeling.
  • Dynamic Gaussian Modeling: Physics-based controllable Gaussian models require intricate object priors such as material properties, reducing practicality for everyday articulated-object reconstruction.
  • Articulation Parameter Estimation: Articulation-parameter estimation methods include prediction-based approaches using sensory inputs from different object configurations and end-to-end models.
  • Articulated Object Reconstruction: Articulated-object reconstruction requires both part-level geometry reconstruction and joint articulation-parameter estimation because each part is piece-wise rigid.

3 PRELIMINARIES

The preliminaries define 3D Gaussian Splatting as a scene representation, its rendering objective, mesh extraction process, and the two-state articulated-object reconstruction setting.

  • 3D Gaussian Splatting: 3D Gaussian Splatting represents a static scene with Gaussians carrying centers, covariance matrices, opacity, and spherical-harmonic coefficients.
  • 3D Gaussian Splatting: Covariance matrices are decomposed into scaling and rotation components so each scene Gaussian has position, rotation, scale, opacity, and appearance parameters.
  • 3D Gaussian Splatting: Rendered images and optional depth maps are produced by projecting Gaussians onto the image plane and aggregating them with alpha blending.
  • 3D Gaussian Splatting: 3DGS learns Gaussians from multi-view RGB-D images using an image, D-SSIM, and optional depth-rendering loss.
  • Mesh Extraction from Gaussians: Meshes are extracted from Gaussian splats by rendering depth maps, fusing them with TSDF, and applying marching cubes.

4 METHOD

ArtGS reconstructs articulated objects from two states by learning canonical Gaussians, part assignments, and articulation transformations jointly. Its method combines motion-aware initialization, center-based unsupervised part discovery, and self-guided joint modeling.

  • Problem Formulation: ArtGS learns canonical Gaussians and articulation parameters from multi-view RGB-D observations at two joint states.
  • Problem Formulation: Canonical Gaussians are placed at the mid-state, with inverse transformations connecting the two observed states and reducing separate-state occlusion problems.
  • 4.1 Coarse-to-Fine Canonical Gaussian Initialization: Motion analysis refines the coarse initialization by identifying static and dynamic Gaussians through cross-state Chamfer distances and adding static Gaussians from the more informative state.
  • 4.1 Coarse-to-Fine Canonical Gaussian Initialization: Coarse initialization matches separately trained single-state Gaussians with Hungarian Matching and averages matched pairs, using FPS to reduce them to 5K before matching.
  • 4.2 Part Discovery for Articulation Modeling: ArtGS discovers parts unsupervised and learns per-part articulation parameters rather than relying on prior part information.
  • 4.2 Part Discovery for Articulation Modeling: Center-based modeling assigns Gaussians using spatial and dynamic information, while a residual MLP refines overlapping boundaries and Gumbel Softmax enforces one-part assignments.
  • 4.2 Part Discovery for Articulation Modeling: Part centers are initialized using motion-aware clustering, with static-part means and spectral clustering for movable parts.
  • 4.3 Self-Guided Articulation Type and Parameter Learning: Per-part articulation parameters use dual-quaternions, and a warm-up rotation heuristic classifies joints as revolute or prismatic before constraining transformations.

5 EXPERIMENTS

ArtGS is evaluated on synthetic and real-world articulated-object datasets using mesh-reconstruction and articulation-estimation metrics, including complex multi-part benchmarks and ablations. It consistently outperforms existing methods, while its canonical initialization, center-based modeling, and motion-prior components support reconstruction quality, articulation estimation, and efficiency.

  • Experimental Setup: ArtGS is evaluated on PARIS, DTA-Multi, and the newly curated ArtGS-Multi dataset using mesh reconstruction and articulation estimation metrics.PARIS contains two-part objects; DTA-Multi and ArtGS-Multi contain increasingly complex multi-part articulated objects.
  • Results on PARIS: ArtGS significantly outperforms existing methods across PARIS metrics, especially in joint articulation parameter estimation, while improving movable-part reconstruction and training efficiency.Its canonical Gaussians connect two states, improving low-visibility reconstruction while retaining competitive high-visibility results.
  • Results on Complex Articulated Objects: ArtGS consistently outperforms existing methods by a large margin on DTA-Multi and ArtGS-Multi for both mesh reconstruction and articulation estimation.On ArtGS-Multi, DTA struggles with movable-part identification and axis prediction as part count increases.
  • Results on Complex Articulated Objects: ArtGS maintains high performance regardless of part count while preserving time efficiency, whereas existing methods’ training time scales with the number of parts.Qualitative comparisons show improved part meshes and joint articulation on Storage-47648 and Table-31249.
  • Ablation Studies: Removing canonical Gaussian initialization degrades all metrics, particularly for movable parts, because malformed canonical Gaussians lead optimization toward suboptimal local minima.The ablations evaluate representative objects with multiple revolute and prismatic joints.
  • Ablation Studies: Center-based part assignment and motion-prior-based center initialization outperform MLP, Slot-Attention, random, and all-Gaussian clustering alternatives, especially for movable-part reconstruction and joint estimation.The full ablation study concludes that each component contributes significantly to joint estimation and part-mesh reconstruction.

6 CONCLUSION

ArtGS reconstructs articulated objects from two states of multi-view images by using 3D Gaussians for state alignment and part-dynamics modeling. Experiments show improved joint parameter estimation and part-mesh reconstruction, particularly for complex multi-part objects, with robustness and efficiency across synthetic and real-world datasets.

  • Conclusion: ArtGS reconstructs articulated objects from two states of multi-view images using 3D Gaussians for state alignment and part-dynamics modeling.The approach targets articulated-object reconstruction for complex multi-part settings.
  • Conclusion: ArtGS improves joint parameter estimation and part-mesh reconstruction, particularly for complex multi-part objects.The conclusion attributes these improvements to the approach’s introduced techniques for state alignment and part-dynamics modeling.
  • Conclusion: Experiments across synthetic and real-world datasets validate ArtGS’s robustness and efficiency.The paper positions ArtGS as a foundation for digital replicas of articulated objects in robotics and augmented reality.

A IMPLEMENTATION AND TRAINING DETAILS

ArtGS uses staged Gaussian initialization, learnable part discovery, temperature annealing, and constrained optimization to model articulated objects. Training combines state alignment, part assignments, joint-type prediction, and Gaussian deformation regularization.

  • Canonical Gaussian Initialization: Single-state Gaussians G0 and G1 are trained for 10K steps before coarse canonical Gaussians are obtained by matching them.The opacity entropy loss encourages Gaussian opacities toward 0 or 1, controlling Gaussian count and accelerating training; coarse initialization takes about 2 minutes per object.
  • Part Discovery for Articulation Modeling: Part discovery computes Gaussian-level part masks from canonical Gaussians and learnable part centers using a learnable hash grid for residual prediction.The formulation uses canonical Gaussians Gc and learnable centers Ck = (pk, Rk, λk).
  • Part Discovery for Articulation Modeling: Gumbel-Softmax temperature is annealed from 1 to 0.1 over 10K steps, retaining soft assignments early and hard assignments later for training stability.This schedule accommodates initially inaccurate part assignments and articulation parameters during joint optimization.
  • Optimization: Joint-type prediction uses a 3K–5K-step warm-up, followed by 20K steps of ArtGS training with joint-type constraints.The warm-up takes 30–50 seconds, while constrained training takes 5–7 minutes per object.
  • Optimization: The Chamfer-distance loss regularizes only objects with revolute joints because nearest-Gaussian matching is ineffective when prismatic-joint Gaussians overlap substantially.Gaussian densification clones or splits a Gaussian when its center gradient exceeds ε_densify.

B ADDITIONAL DISCUSSIONS

Additional analyses show that ArtGS addresses state-dependent visibility failures and that accurate part assignment is central to downstream reconstruction. The appendix evaluates state-wise metrics and reports that part separation supports joint estimation and mesh reconstruction.

  • Visibility Problem: DTA reconstructs high-visibility states well but performs markedly worse in low-visibility states, especially for prismatic-joint objects.The issue involves difficulty capturing geometry and articulation of partially occluded parts.
  • Visibility Problem: Table A.1 reports state-0, state-1, and two-state-average metrics over 10 trials, with Axis Pos. omitted for prismatic joints.Best results are highlighted on the average of the two states.
  • Visibility Problem: ArtGS jointly optimizes multiple articulation states to leverage complementary observations, preserve cross-state part consistency, and infer occluded geometry.These capabilities yield more consistent reconstructions across varying object configurations.
  • Significance of Part Assignment: Accurate part assignment is identified as a prerequisite for high-quality articulated reconstruction because incorrect separation prevents reasonable joint parameter estimation.ArtGS addresses this through center-based segmentation and improved clustering initialization.
  • Significance of Part Assignment: Even inaccurate joint parameter estimation can coexist with correct part separation, whereas failed part division makes reasonable joint estimation impossible.The analysis draws on qualitative and quantitative results from Fig. A.3 and Tab. 3.
  • Significance of Part Assignment: Table A.2 reports Axis Pos. as mean ± std over 10 trials averaged across two states, with values multiplied by 1000.The best results are highlighted.

C LIMITATIONS

ArtGS is limited by its two-state formulation, increasing segmentation difficulty for similar parts, initialization sensitivity, and lower mesh fidelity than NeRF-based methods. The authors identify multi-state, temporal, improved initialization, and Gaussian-reconstruction extensions as future directions.

  • Stability of Randomness: Severe initialization errors in part centers can still cause part mis-segmentation, despite improved stability from canonical-Gaussian and part-assignment initialization.Better initialization of articulation parameters also improves performance, according to the authors.
  • Limited States: ArtGS models only two states, which may not capture the complexity of real-world multi-part objects.The authors propose extending the method to multiple states or continuous monocular video reconstruction.
  • Limited States: Distinguishing parts with similar joint axes and motion patterns becomes harder as the number of parts increases.Parallel drawers are given as an example of this segmentation challenge.
  • Mesh Reconstruction Fidelity: The original Gaussian Splatting implementation has lower mesh reconstruction quality than NeRF-based methods.The authors suggest integrating recent Gaussian-Splatting reconstruction advances to improve fidelity.

D.1 ADDITIONAL QUANTITATIVE COMPARISONS

Additional comparisons evaluate rendering quality and mesh reconstruction, while also documenting limitations and optimization behavior. ArtGS is reported as comparable or superior to PARIS, with stronger advantages on complex and real-world objects.

  • Scaled Axis Pos Metric: ArtGS demonstrates superior performance to DTA when Axis Pos is multiplied by 1000, addressing minimal variation under the standard scaling.The paper multiplies Axis Pos by 10 in Tables 1 and A.1 and additionally reports multiplication by 1000 in Table A.2.
  • Perception-based Metrics: LPIPS, SSIM, and PSNR evaluate rendering quality on PARIS data, with results reported in Table A.3.The evaluation reports results averaged across two states and highlights the best results.
  • Perception-based Metrics: ArtGS achieves comparable or superior performance relative to PARIS on perception-based metrics.The comparison concerns rendering quality, while the primary evaluation focus remains mesh reconstruction and articulation estimation.
  • Limitations: TSDF-based mesh extraction constrains CD-w performance on simple synthetic objects relative to marching-cubes methods using NeRF.The limitation is attributed to fundamental differences between TSDF and marching cubes; ArtGS advantages become evident for complex or real-world objects.
  • Failure Cases: Incorrectly initialized part centers can persist through optimization and degrade reconstruction for misaligned parts in real-world multi-part objects.The errors are associated with sensor noise, occlusion, and varying illumination; manual correction improves results.
  • Failure Cases: Parts with identical motion across states can be merged into a single learned joint, preventing ArtGS from distinguishing independently movable parts.The paper suggests that additional states could provide richer motion information for better part separation.

D.4 ADDITIONAL QUALITATIVE COMPARISONS

Additional qualitative comparisons cover multiple datasets, failure cases, canonical-Gaussian evolution, interpolation, and intermediate time states. The visualizations emphasize how ArtGS initializes, refines, and evaluates articulated reconstructions.

  • Additional Qualitative Comparisons: Additional qualitative comparisons are provided across different datasets.The supplementary section presents further visual comparisons following the main results.
  • Failure Cases: Figure A.1 illustrates ArtGS failure cases, including initialized and optimized canonical Gaussians and manually corrected erroneous part centers.The prefix “M” denotes manual correction of erroneous part centers.
  • Evolution of Canonical Gaussians: Figure A.2 visualizes canonical-Gaussian part assignments and centers as dense static and sparse dynamic Gaussians densify and refine during training.The visualization tracks simultaneous refinement of Gaussian centers and part assignments.
  • Qualitative Results: Additional qualitative results are shown on ArtGS-Multi, PARIS interpolation data, and intermediate states from T = 0 to T = 1.The listed intermediate states include T = 0, 0.25, 0.5, 0.75, and 1.
Loading 2502.19459v2…