PaperlayerView exact revision on arXiv

Source-linked AI summary

PARIS: Part-level Reconstruction and Motion Analysis for Articulated Objects

Jiayi Liu, Ali Mahdavi-Amiri, Manolis Savva

arXiv:2308.07391v1cs.CVcs.AIcs.GR

TL;DR

PARIS addresses joint part-level reconstruction and motion estimation from multi-view images of an articulated object in two states, without 3D or motion supervision. It learns part-level implicit models and jointly estimates motion; experiments report improved reconstruction and motion estimation, with severe occlusion remaining challenging.

  • Problem

    Prior articulated-object methods can require costly 3D supervision and articulation annotation, while category-specific models have difficulty generalizing to unseen objects.

    Source: §1 · p. 1

  • Method

    PARIS jointly learns separate implicit fields for static and movable parts and estimates motion from two-state multi-view RGB images using self-supervision.

    Source: §1 · pp. 1–2, §4 · p. 3

  • Results

    Experiments on synthetic and real data show PARIS significantly outperforms prior work and baselines in shape and appearance quality and motion-estimation accuracy.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The method recovers shape, appearance, and motion better than prior work and baselines, while severe occlusion remains a challenge.

    Source: §6 · p. 9

    Accurate part decoupling and motion estimation rely on sufficient multi-view observations; revolute-joint estimates tend to fail when the movable part is severely occluded or highly symmetric.

    Source: §5.5 · p. 9

Abstract

from arXiv · show

We address the task of simultaneous part-level reconstruction and motion parameter estimation for articulated objects. Given two sets of multi-view images of an object in two static articulation states, we decouple the movable part from the static part and reconstruct shape and appearance while predicting the motion parameters. To tackle this problem, we present PARIS: a self-supervised, end-to-end architecture that learns part-level implicit shape and appearance models and optimizes motion parameters jointly without any 3D supervision, motion, or semantic annotation. Our experiments show that our method generalizes better across object categories, and outperforms baselines and prior work that are given 3D point clouds as input. Our approach improves reconstruction relative to state-of-the-art baselines with a Chamfer-L1 distance reduction of 3.94 (45.2%) for objects and 26.79 (84.5%) for parts, and achieves 5% error rate for motion estimation across 10 object categories. Video summary at: https://youtu.be/tDSrROPCgUc

1. Introduction

Articulated-object perception couples part reconstruction with motion analysis, but prior approaches often require costly 3D supervision or category-specific models. PARIS jointly reconstructs part shape and appearance and estimates motion from multi-view RGB images in two states, without 3D, motion, or semantic annotation.

1. Introduction

  • Prior supervised approaches require 3D supervision and articulation annotation, while category-specific models make generalization to unseen objects difficult.The required supervisory data is expensive and unavailable at scale.Source: §1 · p. 1
  • Reconstruction and motion analysis are intertwined: complete geometry helps motion analysis, while motion parameters provide a signal for reconstruction across articulation states.The paper uses this relationship to avoid explicit 3D data and motion-parameter supervision.Source: §1 · p. 1
  • PARIS jointly reconstructs static and movable part shape and appearance while estimating articulation motion from multi-view RGB images in two states.It uses separate implicit neural fields, composited with estimated motion to form self-supervisory losses from the input images.Source: §1 · pp. 1–2
  • Experiments on synthetic and real data show that PARIS significantly outperforms prior work and baselines in shape and appearance quality and motion-estimation accuracy.Articulated-object understanding is also motivated by applications including robotics, animation, and industrial design.Source: §1 · pp. 1–2

2. Related Work

Prior work addresses articulated-object mobility and reconstruction mainly from 3D inputs, with category generalization and appearance reconstruction remaining distinctions in PARIS’s setting. PARIS instead decouples part shape and appearance and estimates motion from two sets of RGB images.

2. Related Work

  • Earlier mobility methods use point clouds, depth-image sequences, or segmented point clouds, and include supervised, semi-supervised, and user-command approaches.Semantic segmentation does not necessarily align with part mobility; several learned approaches have limited generalization to unseen categories.Source: §2 · p. 2
  • Ditto [20] takes two 3D point clouds, while CARTO [15] uses stereo images; both differences matter because these works produce geometry without appearance, and CARTO cannot reconstruct parts.The related works discussed also require 3D supervision and articulation annotation, unlike PARIS’s image-only self-supervision.Source: §2 · p. 2
  • Prior implicit articulated-object models include 3D-supervised NASA [8] and A-SDF [29], category-level Wei et al. [47], and CLA-NeRF [43], which uses segmentation maps and articulation annotation.PARIS decouples part shape and appearance without category knowledge and estimates motion end-to-end from two sets of RGB images in distinct static states.Source: §2 · p. 2

3. Problem Statement

The task takes multi-view images of an unknown-category object in two arbitrary articulation states and seeks part-level geometry, appearance, and motion estimates. The setup assumes one moving part and rotation-only or translation-only motion.

3. Problem Statement

  • The input is multi-view RGB images with camera parameters at start state t = 0 and end state t = 1, and the object is assumed to have one movable part and one static part.The goal includes decoupling both geometry and appearance so the object can be articulated to unseen states and rendered from arbitrary views.Source: §3 · p. 3
  • The movable part is assumed to exhibit rotation or translation only, and the motion type determines the parameterization used for estimation.A revolute joint uses pivot point p and unit quaternion q; a prismatic joint uses unit axis a and translation distance d.Source: §3 · p. 3
  • The chosen motion transformation is plugged into the training pipeline to jointly optimize the motion parameters.Source: §3 · p. 3

4. Method

PARIS jointly learns static and movable part representations while estimating articulation from object-level observations. It composites the fields for rendering and uses image and mask supervision, with an additional regularizer for field overlap.

4. Method

  • Motion maps moving content from an input state to a canonical state, separating the movable component from the component that remains still.The motion transformation serves as a cue for part separation.Source: §4 · p. 3

4.1. Composite Neural Radiance Fields

  • Two static-scene fields represent the static part and the movable part at canonical t*=0.5; motion transforms the latter between the two observed states.The fields are not conditioned on state, and the movable field does not model continuous dynamics.Source: §4.1 · pp. 3–4
  • For each state, ray marching queries the static and transformed movable fields, sums their densities for sampling, then composites their weighted colors.The additive composition allows a high-density sample from either field to terminate a ray during rendering.Source: §4.1 · pp. 4–5
  • On average, PARIS outperforms other methods for whole-object and movable-part reconstruction, with static-part quality comparable to Ours-ICP.Table 1 reports surface quality for the reconstructed object and parts.Source: Table 1 · p. 5

4.2. Supervision Losses

  • RGB and object-mask losses supervise rendering and opacity; Lprob counters mobile-field noise in overlaps by encouraging each ray's color to come from one field.The authors report overlap noise especially when the static part occupies a large volume.Source: §4.2 · p. 5
  • PARIS performs best on joint-axis prediction across 10 examples, predicts accurate revolute angles, and performs better on prismatic joints.The joint-state metrics are geodesic distance for revolute joints and translation error for prismatic joints.Source: Table 2 · p. 6, Table 3 · p. 6

5. Experiments

The experiments evaluate PARIS on synthetic and real articulated objects against reconstruction and motion baselines, using geometric, motion, and rendering metrics. Results show stronger average part segmentation and motion estimation, alongside limitations from sparse views, occlusion, and geometric symmetry.

5.1. Dataset

  • Synthetic evaluation uses PartNet-Mobility instances from 10 categories with 64–100 rendered views per state; real-world evaluation uses MultiScan objects in two states.Real RGB frames train the method, while reconstructed meshes provide evaluation ground truth.Source: §5.1 · p. 6

5.2. Baselines

  • Baselines include A-SDF, which uses ground-truth SDF samples but cannot predict parts or explicit motion, Ditto, which receives ground-truth point clouds, and Ours-ICP, which uses separate image-trained SDFs, CSG, and registration.Ours-ICP estimates the movable-part transformation with global registration and ICP.Source: §5.2 · pp. 6–7, Figure 4 · p. 7

5.3. Evaluation Metrics

  • Evaluation measures whole-object and part Chamfer-L1 distances, joint-axis and joint-state errors, and novel-view PSNR and SSIM.Chamfer distances use 10,000 sampled surface points; rendering metrics average 50 novel views per instance and state.Source: §5.3 · p. 7

5.4. Experiment Results

  • PARIS outperforms baselines on average for part segmentation, wins 7 of 10 cases for both parts, and estimates revolute and prismatic motion significantly more accurately.Ditto performs well on its three seen categories but drops significantly on unseen categories; its point-cloud input also gives it an advantage for deep-container whole-object reconstruction.Source: §5.4 · p. 7
  • Regularization improves average surface reconstruction for objects and parts, especially movable parts with insufficient views, and the ablation reports gains in geometry, motion, and appearance.The qualitative ablation highlights reduced noise in the movable part.Source: §5.4 · p. 8, Table 4 · p. 8, Figure 7 · p. 8
  • On real-world objects, PARIS has comparable static-part and whole-object surfaces to Ditto, less accurate movable-part surfaces, and better motion estimation.Both methods assume the object geometries in the two states are pre-aligned and are affected by alignment and reconstruction errors.Source: Table 5 · p. 8, §5.4 · pp. 8–9
  • Using state t*=0 instead of the virtual canonical state t*=0.5 reduces performance across aspects except appearance, with the largest degradation in movable-part geometry.The virtual state uses supervision from both observed states for motion and segmentation.Source: §5.4 · p. 8, Table 6 · p. 9
  • Performance declines as input views decrease, with a significant drop below 16 views.The study downsamples the 64-view results to evaluate inputs from 32 to 4 views.Source: §5.4 · p. 8, Table 7 · p. 9

6. Conclusion

The paper addresses joint part-level shape and appearance reconstruction and motion estimation from multiview RGB images of two arbitrary object states. Experiments show better recovery than prior work and baselines, while severe occlusion and extension to multiple parts remain challenges.

6. Conclusion

  • The work tackles joint part-level shape, appearance, and motion estimation for articulated objects observed in two arbitrary states using multiview RGB images.The authors describe it as the first work to tackle this task from such image input.Source: §6 · p. 9
  • Experiments show recovery of shape, appearance, and motion parameters better than prior work and baselines.Source: §6 · p. 9
  • Severe occlusion remains challenging, multiple parts are a future challenge, and the method assumes the two object states are aligned in world coordinates.The alignment assumption is shared with prior work.Source: §6 · p. 9
Loading 2308.07391v1…