PaperlayerView exact revision on arXiv

Source-linked AI summary

Articulated Object Reconstruction from Rest-State Observation

Daeun Lee, Jaeah Lee, Woosung Kim, Haebeom Jung, Jaesik Park

arXiv:2607.27749v1cs.CVcs.RO

TL;DR

Rest-state articulated reconstruction must infer object parts and motion from a single closed configuration, where motion cues are absent. Rest2Art cross-validates pretrained-model predictions against explicit mesh geometry and uses generated motion hypotheses for joint fitting, achieving strong part and joint recovery across synthetic and real-world inputs.

  • Problem

    Rest-state reconstruction seeks to infer articulated parts and motion from closed configurations alone, which are common but provide no observed articulation cues.

  • Method

    Rest2Art cross-validates vision-language and segmentation predictions on an explicit mesh and fits joint parameters from video-diffusion articulation hypotheses using geometric consistency.

  • Results

    The method achieves the highest type accuracy among rest-state methods on both datasets, competitive axis estimation overall, and outperforms reconstruction baselines on real-world type and angular accuracy.

  • Takeaways & Limitations

    Cross-validating inconsistent pretrained-model outputs against mesh geometry provides a viable path to metric-scale rest-state articulation estimation.

  • Takeaways & Limitations

    Joint-axis estimation is ambiguous from monocular motion tracks alone, requiring mesh-based penalties and boundary cues for disambiguation.

Abstract

from arXiv · show

Building interactive digital twins requires recovering both 3D geometry and the kinematic structures that govern how objects articulate. Yet existing methods for articulated object reconstruction require explicitly observable motion from multiple articulation states. We introduce a rest-state formulation that reconstructs articulated objects from a single closed configuration, an inherently ill-posed setting where geometry, semantics, and motion priors compensate for the absence of motion cues. Our framework adopts an explicit mesh as an intermediate representation for cross-model verification and fusion, reconciling noisy outputs from vision-language and segmentation models into spatially consistent part structures. To estimate joint parameters without observed motion, we use a video diffusion model to synthesize articulation hypotheses and validate them through geometric consistency. Our approach achieves accurate part decomposition and physically plausible articulation, performing competitively with motion-observing reconstruction-based, generation-based, and modular pretrained-model baselines.

1 Introduction

This section introduces rest-state articulated object reconstruction: inferring geometry, part structure, and articulation from a single closed configuration without observed motion. The framework addresses this ill-posed problem by reconciling noisy model predictions through an explicit mesh and validating synthesized articulation hypotheses geometrically.

  • Motivation: Existing reconstruction methods generally rely on observed articulation states or prior knowledge of the number of movable parts, unlike the proposed rest-state setting.Some alternatives use feed-forward prediction or foundation models to relax these constraints, but the introduction frames rest-state reconstruction as the target problem.
  • Motivation: Rest-state reconstruction infers how openable objects articulate from their closed configurations alone, where geometry, semantics, and motion remain ambiguous without observed motion.Closed configurations are common in product photos, internet imagery, and scenes, making them a practical input state.
  • Method: An explicit mesh provides spatially consistent part boundaries, occlusion reasoning, and confidence-weighted aggregation of noisy, view-dependent VLM and segmentation predictions.The mesh serves as an intermediate representation for reconciling incomplete part hierarchies and missed parts across views.
  • Method: The framework iteratively co-refines Vision-Language Model and segmentation outputs, using their disagreements to drive mutual correction before lifting validated evidence onto the mesh.This design combines model disagreement with confidence-weighted per-part aggregation to produce a coherent 3D result.
  • Method: A video diffusion model synthesizes articulation hypotheses, which are validated through geometric consistency to estimate joint parameters without observed motion.Naïve video diffusion can hallucinate nonexistent components, motivating geometric validation.

2 Related Work

Prior articulated-object reconstruction methods estimate geometry and kinematics using category priors, scene optimization, foundation models, or implicit representations, but generalization remains challenging. Rest-state articulation additionally depends on identifying movable parts from weak geometric and visual cues, motivating recent zero-shot 3D segmentation approaches.

  • Articulated Object Reconstruction: Articulated-object reconstruction jointly estimates 3D geometry and kinematic structure across movable parts.Existing approaches include category-specific priors, predefined kinematic templates, per-scene optimization, foundation models, and implicit object-centric representations.
  • Articulated Object Reconstruction: Category-specific priors and predefined kinematic templates limit articulation estimation for unseen objects.More recent methods instead use per-scene optimization or foundation models for part-level understanding.
  • Zero-Shot 3D Part Segmentation: Closed configurations make movable-part identification difficult because uniform colors, box-like geometry, and flush boundaries provide little discriminative signal.This challenge is central to articulating objects observed only in their rest state.
  • Zero-Shot 3D Part Segmentation: Recent zero-shot 3D part-segmentation methods lift 2D detections onto point clouds, train feedforward point embeddings on web-scale assets, or learn continuous 3D feature fields through contrastive distillation.These approaches provide alternative mechanisms for part-level understanding in 3D.

3 Method

Rest2Art reconstructs articulated objects from a rest-state mesh through part identification, joint estimation, and volumetric completion. It uses mesh-based cross-view evidence and synthesized articulation videos to recover spatially consistent parts and physically plausible joint parameters without observed motion.

  • Pipeline overview: The pipeline identifies movable parts, estimates joint parameters from synthesized articulation videos, and completes each part into a volumetric mesh for physics simulation.These are the three stages of the method.
  • Joint estimation: Joint parameters are estimated by synthesizing articulation videos, tracking points, and fitting rigid-body models against the 3D mesh despite the absence of real motion.The generated video only needs approximately correct motion type and direction because fitting uses the mesh rather than video geometry.
  • Part identification: A VLM hierarchy and SAM3 masks are iteratively cross-verified by comparing detected part counts with hierarchy-predicted instance counts.Excess detections trigger hierarchy revision, while agreement caches masks as validated evidence.
  • Part identification: Validated 2D masks are lifted onto a mesh and fused across reliable views to produce globally consistent 3D part labels.Mesh adjacency then removes disconnected fragments and propagates labels from dominant seed regions across surface topology.
  • Joint estimation: For revolute joints, mesh-based penalties prevent the fitted axis from driving a child part into its parent, while boundary directions refine the ambiguous monocular estimate.Multiple 3D axes can produce identical 2D projections, so tracks alone cannot determine the true rotation direction.

4 Experiments

Experiments evaluate rest-state reconstruction across segmentation, joint estimation, ablations, real-world data, and diverse input modalities. The method combines cross-model verification with video-based hypothesis sampling and mesh-grounded fitting to achieve accurate, robust articulation recovery.

  • Experimental setup: Experiments assess segmentation, joint estimation, ablations, and generalization on ACD, MultiScan, web images, and scanned meshes.All ACD objects are rendered in closed configuration, while MultiScan evaluates real-world robustness.
  • Part segmentation: Our method produces more accurate part boundaries than segmentation baselines and segments interactable parts, including knobs missing from annotations.The co-refinement loop cross-validates part counts and masks between the VLM and segmentation model before lifting evidence to the mesh.
  • Joint estimation: Our method achieves the highest type accuracy among rest-state-applicable methods on both datasets and competitive axis estimation overall.Video-based hypothesis sampling combined with mesh-grounded geometric fitting enables rest-state articulation estimation at metric scale.
  • Generalization: Despite noisy real-world inputs, the method recovers relatively accurate joints and outperforms reconstruction baselines on type accuracy and angular error.The pipeline also generalizes to single images, online product photos, indoor scenes, and scanned meshes through modality-specific or unchanged reconstruction stages.
  • Ablations: Combining VLM hierarchy prediction with segmentation yields the best hierarchy-tree and per-part-count accuracy because the models provide complementary signals.A VLM alone miscounts parts, while segmentation alone cannot infer hierarchical relationships without semantic grounding.
  • Ablations: Mask overlay reduces hallucinated articulation from 23.1% to 11.5% and raises accurate articulation from 69.2% to 80.8% on ACD-HSSD.VBVR LoRA improves motion plausibility, but mask overlay is necessary for subtle part boundaries.

5 Conclusion … Drawer Interior Generation.

Rest2Art reconstructs articulated objects from rest-state observations by cross-validating pretrained-model outputs against explicit mesh geometry, preserving metric scale and physically meaningful joint parameters. Its implementation combines view selection, hierarchy construction, articulation-video synthesis and tracking, mesh solidification, and drawer-tray generation.

  • 5 Conclusion: Rest2Art cross-validates pretrained-model outputs against explicit mesh geometry to produce robust, metric-scale digital replicas with physically meaningful joint parameters.The framework is designed for articulated-object reconstruction from rest-state observations alone.
  • A.1 Key Parameters; A.2 View Selection Strategy: 12 re-rendered views are generated after aligning the mesh OBB to Blender coordinates, using frontal-centered viewpoints that maximize articulated-component visibility.Standalone meshes use Pytorch3D with fixed camera settings; NVS inputs reuse their renderer.
  • A.2 View Selection Strategy: View weights combine detection confidence, log mask area, and parent-child visibility so selected views expose complete part hierarchies without domination by oversized masks.Each normalized weight is the view score divided by the best score for that part.
  • A.3 Hierarchy Tree Construction: GPT-5.2 selects the highest-part-count prediction as the initial hierarchy, then revises it using mask overlays and prunes parts absent from final segmentation.The hierarchy uses a base-rooted connectivity graph, while part identities are inferred from geometry, handle placement, proportion, and motion affordance.
  • A.4 Video Generation Details; Prompt Generation.: Wan2.2-I2V-A14B with VBVR LoRA synthesizes 81-frame, 960 × 960 articulation videos from colored mask overlays, using R = 2 random seeds and accelerated four-step inference.Qwen3-VL generates prompts requiring synchronized, mechanically plausible motion while appending a fixed-camera constraint.
  • Multi-Seed Consensus.: Each seed undergoes independent tracking and joint fitting, with the best seed selected by median displacement and reprojection error; static parts trigger two fallback videos.This multi-seed procedure is intended to recover missed articulations.
  • A.5 Track Extraction and Filtering: CoTracker3 tracks dense-grid points initialized inside eroded masks, truncating tracks when visible fraction falls below τvis = 0.5 to filter drift and degraded frames.Mask erosion removes unreliable boundary points before bidirectional tracking.
  • A.6 Mesh Completion Details; Solidification.; Drawer Interior Generation.: Because interior geometry is unobservable, each part is volumetrized into a watertight solid by inward offsetting and boundary closure, while drawers receive an inward tray scaled to 0.75× their extents.Base thickness is δ = 0.015 m, adaptively clamped near adjacent parts; tray wall thickness equals δ and depth uses available space.

A.7 Runtime and Memory Analysis

The analysis reports Rest2Art’s per-stage runtime and GPU-memory usage on ACD using one NVIDIA H200 GPU, with video generation as the dominant cost. External mesh reconstruction is excluded from the reported Rest2Art runtime.

  • Measurement setup: Per-stage runtime and GPU-memory usage are measured on ACD with a single NVIDIA H200 GPU, including model-loading time in each entry.Table B summarizes mean runtime per stage, while Fig. A reports the per-stage GPU-memory profile.
  • Runtime breakdown: Video generation is the dominant bottleneck, accounting for approximately half of the total runtime.
  • Runtime scope: Mesh reconstruction uses an external NVS pipeline, such as 2DGS, and is excluded from the Rest2Art runtime.

B Experimental Details … C Additional Experiments and Analyses

The supplementary sections specify how Rest2Art constructs fair two-state inputs, filters camera registrations, preprocesses MultiScan instances, and organizes additional analyses. They also document the per-stage resource profile and the scope of experiments examining convergence, motion-source robustness, segmentation, and failure modes.

  • B Experimental Details: Rest2Art’s per-stage pipeline includes co-refinement, segmentation, prompt generation, video generation, tracking, joint fitting, and mesh completion.The listed stages use 30, 3, 18, 172, 6, 7, and 40 units, respectively.
  • B Experimental Details: The experimental details include per-stage VRAM measurement on the ACD dataset and MASt3R-SfM camera registration.These components are presented in Fig. A and the camera-registration subsection.
  • B.1 Two-State Input Construction: Two-state reconstruction baselines receive synthesized paired observations because they require multi-view images from two distinct articulation states.The construction procedure is described for reconstruction methods.
  • Why Direct Comparison Is Infeasible.: Direct comparison is infeasible from a single closed-configuration observation, while Articulation in Motion additionally requires an interaction video.The preprocessing therefore constructs the best achievable baseline inputs rather than excluding the comparison.
  • Two-State Input Synthesis.: The synthesis pipeline selects the last physically plausible generated frame as the articulated reference and uses fixed-camera video plus Qwen-Image-Edit with a multi-angle LoRA for view-consistent images.It yields up to 13 images per object, including the input view.
  • Two-State Input Synthesis.: Without mask overlays during video generation, synthesized inputs maintain consistent appearance between states but exhibit higher hallucination rates than Rest2Art.The difference is reported in Fig. 8.
  • Camera Estimation and View Filtering.: Camera estimation anchors MASt3R-SfM registrations to the input view and removes inconsistent views through two-pass MAD-based filtering.The first pass rejects views with z-score greater than τ = 3.0, removes at most 30% of views, and retains at least 5.
  • B.2 MultiScan Preprocessing Details: MultiScan preprocessing extracts object-centric openable instances from scene-level annotations, exporting part-segmented meshes with joint origins and axes before selecting visible RGB views.Object crops use mesh-level objectId and partId labels, projected with provided intrinsics and poses.

C.1 Convergence of Co-Refinement · C.2 VDM Output as Motion Hypothesis · Why Video Diffusion, Not a VLM.

The co-refinement loop converges when VLM part counts agree with segmentation detections across all categories, with a two-round cap resolving most scenes. Remaining cases oscillate under visual ambiguity, while direct VLM joint regression is unreliable from a single rest-state image because it lacks reliable 3D spatial reasoning.

  • C.1 Convergence of Co-Refinement: Convergence requires agreement between VLM-predicted part counts and segmentation detections across all part categories.No further refinement rounds are required once this agreement is reached.
  • C.1 Convergence of Co-Refinement: 84.1% of ACD scenes reach agreement under Rmax = 2 refinement rounds.The cap keeps inference cost low while resolving most disagreements.
  • C.1 Convergence of Co-Refinement: Increasing Rmax to 20 raises agreement only modestly, to 92.8%.The reported gain is limited despite allowing substantially more refinement rounds.
  • C.1 Convergence of Co-Refinement: The remaining 7.2% of cases oscillate between two stable configurations rather than diverging.This behavior reflects non-settling decomposition outputs across iterations.
  • C.1 Convergence of Co-Refinement: Oscillation typically occurs on visually ambiguous objects with low-texture surfaces or weak part boundaries.In these cases, the VLM hierarchy and segmentation outputs fail to settle on one consistent decomposition.
  • Why Video Diffusion, Not a VLM.: Directly regressing continuous joint parameters with a VLM is fundamentally unreliable from a single rest-state image.VLMs are trained predominantly on 2D images and lack reliable 3D spatial reasoning.
  • Why Video Diffusion, Not a VLM.: Reported VLM regression successes typically depend on canonical, axis-aligned viewpoints.Under such viewpoints, joint directions reduce to memorized orientations such as ±x, ±y, and ±z.

Robustness to VDM Artifacts.

The pipeline treats video diffusion outputs as motion hypotheses rather than geometric truth, using the 3D mesh as the geometric ground truth. VBVR LoRA, mesh-grounded fitting, and deformation testing address hallucination, unreliable trajectories, nonrigid drift, and deformation artifacts.

  • Artifact handling: VDM outputs serve as motion-hypothesis sources, while the 3D mesh provides geometric ground truth.This design reflects the plausibility of single-image articulation trajectories and the limitations of video generation.
  • Artifact handling: VBVR LoRA suppresses generation hallucination, while mesh-grounded rigid-body fitting handles unreliable trajectories and nonrigid drift.These mechanisms are used to reduce the impact of artifacts in generated articulation sequences.
  • Deformation robustness: 5.2% empirical median deformation on ACD yields negligible axis error, indicating effective operation within realistic-generation conditions.Deformation is synthetically injected into part geometry, and the resulting axis error is measured.

C.3 Segmentation Quantitative Results

On the ACD dataset, Rest2Art substantially outperforms PartField and Find3D in part segmentation across IoU and mAP. The results attribute this advantage to grounding segmentation in reconstructed mesh geometry and VLM-predicted part hierarchies.

  • Segmentation Quantitative Results: Rest2Art substantially outperforms PartField and Find3D across all evaluated part-segmentation metrics on ACD.Evaluation uses IoU and mAP against the two 3D segmentation baselines.
  • Segmentation Quantitative Results: The method evaluates part segmentation quality using IoU and mAP on the ACD dataset.ACD is cited as the evaluation dataset for the segmentation comparison.
  • Segmentation Quantitative Results: Find3D can achieve relatively high mAP despite low IoU because text-prompted parts are often under-segmented, leaving substantial regions unsegmented.This pattern reflects a mismatch between the two segmentation metrics for text-prompted part predictions.

C.4 Failure Case Analysis · D Applications

Rest2Art’s remaining failures arise mainly from corrupted geometry and subtle articulation with insufficient track displacement, while its applications section evaluates generalization across inputs, captures, object categories, and downstream uses.

  • C.4 Failure Case Analysis: 2 out of 69 videos (2.9%) still exhibit corrupted geometry that prevents meaningful track extraction.The co-refinement pipeline with mask overlay suppresses hallucination artifacts but does not eliminate this failure case.
  • C.4 Failure Case Analysis: Degraded joint accuracy primarily affects objects with subtle or short-range articulation, where track displacement is too small to estimate motion parameters reliably.The limitation reflects insufficient observable motion in the generated tracks.
  • C.4 Failure Case Analysis: Increasing the number of generated videos is proposed to enlarge the hypothesis pool and improve the chance of recovering usable articulation evidence.The passage identifies this as the first direction for improving failure cases.
  • D.1: Rest2Art generalizes to articulated-state inputs without any modification.This is evaluated in Section D.1 beyond the standard rest-state setting.
  • D.2: Rest2Art is validated on real-world captures spanning diverse input modalities and articulation states.This evaluation is presented in Section D.2.
  • D.3: A metaprompt mechanism extends Rest2Art to broader object categories.This extension is described in Section D.3.
  • D.4: Rest2Art is demonstrated in downstream applications.These applications are presented in Section D.4.

D.1 Single Articulated-State Input … D.4 Downstream Applications

Rest2Art generalizes beyond rest-state inputs, producing consistent articulation estimates from scanned meshes and single images, including partially open objects. Its outputs support broader object categories and downstream physics simulation, robotics, and virtual asset integration.

  • D.1 Single Articulated-State Input: Rest2Art generalizes to partially or fully open configurations, despite being designed for rest-state observation.Articulated-state inputs provide part displacement directly, reducing reliance on motion-hypothesis generation compared with closed configurations.
  • D.1 Single Articulated-State Input: A single casually captured image can yield an interactable replica with estimated articulation for both a rest-state cabinet and a partially open microwave.The results use SAM3D for reconstruction before articulation estimation.
  • D.2 Real-World Demonstrations: Real-world demonstrations span raw scanned meshes and single captured images reconstructed via SAM3D, with consistent articulation estimates across modalities.The demonstrations include the same ScanNet++ scene and casually captured in-the-wild images.
  • D.4 Downstream Applications: The downstream assets enable physics-based manipulation of reconstructed furniture and integration of virtual interactable replicas into real-world scenes.The illustrated cabinet includes operable drawers in Isaac Sim [35].
  • D.2 Real-World Demonstrations: Rest2Art exports simulation-ready URDF assets suitable for embodied AI and robotics applications.These assets are produced from the reconstructed interactable replicas.
  • D.3 Generalization to General Object Categories: A metaprompt mechanism broadens the framework beyond openable furniture with prismatic and revolute joints to categories such as laptops and robots.The pipeline uses two VLM calls to adapt part vocabulary to the input object.
  • D.4 Downstream Applications: Rest2Art’s simulation-ready URDF assets can be directly deployed for physics simulation and virtual asset integration.Figure I illustrates physics-based manipulation in Isaac Sim [35] and placement of a reconstructed interactable replica into a real-world scene.
Loading 2607.27749v1…