Source-linked AI summary
Ditto: Building Digital Twins of Articulated Objects from Interaction
Zhenyu Jiang, Cheng-Chun Hsu, Yuke Zhu
TL;DR
Articulated-object digital twins need geometry, part structure, and kinematic relations, while existing approaches and asset-authoring workflows limit scalable simulation-ready reconstruction. Ditto jointly learns part-level geometry and articulation from before-and-after interaction observations using implicit neural representations. It achieves more accurate geometry and articulation reasoning than baselines, generalizes to real-world objects, and spawns the recreated twins in interactive simulation.
Problem
Building articulated-object digital twins is challenging because they require geometry, part composition, and kinematic relations, while existing workflows limit scalable asset acquisition.
Method
Ditto is an end-to-end implicit-neural-representation model that jointly reconstructs part-level geometry and estimates articulation from two visual inputs before and after motion.
Results
Ditto achieves significantly more accurate geometry and articulation results than baselines, generalizes to real-world objects, and directly spawns recreated twins in interactive simulation.
Takeaways & Limitations
The results support autonomous digital-twin building for embodied-AI research and AR/VR applications within the demonstrated scope.
Takeaways & Limitations
Ditto currently segments only mobile and static parts rather than reconstructing a full kinematic tree with multiple joints and parts.
Abstract
from arXiv · showhide
Digitizing physical objects into the virtual world has the potential to unlock new research and applications in embodied AI and mixed reality. This work focuses on recreating interactive digital twins of real-world articulated objects, which can be directly imported into virtual environments. We introduce Ditto to learn articulation model estimation and 3D geometry reconstruction of an articulated object through interactive perception. Given a pair of visual observations of an articulated object before and after interaction, Ditto reconstructs part-level geometry and estimates the articulation model of the object. We employ implicit neural representations for joint geometry and articulation modeling. Our experiments show that Ditto effectively builds digital twins of articulated objects in a category-agnostic way. We also apply Ditto to real-world objects and deploy the recreated digital twins in physical simulation. Code and additional results are available at https://ut-austin-rpl.github.io/Ditto
1. Introduction
Ditto targets scalable, category-agnostic reconstruction of interactive articulated-object digital twins from before-and-after interaction observations. It jointly estimates part geometry and articulation so the resulting twins can support simulation and AR/VR applications.
- Synthetic data can provide abundant training data, but reality gaps limit models trained in simulation, motivating digitization of physical objects.
- Articulated-object digital twins require overall geometry, part composition, and kinematic relations between parts.
- Existing simulated articulated-object platforms depend heavily on graphics designers and engineers, limiting scalable asset acquisition.
- Ditto infers digital twins from visual observations before and after motion, addressing motion-based segmentation, partial-point-cloud reconstruction, and unknown joint-type estimation.
- Ditto uses implicit neural representations to jointly predict part-level geometry and kinematic articulation from partial point clouds.
- Ditto reports superior results across datasets and metrics against baselines and is applied to real-world objects for simulation-ready digital twins.
2. Related Work
Prior work estimates articulation and reconstructs articulated objects, but Ditto combines interactive perception with an architecture that separately supports geometry and articulation prediction. Its design uses fused point-cloud features and local implicit representations for these outputs.
- Interactive-perception methods estimate articulation from action-generated visual stimuli because articulation can be ambiguous from a single observation.
- Learning-based reconstruction methods predict part segmentation and joint parameters, but point-cloud geometry is unsuitable for physical simulation requiring compact collision geometry.
- Unlike A-SDF's implicit articulation-state code, Ditto builds full 3D meshes for each part and models articulations explicitly for virtual physical interaction.
- Implicit neural representations encode complex 3D shapes smoothly and continuously at high resolution, while local features improve scalability and fine detail.
- Ditto's architecture fuses before-and-after point clouds, then uses separate decoder branches for geometry reconstruction and articulation estimation.
3. Problem Formulation
The task is to reconstruct a simulation-importable digital twin of an unknown-category articulated object from point clouds captured before and after interaction. The output includes part geometry, joint parameters, and relative state change for static and mobile parts.
- The digital twin is represented in standard 3D formats such as URDF and contains a kinematic tree with part geometry and physical properties.
- The formulation assumes an unknown-category object whose articulation state changes through interaction, with one moved part designated mobile.
- The input is a pair of point clouds P1, P2 ∈ R^N×3 captured before and after interaction.
- The objectives are to segment and reconstruct static and mobile geometry, estimate their connecting joint parameters, and recover relative joint-state change.
- Ditto considers one-dimensional revolute and prismatic joints, parameterized respectively by an axis and pivot or a translation direction and joint state.
4. Method
Ditto builds digital twins by jointly reconstructing part-level geometry and estimating articulation from paired point-cloud observations before and after interaction. Its two-stream encoder and implicit decoders separately support geometry, segmentation, joint-type, and joint-parameter prediction.
- Two-Stream Encoder: Ditto fuses two input point clouds with a two-stream encoder and attention, then propagates fused features into dense geometry and articulation features.PointNet++ subsampling reduces the quadratic attention cost for denser point clouds.
- Geometry Implicit Decoder: The geometry decoder maps a 3D coordinate and local feature to occupancy probability for continuous object reconstruction.Local features are sampled from a structured feature grid using trilinear sampling.
- Articulation Implicit Decoders: Ditto predicts binary static-versus-mobile segmentation because one joint state is assumed to change during interaction.The segmentation decoder conditions its prediction on local articulation features.
- Articulation Implicit Decoders: The articulation decoders predict segmentation labels, joint type, and prismatic or revolute joint parameters from points inside the object.Prismatic joints use translation-axis direction and displacement, while revolute joints use rotation-axis direction, axis position, and rotation angle.
- Training: Training combines geometry and joint losses, including binary cross-entropy for occupancy, segmentation, and joint type, plus orientation and state losses for prismatic joints.The geometry loss optimizes part-level reconstruction, while the joint loss optimizes articulation estimation.
- Explicit Articulated Object Extraction: Explicit part meshes are extracted by masking occupancy queries with segmentation results and applying multiresolution isosurface extraction and Marching Cubes.This converts the learned implicit representation into per-part surface meshes for interactive digital twins.
5. Experiments
Ditto is evaluated on synthetic and human-designed articulated-object datasets, where it reconstructs part-level geometry and estimates articulation more accurately than the baselines. Real-world experiments further demonstrate digital-twin reconstruction and simulation-based robot interaction, while current scope remains limited to two-part objects.
- 5.1. Experimental Setup: Ditto is evaluated on the synthetic and Shape2Motion datasets using quantitative geometry and articulation metrics.The datasets contain procedurally generated and human-designed articulated objects, respectively.
- 5.2. Baselines: Ditto significantly outperforms baselines on all reported metrics across both datasets.The Correspondence and Global Joint baselines perform poorly on articulation estimation.
- 5.2. Baselines: Ditto produces precise mobile-part geometry and accurate joint estimates, whereas baselines show artifacts in reconstructed mobile parts.Dense predictions are aggregated into the final joint estimate, improving robustness and accuracy.
- 5.3. Comparison with A-SDF: Ditto surpasses A-SDF on whole-object Chamfer distance and reconstructs fine-grained details, including cabinet and drawer handles.A-SDF uses category-specific training and test-time optimization, while Ditto is feedforward and category-agnostic.
- 5.5. Ablation Studies: Mobile Chamfer distance is substantially lower for Ditto than for ablated versions, with shared 3D and 2D features producing the worst Mobile CD.Sharing features destabilizes training and harms reconstruction of the interactable mobile part.
- 5.4. Articulated Object Reconstruction: Ditto reconstructs toy cabinets, laptops, and faucets from noisy, incomplete depth-camera point clouds and enables simulated faucet interaction with transferred real-world actions.The faucet digital twin is imported into Robosuite for robot interaction.
- 5.7. Limitations: Ditto currently segments objects into only static and mobile parts rather than reconstructing a full multi-joint kinematic tree.The authors propose consecutive interactions and aggregation as a path toward handling composite objects with multiple joints and parts.
- 5.7. Limitations: Current interactions are manually specified in simulation or performed by humans in the real world, rather than autonomously selected by an agent.The authors identify autonomous active perception as future work.
6. Conclusion
The conclusion presents Ditto as an end-to-end approach for reconstructing articulated-object geometry and articulation from observations before and after motion. It reports improved geometry and articulation reasoning over baselines, real-world generalization, and direct deployment in interactive simulation.
- 6. Conclusion: Ditto jointly learns full-fledged geometry reconstruction and articulation estimation from two visual inputs before and after articulated motion.The model uses an implicit neural representation and interactive perception.
- 6. Conclusion: Ditto achieves significantly more accurate geometry and articulation reasoning than the baselines.The conclusion reports this improvement as the main experimental result.
- 6. Conclusion: Ditto generalizes to real-world objects and allows recreated digital twins to be spawned directly in interactive simulation.The conclusion connects these demonstrations to embodied AI and AR/VR applications.
A. Implementation Details
The implementation uses standard datasets and point-cloud sampling procedures, with PyTorch and Adam for model training.
- A. Implementation Details: The experiments use Shape2Motion and Synthetic datasets, sampling 8,192 points for each input point cloud.Training also samples occupancy, segmentation, and joint-parameter query pairs.
- A. Implementation Details: The models are implemented in PyTorch and trained with Adam using a learning rate of 10^-4 and batch size 8.These are the reported optimization settings.
B. Ablation Study
The ablation study shows that Ditto’s architectural choices improve articulated-object reconstruction while reducing computational cost relative to an alternative feature design.
- Feature design: Using shared 3D feature grids for geometry and articulation achieves similar performance but requires around 20% more memory usage and training time than Ditto.Ditto instead uses a 3D feature grid for geometry and a 2D feature plane for articulation.
- Architecture: Ditto reconstructs articulated objects more accurately than its ablated versions, especially for mobile parts, due to attention-based fusion, separate decoders, and separate feature representations.Concat Fusion and Share Feature produce incomplete or artifact-laden surfaces, while Share Decoder can missegment mobile and static parts.
C. Comparison with A-SDF
Ditto outperforms A-SDF in reconstruction and articulated motion synthesis by explicitly representing part meshes and articulation parameters, avoiding distortions after motion.
- Analysis: Fixing A-SDF’s articulation code to the ground-truth value improves reconstruction close to Ditto, indicating interference between its articulation and shape codes during test-time optimization.A shape code can degrade when the articulation code reaches a local minimum far from ground truth.
- Motion synthesis: Ditto achieves significantly better articulated motion synthesis results than A-SDF.The comparison reconstructs an object, changes its joint state, and measures whole Chamfer distance after manipulation.
- Motion consistency: After motion, A-SDF can produce geometry inconsistent with the initial state, whereas Ditto introduces no unexpected distortion beyond the rigid transformation induced by articulation.A-SDF’s latent articulation representation can widen the drawer body after motion; Ditto explicitly extracts the mobile-part mesh and joint parameters.
D. Joint Type Prediction
Ditto predicts articulation joint types accurately on both evaluated datasets, with perfect accuracy on Synthetic and near-perfect performance on Shape2Motion.
- Synthetic dataset: 100% joint type accuracy is achieved by all methods on the Synthetic dataset.
- Shape2Motion dataset: On Shape2Motion, most methods also achieve 100% joint type accuracy, except the Global Joint baseline and Concat Fusion ablation.Ditto is among the methods achieving perfect accuracy on this dataset.
E. Generalization to Unseen Categories
Ditto generalizes robustly from Shape2Motion training categories to geometrically similar unseen categories, with weaker performance on a more different category.
- Similar categories: Ditto generalizes robustly to unseen drawer, microwave, and fridge categories that are geometrically similar to the training data.The model was trained on four Shape2Motion categories and evaluated on four unseen categories.
- Different categories: Ditto performs slightly worse on the unseen stapler category, which differs more significantly from the training categories.The authors attribute this to shape priors learned from the training data.