PaperlayerView exact revision on arXiv

Source-linked AI summary

AuRA: From Atoms to Bits through Autonomous Robot Interaction

Xinchen Yu, Yoonkyo Jung, Haoran Chen, Zikui Cai, Aayush Talreja, Seungjae Lee, Hyunjin Kim, Furong Huang

arXiv:2610.34452v1cs.RO

TL;DR

Real-to-sim reconstruction requires observations of object geometry, appearance, and physical behavior, but acquisition often relies on human manipulation or annotation. AuRA automates interaction-data collection and reconstruction using feasibility-screened grasps, robot-state-derived controller trajectories, and registered pre-interaction priors. Across 13 deformable objects, it reduces collection effort and improves most future-prediction metrics over human-collected PhysTwin, with similar mean silhouette IoU and Chamfer distance on evaluated unseen interactions.

  • Problem

    Reconstruction workflows often rely on human manipulation or annotation, and autonomous acquisition must address feasible interactions, reliable gripper-motion inputs, and robot-induced occlusion.

    Source: §I · p. 1

  • Method

    AuRA automates object-interaction data collection and reconstruction using feasibility-checked grasps, robot-state-derived controller trajectories, and registered pre-interaction geometry and appearance priors.

    Source: §I · pp. 1–2

  • Results

    Across 13 deformable objects, AuRA reduces active human labor and wall-clock collection time per successfully reconstructed asset by 88.5% and 55.4%, respectively, and improves seven of nine future-prediction metrics over human-collected PhysTwin.

    Source: §I · p. 2

  • Takeaways & Limitations

    Across the evaluated objects, autonomous robot interaction makes real-to-sim asset acquisition more efficient and reliable while achieving comparable reconstruction quality to human-operated acquisition.

    Source: §V · p. 7

Abstract

from arXiv · show

Simulation enables robot-learning experiments at scale, but acquiring physics-informed digital twins often requires human manipulation and manual annotation of recorded interactions. To reduce this reliance on human effort, we present AuRA, a system that automates robot interaction and data preparation for digital-twin reconstruction. AuRA combines execution-aware grasp selection, gripper-motion measurements derived from robot states, and registered pre-interaction geometry and appearance priors. These choices enable autonomous collection, avoid reliance on visual gripper tracking, and recover object information obscured by the robot. Across 13 deformable objects, AuRA reduces active human labor and wall-clock collection time per successfully reconstructed asset by 88.5% and 55.4%, respectively. Relative to the human-collected PhysTwin baseline, AuRA improves seven of nine metrics for future prediction under known gripper motion, including a 28.1% reduction in one-way $L_1$ Chamfer distance. It also achieves similar mean silhouette IoU and Chamfer distance under the evaluated unseen interactions. These results highlight the potential of autonomous robot interaction to support more efficient digital-twin acquisition with comparable reconstruction quality.

I. INTRODUCTION

Digital-twin acquisition often depends on human manipulation and annotation, while autonomous collection must address feasible interactions, reliable gripper-motion inputs, and robot-induced occlusion. AuRA combines autonomous interaction with robot-state-derived controller trajectories and registered pre-interaction priors, reducing collection effort while maintaining reconstruction quality.

I. INTRODUCTION

  • Human manipulation and annotation are common in reconstruction workflows, limiting how quickly asset collections can expand.Autonomous robot interaction is proposed as a way to reduce this effort by collecting the needed observations.Source: §I · p. 1
  • Autonomous acquisition must handle robot and workspace constraints, reliable gripper-motion inputs, and geometry and appearance obscured by the robot.The passages note that visual gripper tracking errors can bias inferred physical parameters and missing surface observations can leave gaps in the physical model and rendered appearance.Source: §I · p. 1
  • AuRA generates grasp candidates from pre-interaction RGB-D observations, checks robot and workspace feasibility, then records RGB-D video and robot states during interaction.Collection proceeds without manual manipulation or recording-boundary annotation after placement and initialization.Source: §I · pp. 1–2
  • Robot states and calibrated gripper geometry provide controller trajectories, separating measurement of robot action from visual tracking of object motion.This avoids requiring visual gripper segmentation, tracking, or depth lifting.Source: §I · p. 2
  • Registered pre-interaction point-cloud and Gaussian appearance priors supplement information missing from interaction observations because of robot occlusion.The captures support both physical acquisition and reconstruction without a prebuilt object asset or separate manual capture stage.Source: §I · p. 2
  • 88.5% less active human labor and 55.4% less wall-clock collection time per successfully reconstructed asset; AuRA also improves seven of nine future-prediction metrics over human-collected PhysTwin.On evaluated unseen interactions, AuRA and PhysTwin achieve similar mean silhouette IoU and Chamfer distance.Source: §I · p. 2

II. RELATED WORK

Research on deformable-object digital twins includes both learned dynamics and explicit physical-model fitting. The cited approaches differ in their representations and reconstruction scope, while AuRA addresses autonomous acquisition of interaction data for reconstruction.

A. Reconstruction and Simulation of Deformable Objects

  • Deformable-object digital twins need dynamics that predict shape changes under applied motion or forces, beyond recovering shape and appearance.Geometric and visual methods use representations including point clouds, neural radiance fields, and 3D Gaussians.Source: A. Reconstruction and Simulation of… · p. 2
  • Learned approaches use graph-based particle or node simulators to predict motion, with recent systems coupling learned dynamics to Gaussian appearance representations.Source: A. Reconstruction and Simulation of… · p. 2
  • Explicit physical-model fitting includes differentiable mesh simulation, spring–mass systems, and material point methods.PhysTwin reconstructs geometry, appearance, and dynamics from sparse-view RGB-D videos; the described acquisition pipelines rely on human manipulation or teleoperation.Source: A. Reconstruction and Simulation of… · p. 2
  • AuRA addresses the complementary challenge of autonomously acquiring interaction data for reconstruction within the explicit physical-model-fitting paradigm.The related work characterizes this as distinct from the reconstruction methods’ acquisition pipelines.Source: A. Reconstruction and Simulation of… · p. 2

B. Robotic Interaction for Object Modeling

Prior robotic-interaction work addresses geometry, appearance, physical-property estimation, or articulated-object modeling, with stated gaps in jointly reconstructing deformation dynamics and renderable appearance. AuRA combines autonomous acquisition, trajectory extraction, registered geometry priors, spring–mass fitting, and Gaussian appearance modeling.

B. Robotic Interaction for Object Modeling

  • Prior active object-modeling methods target shape or appearance without jointly modeling deformation dynamics, while other interaction systems omit a jointly reconstructed renderable appearance model or use rigid-link representations.AuRA brings autonomous acquisition to digital-twin reconstruction encompassing geometry, appearance, and deformation dynamics.Source: B. Robotic Interaction for Object… · p. 2

III. AURA: AUTONOMOUS DATA PREPARATION AND RECONSTRUCTION

  • The pipeline’s multi-view RGB-D capture supports grasp generation and feasibility filtering; robot states and calibrated gripper geometry provide controller trajectories, while registered priors supplement robot-occluded regions.The reconstruction backend uses these inputs for spring–mass optimization and Gaussian rendering.Source: Figure 2 · p. 3
  • AuRA’s pipeline uses grasp-and-lift interaction data, extracted object and controller trajectories, and registered pre-interaction geometry to prepare physical reconstruction inputs.The resulting spring–mass model and aligned Gaussians represent geometry, appearance, and dynamics.Source: §III · pp. 2–3

A. Data Acquisition

AuRA captures a stationary object from multiple RGB-D viewpoints, fuses filtered depth measurements into a world-frame point cloud, and uses it to generate and screen grasp candidates before execution.

A. Data Acquisition

  • Before manipulation, calibrated RGB-D cameras capture the stationary object from multiple viewpoints, and masked depth measurements are fused into a filtered point cloud in the common world frame.Table I lists point-cloud preprocessing parameters for grasp-pose generation.Source: A. Data Acquisition · p. 3, Table I · p. 3
  • AnyGrasp generates candidate grasp poses and opening widths from the point cloud, which AuRA screens for tabletop clearance, inverse-kinematics feasibility, and joint-limit margins.Source: A. Data Acquisition · p. 3
  • The robot executes the highest-confidence candidate that passes the checks while AuRA records synchronized RGB-D videos and robot joint states, stopping automatically when the lift is complete.Source: A. Data Acquisition · p. 3

B. Data Post-processing

AuRA prepares interaction observations and robot-state-derived controller trajectories for reconstruction, then combines the fitted physical model with registered geometry and appearance priors to form a digital twin.

B. Data Post-processing

  • Controller trajectories derived from robot joint states provide gripper motion even when the gripper is occluded, without visual gripper segmentation, tracking, or depth lifting.Calibrated fingertip geometry and robot-to-world transformation determine the paths of gripper-attached controller points.Source: B. Data Post-processing · p. 3
  • A registered pre-interaction point cloud adds surface points only where tracked object points are absent, filling geometry obscured by the robot.Rigid alignment uses PCA initialization and ICP refinement; failed geometric checks retain the identity transformation.Source: B. Data Post-processing · p. 4

C. Asset Reconstruction

  • Object trajectories and controller trajectories drive spring–mass optimization, with tracked object motion supervising the physical-model fit.Gripper-attached controller points connect to object nodes through springs.Source: C. Asset Reconstruction · p. 4
  • A static 3D Gaussian appearance model is trained from pre-interaction RGB-D observations and aligned to the interaction sequence using the point-cloud registration transform.The pre-interaction observations retain appearance information from regions later obscured by the robot.Source: C. Asset Reconstruction · p. 4
  • Weighted blending binds aligned Gaussians to the fitted physical model, transferring simulated deformation for rendering.The resulting digital twin represents geometry, appearance, and dynamics.Source: C. Asset Reconstruction · p. 4

IV. EXPERIMENT

Across 13 deformable objects, AuRA reduces collection time and labor per successfully reconstructed asset and achieves higher end-to-end yield than human-operated collection overall. The study evaluates these outcomes against a human-collected PhysTwin baseline for reconstruction quality.

IV. EXPERIMENT

  • Evaluation covered 13 deformable objects and compared collection efficiency and reliability with human-operated acquisition, while reconstruction quality used human-collected PhysTwin as its baseline.The setup used a Franka arm and three OAK-D RGB-D cameras calibrated to a common world frame.Source: §IV · p. 4

A. Data Collection Efficiency and Reliability

  • 74.3% of AuRA acquisition attempts yielded reconstructed assets, versus 26.9% for human-operated collection overall; time and labor fell 55.4% and 88.5%, respectively.AuRA required 126.7s wall-clock collection time and 32.7s active human labor per asset, compared with 284.4s for both measures in human collection overall.Source: A. Data Collection Efficiency and… · p. 5
  • AuRA detected failures after 88.3s on average, compared with 133.0s for human collection overall.The timed collection comparison includes three human experience groups, with two trajectories per object from each group; untimed backup trials were excluded.Source: A. Data Collection Efficiency and… · pp. 4–5
  • The authors attribute AuRA’s higher reliability to feasibility filtering before execution and controller trajectories derived deterministically from robot states.These choices reject infeasible interaction candidates and avoid downstream reconstruction failures caused by visual gripper tracking.Source: A. Data Collection Efficiency and… · p. 5

1) Resimulation and Future Prediction:

For future prediction under known controller motion, AuRA outperforms PhysTwin on seven of nine metrics, while PhysTwin performs better on most resimulation metrics. Evaluation trains on the first 70% of each sequence and tests on the remaining 30%.

1) Resimulation and Future Prediction:

  • Future prediction evaluates held-out frames under known controller motion, after training on the first 70% of each interaction sequence.Resimulation measures agreement over training frames.Source: 1) Resimulation and Future Prediction · pp. 5–6
  • AuRA outperforms PhysTwin on seven of nine future-prediction metrics, while PhysTwin performs better on eight of nine resimulation metrics.For future prediction, mask IoU rises from 0.4607 to 0.5001 and one-way L1 Chamfer distance falls from 23.335 mm to 16.783 mm; PhysTwin retains higher SSIM and lower depth MAE.Source: 1) Resimulation and Future Prediction · p. 6

2) Unseen Interactions:

On evaluated unseen interactions, AuRA with static Gaussians and PhysTwin achieve similar mean performance. AuRA’s pre-interaction priors preserve information from regions obscured by the robot.

2) Unseen Interactions:

  • The evaluation collected one new interaction per asset across 13 objects, using a teleoperated drag-and-lift and recorded robot states to drive reconstructed dynamics.Assets were manually aligned to the physical objects before each interaction.Source: 2) Unseen Interactions · p. 6
  • AuRA with static Gaussians and PhysTwin achieve similar mean performance on unseen interactions.Mask IoU is 0.3861 for AuRA and 0.3849 for PhysTwin; one-way L1 Chamfer distance is 43.554 mm and 43.43 mm, respectively.Source: 2) Unseen Interactions · p. 6
  • Static Gaussians improve all reported appearance and silhouette metrics over first-frame Gaussians, preserving visual information from regions occluded by the robot.The registered static point cloud supplies surface observations missing from the interaction sequence, complementing the appearance prior.Source: 2) Unseen Interactions · p. 6

1) Controller trajectory source:

Robot-state-derived controller trajectories were substantially more reliable than visually tracked controllers on the tested trajectories. Ablations also found that geometry priors improved most metrics, while tracking-point density did not uniformly predict future-prediction performance.

1) Controller trajectory source:

  • Robot-state-derived controllers reconstructed all 26 trajectories (100.0%), versus two cases (7.7%) for visually tracked controllers.The comparison used the same 26 robot-collected trajectories.Source: 1) Controller trajectory source · pp. 6–7

2) Static geometry prior and object tracking methods:

  • The geometry prior improved most metrics across tracking configurations; with it, CoTracker3-m led most resimulation metrics and MV-TAP led most future-prediction metrics.The ablation used the same 26 robot-collected trajectories and static Gaussians; each table cell reports results without and with the prior.Source: 2) Static geometry prior and object… · p. 7, Table IV · p. 7
  • MV-TAP and MVTracker retained 615.96 and 625.46 tracking points per trajectory, while CoTracker3-m and CoTracker3-p retained 2,853.46 and 1,313.92, respectively.With the geometry prior, CoTracker3-m generally performed better than CoTracker3-p in resimulation but worse in future prediction; the authors hypothesize denser supervision fits observed motion without necessarily improving future prediction.Source: 2) Static geometry prior and object… · p. 7

3) Static versus first-frame Gaussians:

On 13 unseen interactions with simulated motion held fixed, static Gaussians improved appearance and surface coverage over first-frame Gaussians, with a modest increase in depth error. The conclusion presents AuRA as an autonomous acquisition and reconstruction pipeline that reduces collection effort and time while achieving higher yield and comparable reconstruction quality.

3) Static versus first-frame Gaussians:

  • Static Gaussians raised mask IoU from 0.3294 to 0.3861 and PSNR from 9.622 to 10.379 dB, while mask recall and depth coverage rose from 0.5821 to 0.6522.The comparison used the same 13 unseen interactions and fixed simulated motion; depth MAE increased from 18.478 to 19.522 mm.Source: 3) Static versus first-frame Gaussians · p. 7
  • The comparison attributes improved appearance and surface coverage to pre-interaction captures, alongside a modest increase in depth error.The reported conclusion is limited to the evaluated unseen interactions.Source: 3) Static versus first-frame Gaussians · p. 7

V. CONCLUSION

  • AuRA combines execution-aware grasp selection, robot-state-derived controller trajectories, and pre-interaction geometry and appearance priors in an autonomous reconstruction pipeline.Across 13 deformable objects, it reduced human labor and collection time while achieving higher end-to-end reconstruction yield than human-operated acquisition and comparable reconstruction quality to PhysTwin.Source: §V · p. 7

Related papers

Loading 2610.34452v1…