PaperlayerView exact revision on arXiv

Source-linked AI summary

Factory: Fast Contact for Robotic Assembly

Yashraj Narang, Kier Storey, Iretiayo Akinola, Miles Macklin, Philipp Reist, Lukasz Wawrzyniak, Yunrong Guo, Adam Moravanszky, Gavriel State, Michelle Lu, Ankur Handa, Dieter Fox

arXiv:2205.03532v1cs.ROcs.GRcs.LG

TL;DR

Robotic assembly remains difficult to simulate accurately and efficiently when interactions are contact-rich. Factory combines physics-simulation methods with robot-learning tools and assembly resources; it reports real-time simulation of more than 1,000 nut-and-bolt assemblies and proof-of-concept policies. The authors identify collision-scheme improvements and broader assets, scenes, and policy training as future work.

  • Problem

    Accurately and efficiently simulating the diverse contact-rich interactions in robotic assembly remains a longstanding challenge.

    Source: §I · p. 1

  • Method

    Factory combines SDF-based collisions, contact reduction, and a Gauss-Seidel solver with assembly assets, environments, controllers, and proof-of-concept reinforcement-learning policies.

    Source: §I · p. 2, §VI · p. 10

  • Results

    Factory reports real-time simulation of more than 1,000 nut-and-bolt assemblies and stable, high-success proof-of-concept policies with realistic forces and torques.

    Source: B. Contact Reduction and Solver… · p. 5, §VI · p. 10

  • Takeaways & Limitations

    The authors treat successful policy training on contact-rich nut-and-bolt assembly as qualitative evidence of simulator robustness.

    Source: §V · p. 8

    Planned SDF improvements include handling thin-shell and low-tessellation meshes more robustly and reducing the SDF memory footprint.

    Source: §VII · p. 11

Abstract

from arXiv · show

Robotic assembly is one of the oldest and most challenging applications of robotics. In other areas of robotics, such as perception and grasping, simulation has rapidly accelerated research progress, particularly when combined with modern deep learning. However, accurately, efficiently, and robustly simulating the range of contact-rich interactions in assembly remains a longstanding challenge. In this work, we present Factory, a set of physics simulation methods and robot learning tools for such applications. We achieve real-time or faster simulation of a wide range of contact-rich scenes, including simultaneous simulation of 1000 nut-and-bolt interactions. We provide $60$ carefully-designed part models, 3 robotic assembly environments, and 7 robot controllers for training and testing virtual robots. Finally, we train and evaluate proof-of-concept reinforcement learning policies for nut-and-bolt assembly. We aim for Factory to open the doors to using simulation for robotic assembly, as well as many other contact-rich applications in robotics. Please see https://sites.google.com/nvidia.com/factory for supplementary content, including videos.

I. INTRODUCTION

Factory targets the difficulty of simulating assembly’s complex contacts at useful speed, combining collision and solver methods with robot-learning tools. Its introduction presents a contact-rich simulation module, an assembly suite, and proof-of-concept nut-and-bolt policies.

I. INTRODUCTION

  • Assembly is difficult to automate, while industrial methods can be restrictive and research approaches often trade reliability or cycle time for robustness to variation.Assembly spans operations from peg and connector insertion to threaded fastening, wire processing, cable routing, and soldering.Source: §I · p. 1
  • Contact-rich assembly simulation must handle tight-clearance geometry and many constraints efficiently; the introduction says no existing simulator achieves real-time simulation of 1,000 threaded-fastener environments.Thread collisions matter because nuts can move in six degrees of freedom and encounter cross-threading or jamming; mesh-based collision can exceed memory and compute limits.Source: §I · p. 1
  • 1,000 simultaneous nut-and-bolt assemblies run in real time on a single GPU in Factory’s example.The module combines signed-distance-function-based collisions, contact reduction, and a Gauss-Seidel solver.Source: §I · p. 2
  • Factory provides 60 designed assets, three robotic assembly environments, and seven classical controllers, with support for user-defined additions.The suite includes the Franka robot and rigid-body assemblies from the NIST Assembly Task Board 1.Source: §I · p. 2
  • Proof-of-concept Franka policies are trained and tested in Isaac Gym for nut-and-bolt assembly, with simulated contact forces compared against real-world literature values.The authors report strong consistency between those forces and literature values.Source: §I · p. 2

1) Geometric Representations:

  • Factory uses discrete voxel SDFs for collision detection; SDFs represent complex geometry and provide distance and gradient lookups, but require offline preprocessing that can be time- and memory-intensive.The paper frames contact generation and solving non-penetration constraints as core challenges in contact-rich simulation.Source: 1) Geometric Representations · p. 2, A. Contact-Rich Simulation · p. 2

2) Contact Reduction:

Contact reduction limits contact counts while preserving dynamics, making a Gauss-Seidel solver practical for contact-rich scenes. The section relates solver choices to nut-and-bolt simulation, whose prior benchmark was one pair at 1/20 real-time.

2) Contact Reduction:

  • Factory combines normal similarity, penetration depth, and an area-based metric to reduce contacts; a nut-and-bolt scheme can shrink from about 1e6 contacts to about 1e4.Contact reduction clusters contacts and reduces them within bins; the stated goal is to preserve accurate dynamics while limiting memory and stability costs.Source: 2) Contact Reduction · p. 2

3) Solvers:

  • Contact reduction lets Gauss-Seidel outperform Jacobi on numerous contact-rich interactions, despite Jacobi’s parallel operations.Gauss-Seidel performs serial operations, whereas Jacobi can leverage GPU acceleration but converges more slowly.Source: 3) Solvers · p. 3

4) Benchmark Problem:

  • The paper focuses on rigid-body nut-and-bolt simulation for real-time learning, rather than FEM, which captures deformation but may take about 1–10 minutes per pair on CPU.Finite thread clearances allow six-degree-of-freedom motion, making the assembly a challenging contact-rich benchmark.Source: 4) Benchmark Problem · p. 3, Figure 2 · p. 3
  • Factory demonstrates real-time simulation of 1,000 nut-and-bolt pairs, compared with the established general-purpose simulator benchmark of one pair at 1/20 real-time.The paper notes that reinforcement-learning applications benefit from roughly 10^2–10^3 contact-rich interactions in real time.Source: 4) Benchmark Problem · p. 3

B. Robotic Assembly Simulation

  • Earlier nut-and-bolt studies used different contact formulations and scene settings; Factory identifies one pair at 1/20 real-time as the general-purpose simulator benchmark.The cited approaches include symbolic elimination, particle-based contact, incremental potential contact, and configuration-space contact models.Source: 4) Benchmark Problem · p. 3, B. Robotic Assembly Simulation · p. 3

1) Simulation:

Factory addresses the shortage of realistic, tight-clearance assembly simulation in robotics simulators with contact-rich simulation methods and robot-learning tools. Its approach combines SDF collisions, contact reduction, and a Gauss-Seidel solver, while targeting parallel on-policy learning without demonstrations.

1) Simulation:

  • Few robotics-simulator studies handle realistic scales, clearances, and complex assembly geometries; Factory’s PhysX and Isaac Gym module simulates NIST Task Board 1 components with accurate models and real-world clearances.Prior simulated assembly tasks generally used larger clearances and simpler geometries.Source: 1) Simulation · pp. 3–4

2) Reinforcement Learning:

  • Factory targets parallel on-policy reinforcement learning to avoid demonstrations and corrections, addressing the sample inefficiency of on-policy methods and brittleness of common off-policy approaches.Model-based methods are sample-efficient but difficult to apply to contact-rich tasks with discontinuous, nonlinear dynamics and unknown material parameters.Source: 2) Reinforcement Learning · p. 4

III. CONTACT-RICH SIMULATION METHODS

  • The PhysX module combines SDF collisions, contact reduction, and a Gauss-Seidel solver to simulate highly detailed interactions faster than previous efforts.The combined methods target fast, accurate, and robust simulation of contact-rich interactions.Source: §III · p. 4

A. SDF Collisions

  • Signed distance functions represent surfaces implicitly by distance values, with their gradients providing contact normals; Factory samples mesh SDFs into 3D textures for GPU lookups.SDF collisions generate one contact per mesh face using iterative minimization to find the closest point on that face.Source: A. SDF Collisions · p. 4

B. Contact Reduction and Solver Selection

Contact reduction lowers the contact and memory burden enough to make Gauss-Seidel solving and highly parallel simulation practical. The implementation reduces contacts on the GPU, and evaluations report large reductions and real-time execution across contact-rich scenes.

B. Contact Reduction and Solver Selection

  • Reducing contacts addresses the memory burden of constraints: 16k contacts require about 2.5 MB per timestep, and a Jacobi solver needs 8 substeps and 64 iterations for stable simulation.With reduced contacts, the text estimates Jacobi memory-bandwidth requirements fall from 76.8 GB/s to 1.44 GB/s.Source: B. Contact Reduction and Solver… · pp. 4–5
  • With contacts reduced to about 300 from 16k, a Jacobi solver could hypothetically simulate 1100 assemblies in real time; a Gauss-Seidel solver then uses 1 substep and 16 iterations for stable simulation of over 1000 assemblies.The contact reduction process operates in GPU shared memory and can make contact generation faster by avoiding extensive global-memory writes.Source: B. Contact Reduction and Solver… · p. 5, C. Implementation of Contact Reduction · p. 5

C. Implementation of Contact Reduction

D. Performance Evaluations

  • Contact reduction can reduce counts by over 2 orders-of-magnitude; the M16 scene has 1024 parallel nut-and-bolt interactions executing in real time, with a 3-orders-of-magnitude speed-up over real-time single-threaded computation.Contact handling time is typically dominant compared to solution time.Source: Figure 3 · p. 5, D. Performance Evaluations · p. 6

IV. ROBOT LEARNING TOOLS

Factory integrates its contact-rich simulation methods into Isaac Gym and supplies assets, assembly scenes, and controllers for robotic learning. The assets address geometry and tolerance shortcomings in publicly available NIST Task Board models.

IV. ROBOT LEARNING TOOLS

  • The PhysX contact methods integrate into Isaac Gym, where users can enable them for arbitrary assets by adding an SDF tag to URDF descriptions.Factory provides 60 assets, 3 robotic assembly scenes for RL training, and 7 classical controllers in Isaac Gym.Source: §IV · p. 6

A. Assets

  • Public NIST Task Board CAD models often lack real-world tolerances or have incompatible, incomplete, or hand-measured connector geometry, which can destabilize or distort simulated dynamics.The simulated board includes assets and RL training environments for all its rigid components.Source: A. Assets · p. 6, Figure 5 · p. 6
  • Factory provides 60 simulation-ready part models, including NIST Task Board parts and dimensional variations, with ISO-conforming nuts, bolts, pegs, and gearshafts.Connector models use manufacturer CAD; their collision meshes simplify external geometry while preserving mating features.Source: A. Assets · p. 6

B. Scenes

Factory provides three Isaac Gym assembly scenes built around NIST Task Board 1 assemblies and a suite of classical controllers for robot motion and force control. The scenes support varied assembly goals, while the controllers offer different ways to command the robot.

B. Scenes

  • Three Isaac Gym scenes cover nut-and-bolt tightening, insertion, and four-part gear meshing, with randomized assets and locations.All scenes were tested with up to 128 simultaneous environments on an NVIDIA RTX 3090 GPU.Source: B. Scenes · pp. 6–7
  • FrankaInsertionEnv displays randomized initial part positions and a goal state with all parts inserted.Source: Figure 6 · p. 7

C. Controllers

  • Seven classical controllers provide joint- and task-space motion, force, and hybrid force-motion control options.Their actions are executed using an explicit integrator to avoid undesired damping.Source: C. Controllers · pp. 7–8

V. REINFORCEMENT LEARNING

The paper tests proof-of-concept reinforcement learning for nut-and-bolt assembly by splitting the long-horizon task into Pick, Place, and Screw phases. In the Pick phase, a keypoint-distance reward supports a policy that succeeds throughout its tested randomization bounds.

V. REINFORCEMENT LEARNING

  • The nut-and-bolt task was divided into Pick, Place, and Screw subpolicies because it is long-horizon and challenging to learn end-to-end.Source: §V · p. 8

A. Shared Framework

  • Pick, Place, and Screw were trained in Isaac Gym using PPO, typically in batches of 3–4 policies on one RTX 3090 GPU.Each policy used 128 parallel environments; a batch required 1–1.5 hours for 1024 policy updates.Source: A. Shared Framework · p. 8

B. Subpolicy: Pick

  • The Pick policy used damped-least-squares IK targets and a keypoint-distance reward between nut and fingertip approach axes.The keypoint formulation avoided tuning a translation-rotation weighting parameter and was equivariant to hand yaw about the nut axis.Source: B. Subpolicy: Pick · pp. 8–9
  • 100% Pick success was achieved within the tested randomization bounds.The policy learned a fast straight-line approach followed by slow pose refinement.Source: B. Subpolicy: Pick · p. 9

C. Subpolicy: Place

The Place and Screw subpolicies achieved high individual success rates, and their randomized initial states enabled sequential execution. Chaining all three phases reached 74.2% success, while the Screw experiments exposed training challenges and a simplifying no-regrasp assumption.

C. Subpolicy: Place

  • 98.4% Place success was achieved within the tested randomization bounds.An action-gradient penalty smoothed trajectories without materially affecting steady-state error.Source: C. Subpolicy: Place · p. 9

D. Subpolicy: Screw

  • Screw training assumed the end-effector joint limit was removed to avoid regrasping, while exploration encountered collisions, thread misalignment, jamming, and gripper slip.Most experimental groups failed from these pathologies, although complex nut-and-bolt collisions remained stable.Source: D. Subpolicy: Screw · pp. 9–10
  • Screw configuration selection compared controllers and gains, observation spaces, action spaces, and baseline rewards; the best agents used OSC motion control, low proportional gains, and a 2-DOF action space.Success meant the nut was less than one thread away from the bolt base; future work was to investigate asymptotic performance with more updates.Source: D. Subpolicy: Screw · pp. 9–10
  • 85.6% Screw success was achieved over 1024 episodes after training the final policy for 4096 gradient updates.Source: D. Subpolicy: Screw · p. 10

E. Sequential Policy Execution

  • 74.2% end-to-end success was achieved by chaining the Pick, Place, and Screw subpolicies.Initial states for each subpolicy were randomized to span the preceding policy’s observed final-state distribution; the approach does not scale to long sequences.Source: E. Sequential Policy Execution · p. 10

F. Contact Forces

Simulated Screw-policy forces were compared with human nut-tightening forces, and the reported force similarity does not establish sim-to-real policy transfer. The discussion presents Factory’s simulation and learning tools, while the limitations and future-work sections identify collision, asset, and policy areas still to address.

F. Contact Forces

  • MMD was 0.01269 for simulated Screw-policy forces versus analogous human nut-tightening forces; less-related tasks had MMD values of approximately 0.1–1.0.The simulated forces were in the middle of human ranges, but the comparison does not guarantee sim-to-real policy transfer.Source: Figure 9 · p. 10, F. Contact Forces · p. 10

VI. DISCUSSION

  • Factory provides 60 assets, three assembly scenes, and seven controllers, alongside proof-of-concept nut-and-bolt policies trained in 1–1.5 hours for four policies on one GPU.The authors state that the methods can also be used for robotics tasks beyond assembly.Source: §VI · p. 10

VII. LIMITATIONS

  • Planned simulation improvements address thin-shell collisions, low-tessellation meshes, SDF memory use, and stiff deformable features.The deformable-feature example is a flexible RJ45 connector tab.Source: §VII · p. 11

A. Future Work

  • Future work includes sim-to-real transfer, image-based policies, tactile sensing, and reducing reliance on human demonstrations or corrections.Proposed image-policy routes include distillation from privileged-information teachers or image inputs for the actor with privileged information for the critic.Source: A. Future Work · p. 11

VIII. CONCLUSION

  • The authors present Factory as an existence proof that efficient simulation and learning of contact-rich interactions is possible in robotics.Source: §VIII · p. 11

APPENDIX

Initial nut-and-bolt experiments compare collision representations, finding that only SDFs enabled accurate, efficient, and robust simulation. The appendix also describes assembly scenes and their rendering.

1) Initial Nut-and-Bolt Experiments:

  • Convex hulls were insufficient, while artifacts in the 951-hull V-HACD decomposition caused inaccurate and unstable M4 nut-and-bolt contact.The decomposition appeared faithful overall, but a closer view revealed spatial artifacts.Source: 1) Initial Nut-and-Bolt Experiments · p. 16, Figure S11 · p. 16
  • Only SDF representations enabled accurate, efficient, and robust simulation; trimesh collisions required 1e-3 s timesteps with 10 substeps and ran at 1/80 real-time.The M4 bolt SDF contained 5.3 million floating-point distance samples.Source: 1) Initial Nut-and-Bolt Experiments · p. 16, Figure S12 · p. 16
  • The appendix describes a vibrating feeder and scripted Franka controller that grasp a nut and tighten it onto a bolt, plus a flange scene that tightens multiple nuts in sequence.The flange scene is only qualitatively evaluated.Source: 1) Initial Nut-and-Bolt Experiments · p. 17

3) Rendering:

  • Rendering notes distinguish path-traced scenes from real-time rendering: Fig. 3 is real-time, while the listed other scenes are path-traced.For simulation-related figures not listed, images were captured directly from PhysX, Isaac Gym, or Onshape.Source: 3) Rendering · p. 17

D. Robot Learning Tools

The robot-learning tools include assembly assets, SDF collision configuration guidance, and utilities for setting goals and generating stable randomized robot-object poses.

1) Assets:

  • The paper says Tables VII and VIII provide details on all assets supplied in this work, and the authors intend the database to grow through community contributions.Source: 1) Assets · p. 17

2) Using Contact Simulation Methods:

  • An <sdf> tag enables SDF-mesh collision when both objects have it; with only one tag, convex-mesh collision applies, and with neither, default convex-convex collision applies.The example specifies an SDF resolution of 256 or 512 voxels along the object's longest dimension.Source: 2) Using Contact Simulation Methods · p. 17

3) Helpful Additions:

  • Assembly assets use zero-positions corresponding to fully assembled parts, while FrankaCalibrate generates stable robot-and-object poses at randomized robot poses for policy training.This supports training short-horizon sub-policies that begin with an object already grasped.Source: 3) Helpful Additions · pp. 17–18

4) Controllers:

The section presents seven robot controllers based on prior mathematical formulations, spanning joint- and task-space motion control, force control, and hybrid control.

4) Controllers:

  • The controller formulations in this work are based on mathematical formulations articulated in.Source: 4) Controllers · p. 18
  • Joint-space inverse differential kinematics uses task-space error as input to an IK algorithm to compute the target joint-position error for torque control.Implemented IK options include Jacobian pseudoinverse, Jacobian transpose, damped least-squares, and adaptive SVD.Source: 4) Controllers · p. 18
  • Joint-space inverse dynamics combines motion control with inertia and gravity terms; the Coriolis and centrifugal vector is ignored during assembly.The passage attributes this omission to small joint velocities.Source: 4) Controllers · p. 18
  • Task-space impedance control maps task-space position and velocity feedback through the Jacobian, with rotational error defined according to the geometric or analytic Jacobian.For the geometric Jacobian, the orientation components of velocity are angular velocity; for the analytic Jacobian, they are axis-angle derivatives.Source: 4) Controllers · p. 18
  • Operational-space motion control uses task-space inertia, Coriolis and centrifugal terms, and gravity; the Coriolis term is ignored for the small velocities of assembly.The task-space inertia matrix can be computed from the joint-space inertia matrix.Source: 4) Controllers · p. 18
  • The open-loop force controller uses a target wrench, while the closed-loop P force controller uses task-space proportional gains.The gains are specified as a diagonal matrix.Source: 4) Controllers · p. 18
  • Hybrid force-motion control uses Boolean diagonal selection matrices to designate motion- and force-controlled axes, which need not be complementary.Source: 4) Controllers · p. 18

E. Reinforcement Learning

The reinforcement-learning section documents PPO settings and comparisons for the Screw task, alongside simulation resource and asset details. The supplied table captions define metrics and comparison scope but do not provide outcome values.

E. Reinforcement Learning

  • The simulation tables cover solver memory-bandwidth requirements before and after contact reduction and geometric representations used across simulator test scenes.The solver caption notes that other limitations can reduce the theoretical maximum scene size by an order of magnitude.Source: Table V · p. 19, Table VI · p. 19
  • The supplied materials specify manufacturing-tolerance clearance ranges for assembly parts and include a histogram of torque and force norms collected during Screw-policy inference.The histogram data comes from 128 environments and 1024 episodes; extreme outliers were rejected for visualization.Source: Figure S17 · p. 19, E. Reinforcement Learning · p. 20
  • The appendix provides PPO parameters for Pick, Place, and Screw sub-policies and reports that a small fixed learning rate consistently achieved higher returns than an adaptive KL-threshold schedule.The actor and critic networks are MLPs with a shared trunk, and observations, actions, advantages, and values were normalized.Source: Table IX · p. 20
  • The Screw-task controller-and-gain comparison reports success rate, time to success, reward, and joint torque, with each cell averaged over 3 seeds.The caption defines success rate as the fraction of successful episodes and time to success as mean timesteps to success.Source: Table X · p. 21
  • The Screw-task action-space comparison includes observation-space tests and reports success rate, time to success, reward, and joint torque averaged over 3 seeds.Source: Table XI · p. 21
  • The baseline-reward comparison reports success rate, time to success, and joint torque averaged over 3 seeds; mean reward is omitted because baseline reward ranges differ.The caption defines ||d|| as keypoint distance.Source: Table XII · p. 21
Loading 2205.03532v1…