PaperlayerView exact revision on arXiv

Source-linked AI summary

Robust Multi-Modal Policies for Industrial Assembly via Reinforcement Learning and Demonstrations: A Large-Scale Study

Jianlan Luo, Oleg Sushkov, Rugile Pevceviciute, Wenzhao Lian, Chang Su, Mel Vecerik, Ning Ye, Stefan Schaal, Jon Scholz

arXiv:2103.11512v4cs.AIcs.RO

TL;DR

Industrial insertion has seen limited adoption of learning-based methods, and the paper argues that the design space is a central obstacle. It presents a demonstration-based DDPGfD system, evaluates it on NIST assembly and challenge tasks, and reports high reliability alongside strong perturbation generalization and human-comparable cycle time.

  • Problem

    Industrial insertion has not widely adopted learning-based methods, and DRL has not been systematically evaluated against industry-acceptable benchmarks and solutions.

    Source: §I · p. 1

  • Method

    The paper develops SHIELD, a collection of DDPGfD design choices using human demonstrations as priors, and evaluates policies across assembly tasks and observation modalities.

    Source: §I · p. 2, 1) Experiment setup · p. 5

  • Results

    SHIELD reports 99.8% success across 13K NIST trials, while moving-target HDMI insertion achieved 100% over 1637 episodes and matched human cycle time.

    Source: 3) Results · p. 6, 2) Agent Results · p. 7, §VIII · p. 8

  • Takeaways & Limitations

    The authors present demonstrations and training-time randomization as a general template that outperforms traditional solutions in perturbation handling and generalization with little task-specific tuning.

    Source: §VIII · p. 8

Abstract

from arXiv · show

Over the past several years there has been a considerable research investment into learning-based approaches to industrial assembly, but despite significant progress these techniques have yet to be adopted by industry. We argue that it is the prohibitively large design space for Deep Reinforcement Learning (DRL), rather than algorithmic limitations per se, that are truly responsible for this lack of adoption. Pushing these techniques into the industrial mainstream requires an industry-oriented paradigm which differs significantly from the academic mindset. In this paper we define criteria for industry-oriented DRL, and perform a thorough comparison according to these criteria of one family of learning approaches, DRL from demonstration, against a professional industrial integrator on the recently established NIST assembly benchmark. We explain the design choices, representing several years of investigation, which enabled our DRL system to consistently outperform the integrator baseline in terms of both speed and reliability. Finally, we conclude with a competition between our DRL system and a human on a challenge task of insertion into a randomly moving target. This study suggests that DRL is capable of outperforming not only established engineered approaches, but the human motor system as well, and that there remains significant room for improvement. Videos can be found on our project website: https://sites.google.com/view/shield-nist.

I. INTRODUCTION

The paper frames industrial DRL around efficiency, economy, and rigorous evaluation, and presents SHIELD as a demonstration-based DDPGfD system. It evaluates assembly tasks ranging from the NIST benchmark to moving-target insertion and blind key-lock search.

I. INTRODUCTION

  • Industrial insertion remains difficult to adopt with DRL amid concerns about complexity, brittleness, and long training, while prior studies varied in tasks, actions, connectors, and observations.The NIST benchmark offers representative assembly tasks with defined metrics for comparison.Source: §I · pp. 1–2
  • The paper identifies manually programmed force- or vision-control primitives as the prevailing approach, requiring substantial engineering for each skill and limiting access to automation.It notes that industrial insertion robots commonly operate in constrained environments.Source: §I · p. 1
  • SHIELD combines design choices for DDPGfD with three industry-oriented criteria: efficient learning, economical deployment, and evaluation against industry solutions.The paper identifies human demonstrations as a natural task-specific prior and says the choices collectively improve speed and robustness to scene variability.Source: §I · p. 2
  • 99.8% success across 13K trials is reported as evidence of SHIELD’s reliability and robustness on the NIST assembly challenge.The paper also lists an open-source interaction dataset and two challenge tasks among its contributions.Source: §I · p. 2

II. PROBLEM STATEMENT

  • The NIST study covers three connector-insertion tasks, while the challenge tasks include HDMI insertion into a moving socket and blind key-lock insertion.The moving-target task is evaluated against human cycle time; the key-lock task requires search without visual input.Source: §II · pp. 2–3

III. RELATED WORK

Related work spans learning-based insertion and hybrid control, with reported limits in hardware realism, action scope, or reliability. The paper chooses DDPGfD and formulates manipulation as maximizing sparse task rewards under learned dynamics.

III. RELATED WORK

  • Prior work trained visual-force insertion on synthetic parts [3, 2], while a sim2real meta-learning approach used position-only actions in a 3mm workspace [6].The former demonstrated learning-based insertion without simulation; the latter adapted to real-world errors but used a height-based reward limited to the socket’s immediate neighborhood.Source: §III · p. 3
  • A guided-policy-search controller combining iLQG and operational-space force control generalized to unseen peg-hole configurations, but reliability on unseen displacements was limited [5].Source: §III · p. 3
  • A residual-RL approach completed displaced-block insertions where its scripted controller failed, but did not demonstrate high reliability or industrial-connector-level precision [9].Source: §III · p. 3
  • The authors select DDPGfD for its comparative simplicity while searching the design space for their setting.Source: §III · p. 3

IV. PRELIMINARIES

  • The tasks concern moving already-grasped objects to goal poses and use sparse rewards: terminal binary success without vision, or a visual reward classifier with vision.Source: §IV · p. 3
  • The formulation seeks task reward over a finite horizon with unknown dynamics; states and actions may also be subject to algebraic constraints.Source: §IV · p. 3

V. METHODS

The framework combines demonstrations with off-policy DDPGfD, then iteratively collects policy data and applies training modifications for industrial insertion tasks. Human operators can correct failing behavior so the off-policy learner can use those interventions in future policies.

V. METHODS

  • The workflow collects demonstrations, optionally pretrains visual features, gathers policy data, applies corrections or curricula, and trains actor and critic asynchronously.The framework describes these steps as its overall workflow.Source: §V · p. 3

A. Reinforcement learning from demonstration

  • DDPGfD uses human demonstrations to bootstrap a policy for sparse-reward, contact-rich manipulation without extensive reward shaping.Demonstrations provide candidate solutions where sparse rewards make exploration difficult.Source: A. Reinforcement learning from… · p. 3
  • A task and action-space curriculum gradually increases learning difficulty.The method includes this curriculum among its modifications to DDPGfD.Source: A. Reinforcement learning from… · pp. 3–4
  • Removing Gaussian action noise makes the policy deterministic; the authors report that learning stochasticity and demonstrations still provide sufficient exploration.They argue random noise can hinder navigation through the narrow passages found in insertion tasks.Source: A. Reinforcement learning from… · p. 4
  • The replay buffer keeps all experience and samples transitions uniformly, because prioritization schemes consistently hurt performance in this setting.Human actions are treated as the agent’s own experience.Source: A. Reinforcement learning from… · p. 4
  • On-policy correction lets an operator intervene when the agent fails, and the off-policy learner can incorporate those corrections into future policies.The authors report that around ten corrections were enough for the agent to succeed in the described task; Fig. 3 illustrates the intervention.Source: A. Reinforcement learning from… · p. 4, Figure 3 · p. 4

B. Relative Coordinates and Goal Randomization

The method expresses proprioceptive information relative to the robot’s reset pose, treating that pose as the task origin. This parameterization supports new-location generalization and fine-grained control of pose distributions.

B. Relative Coordinates and Goal Randomization

  • Using the reset pose as the task origin avoids relying on absolute robot-base coordinates, helping prevent overfitting to a particular position.Relevant proprioceptive information is expressed relative to that origin.Source: B. Relative Coordinates and Goal… · p. 4
  • The relative-coordinate parameterization makes moving the goal equivalent to physically moving it from the agent’s perspective, without extra mechanical apparatus.The robot itself is the only additional apparatus required.Source: B. Relative Coordinates and Goal… · p. 4
  • The parameterization permits fine-grained control over pose distributions, which the authors use for a curriculum.The curriculum is described in Section VI-B1a.Source: B. Relative Coordinates and Goal… · p. 4

C. Pre-trained Visual Features

The paper pretrains visual features with an unsupervised objective rather than learning image features from reward. It uses a variational autoencoder (VAE), with additional images collected to broaden visual coverage beyond near-optimal demonstrations.

C. Pre-trained Visual Features

  • The authors advocate unsupervised pretraining of visual features instead of training image features from reward; their implementation uses a variational autoencoder (VAE).They contrast this with a cited pixel-based insertion approach that took hours to reach 80% performance.Source: C. Pre-trained Visual Features · pp. 4–5
  • Additional camera images are collected by jogging the robot because demonstrations typically cover only states visited by a near-optimal policy.Broader image coverage matters because the human-labeled reward is regressed on the VAE features.Source: C. Pre-trained Visual Features · p. 5
  • The VAE objective combines input reconstruction with a penalty that keeps the encoded posterior close to a prior distribution.The inputs are camera images, and the prior is often chosen as a unit Gaussian.Source: C. Pre-trained Visual Features · p. 5
  • After VAE training, the decoder is discarded and the fixed encoder supplies latent codes to the agent.The encoder extracts those codes from images.Source: C. Pre-trained Visual Features · p. 5

D. Multi-modal policies

The policies receive different combinations of proprioceptive, force/torque, and visual inputs. The visual-feature variants differ in whether the VAE also serves as a reward detector.

D. Multi-modal policies

  • The modality comparison includes proprioception alone, proprioception plus wrist force/torque, and variants with pretrained visual features.Fig. 4 illustrates the different modality inputs to the policy.Source: D. Multi-modal policies · p. 5, Figure 4 · p. 5
  • In one visual variant, the VAE also acts as the reward detector, while that policy relies on vision and force/torque inputs.This distinguishes it from the other variant that makes pretrained visual features available to the agent.Source: D. Multi-modal policies · p. 5

VI. EXPERIMENTS

The experiments evaluate insertion policies across multiple input modalities and altered conditions, with vendor solutions serving as the industrial baseline.

VI. EXPERIMENTS

  • The experiments cover NIST board insertions, moving-target HDMI insertion, and key-lock insertion with long-range searching.The NIST tasks use a Kuka iiwa arm with Cartesian end-effector velocity control and a customized impedance controller.Source: §VI · p. 5, 1) Experiment setup · p. 5

1) Experiment setup:

  • The NIST setup tests proprioception alone, proprioception with wrist force/torque, and those signals with vision, including new locations, orientations, and manually moved boards.The moving-board condition was not part of training and tests generalization under changes within the training randomization range.Source: 1) Experiment setup · p. 5

2) Baselines:

  • Two independent vendors provide the NIST challenge baselines; pure RL methods without demonstrations were excluded because none solved the tasks.Vendors used off-the-shelf tools and additional engineering.Source: 2) Baselines · pp. 5–6
  • The vendor procedures use force-regulated spiral search or repeated insertion attempts over an expanding x-y perturbation grid.One procedure searches after force detection; the other repeats trials while increasing translation perturbations in 0.5 mm steps.Source: 2) Baselines · p. 6

3) Results:

The learned policies outperform vendor solutions in perturbation tolerance, achieve high overall reliability, and solve moving-board insertions under challenging conditions.

3) Results:

  • 99.8% success across 13,096 trials; most failures involved USB connectors with floating metal parts and changing contact dynamics.The authors plan to encourage gentler insertion policies to address these failures.Source: 3) Results · p. 6
  • SHIELD handles larger perturbation ranges than vendor solutions while maintaining nearly 100% success rates.Fig. 6 compares maximum perturbation for the method and vendors.Source: Figure 6 · p. 6, 3) Results · p. 6

4) Dynamic insertion:

  • The vision-based policy exceeded 95% success over 50 moving-board insertion trials, with failures linked to motion beyond controller frequency or training range.The moving-board task requires rapid reactions to environmental changes.Source: 4) Dynamic insertion · p. 6, Figure 7 · p. 7

B. Moving HDMI Insertion

  • The moving HDMI experiment evaluates cycle time at maximum speed to compare the approach’s reliability and speed with human hand-eye coordination.The target moves during insertion.Source: B. Moving HDMI Insertion · p. 7

1) Experiment Setup:

The paper evaluates a vision-guided policy on moving-target HDMI insertion, compares its speed and reliability with human performance, and tests long-range key-lock search.

1) Experiment Setup:

  • The HDMI socket moves in a randomized circular path, and the agent receives no socket pose or velocity, relying on local TCP velocity, wrist force/torque, and vision.The plug begins 15 cm from the mean socket pose.Source: 1) Experiment Setup · p. 7
  • Training progressively increases socket-position randomization to ±6 cm and speed to 2.5 cm/s, while the action space grows as policy reliability improves.The action-space curriculum starts with conservative velocities and expands exponentially to a maximum.Source: 1) Experiment Setup · p. 7

2) Agent Results:

  • The agent achieved 100% success over 1,637 episodes at 1,093 ms mean insertion time, versus humans’ 100% over 33 attempts at 1,041 ms.Fig. 8 compares insertion completion times; the authors characterize the agent’s reliability and insertion time as comparable to human hand-eye coordination.Source: 2) Agent Results · p. 7, Figure 8 · p. 7, 3) Human Results · p. 7

3) Human Results:

  • The human comparison involved two participants completing 33 attempts, with 100% insertion success and a 1,041 ms mean time.Participants started with eyes closed and inserted the plug as quickly as possible after the start signal.Source: 3) Human Results · p. 7

C. Key-lock insertion

  • The key-lock task targets small-clearance insertion and long-range searching without vision, which the authors say reduces system complexity relative to industry-standard search algorithms.The experiment uses a household metal lock and key rather than the industrial connectors used earlier.Source: C. Key-lock insertion · pp. 7–8

1) Experiment Setup:

The key-lock experiment tests whether a learned policy can search from hidden randomized offsets, with a curriculum supporting greater insertion tolerance.

1) Experiment Setup:

  • The key starts 5 cm from the lock at a randomized, unobserved x-y offset, requiring the policy to search for the opening.The agent receives pose relative to the episode start, so the offset is not observable.Source: 1) Experiment Setup · p. 8
  • Episodes end upon successful insertion, leaving the workspace, exceeding 15 seconds, or reaching a force/torque limit.The force/torque limit is intended to avoid damaging the key.Source: 1) Experiment Setup · p. 8
  • A starting-pose curriculum enabled insertion with up to 0.5 cm x-y offset; the agent failed at that offset without the curriculum.The randomization magnitude increases gradually from zero.Source: 1) Experiment Setup · p. 8

2) Results:

  • The RL agent extends the search range to 5 mm versus vendors’ 2 mm while learning insertion for small-clearance objects.Table II summarizes success rates at different maximum starting-pose offsets.Source: Table II · p. 8, 2) Results · p. 8

VII. OPEN-SOURCE DATASET

The paper releases data from its NIST board evaluations and reports robust benchmark and challenge-task results, while identifying evaluation scale and visual robustness as areas for further work.

VII. OPEN-SOURCE DATASET

  • The open-source dataset contains robot interaction data collected during NIST board evaluations.It covers around 50 hours across different tasks and input modalities, with reward labels included.Source: §VII · p. 8

VIII. DISCUSSION AND FUTURE WORK

  • 99.8% overall success across 13K trials on the NIST board; the authors also report robustness across varied testing scenarios.Source: §VIII · p. 8
  • On moving-target HDMI insertion, the agent achieved 100% success over 1637 trials and matched human cycle time; it also learned randomized key-lock insertion.Source: §VIII · p. 8
  • The NIST assembly challenge is presented as a general template for similar tasks, requiring few demonstrations and almost no parameter tuning.The authors report better handling of perturbations and generalization than traditional solutions in real-world trials.Source: §VIII · p. 8
  • The approach is proposed as a way for robots to enter unconstrained manufacturing environments and industries with low automation levels.Source: §VIII · p. 9
  • The authors call for larger-scale evaluation to verify a 99.999% success rate and report that vision robustness suffers under environmental changes and when task objects appear smaller.They note that the VAE reconstruction objective models the entire scene rather than task-relevant objects.Source: §VIII · p. 9
  • The authors identify offline reinforcement learning with online fine-tuning as a possible way to further shorten training time.Source: §VIII · p. 9
Loading 2103.11512v4…