Source-linked AI summary
Deep Reinforcement Learning for Industrial Insertion Tasks with Visual Inputs and Natural Rewards
Gerrit Schoettler, Ashvin Nair, Jianlan Luo, Shikhar Bahl, Juan Aparicio Ojea, Eugen Solowjow, Sergey Levine
TL;DR
Industrial insertion is difficult to control because contact dynamics are hard to model, while real-world RL lacks perfect state information and dense rewards and must remain sample-efficient. The paper combines image-based or sparse natural rewards with prior control information, evaluating the approach on tight-tolerance connector tasks. It reports successful learning across three tasks, including sparse electrical rewards and goal-image rewards, while also studying robustness to noisy goal locations.
Problem
Industrial insertion involves complex contact dynamics, while practical real-world RL must address limited sample efficiency, safety, imperfect state information, and unavailable dense rewards.
Method
The paper learns image-based policies from goal images or sparse outcome rewards, incorporating prior information such as human-engineered controllers to support real-world learning.
Results
The experiments successfully learn tight-tolerance insertion policies across three connectors using convenient image-based or sparse rewards, with residual RL producing consistent vision-based insertion.
Takeaways & Limitations
Convenient goal-image and outcome-based rewards can support real-world learning of complex assembly tasks, including under noisy goal specifications and connector-specific challenges.
Takeaways & Limitations
Extending the approach to more complex multi-stage assembly remains challenging because visual backgrounds become more complex and policies must adapt to earlier mistakes.
Abstract
from arXiv · showhide
Connector insertion and many other tasks commonly found in modern manufacturing settings involve complex contact dynamics and friction. Since it is difficult to capture related physical effects with first-order modeling, traditional control methods often result in brittle and inaccurate controllers, which have to be manually tuned. Reinforcement learning (RL) methods have been demonstrated to be capable of learning controllers in such environments from autonomous interaction with the environment, but running RL algorithms in the real world poses sample efficiency and safety challenges. Moreover, in practical real-world settings we cannot assume access to perfect state information or dense reward signals. In this paper, we consider a variety of difficult industrial insertion tasks with visual inputs and different natural reward specifications, namely sparse rewards and goal images. We show that methods that combine RL with prior information, such as classical controllers or demonstrations, can solve these tasks from a reasonable amount of real-world interaction.
1 Introduction
The paper targets industrial tasks where complex contacts challenge conventional control, while real-world deep RL faces sample-efficiency and reward-specification obstacles. It proposes learning from convenient visual or sparse rewards and shows successful real-world insertion across multiple tasks.
- Industrial tasks requiring adaptability and tight visual perception-control loops often exceed conventional positioning controllers, especially under complex contact dynamics.
- Deep RL deployment remains difficult because real-world interaction must limit time and robot wear, while dense reward shaping requires additional perception engineering.
- Goal images can provide both state representation and goal specification, enabling direct visual feedback with minimal manual goal-definition effort.
- Sparse rewards can be obtained from human labels or simple instrumentation, but task-specific prior information is needed to address difficult exploration.
- Across three tasks, the experiments successfully learn tight-tolerance assembly from image-based RL using convenient sparse or goal-image rewards without extra engineering.
2 Related Work
Prior robotics work incorporates task information through reward shaping, planning, demonstrations, or richer sensing, but these approaches can demand difficult supervision, geometry, or high-dimensional measurements. The paper situates its approach among efforts to make precise assembly learning more practical.
- Reinforcement learning has been applied across robotics domains including driving, locomotion, stacking, grasping, navigation, and balancing.
- Reward shaping becomes increasingly difficult as task complexity grows, while trajectory planners require object and geometry information that can be hard to provide.
- Demonstration-based methods can initialize policies, infer rewards, or improve learning, but multiple demonstrations are difficult to collect for assembly tasks.
- High-precision assembly methods often use geometry, forces, joint states, and velocities that increase experimental complexity and supervision requirements.
3 Electric Connector Plug Insertion Tasks
The study benchmarks three connector insertion tasks with increasing mechanical and alignment difficulty, then evaluates visual, sparse, and shaped-reward settings. The visual setting uses image-only state and goal specification, while the sparse setting uses electrical connectivity for USB insertion.
- Electric Connector Plug Insertion Tasks: The benchmark includes USB, D-Sub, and waterproof Model-E connectors, each presenting different precision and force requirements.
- Control: The robot executes actions through a cascade controller: actions run at up to 10 Hz, inverse kinematics produces joint angles, and impedance-PID torque control runs at 1000 Hz.
- Adapters: Insertion difficulty increases from USB to D-Sub to Model-E, with tighter fits, more alignment constraints, edges, and grooves requiring greater downward force.
- Visual: In the visual setting, the policy receives a 32 × 32 grayscale image and uses pixelwise L1 distance to a goal image as reward, without state estimation.
- Sparse: The sparse setting rewards detection of an active electrical connection and supplies Cartesian end-effector coordinates plus vertical force as state, evaluated only on USB.
- Shaped reward: The shaped-reward baseline combines L1 and L2 position errors with a force term whose sign changes after insertion, using calibrated force measurements.
4 Methods
The paper evaluates off-policy RL algorithms for connector insertion and combines them with prior controllers or demonstrations to improve sample efficiency and exploration safety. Residual RL adds a learned policy to a fixed controller, while demonstration-guided learning uses behavior cloning during TD3 training.
- The study evaluates multiple reinforcement learning algorithms for connector insertion tasks.
- Off-policy RL methods use replayed transitions to support sample-efficient real-world learning.The paper considers TD3 and SAC, which are described as sample efficient, stable, and requiring little hyperparameter tuning.
- TD3 reduces value overestimation with two Q-function approximators and stabilizes training through delayed policy updates.
- SAC is an off-policy value-based method using maximum-entropy reinforcement learning with a stochastic policy.
- Residual RL combines a fixed policy with a learned parametric policy, using SAC or TD3 to learn the residual parameters.The experiments use a Cartesian-space P-controller as the fixed policy; it centers the end-effector and reaches the goal in about 10 time steps from a reset 5 cm above it.
- Demonstration-guided RL adds a behavior-cloning loss from joystick demonstrations while training with TD3.
5 Experiments
The experiments evaluate residual RL with goal-image and binary sparse rewards on real connector assembly tasks, alongside oracle state-based and robustness comparisons. They also compare SAC and TD3 exploration, including their residual-RL variants.
- The study evaluates residual RL with natural rewards on real connector assembly tasks, measuring task success and final distance to the goal.
- Vision-based Learning: Vision-based learning uses raw 32 × 32 grayscale images and the ℓ1 distance between current and goal images as the reward.
- Learning from Sparse Rewards: Sparse-reward learning uses a binary electrical-connection signal and is evaluated only on the USB connector.This setting targets electronic manufacturing tasks where connector functionality can be directly measured.
- Perfect State Information: Full-state experiments use dense distance-and-force rewards as an oracle for comparing the difficulty of sparse and image rewards.
- Robustness: Robustness tests introduce small goal perturbations to assess tolerance to measurement, calibration, and object-placement errors.
- Exploration Comparison: SAC explores with an adaptive stochastic policy, whereas TD3 uses a deterministic policy plus Gaussian training noise; both are compared with residual RL.
6 Results
Across visual, sparse-reward, state-based, and robustness experiments, residual RL generally produced stronger, faster, or more robust connector insertion policies than standard RL and alternative prior-informed methods. It also handled perturbed goals and, with SAC, performed particularly well across connectors.
- 6.1 Vision-based Learning: Residual RL learned a successful and consistent vision-based insertion policy from relatively few samples.
- 6.5 Exploration Comparison: With SAC, residual RL tended to perform best across all three connectors, while TD3 was comparably effective but often substantially less robust.
- 6.1 Vision-based Learning: Image-distance rewards provided sufficient stage information for vision-based insertion, while residual RL prevented reward-hacking behaviors through stabilizing horizontal control.
- 6.2 Learning From Sparse Rewards: All methods achieved very high USB insertion success with sparse electrical-connection rewards, showing precise insertion can be learned without dense shaping.
- 6.3 Perfect State Information: Residual RL outperformed standard RL with perfect state information and dense rewards, although standard RL made more initial progress with the better-shaped reward.
- 6.2 Learning From Sparse Rewards: Residual RL and RL with demonstrations solved sparse-reward USB insertion relatively quickly, whereas RL alone took about twice as long at comparable performance.
- 6.4 Robustness: Residual RL remained robust to ±1 mm goal perturbations, overcoming the deterministic controller on difficult connectors and succeeding on Model-E in 16/25 trials.
7 Conclusion
The paper demonstrates deep RL for tight-tolerance industrial assembly using raw images and convenient reward specifications. It identifies broader automation potential while noting that more complex, multi-stage environments remain challenging.
- Deep RL can solve complex industrial assembly tasks with tight tolerances using raw image observations and binary or goal-image rewards.
- Easy-to-specify reward functions give RL the potential to automate a wide range of assembly tasks with flexible robotic manipulators.
- Extending the approach to multi-stage assembly remains challenging because visually complex backgrounds and adaptation to prior mistakes complicate goal-based policies.