Source-linked AI summary
Variable Compliance Control for Robotic Peg-in-Hole Assembly: A Deep Reinforcement Learning Approach
Cristian C. Beltran-Hernandez, Damien Petit, Ixchel G. Ramirez-Alpizar, Kensuke Harada
TL;DR
High-precision peg-in-hole assembly remains difficult for position-controlled industrial robots in uncertain environments, motivating a learning-based approach. The paper combines off-policy reinforcement learning with sim2real transfer and domain randomization to learn motion and force control for uncertain goal positions. Experiments on real contact-rich insertion tasks report high success rates under varying goal-position uncertainty, environmental stiffness, and novel tasks, while force-range selection remains partly manual.
Problem
Safely performing complex, high-precision peg-in-hole assembly in unstructured environments remains an open problem, especially for position-controlled industrial robots facing contact forces and uncertainty.
Method
The framework uses an off-policy reinforcement-learning policy to jointly control motion trajectories and variable compliance-controller force gains, accelerated with sim2real transfer and domain randomization.
Results
The learned policies achieved high success rates under varying goal-position uncertainty, environmental stiffness, and novel insertion tasks in extensive real-robot experiments.
Takeaways & Limitations
The framework supports learning high-dimensional contact-rich assembly policies on real position-controlled robotic systems across varied environments.
Takeaways & Limitations
The force-control parameter range remains a manually selected trade-off between adaptation to different environments and learning difficulty or generalization.
Abstract
from arXiv · showhide
Industrial robot manipulators are playing a more significant role in modern manufacturing industries. Though peg-in-hole assembly is a common industrial task which has been extensively researched, safely solving complex high precision assembly in an unstructured environment remains an open problem. Reinforcement Learning (RL) methods have been proven successful in solving manipulation tasks autonomously. However, RL is still not widely adopted on real robotic systems because working with real hardware entails additional challenges, especially when using position-controlled manipulators. The main contribution of this work is a learning-based method to solve peg-in-hole tasks with position uncertainty of the hole. We proposed the use of an off-policy model-free reinforcement learning method and bootstrap the training speed by using several transfer learning techniques (sim2real) and domain randomization. Our proposed learning framework for position-controlled robots was extensively evaluated on contact-rich insertion tasks on a variety of environments.
1. Introduction
High-precision peg-in-hole assembly remains difficult for position-controlled industrial robots because uncertain environments and contact forces challenge manually engineered controllers. This work introduces a learning-based framework that jointly learns motion and variable force control, accelerated through sim2real and domain randomization, and evaluates it on real contact-rich tasks.
- High-precision peg-in-hole assembly remains an open problem in unstructured environments despite its widespread industrial use.
- Position errors can generate unsafe assembly forces, while manually identifying model parameters and tuning controller gains is time-consuming and task-specific.These approaches are also described as insufficiently robust to uncertainty and environmental variation.
- Prior learning methods either discretized continuous robot actions or controlled motion trajectories while requiring manual force-gain tuning.The cited approaches therefore have limitations for continuous, high-dimensional robot control and environmental variation.
- The proposed framework learns motion trajectories and variable force-control gains for position-controlled robots facing uncertain goal positions.It is designed to learn contact-rich manipulation without requiring prior reference trajectories or force profiles, while still supporting them to accelerate learning.
- Sim2real transfer learning is used to improve training efficiency on real robots, while domain randomization improves policy generalization from simulation.The paper identifies their combination for contact-rich applications with position-controlled robots as a studied contribution.
- The method was extensively evaluated on a real robotic system across varied contact-rich peg-in-hole insertion tasks.
2.1. System Overview
The framework uses estimated target positions to learn an outer-loop policy that sets motion and compliance parameters for an industrial robot. An adaptive compliance controller executes these commands through a faster inner control loop.
- The learned policy uses the estimated target position to define motion trajectories and force-control parameters for an adaptive compliance controller.
- The outer reinforcement-learning loop runs at 20 Hz and provides subgoal positions plus compliance-controller parameters.Its slower frequency allows policy computation from the robot state before the next action.
- The inner loop is a parallel position-force adaptive compliance controller running at 500 Hz for contact-rich manipulation.The stated frequency is the maximum available on the Universal Robots e-series arms used in the system.
- Motion commands sent to the compliance controller specify the robot end-effector pose, consisting of position and orientation.The orientation is represented with unit quaternions, whose scalar and vector parts are defined in the method description.
2.2. Learning Adaptive-Compliance Control
The method models robotic assembly as a maximum-entropy reinforcement-learning problem and combines multimodal sensing with an adaptive parallel position-force controller. The policy predicts actions from proprioception and force/torque history, including motion subgoals and controller parameters.
- Reinforcement-learning algorithm: Soft actor critic (SAC) learns a maximum-entropy policy from replayed rollouts for sample-efficient robotic control.The implementation also uses distributed prioritized experience replay.
- Multimodal policy architecture: The policy encodes proprioception and force/torque feedback separately, concatenates their features, and predicts the next action.Proprioception produces a 32-dimensional feature vector, while the temporal convolutional network produces another 32-dimensional vector before concatenation.
- Adaptive-compliance control: The policy outputs position/orientation subgoals and parameters for a parallel position-force controller.The action is represented as a pair of position actions and controller-parameter actions.
- Adaptive-compliance control: The controller combines PD position control, PI force control, a selection matrix, and the policy’s position action.The selection matrix determines the degree of position and force control in each direction.
- Adaptive-compliance control: The controllable force-control parameters are bounded around baseline gains to reduce unstable behavior and system complexity.Each parameter is mapped from the policy-action range [-1, 1] into [Pbase − Prange, Pbase + Prange].
2.3. Task’s reward function
The reward function combines contact-force tracking with task progress and completion outcomes. It rewards reaching the goal with low contact force, faster completion, and collision avoidance.
- Reward design: The reward increases as the end-effector approaches the goal and the contact force decreases.The force term compares the external force with the desired force and normalizes it by the allowed maximum contact force.
- Task completion: Task completion requires the end-effector to be within 1 mm of the true goal position.An additional reward is provided for each unused time step relative to the episode limit.
- Safety constraint: Collisions receive a negative reward and terminate the episode early.The collision constraint is intended to encourage safer exploration during training.
2.4. Speeding Up Learning
Learning is accelerated by combining prior control knowledge with simulation-to-real transfer. Residual reinforcement learning supplies a manually defined controller, while domain randomization exposes the policy to varied simulated conditions before deployment.
- Simulation-to-real transfer: A physics simulator trains the insertion policy before transferring it directly to the real robot.The approach uses Gazebo 9 and applies domain randomization to reduce the reality gap.
- Residual reinforcement learning: Residual reinforcement learning combines a manually defined controller with policy actions to avoid learning the insertion task from scratch.The manually defined controller can use an estimated target position or reference trajectory, while the policy learns adjustments.
- Domain randomization: Domain randomization varies initial and goal positions, object-surface stiffness, and goal-pose uncertainty during simulation training.These variations target generalization across initial conditions and dynamic contact conditions.
3. Experiments and results
Experiments combined Gazebo simulation with real-robot validation on a UR3e platform. Training randomized task conditions, then retrained the simulated policy briefly on the real robot to reduce the simulation-to-reality gap.
- Experimental platform: The platform used a Universal Robot 3 e-series arm with a wrist force/torque sensor and Robotiq Hand-e gripper.The real robot supported control frequencies up to 500 Hz and was operated through ROS.
- Experimental platform: Real experiments used a 6-degree-of-freedom UR3e arm and a 3D-printed cuboid peg with 1.0 mm clearance.The peg and task-board hole had nonsmooth surfaces.
- Simulation training: Simulation training inserted a cuboid peg into a task-board hole for 500,000 time steps, taking about 5 hours on average.Training conditions were randomized after each episode, with goal positions selected from defined insertion planes.
- Simulation-to-real transfer: After simulation training, the policy was retrained on the real robot for 3% of the simulation time steps, requiring about 20 minutes.This refinement was intended to account for differences between simulated and real-world dynamics.
3.3. Evaluation
Evaluation tested the learned policy on real cuboid-peg insertion under true and uncertain goal poses. Sim2real with real-robot retraining generally performed best, while direct transfer could struggle with real-world dynamics.
- Cuboid-peg evaluation: The learned policy used relative end-effector position, contact force, and normalized Cartesian actions across search and insertion phases.The insertion direction was aligned with the robot’s y axis, and the policy adjusted force-control parameters after contact.
- Cuboid-peg evaluation: Three policies were tested 20 times from random initial positions with a true goal pose, and all achieved a very high success rate.The sim2real policy without retraining had difficulty with real-world physics, whereas sim2real plus retraining achieved the best overall performance time.
- Goal-pose uncertainty: Goal-pose uncertainty was evaluated using translation offsets of [-1, 1] millimeters and orientation offsets of [-5°, 5°] across all directions.Each condition used 20 trials from random initial positions, with results reported in Table 3.
- Goal-pose uncertainty: Sim2real with domain randomization and real-robot fine-tuning gave the best results across the tested goal-pose uncertainty conditions.When simulation and real-world dynamics differed substantially, learning from scratch could outperform direct policy transfer, especially at 5° orientation uncertainty.
Varying Environment Stiffness
The evaluation extended beyond the training setup to different stiffnesses and novel insertion tasks. The policy generalized well overall, but rough search surfaces and complex geometry limited performance in the hardest cases.
- Varying stiffness: Stiffness evaluation used high-, medium-, and low-stiffness environments, with results reported for 20 initial positions per condition.Medium stiffness used a rubber band, while low stiffness added a soft foam surface.
- Novel insertion tasks: Novel tasks included a high-stiffness metal ring, a high-force electric outlet, a delicate LAN port, and a USB connector.These tasks varied surface stiffness, required insertion force, and peg-shape complexity beyond the cuboid training task.
- Novel insertion tasks: The policy achieved a high success rate on novel and complex insertion tasks despite training mainly on a simpler cuboid-peg task.Evaluation used 20 executions from random initial positions with perfect goal-pose estimation.
- Failure boundary: Rough surfaces and crevices increased the chance of the peg becoming stuck during search, especially for the LAN port’s complex endpoint.A single stuck LAN-adapter corner prevented completion even when large force was applied.
- Insertion-plane generalization: Success rates on electric-outlet and LAN-port insertions were similar across different insertion planes.The policy was slightly better on the y-axis plane, where it had received real-robot retraining.
3.5. Ablation Studies
Ablation studies examined simulation transfer, policy architecture, and input choices. Simulation pretraining greatly reduced the real-robot training needed for successful behavior, while prior-action and desired-force inputs supported faster learning.
- Learning from scratch vs sim2real: Learning from a pretrained simulation policy reached comparable performance in under 5,000 real-robot steps, versus at least 50,000 steps when learning from scratch.The pretrained policy still required real-robot fine-tuning because simulated dynamics did not fully match reality.
- Policy architecture: The policy architecture was compared with a two-layer fully connected neural network over 70,000 simulation training steps.The supplied passage identifies the comparison but does not report the numerical outcome of the learning curves.
- Policy inputs: Removing prior action at−1 produced poor performance, indicating that prior-action information was critical for faster convergence.The comparison used random initial positions and random desired insertion forces in simulation.
- Policy inputs: Including desired insertion force Fg enabled higher cumulative rewards and faster learning.This input comparison was performed on the simulated cuboid-peg insertion task.
4. Discussion
The learned framework achieved high insertion success under a perfect goal-position estimate and across uncertainty in goal position, environmental stiffness, and novel insertion tasks. Its force-control parameter range remains a manual design choice that trades learning difficulty against generalization.
- Under a perfect goal-position estimate, the policy correctly learned the nominal trajectory and force-control parameters needed for successful insertion.
- Figure 12 compares learning curves for policies with different inputs on cuboid-peg insertion with random initial positions and desired insertion force.
- High success rates were achieved under varying goal-position uncertainty, environmental stiffness, and novel insertion tasks.The policy learned both the nominal trajectory and appropriate force-control parameters for insertion.
- The framework can be adapted to other RL algorithms, although this study used SAC for its sample efficiency as an off-policy method.The authors identify comparing alternative learning algorithms as future work.
- The manually selected force-control parameter range trades adaptability across environments against learning difficulty and training speed.Wide ranges may generalize better but make learning harder, whereas narrow ranges may learn faster but generalize less well.