Source-linked AI summary
Deep Reinforcement Learning for High Precision Assembly Tasks
Tadanobu Inoue, Giovanni De Magistris, Asim Munawar, Tsuyoshi Yokoya, Ryuki Tachibana
TL;DR
High-precision assembly can require accuracy beyond robot precision, while conventional programming and simulation involve substantial setup and tuning. The paper trains an LSTM with reinforcement learning from robot sensor feedback for tight-clearance peg-in-hole assembly. In 100 trials across four cases, the learned approach achieved a 100% success rate and robustness to position and angle errors.
Problem
Conventional programming and simulation are tedious or insufficient when assembly precision exceeds robot accuracy and environmental variations must be handled.
Method
The approach trains an LSTM with reinforcement learning using robot sensor feedback, position-error modeling, and hybrid position/force control for peg-in-hole fitting.
Results
100% success rate was achieved in 100 trials for all four tested cases, with robustness to position and angle errors.
Takeaways & Limitations
The proposed teachless approach provides precise peg-in-hole fitting while reducing manual setup effort and handling position and angle errors.
Takeaways & Limitations
The high-precision fitting task is learned separately for each configuration; broader generalization to materials, manipulators, angles, and shapes is future work.
Abstract
from arXiv · showhide
High precision assembly of mechanical parts requires accuracy exceeding the robot precision. Conventional part mating methods used in the current manufacturing requires tedious tuning of numerous parameters before deployment. We show how the robot can successfully perform a tight clearance peg-in-hole task through training a recurrent neural network with reinforcement learning. In addition to saving the manual effort, the proposed technique also shows robustness against position and angle errors for the peg-in-hole task. The neural network learns to take the optimal action by observing the robot sensors to estimate the system state. The advantages of our proposed method is validated experimentally on a 7-axis articulated robot arm.
I. INTRODUCTION
Conventional programming and simulation are time-consuming and insufficient for assembly requiring precision beyond robot accuracy. The paper proposes reinforcement-learning-based skill acquisition using sensor feedback and an LSTM for tight-clearance peg-in-hole assembly.
- Motivation: Teach-pendant programming is tedious, time-consuming, and requires parameter tuning when deploying robots to new factory lines.Environment variations make setup difficult even after initial programming.
- Motivation: Offline programming reduces robot downtime but requires simulation development and testing, while imperfect models cannot fully represent environmental variation.The paper identifies precision machining and flexible material handling as applications where this method is insufficient.
- Approach: The proposed skill-acquisition approach compensates for low programming accuracy through learning without parameter tuning and explicit teaching.The robot learns a high-precision fitting task from sensor feedback.
- Related work: Prior reinforcement-learning studies addressed robotic grasping with cameras and encoders, but the paper states they cannot be directly applied to high-precision industrial applications.Figure 1 presents the abstract concept of robots learning new skills through deep reinforcement learning.
- Approach: The study applies reinforcement learning with an LSTM to tight-clearance cylindrical peg-in-hole assembly using common force and position sensors.The task includes tilted holes and avoids relying on super-precise force-torque sensors or cameras.
II. PROBLEM FORMULATION
The paper formulates high-precision cylindrical peg-in-hole assembly as two separately learned phases: search and insertion.
- Task decomposition: The target is a high-precision cylindrical peg-in-hole task for force-controlled robotic assembly.The task is divided into search and insertion phases.
- Search: During search, the robot places the peg center within the hole center’s clearance region.
- Insertion: During insertion, the robot adjusts peg orientation relative to the hole and pushes the peg to the desired position.The paper studies and learns search and insertion separately.
A. Search Phase
The search phase addresses localization under limited robot and sensor accuracy, using force-torque sensing to estimate the hole location relative to the peg.
- Search challenge: Position control makes setting the peg and hole within a few tens of µm difficult despite industrial robot accuracy.
- Sensing: The method uses a common 6-axis force-torque sensor to learn hole location relative to peg position.Visual servoing is described as impractical because of camera resolution limits and occlusion during assembly.
- Prior approaches: Earlier approaches interpret peg position from sensor moments or depth profiles, but the paper notes difficulty generalizing them to real-world physical interactions.
B. Insertion Phase
The insertion phase builds on prior force-, associative-learning-, and neural-network-based approaches while targeting a much tighter clearance that is difficult for both humans and robots.
- Prior approaches: Prior insertion research includes associative reinforcement learning, neural-network control, tilted-mode recovery, and force-moment alignment.
- Precision challenge: The paper targets insertion with 10 µm clearance, which is difficult because the peg can become stuck at the initial insertion stage.The paper states that clearance tighter than robot position accuracy requires the robot to learn the task.
III. REINFORCEMENT LEARNING WITH LONG SHORT TERM MEMORY
The method trains recurrent Q-learning networks to select force and rotation actions from sensor-derived states for separate search and insertion skills. Position rounding, experience replay, and LSTM memory support learning under uncertain positions and delayed observations.
- The RL agent observes a sensor-derived system state and learns a policy that selects the best action at each state.The state includes average force and moment measurements, while actions are sent to a hybrid position/force controller.
- The peg position is computed from encoder-based forward kinematics, and rounded position values provide auxiliary information for learning under hole-position errors.The hole center may vary within −c < x < c and −c < y < c; rounding is intended to accelerate convergence.
- The reward favors completing an episode in few steps and assigns penalties based on final distance or insertion depth when the task is incomplete.The reward is bounded by −1 ≤ r < 1; failure beyond the safe search boundary or a peg stuck at the hole entry receives −1.
- Q-learning uses ε-greedy exploration, a deep recurrent Q-function approximation, and replay-memory sampling to update network parameters.The action thread stores transitions while the learning thread samples minibatches from a FIFO memory pool for training.
- Search and insertion are treated as distinct skills, with two LSTM-based neural networks trained sequentially for the two phases.Both networks use two LSTM layers, and insertion training starts after the search skill is in place.
- LSTM layers use sensed-data history to compensate for the two-cycle delay between issuing an action and observing its environmental effect.The delayed response occurs because the robot controller executes commands at a later cycle, while contact states are not clearly identified.
IV. EXPERIMENTS
The experiments use a 7-axis robot platform with force-torque sensing and separately trained recurrent networks for search and insertion. The setup includes adjustable hole orientation and staged reinforcement-learning parameters.
- Platform: The experimental platform uses a 7-axis articulated robot arm, a 6-axis force-torque sensor, and a gripper for cylindrical pegs.The sensor is rated for 200 N force and 4 N m moment, with 0.024 N force resolution.
- Platform: The peg-and-hole apparatus includes different component sizes, while the robot arm accuracy is ± 60 µm.A 1D goniometer stage adjusts the hole plate angle relative to the ground.
- Experimental architecture: The external computer controls the robot over UDP, averages sensor data, and deploys the learned model on a Raspberry Pi 3.Sensors are sampled every 2 ms, while the computer polls every 40 ms and averages 20 data points.
- Learning setup: Search and insertion are treated as distinct skills, with two networks using two LSTM layers of sizes h1 = 20 and h2 = 15.The insertion network is trained after the search skill has been learned.
- Learning setup: Training uses replay memory of 20,000 steps, 230 episodes, and maximum episode lengths of 100 search steps and 300 insertion steps.Learning begins after 10 episodes with batches of 64 randomly selected experiences.
- Learning setup: Exploration starts at 1.0 and decreases by 0.005 per episode until reaching 0.1, transitioning from random action selection toward exploitation.The exploration schedule is applied during network training.
A. Search Phase
The search phase uses force and moment information to locate the hole before insertion. Experiments select a downward force and train the network under randomized position errors, with learning improving search performance.
- Force selection: The search phase uses moment measurements from preliminary experiments to estimate the hole direction from peg position and downward force.The experiments evaluate three peg locations and downward-force conditions after calibrating the force-torque sensor.
- Force selection: 20 N downward force is selected for subsequent search experiments because it provides detectable moments while reducing wear relative to stronger force.At 10 N, the experiments could not obtain a detectable moment; 20 N and 30 N supported hole-direction estimation.
- Robustness training: Position robustness is trained by adding randomly selected errors in one of 16 directions to the peg or hole position.The procedure addresses sensor error and manually introduced hole-position error.
- Robustness training: Staged training begins with a 1 mm initial peg offset, then increases it to 3 mm while reusing prior knowledge and reducing the later-stage exploration rate to 0.5.The staged procedure is applied to search-phase learning.
- Search control: The search network receives a 7-dimensional state vector and selects among four elementary-action combinations.The position grid uses c = 3 mm for d0 = 1 mm and c = 5 mm for d0 = 3 mm.
- Search control: Continuous force-controlled motion maintains peg contact with the hole plate and can avoid static friction compared with stepwise movements.The peg remains under a constant downward force during motion.
- Learning outcome: With 10 µm clearance, 0° tilt, and 1 mm initial offset, learning converges and the steps needed to complete search decrease significantly.Figure 9 reports means and 90% confidence bounds using a moving window of 20 episodes.
B. Insertion Phase
The insertion phase is learned as a separate skill after successful search. It uses force and moment sensing to select insertion actions and detects completion from vertical peg displacement.
- Skill structure: Insertion is trained with a separate network after the search network has been trained successfully.Successful searching is a prerequisite for beginning insertion-phase training.
- State and actions: The insertion state uses moment measurements to sense peg orientation and vertical force to indicate whether the peg is stuck.These sensor signals are part of the insertion network’s state representation.
- State and actions: The insertion controller selects from five actions defined for the insertion phase.The actions are based on the insertion network’s 7-dimensional state representation.
- Completion criterion: Insertion completion is detected when the peg’s downward displacement exceeds the 19 mm stroke threshold Z.The reward for a successful insertion episode is similar to the search-phase reward.
C. Results
Experiments evaluated the learned peg-in-hole skill across different clearances, offsets, and tilt angles. The proposed approach achieved 100% success in all tested cases and acquired robustness against position and angle errors.
- The experiments tested different peg clearances and tilted hole plates to assess robustness.A 1D goniometer stage was used to tilt the hole plate.
- The 3 mm-offset, 10 µm-clearance, 0° case had a wider, more right-shifted execution-time distribution than the 1 mm-offset, 20 µm-clearance, 1.6° case.The insertion time becomes longer with larger tilt because the peg must align with the hole.
- 100% success rate was achieved in all four peg-in-hole cases over 100 trials.The cases varied initial offset, clearance, and hole tilt.
- Average execution times were summarized for four cases and compared with a conventional fixed-search-pattern approach using force-sensing control.The conventional approach allows maximum initial position and angle errors of 1 mm and 1° respectively.
- The results show that the learning technique acquired fitting skills robust to position and angle errors.
- Execution-time histograms compared search, insertion, and total time for a 10 µm-clearance case and a 20 µm-clearance, 1.6°-tilted case.Case A used a 3 mm initial offset; Case B used a 1 mm initial offset.
V. CONCLUSIONS AND FUTURE WORK
The paper presents a teachless approach for precise peg-in-hole fitting and validates it on a 7-axis articulated robot arm. Future work targets broader generalization and cloud-based skill delivery, while the current approach uses discrete actions.
- The teachless approach was validated for precise peg-in-hole tasks using a 7-axis articulated robot arm.Results showed robustness against position and angle errors for a fitting task.
- The task is currently learned online for each configuration, limiting immediate generalization across materials, manipulators, angles, and shapes.Future work proposes learning a more general model from trial data gathered across multiple robots.
- The proposed peg-in-hole controller uses a discrete number of actions, motivating comparison with continuous-space methods such as A3C and DDPG.