Source-linked AI summary
A Practical Approach to Insertion with Variable Socket Position Using Deep Reinforcement Learning
Mel Vecerik, Oleg Sushkov, David Barker, Thomas Rothörl, Todd Hester, Jon Scholz
TL;DR
Insertion methods can require engineered models or high-level object features, while real tasks also involve uncertain socket positions and deformable objects. EDRIAD combines deep reinforcement learning with human demonstrations and visual inputs. The paper reports reliable real-robot insertion under socket-position variation, with the most challenging task solved in under 20 minutes of interaction time.
Problem
Insertion remains difficult under socket-pose uncertainty and deformability, while existing approaches can assume fixed poses or require engineered features and behaviors.
Source: §I · p. 1Method
EDRIAD combines visual rewards and policy inputs with demonstration-based actor and critic losses and a distributional critic loss.
Source: §I · pp. 1–2Results
Takeaways & Limitations
The real-robot policies reliably solve the evaluated tasks and are robust to variation in socket pose.
Source: §VI · p. 8EDRIAD requires a carefully engineered compliance controller and considerable hyper-parameter tuning, and assumes the plug is already firmly grasped.
Source: §VI · p. 8
Abstract
from arXiv · showhide
Insertion is a challenging haptic and visual control problem with significant practical value for manufacturing. Existing approaches in the model-based robotics community can be highly effective when task geometry is known, but are complex and cumbersome to implement, and must be tailored to each individual problem by a qualified engineer. Within the learning community there is a long history of insertion research, but existing approaches are typically either too sample-inefficient to run on real robots, or assume access to high-level object features, e.g. socket pose. In this paper we show that relatively minor modifications to an off-the-shelf Deep-RL algorithm (DDPG), combined with a small number of human demonstrations, allows the robot to quickly learn to solve these tasks efficiently and robustly. Our approach requires no modeling or simulation, no parameterized search or alignment behaviors, no vision system aside from raw images, and no reward shaping. We evaluate our approach on a narrow-clearance peg-insertion task and a deformable clip-insertion task, both of which include variability in the socket position. Our results show that these tasks can be solved reliably on the real robot in less than 10 minutes of interaction time, and that the resulting policies are robust to variance in the socket position and orientation.
I. INTRODUCTION
EDRIAD targets insertion under socket and dynamics uncertainty, combining deep reinforcement learning with demonstrations for contact-rich tasks. It is evaluated on peg and deformable-clip insertion with variable socket positions.
I. INTRODUCTION
- Existing insertion work often assumes a fixed socket pose, while real tasks add position uncertainty, deformable objects, and demands for end-user configurability.The paper identifies modeling, simulation, object tracking, reward shaping, and manual control-parameter tuning as requirements an ideal approach would avoid.Source: §I · p. 1
- EDRIAD is presented as a way to teach insertion with a small number of human demonstrations and quickly improve performance.The stated long-term vision is for end-users to teach complex insertion tasks in natural settings with only a few examples.Source: §I · p. 1
- EDRIAD combines visual rewards and policy inputs with demonstration-based actor and critic losses and a distributional critic loss.The critic losses classify positive versus negative demonstration transitions and predict episode progress.Source: §I · pp. 1–2
- Roughly 60x less training time than [24], with the most challenging task solved in under 20 minutes of robot interaction time.The real-robot evaluation covers two insertion tasks with randomized target positions.Source: §I · p. 2
II. TASK DESCRIPTION
- The last-inch task assumes the end effector is near the target and defines state and goal or reward entirely from raw robot sensors.Episodes begin with the plug in hand roughly 5 cm from the socket; socket starting pose varies by 2–4 cm.Source: §II · p. 2
- The evaluation contrasts a 0.5 mm-clearance round peg insertion requiring force-sensitive probing and visual alignment with a deformable two-prong clip task requiring prying and visual alignment.For the clip, multiple object-part poses cannot be inferred deterministically from the gripper pose.Source: Figure 1 · p. 1, Figure 2 · p. 1, §II · p. 2
III. RELATED WORK
Prior insertion methods include phase-based control, learned compliance, simulation transfer, and demonstration-based reinforcement learning. EDRIAD is positioned against their engineering, feature, and robustness constraints.
III. RELATED WORK
- Phase-based control decomposes insertion into approach, search, align, and insert, using state machines and Cartesian impedance controllers.Search and alignment must handle socket-pose uncertainty; narrow-clearance insertion can bind, motivating wiggling and recovery behaviors.Source: §III · p. 2
- Earlier learned compliance work used continuous neural-network actions and haptic feedback, while this approach adds deep networks and a value function to accelerate training.Its joint-velocity actions avoid writing a Cartesian admittance controller and allow compliance to contact anywhere on the arm.Source: §III · p. 3
- Sim2Real insertion-related manipulation requires simulator engineering, and the authors report difficulty producing stable simulations of grasp dynamics and force feedback.The cited folding work also used fake anchors between cloth and gripper to sidestep grasping.Source: §III · p. 3
- GPS can train raw-pixel insertion policies from trajectory-optimized policies, but requires low-dimensional features and a smooth optimization cost.Another cited method handling socket-position uncertainty uses hand-defined discrete Cartesian actions, lacks vision, and computes reward from robot kinematics, restricting deformable-object use.Source: §III · p. 3
- DDPGfD applies demonstrations to continuous actions, but the paper distinguishes EDRIAD by pretraining, stochastic-environment robustness, and sample efficiency.The cited prior method does not include pretraining and is described as not robust to environment stochasticity or as sample-efficient.Source: §III · p. 3
IV. APPROACH
The approach modifies DDPG with demonstration-based losses and image-derived rewards and features. Its visual pipeline reuses insertion-detector features for control while addressing sensitivity to visual noise and false-positive rewards.
IV. APPROACH
- DDPG retains its off-policy actor–critic mechanism while adding image-based rewards, positive and negative demonstrations, actor behavior cloning, critic demonstration and progress losses, and a distributional critic loss.These modifications target the task setting and sample efficiency.Source: §IV · p. 3
A. Visual Features and Rewards
- A convolutional network detects insertion from in-hand camera images, avoiding the need for dedicated socket and plug-deformation tracking.The detector’s features are also reused for control; the authors report accurate detectors from a few minutes of teleoperation images.Source: A. Visual Features and Rewards · p. 3
- Reused classification features were sensitive to visual noise and behavior variation, so the authors regularized their magnitudes, covariance, and invariance to image noise and color or brightness changes.The feature-distribution regularizers use batch statistics and are described as analogous to a KL-divergence from a standard Normal distribution.Source: A. Visual Features and Rewards · pp. 3–4
- Positive reward requires both of two insertion detectors to exceed 88% probability, which the authors report prevented reward delusions and reduced false positives.Only one detector’s features are used as policy observations.Source: A. Visual Features and Rewards · p. 4
- The reward detector and visual features are trained before and held fixed during reinforcement learning, with computed features and rewards stored in replay.The authors state this avoids repeating full-batch forward and backward passes during learning updates.Source: A. Visual Features and Rewards · p. 4
B. Collecting Demonstrations
Human demonstrations address sparse-reward exploration by providing examples of successful insertion and of failure modes. The authors collect them using a 6D mouse and report that teaching takes less than 30 minutes.
B. Collecting Demonstrations
- Sparse goal-classification rewards provide no strong exploration cue, making socket discovery difficult for the agent.The detector’s visual features and rewards do not encode a smooth goal-distance representation.Source: B. Collecting Demonstrations · p. 4
- The authors collect 20–30 successful and an equal number of negative trajectories, including probes of failure modes such as inserting only one clip-prong.A 6D mouse supplies Cartesian commands, translated into joint actions using the arm’s known kinematics.Source: B. Collecting Demonstrations · p. 4
- Teaching the robot with these demonstrations takes less than 30 minutes of wall-clock time.Source: B. Collecting Demonstrations · p. 4
C. Demonstration Critic Losses
The critic extracts additional information from demonstrations through transition classification and episode-progress prediction. Its Q-function uses a distributional loss over projected one-step returns.
C. Demonstration Critic Losses
- The critic’s demonstration-classification loss trains it to distinguish transitions from positive versus negative demonstrations.The classification head predicts demonstration-label probabilities for expert transitions.Source: C. Demonstration Critic Losses · p. 4
- The critic’s episode-progress loss predicts the fraction of a successful demonstration episode completed at each transition, using an L2 error loss.The progress fraction is assigned to every step in the successful demonstration.Source: C. Demonstration Critic Losses · p. 4
D. Distributional Q-function
- The distributional critic predicts Q-values over 60 evenly spaced bins from 0 to 1 and minimizes KL-divergence to the projected one-step return distribution.The projection is onto the current bins.Source: D. Distributional Q-function · pp. 4–5
E. Behaviour cloning
The actor imitates actions from positive expert demonstrations, especially during pretraining and early learning. The combined loss schedule decays demonstration-based losses over time.
E. Behaviour cloning
- The actor’s behavior-cloning loss imitates actions from positive expert demonstrations and is especially important during pretraining and early learning.Source: E. Behaviour cloning · p. 5
F. Summary of losses
- The actor and critic losses follow schedules that decay the effect of demonstration-based losses over time.The authors used λe and λAC values of 500.Source: F. Summary of losses · p. 5
G. DDPG Details
The setup specifies the agent’s inputs, learning procedure, and robot-control conditions, then evaluates performance across randomized insertion tasks using repeated trials and defined metrics.
G. DDPG Details
- The agent combines joint positions, velocities, and torques, end-effector position, and visual features into a scaled 32-dimensional observation vector.Position inputs are offset to have zero mean, and input types are scaled to similar magnitudes.Source: G. DDPG Details · p. 5
- Demonstrations are loaded into a uniform-sampling replay buffer before interaction, and stored samples are never discarded.The agent uses one-step transitions and demonstration pretraining.Source: G. DDPG Details · p. 5
- The actor’s critic-gradient contribution is scaled down early in training and increased over time because the initial critic gradient can harm the policy.Supervised losses may initially suffice to train a successful policy, before the critic has a useful action gradient.Source: G. DDPG Details · p. 5
V. EXPERIMENTS
- The experiments ask how visual features, behavior cloning, auxiliary losses, and pretraining affect performance.Source: §V · p. 5
A. Experimental setup
- Socket positions are randomized between episodes, with full randomization corresponding to 2–4 cm of opening movement; each experiment runs for 6000 steps.A full run represents about 20 minutes of environment interaction, while episodes last at most 40 steps (8 seconds).Source: A. Experimental setup · p. 5
- The agent uses joint-velocity commands limited to ±0.1 rad/s, which pass through a 100 Hz admittance layer.The velocity command combines the agent’s requested velocity with a compliance-scaled, gravity-compensated joint torque.Source: A. Experimental setup · p. 5
- Hyperparameters are tuned in MuJoCo across eight runs, while task performance is reported using success rate and per-step reward across repeated real-robot trials.Success rate is the ratio of successful episodes over the last 500 steps; experiments are repeated four times for peg insertion and six times for clip insertion.Source: A. Experimental setup · p. 5
B. Visual Features
EDRIAD learns both insertion tasks with randomized socket positions, but visual features are needed for that variability. Longer clip-task training continues to improve success without instability.
B. Visual Features
- 97% peg and 77% clip are the average final success rates on randomized tasks; the clip is harder because its flexibility makes the environment less observable.Source: B. Visual Features · p. 5, Figure 4 · p. 6
- Visual features are necessary for solving randomized socket positions, while fixed-position tasks can be solved with or without them.For both tasks, the best performance occurs with a fixed socket and vision; those tasks are learned within about 2000 steps.Source: B. Visual Features · pp. 5–6
- Without vision, agents use probing to search for openings; clip policies try one prong and then the other, producing slower and less reliable behavior than vision-based policies.Source: B. Visual Features · p. 6
C. Learning stability
- At 30000 steps, clip-task success reaches 98%, continuing to improve beyond 6000 steps without deterioration or instability.The longer run took about 8 hours of wall-clock time, although hyperparameters were tuned for 6000-step runs.Source: C. Learning stability · p. 6, Figure 5 · p. 6
D. Behaviour cloning
On the real robot, EDRIAD learned clip insertion faster than behavioral cloning, including with randomized socket positions. Ablations showed that the added losses and pretraining support performance, especially early in learning.
D. Behaviour cloning
- EDRIAD exceeded behavioral cloning within 2,000 steps (6 minutes) on both real-robot clip tasks; fixed-socket insertion took ~3.2s versus ~7.3s for the demonstrator and ~7.6s for cloning.The behavioral-cloning agent eventually overfit; the reported comparison uses the best performance from each of three runs.Source: D. Behaviour cloning · p. 6, Figure 6 · p. 7
- The behavioral-cloning agent achieved about 40% success on randomized clip insertion and about 80% on the fixed-socket task, but could not improve beyond the demonstrations.EDRIAD learned to exceed the supervised policy within 2,000 steps (6 minutes) in both variants.Source: D. Behaviour cloning · p. 6
E. Effects of auxiliary losses
- Removing EDRIAD’s contributions reduced real-robot performance, particularly early in training; in simulation, removing combinations of auxiliary losses impaired learning more than individual removals.Episode-progress and classification losses had small but significant effects; removing both had a larger effect, and removing the supervised loss had a large effect even with other losses present.Source: E. Effects of auxiliary losses · p. 7, Figure 7 · p. 7, Figure 8 · p. 7
F. Effectiveness of Pretraining
- With the full algorithm, pretraining significantly improved performance through 1,000 updates, whereas pretraining without the additional losses harmed performance, increasingly with more updates.Without stabilizing losses, pretraining was also reported to be sensitive to the number of pretraining updates.Source: F. Effectiveness of Pretraining · pp. 7–8, Figure 9 · p. 8
- Q-filtering provided no observed benefit, while standard convolutional networks were prohibitively slow to train and small networks were ineffective.Source: F. Effectiveness of Pretraining · p. 8
VI. DISCUSSION AND FUTURE WORK
The authors report that trained policies were robust to several perturbations, including cases not used in training. They also identify controller, tuning, and initial-grasp requirements as restrictive assumptions, and outline areas for future work.
VI. DISCUSSION AND FUTURE WORK
- The method still required a carefully engineered compliance controller, considerable hyperparameter tuning, and a plug firmly grasped before insertion.The authors identify these as restrictive assumptions that must be addressed for wider reach.Source: §VI · p. 8
- Policies tolerated lighting changes, human perturbations, and dynamic socket movement during an episode, despite not being trained for those cases.The authors describe the robustness of final policies after RL training as an anecdotal observation.Source: §VI · p. 8
- Future work will examine simpler losses, reduced hyperparameter sensitivity and simulation parameter sweeps, and more principled visual-feature extraction.Source: §VI · p. 8
APPENDIX
The appendix specifies separate actor and critic architectures, an image-based reward network, and the learning schedule used for training.
A. Network architectures
- The actor uses linear layers of 600, 400, and 7 nodes with ReLU activations and outputs actions; the critic uses 800, 600, 60, and 60 nodes and outputs a softmax distribution over Q.Source: A. Network architectures · p. 8
- The reward network processes 128 by 128 RGB images through three convolutional layers, then uses an 8-unit feature layer and predicts insertion with a single-node output.The convolutional layers use stride 3, kernel size 5 by 5, and 16, 32, and 16 channels.Source: A. Network architectures · p. 8
- Before environment interaction, the agent performs 1,000 demonstration-data updates with batches of 256; training then uses 40 updates per environment step, performed after each episode.Actor and critic target networks are updated every 10 learning updates.Source: A. Network architectures · p. 8