PaperlayerView exact revision on arXiv

Source-linked AI summary

Reinforcement Learning on Variable Impedance Controller for High-Precision Robotic Assembly

Jianlan Luo, Eugen Solowjow, Chengtao Wen, Juan Aparicio Ojea, Alice M. Agogino, Aviv Tamar, Pieter Abbeel

arXiv:1903.01066v2cs.RO

TL;DR

High-precision robotic assembly requires policies that handle tight contacts, friction, and force-sensitive behavior. The paper combines reinforcement learning with operational-space force control and a force/torque-aware neural architecture, reporting successful force-based assembly behavior and improved generalization to local variations while noting future extensions to multimodal sensing and contact modeling.

  • Problem

    High-precision gear-set assembly requires force-sensitive behavior under tight tolerances, contacts, and friction, motivating learned policies beyond manually designed heuristics.

  • Method

    The paper combines iLQG reinforcement learning with operational-space motion/force control and a neural architecture that explicitly incorporates force/torque information.

  • Results

    The method performs best among the evaluated high-precision approaches and its force/torque-aware neural architecture produces better generalization to local environment variations.

  • Takeaways & Limitations

    Learning force actions can implicitly generate task constraints, while force/torque-aware policies support adaptive compliant behavior and robustness to local variations.

Abstract

from arXiv · show

Precise robotic manipulation skills are desirable in many industrial settings, reinforcement learning (RL) methods hold the promise of acquiring these skills autonomously. In this paper, we explicitly consider incorporating operational space force/torque information into reinforcement learning; this is motivated by humans heuristically mapping perceived forces to control actions, which results in completing high-precision tasks in a fairly easy manner. Our approach combines RL with force/torque information by incorporating a proper operational space force controller; where we also exploit different ablations on processing this information. Moreover, we propose a neural network architecture that generalizes to reasonable variations of the environment. We evaluate our method on the open-source Siemens Robot Learning Challenge, which requires precise and delicate force-controlled behavior to assemble a tight-fit gear wheel set.

I. INTRODUCTION

The paper targets autonomous learning of adaptable, robust control policies for high-precision assembly, where tight tolerances, contacts, friction, and incomplete system models challenge conventional controllers. It combines reinforcement learning with force-aware control and evaluates the approach on a four-step gear-set assembly benchmark.

  • Industrial robot performance depends on accurately tracking predefined trajectories, while adaptable control must handle unmodeled behaviors and generalize across conditions.
  • Humans can perform high-accuracy insertion by feeling contacts and using force-feedback heuristics such as probing or searching along a surface.
  • Reinforcement learning acquires reactive policies through trial-and-error interaction, but sparse rewards can leave it stuck in local optima.Operational space control can shape actions toward task-space goals and restrict policy search to promising solution regions.
  • The paper asks whether operational-space actions, learned adaptive impedance, and force/torque-aware neural policies improve high-precision assembly and generalization.
  • The benchmark assembles a gear set through four sequential tasks involving peg insertion, gear placement, squared-shaft assembly, and gear-tooth alignment.The second step requires tolerances tighter than 0.1 mm, while later steps require precise orientation and rotation.
  • Prior work includes model-based and model-free RL, contact-rich guided policy search, explicit contact-dynamics methods, and planner-shaped torque control, but high-precision limitations remain.

III. PRELIMINARIES

The paper formulates assembly as finite-horizon control of already-grasped objects moving toward goal positions under unknown dynamics and possible algebraic constraints. Reinforcement learning optimizes operational-space force and torque actions for this problem.

  • The considered tasks move already-grasped objects to goal positions, with success measured by minimizing distance to those goals.
  • Robot states x_t and actions u_t are optimized over a task horizon T using a task-related cost ℓ(x_t, u_t).
  • The system dynamics f are unknown, and states or actions may additionally satisfy algebraic constraints.
  • The control action is an operational-space wrench F_tip = [F_x, F_y, F_z, M_x, M_y, M_z], representing desired force, torque, or impedance.The paper optimizes this wrench through reinforcement learning.

IV. REINFORCEMENT LEARNING WITH FORCE CONTROL

The method combines a sample-efficient model-based RL algorithm with operational-space force control, using learned force actions to encode task constraints and explicitly feeding force/torque measurements into a neural policy. This design is intended to improve precision and robustness to local variations.

  • The paper combines iterative Linear-Quadratic-Gaussian reinforcement learning with operational-space force and hybrid motion/force controllers.The neural network architecture explicitly incorporates force information for generalization.
  • High-precision insertion is simplified by constraining motion and forces according to contact conditions, such as preventing downward motion before alignment.
  • The method implicitly generates Pfaffian constraints by combining natural environmental constraints with artificial task constraints.
  • The proposed neural architecture explicitly uses current-time force/torque measurements to derive control actions and adapt to local environment variations.

A. Operational Space Motion/Force Controller

The operational-space controller maps desired end-effector wrenches into joint torques, while a low-gain position loop stabilizes motion in free space. Its learned force component supports configuration-dependent adaptive impedance behavior.

  • A. Operational Space Motion/Force Controller: The operational-space force controller uses the desired end-effector wrench F_tip to generate joint-space control through the robot Jacobian and applied joint torques.
  • A. Operational Space Motion/Force Controller: For the 7-DOF Sawyer arm, the joint torque command combines gravity compensation, Jacobian-transformed force, and a nullspace torque component.
  • A. Operational Space Motion/Force Controller: Because direct force control could continuously accelerate the robot during free-space motion, the experiments wrap it with a small-gain position loop.
  • A. Operational Space Motion/Force Controller: The hybrid controller weights motion and force components using diagonal matrices and desired joint positions and velocities obtained through inverse kinematics.
  • A. Operational Space Motion/Force Controller: The learned time-varying force controller is intended to yield high impedance in free space and high admittance during contact.

B. Iterative Linear-Quadratic-Gaussian Controller

The paper uses iLQG as a sample-efficient model-based reinforcement-learning method that iteratively improves controllers by optimizing local linear-Gaussian dynamics and costs around nominal trajectories.

  • iLQG is selected because it is sample efficient and has convenient second-order methods for rapid solution.
  • The algorithm represents a trajectory as ω = {x_1, u_1, ... , x_T, u_T} and seeks to minimize expected trajectory cost.The state typically includes joint angles, end-effector pose, and their time derivatives.
  • iLQG iteratively linearizes dynamics around the current nominal trajectory and constructs a quadratic cost approximation.It then computes actions by dynamic programming and refits linear-Gaussian dynamics.

C. Interpretation as Learning Pfaffian Constraints

The method interprets force-based reinforcement learning as learning task-specific Pfaffian constraints through interaction, while combining contact constraints with operational-space motion dynamics.

  • Contact interactions with a rigid environment are formulated as holonomic and nonholonomic Pfaffian constraints.
  • The operational-space twist V belongs to SE(3) with six components, while A(q) encodes k natural constraints.
  • Operational-space motion dynamics are expressed as F = Λ(q) V̇ + η(q, V), where Λ and η represent transformed inertia and dynamic terms.This is the robot’s motion dynamics expressed in operational space without the tip wrench F_tip.
  • Combining motion and constraint equations yields constrained dynamics with an additional A^T(q)λ term, requiring requested wrenches to lie in the column space of A^T(q).
  • Learning F_tip through continuous reinforcement-learning interaction can be viewed as iteratively improving A(q) to describe manipulation tasks more accurately.The paper identifies this learned constraint description as important for high-precision settings.

D. Training Neural Network controller using MDGPS

The neural controller explicitly processes wrist force/torque measurements while MDGPS trains it to match local linear-Gaussian controllers learned by iLQG.

  • The architecture filters noisy force/torque measurements with a low-pass filter and concatenates them at the second-last neural-network layer.
  • Force/torque readings are introduced as direct haptic features while avoiding an early coupling between external measurements and robot internal states.
  • The architecture is designed so free-space behavior remains robot-configuration-dependent rather than force-dependent.The paper notes that directly feeding noisy force/torque inputs into the first layer can produce random motions.
  • MDGPS trains π_θ(u_t|x_t) by minimizing its KL-divergence from the linear-Gaussian controller learned through iLQG.
  • Guided policy search can implement this training through supervised learning and add a KL-divergence cost to enforce agreement between local and global policies.

V. EXPERIMENTS

The experiments evaluate force-based reinforcement-learning controllers and their ablations on four assembly tasks using a Sawyer robot, with training performed through iterative local-controller and neural-policy updates.

  • The experiments ask how proposed iLQG with force control performs, whether it exploits contact constraints, and how it compares with force-processing ablations.
  • Algorithm 1 trains a local iLQG controller using an operational-space force controller, projects it to joint torque, and then trains the neural controller with MDGPS.
  • The study also evaluates whether the proposed neural architecture improves local generalization relative to its ablations.
  • Four assembly tasks are evaluated on a Rethink Robotics Sawyer robot with wrist force/torque measurements and ROS control at 20 Hz.During training, four roll-outs are collected per iteration.

B. Assembly Performance Results

The study compares kinematics and torque-control baselines with an operational-space controller across four tight-tolerance assembly tasks. Experiments show that learned force-based behavior reproduces probing, forceful insertion, and fine-tuning strategies.

  • Experimental setup: The evaluation compares kinematics-only control, direct-torque iLQG, augmented-state torque control, and operational-space controller variants.Each method is evaluated on four assembly tasks, with success defined as reaching the desired pose within a specified tolerance.
  • Experimental setup: Success rates are reported separately for each task because an independent policy is trained for every task.Overall success could be obtained by multiplying the independently trained policies’ success rates.
  • Baseline comparison: Kinematics-only control fails consecutively, while direct-torque iLQG without extensive cost shaping produces only one success on the easiest task.The authors attribute that isolated success to Gaussian controller noise producing a lucky insertion motion.
  • Baseline comparison: Augmenting the state with force/torque measurements does not reliably improve iLQG, because the signals are non-Markovian for the fitted dynamics model.The authors report that time-correlated modeling of these measurements does not produce meaningful information.
  • Learned assembly behavior: During task 1, the learned robot moves quickly to contact, probes slowly, then applies a large downward force after estimating the hole position.This sequence combines exploratory contact behavior with an aggressive final insertion motion.
  • Learned assembly behavior: During task 3, downward force stabilizes the gear while rotating torque aligns it, followed by iterative insertion and horizontal pose fine-tuning.The observed behavior roughly aligns with human heuristics for contact-rich assembly.
  • Learned assembly behavior: The policy’s task-2 insertion behavior varies exploration noise over time: it explores initially, becomes aggressive near the goal, and injects noise again for friction compensation.The force-based pattern is discovered through interaction and is reported to match human intuition.

C. Generalization Results

The paper tests whether a force/torque-aware neural controller generalizes when the assembly base moves. The proposed architecture is compared with direct force/torque input and its underlying iLQG controller, with mixed results and high variance.

  • Experimental design: Generalization experiments train three policies on task 2 with a fixed base, then evaluate them unchanged after moving the base.The compared policies are the proposed architecture, a network receiving force/torque data at its first layer, and the iLQG controller used for training.
  • Architecture comparison: The proposed architecture filters force/torque readings and concatenates them in the second-last network layer rather than forcing early correspondence with internal robot states.This design treats tool-space haptic information as direct features for control.
  • Experimental design: The base is moved by 1cm, 2cm, and 5cm to test success-rate generalization across local position variations.Success rates for these settings are reported in Table II.
  • Architecture comparison: Directly inputting force/torque data into the network’s first layer often aborts training because of large KL-divergence from iLQG.Policies that did train sometimes produced undesirable free-space motions and behaved poorly in three experiments.
  • Architecture comparison: The proposed neural controller performs slightly better than iLQG on average, but its result variance is high and it does not consistently outperform iLQG.The authors suggest that a more accurate six-axis force/torque controller might mitigate this issue.

VI. CONCLUSIONS AND FUTURE WORK

The paper combines reinforcement learning with operational-space force control for high-precision assembly and reports improved generalization from explicitly processing force/torque information. Future work extends the architecture toward multimodal sensing and more structured contact modeling.

  • VI. CONCLUSIONS AND FUTURE WORK: Combining RL with an operational space force controller solves high-precision assembly by automating operational-space Pfaffian constraints.The authors regard these constraints as crucial for high-precision tasks.
  • VI. CONCLUSIONS AND FUTURE WORK: The proposed method performs best among the evaluated ablations in the high-precision setting.
  • VI. CONCLUSIONS AND FUTURE WORK: A neural network architecture that explicitly uses force/torque information improves generalization.
  • VI. CONCLUSIONS AND FUTURE WORK: Future work proposes adding raw vision and tactile inputs to create an end-to-end multimodal neural network.
  • VI. CONCLUSIONS AND FUTURE WORK: Future work also considers explicitly modeling contact to structure Pfaffian constraints, reduce sample complexity, and support policy transfer.
Loading 1903.01066v2…