PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Variable Impedance Control via Inverse Reinforcement Learning for Force-Related Tasks

Xiang Zhang, Liting Sun, Zhian Kuang, Masayoshi Tomizuka

arXiv:2102.06838v1cs.RO

TL;DR

Variable-impedance skills learned for contact-rich manipulation can be task-specific, limiting transfer when settings change. This paper uses IRL to recover policies and rewards from expert demonstrations and compares reward action spaces. Across simulation and real-robot evaluations, gain-space learning transfers better than behavior cloning and force-based IRL in the reported testing scenarios.

  • Problem

    Learned variable-impedance skills may be task-specific and sensitive to changes in task settings, motivating a more general way to learn them.

    Source: §I · p. 1

  • Method

    The approach uses IRL to recover a variable-impedance policy and reward from demonstrations, then applies RL to optimize policies for different task settings.

    Source: §I · p. 1

  • Results

    In simulation and on the real robot, the approach recovered expert behavior in training and showed better transfer than behavior cloning and force-based IRL in the reported testing scenarios.

    Source: §I · p. 2, §V · p. 8

  • Takeaways & Limitations

    The authors conclude that using impedance gain as the action can improve reward transfer performance, with experiments spanning Peg-in-Hole and Cup-on-Plate tasks.

    Source: §I · p. 1, §V · p. 8

    The method uses a simplified impedance law without measuring external force and assumes fixed, given waypoints, limiting its scope with respect to force-aware dynamics and time-varying goals.

    Source: §V · p. 8

Abstract

from arXiv · show

Many manipulation tasks require robots to interact with unknown environments. In such applications, the ability to adapt the impedance according to different task phases and environment constraints is crucial for safety and performance. Although many approaches based on deep reinforcement learning (RL) and learning from demonstration (LfD) have been proposed to obtain variable impedance skills on contact-rich manipulation tasks, these skills are typically task-specific and could be sensitive to changes in task settings. This paper proposes an inverse reinforcement learning (IRL) based approach to recover both the variable impedance policy and reward function from expert demonstrations. We explore different action space of the reward functions to achieve a more general representation of expert variable impedance skills. Experiments on two variable impedance tasks (Peg-in-Hole and Cup-on-Plate) were conducted in both simulations and on a real FANUC LR Mate 200iD/7L industrial robot. The comparison results with behavior cloning and force-based IRL proved that the learned reward function in the gain action space has better transferability than in the force space. Experiment videos are available at https://msc.berkeley.edu/research/impedance-irl.html.

I. INTRODUCTION

Variable impedance is needed for safe interaction across task phases, but existing learned skills may not transfer when task settings change. The paper applies IRL to recover variable-impedance policies and rewards, and reports better transfer performance for gain-space learning than behavior cloning and force-based IRL.

  • Robots in unstructured environments must interact safely and stably, and complex tasks can require impedance changes across task phases.Source: §I · p. 1
  • Prior learning-from-demonstration skills may fail when task settings change, while designing a suitable reward for deep RL is challenging.Source: §I · p. 1
  • The paper recovers both the variable-impedance policy and reward from demonstrations, then uses RL to generate policies for different task settings.The learned reward depends only on the current observation and action, which the authors describe as agnostic to task settings.Source: §I · p. 1
  • The paper examines whether rewards for force-related tasks should use force or impedance gain as the action space, reporting better transferability for gain-space learning.The experiments compare behavior cloning and study impedance-gain and Cartesian-force action spaces in Peg-in-Hole and Cup-on-Plate.Source: §I · pp. 1–2
  • The approach successfully recovers expert variable-impedance policies and achieves better transfer performance than behavior cloning and force-based IRL in the testing scenarios.Source: §I · p. 2

A. Force-control related tasks in robotics

The background reviews force-control approaches, variable-impedance learning, and IRL before introducing Cartesian impedance control. The paper positions recovered rewards and gain-based actions as ways to generate skills for changed task settings.

A. Force-control related tasks in robotics

B. Variable Impedance Control

  • Earlier variable-impedance demonstrations learned task-specific skills, whereas this approach recovers a reward that can be reoptimized with RL for new settings.Source: B. Variable Impedance Control · p. 2

C. Inverse reinforcement learning

  • IRL infers a reward or cost from demonstrations for forward RL, and this paper examines gain actions as a more general expert-policy representation than force actions.Source: C. Inverse reinforcement learning · p. 2

A. Cartesian space impedance control

  • The Cartesian impedance law maps tracking error and velocity through stiffness and damping gains to feedback force, alongside a feed-forward term for nonlinear dynamics.The controller observes tracking error and tracking velocity; the policy outputs either gains or feedback force, depending on the action-space design.Source: A. Cartesian space impedance control · p. 3

B. Learning variable impedance skills with AIRL

The approach uses AIRL to learn a variable-impedance policy and reward from demonstrations, then holds the learned reward fixed while reoptimizing a policy for a different task setting.

B. Learning variable impedance skills with AIRL

IV. EXPERIMENTS

The evaluation includes simulated and real-robot experiments, with human demonstrations for the real Cup-on-Plate task. Generalizability is assessed against four baseline methods.

IV. EXPERIMENTS

  • Experiments evaluate the approach in simulation and on a real robot.Source: §IV · p. 3
  • For real-robot validation, human expert data for Cup-on-Plate were collected by kinesthetic teaching and learned policies were tested on the robot.Source: §IV · p. 4
  • The generalizability comparison includes force-based AIRL, gain-based BC, force-based BC, and constant gain.The authors hypothesize that gain actions improve transferability and that AIRL is more general than BC.Source: §IV · p. 4

A. Tasks in simulator

The simulator evaluation uses Peg-in-Hole and Cup-on-Plate tasks, including a simulated FANUC industrial robot for cup placement.

A. Tasks in simulator

  • The simulator evaluation covers Peg-in-Hole and Cup-on-Plate, with the goal of placing the cup on the plate quickly and steadily.Source: 1) Task setups · p. 4
  • The Peg-in-Hole peg and hole diameters are 25.37 mm and 25.40 mm, respectively.Source: 1) Task setups · p. 4
  • The Cup-on-Plate simulation includes a FANUC LR Mate 200iD/7L, and Mujoco automatically calculates the model parameters from the simulation model.Source: 1) Task setups · p. 4

2) Observation spaces:

Observations combine tracking error and velocity, with a five-step history also evaluated. The action design and expert data generation differ by task, including task-specific impedance outputs and controller phases.

2) Observation spaces:

  • Observations use tracking error and tracking velocity, and an augmented version includes their histories from the previous five time steps.The end-effector is located at the peg COM for Peg-in-Hole and on the cup for Cup-on-Plate.Source: 2) Observation spaces · p. 4

3) Action spaces:

  • Policies can output Cartesian force or impedance gains; diagonal positive stiffness and damping matrices reduce gain-action dimensionality and ensure positive definiteness.Peg-in-Hole outputs six stiffness values with fixed damping, whereas Cup-on-Plate adds one damping factor; the full-matrix extension can use Cholesky decomposition.Source: 3) Action spaces · p. 4

4) Generating expert data:

  • Fifty trajectories per task were collected from designed variable-impedance controllers, including a fixed tip-stiffness expert for Peg-in-Hole.Because the tip-to-COM Jacobian depends on peg configuration, fixed tip stiffness becomes variable stiffness at the COM.Source: 4) Generating expert data · p. 4
  • The Cup-on-Plate expert controller accelerates with the largest gain, switches to a smaller gain, then approaches the plate at minimum speed for safety.The switching points are 0.4 m and 0.2 m.Source: 4) Generating expert data · p. 4

5) Performance score:

The performance functions score task-specific tracking behavior: Peg-in-Hole penalizes misalignment or vertical error, while Cup-on-Plate penalizes position and velocity errors according to task phase.

  • Peg-in-Hole receives a constant penalty when the peg is misaligned; otherwise, the function penalizes vertical distance to the target.Source: 5) Performance score · p. 4
  • Cup-on-Plate penalizes tracking error during acceleration, tracking velocity during reaching, and both during switching.Source: 5) Performance score · p. 5

6) Training details:

The evaluations compare imitation in training with transfer to altered settings, using Peg-in-Hole simulation tests, Cup-on-Plate tests, and a real-robot evaluation. Gain-based AIRL generally transfers well, with results varying by task and test condition.

6) Training details:

  • The reward network used two layers with 32 units; TRPO batch size and trajectory length were 8000 and 200 for Peg-in-Hole, and 10000 and 500 for Cup-on-Plate.Source: 6) Training details · p. 5

7) Evaluations:

  • All Peg-in-Hole policies achieved high, similar success rates in training, but gain-based AIRL achieved the highest success rate as peg tilt changed.Source: 7) Evaluations · p. 5, Table I · p. 5
  • With smaller Peg-in-Hole mesh scales, gain-based AIRL achieved a higher success rate, while gain actions improved transferability for both AIRL and behavior cloning.Source: 7) Evaluations · pp. 5–6, Table II · p. 5
  • Augmenting observations with five past tracking errors and velocities did not improve the approaches; force-based AIRL became harder to train and generalized worse.Source: 7) Evaluations · p. 6
  • In Cup-on-Plate training, all four policies performed similarly to the expert; AIRL recovered many expert features, while behavior cloning best imitated expert actions.Source: 7) Evaluations · p. 6, Table III · p. 6
  • Gain-based AIRL outperformed other policies in Peg-in-Hole tilt tests T1, T2, and T4, falling behind only the constant-gain baseline in T3.Source: 7) Evaluations · p. 6, Table III · p. 6
  • On the real robot, Fig. 6 compares gain-based AIRL and BC policies in training and testing, while Tables IV and V report average trajectory deviation and final deviation, respectively.Source: Figure 6 · p. 7, Table IV · p. 7, Table V · p. 7
  • In the Cup-on-Plate test, force-based AIRL applied forces in the wrong x and y directions and failed, while gain-based AIRL recovered expert gain trends and forces in those axes.Source: 7) Evaluations · pp. 6–7, Figure 5 · p. 6

B. Experiments on the real robot

The paper evaluates its method on a real-world Cup-on-Plate task using expert demonstrations collected on the real robot.

1) Task setup:

The task setup uses Cartesian variable impedance control and records expert state-force demonstrations. Expert gains are then estimated from short sliding windows of those data.

1) Task setup:

  • The real-robot setup used a host PC to run Cartesian variable impedance control and a target PC connected through Simulink Real-Time; robot dynamics parameters were obtained by the Euler-Lagrange method.Source: 1) Task setup · p. 7

2) Collecting human expert data:

  • For Cup-on-Plate demonstrations, a human applied force and torque measured by an F/T sensor; the recorded data paired tracking state with expert force across 30 trajectories.Source: 2) Collecting human expert data · p. 7

3) Gain estimation using sliding window method:

4) Evaluation:

Gain-based AIRL transferred better than the baselines in simulation and outperformed behavior cloning at a different real-robot initial position. The conclusion notes limits from simplified impedance dynamics and fixed, task-given waypoints.

4) Evaluation:

  • In real-robot training, gain-based AIRL and BC both recovered expert gains, with average expert deviations close to 10 mm and final target deviations of 12 mm.The evaluation used human expert data and trained the policy and reward function in simulation.Source: 4) Evaluation · pp. 7–8
  • In the real-robot testing scenario, average expert deviation was 13.4 mm and final deviation was 10.8 mm, both much smaller than gain-based BC.The deviations were in the same range as in training.Source: 4) Evaluation · p. 8
  • Force-based AIRL failed to finish Cup-on-Plate in training and testing; its simulated velocity responses differed from the real robot, while gain feedback limited trajectory deviation.Force-based BC also failed in testing, despite reaching the target in training with a larger final deviation than the gain-based methods.Source: 4) Evaluation · p. 8

V. CONCLUSIONS

  • The approach recovers variable-impedance policies and rewards from demonstrations, then optimizes the learned reward for new task settings; the authors argue gain actions improve transfer.They attribute this to the feedback control law.Source: §V · p. 8
  • Gain-based AIRL had better transfer results than all baselines in simulation and outperformed BC at a different real-robot initial position despite a sim-to-real gap.In real-robot training, it also recovered the expert trajectory.Source: §V · p. 8
  • The simplified impedance law omits external-force measurement and leaves coupled Cartesian dynamics; the approach also assumes fixed, task-given waypoints.The authors propose adding force to the control law and making expert waypoints policy outputs to extend the method to time-varying goals.Source: §V · p. 8
Loading 2102.06838v1…