PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Variable Impedance Control for Contact Sensitive Tasks

Miroslav Bogdanovic, Majid Khadiv, Ludovic Righetti

arXiv:1907.07500v2cs.ROcs.AIcs.LG

TL;DR

Contact-rich robotic tasks are difficult for reinforcement learning, and the paper examines whether policy action-space structure improves robustness. It learns joint-space desired positions and impedance, finding better performance under contact uncertainty and demonstrating real-system deployment.

  • Problem

    Policy parameterization for reinforcement learning in contact-rich tasks remains an important question, as direct torque policies can require learning complex control behavior without explicit structure.

    Source: §I · p. 1, §II · p. 2

  • Method

    The paper learns policies that output joint-space desired positions and impedance, and regularizes variable-gain policies to aid interpretation and transfer.

    Source: §I · p. 1, §VI · p. 7

  • Results

    Variable gain control outperforms learned fixed-gain and direct-torque policies, especially under contact uncertainty, and the learned policies preserve performance and robustness on a real system.

    Source: §I · p. 2, §VII · p. 8

  • Takeaways & Limitations

    Trajectory-tracking regularization limits high-frequency desired-position changes and makes desired trajectories and feedback gains interpretable for contact-rich tasks.

    Source: §VI · p. 7

    Simulation omits robot flexibility, friction, drivetrain dynamics, delays, and finite bandwidth, so optimized policies may induce problematic real-robot oscillations.

    Source: C. Experiments on the real system · p. 4

Abstract

from arXiv · show

Reinforcement learning algorithms have shown great success in solving different problems ranging from playing video games to robotics. However, they struggle to solve delicate robotic problems, especially those involving contact interactions. Though in principle a policy directly outputting joint torques should be able to learn to perform these tasks, in practice we see that it has difficulty to robustly solve the problem without any given structure in the action space. In this paper, we investigate how the choice of action space can give robust performance in presence of contact uncertainties. We propose learning a policy giving as output impedance and desired position in joint space and compare the performance of that approach to torque and position control under different contact uncertainties. Furthermore, we propose an additional reward term designed to regularize these variable impedance control policies, giving them interpretability and facilitating their transfer to real systems. We present extensive experiments in simulation of both floating and fixed-base systems in tasks involving contact uncertainties, as well as results for running the learned policies on a real system.

I. INTRODUCTION

Contact transitions and uncertain contact properties make delicate robotic tasks difficult, while prior work leaves policy parameterization as an important question. The paper investigates joint-space variable impedance policies and reports stronger performance under contact uncertainty than learned fixed-gain or direct-torque policies.

I. INTRODUCTION

  • Contact establishment and breakage switch system dynamics, while unforeseen changes in contact location, friction, or stiffness can degrade robot behavior.These challenges arise in manipulation and locomotion.Source: §I · p. 1
  • The paper focuses on policy parameterization because end-to-end learning has emphasized observation complexity, especially vision, more than physical interaction.Prior work also motivates joint-space learning because task-space impedance can constrain redundancy resolution and possible motions.Source: §I · p. 1
  • The paper studies how policy parameterization affects reinforcement learning for contact-rich tasks and reports robust behaviors from policies that jointly generate desired positions and joint impedance.Experiments include a floating-base single-leg hopper and a fixed-base manipulator, with simulation and real-hardware results.Source: §I · p. 1
  • Variable gain control outperforms a wide range of learned fixed-gain and direct-torque policies, especially with contact uncertainty.The comparison includes two delicate force-control cases.Source: §I · p. 2

II. CONTROL POLICY PARAMETRIZATION

The section compares direct torque, fixed-gain PD, and variable-gain PD policy parameterizations. Variable gains let the policy adjust both desired joint positions and impedance, adding feedback flexibility at the cost of a larger action space.

II. CONTROL POLICY PARAMETRIZATION

  • Direct torque policies impose no control structure, but may need to learn desired motion and PD-like correction together, complicating meaningful exploration.Small torque changes can produce very different movements and task costs.Source: §II · p. 2
  • Fixed-gain PD policies output desired joint positions and use predefined gains; they are easier to learn for free-space motion but interaction becomes harder, especially under uncertainty.The controllers are state-based rather than tied to predefined time-based trajectories, allowing them to handle uncertainty in contact location and timing.Source: §II · p. 2
  • Variable-gain PD lets the policy modulate each joint’s desired position and impedance, adding a gain degree of freedom that can separate feedforward behavior from feedback response.The experiments use one output for stiffness and damping, with damping varied according to the square root of stiffness.Source: §II · p. 2
  • Doubling the action-space size is a drawback of variable-gain control, although the authors report that it rarely reduces learning performance.The parameterization is intended to retain fixed-gain PD’s ease of exploration while giving finer control over contact interactions.Source: §II · p. 2

III. EVALUATION PROCEDURE

The evaluation examines how controller parametrization affects performance under contact uncertainty, using contact-sensitive tasks across floating- and fixed-base systems. It also tests whether learned hopping policies transfer to hardware.

Evaluation goals

  • The study compares controller parametrizations under contact uncertainty because handling variation in critical environment parameters is relevant to simulation-to-real transfer.The evaluation seeks solutions able to handle environmental variation, which the paper identifies as important for transfer.Source: §III · pp. 2–3

Evaluation environments

  • Evaluations use a floating-base hopper and a fixed-base manipulator, whose tasks require distinct contact behaviors, with hopping policies also transferred to the real system.Manipulation requires fine interaction; hopping requires soft, reactive landings and quick force exertion.Source: §III · p. 3, Figure 2 · p. 3

Reinforcement learning algorithm

  • Policies are trained with DDPG, chosen as an off-policy algorithm for the challenges posed by joint-space control, contact discontinuities, and multipart rewards.The authors expect their results to remain consistent with other learning algorithms.Source: §III · p. 3

A. Task description

The task evaluates stable hopping while limiting hard impacts, under randomized ground-height changes. The reward combines height, force, and torque-smoothness considerations, creating a challenging learning problem.

Setup

  • The hopper is a two-degree-of-freedom leg system with its base restricted to vertical motion, retaining base movement and intermittent contacts during jumping.The restriction eliminates the falling-down effect.Source: A. Task description · p. 3

Task

  • Policies must achieve stable periodic hops with smooth landings and push-offs, while ground height varies randomly from −5 cm to 5 cm.This range is 31% of the hopper’s 32 cm fully stretched height.Source: A. Task description · p. 3
  • The state includes joint and base positions and velocities, but no explicit contact information.Source: A. Task description · p. 3

Reward function design

  • The reward uses base height, penalizes large forces and adds a torque-smoothness penalty, but the force penalty makes early hopping attempts difficult to reward.Initial bad landings can incur penalties larger than the reward for jumping.Source: A. Task description · p. 3

Quantitative comparison

  • Figure 3 presents final training scores, averaged learning curves from 100 runs per parametrization, and a view of final policies scoring above 1000.Source: Figure 3 · p. 4

B. Simulation results

In simulation, variable-gain control outperformed the best fixed-gain setting in average learning and found better solutions; on hardware, it outperformed fixed-gain policies across tested ground positions. A trajectory-tracking reward favored trackable desired positions and interpretable gain outputs.

B. Simulation results

  • Direct torque policies failed the task, while fixed-gain performance fell at gains above or below Kp = 5.0.Kp = 5.0 had the best average performance among the fixed-gain values examined.Source: B. Simulation results · p. 4
  • Across 100 runs, variable-gain control exceeded best-fixed-gain average performance, converged adequately more often and faster, and found solutions better than any fixed-gain policy.The fixed-gain comparison used its best-performing gain value.Source: B. Simulation results · p. 4

C. Experiments on the real system

  • Simulation policies can excite problematic robot oscillations through rapid control changes; the authors explore control parametrization rather than relying on dynamics randomization alone.The paper attributes the transfer concern to omitted or simplified robot dynamics and control delays in simulation.Source: C. Experiments on the real system · p. 4

B. Simulation results

  • In the learned hopping behavior, gains fell to their minimum before contact and rose quickly for push-off, while torque stayed near zero except at landing and push-off.The reported behavior absorbs impact and applies force smoothly without losing contact.Source: B. Simulation results · p. 4, Figure 4 · p. 5

C. Experiments on the real system

  • The tracking reward penalizes desired-position mismatch at the next step, favoring trackable setpoints; with it, variable-gain outputs tracked the trajectory more closely while fixed-gain outputs changed little.Both policy types still produced hopping with similar performance; the variable-gain policy retained low gains before contact and increased them for push-off.Source: C. Experiments on the real system · pp. 4–6, Figure 5 · p. 5
  • On hardware, all six evaluated policies achieved reasonable consecutive jumps on the first try; variable gain outperformed every fixed-gain policy and exceeded prior model-based height.The variable-gain policy also reached gains of Kp = 10.0 without the instability observed for fixed-gain policies at those values.Source: Figure 7 · p. 6, C. Experiments on the real system · p. 6

A. Task description

The fixed-base task trains a 7-DoF KUKA arm to trace a circle while maintaining table contact and a desired vertical force. Its reward encodes trajectory tracking, end-effector orientation, desired contact force, and avoidance of unwanted robot-table contact.

A. Task description

  • Gravity is compensated with a feedforward term, and the state includes all seven joint positions and velocities plus measured end-effector force.Source: A. Task description · p. 6
  • The 7-DoF KUKA task traces a circle with the end effector against a table while applying a desired constant vertical force.The robot starts from a random position and must reach the table, establish safe contact, and exert the force.Source: A. Task description · p. 6
  • The reward combines distance and tangential-velocity tracking with end-effector orientation, table contact and force, and a penalty for other robot-table contact.The trajectory terms use the closest point on the circle to support constant angular velocity without incentivizing a specific behavior.Source: A. Task description · pp. 6–7

B. Simulation results

Across simulations with table-height, friction, and stiffness variation, variable-gain control outperforms fixed-gain and direct-torque parametrizations. In a representative contact-transition experiment, it produces smoother forces without losing contact, unlike the comparison policies.

B. Simulation results

  • The comparison averages learning curves and standard deviations across six trainings per controller, using different circular trajectories.Training used a fixed, predefined number of episodes.Source: B. Simulation results · p. 7
  • Variable-gain control outperforms both other parametrizations; splitting control into motion and impedance parts makes good behavior easier to find, with one term handling circular motion and the other managing uncertainty or friction.Robustness was tested in separate simulations for table height (0.8–1.0 m), Coulomb friction (0–1), and surface stiffness (50–500 N/m), with environments uniformly sampled each training episode.Source: B. Simulation results · p. 7
  • For a representative contact-transition experiment, variable-gain control applies smooth forces without losing contact; direct torque loses contact frequently and fixed gains oscillate at high frequency.The authors note that the smoother forces could realistically be applied on a real robot.Source: B. Simulation results · p. 7

VI. DISCUSSION

The discussion links trajectory-tracking regularization to smoother desired-position changes and interpretable impedance policies. It also describes how the real-robot stiffness and damping range was bounded before learning state-dependent impedance.

VI. DISCUSSION

  • The trajectory-tracking reward term was crucial to sim-to-real transfer because it prevents high-frequency changes in desired position.It incentivizes changes consistent with system dynamics and constraints, provided they do not degrade task achievement.Source: §VI · p. 7
  • The regularizer makes desired positions interpretable as a feasible trajectory and the multiplier to tracking error as its feedback gains.This representation can provide insight into impedance modulation for contact-rich tasks, which the authors identify as an open problem.Source: §VI · p. 7
  • Decoupling feedforward and feedback paths can let a learned policy rely mostly on one term, undermining the intended physical meaning of the control terms.The discussion says an incentive is needed to favor one combination of control terms when multiple combinations can produce the same behavior.Source: §VI · p. 7
  • The authors first identified a real-system stiffness and damping range without instability, then constrained learned state-dependent impedance to that range.They report no instability when applying the resulting policies directly to the real robot.Source: §VI · p. 7

VII. CONCLUSION

The paper concludes that variable-impedance control improves policy performance and reliability on floating-base hopping and fixed-base table-wiping tasks, and that regularization gives desired trajectories and impedance physical meaning. It also reports direct deployment on a real system while preserving performance and robustness.

VII. CONCLUSION

  • Variable-impedance control produced better-performing and more reliable policies on both floating-base hopping and fixed-base table-wiping tasks.Source: §VII · p. 7
  • The proposed regularization term imposes physical meaning on the desired trajectory and impedance outputs.Source: §VII · p. 7
  • Figure 8 covers fixed-base evaluations varying table height, friction, and stiffness; Figure 9 depicts table-force interactions during circle tracking.Source: Figure 8 · p. 8, Figure 9 · p. 8
  • The learned policies were deployed directly on a real system while preserving performance and robustness.Source: §VII · p. 8
Loading 1907.07500v2…