PaperlayerView exact revision on arXiv

Source-linked AI summary

Variable Impedance Control in End-Effector Space: An Action Space for Reinforcement Learning in Contact-Rich Tasks

Roberto Martín-Martín, Michelle A. Lee, Rachel Gardner, Silvio Savarese, Jeannette Bohg, Animesh Garg

arXiv:1906.08880v2cs.ROcs.AIcs.LG

TL;DR

Research on RL has focused more on observation spaces and rewards than on action-space choice, even though manipulation tasks have differing control requirements. This paper compares action spaces and proposes VICES, reporting improved sample efficiency, energy efficiency, and safety across three tasks, as well as transfer to other robots and the real robot.

  • Problem

    The effect of action-space choice on deep RL for contact-rich manipulation has received limited empirical study, despite task-dependent control needs.

    Source: §I · pp. 1–2

  • Method

    The paper compares common joint- and end-effector-space action spaces across three manipulation tasks and introduces VICES, which gives the policy control over end-effector impedance.

    Source: §I · p. 2, §IV · p. 4

  • Results

    Across tasks, VICES outperforms other action spaces on sample efficiency, energy consumption, and safety, and policies transfer to other simulated robots and a real robot without fine-tuning.

    Source: §VI · p. 8

  • Takeaways & Limitations

    The authors present VICES as an efficient action-space choice for RL under dynamically variable contact conditions.

    Source: §VI · p. 8

Abstract

from arXiv · show

Reinforcement Learning (RL) of contact-rich manipulation tasks has yielded impressive results in recent years. While many studies in RL focus on varying the observation space or reward model, few efforts focused on the choice of action space (e.g. joint or end-effector space, position, velocity, etc.). However, studies in robot motion control indicate that choosing an action space that conforms to the characteristics of the task can simplify exploration and improve robustness to disturbances. This paper studies the effect of different action spaces in deep RL and advocates for Variable Impedance Control in End-effector Space (VICES) as an advantageous action space for constrained and contact-rich tasks. We evaluate multiple action spaces on three prototypical manipulation tasks: Path Following (task with no contact), Door Opening (task with kinematic constraints), and Surface Wiping (task with continuous contact). We show that VICES improves sample efficiency, maintains low energy consumption, and ensures safety across all three experimental setups. Further, RL policies learned with VICES can transfer across different robot models in simulation, and from simulation to real for the same robot. Further information is available at https://stanfordvl.github.io/vices.

I. INTRODUCTION

The introduction argues that action-space choice affects learning and control in manipulation, and presents VICES as an action space for contact-rich tasks. It motivates comparing action spaces across tasks and highlights VICES’s reported efficiency and transfer benefits.

  • Robot control separates reference generation from tracking: an outer map g(o) converts observations to reference signals, while an inner map f(a) converts them to actuation commands.Source: §I · p. 1
  • Because action space determines what the inner loop tracks, it affects tracking error and thereby robustness, task performance, learning efficiency, and exploration.Source: §I · p. 1
  • Contact-rich tasks can require time-varying stiffness and compliance across directions, making manual specification of task constraints difficult and unscalable.Source: §I · p. 1
  • The study compares joint position, velocity, torque, and variable impedance with fixed and variable end-effector impedance across path following, door opening, and surface wiping.These tasks represent no contact, constrained mechanisms, and continuous unconstrained contact, respectively.Source: §I · p. 2
  • VICES improves RL exploration and energy efficiency, reduces applied forces, and supports robust transfer across robots with different dynamics in simulation and the real world.Source: §I · p. 2

II. RELATED WORK

Prior approaches to impedance control and learning often rely on task knowledge, demonstrations, or initial trajectories. This work instead studies learning from scratch with control functions that map policy actions to robot commands across manipulation tasks with different contact properties.

Robot Motion Control

  • Impedance control supports safe contact with unknown environments by regulating force as the robot deviates from a kinematic goal, avoiding a hard separation of task dimensions [10].Earlier abstractions split kinematic and dynamic control, but that decoupling can require unavailable task knowledge.Source: §II · p. 2

Variable Impedance and Learning from Demonstrations

  • Existing variable-gain methods schedule impedance for a given trajectory, whereas this work predicts end-effector displacement and gains from observations.In learning from demonstrations, variable impedance has been tied to trial coherence or sensed force, and applied to adaptive grasping, deformable-object manipulation, and human co-adaptation.Source: §II · p. 2

Reinforcement Learning

  • Prior RL methods learned variable impedance with initial trajectories: Kim et al. required one, while Buchli et al. also used joint-space actions, limiting transferability.Kim et al. demonstrated convergence, robustness, and energy efficiency in simulated planar-arm manipulation; Buchli et al. optimized energy while accomplishing a task, but also required an initial solution estimate.Source: §II · p. 2
  • Rey et al. learned task-space impedance from exploration alongside demonstrated trajectories, while refined trajectories with force profiles; this work learns trajectories and stiffness profiles through impedance.Rey et al. used Gaussian Mixture Regression and one stiffness parameter, demonstrating simulation and one real-world planar task; used PI2 [44] and force-torque readings.Source: §II · p. 2
  • Unlike prior approaches bootstrapped with demonstrations, this work explores learning from scratch; Viereck et al. found controller-structured actions suited hopping with intermittent contact.Viereck et al. compared desired torques with feedback parameters and desired configuration transformed analytically into torques; the latter action space performed best and made network outputs more interpretable.Source: §II · p. 2
  • Peng et al. studied action representations for imitation learning in simulated bipedal locomotion; this work extends the action-space question to contact-rich manipulation and preliminary real-world transfer.Source: §II · p. 2

III. REINFORCEMENT LEARNING

The section formulates reinforcement learning as policy optimization in a continuous-state, continuous-action Markov decision process, with policies acting on states or observations. It uses PPO for evaluating action spaces and defers comparisons of algorithm sensitivity to future work.

III. REINFORCEMENT LEARNING

  • The assumed problem is a discrete-time continuous Markov decision process, and the goal is to find a policy that maximizes expected reward from environmental interactions.The process is defined by state and action spaces, transition and reward functions, a discount factor, and an initial-state distribution.Source: §III · p. 3
  • When states are not directly observable, the policy conditions on observations, and policy gradients optimize its parameters to maximize expected future return.At each time step, the agent receives an observation, acts, receives a reward, and obtains a new observation.Source: §III · p. 3
  • The policy-gradient theorem underlies algorithms including TRPO, DDPG, and A2C; the action-space evaluation uses PPO, while algorithm sensitivity is deferred to future work.The action-value function is associated with the current policy.Source: §III · p. 3

IV. ACTION SPACES IN RL FOR ROBOT MANIPULATION

The section treats an RL action as a reference mapped to robot commands by a controller, and introduces VICES, which lets the policy choose both end-effector pose and impedance gains. It describes common action spaces and the controllers that translate them into joint torques.

Common action spaces

  • Common manipulation action spaces include joint torques, velocities, positions, and end-effector pose; specified desired torques limit applied forces, making torque control safer for contact-rich tasks.The section assumes joint torques are the lowest-level commands for its physical agent.Source: §IV · p. 3

Motivation

  • Contact and force constraints can make motion control alone insufficient; success may require dynamically modulating environmental force through joint torques.The section gives articulated-object manipulation and applying the minimum force to wipe a surface as examples.Source: §IV · p. 3

Action-to-command mapping

  • Analytic controllers map policy actions and robot state to low-level commands, and making controller parameters policy outputs gives the agent control over manipulation behavior.Policies act at 20 Hz while controllers generate robot commands at 500 Hz, with linear interpolation between policy actions.Source: §IV · p. 3, Figure 2 · p. 3

End-Effector Pose

  • An impedance PD controller converts end-effector pose error and velocity into acceleration, then operational-space dynamics and the Jacobian map the desired wrench to joint torques.The controller separates position and orientation terms using the end-effector pose and velocity.Source: §IV · pp. 3–4

Variable Impedance End-Effector Space (VICES)

  • VICES augments end-effector pose actions with policy-selected impedance gains, enabling adaptation along six translation and rotation axes as task phases change.The paper extends an approach previously explored for joint-space gains.Source: §IV · p. 4

V. EXPERIMENTS

The experiments compare action spaces across manipulation tasks spanning free motion to constrained, contact-rich interaction, assessing learning, physical efficiency, and transfer.

V. EXPERIMENTS

  • The study evaluates action spaces in Path Following, articulated-mechanism manipulation, and surface wiping, spanning free motion to highly constrained contact-rich tasks.These domains are used to compare action-space characteristics for policy learning.Source: §V · p. 4
  • The evaluation covers RL suitability, sample efficiency and task completion, torque-proportional energy use, contact wrenches, cross-robot transfer, and sim-to-real transfer.Physical effort is assessed through environmental wrenches during contact-rich manipulation; energy is estimated assuming joint torque is proportional to electric power.Source: §V · p. 4
  • Policies output actions at 20 Hz while joint torque commands run at 500 Hz, with commands generated using current robot state and interpolated linearly between policy steps.The controller holds the policy's desired goal constant while updating the robot state to generate higher-frequency torque commands.Source: §V · p. 4

A. Free-Space Motion - Path Following

In free-space four-via-point path following, VICES in end-effector space consistently improved energy efficiency while maintaining task performance. End-effector policies also transferred zero-shot between simulated robots, unlike joint-space policies.

A. Free-Space Motion - Path Following

  • The fixed four-via-point path was trained first for completion, then with an energy penalty, using via-point crossings and torque-based energy estimates as outcomes.The energy estimate assumed motor torque proportional to current and constant voltage, with energy calculated as the time integral of power.Source: A. Free-Space Motion - Path Following · p. 5
  • All policies except reference joint-torque and low-fixed-impedance end-effector policies learned to pass through all four via-points.Success was measured as the fraction of the four fixed via-points crossed on a free-space trajectory.Source: A. Free-Space Motion - Path Following · p. 5
  • As each via-point was checked, VICES adjusted stiffness and paired it with desired displacement to move toward the next point while avoiding overshoot.The four via-points shared an x-coordinate, and the policy learned not to exert force in that dimension.Source: Figure 4 · p. 5, A. Free-Space Motion - Path Following · p. 5
  • VICES in end-effector space was the only end-effector action space that consistently improved energy efficiency while maintaining path-following performance.The medium- and high-fixed-impedance end-effector models became unstable when optimizing energy; the low-impedance model was not evaluated with the energy penalty because it never completed the task.Source: A. Free-Space Motion - Path Following · p. 5
  • Joint-space variable-impedance policies reduced energy more than fixed-impedance controllers and end-effector variable-impedance policies, partly because they began with higher energy consumption.The absolute optimization difference was attributed to joint-space policies moving farther per time step and completing the task faster in the first phase, leaving more room to reduce energy.Source: A. Free-Space Motion - Path Following · pp. 5–6
  • After convergence, only fixed- and variable-impedance end-effector policies transferred zero-shot from simulated Panda to Sawyer; joint-space policies did not.End-effector control factors out robot embodiment effects, while joint-space transfer was impeded by differences in the robots’ kinematics and dynamics.Source: Figure 5 · p. 6, A. Free-Space Motion - Path Following · p. 6

B. Manipulation of Constrained Mechanisms - Door Opening

In door opening, end-effector-space policies achieve stronger task completion than other action spaces, while transferring to a different robot more successfully than joint-space policies. The task targets a controlled door configuration, and end-effector-space transfer remains limited by differing robot joint constraints.

B. Manipulation of Constrained Mechanisms - Door Opening

  • The robot must cage a vertical bar handle with its closed two-finger gripper and move a one-degree-of-freedom door to a 60° target configuration.Success is measured as the fraction of the desired door-joint state achieved.Source: B. Manipulation of Constrained… · p. 6
  • Training uses Panda embodiments across five random seeds, with end-effector pose and velocity plus door angle and angular velocity as observations.The reward favors progress toward the target, penalizes forces above 40 N, non-gripper collisions, and joint-limit violations, and terminates episodes for limit violations.Source: B. Manipulation of Constrained… · p. 6
  • End-effector-space policies with medium, variable, or high impedance achieve close to 100% task success and higher rewards than policies in other action spaces.Fixed medium impedance learns faster than variable impedance, but they reach similar final rewards and success rates.Source: B. Manipulation of Constrained… · p. 6
  • Fixed high impedance averages 100% task success but earns lower rewards; low impedance cannot overcome door friction, and joint-space actions fail or perform worse.Joint-velocity actions reach up to 75% task success but often hit joint limits; torque and position actions cannot exert enough force without reaching those limits.Source: B. Manipulation of Constrained… · p. 6
  • End-effector-space policies transfer from Panda to Sawyer more successfully than joint-space policies, which cannot transfer because the robots’ kinematics differ substantially.End-effector-space actions abstract away robot-specific dynamics and kinematics.Source: B. Manipulation of Constrained… · p. 6
  • End-effector-space transfer still drops in performance because policies do not account for the robots’ differing kinematic constraints, including joint limits.Source: B. Manipulation of Constrained… · p. 7

C. Contact-Rich Manipulation - Surface Wiping

VICES performed best among the tested action spaces for simulated wiping and transferred to a real whiteboard-wiping task without retraining. The real-robot policy succeeded in 8 of 10 trials.

C. Contact-Rich Manipulation - Surface Wiping

  • The task required pressing a soft wiping tool against a table with an unknown stain location, randomized initial robot location, and friction sampled from 0.1 to 1.0.Success was measured as the fraction of stain elements wiped; the policy received 48×48×3 RGB images plus end-effector pose and velocity.Source: C. Contact-Rich Manipulation - Surface… · p. 7
  • Reward emphasized wiping, with added reward for clearing all elements and maintaining table contact, while penalizing excessive force, joint-limit violations, and non-tool table collisions.Joint-limit violations and non-tool collisions ended the episode and restarted the task.Source: C. Contact-Rich Manipulation - Surface… · p. 7
  • In simulation, VICES converged faster, achieved higher reward, and wiped the highest ratio of units among the tested action spaces.VICES policies applied a mean force of 28 N, below the robot’s payload; other action spaces applied higher mean force or too little force to wipe.Source: C. Contact-Rich Manipulation - Surface… · p. 7
  • The simulation-trained VICES policy wiped more than 3/4 of the painted line in 8 of 10 real-robot trials without retraining.The robot used no direct force sensing, and it was stopped if payload was exceeded.Source: Figure 6 · p. 7, C. Contact-Rich Manipulation - Surface… · p. 7
  • VICES policies transferred better between Panda and Sawyer embodiments than policies in other action spaces, though performance dropped because the robots generated different forces.The direct-transfer evaluation used policies trained with a Panda embodiment and tested them on Sawyer in simulation.Source: C. Contact-Rich Manipulation - Surface… · p. 7

VI. CONCLUSION

The paper evaluates how action-space choice affects RL across tasks without contact, with kinematic constraints, and with contact-rich manipulation, and presents VICES as an efficient choice. VICES outperforms other action spaces on sample efficiency, energy consumption, and safety, while enabling transfer without fine-tuning.

VI. CONCLUSION

  • The conclusion identifies a lack of principled evaluation of how to choose action spaces for learning control policies in RL.Source: §VI · pp. 7–8
  • VICES outperforms other action-space choices in sample efficiency, energy consumption, and safety, even when contact conditions vary dynamically during the task.The conclusion presents VICES as an efficient action space for RL.Source: §VI · p. 8
  • The evaluation covers RL policies for tasks without contact, with kinematic constraints, and with contact-rich manipulation.Source: §VI · p. 8
  • By subtracting embodiment dynamics, VICES enables policies learned in simulation to transfer to other simulated robots and a real robot without fine-tuning.Source: §VI · p. 8

Related papers

Loading 1906.08880v2…