Source-linked AI summary
Variable Impedance Control in End-Effector Space: An Action Space for Reinforcement Learning in Contact-Rich Tasks
Roberto Martín-Martín, Michelle A. Lee, Rachel Gardner, Silvio Savarese, Jeannette Bohg, Animesh Garg
TL;DR
The paper addresses limited study of action-space choice in reinforcement learning for contact-rich manipulation. It compares action spaces across free-space, constrained, and continuous-contact tasks and finds that VICES improves learning efficiency, physical efficiency, safety, and transfer across robot embodiments.
Problem
Reinforcement-learning studies have rarely justified action-space choice, despite evidence that task-conforming interfaces can simplify learning and support robust manipulation.
Method
The paper compares joint and end-effector action spaces, including fixed and variable impedance, across Path Following, Door Opening, and Surface Wiping.
Results
VICES outperforms other action-space choices across the evaluated tasks on sample efficiency, energy consumption, safety, and transfer to other simulated robots and a real robot.
Takeaways & Limitations
Variable impedance in end-effector space provides an action interface that supports efficient learning and cross-embodiment policy transfer without fine tuning.
Abstract
from arXiv · showhide
Reinforcement Learning (RL) of contact-rich manipulation tasks has yielded impressive results in recent years. While many studies in RL focus on varying the observation space or reward model, few efforts focused on the choice of action space (e.g. joint or end-effector space, position, velocity, etc.). However, studies in robot motion control indicate that choosing an action space that conforms to the characteristics of the task can simplify exploration and improve robustness to disturbances. This paper studies the effect of different action spaces in deep RL and advocates for Variable Impedance Control in End-effector Space (VICES) as an advantageous action space for constrained and contact-rich tasks. We evaluate multiple action spaces on three prototypical manipulation tasks: Path Following (task with no contact), Door Opening (task with kinematic constraints), and Surface Wiping (task with continuous contact). We show that VICES improves sample efficiency, maintains low energy consumption, and ensures safety across all three experimental setups. Further, RL policies learned with VICES can transfer across different robot models in simulation, and from simulation to real for the same robot. Further information is available at https://stanfordvl.github.io/vices.
I. INTRODUCTION
The paper frames action-space selection as a task-dependent design choice for reinforcement learning, especially in contact-rich manipulation. It introduces VICES, which combines end-effector motion and impedance control to support learning, efficiency, safety, and transfer.
- Motivation: Different manipulation tasks may benefit from different action spaces because their required contacts, trajectories, velocities, and moments differ.The paper distinguishes free-space, constrained, and contact-interaction requirements.
- Control Interface: Robot control separates reference generation from reference tracking through outer and inner feedback loops.The outer map g(o) produces abstract reference signals, while f(a) converts them into actuator commands.
- Research Gap: Prior reinforcement-learning studies often vary observations or use joint and task-space commands without fully justifying the action space.The action space determines the quantity around which the inner control loop is closed.
- Research Gap: Contact-rich tasks can require direction-dependent and time-varying stiffness or compliance, making manual task-constraint specification difficult to scale.Surface wiping exemplifies the need to control forces in some directions and motion in others.
- Study Design: The paper empirically compares joint position, velocity, torque, and impedance spaces with fixed and variable end-effector impedance across three task types.The tasks are Path Following, Door Opening, and Surface Wiping.
- Contribution: VICES is proposed for Deep RL because it targets motion and impedance in end-effector space while improving sample efficiency, energy efficiency, force reduction, and robot transfer.The transfer claim relies on dynamically consistent operational-space control.
II. RELATED WORK
Related work develops compliant and variable-impedance control, learning from demonstrations, and action-space comparisons, but leaves open learning variable task-space behavior from scratch for contact-rich manipulation. The paper positions VICES as addressing this gap while studying transfer across robots and to the real world.
- Robot Motion Control: Compliant robot control helps accommodate environmental uncertainty, while some tasks require direct regulation of contact interactions.Examples include unknown surfaces, mechanism constraints, and limiting wiping forces.
- Robot Motion Control: Existing variable-impedance methods often schedule gains for a given kinematic trajectory, whereas VICES predicts end-effector displacement and impedance gains from observations.The proposed approach does not assume that the reference trajectory is already supplied.
- Learning from Demonstrations: Demonstration-based methods may reflect trajectory variability rather than environmental constraints and are limited to tasks for which expert demonstrations are feasible.The cited approach is especially constrained to kinematic tasks with changing precision requirements.
- Related Learning Methods: Prior variable-impedance work optimized energy in joint space, which the paper associates with limited transferability across robots and less task-space trajectory optimality.This comparison concerns Buchli et al.'s path-integral approach.
- Related Learning Methods: Other work learned task-space impedance from demonstrations in simulation and one planar real-world task, including only one stiffness parameter.Related force-profile refinement methods also start from given trajectories.
- Related Learning Methods: The paper differs from earlier approaches by exploring learning from scratch rather than bootstrapping with initial demonstrations.It also relates to studies comparing torque and feedback-control action representations.
- Action-Space Studies: The study extends action-representation analysis from simulated bipedal locomotion to contact-rich robot manipulation with preliminary real-world transfer.The cited locomotion work focused on imitation learning.
III. REINFORCEMENT LEARNING
The paper formulates reinforcement learning as policy optimization in a continuous Markov decision process, with policies selecting actions from observations to maximize expected future reward. It uses policy-gradient methods and evaluates action-space sensitivity with PPO.
- RL Formulation: Reinforcement learning seeks a policy that selects actions from current observations to maximize expected reward from environment interactions.The formulation uses a discrete-time continuous Markov decision process.
- RL Formulation: The Markov decision process includes continuous states and actions, transition dynamics, rewards, a discount factor, and an initial-state distribution.A probabilistic policy represents the density of actions given a state.
- Observation-Based Policies: When the state is unobservable, the policy is conditioned on observations and updated through interactions producing rewards and new observations.The policy is parameterized by θ in the described learning setup.
- Policy Optimization: Policy-gradient optimization adjusts parameters using the policy-gradient theorem and the action-value function associated with the current policy.The supplied theorem fragment connects parameter updates to action values.
- Algorithm Choice: The evaluation uses Proximal Policy Optimization, while sensitivity of different algorithms to action-space choice is deferred to future work.Other listed policy-gradient approaches include TRPO, DDPG, and A2C.
IV. ACTION SPACES IN RL FOR ROBOT MANIPULATION
The paper compares action spaces that map policy outputs into joint torques, emphasizing end-effector references with impedance parameters for contact-rich manipulation. VICES lets policies jointly predict end-effector motion and dynamically adapt impedance gains across task axes.
- Compared action spaces: Common action spaces include joint torques, velocities, positions, and end-effector pose commands, with each using a corresponding controller.Joint velocity and position policies use feedback controllers; end-effector pose policies use impedance-based operational-space control.
- Controller framework: Policies map abstract actions into joint torque commands through analytic controllers parameterized by control gains.The policy supplies reference signals, while the controller uses the robot state and parameters to produce actuation commands.
- Task-dependent action spaces: In training curves, variable impedance is unnecessary for high reward in path following and door opening, but improves surface-wiping reward by adapting applied force to task constraints.The reported comparison spans free-space, kinematically constrained, and contact-rich tasks.
- End-effector control: End-effector control converts pose errors and velocities into desired accelerations, then maps end-effector wrenches to joint torques using the Jacobian.The operational-space formulation uses robot kinematic and dynamic models to generate torque commands.
- Variable Impedance End-Effector Space (VICES): VICES augments end-effector actions with impedance gains, allowing the policy to predict motion references and adapt stiffness and compliance along six axes.The gains can vary with the phase of the task rather than being manually fixed.
V. EXPERIMENTS
The experiments compare action spaces across tasks ranging from free motion to highly constrained contact-rich manipulation. They evaluate learning, physical efficiency and effort, robot-to-robot transfer, and sim-to-real transfer using policies operating at low frequency and controllers issuing high-frequency torque commands.
- Experimental domains: Experiments cover free-space path following, articulated-door manipulation, and contact-rich surface wiping.These tasks span different levels of task-space constraint.
- Evaluation criteria: The evaluation asks whether action spaces support model-free RL, physical efficiency, physical effort, transfer across robots, and sim-to-real transfer.Metrics include samples to success or convergence, energy consumption, contact wrenches, cross-robot task success, and real-robot performance.
- Control and execution: Policies output actions at 20 Hz, while controllers update joint torque commands at 500 Hz and linearly interpolate between consecutive policy commands.The interpolation is used to produce smoother commands and motions.
A. Free-Space Motion - Path Following
The Path Following task evaluates whether policies can pass through four fixed via-points in free space and optimize energy after learning task completion. VICES adapts impedance during motion, while variable-impedance joint-space policies reduce energy more strongly partly because their initial solutions consume more energy.
- Task setup: The agent follows a fixed free-space trajectory through four via-points, with success measured as the fraction passed.The task can be solved kinematically without impedance control.
- Training objectives: Training uses task-completion rewards first, then adds an energy penalty to optimize consumption without reducing via-point performance.The energy estimate assumes torque scales with electric power and integrates consumption over time.
- VICES behavior: VICES changes stiffness after each via-point and combines it with the appropriate desired displacement to reach the next point without overshooting.The policy uses impedance differently across trajectory portions, increasing stiffness after checking a via-point.
- Energy efficiency: VICES was the only end-effector-space policy that consistently improved energy efficiency while maintaining task performance.This comparison concerns optimization between the first and second experiment phases.
- Energy efficiency: Joint-space variable-impedance policies reduced energy more than end-effector-space variable-impedance policies, reflecting higher initial energy use and greater optimization headroom.The authors describe the difference in absolute optimization as an artifact of different task-completion speeds and initial energy consumption.
- Transferability: In zero-shot simulation transfer from Panda to Sawyer, fixed- and variable-impedance end-effector policies transferred, whereas joint-space policies did not.The evaluation covers Path Following, door opening, and surface wiping, with transfer performance becoming harder as environmental contact increases.
B. Manipulation of Constrained Mechanisms - Door Opening
The Door Opening task requires manipulating a constrained one-degree-of-freedom mechanism to 60° while limiting force, collisions, and joint-limit violations. End-effector-space policies achieved near-complete task success, but transfer remained limited by embodiment-specific kinematic constraints.
- Task setup: The robot manipulates a one-degree-of-freedom door mechanism to a target configuration using a gripper that cages the handle while permitting rotation.The target is θ_goal = 60°.
- Safety and reward: The reward encourages approaching the target, penalizes excessive forces, collisions, and joint-limit violations, and terminates episodes when joint limits are exceeded.Excessive force is defined relative to the robot’s 40 N physical payload.
- Task performance: End-effector policies with medium, variable, or high impedance reached close to 100% task success and higher rewards than other action spaces.Fixed medium impedance learned faster initially, while variable impedance reached similar final rewards and success after learning a suitable impedance.
- Task performance: Fixed high impedance also averaged 100% success, but received lower rewards because it exerted higher penalized forces.Fixed low impedance could not overcome door friction, while joint-velocity policies reached at most 75% success and often hit joint limits.
- Transferability: End-effector policies transferred more successfully than joint-space policies, because they abstracted away robot-specific dynamics and kinematics.Door transfer is harder because different embodiments impose different task-space and joint-limit constraints.
- Transferability: Transfer still shows a performance drop because end-effector policies do not account for the robots’ different kinematic constraints, including joint limits.The limitation concerns cross-embodiment transfer rather than the original-robot task result.
C. Contact-Rich Manipulation - Surface Wiping
The surface-wiping experiment tests whether policies learned with VICES transfer between robot embodiments and from simulation to a real robot. VICES policies transfer better across simulated robots and achieved successful real-world wiping in most trials.
- Setup: The experiment wipes a table with an unknown surface location using a soft-material wiping tool that generates penetration-dependent contact forces.The simulator models the tool as soft, with contact forces increasing proportionally to penetration.
- Transfer between robots: VICES policies transfer better between Panda and Sawyer robots than policies using other action spaces.The transfer advantage is attributed to policy independence from robot embodiment, although differing generated forces cause a significant performance drop.
- Simulation-to-real transfer: The simulation-trained policy was applied to the real robot without retraining for whiteboard wiping.Real images were converted into simulated images, and the robot was stopped when the payload was exceeded; no direct force sensing was used.
- Simulation-to-real transfer: 8 of 10 trials successfully wiped more than 3/4 of the painted line.One failure triggered the safety mechanism after abrupt motion, while another left the mark partially unwiped.
VI. CONCLUSION
The paper addresses the lack of principled evaluation for action-space choice in reinforcement-learning control policies. It evaluates VICES across manipulation settings and reports improved efficiency, energy consumption, safety, and transfer without fine tuning.
- VI. CONCLUSION: The study evaluates action-space choice for RL policies in tasks without contact, with kinematic constraints, and with contact-rich manipulation.This conclusion frames the evaluation across three task regimes.
- VI. CONCLUSION: VICES outperforms other action spaces on sample efficiency, energy consumption, and safety, including tasks with dynamically variable contact conditions.The conclusion presents this as an empirical result of the paper’s evaluation.
- VI. CONCLUSION: VICES enables policies learned in simulation to transfer to other simulated robots and a real robot without fine tuning.The conclusion attributes this transfer to subtraction of the embodiment’s dynamic effects.