PaperlayerView exact revision on arXiv

Source-linked AI summary

Safety-aware Skill Adaptation for Reinforcement Learning in Dynamic Environments

A K M Nadimul Haque, Sheila Sutjipto, Marc G. Carmichael, Teresa Vidal-Calleja

arXiv:2609.11433v1cs.RO

TL;DR

Safe adaptation of learned robot skills remains challenging in cluttered, dynamic environments because unrestricted exploration can be unsafe and unstable. Dist-GPRL combines local GP-based trajectory updates with covariance-correlated actions, HAP safe-subspace guidance, distance-field feedback, and kinematic regularisation; across dynamic manipulation tasks, it achieves stronger overall task success, collision avoidance, and kinematic retention than evaluated alternatives.

  • Problem

    Safe skill adaptation in cluttered and dynamic environments lacks sufficiently integrated geometry-aware exploration while preserving demonstrated motion characteristics.

  • Method

    Dist-GPRL sequentially adapts overlapping local windows of GP-parameterised via-points using covariance-transformed actions, HAP safe-subspace guidance, distance-field rewards, and kinematic regularisation.

  • Results

    Across two dynamic manipulation tasks, Dist-GPRL achieves the strongest overall balance among task success, collision avoidance, and kinematic retention across evaluated methods.

  • Takeaways & Limitations

    The framework improves learning and reduces collisions while retaining demonstrated kinematic characteristics, including during real-world robot execution.

  • Takeaways & Limitations

    The approach remains limited by GP extrapolation and reliance on accurate sensor information without accounting for sensor noise or distance-field uncertainty.

Abstract

from arXiv · show

Skill adaptation frameworks based on reinforcement learning often require restrictive assumptions to maintain stability, such as fixed observations or tightly controlled exploration schedules. In cluttered and dynamic environments, however, unrestricted exploration can lead to unsafe behaviour and unstable learning, particularly when task-relevant observations lie near obstacles or involve moving objects. In this work, we present Dist-GPRL, a distance-aware and safety-guided reinforcement learning framework for structured robot skill adaptation. Building upon Gaussian Process (GP)-based skill parameterisation, our framework sequentially adapts overlapping local windows of sparse trajectory via-points rather than modifying the complete skill at every policy step. Raw policy outputs are correlated through the GP covariance structure, producing temporally coherent trajectory updates while reducing the action-space and credit-assignment difficulties associated with global trajectory adaptation. Safety is incorporated through two complementary forms of guidance. A safe-subspace prior derived from the Hausdorff Approximation Planner (HAP) biases policy exploration toward feasible regions, while dynamically updated distance field clearance and gradient rewards provide local obstacle awareness. A trajectory-kinematics similarity regulariser further preserves the demonstrated velocity and acceleration characteristics during adaptation. We evaluate the framework on two dynamic object-manipulation tasks in simulation and transfer the learned policy to real-world robot execution. Experimental results demonstrate higher task success, lower collision frequency, and more stable learning than the baselines, while preserving the kinematic characteristics of the demonstrated skill.

I. INTRODUCTION

Dist-GPRL addresses safe adaptation of learned robot skills across changing task configurations by combining structured GP-based updates with geometry-aware exploration. It uses local windowed adaptation, covariance-correlated actions, HAP safe-subspace guidance, distance-field feedback, and kinematic regularisation, with simulation and hardware validation.

  • I. INTRODUCTION: Dist-GPRL combines local windowed GP skill adaptation, covariance-transformed actions, HAP safe-subspace guidance, distance-field feedback, and kinematic regularisation for dynamic environments.The framework extends structured GP-based adaptation while addressing safety, exploration, and motion consistency.
  • I. INTRODUCTION: Sequentially updating overlapping temporal windows reduces the action-space and credit-assignment difficulties of adapting the complete via-point set simultaneously.The local formulation retains the global GP trajectory representation while making policy updates more structured.
  • I. INTRODUCTION: GP covariance transforms independent policy outputs into coherent via-point modifications that follow the demonstrated skill’s temporal structure.This coupling produces temporally coherent trajectory adaptations and improves learning stability.
  • I. INTRODUCTION: HAP-derived safe-subspace guidance supports feasible exploration, while dynamically updated distance-field rewards provide local obstacle awareness as environments change.The two safety mechanisms provide complementary global feasibility and local clearance guidance.
  • I. INTRODUCTION: Experiments evaluate the framework on dynamic manipulation tasks in simulation and real hardware, including ablation studies and safety metrics.The supplied introduction identifies both simulation and hardware validation.

II. RELATED WORK

Related work spans trajectory representations, reinforcement-learning adaptation, safe-subspace exploration, and distance-field control. The paper positions GP-based structured adaptation as expressive and analytically useful, while noting that combining it with dynamic safety guidance remains underexplored.

  • II. RELATED WORK: GPs offer expressive probabilistic skill representations with analytical derivatives and support sparse via-point control of globally smooth trajectory modifications.Derivatives provide direct access to velocity and acceleration profiles, supporting kinematic-aware adaptation.
  • II. RELATED WORK: HAP-based safe subspaces guide exploration toward safe, feasible, and task-optimal poses, but dense construction and updating are computationally expensive and robot-specific.These costs reduce reliability when environments change dynamically.
  • II. RELATED WORK: Distance fields provide continuous obstacle-clearance information and gradients for safer motion, yet their combination with structured demonstrated-skill adaptation remains relatively unexplored.Prior work has used distance fields in navigation and manipulation, including hierarchical or constrained DRL systems.
  • II. RELATED WORK: GP conditioning on sparse via-points uses kernel temporal correlations to produce smooth trajectory updates from a reduced set of control variables.The covariance structure also supports analytical trajectory derivatives.

B. Soft Actor-Critic

This section introduces the safety-aware skill-adaptation pipeline and its supporting control representations. SAC supplies stochastic off-policy learning for continuous actions, while Euclidean distance fields encode clearance and locally safer directions.

  • B. Soft Actor-Critic: Soft Actor-Critic is a stochastic, off-policy reinforcement-learning algorithm for continuous action spaces that maximises expected return with entropy regularisation.Entropy regularisation encourages exploration and improves stability in high-dimensional continuous control.
  • C. Euclidean Distance Fields: An Euclidean Distance Field represents the environment as a scalar clearance field around obstacle surfaces.The field is non-negative in free space and approaches zero at contact.
  • C. Euclidean Distance Fields: The distance-field gradient points toward maximal clearance increase, providing a locally optimal obstacle-avoidance direction for planning and control.Smoothness and differentiability make distance fields useful in motion planning and optimisation-based control.
  • B. Soft Actor-Critic: The proposed safety-aware skill-adaptation pipeline is presented through an overview in Fig. 2.The figure provides the high-level formulation context for the method.

A. Problem Formulation

The formulation adapts a GP-parameterised demonstrated skill by modifying sparse via-points for a new task configuration while retaining kinematics and safety. Dist-GPRL restricts each RL step to a phase-selected local window and correlates its corrective updates through the GP prior.

  • A. Problem Formulation: The demonstrated skill is represented by six separate GPs using sparse position and orientation via-points.The via-point set contains position and orientation controls for the trajectory.
  • A. Problem Formulation: The adaptation objective preserves the demonstrated velocity profile while requiring the adapted trajectory to remain collision-free.Safety is expressed through positive distance from obstacles using a differentiable Euclidean Distance Field.
  • A. Problem Formulation: Each RL step adapts a contiguous trajectory segment by updating a phase-selected local window of L via-points.The local update is formulated within an MDP, with the adapted segment defined by the active window.
  • A. Problem Formulation: The policy observes windowed via-points together with obstacle clearances and task vectors, then outputs a corrective action for the active window.The task vectors encode distances from the end-effector to the manipulated object and goal.
  • A. Problem Formulation: The GP covariance transforms raw window actions into correlated via-point updates, preserving the global trajectory representation while structuring local modifications.The principal covariance submatrix indexed by the active window produces the correlated corrective actions.

B. Action Biasing with Cosine Similarity and HAP Prior

The actor objective combines return maximisation with two priors: kinematic similarity to the demonstration and proximity to HAP-derived safe Cartesian regions. Together, these terms bias via-point adaptation toward motion-consistent and feasible trajectories.

  • B. Action Biasing with Cosine Similarity and HAP Prior: The actor objective jointly promotes return maximisation, kinematic retention, and safety-aware exploration in via-point space.It augments the SAC actor objective with a cosine similarity loss and a HAP-based safe-subspace prior.
  • B. Action Biasing with Cosine Similarity and HAP Prior: The cosine similarity loss compares demonstrated and adapted velocity and acceleration profiles obtained through GP derivative maps.The loss constrains motion directionality and smoothness during via-point adaptation.
  • B. Action Biasing with Cosine Similarity and HAP Prior: The HAP prior penalises sampled adapted positions according to their soft-min distance from the traversable safe Cartesian subspace.The distance uses nearest neighbours in the HAP-derived safe-position set.

C. Safety-aware Policy Exploration

Distance-field guidance adds local obstacle awareness to policy exploration through clearance penalties and gradient alignment. The combined reward encourages task completion, maintained clearance, and safety-directed exploration, but does not formally guarantee safety.

  • C. Safety-aware Policy Exploration: The distance-based reward discourages unsafe adaptation by penalising proximity to obstacles through a dynamically evaluated Euclidean Distance Field.The hyperbolic penalty increases rapidly near contact and provides a stronger safety gradient than linear shaping.
  • C. Safety-aware Policy Exploration: EDF gradients at active via-points define locally safer directions, and gradient-alignment rewards favour corrective actions that follow those directions.A higher gradient-alignment value reflects actions aligned with obstacle-avoidance directions.
  • C. Safety-aware Policy Exploration: The complete reward combines the task-completion reward with the gradient-aligned exploration reward while also incorporating clearance-based safety shaping.The formulation encourages task completion, clearance maintenance, and safety-directed exploration during policy learning.
  • C. Safety-aware Policy Exploration: The distance penalty and gradient-aligned reward encourage collision-free behaviour but do not provide formal safety guarantees.This limitation applies despite their combined safety guidance during policy learning.

D. Training Details

Training initialises a GP-conditioned demonstration and updates it online through overlapping local windows during each rollout. The executed adapted segments generate safety-shaped transitions used to update the SAC policy, after which the best policy queries the GP for the final trajectory.

  • D. Training Details: At episode start, the demonstrated trajectory is GP-parameterised and conditioned on the current task configuration to initialise the via-points.The Euclidean Distance Field is initialised from the workspace model before being updated online.
  • D. Training Details: During each rollout, the active window is selected from the phase variable, and the policy uses windowed via-points, EDF clearances, and task vectors to propose corrections.The EDF is updated according to the moving obstacle state.
  • D. Training Details: GP conditioning converts each corrective action into a correlated local via-point update that defines the executed adapted trajectory segment.One transition is stored per time step after segment execution.
  • D. Training Details: SAC parameters are updated using task and safety rewards, and the GP queried by the best policy produces the final adapted trajectory.The training loop stores transitions and updates the actor and critics through the regularised objective.

E. Hyperparameter Selection

The implementation uses overlapping local windows and empirically selected regularisation weights to balance adaptation flexibility, smoothness, safety guidance, and task completion.

  • E. Hyperparameter Selection: Three overlapping windows of L = 5 six-DoF via-points create a 30-dimensional action per reinforcement-learning step.The GP parameterisation contains 15 via-points per axis, or 90 across six axes; one-point overlap maintains continuity between adjacent segments.
  • E. Hyperparameter Selection: Smaller windows limit adaptation flexibility, whereas larger windows approach global adaptation and can reduce learning stability.
  • E. Hyperparameter Selection: The covariance transformation uses λ = 10−2, while λhap = 0.2, λsim = 0.5, and λga = 0.2 were selected empirically and fixed across experiments.Increasing safe-subspace or similarity regularisation strengthens guidance or kinematic retention, but excessive regularisation can restrict task adaptation.

A. Experimental Setup

The evaluation uses two dynamic manipulation tasks in PyBullet, compares Dist-GPRL with five structured-adaptation baselines, and transfers the bar-manipulation policy to hardware.

  • A. Experimental Setup: Simulation evaluates dynamic cube pushing and dynamic bar manipulation with varied object and obstacle configurations.Cube pushing varies the object pose by up to 20 cm and the obstacle start pose by up to 10 cm; bar poses vary by up to 20 cm.
  • A. Experimental Setup: The dynamic bar-manipulation policy is transferred from simulation to hardware in one shot using an updating distance field and ArUco-based object localisation.
  • A. Experimental Setup: The benchmark includes GPRL, ProMP-RRL, Dist-GPRL-Global, GPRL-D, and GPRL-H, isolating global adaptation, windowing, covariance transformation, HAP guidance, and distance-based rewards.GPRL directly updates all via-points; the other variants remove or retain selected components of the proposed framework.
  • A. Experimental Setup: All compared frameworks use the same SAC configuration.

C. Simulation Results

Across dynamic manipulation simulations and hardware evaluation, the complete windowed framework combines high task success with low collision frequency, while its HAP and covariance components improve learning behaviour.

  • C. Simulation Results: Global via-point adaptation improved over baseline GPRL but still produced collisions and manipulation failures, whereas windowed variants performed better.
  • C. Simulation Results: 89% success in dynamic cube pushing and 98% in dynamic bar manipulation were achieved by the proposed framework, with 5% and 1% collision rates, respectively.The method also maintained competitive minimum clearance across both tasks.
  • C. Simulation Results: The HAP subspace consistently increased success and episode returns while reducing collision frequency across three seeds.The corresponding figure reports its effect on learning across training.
  • C. Simulation Results: Covariance-based action transformation improved learning stability and sample efficiency, producing better performance earlier than direct action updates.The figure reports higher final task success, lower collision frequency, and higher episodic returns.
  • C. Simulation Results: Runtime measurements yielded approximately 19.3 ms of computational adaptation time per episode, supporting real-time adaptation feasibility.
  • D. Hardware Results: The complete framework achieved the highest hardware success rate while maintaining low collision frequency in real-world bar manipulation.

E. Discussion

Dist-GPRL combines local GP-based adaptation with HAP-subspace and distance-field guidance to improve learning and safety in dynamic manipulation. Across simulation and real-world execution, it achieved the strongest overall balance of task success, collision avoidance, and kinematic retention, while GP extrapolation and sensing remain limitations.

  • Full via-point adaptation caused unstable learning, low success, and high collisions because its high-dimensional actions created a credit-assignment problem.
  • Local updates improved credit assignment, while HAP guidance and distance-based rewards supported safer exploration, faster convergence, and fewer collisions.
  • Covariance-coupled smoothing kept via-points moving coherently within each local window and preserved the demonstrated geometric and kinematic structure.
  • GP extrapolation can produce erratic late-trajectory movements, while sensor noise, distance-field uncertainty, and the learned collision-avoidance policy limit formal safety guarantees.
  • Dist-GPRL achieved the strongest overall balance of task success, collision avoidance, and kinematic retention across two dynamic manipulation tasks, including real-world execution.
Loading 2609.11433v1…