PaperlayerView exact revision on arXiv

Source-linked AI summary

Smooth Exploration for Robotic Reinforcement Learning

Antonin Raffin, Jens Kober, Freek Stulp

arXiv:2005.05719v2cs.LGcs.ROstat.ML

TL;DR

Deep RL’s stepwise exploration can produce jerky, damaging, and inefficient behavior on real robots. The paper generalizes state-dependent exploration into gSDE, whose sampling interval balances smoothness and performance, enabling direct training on three robots.

  • Problem

    Independent exploration noise in Deep RL causes shaky motion, wear-and-tear, poor exploration, and high variance on real robots.

  • Method

    The paper extends SDE with learned features and periodic noise resampling to create generalized state-dependent exploration, or gSDE.

  • Results

    gSDE achieves competitive continuous-control results and is successfully combined with SAC for direct training on three real robots.

  • Takeaways & Limitations

    The noise sampling interval provides a compromise between smoothness and performance, supporting direct real-robot training without filters or simulation.

  • Takeaways & Limitations

    The original SDE formulation requires state normalization, has linear state-to-noise dependence, and needs variance tuning that grows with state dimension.

Abstract

from arXiv · show

Reinforcement learning (RL) enables robots to learn skills from interactions with the real world. In practice, the unstructured step-based exploration used in Deep RL -- often very successful in simulation -- leads to jerky motion patterns on real robots. Consequences of the resulting shaky behavior are poor exploration, or even damage to the robot. We address these issues by adapting state-dependent exploration (SDE) to current Deep RL algorithms. To enable this adaptation, we propose two extensions to the original SDE, using more general features and re-sampling the noise periodically, which leads to a new exploration method generalized state-dependent exploration (gSDE). We evaluate gSDE both in simulation, on PyBullet continuous control tasks, and directly on three different real robots: a tendon-driven elastic robot, a quadruped and an RC car. The noise sampling interval of gSDE permits to have a compromise between performance and smoothness, which allows training directly on the real robots without loss of performance. The code is available at https://github.com/DLR-RM/stable-baselines3.

1 Introduction

Deep RL commonly uses independently sampled action noise, which can work in simulation but causes shaky, damaging, and inefficient exploration on real robots. The paper adapts SDE into a broader method and evaluates smoothness–performance trade-offs, including direct learning on three robots.

  • 1 Introduction: Independent stepwise Gaussian noise produces shaky trajectories, motor wear, canceled perturbations, and increasing variance on real robots.These drawbacks were observed across three robots, making unstructured-noise Deep RL impractical for David.
  • 1 Introduction: SDE keeps exploration state-dependent and consistent within an episode, making it smoother and less variable than unstructured action noise.The paper motivates SDE as a compromise between action-space and parameter-space exploration.
  • 1 Introduction: The paper argues that SDE has not yet been successfully combined with Deep RL because shaky motion is less noticeable in simulation.This motivates renewed attention to exploration methods suitable for real robots.
  • 1 Introduction: The paper adapts SDE to recent Deep RL algorithms and addresses limitations of its original formulation.The contributions include evaluating smoothness versus performance and examining the effect of the noise sampling interval.
  • 1 Introduction: gSDE is proposed as a generalized exploration method for addressing real-robot problems caused by unstructured Gaussian noise.Its evaluation spans simulation and direct reinforcement learning on a tendon-driven robot, a quadruped, and an RC car.

2 Background

The background defines reinforcement learning and contrasts action-space, parameter-space, and state-dependent exploration. SDE preserves smoother, episode-consistent exploration while remaining compatible with policy-gradient training, but its original formulation has important scope limitations.

  • 2 Background: Reinforcement learning models an agent interacting with an MDP to maximize expected discounted long-term reward over trajectories.The MDP includes state and action spaces, transition dynamics, and rewards; interactions are organized into episodes.
  • 2.1 Exploration in Action or Policy Parameter Space: Continuous-control methods commonly add independently sampled Gaussian noise to the deterministic policy output at every time step.The resulting stochastic policy is centered on the deterministic policy, with diagonal covariance and action-dimension scale parameters.
  • 2.1 Exploration in Action or Policy Parameter Space: Parameter-space exploration samples a perturbation once per episode, producing consistent exploration but becoming difficult to tune as policy dimensionality grows.This approach requires perturbing the policy parameters rather than adding noise directly to actions.
  • 2.2 State-Dependent Exploration: SDE adds noise through a state-dependent exploration function whose parameters are sampled at the beginning of an episode.For a given state during one episode, the resulting action is consistent, making exploration smoother and more consistent than step-based noise.
  • 2.2 State-Dependent Exploration: SDE is distinct from SAC-style noise, where variance may depend on state but noise is still sampled at every step.Its known policy distribution also permits variance adaptation through the likelihood-ratio gradient estimator.
  • 2.2 State-Dependent Exploration: For nonlinear exploration functions, the resulting action distribution is usually unknown, so exact variance derivatives are difficult and may require approximate inference.The paper leaves this extension for future work in favor of simplicity.

3 Generalized State-Dependent Exploration

gSDE extends State-Dependent Exploration to address limitations in the original formulation by periodically resampling noise and using learned policy features. The resulting method generalizes the exploration schedule and integrates with probability-based Deep RL algorithms.

  • Limitations of original SDE: The original SDE formulation has limitations involving fixed episode-wide noise, state-dimension-dependent variance, linear state dependence, and required state normalization.These constraints can limit exploration in long episodes, require problem-specific tuning, restrict the exploration relationship, and make gradient and noise magnitude depend on state scaling.
  • Proposed extensions: gSDE resamples exploration parameters every n steps and uses learned policy features instead of raw states as inputs to the noise function.The chosen features are taken from the layer before the deterministic policy output.
  • Sampling interval: The sampling interval n unifies unstructured exploration at n = 1 with original per-episode SDE at n = episode length.This creates a single framework spanning the two exploration schedules and exposes n as a key design parameter.
  • Policy features: Using policy features makes the state-noise relationship nonlinear, ties variance to network architecture, and supports image inputs without state normalization.The formulation includes original SDE as a special case when raw state features or a linear policy are used.
  • Integration with Deep RL: The resulting approach is called generalized State-Dependent Exploration (gSDE) and can replace the Gaussian distribution in probability-based algorithms such as SAC or PPO.Its analytical log-likelihood permits integration into these algorithms.

4 Experiments

The experiments evaluate gSDE's smoothness–performance trade-off in PyBullet and test direct real-robot training on three platforms. Results show that periodic noise resampling and learned policy features improve exploration, while SAC with gSDE transfers successfully without simulation or filtering.

  • 4.1 Compromise Between Smoothness and Performance: SAC with gSDE-8 achieves better performance with lower training continuity cost, providing the desired compromise for real-robot learning.The comparison covers four PyBullet locomotion tasks and contrasts unstructured, correlated, parameter, and gSDE exploration.
  • 4.2 Comparison to the Original SDE: The sampling interval interpolates between unstructured stepwise exploration and episode-level SDE, but intervals that are too large reduce exploration.For PPO on WALKER2D, the sampling frequency is especially important; SAC permits relatively free interval choices.
  • 4.2 Comparison to the Original SDE: Using latent policy features as the exploration input is usually beneficial, especially for PPO, while requiring less tuning and enabling pixel-based learning.The original formulation uses the state as input, whereas gSDE can use learned features from the policy network.
  • 4.3 Learning to Control a Tendon-Driven Elastic Robot: On the tendon-driven robot, unstructured exploration damaged the tendons, whereas SAC with gSDE achieved evaluation errors below 3° orientation and 3 mm position.The learned gSDE controller was compared with both a low-pass-filtered SAC policy and a model-based controller.
  • 4.3 Learning to Control a Tendon-Driven Elastic Robot: SAC with gSDE also successfully trained an elastic quadruped and an RC car directly on real robots without simulation or filtering.

5 Related Work

Related work addresses continuous-control exploration through correlated noise, parameter-space exploration, population methods, and smoothing losses. These approaches improve smoothness or exploration but can require history, expert knowledge, extra hyperparameters, or simulation-only training.

  • Correlated-noise methods smooth exploration but require temporal history, changing the problem definition.Examples use autoregressive processes or Markov-chain coherence to interpolate between step- and episode-based exploration.
  • Parameter-space exploration avoids some unstructured-noise problems but can require motor primitives, expert knowledge, or mechanisms for high-dimensional spaces.Deep RL adaptations define action-space distances and use layer normalization to handle high-dimensional parameterizations.
  • Population-based methods can be competitive in training time through massive parallelization, but they are sample inefficient and may add computational overhead and hyperparameters.Hybrid evolution-strategy and RL-gradient approaches address sample inefficiency while increasing complexity.
  • Continuity and smoothing losses can produce smoother test-time controllers and reduce energy use, but they do not solve smooth exploration during training.Consequently, the cited approach limits training to simulation rather than directly training on real robots.

6 Conclusion

The paper argues that independently sampled exploration noise prevents direct Deep RL training on real robots because it causes damaging, ineffective motion. It proposes gSDE and reports competitive benchmarks plus successful direct training on three robots.

  • Independent Gaussian exploration creates shaky motion, poor exploration, and potential robot damage, preventing direct application of Deep RL to real-world robots.
  • gSDE extends SDE by resampling noise every n steps and using learned policy features as the exploration-function input.The method is presented as a simple and efficient alternative to unstructured Gaussian exploration.
  • gSDE achieves competitive continuous-control benchmark results, reduces training wear-and-tear, and uses the sampling interval to trade off performance against smoothness.The ablation identifies the sampling interval as the most influential modification, while SAC is robust to its hyperparameter choice.
  • SAC with gSDE is successfully applied directly to a tendon-driven robot, a quadruped, and an RC car.The experiments are positioned as a step toward learning directly on real systems despite safety and duration challenges.

A.1 State Dependent Exploration

State-dependent exploration can be expressed as parameter-space exploration in the linear case and integrated with policy-gradient learning. Its state-dependent noise produces consistent exploration while addressing shortcomings of independently sampled action noise.

  • In the linear case, parameter-space exploration and SDE are equivalent because both perturb the policy parameters multiplying the state.The displayed derivation rewrites the noisy action as a perturbed linear policy.
  • SDE's known policy distribution permits computing the variance gradient and adapting the exploration scale during training.
  • Because SDE can use a likelihood-ratio gradient estimator, it remains compatible with standard policy-gradient methods while reducing shortcomings of unstructured exploration.

A.2 Algorithms

The paper uses state-of-the-art model-free continuous-control algorithms, contrasting fast parallel on-policy methods with more sample-efficient off-policy methods. SAC is selected for robotics experiments because it integrates exploration into its objective and is robust across hyperparameters.

  • A2C: A2C performs synchronous parallel actor-critic updates from n-step rollouts but is fast and not sample efficient.It uses the REINFORCE estimator to compute policy gradients.
  • PPO: PPO limits destabilizing policy changes by clipping the importance-ratio objective and uses workers plus Generalized Advantage Estimation.PPO relaxes TRPO's constrained optimization approach while retaining a trust-region-like effect.
  • TD3: TD3 stabilizes deterministic policy learning with clipped double Q-learning, delayed policy updates, and target-policy smoothing, while relying on external exploration noise.These mechanisms address overestimation, premature policy updates, and value-function overfitting.
  • SAC: SAC maximizes a maximum-entropy objective, learns a squashed Gaussian stochastic policy, and automatically adjusts its entropy coefficient α.Automatic entropy adjustment removes the need to tune this crucial hyperparameter in the latest SAC formulation.
  • Which algorithm for robotics?: SAC and TD3 are preferred for robotics because they are more sample efficient than A2C and PPO, while SAC is easier to tune through its integrated exploration objective.A2C and PPO are easily parallelized and fast in wall-clock time, whereas SAC and TD3 achieve equivalent performance with fewer samples.

A.3 Real Robot Experiments

The real-robot experiments use the same RL algorithm as in simulation while adding observation history, action history, and continuity costs to improve smoothness and handle communication delays. They cover a tendon-driven robot, an elastic quadruped, and an RC car, with learned control comparable to a model-based controller on the reported evaluation trajectory.

  • Common Setup: The real-robot setup augments observations with the previous observation and action, adds a continuity reward cost, and uses the same algorithm as simulation without a filter.These additions target controller smoothness and communication delays that would otherwise break the Markov assumption.
  • Learning to control an elastic neck: An episode is successful when the desired pose is reached within 10 mm position and 5° orientation thresholds; otherwise, it terminates after a 5 s timeout, with 200 steps maximum.
  • Real-robot tasks: Training is conducted directly on an elastic quadruped and an RC car, using onboard-sensor observations for walking and camera input for driving, with robot-issued actions and task-specific rewards.The quadruped training totals around 8 hours, with episodes ending after falls, leaving bounds, or 5 seconds; the car requires less than 30 minutes, ending after intervention or 1 minute.
  • Evaluation: The model-based and learned controllers yield comparable mean position and orientation errors on the evaluation trajectory.The comparison is reported in Table 3 and Figure 6.

A.4 Implementation Details

The implementation combines Stable-Baselines tooling with SAC, PPO, TD3, and A2C configurations, then evaluates learning curves, ablations, and hyperparameter sensitivity across PyBullet tasks. Results emphasize exploration and sampling choices, while hyperparameter tuning and network design materially affect performance.

  • A.4 Implementation Details: The implementation provides SAC with gSDE through Algorithm 1, sampling noise-function parameters and updating the actor, critic, entropy temperature, noise variance, and target networks.The SAC implementation clips the policy mean to [−2, 2] for numerical stability and uses the expln trick only for specific environments where useful.
  • A.6 Ablation Study: Additional Plots: SAC is robust to the initial exploration variance, whereas PPO performance depends strongly on the noise sampling interval.These findings come from the ablation study on the remaining PyBullet tasks.
  • A.6 Ablation Study: Additional Plots: Parallel noise-parameter sampling improves PPO performance on every task and produces a less significant improvement for A2C by enabling more diverse exploration.The authors conclude that parallel workers improve both exploration and final performance.
  • A.7 Hyperparameter Optimization: Hyperparameters are tuned with Optuna using a TPE sampler, median pruning, 500 candidates, and a maximum of 3 · 10^5 HALFCHEETAH time-steps, followed by manual stability adjustments.The tuned SAC gSDE hyperparameters are reused for other exploration types for fair comparison.
  • A.8 Hyperparameters: For SAC, larger networks usually improve PyBullet results, but gains become minimal beyond two layers with 256 units per layer.Time features also substantially affect performance in time-limited experiments, while learning curves compare on- and off-policy algorithms across PyBullet tasks.
Loading 2005.05719v2…