Source-linked AI summary
Regularizing Action Policies for Smooth Control with Reinforcement Learning
Siddharth Mysore, Bassel Mabsout, Renato Mancuso, Kate Saenko
TL;DR
Oscillatory actions in deep RL controllers can undermine practical control through high power use and system wear. CAPS regularizes action policies for temporal and spatial smoothness; on a real quadrotor, the paper reports lower power consumption and smoother policies while training flight-worthy controllers.
Problem
Deep RL controllers can learn oscillatory actions, a practical concern associated with poor control, high power consumption, and system wear.
Source: §I · p. 1Method
CAPS directly regularizes action policies for temporal smoothness between actions and spatial smoothness between similar states.
Source: §I · p. 1Results
Almost 80% reduction in power consumption and 96% improvement in policy smoothness were reported, alongside flight-worthy drone controllers trained with a simple reward structure.
Source: §I · p. 2Takeaways & Limitations
CAPS enabled consistent training of flight-worthy drone controllers with a simple reward structure, and the authors report significantly reduced training time.
Source: §I · p. 2
Abstract
from arXiv · showhide
A critical problem with the practical utility of controllers trained with deep Reinforcement Learning (RL) is the notable lack of smoothness in the actions learned by the RL policies. This trend often presents itself in the form of control signal oscillation and can result in poor control, high power consumption, and undue system wear. We introduce Conditioning for Action Policy Smoothness (CAPS), an effective yet intuitive regularization on action policies, which offers consistent improvement in the smoothness of the learned state-to-action mappings of neural network controllers, reflected in the elimination of high-frequency components in the control signal. Tested on a real system, improvements in controller smoothness on a quadrotor drone resulted in an almost 80% reduction in power consumption while consistently training flight-worthy controllers. Project website: http://ai.bu.edu/caps
I. INTRODUCTION
Deep RL controllers can produce oscillatory control, especially in continuous control and sim-to-real settings, while reward engineering and run-time filters have limitations. CAPS directly regularizes action policies for temporal and spatial smoothness, with reported improvements in drone power consumption and policy smoothness.
I. INTRODUCTION
- Oscillatory RL control can cause visible system oscillations, increased power consumption or overheating, and even hardware failure [5], [7],, [10].The paper identifies this as an issue that had received little attention.Source: §I · p. 1
- In continuous control, varied acceptable outputs and sim-to-real domain gaps can accentuate the problem; this work focuses on low-level quadrotor attitude control.The controllers track a pilot’s desired angular-velocity rates on the drone’s three rotation axes.Source: §I · p. 1
- Because neural-network controllers are black boxes and filters behaved inconsistently in the authors’ experiments, they argue oscillation should be addressed during policy training.Reward engineering conditions behavior indirectly and may not reliably produce the desired trait, motivating direct action-policy optimization.Source: §I · p. 1
- CAPS regularizes policies for temporal smoothness between actions and spatial smoothness from similar states to similar actions.The authors report consistent training of smooth action policies.Source: §I · p. 1
- Almost 80% reduction in power consumption and 96% improvement in policy smoothness were reported for CAPS, which also trained flight-worthy drone controllers with a simple reward structure.The authors also report significantly reduced training time; marginally worse rewards sometimes reflected more conservative responses, without loss of learnability or practical utility.Source: §I · p. 2
II. BACKGROUND
Prior work has often sought smoother RL behavior through reward engineering, but this can require inaccessible state information and relies on learned surrogate value functions. The paper distinguishes its direct regularization of action mappings from reward-based and weight-based approaches, and from work focused on spatial smoothness or noise robustness.
II. BACKGROUND
- Reward-engineering approaches to smoothness may require state information unavailable to the designer, while black-box environments leave the true policy value functions unknown.RL algorithms learn surrogate value functions to evaluate policy behavior.Source: §II · p. 2
- Because surrogate optimization can diverge from the true value function, optimizing it may worsen policy performance as training progresses.The paper uses this concern to motivate direct policy optimization rather than relying on nuanced transfer from rewards.Source: §II · p. 2
- Prior work regularized neural-network weights to improve reward metrics, whereas this paper instead regularizes action-policy mappings, focusing on characteristics of learned behavior.Source: §II · p. 2
- Concurrent work regularized policies for spatial smoothness and noise robustness, but did not report whether learned mappings were smoother; the authors therefore pursue both spatial and temporal smoothing.They anticipated spatial smoothing alone would not suffice for minimizing oscillatory responses, particularly after simulation transfer.Source: §II · p. 2
III. CAPS : CONDITIONING FOR ACTION POLICY SMOOTHNESS
CAPS conditions behavior at the policy network by adding temporal and spatial regularization to policy optimization, rather than modifying reward engineering. Its penalties act on successive actions and on actions for nearby states, with distances and perturbation distributions chosen for the data space.
III. CAPS : CONDITIONING FOR ACTION POLICY SMOOTHNESS
- CAPS conditions action policies for smooth behavior through regularization, addressing policy behavior at the neural network rather than through reward signals.The method focuses on policy optimization without directly changing how value functions are learned.Source: §III · p. 2
- The temporal term penalizes dissimilar actions across a system transition, while the spatial term encourages similar actions for states sampled near the current state.The authors hypothesize spatial smoothing can improve robustness to domain shifts and unmodeled dynamics; they report that Section IV supports this hypothesis.Source: §III · pp. 2–3
- The temporal and spatial penalties approximate local Lipschitz regularization of the policy mappings around regions of interest.The paper notes that exact Lipschitz-constant optimization for neural networks is NP-hard [27].Source: §III · p. 3
- CAPS uses distance measures and state perturbations selected for the data space, with Euclidean actuator-output distances and normally distributed state perturbations in the considered tasks.The perturbation standard deviation is based on expected measurement noise or tolerance.Source: §III · p. 3
- On the toy problem, CAPS-regularized TD3 actions are closer to the ideal linear mapping, whereas vanilla TD3 actions are closer to a binary policy.The caption links the vanilla policy’s binary-like actions to high-frequency oscillation and says other tested RL algorithms had similar responses.Source: Figure 2 · p. 3
IV. EVALUATION
Evaluation spans a toy tracking task, four OpenAI Gym tasks, and quadrotor flight. CAPS improves policy smoothness across these settings, with real-drone tests also examining transfer and practical control outcomes.
Toy Problem
- In the simple one-dimensional toy task, RL learned aggressive oscillatory actions, while CAPS produced behavior closer to the known ideal control.The authors suggest early oscillatory behavior can act as a local minimum that agents fail to escape.Source: §IV · p. 3
Measuring Smoothness
- The FFT-based smoothness measure weights frequency components by amplitude; higher scores indicate more high-frequency signal and typically more expensive actuation.Smoothness scores should not necessarily be compared across different control problems.Source: §IV · p. 4
Gym Benchmarks
- Across four Gym tasks, CAPS policies were smoother than vanilla counterparts; reward hits appeared on Pendulum and LunarLander, while Reacher and Ant gains were dynamics-limited.The evaluation used four RL algorithms and 11 independently trained agents per case.Source: §IV · p. 4
Sim-to-Real Policy Transfer
- CAPS reduced motor amplitude and oscillation while maintaining tracking comparable to vanilla PPO, with peak motor usage below 20%.CAPS-trained agents were flight-worthy; stable real-world flight required roughly 30% baseline thrust to compensate for gravity.Source: Figure 3 · p. 5, §IV · p. 5
- CAPS training completed within 1 million time-steps, reducing data intensity by 90% and wall-time by 8× versus Neuroflight.The comparison uses the same PPO algorithm, network architecture, and deployment pipeline as Neuroflight.Source: §IV · pp. 4–5
- CAPS signals had virtually no high-frequency components and consumed less power than PID, while overall mean tracking error remained under 10 deg/s.The drone results compare motor usage and smoothness against PID, Neuroflight, and temporal- or spatial-only smoothing.Source: Table II · p. 5, §IV · p. 5, Figure 4 · p. 6
- Temporal and spatial smoothing together in CAPS outperformed either alone in training and transfer; temporal smoothing remained noisy after transfer, while spatial smoothing forced control into bands.The authors could not make a thorough comparison with Shen et al. because they lacked access to its code.Source: §IV · p. 5, Figure 4 · p. 6
Why not just use a filter?
- Output filtering can cause neural-network controllers to overshoot or oscillate, and the tested FIR filter caused catastrophic loss of control.The filters were tuned for critical damping on a PID controller, but neural networks may overcompensate for the unfamiliar dynamics.Source: Figure 5 · p. 6
V. CONCLUSION
CAPS regularizes sequential and state-conditioned action smoothness, improving learned-policy smoothness without significant performance loss. In simulation-to-quadrotor transfer, it reduced power consumption by almost 80% while maintaining good input tracking.
V. CONCLUSION
- Almost 80% lower power consumption accompanied simulation-to-real transfer of smoother attitude-control agents, with good tracking of input controls maintained.The conclusion also reports significantly reduced motor heating and wear during flights.Source: §V · p. 6
- CAPS regularizes both sequential actions and the mapping from similar states to similar actions, without requiring reward engineering or training-environment changes.The method can be added to practically any continuous-control RL algorithm.Source: §V · p. 6
- Across learned policies, CAPS improved smoothness without significant negative impact on performance, and the learned smoothness transferred between domains.The conclusion frames these results as support for developing more robust controllers.Source: §V · p. 6