PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning to Utilize Shaping Rewards: A New Approach of Reward Shaping

Yujing Hu, Weixun Wang, Hangtian Jia, Yixiang Wang, Yingfeng Chen, Jianye Hao, Feng Wu, Changjie Fan

arXiv:2011.02669v1cs.LGcs.AI

TL;DR

The paper addresses the unreliability of fully using human-designed shaping rewards by learning how to utilize a given shaping function adaptively. It formulates this as bi-level optimization, derives gradients, and proposes three learning algorithms. Experiments in cartpole and MuJoCo show that the methods exploit beneficial rewards while ignoring or transforming unbeneficial ones.

  • Problem

    Existing reward-shaping methods generally assume human-transformed shaping rewards are fully helpful, although numeric reward design can be subjective, biased, or unreliable.

  • Method

    BiPaRS alternates lower-level policy optimization using modified rewards with upper-level optimization of a parameterized shaping weight function for true-reward maximization.

  • Results

    The cartpole and MuJoCo experiments show that the algorithms exploit beneficial shaping rewards, ignore unbeneficial rewards, and can transform harmful rewards into beneficial ones.

  • Takeaways & Limitations

    Adaptive shaping can reduce reliance on repeatedly tuning imperfect reward values and provide an indication of designed reward quality.

  • Takeaways & Limitations

    The gradient theorem assumes that the policy gradient with respect to shaping parameters exists, while that gradient is not directly computable and must be approximated.

Abstract

from arXiv · show

Reward shaping is an effective technique for incorporating domain knowledge into reinforcement learning (RL). Existing approaches such as potential-based reward shaping normally make full use of a given shaping reward function. However, since the transformation of human knowledge into numeric reward values is often imperfect due to reasons such as human cognitive bias, completely utilizing the shaping reward function may fail to improve the performance of RL algorithms. In this paper, we consider the problem of adaptively utilizing a given shaping reward function. We formulate the utilization of shaping rewards as a bi-level optimization problem, where the lower level is to optimize policy using the shaping rewards and the upper level is to optimize a parameterized shaping weight function for true reward maximization. We formally derive the gradient of the expected true reward with respect to the shaping weight function parameters and accordingly propose three learning algorithms based on different assumptions. Experiments in sparse-reward cartpole and MuJoCo environments show that our algorithms can fully exploit beneficial shaping rewards, and meanwhile ignore unbeneficial shaping rewards or even transform them into beneficial ones.

1 Introduction

Reward shaping adds domain-knowledge rewards to improve RL learning, but imperfect and biased human-designed values make full utilization unreliable. This paper instead learns how to adaptively use a given shaping function.

  • Motivation: Reward shaping combines original rewards with additional rewards derived from domain knowledge to guide RL toward faster and better learning.It has been applied from early robot and bicycle-learning tasks to complex environments such as Doom and Dota 2.
  • Limitations of Existing Approaches: PBRS guarantees policy invariance when the shaping reward is defined as a difference of potential values.Existing approaches mainly focus on generating shaping rewards and generally assume those rewards are helpful.
  • Limitations of Existing Approaches: Human conversion of rules or other knowledge into numeric rewards is subjective, cognitively biased, and potentially unreliable.Reward designers may need to try many reward versions, especially without expert knowledge of the task.
  • Paper Approach: The paper formulates adaptive shaping-reward utilization as bi-level optimization and derives gradients for learning a parameterized shaping weight function.The lower level optimizes policy using shaping rewards, while the upper level maximizes expected accumulative true reward.
  • Findings: Experiments in cartpole and MuJoCo show that the algorithms identify shaping-reward quality, exploit beneficial rewards, and ignore or transform harmful ones.The authors report that some tests transform harmful shaping rewards into beneficial ones.

2 Background

Reward shaping modifies an RL reward function with domain knowledge, while background RL is modeled through policy gradients in a Markov decision process. Prior work ranges from potential-based guarantees to automatic and specialized shaping methods.

  • RL Framework: An MDP is represented by states, actions, transitions, rewards, initial-state probabilities, and a discount rate, with a parameterized policy πθ.The policy-gradient objective maximizes expected accumulative reward.
  • Reward Shaping: Reward shaping adds a shaping function F to the original reward, producing the modified reward r′ = r + F.The shaping function incorporates domain knowledge into the reward signal.
  • Reward Shaping: PBRS defines F as γΦ(s′) − Φ(s) and guarantees policy invariance, while variants extend potentials to state-action or time-dependent representations.These variants include potential-based advice and dynamic PBRS.
  • Related Work: Other reward-shaping research includes theoretical PBRS analysis, automatic shaping, multi-agent shaping, belief and ethics shaping, and meta-learning approaches.ASR optimizes weights for multiple auxiliary shaping rewards but assumes all shaping rewards are useful.

3 Parameterized Reward Shaping

Parameterized reward shaping introduces state-action-dependent weights so beneficial and unbeneficial shaping rewards can be treated differently. BiPaRS alternates policy optimization under modified rewards with weight optimization for true-reward performance.

  • Parameterized Reward Shaping: The modified reward is formed by combining the true reward with the weighted shaping reward, allowing different shaping contributions to be utilized differently.The stated goal is to distinguish beneficial from unbeneficial rewards supplied by f.
  • Parameterized Reward Shaping: BiPaRS introduces a shaping weight function zφ that assigns parameterized weights to shaping rewards for each state-action pair.With multiple shaping functions, zφ can be a weight vector whose elements correspond to individual shaping rewards.
  • Bi-level Optimization: The lower-level objective optimizes policy πθ according to the modified reward ˜r.For fixed zφ, this is a standard policy-optimization problem using the modified MDP.
  • Bi-level Optimization: The upper-level objective optimizes zφ by evaluating the true reward achieved by the policy produced through shaping.Thus, policy and shaping-weight optimization form a bi-level problem called BiPaRS.
  • Gradient Computation: A theorem gives the gradient of true-reward performance with respect to φ, but computing it requires the policy gradient with respect to φ.The paper assumes this gradient exists because the policy is an implicit result of applying zφ.

4 Gradient Approximation

The paper approximates the unavailable policy-to-weight gradient through explicit policy inputs, meta-gradients, and incremental history tracking. These methods trade gradient accuracy against computational complexity.

  • 4.1 Explicit Mapping: Explicit mapping feeds shaping weights into a hyper-policy, creating a direct mapping from zφ to πθ for chain-rule gradient computation.The resulting method is called explicit mapping (EM).
  • 4.1 Explicit Mapping: The hyper-policy extends the state space with shaping weights while preserving equivalent rewards, transitions, and initial-state probabilities.For a fixed shaping weight function, the original and extended modified MDPs are described as equivalent.
  • Meta-Gradient Approximation: The second method approximates the meta-gradient ∇φθ so that the policy gradient with respect to shaping parameters can be computed from ∇θπθ.A batch of N samples is used for the policy-parameter update, with α as the learning rate.
  • Meta-Gradient Approximation: The meta-gradient computation can use an unbiased state-action-value estimate such as a Monte Carlo return from sampled trajectories.The trajectory construction starts from each sampled state-action pair.
  • 4.3 Incremental Meta-Gradient Learning: IMGL treats policy and shaping-weight parameters as related across prior updates and computes their gradient relation incrementally.It initializes the policy-parameter gradient with respect to φ to zero and updates it iteratively; the methods differ in accuracy-complexity tradeoffs.

5 Experiments

Across cartpole and MuJoCo experiments, BiPaRS methods adapt shaping weights to exploit helpful rewards and mitigate harmful or state-dependent rewards.

  • 5.1 Sparse-Reward Cartpole: In discrete-action cartpole, shaping methods raise PPO’s converged ASPE from 170 to nearly 200.All methods maintain positive shaping weights; BiPaRS-MGL and BiPaRS-IMGL increase average weights from 1.0 to 1.5 and 1.8.
  • 5.2 MuJoCo: Across five MuJoCo tasks, BiPaRS methods adapt to the shaping reward, with BiPaRS-MGL and BiPaRS-IMGL outperforming other methods in Hopper-v2.Torque curves track reward curves, and the methods can reverse an initially low-torque strategy when it proves unbeneficial.
  • 5.3 Adaptability Test: With harmful shaping rewards, BiPaRS methods reduce their influence and can achieve ASPE values near or above PPO’s reference performance.BiPaRS-EM reaches 130, while BiPaRS-MGL and BiPaRS-IMGL approach 180 in continuous-action cartpole; negative weights indicate reward transformation.
  • 5.3 Adaptability Test: Reloaded shaping weights let BiPaRS learn faster and better, while random shaping rewards still yield ASPE values of about 170 and 150 in continuous- and discrete-action cartpole.The reported values apply to BiPaRS-EM in continuous-action cartpole and all three BiPaRS methods in discrete-action cartpole.
  • 5.4 Learning State-Dependent Shaping Weights: When helpful and harmful rewards differ by state, BiPaRS methods outperform the single-weight baseline and learn state-dependent shaping weights.BiPaRS-EM’s heat map visualizes weights across 100 states while fixing car and pole velocities.

6 Conclusions

The paper proposes BiPaRS, a bi-level approach that adaptively learns how to use a given shaping reward function.

  • 6 Conclusions: BiPaRS formulates shaping-reward utilization as bi-level optimization and provides gradient results with three learning algorithms.Experiments show the methods exploit beneficial rewards while ignoring or transforming unbeneficial rewards.

Broader Impact

The paper targets difficult reward design by adaptively using imperfect numeric representations of prior knowledge in reinforcement learning.

  • Broader Impact: The adaptive approach aims to reduce repeated reward tuning, while learned shaping weights indicate the quality of designed rewards.The paper presents this as a way to address cognitive error when qualitative knowledge is converted into numeric learning signals.

A.1 Complexity Analysis

BiPaRS alternates policy and shaping-weight optimization, with EM, MGL, and IMGL providing gradient approximations that trade computational cost against modeling detail.

  • Explicit Mapping: EM explicitly maps shaping weights into the policy input and has computational complexity O(m).The method approximates ∇φ log πθ through the shaping-weight input.
  • Meta-Gradient Learning: MGL uses policy parameters before and after one low-level update to compute a meta-gradient for shaping-weight optimization.Its complexity can be reduced from O(Nnm) to O(N(n + m)).
  • Incremental Meta-Gradient Learning: IMGL generalizes MGL by treating policy parameters as dependent on shaping-weight parameters across prior optimization rounds.This incorporates additional historical dependence into the gradient approximation.
  • Complexity Trade-off: IMGL is more computationally expensive because its Hessian term has O(Nn3) complexity.The paper notes OPG approximations, smaller policy models, or omitting the second-order term as possible reductions.
  • Algorithm: BiPaRS iteratively optimizes policy parameters θ and shaping-weight parameters φ through alternating updates.The general algorithm instantiates EM, MGL, or IMGL for gradient approximation.

B Theorem Proof

The proof derives the gradient of the expected true reward with respect to shaping-weight parameters under continuity and bounded-reward assumptions, using standard differentiation and integration arguments.

  • Assumptions: The derivation assumes continuity of the MDP components and policy, plus an upper bound on the original reward.These are stated as Assumptions 1 and 2.
  • Theorem: Theorem 2 gives the gradient of J(zφ) with respect to φ for a stochastic policy in the upper-level BiPaRS problem.The gradient is expressed using the state-action value function Qπ in the original MDP.
  • Proof Strategy: The proof expands the Bellman equation, exchanges derivative and integral orders, applies Fubini’s theorem, and uses the definition of state visitation probability.The derivative–integral exchange requires Assumption 1, while the Fubini step requires the stated assumptions.
  • Proof Strategy: The log-derivative trick yields the final stochastic-policy gradient expression.The proof sketch identifies this as the final step in obtaining Equation (B.14).

C BiPaRS for Deterministic Policy Setting

For deterministic policies, BiPaRS extends adaptive shaping-weight optimization, derives an upper-level gradient theorem, and develops EM, MGL, and IMGL approximations under continuity and bounded-reward assumptions.

  • Problem Formulation: BiPaRS for deterministic policies optimizes the expected true reward through a shaping-weight function applied to a given shaping reward.The framework is formulated as a bi-level problem in which policy optimization uses modified rewards.
  • Gradient Theorem: Theorem 3 gives the gradient of the deterministic-policy objective J(zφ) with respect to φ.The corresponding state-action value function Qµ is defined in the original MDP.
  • Assumptions: The deterministic-policy proof assumes continuity of transition and reward derivatives, policy continuity, and bounded rewards.These assumptions support the theorem’s gradient derivation.
  • Gradient Approximations: EM makes zφ an input to a hyper policy, while MGL and IMGL differentiate through one or multiple low-level policy updates.The methods approximate ∇φπθ or the induced policy-parameter gradient using different dependencies.
  • Gradient Approximations: IMGL can use a cheaper approximation by treating the shaping state-action value as constant with respect to φ.This avoids extending the input space of the shaping value function.

D Experiments

Experiments evaluate BiPaRS in cartpole with PPO and learned shaping-weight networks, using settings designed to test adaptive utilization of shaping rewards.

  • Cartpole Setup: The cartpole task uses OpenAI Gym-v1, with randomized initial cart positions and pole angles below 3 degrees.The task requires controlling horizontal cart motion to balance the pole.
  • Hyperparameters: PPO uses two-layer policy and value networks, updated every 20,000 steps with 50 optimizing epochs and batch size 1024.The policy network has 8 units per layer, while the value network has 32 units per hidden layer.
  • Optimization: All networks use Adam, with BiPaRS shaping-weight learning rates of 10^-5 for original shaping-reward tests and different rates for harmful-reward tests.The policy and value-function learning rates are 10^-4 and 2 × 10^-4, respectively.

D.2 MuJoCo

The MuJoCo experiments use PPO-based learners with task-specific shaping weights and neural-network configurations. Humanoid-v2 requires a smaller policy network and an approximation that omits a computationally expensive meta-gradient term.

  • The MuJoCo shaping reward uses a task-specific weight w to scale torque-based penalties relative to the true reward.
  • PPO and RCPO use three hidden layers with 64 units and ReLU activations for their policy and value networks.
  • The BiPaRS shaping-weight network has two tanh hidden layers with 16 and 8 units, respectively.
  • All MuJoCo algorithms update every 20,000 training steps using 50 epochs of 1,024-sample batches, with γ = 0.999 and GAE λ = 0.95.
  • Humanoid-v2 uses a two-layer 32-unit policy network, and BiPaRS-IMGL ignores one meta-gradient term because directly computing the Hessian has O(n^3) complexity.
Loading 2011.02669v1…