PaperlayerView exact revision on arXiv

Source-linked AI summary

On Learning Intrinsic Rewards for Policy Gradient Methods

Zeyu Zheng, Junhyuk Oh, Satinder Singh

arXiv:1804.06459v2cs.AIcs.LGstat.ML

TL;DR

Designing rewards that efficiently produce desirable behavior remains difficult, especially when objectives are unclear and learning agents are bounded. The paper introduces LIRPG, which learns intrinsic rewards jointly with policy-gradient updates while evaluating policies by extrinsic reward. Across Atari and Mujoco experiments, intrinsic-reward learning improved performance in most tested domains, though not all.

  • Problem

    Reward functions are difficult to design for sequential decision tasks, and learning intrinsic rewards for policy-gradient agents remained an open problem.

  • Method

    LIRPG jointly updates policy parameters using intrinsic plus extrinsic rewards and intrinsic-reward parameters using extrinsic rewards.

  • Results

    LIRPG improved learning performance in all 15 tested Atari games and 4 of 5 tested Mujoco domains, using the same A2C or PPO architecture and hyperparameters as baselines.

  • Takeaways & Limitations

    The results support intrinsic-reward learning as a promising way to improve modern policy-gradient agents such as A2C and PPO.

Abstract

from arXiv · show

In many sequential decision making tasks, it is challenging to design reward functions that help an RL agent efficiently learn behavior that is considered good by the agent designer. A number of different formulations of the reward-design problem, or close variants thereof, have been proposed in the literature. In this paper we build on the Optimal Rewards Framework of Singh et.al. that defines the optimal intrinsic reward function as one that when used by an RL agent achieves behavior that optimizes the task-specifying or extrinsic reward function. Previous work in this framework has shown how good intrinsic reward functions can be learned for lookahead search based planning agents. Whether it is possible to learn intrinsic reward functions for learning agents remains an open problem. In this paper we derive a novel algorithm for learning intrinsic rewards for policy-gradient based learning agents. We compare the performance of an augmented agent that uses our algorithm to provide additive intrinsic rewards to an A2C-based policy learner (for Atari games) and a PPO-based policy learner (for Mujoco domains) with a baseline agent that uses the same policy learners but with only extrinsic rewards. Our results show improved performance on most but not all of the domains.

1 Introduction

Reward design is difficult because task objectives may be unclear or multi-criteria, and reward transformations can alter how efficiently bounded agents learn. The paper introduces a stochastic-gradient method for learning intrinsic rewards alongside policy-gradient learning.

  • Reward functions can be hard to specify when objectives involve human satisfaction or multiple criteria that must be combined into one scalar.
  • Policy-order-preserving reward transformations leave the optimal policy unchanged but can improve or worsen sample and computational learning complexity.
  • Bounded agents may benefit from reward transformations that change the theoretical optimum but produce more desirable policies under limited memory, computation, representation, or data.
  • Reward design is often iterative and error-prone because seemingly appropriate rewards can produce unexpected behaviors such as reward hacking.
  • The proposed method learns parametric intrinsic rewards while a policy-gradient learner optimizes combined intrinsic and extrinsic rewards, and the intrinsic-reward parameters optimize extrinsic performance.

2 Background and Related Work

Prior work studies reward transformations, auxiliary rewards, exploration bonuses, and hierarchical structures, but these rewards are generally expert-designed or limited to planning agents. This paper instead learns intrinsic reward mappings for policy-gradient agents.

  • Optimal rewards and reward design: The Optimal Rewards Framework defines an intrinsic reward as optimal when it leads a particular agent-environment pair to maximize extrinsic reward.
  • Optimal rewards and reward design: Exhaustive search does not scale, while PGRD provides a scalable gradient method restricted to lookahead-search planning agents.
  • Reward shaping and Auxiliary rewards: Potential-based shaping preserves optimal policies, whereas auxiliary rewards, exploration bonuses, and prediction errors support learning through expert-designed transformations.
  • Hierarchical RL: Hierarchical RL commonly uses higher-level goals as intrinsic rewards, while this paper uses general mappings from observation-action pairs to rewards.

3 Gradient-Based Learning of Intrinsic Rewards: A Derivation

LIRPG extends policy-gradient learning by jointly adapting policy parameters using intrinsic plus extrinsic rewards and intrinsic-reward parameters using extrinsic performance. The derivation uses chain-rule gradients and an off-policy importance-sampling correction to improve data efficiency.

  • Motivation: LIRPG addresses policy-gradient agents, extending prior intrinsic-reward learning beyond planning agents that use lookahead search.The underlying learner is a policy-gradient agent rather than a planning agent such as one using UCT.
  • Policy Gradient based RL: The policy-gradient formulation represents the policy as a parameterized state-to-action distribution and optimizes expected discounted returns.The paper presents the basic formulation to simplify the derivation, while noting compatibility with actor-critic methods such as A2C and PPO.
  • Reward formulation: The method separates extrinsic rewards, which define the task, from additive intrinsic rewards that influence policy-parameter updates.The agent’s performance is evaluated by the extrinsic rewards achieved, while intrinsic rewards shape learning.
  • Algorithm Overview: At each iteration, the policy parameters θ are updated toward the weighted sum of intrinsic and extrinsic rewards, while η is updated toward extrinsic rewards alone.This jointly updates both modules, with θ optimized through Jex+in and η through Jex.
  • Updating Intrinsic Reward Parameters (η): The intrinsic-reward update estimates how changing η affects extrinsic value through the resulting change in policy parameters, using the chain rule.The update multiplies an extrinsic-value gradient with the sensitivity of updated policy parameters to intrinsic-reward parameters.
  • Data-efficient update: Reusing the original episode with importance sampling avoids the two-episode requirement of the on-policy intrinsic-reward update.The correction addresses the mismatch between behavior generated by πθ and updates evaluated under the changed policy.

4 Experiments on Atari Games

The experiments evaluate LIRPG-augmented A2C agents against extrinsic-reward and live-bonus baselines across Atari games, with analyses of performance and learned action rewards.

  • Overall Performance: LIRPG-augmented agents were compared with A2C agents using only extrinsic rewards and with A2C agents receiving a constant live bonus.The augmented and baseline agents used the same A2C implementation and hyper-parameters; LIRPG adds intrinsic-reward parameters.
  • Overall Performance: The evaluation covered 15 Atari games, using five runs of 50 million time steps per agent and game.The games were selected where baseline performance was good but learning remained slow enough to leave room for improvement.
  • Overall Performance: LIRPG outperformed the A2C baseline on all 15 games and improved human-score-normalized performance by more than 10% on 9 games.Figure 2 reports human score normalized improvements for the augmented agents over the A2C baseline.
  • Overall Performance: Against the stronger live-bonus baseline, LIRPG performed better on every game except SpaceInvaders and Asterix.Most Atari games are shooting games, making the live-bonus baseline a comparatively strong reference.
  • Analysis of the Learned Intrinsic Reward: Intrinsic rewards varied across states for most actions rather than behaving as state-independent action biases, and aggregate rewards correlated with action-selection frequency.Figure 3 summarizes mean rewards, standard deviations, and action frequencies over 100 thousand steps in five games.

5 Mujoco Experiments

The Mujoco experiments test LIRPG with PPO under delayed rewards and compare mixed-reward, baseline, live-bonus, and intrinsic-only training. LIRPG improved PPO in most domains, while intrinsic-only training sometimes matched or exceeded mixed training.

  • Overall Performance: The study evaluated LIRPG with PPO in five Mujoco domains under reward delays of 10, 20, and 40 steps.The environments were Hopper, HalfCheetah, Walker2d, Ant, and Humanoid, with agents trained for 1 million steps per environment.
  • Overall Performance: Figure 4 plots average reward over the last 100 training episodes against time steps for PPO, PPO-bonus, mixed LIRPG, and intrinsic-only LIRPG agents.Curves average 10 random-seed runs, with shaded standard errors.
  • Overall Performance: With a delay of 20, learning intrinsic rewards significantly improved PPO in 4 out of 5 domains, but degraded performance in Ant.The reported results did not use domain-specific hyper-parameter optimization.
  • Overall Performance: LIRPG outperformed the live-bonus baseline in 4 out of 5 domains, with comparable performance to it on HalfCheetah.The live bonus helped on Hopper and Walker2d.
  • Training with Only Intrinsic Rewards: Training the policy with only intrinsic rewards achieved performance similar to mixed-reward LIRPG in 3 out of 5 domains, performed worse on Hopper, and performed better on HalfCheetah.The intrinsic reward module was trained to optimize extrinsic reward, whereas intrinsic-only live-bonus training would fail without a forward-movement signal.

6 Discussion and Conclusion

The paper concludes that LIRPG is a practical algorithm for learning intrinsic rewards for policy-gradient agents and reports improvements across Atari and most Mujoco domains. The authors characterize the results as promising while noting that robustness and conditions for success require further study.

  • Discussion and Conclusion: LIRPG improved learning performance in all 15 tested Atari games and in 4 of 5 Mujoco domains with PPO at a delay of 20.The augmented and baseline agents used the same A2C or PPO architecture and hyper-parameters.
  • Discussion and Conclusion: LIRPG is presented as a novel practical algorithm for learning intrinsic rewards with policy-gradient agents in high-dimensional-observation problems.The paper describes it as the first such algorithm to the authors’ knowledge.
  • Discussion and Conclusion: The results support using intrinsic-reward learning as a form of meta-learning to improve modern policy-gradient architectures such as A2C and PPO.This conclusion is stated as evidence of promise rather than as a claim of universal effectiveness.

A.1 Atari Experiments

The Atari implementation follows standard frame-stacking and action-repeat preprocessing, retains the OpenAI A2C policy architecture and hyper-parameters, and adds a separate intrinsic-reward module.

  • Atari Implementation: Each Atari episode begins with 0 to 30 uniformly sampled no-op steps, repeats each chosen action for 4 frames, and ends at game over or life loss.These procedures follow the setup described by Mnih et al. [2015].
  • Atari Implementation: Four consecutive frames are max-pooled per pixel, converted to an 84 × 84 grayscale image, and stacked across the last 4 observations.The stack captures observations over 16 frames, and extrinsic rewards are clipped to [−1, 1].
  • Atari Implementation: The policy module uses two convolutional neural networks with three convolutional layers and one fully connected layer, matching the OpenAI implementation.The policy module is unchanged between augmented and baseline agents.
  • Atari Implementation: The intrinsic-reward module also has policy and value networks, with its policy network producing a scalar reward per action through tanh in [−1, 1].Its value network estimates Gex for intrinsic-reward learning.
  • Atari Implementation: The intrinsic-reward module uses RMSProp, initializes β at 0.0007, anneals β to zero over 50 million time steps, and searches λ separately for each game.The policy-module hyper-parameters remain at the original OpenAI defaults.

A.2 Mujoco Experiments

The Mujoco augmented agent retains the PPO policy module while adding separate intrinsic-reward networks that take observations and actions as input. PPO hyperparameters remain unchanged, while the intrinsic-reward module uses its own optimization settings and reward scaling.

  • Policy module: The policy network uses a two-hidden-layer MLP with 64-unit tanh layers, while its value network outputs a single scalar and shares no parameters.
  • Intrinsic reward module: The intrinsic-reward networks are two-hidden-layer MLPs that concatenate observation and action vectors and produce one scalar output.
  • The policy module is unchanged from the OpenAI PPO implementation for both augmented and baseline agents.
  • Intrinsic reward module: Adam optimizes the intrinsic-reward networks with β initialized to 0.0001 and fixed for 1 million time steps.
  • Intrinsic reward module: The augmented PPO agent fixes λ at 1.0, scales extrinsic rewards by 0.01, and retains PPO gradient clipping at norm 0.5.

B More Experimental Results

The additional results visualize learning curves and action-level intrinsic rewards across Atari and delayed-reward domains. The figures specify how performance, variability, action selection, domains, and delays are compared between baseline and LIRPG-augmented agents.

  • PPO settings: The PPO experiments use 2048 interaction steps per iteration, batch size 32, 10 training epochs, learning rate 0.0003, and discount factor γ = 0.99.
  • Figure 5: Figure 5 plots average game score over the last 100 episodes against learning time, comparing blue baseline and red LIRPG-augmented curves.Dark curves average four random-seed runs, while shaded areas show standard errors of five individual runs.
  • Figure 5: Figure 5 searches intrinsic-reward weight λ over {0.003, 0.005, 0.01, 0.02, 0.03, 0.05} and ξ over {0.001, 0.01, 0.1, 1}.
  • Figure 6: Figure 6 compares each available action’s intrinsic-reward mean and standard deviation with its selection frequency.The data come from selected runs played without further learning for 100 thousand steps, or 400 thousand frames.
  • Figure 7: Figure 7 plots average reward over the last 100 episodes for each domain and delays of 10, 20, and 40 steps, comparing baseline and LIRPG-augmented PPO.Dark curves average ten random-seed runs, with shaded standard errors across those runs.
Loading 1804.06459v2…