Source-linked AI summary
Understanding Learned Reward Functions
Eric J. Michaud, Adam Gleave, Stuart Russell
TL;DR
Manually specifying rewards is difficult in complex real-world tasks, and learned rewards can rely on spurious features. The paper applies saliency and counterfactuals to interpret learned reward functions. It finds that these methods can expose surprising reward behavior but often cannot predict whether reward changes will affect policy performance.
Problem
In complex real-world tasks, manually specified rewards can be difficult to align with human preferences, while learned rewards may rely on brittle spurious features.
Source: §1.1 · p. 1Method
The authors adapt saliency methods and counterfactual inputs to inspect learned reward functions and test hypotheses about their behavior.
Source: §1.3 · p. 2, §3 · p. 3Results
Learned rewards can implement surprising algorithms, and detected changes in reward output do not necessarily prevent policies from achieving high return.
Source: §5 · p. 8, §4.3 · p. 8Takeaways & Limitations
The authors call for techniques to verify whether learned reward functions truly reflect user preferences.
Source: §5 · p. 8Saliency can reveal reward-output sensitivity but cannot directly predict policy effects; a goal-destroying object could cause catastrophic transfer failure.
Source: §4.1 · pp. 5–6
Abstract
from arXiv · showhide
In many real-world tasks, it is not possible to procedurally specify an RL agent's reward function. In such cases, a reward function must instead be learned from interacting with and observing humans. However, current techniques for reward learning may fail to produce reward functions which accurately reflect user preferences. Absent significant advances in reward learning, it is thus important to be able to audit learned reward functions to verify whether they truly capture user preferences. In this paper, we investigate techniques for interpreting learned reward functions. In particular, we apply saliency methods to identify failure modes and predict the robustness of reward functions. We find that learned reward functions often implement surprising algorithms that rely on contingent aspects of the environment. We also discover that existing interpretability techniques often attend to irrelevant changes in reward output, suggesting that reward interpretability may need significantly different methods from policy interpretability.
1 Introduction
Manual reward specification is difficult in complex real-world tasks, and learned rewards can rely on brittle features rather than user-relevant objectives. The paper investigates direct interpretation of reward functions while noting that changes in reward outputs do not necessarily predict policy behavior.
1.1 Reward Learning
- Reward learning uses demonstrations or trajectory preferences when human preferences are difficult to specify manually.The paper motivates auditing learned objectives to check whether they reflect relevant features and user preferences.Source: §1.1 · pp. 1–2
- Learned rewards may depend on spurious features that correlate with return but are not causally connected to it, making them brittle in novel situations.Manual specification can also omit variables users care about or encode exploitable quirks.Source: §1.1 · p. 1
1.2 On the simplicity of reward functions vs. policies
- Reward functions may be easier to understand than policies because objectives can be simpler than the algorithms agents learn to achieve them.The authors expect objective complexity to remain constant or grow slowly while agent complexity could grow quickly.Source: §1.2 · p. 2
1.3 Evaluating learned reward functions
- Policy rollouts alone can leave unclear whether failure came from an unsuitable learned reward or from the RL algorithm failing to optimize it.EPIC compares learned rewards with ground truth, but ground truth is often unavailable precisely where reward learning is useful.Source: §1.3 · p. 2
- The authors adapt saliency and counterfactual techniques to inspect learned rewards, finding surprising algorithms and limits to predicting transfer from reward-output changes.They report that detected changes may not prevent successful policies, including under potential-based shaping.Source: §1.3 · pp. 2–3
2 Related Work
Prior interpretability work in reinforcement learning has focused on policies, while the closest cited reward-analysis approach fits a simplified model. This paper instead analyzes learned reward functions directly.
2 Related Work
- Policy interpretability methods visualize states, perturb observation regions, or expose policy inputs through an attention bottleneck.The cited examples include state visualization, image blurring with output-change heat maps, and an architectural attention bottleneck.Source: §2 · p. 3
- Russell and Santos fit decision trees to reward functions and use Gini importance and LIME [20], whereas this work applies interpretability techniques directly to learned rewards.The authors argue direct analysis can reveal reward failure modes that a simplified model may not capture.Source: §2 · p. 3
3 Methods
The methods combine saliency maps with counterfactual inputs to inspect learned reward functions. Saliency generates hypotheses about relevant features, while counterfactuals test those hypotheses in more detail.
3 Methods
- Raw gradient saliency computes reward gradients with respect to transition inputs, yielding local sensitivity maps that can be interpretable in practice.Figure 1 shows greater learned-reward sensitivity to next-state pixels than current-state pixels in the illustrated gridworld transitions.Source: §3 · p. 3, Figure 1 · p. 4
- Occlusion maps blur observation regions and record resulting reward changes, producing heat maps that are best suited to large image-based observation spaces.The perturbations indicate sensitivity to information such as feature location or changes across images.Source: §3 · p. 3
- Counterfactual inputs use hand-crafted, potentially unnatural examples to test hypotheses about reward computation and can provide more insight than saliency alone.They take more time to construct than saliency maps.Source: §3 · p. 3
- Method choice depends on feature scale: raw gradients suit largely independent input dimensions, while appropriately scaled occlusion can clarify contiguous image features.Integrated gradients, Guided Grad-CAM, DeepLift, and Shapley values performed poorly in this setting.Source: §3 · p. 4
4 Experiments
Across gridworld and Atari experiments, learned rewards often relied on contingent environmental features, and saliency plus counterfactuals exposed surprising reward algorithms. Large output changes did not reliably predict policy changes, while some environmental shifts could still cause catastrophic transfer failures.
4 Experiments
- Reward regression on expert-policy transitions in gridworld and Atari often produced surprising algorithms that could fail under distribution shift.The authors note that regression on synthetic ground-truth labels might be expected to yield higher-fidelity rewards than learning from noisy, biased human data.Source: §4 · p. 4
4.1 Gridworld Environments
- The model focused on next-state pixels and rewarded goal absence; counterfactual goal removal, duplication, and addition confirmed predictions unlike the true goal-reaching reward.The true reward checks whether any goal block visible in the current state is covered in the next state; the learned rule instead appeared to reward the absence of a visible goal.Source: §4.1 · pp. 4–5, Figure 2 · p. 5
- Despite large output shifts in TwoGoals, learned-reward policies retained almost identical returns, but a goal-destroying object could send the agent toward zero true return.The TwoGoals environment changes the number and positions of goals while preserving the goal-reaching reward structure.Source: §4.1 · pp. 5–6, Figure 3 · p. 6
4.2 Atari 2600
- Breakout reward saliency centered on the ball, while Seaquest saliency centered on the score, whose immediate update let the model predict reward without reading digits.In Breakout, the score updates only after the reward is returned; in Seaquest, it updates immediately in the next state.Source: §4.2 · p. 6, Figure 4 · p. 7
- Removing the score left Breakout outputs unchanged but made Seaquest outputs noisy; a policy trained on the noisy Seaquest reward still achieved somewhat comparable true-reward performance.The reward models were trained with the score displayed, then evaluated without it; Table 1 reports PPO returns after 10M training steps.Source: Figure 5 · p. 7, §4.2 · p. 7, Table 1 · p. 8
4.3 Summary of results
- Saliency narrows hypotheses about reward mechanisms, but output changes alone do not predict policy effects because reward interpretability methods treat all output changes as equally important.The authors suggest that reward-specific measures such as EPIC could be a useful starting point for methods that account for reward functions’ mathematical structure.Source: §4.3 · p. 8
5 Conclusions and Future Work
Saliency maps and counterfactuals reveal how learned rewards work, but they are poorly suited to predicting transfer because changed reward outputs can still produce high-return policies. The results motivate new work to verify that learned rewards reflect user preferences.
5 Conclusions and Future Work
- Saliency maps and counterfactuals reveal surprising reward algorithms and feature dependencies, but poorly predict transfer because changed outputs can still yield high-return policies.The techniques highlight both relevant differences and irrelevant ones.Source: §5 · p. 8
- Reward outputs can change while policies trained on them still achieve high return in a modified environment, making current interpretability techniques conservative indicators of transfer.Source: §5 · p. 8
- Verifying that learned rewards reflect user preferences is tractable and important as increasingly capable RL systems are deployed in complex real-world environments.The paper argues objective complexity is likely to remain fixed while ML capability grows.Source: §5 · p. 8
Broader Impact
For complex real-world tasks, human preferences are difficult to translate manually into rewards, motivating reward learning and audits. The authors caution that their interpretability methods may miss problems and should not be trusted too much.
Broader Impact
- Manually translating the totality of human preferences into reward functions is difficult, motivating reward learning for complex real-world tasks.Source: Broader Impact · p. 9
- The authors hope interpretability methods such as theirs will help audit learned rewards for real-world tasks.Source: Broader Impact · p. 9
- The methods can elucidate problems but are not guaranteed to reveal them, so engineers should not place too much trust in them.The authors identify a potential false sense of security as a downside.Source: Broader Impact · p. 9