Source-linked AI summary
Preprocessing Reward Functions for Interpretability
Erik Jenner, Adam Gleave
TL;DR
Learned rewards may fail to capture user preferences, and evaluating them across policies and environment dynamics can be difficult. The paper preprocesses rewards into simpler policy-equivalent forms before visualization, and reports that preprocessing often improves interpretability. It also notes that policy optimization may behave differently under the original and equivalent rewards.
Problem
Learned rewards may not represent user preferences, while evaluating their quality and transfer across environment dynamics can be difficult.
Source: §1 · p. 1Method
The framework searches policy-equivalent transformations for a lower-cost reward, treating the learned reward as a black box and visualizing the result.
Source: §1 · p. 2, §3.1 · p. 3Results
Takeaways & Limitations
Some low-quality learned rewards remain difficult to understand, suggesting reward-learning algorithms can converge to models significantly different from user preferences.
Source: §7 · p. 8Potential shaping can make policy optimization easier or harder, so an RL algorithm may learn different policies with the original and visualization rewards, especially in challenging optimization settings.
Source: §6 · p. 8
Abstract
from arXiv · showhide
In many real-world applications, the reward function is too complex to be manually specified. In such cases, reward functions must instead be learned from human feedback. Since the learned reward may fail to represent user preferences, it is important to be able to validate the learned reward function prior to deployment. One promising approach is to apply interpretability tools to the reward function to spot potential deviations from the user's intention. Existing work has applied general-purpose interpretability tools to understand learned reward functions. We propose exploiting the intrinsic structure of reward functions by first preprocessing them into simpler but equivalent reward functions, which are then visualized. We introduce a general framework for such reward preprocessing and propose concrete preprocessing algorithms. Our empirical evaluation shows that preprocessed rewards are often significantly easier to understand than the original reward.
1 Introduction
Learned rewards need validation before deployment, but policy-based evaluation can be costly and makes failures hard to diagnose. The paper proposes simplifying rewards through policy-preserving transformations before visualization.
1 Introduction
- Policy-based reward evaluation can be expensive and brittle: poor policy performance does not reveal whether the reward or policy training caused it.Testing transfer across environment dynamics is also difficult because specifying the relevant dynamics and evaluating policies across them may be infeasible.Source: §1 · p. 1
- EPIC distance [10] can compare a learned reward with ground truth, but without ground truth it cannot establish whether learned models are correct.It may still help compare or cluster multiple learned reward models.Source: §1 · p. 1
- Reward preprocessing transforms a learned reward into a simpler equivalent that preserves optimal policies, then visualizes the simplified reward.The framework optimizes an interpretability objective over equivalent transformations while treating the reward model as a black box.Source: §1 · p. 2
- The paper contributes a general preprocessing framework, two concrete applications, and empirical evidence that the methods often significantly improve interpretability.The motivation is that real-world rewards are hard to specify by hand and may be learned from demonstrations, preferences, or initial states.Source: §1 · pp. 1–2
2 The Reward Preprocessing Framework
Reward preprocessing seeks a minimum-cost reward within a chosen equivalence class. Equivalence and interpretability costs should reflect the intended application and visualization.
2 The Reward Preprocessing Framework
- The framework turns a reward into a simpler equivalent reward for visualization, requiring both an equivalence relation and an interpretability cost function.It seeks the minimum-cost reward within the chosen equivalence class.Source: §2 · p. 2
2.1 Equivalence Relation
- Potential shaping and positive rescaling preserve policy ordering, while broader equivalences can be appropriate when the downstream task and dynamics are fixed.S′-redistribution preserves the optimal policy when the transition dynamics determining S′ remain fixed.Source: §2.1 · p. 2
2.2 Choosing Cost Functions
- No single cost captures interpretability: sparse rewards and dense rewards with simple structure can both be understandable.The authors propose multiple sufficient-but-not-necessary interpretability costs and presenting the resulting rewards for user choice; visualization method also affects which costs are appropriate.Source: §2.2 · pp. 2–3
3 Methodology
The methodology searches potential-shaped versions of a reward for lower-cost alternatives, using sparsity or temporal smoothness objectives. It preserves reward scale during preprocessing.
3 Methodology
- The concrete methods are presented as simple instances that establish a lower bound on the benefit obtainable from reward preprocessing.The authors note that these choices are likely far from optimal.Source: §3 · p. 3
3.1 Potential Shaping Equivalence Relation
- Potential shaping preserves policy ordering under arbitrary transition dynamics and initial-state distributions, enabling equivalence even when transferring across dynamics.The shaped reward is optimized over a neural-network-parameterized potential using gradient descent; the reward itself need not be differentiable, but the cost must be.Source: §3.1 · p. 3
- Although positive rescaling preserves policy ordering, the methods exclude rescaled rewards to preserve scale for visualization.Source: §3.1 · p. 3
3.2 Cost Functions
- The methods use sparsity costs based on |r| or log(1 + |r|), and smoothness costs based on absolute deviations between temporally adjacent rewards.Costs are evaluated on uniform transitions in gridworlds and on the visualization transition distribution in continuous control.Source: §3.2 · pp. 3–4
4 Results
In gridworld, preprocessing recovers interpretable patterns from shaped rewards, while mountain-car rewards become simpler but remain noisy. For learned rewards, residual noise may reflect errors that potential shaping cannot remove.
4 Results
- The evaluation covers gridworld and mountain car, using hand-designed shaped or noisy rewards and learned rewards trained with AIRL or DRLHP.The gridworld permits direct visualization of the full reward function; mountain-car rewards are plotted over expert trajectories.Source: §4 · p. 4, §4.1 · p. 4, §4.3 · p. 5
4.1 Simplifying Shaped Rewards
- Preprocessing recovers simple Goal and Path patterns from shaped gridworld rewards, including Manhattan-distance and random shaping.The original Path reward is dense, yet preprocessing recovers a similar pattern.Source: §4.1 · p. 4
4.2 Understanding Learned Rewards
- DRLHP rewards remain noisy after preprocessing, and potential shaping cannot bridge AIRL rewards’ greater gap from ground truth.A larger equivalence class might remove more noise, but could stop preserving the same optimal policy under some dynamics.Source: §4.2 · pp. 4–5
4.3 Mountain Car
- Mountain-car preferences are generated from ground-truth rewards, including a case shaped with an optimal value function to simulate incremental-progress feedback.The learned model uses synthetic preferences between trajectory fragments; the shaped-ground-truth case also uses DRLHP.Source: §4.3 · p. 5
- In mountain car, preprocessing makes noisy learned rewards significantly simpler, especially for shaped rewards; sparsity performs better than smoothness.Residual noise is likely not removable by potential shaping and may accurately depict learned-reward errors.Source: §4.3 · p. 5
5 Related Work
Prior work applies general-purpose interpretability methods to reward models, while this paper exploits reward structure to simplify them without changing their equivalence. Policy interpretation and interpretable reward design address related but distinct problems.
5 Related Work
- Russell and Santos use feature-importance estimates from a fitted global model and LIME [19], but their simpler fitted model is usually not equivalent to the original reward.Reward preprocessing instead exploits reward structure to obtain an equivalent, simplified model.Source: §5 · p. 5
- Michaud et al. [14] interpret rewards directly with saliency, occlusion, and counterfactual inputs, methods that do not exploit reward structure.The paper presents reward preprocessing as complementary: simplify an equivalent reward first, then visualize or interpret it.Source: §5 · p. 5
- Policy-interpretability work includes intrinsically simple policies [25], additive Q-value explanations, and causal models [13] [8].This differs from interpreting learned reward functions.Source: §5 · p. 5
- Devidze et al. design rewards that trade off interpretability, operationalized as sparsity, against ease of policy optimization.Their goal differs from simplifying an already learned reward model.Source: §5 · p. 5
6 Limitations and Future Work
The paper identifies limits in potential shaping and in the cost functions used to simplify rewards. These limits constrain how preprocessing relates to policy optimization and visualization choices.
6 Limitations and Future Work
- Potential shaping preserves the optimal policy but can make policy optimization easier or harder, so an RL algorithm may learn different policies under shaped and unmodified rewards.This issue is most significant when policy optimization is challenging and the tool is used by trained practitioners.Source: §6 · p. 8
- Potential shaping may define an equivalence class that is too broad for practical transfer needs, while known dynamics invariants could allow a larger class that still preserves optimal policies.The paper notes that rewards need not transfer across all possible transition dynamics in practice.Source: §6 · p. 8
- The proposed cost functions target this paper’s visualizations, so other visualizations may benefit from different costs, such as sparse gradients for gradient saliency maps.The authors expect sparsity and smoothness to remain useful in many settings.Source: §6 · p. 8
7 Conclusion
The paper presents reward preprocessing as a way to simplify rewards before visualization. It reports recovery of simple rewards from shaped versions and simplification of noisy learned models, while noting that low-quality models can remain difficult to understand.
7 Conclusion
- Preprocessing recovers simple reward functions from shaped versions of ground-truth rewards and can substantially simplify noisy learned reward models.Some low-quality learned models remain difficult to understand, suggesting reward-learning algorithms can converge to models significantly different from user preferences.Source: §7 · p. 8
- The paper introduces a framework for preprocessing reward functions before visualization.Source: §7 · p. 8
- Some low-quality learned reward models remain difficult to understand even after preprocessing.The authors suggest these models may differ significantly from users’ intended preferences.Source: §7 · p. 8
A Appendix
The appendix shows reward maps and mountain-car reward curves under several preprocessing costs and reward-learning settings. Its captions report simplification across settings, with differing outcomes for logarithmic and L1 costs in some cases.
A Appendix
- Figures 4–7 show Goal and Path rewards preprocessed with L1 or logarithmic sparsity and smoothness costs; the log-cost Path result is less symmetric than the L1 result but remains significantly simplified.The Goal results are described as similar to Fig. 1, and the L1 Path results as similar to Fig. 2.Source: Figure 4 · p. 11, Figure 5 · p. 12, Figure 6 · p. 13
- Figures 8–11 show preference-trained Goal and Path reward models alongside preprocessed versions using L1 or logarithmic sparsity and smoothness costs.Each heatmap represents all possible transitions in a 10 × 10 gridworld.Source: Figure 8 · p. 15, Figure 9 · p. 16, Figure 10 · p. 17
- Figures 12–15 show AIRL-learned Goal and Path reward models alongside preprocessed versions using L1 or logarithmic sparsity and smoothness costs.Each heatmap represents all possible transitions in a 10 × 10 gridworld.Source: Figure 12 · p. 19, Figure 13 · p. 20, Figure 14 · p. 21
- For mountain car, logarithmic-cost preprocessing recovers the ground truth exactly up to a constant shift for two linear shapings and significantly simplifies the more complex shaping.The L1 version works reasonably well for simple shaped rewards but appears less effective than the logarithmic version on the complex case.Source: Figure 16 · p. 23, Figure 17 · p. 24
- For learned mountain-car rewards, sparsity and smoothness preprocessing produces simpler, less noisy curves, especially for the reward shaped with an optimal value function.The results are described as extremely similar, though perhaps slightly worse, than the logarithmic version in Fig. 3.Source: Figure 18 · p. 25