Dynamics-Aware Comparison of Learned Reward Functions

1 INTRODUCTION

• Reward functions can be learned from demonstrations, preferences, or other supervision, but evaluating their quality remains an open challenge.This is especially relevant when learned rewards are too complex to inspect manually.Source: §1 · p. 1
• Policy-based reward comparisons require learning a policy and evaluating its returns, and can conflate reward errors with policy-optimization errors.The evaluation may also omit transitions needed to quantify reward differences.Source: §1 · pp. 1–2
• DARD uses a transition model to compare rewards invariantly to reward shaping while evaluating learned rewards on approximately physically realizable transitions.Experiments in simulated multi-agent navigation and robotic-arm manipulation suggest it better reflects learned-reward similarity than EPIC.Source: §1 · p. 2

2.1 LEARNING REWARD FUNCTIONS

• Reward-learning methods use explicit labels, per-timestep reward labels, or expert demonstrations to infer reward functions.Inverse reinforcement learning is underconstrained because many rewards can make an expert policy appear optimal.Source: §2.1 · p. 2

2.2 EVALUATING METHODS OF LEARNING REWARD FUNCTIONS

• Policy-value evaluation estimates the value of an induced policy, while EPIC compares rewards directly to avoid policy-learning complexity and obscure sources of error.Online rollouts can be prohibitive in safety-critical domains; this work extends EPIC with an environment transition model to improve comparisons involving learned rewards.Source: §2.2 · pp. 2–3

3.1 MARKOV DECISION PROCESSES

• An MDP specifies state and action spaces, transitions, rewards, a discount factor, and an initial-state distribution; policies aim to maximize expected discounted return.A trajectory return is the sum of discounted rewards along its transitions.Source: §3.1 · p. 3

3.2 REWARD FUNCTION EQUIVALENCE

• The paper seeks a pseudometric reflecting whether reward functions induce the same optimal policies, since potential-based shaping can preserve those policies.The focus is on rewards depending on current state, action, and next state.Source: §3.2 · pp. 3–4

3.3 EQUIVALENT-POLICY INVARIANT COMPARISON (EPIC)

• EPIC canonicalizes rewards to remove shaping effects, then compares canonicalized rewards using correlation-based distance.Canonicalization is invariant to potential-based shaping; states and actions are independent during canonicalization, while the joint transition distribution is used for correlation.Source: §3.3 · p. 4
• EPIC applies its canonicalization using marginal state and action distributions from an offline dataset, which can require evaluating learned rewards outside their training distribution.The out-of-distribution evaluations can produce inaccurate reward values.Source: §3.3 · p. 4

4 DARD: DYNAMICS-AWARE REWARD DISTANCE

• EPIC’s canonicalization samples independent state combinations that may be impossible under the environment’s transition model, making learned-reward evaluations out of distribution and potentially unreliable.This trade-off supports robustness to arbitrary changes in transition dynamics but can introduce inaccuracies for rewards learned from data.Source: §4 · p. 5
• DARD assumes access to a transition model and transforms rewards using transitions sampled conditionally from that model, aiming to preserve shaping invariance while staying closer to observed data.The transformation normalizes a transition’s reward using nearby model-sampled transitions; the sampling in one term is not perfectly in-distribution, but is closer to physically realizable than EPIC’s samples.Source: §4 · pp. 5–6
• DARD selects actions to cover possible actions because optimized policies may choose a wide range, and the transformed rewards remain invariant to potential shaping.The figure contrasts EPIC’s global sampling with DARD’s local, model-based sampling; both recover the unshaped value in the example.Source: Figure 1 · p. 5, §4 · p. 6
• The paper states that DARD is a pseudometric and notes that its accuracy depends on transition dynamics remaining similar during deployment.It presents this assumption as a basis for more accurate and predictive learned-reward distances in practice.Source: §4 · p. 6

5.1 EXPERIMENTAL SETUP

• The experiments use Bouncing Balls and Reacher, sampling transitions from uniform-random or expert-policy coverage distributions.Bouncing Balls models navigation among moving balls; Reacher reflects robotic manipulation with changing joint configurations.Source: §5.1 · pp. 6–7
• DARD is tested with approximate ground-truth and learned transition models, alongside EPIC and Pearson distance; reward models include feasibility, regression, and preference-based variants.Learned transition-model results are called DARD-L; preferences are synthesized from ground-truth return.Source: §5.1 · pp. 7–8

5.2 RESULTS AND DISCUSSION

• For practical learned reward models, DARD distances were more predictive of episode return than baseline methods in both environments.The comparison covers Bouncing Balls and Reacher; Table 1 reports the distance metrics and corresponding policy returns.Source: Table 1 · p. 8, §5.2 · p. 9
• EPIC was not predictive of episode return when comparing learned models, partly because out-of-distribution evaluation misranked REGRESS and PREFERENCES in Reacher.In both environments, EPIC also assigned large distances to REGRESS despite policy performance resembling ground truth, disregarding shaping.Source: §5.2 · p. 8
• Across uniform-random and expert coverage, DARD’s distance scales differed, but its model ordering remained predictive of episode return in both environments.DARD is sensitive to the coverage distribution because sampled transitions cluster around it.Source: §5.2 · p. 9
• DARD-L was predictive across both environments and coverage distributions, usually matching DARD, but Reacher expert coverage exposed a false-positive equivalence from transition-model error.The authors attribute the exception to reward sensitivity to small errors in predicted states and emphasize using well-fit learned transition models.Source: §5.2 · p. 9
• DARD is poorly suited to environments where nearby transitions have nearly identical rewards, because it can then be sensitive to reward-model noise.The authors recommend settings where some nearby transitions yield well-differentiated reward values.Source: §5.2 · p. 9

• DARD is a reward pseudometric that compares rewards directly, is invariant to reward shaping, and uses transition dynamics to avoid far out-of-distribution evaluations.Source: §6 · p. 9
• Experiments suggest DARD reliably predicts associated policy performance, even for learned rewards, when transition dynamics are not expected to change significantly between evaluation and deployment.Source: §6 · p. 9
• Future work could use DARD to benchmark reward-learning methods or apply it in high-dimensional domains where learned transition models are necessary.Source: §6 · p. 9

• The supplementary material provides source code needed to reproduce the results.Source: REPRODUCIBILITY STATEMENT · p. 10
• The software is designed to reproduce the main-body results through a single program execution.Source: REPRODUCIBILITY STATEMENT · p. 10
• Results are computed over five random seeds, with standard errors reported in Appendix A.1.7.Source: REPRODUCIBILITY STATEMENT · p. 10
• The paper provides extensive experimental-setup details in Appendix A.1.Source: REPRODUCIBILITY STATEMENT · p. 10

• Evaluating rewards without executing policies in the environment could help assess objective alignment before safety-critical deployment.Source: ETHICS STATEMENT · p. 10
• Potential application areas include healthcare, finance, and robotics, where the statement says reinforcement learning may otherwise be too risky.Source: ETHICS STATEMENT · p. 10
• The authors identify inappropriate reliance on these methods, including use outside their applicability or instead of further validation, as a risk for unsafe deployment.Source: ETHICS STATEMENT · p. 10

• The following subsections provide details on the different components involved in the experiments.Source: §A.1 · p. 12

• Bouncing Balls has an agent navigate to a random goal while avoiding bouncing agents, with positions and velocities plus the goal forming the state.Source: §A.1.1 · p. 12
• Its 400-timestep horizon approximates a continuing MDP so S and S′ are identically distributed as EPIC requires.Sampling states independently from dataset marginals can produce transitions with entirely different agent positions and velocities.Source: §A.1.1 · p. 12
• Reacher adds a goal-based reward and increases the interval between states from two to five timesteps so nearby-transition rewards are sufficiently differentiated.Source: §A.1.1 · p. 12

• PPO, implemented with RLlib, is used to learn control policies.Source: §A.1.2 · p. 12
• Table 2 lists PPO parameters, with superscripts distinguishing Bouncing Balls and Reacher.Source: Table 2 · p. 12

A.1.3 DATA COLLECTION; A.1.4 TRANSITION MODEL LEARNING

• The experiments collect train, validation, and evaluation datasets using uniform-action and expert policies, and train transition models on uniform-policy data.The transition models predict state changes: Bouncing Balls models each ball separately using velocity (and action for the controlled ball), while Reacher uses concatenated state and action; both use mean-squared error against the next state.Source: §A.1.3 · p. 12, §A.1.4 · p. 13

A.1.5 REWARD LEARNING; A.1.6 REWARD EVALUATION; A.1.7 STANDARD ERROR OF RESULTS

• Reward learning compares direct regression, EPIC-distribution regression, and preference learning; learned rewards are evaluated with EPIC, DARD, and policy returns.Policy returns are averaged over five seeds after one million environment steps; standard errors for distances and returns are computed over five seeds. DARD’s sample approximation assumes deterministic dynamics in these experiments, with one next-state sample per action.Source: §A.1.5 · p. 13, §A.1.6 · pp. 14–15, §A.1.7 · p. 15

A.2.2 METRIC SAMPLE SIZE SENSITIVITY; A.2.1 POINT MAZE EXPERIMENTS

• For noisy ground-truth rewards, DARD and EPIC have similar sample-size sensitivity, while their relative sensitivity for learned rewards depends on the reward model’s training distribution.For REGRESS, DARD has smaller confidence-interval widths because it avoids out-of-distribution reward evaluations; for REGRESS OOD, EPIC has smaller widths because its evaluation distribution matches training. In Point Maze, DARD performs poorly when nearby transitions have nearly identical rewards, but its distances move closer to EPIC when the timestep is increased fivefold; reward models also ignore next states and exhibit little shaping in that experiment.Source: §A.2.1 · p. 16, §A.2.2 · pp. 17–18

A.2.3 RANDOMIZED REWARD FUNCTIONS

• DARD remains predictive of policy return across 128 randomized Reacher rewards under both uniform and expert coverage distributions.Small DARD distances are associated with larger returns; the relationship varies by coverage, and outliers arise when extreme reward weights cause policy-optimization difficulties.Source: §A.2.3 · p. 18, Figure 3 · p. 20

A.3.1 THE IMPACT OF TRANSITION MODEL ERRORS ON DARD

• Transition-model errors cannot separate rewards equivalent over all transitions, but can cause false negatives or false positives for rewards that are not everywhere equivalent.False negatives can arise from sampled unrealizable transitions where rewards differ; false positives can arise when the model misses realizable transitions where they differ, making comprehensive dataset coverage important.Source: §A.3.1 · pp. 20–21

A.3.2 ANALYSIS OF DARD-L IN REACHER WITH π∗COVERAGE DISTRIBUTION

• In Reacher with expert coverage, the learned transition model’s larger predicted displacements produce lower DARD-L distances than the ground-truth model.Mean-squared-error training underweights near-zero displacements; larger predictions make states easier to classify relative to the goal threshold and reduce sampling of states where rewards differ.Source: §A.3.2 · p. 21

A.3.3 APPLICATIONS OF REWARD COMPARISON METRICS

• Reward-comparison metrics could benchmark IRL algorithms against a reward model learned from a small set of manually labeled demonstrations, avoiding policy learning and evaluation.The proposed setting assumes plentiful potentially suboptimal demonstrations, costly or risky online policy evaluation, and challenging policy learning; a benchmark reward could be learned by supervised learning from the labeled examples.Source: §A.3.3 · p. 21
• A small distance between a newly learned reward and the previous high-quality reward supports continued deployment, while a large distance calls for inspecting differing cases.The comparison uses the old reward’s validation dataset and provides part of a validation procedure for continuous reward learning and deployment.Source: §A.3.3 · pp. 21, 23

A.4.1 COMPARING LEARNED REWARD FUNCTIONS

• The dynamics-aware transformation is invariant to potential-based reward shaping: shaped and unshaped rewards have equal transformed rewards.For R′(s, a, s′) = R(s, a, s′) + γΦ(s′) − Φ(s), the proposition states CT(R′) = CT(R).Source: §A.4.1 · p. 23
• DARD defines a pseudometric between reward functions under a coverage transition distribution and the corresponding environment transition model.Source: §A.4.1 · p. 23