Quantifying Differences in Reward Functions

1 INTRODUCTION

• Rollout evaluation can falsely reject a preference-aligned reward when reinforcement learning fails to optimize it, and is computationally expensive.It also may accept a reward that yields the right behavior in one environment but misbehaves when initial states or transition dynamics change.Source: §1 · p. 1
• EPIC compares reward functions directly without policy training, canonicalizing potential-based shaping before correlating canonical rewards over a coverage distribution.The paper also introduces NPEC and ERC as baselines that partially satisfy the stated criteria.Source: §1 · p. 2
• EPIC produces similar results across a range of coverage distributions, whereas ERC and especially NPEC are highly sensitive to the choice of distribution.EPIC works best when the coverage distribution supports all realistic transitions; the experiments use uninformative priors such as random-action rollouts.Source: §1 · p. 2
• Low EPIC distance predicts low regret for policies optimized under learned rewards, including in unseen environments, while rollout evaluation only determines regret in its evaluation environment.The paper reports empirical confirmation of this prediction.Source: §1 · p. 2

2 RELATED WORK

• Inverse reinforcement learning infers rewards from demonstrations, but the problem is underconstrained because many rewards can explain the same demonstrations.Bayesian IRL infers a posterior over rewards, while Maximum Entropy IRL selects the highest-entropy consistent reward.Source: §2 · p. 2
• Preference-based methods learn from trajectory comparisons, while T-REX learns from ranked demonstrations and Cabi et al. learn from episode-level reward sketches [5].Source: §2 · p. 2
• The authors identify no prior work focused on direct reward-function evaluation; related approaches include rollout evaluation, transfer-environment testing, and scatterplots of returns.Ng et al. identify reward transformations that preserve the optimal policy, inspiring the paper’s treatment of reward equivalence.Source: §2 · p. 2

3 BACKGROUND

• An MDP specifies states, actions, discounting, initial-state and transition distributions, and a reward function; its objective is a policy maximizing expected discounted return.The paper assumes an infinite-horizon discounted MDP, with undiscounted results requiring convergence regularity conditions.Source: §3 · p. 3
• Reward equivalence groups bounded rewards that induce the same optimal policies for every initial-state distribution and transition dynamic with fixed state, action, and discount factor.The paper defines an equivalence relation with reflexivity, symmetry, and transitivity.Source: §3 · p. 3
• Potential shaping adds γΦ(s′) − Φ(s) to reward without changing optimal policies, and positive reward scaling also leaves the optimal-policy set unchanged.For discounted infinite-horizon trajectories, the shaping return depends on the initial state while the terminal term tends to zero.Source: §3 · p. 3
• A pseudometric permits distinct but equivalent rewards to have zero distance while retaining symmetry and the triangle inequality.A metric would require zero distance to imply that the two rewards are identical.Source: §3 · pp. 3–4

4 COMPARING REWARD FUNCTIONS WITH EPIC

• EPIC canonicalizes potential-based shaping and compares the resulting rewards with Pearson correlation, which is invariant to scale.This construction makes EPIC invariant across reward-equivalence classes.Source: §4 · p. 4
• The canonical reward uses state and action distributions rather than the MDP’s initial-state distribution or transition dynamics, allowing evaluation on transitions that may arise in deployment.The canonicalization is invariant to potential shaping, and broad-support state and action distributions make it more robust to perturbations of individual transitions.Source: §4 · p. 4
• The triangle inequality allows a proxy reward within EPIC distance ε of ground truth to stand in for expensive direct comparisons, with comparison error bounded by ε.This can make benchmarks expensive to build but cheaper to use.Source: §4 · pp. 4–5
• EPIC distance upper-bounds the difference in returns between optimal policies under either reward, subject to coverage of transitions induced by those policies.The bound extends to continuous spaces under a Lipschitz assumption, with Wasserstein distance replacing the coverage factor; returns converge as EPIC distance approaches zero.Source: §4 · p. 5
• The regret guarantee requires adequate coverage of optimal-policy transitions; broad coverage supports the bound, while excluding unlikely transitions can tighten it.A uniform coverage distribution meets the stated requirement in finite settings, with K ≤ |S|^2|A|.Source: §4 · p. 5
• EPIC upper-bounds regret but cannot lower-bound it in arbitrary environments, since action-independent transitions make all policies’ expected returns equal regardless of rewards.Source: §4 · p. 5
• EPIC assigns zero distance to rescaled and shaped Sparse and Dense rewards, but distance 0.27 to Path and Cliff despite their shared optimal policy in deterministic gridworlds.With a significant risk of slipping, Cliff’s optimal policy takes a different route to avoid the larger penalty for falling.Source: §4 · p. 5

5 BASELINE APPROACHES FOR COMPARING REWARD FUNCTIONS

• EPIC assigns zero distance to equivalent Sparse and Dense rewards, but distance 0.27 to Path and Cliff, whose optimal policies diverge when the gridworld is slippery.The figure contrasts rewards that look different but are equivalent with rewards whose shared deterministic optimal policy changes under slippery dynamics.Source: Figure 1 · p. 6

5.1 EPISODE RETURN CORRELATION (ERC)

• ERC measures Pearson distance between episode returns and is shaping-invariant when initial and terminal states are fixed, or only the initial state in infinite-horizon discounted MDPs.The fixed-terminal-state result follows because shaping adds a constant to each trajectory's return.Source: §5.1 · p. 6
• With stochastic initial states, ERC distance can take arbitrary values under potential shaping.The section describes shaping choices that make the distance approach zero as the potential magnitude grows.Source: §5.1 · p. 6

5.2 NEAREST POINT IN EQUIVALENCE CLASS (NPEC)

• NPEC takes the minimum Lp distance between reward equivalence classes, but remains sensitive to the scale of one reward.The underlying Lp distance is computed over a coverage distribution of transitions.Source: §5.2 · pp. 6–7
• After normalization, NPEC is a premetric invariant to equivalent reward representations, but it may be asymmetric.The theorem bounds the distance between zero and one for bounded rewards.Source: §5.2 · p. 7
• NPEC can be computed exactly in tabular settings; otherwise, gradient descent gives an upper bound whose quotient may be too high or too low.The approximation is not guaranteed to preserve the direction of error in the resulting NPEC distance.Source: §5.2 · p. 7

6.1 COMPARING HAND-DESIGNED REWARD FUNCTIONS

• Across four tasks, EPIC is more reliable than NPEC and ERC, and more computationally efficient than NPEC.The Gridworld, HalfCheetah, and Hopper results are described as qualitatively similar to PointMass.Source: §6.1 · p. 7

6.3 PREDICTING POLICY PERFORMANCE FROM REWARD DISTANCE

• Low distance from GT predicts high policy return in PointMaze-Train and the unseen PointMaze-Test variant, where the wall position differs.Table 2 reports reward distances and mean GT returns for policies trained on each reward; AIRL generator-policy returns are also shown.Source: Table 2 · p. 8, §6.3 · p. 9

6.1 COMPARING HAND-DESIGNED REWARD FUNCTIONS

• EPIC’s estimated distance for equivalent rewards is < 1 × 10−3, versus 0.58 for NPEC and 0.56 for ERC; NPEC’s Figure 2(b) computation took 31 hours.EPIC and ERC figures were generated in under two hours; a lower-precision EPIC approximation took 17 seconds.Source: §6.1 · p. 8

6.2 SENSITIVITY OF REWARD DISTANCE TO COVERAGE DISTRIBUTION

• EPIC’s learned-reward distances vary by less than 2× with D and preserve model rankings, while NPEC and ERC are more sensitive.Under πuni versus π∗, NPEC’s AIRL SO-to-Pref ratio shifts from 96:1 to 2:1; ERC’s shifts from 22:1 to 3:2.Source: §6.2 · p. 8
• With pathological coverage distributions, EPIC’s ranking changes in one case, compared with two for NPEC and all cases for ERC.Ind samples physically impossible transitions; Jail restricts rollouts to a small region that excludes the goal.Source: §6.2 · p. 8
• EPIC is sensitive to D on Mirage, whose reward is higher near a distant mirage state than at the goal but otherwise matches GT.Random and expert policies rarely visit the mirage, producing small EPIC distances under those distributions.Source: §6.2 · p. 9
• Mix’s broader coverage correctly identifies Mirage as far from GT; the experiments recommend broad coverage over plausible transitions rather than narrow expert paths or impossible transitions.Mix combines expert-policy navigation around the wall with random-policy deviations from the goal.Source: §6.2 · p. 9

6.3 PREDICTING POLICY PERFORMANCE FROM REWARD DISTANCE

• Regress and Pref reach low distances and near-expert performance, while AIRL SO and AIRL SA have distances an order of magnitude higher and poor performance.Across training checkpoints, reward distance tracks policy regret; the AIRL methods are slower and more unstable, although their generator policies perform reasonably in PointMaze-Train.Source: §6.3 · p. 9

7 CONCLUSION

• EPIC is a pseudometric that is bounded, invariant to equivalent rewards, and bounds regret for optimal policies.These properties are established in Theorems 4.7, 4.8, and 4.9.Source: §7 · p. 9
• Empirically, EPIC assigns zero distance to equivalent rewards that NPEC and ERC consider dissimilar, and distance to GT predicts policy return even in unseen environments.Source: §7 · p. 9
• The authors recommend reporting EPIC alongside policy-based metrics because traditional policy-based metrics do not guarantee learned-reward fidelity.Source: §7 · p. 9

A.1.1 SAMPLE-BASED APPROXIMATION FOR EPIC DISTANCE

• The approximation draws BV from coverage distribution D and BM from the joint state-action distributions DS × DA.Source: §A.1.1 · p. 12
• For each transition in BV, it estimates the canonically shaped reward by averaging over BM.The displayed expression adds an expectation of shaping terms to R(s, a, s′).Source: §A.1.1 · p. 12
• The method computes Pearson distance between the approximate canonical rewards on BV in O(NV NM) time.The constant c is dropped because it does not affect Pearson distance.Source: §A.1.1 · p. 12

A.1.2 OPTIMIZATION-BASED APPROXIMATION FOR NPEC DISTANCE

• NPEC is defined as the infimum of Lp distance over equivalent rewards and approximated with gradient descent on the reward model.Source: §A.1.2 · p. 12
• The search parameterizes rewards as a scaled RA plus a constant and potential-based shaping, then minimizes mean Lp cost against RB on samples from D.The potential is represented by a deep neural network Φw.Source: §A.1.2 · p. 12
• Because gradient descent converges reliably only when ν and c start near their optima, they are initialized from a constrained least-squares fit.The active-set method solves the fit; with the potential fixed at zero and p = 2, these affine parameters minimize the Lp distance.Source: §A.1.2 · p. 12

• Bootstrapping provides distribution-free confidence intervals because approximate distances may not be normally distributed.The paper reports intervals to help measure error introduced by the approximations.Source: §A.1.3 · p. 12
• EPIC uses 30 seeds, compared with 9 for Episode Return and 3 for NPEC, reflecting their greater computational requirements.Source: §A.1.3 · p. 13
• ERC confidence intervals are bootstrapped over collected episodes, with the distance recomputed for each bootstrapped episode sample.Source: §A.1.3 · p. 13

A.2.1 HYPERPARAMETERS FOR APPROXIMATE DISTANCES

• Approximate-distance coverage uses uniform random transitions in deterministic GridWorld, policy rollouts elsewhere, and uniform, expert, or mixed-policy rollouts in PointMaze.The mixed PointMaze policy switches between uniform-random and expert actions with probability 0.05 per time step.Source: §A.2.1 · p. 13

A.2.2 TRAINING LEARNED REWARD MODELS

• Reward-model experiments train AIRL, preference-comparison, and regression models on synthetic ground-truth data, using expert demonstrations for AIRL and fresh coverage-distribution samples for the other methods.AIRL used five seeds and selected the seed with the highest ground-truth return, introducing positive bias; regression converged slowly and required 10 × 10^6 time steps for good performance.Source: §A.2.2 · p. 15

A.2.3 COMPUTING INFRASTRUCTURE

• Experiments ran on an Intel i9-7920X workstation and a small number of AWS r5.24xlarge instances; all experiments took less than three weeks on one such instance.Source: §A.2.3 · p. 15

A.2.4 COMPARING HAND-DESIGNED REWARD FUNCTIONS

• EPIC closely matches intuitive reward similarities; NPEC suffers optimization error, while ERC can mislead under shaping despite tight confidence intervals.NPEC often exceeds its theoretical upper bound of 1.0 in HalfCheetah and Hopper, with confidence intervals frequently wider than 0.2. ERC performs reasonably when rewards are similarly shaped, but can assign large distances to equivalent shaped rewards.Source: §A.2.4 · pp. 15–16

A.2.5 COMPARING LEARNED REWARD FUNCTIONS

• EPIC and ERC distance intervals have less than 1% relative error, whereas NPEC intervals can reach 50% because of high variance and few seeds.The intervals are bootstrapped 95% bounds for approximate distances from the ground-truth in PointMaze; limited seeds reflect NPEC’s computational expense.Source: §A.2.5 · p. 16, Table A.7 · p. 21

A.2.6 RUNTIME OF DISTANCE METRICS

• In PointMass, EPIC returns a 95% confidence interval below 0.023 in 17 seconds, while NPEC takes over 8 hours and PPO training takes over 4 hours.ERC is relatively fast but needs many samples for tight intervals and can take arbitrary values under shaping with stochastic initial states.Source: §A.2.6 · p. 16, Table A.6 · p. 16

A.3.1 BACKGROUND

• Reward equivalence is an equivalence relation: it is reflexive, symmetric, and transitive.The relation is defined through positive reward scaling and bounded potential shaping.Source: §A.3.1 · p. 25

A.3.2 EQUIVALENT-POLICY INVARIANT COMPARISON (EPIC) PSEUDOMETRIC

• EPIC is a pseudometric, and canonical shaping makes rewards equal under the canonical transformation.The Pearson distance used in the construction is invariant to positive affine transformations and bounded between 0 and 1.Source: §A.3.2 · pp. 26–27
• EPIC is bounded between 0 and 1, and its value is unchanged when rewards are positively scaled before canonicalization.The proof expresses EPIC as the Pearson distance between canonicalized rewards, with scaling factors removed.Source: §A.3.2 · pp. 27–28

A.3.3 NEAREST POINT IN EQUIVALENCE CLASS (NPEC) PREMETRIC

• NPEC is a premetric invariant to replacing either reward with an equivalent reward, with values between 0 and 1; it is not symmetric in the undiscounted case.When the underlying Lp distance is a metric, NPEC(X, Y) = 0 if and only if X = Y.Source: §A.3.3 · pp. 31–32

A.4 FULL NORMALIZATION VARIANT OF EPIC

• With the L2 norm, the direct distance between fully normalized canonicalized rewards is equivalent to EPIC.Canonical shaping centers rewards, and L2 normalization gives them unit variance.Source: §A.4 · pp. 33–34

A.5 REGRET BOUND

• Regret from using the policy optimal for one reward under another is bounded as a function of EPIC distance, and tends to zero as that distance approaches zero.For finite-state and finite-action MDPs, the analysis assumes the coverage distribution dominates the transition distributions of both rewards’ optimal policies by a factor K.Source: §A.5 · p. 34, §A.5.1 · p. 35

A.5.1 DISCRETE MDPS

• In the discrete-MDP analysis, a coverage distribution uniform over transitions gives K ≤ |S|^2|A|.The bound’s coverage assumption requires K times the coverage probability to dominate each optimal policy’s transition probability at every time step.Source: §A.5.1 · pp. 35–36
• For finite MDPs, Theorem 4.9 establishes a regret bound when the coverage distribution dominates transition distributions induced by the rewards’ optimal policies.The stated setup assumes finite state and action spaces and a coverage factor K > 0 that applies across time and transitions.Source: §A.5.1 · p. 37

A.6 LIPSCHITZ REWARD FUNCTIONS

• For continuous MDPs with bounded Lipschitz rewards, Theorem A.16 bounds regret using EPIC distance plus a relaxed Wasserstein-distance term.The theorem’s bound also depends on coverage of the optimal-policy transition distributions relative to the stationary coverage distribution.Source: §A.6 · pp. 38–39

A.7 LIMITING BEHAVIOR OF REGRET

• Regret tends to zero as EPIC distance tends to zero for finite MDPs and for continuous MDPs with Lipschitz rewards, under the theorem’s respective coverage and regularity assumptions.For the continuous case, the assumptions include a totally bounded transition space, rewards that are Lipschitz, coverage bounded below by δ, and non-degenerate trajectory distributions.Source: §A.7 · pp. 40–41