PaperlayerView exact revision on arXiv

Source-linked AI summary

STARC: A General Framework For Quantifying Differences Between Reward Functions

Joar Skalse, Lucy Farnik, Sumeet Ramesh Motwani, Erik Jenner, Adam Gleave, Alessandro Abate

arXiv:2309.15257v3cs.LGcs.AI

TL;DR

Reward learning is used when intended rewards are difficult to specify, but comparing a learned reward with the true reward is difficult to do in a principled way. The paper introduces STARC metrics, which standardize reward functions before measuring their distance. It shows that these metrics bound worst-case regret from both above and below, and that metrics with the same guarantees are bilipschitz equivalent.

  • Problem

    Comparing learned and true rewards is difficult: L2 distance can misrepresent policy ordering, test sets only support guarantees on their data distribution, and policy-optimization evaluation is costly and empirical.

    Source: §1 · p. 2

  • Method

    STARC standardizes reward functions that differ by potential shaping and S1-redistribution, ignores positive scaling, and then measures distance.

    Source: §2.1 · pp. 4–5

  • Results

    STARC metrics induce both upper and lower bounds on worst-case regret, and any pseudometric with both guarantees is bilipschitz equivalent to them.

    Source: §3 · p. 7

  • Takeaways & Limitations

    Their closed-form computation and low cost make STARC metrics suitable for empirical evaluation and theoretical analysis of reward learning methods.

    Source: §1 · p. 2

    The theoretical results assume finite state and action spaces and an MDP setting; experiments are indicative but not conclusive about which STARC metrics work best.

    Source: §5 · p. 9

Abstract

from arXiv · show

In order to solve a task using reinforcement learning, it is necessary to first formalise the goal of that task as a reward function. However, for many real-world tasks, it is very difficult to manually specify a reward function that never incentivises undesirable behaviour. As a result, it is increasingly popular to use reward learning algorithms, which attempt to learn a reward function from data. However, the theoretical foundations of reward learning are not yet well-developed. In particular, it is typically not known when a given reward learning algorithm with high probability will learn a reward function that is safe to optimise. This means that reward learning algorithms generally must be evaluated empirically, which is expensive, and that their failure modes are difficult to anticipate in advance. One of the roadblocks to deriving better theoretical guarantees is the lack of good methods for quantifying the difference between reward functions. In this paper we provide a solution to this problem, in the form of a class of pseudometrics on the space of all reward functions that we call STARC (STAndardised Reward Comparison) metrics. We show that STARC metrics induce both an upper and a lower bound on worst-case regret, which implies that our metrics are tight, and that any metric with the same properties must be bilipschitz equivalent to ours. Moreover, we also identify a number of issues with reward metrics proposed by earlier works. Finally, we evaluate our metrics empirically, to demonstrate their practical efficacy. STARC metrics can be used to make both theoretical and empirical analysis of reward learning algorithms both easier and more principled.

1 INTRODUCTION

Reward learning is increasingly used when intended rewards are difficult to specify, but evaluating learned rewards against true rewards is not straightforward. The paper introduces STARC metrics to compare reward functions and establish their theoretical properties relative to earlier metrics.

1 INTRODUCTION

  • Reward learning can use trajectory preferences, expert demonstrations, or both, but L2 distance, test-set evaluation, and policy optimization each leave important evaluation problems.L2 distance can misrepresent policy ordering; test sets are distribution-specific, while policy-optimization evaluation is expensive and purely empirical.Source: §1 · pp. 1–2
  • STARC is a family of reward-function pseudometrics with upper and lower worst-case-regret bounds, making small distances track similar policy behavior and requiring equivalent metrics to be bilipschitz equivalent.The authors also report compelling empirical performance and say the metrics are cheap to compute and have a closed-form expression.Source: §1 · p. 2

1.1 RELATED WORK

  • EPIC (Gleave et al., 2020) bounds regret for optimal policies, while DARD (Wulfe et al., 2022) incorporates transition dynamics but derives no regret bound.The authors say their bound covers all policy pairs and is stronger and more general than EPIC's; their experiments report tighter correlation with worst-case regret than EPIC and DARD.Source: §1.1 · pp. 2–3

1.2 PRELIMINARIES

  • Policy ordering is defined through the evaluation function J, and rewards with the same ordering are characterized by potential shaping, positive linear scaling, and S1-redistribution.The paper defines these reward transformations and states that they preserve policy ordering in combination.Source: §1.2 · pp. 3–4
  • EPIC compares standardized reward functions using a Pearson distance, whereas DARD uses transition dynamics in its reward transformation and distance definition.Both metrics are parameterized by state and action distributions; DARD additionally depends on the transition function.Source: §1.2 · p. 4

2 STARC METRICS

STARC metrics standardise rewards by collapsing reward functions that differ through policy-order-preserving transformations, normalising scale, and then measuring distance. The section gives concrete construction choices and explains how the framework applies when transition dynamics are unknown or environments are continuous.

2.1 A FORMAL DEFINITION OF STARC METRICS

  • STARC metrics first canonicalise rewards differing by potential shaping and S1-redistribution, then normalise positive scaling before measuring distance.The definitions are parameterised by the environment and apply to choices satisfying the paper’s assumptions.Source: §2.1 · pp. 4–5

2.2 EXAMPLES OF STARC METRICS

  • For any policy π, Value-Adjusted Levelling (VAL) is a canonicalisation function that is easy to evaluate when Vπ is easy to approximate.The section also notes that the choice of canonicalisation, norm, and admissible metric can depend on computational and theoretical convenience and regret tracking.Source: §2.2 · pp. 5–6
  • Minimal canonicalisation functions yield tighter regret bounds; for every weighted L2-norm, one exists and is unique.Minimality means choosing the representative with the smallest norm among rewards equivalent under potential shaping and S1-redistribution.Source: §2.2 · p. 5
  • The normalization norm may be any norm on Im(c), including L1, L2, and L∞; maxπ J(π) − minπ J(π) is also such a norm.The paper permits other admissible metrics for the final distance step as well.Source: §2.2 · pp. 5–6

2.3 UNKNOWN TRANSITION DYNAMICS AND CONTINUOUS ENVIRONMENTS

  • Although STARC metrics depend on transition dynamics τ, VAL can be computed from transition samples, and STARC metrics remain straightforward to use in continuous environments.The theoretical results assume finite state and action sets.Source: §2.3 · p. 6

3 THEORETICAL RESULTS

STARC metrics are pseudometrics whose distance is zero exactly when two rewards induce the same policy ordering. They bound worst-case regret from both above and below, making small distance necessary and sufficient for low regret.

3 THEORETICAL RESULTS

  • STARC metrics are pseudometrics, and assign distance zero exactly to rewards with the same ordering of policies.This gives the distance a policy-ordering interpretation rather than treating raw reward differences as equivalence.Source: §3 · p. 6
  • Soundness gives every STARC metric an upper bound on worst-case regret, including when arbitrary policies are compared.The arbitrary-policy setting includes the special case of comparing optimal policies.Source: §3 · p. 7
  • Completeness supplies a lower bound, so small STARC distance is necessary as well as sufficient for low worst-case regret.The definition includes a condition intended to rule out pathological edge cases.Source: §3 · p. 7
  • Any pseudometrics that are both sound and complete are bilipschitz equivalent, including all STARC metrics and any other metrics with both regret bounds.Thus the guarantees constrain the possible metric up to bilipschitz equivalence.Source: §3 · p. 7
  • The paper reports that EPIC and DARD fail to provide theoretical guarantees similar to those of STARC metrics.The paper describes further analysis of both metrics in Appendices A and B.Source: §3 · p. 7

4 EXPERIMENTAL RESULTS

The experiments compare STARC metrics with existing pseudometrics in random MDPs and estimate distances among hand-crafted rewards in Reacher. VAL-based STARC metrics correlate more strongly with regret than the tested alternatives, while the Reacher estimates preserve the expected reward ordering despite inflated magnitudes.

4.1 LARGE NUMBERS OF SMALL RANDOM MDPS

  • The random-MDP experiment compares 223 pseudometrics using 49,152 reward-function pairs to estimate correlation with regret.The MDPs have 32 states and 4 actions, with γ = 0.95; they use uniformly distributed initial states and randomly sampled sparse, nondeterministic transitions.Source: §4.1 · pp. 7–8
  • VAL is the only tested canonicalisation that removes both potential shaping and S1-redistribution, and it is the one identified as STARC-VAL.The experiment also includes None, CEPIC, CDARD, MinimalPotential, and VALPotential canonicalisations.Source: §4.1 · p. 8
  • 0.856 for VAL versus 0.778 for EPIC and 0.782 for DARD (L2 normalization and distance); VAL beats EPIC in 40 of 42 settings.Among the ten best pseudometrics, eight use VAL and two use VALPotential.Source: Figure 1 · p. 8, §4.1 · p. 8
  • With VAL canonicalisation, L1 for both normalization and distance raises regret correlation from 0.856 to 0.873.For EPIC, weighted L8 normalization with unweighted L8 distance lowers correlation from 0.778 to 0.052, compared with L2 for both; VALPotential can perform as well as VAL with a τ-weighted norm.Source: §4.1 · p. 8

4.2 THE REACHER ENVIRONMENT

  • In Reacher, estimated distances preserve the expected relative ordering of GroundTruth, PotentialShaped, SecondPeak, Random, and Negative rewards.GroundTruth and PotentialShaped have an actual distance of zero but an estimated distance of about 0.9; the authors attribute inflated magnitudes likely to summing absolute values, while noting this does not fundamentally impair ranking.Source: §4.2 · p. 9, Table 1 · p. 9

5 DISCUSSION

The discussion presents STARC metrics as theoretically tight and empirically promising tools for comparing rewards. It also identifies open questions about metric choice and extending the theory beyond finite MDP settings.

5 DISCUSSION

  • STARC metrics are sound and complete, and any pseudometric with both guarantees is bilipschitz equivalent to them.The paper also reports better empirical performance than existing pseudometrics across a wide range of environments and describes the metrics as easy to compute and work with mathematically.Source: §5 · p. 9
  • The empirical results are indicative but not conclusive about which STARC metrics work best in practice.The paper also leaves open generalisation to continuous state and action spaces, weaker regret criteria, and environment classes beyond MDPs, including multi-agent settings.Source: §5 · p. 9

A COMMENTS ON EPIC

The appendix argues that EPIC’s regret guarantees do not match the form of regret the authors consider practically relevant. It reports that EPIC is unsound and cannot provide a lower bound on worst-case regret for almost any definition considered.

A COMMENTS ON EPIC

  • EPIC is not sound for any choice of τ or µ0 under the regret bound in Definition 5.Theorem 3’s bound concerns a different form of regret.Source: A COMMENTS ON EPIC · p. 12
  • The authors favor regret measured relative to the range of achievable reward over an absolute reward difference or L2 norm.The absolute difference depends on reward scale, while the range captures how much reward could be lost relative to what could be achieved.Source: A COMMENTS ON EPIC · p. 12
  • EPIC distance can be positive even when two rewards induce the same policy ordering, so it gives no lower bound on worst-case regret.The counterexample holds for any choice of τ or µ0.Source: A COMMENTS ON EPIC · p. 12
  • The appendix concludes that EPIC lacks the desired theoretical guarantees, including a lower bound for almost any considered definition of regret.Theorem 3’s upper bound does not establish soundness under Definition 5.Source: A COMMENTS ON EPIC · p. 12

B COMMENTS ON DARD

The appendix identifies limitations in DARD’s canonicalisation and theoretical guarantees. Its examples show that DARD can distinguish rewards with identical policy orderings, including through rewards assigned to impossible transitions.

B COMMENTS ON DARD

  • DARD’s canonicalisation CDARD is designed to remove potential shaping, but the authors note that R and CDARD(R) need not differ by potential shaping.The canonicalisation term involving γR(S1, A, S2) depends on both s and s1.Source: B COMMENTS ON DARD · p. 12
  • It is unclear whether CDARD(R) preserves policy ordering, and applying CDARD again may produce a different result.The authors describe this lack of monotonicity as undesirable.Source: B COMMENTS ON DARD · p. 13
  • DARD’s regret upper bound remains unestablished, and the authors warn that bounded DARD distance could coexist with unbounded regret.They note that a nonzero expected value of CDARD(R) does not itself settle whether DARD has an upper bound.Source: B COMMENTS ON DARD · p. 13
  • DARD can be influenced by rewards assigned to transitions that are impossible under the environment dynamics.The final term of CDARD(R) can depend on impossible transitions.Source: B COMMENTS ON DARD · p. 13
  • DARD can assign positive distance to reward functions that induce exactly the same policy ordering.The example assigns zero rewards to possible transitions and differing rewards to impossible transitions.Source: B COMMENTS ON DARD · p. 13
  • The authors argue that STARC offers a better way to incorporate transition-dynamics information into a reward metric.They connect this preference to their results in Section 3 and concerns that DARD may yield misleading measurements.Source: B COMMENTS ON DARD · p. 13

C A GEOMETRIC INTUITION FOR STARC METRICS

The geometric intuition presents reward functions as vectors and treats policy-equivalent transformations as directions to remove. After canonicalisation and normalisation, STARC distance corresponds to an angle that tracks worst-case regret.

C A GEOMETRIC INTUITION FOR STARC METRICS

  • Rewards differing by potential shaping and S1-redistribution induce the same policy ordering, and those transformations form a linear subspace of reward space.Reward functions equivalent under these transformations lie in an affine subspace relative to a given reward.Source: C A GEOMETRIC INTUITION FOR STARC… · p. 13
  • A canonicalisation function removes the dimensions associated with potential shaping and S1-redistribution, mapping rewards to a lower-dimensional subspace.Its null space is the set of reward functions corresponding to those transformations; the L2-minimal choice is orthogonal.Source: C A GEOMETRIC INTUITION FOR STARC… · p. 13
  • Normalising canonicalised rewards places them on the surface of a norm’s unit ball, where the metric measures their distance.The zero reward remains at the origin.Source: C A GEOMETRIC INTUITION FOR STARC… · p. 13
  • The non-sequential utility example illustrates normalising away affine-equivalent utilities, but the analogy is not exact because not all trajectory distributions are realisable by policies in an MDP.The paper relates affine transformations of trajectory returns to potential shaping and positive linear scaling of rewards.Source: C A GEOMETRIC INTUITION FOR STARC… · p. 14
  • For a policy π, occupancy measures make policy value linear in rewards: J(π) = ηπ · R.The set Ω of occupancy measures is a polytope, and reward functions define linear functions over it.Source: C A GEOMETRIC INTUITION FOR STARC… · pp. 14–15
  • STARC metrics are bilipschitz equivalent to the angle between reward functions projected onto the occupancy-measure set Ω, which connects their distance to worst-case regret.The authors say worst-case regret should be proportional to this angle.Source: C A GEOMETRIC INTUITION FOR STARC… · p. 15

D APPROXIMATING STARC METRICS IN LARGE ENVIRONMENTS

The appendix describes sampling-based approximations for STARC metrics in large or continuous environments. It outlines estimating canonicalisation and normalisation from sampled transitions, and gives a Pearson-distance alternative under a centering condition.

D APPROXIMATING STARC METRICS IN LARGE ENVIRONMENTS

  • STARC metrics can be computed exactly in small MDPs in time polynomial in |S| and |A|, while larger environments call for approximation.The discussion includes continuous environments.Source: D APPROXIMATING STARC METRICS IN LARGE… · p. 15
  • The VAL canonicalisation can be approximated by keeping a policy fixed, estimating its value function, and sampling transitions to estimate the canonicalised reward.Direct access to τ is unnecessary; sampling from it and approximating Vπ suffice.Source: D APPROXIMATING STARC METRICS IN LARGE… · p. 15
  • The Lp normalisation and distance steps can be estimated by sampling random transitions to approximate the required expectations, including with continuous state and action spaces.The derivation uses cancellation of the n^(1/p) terms.Source: D APPROXIMATING STARC METRICS IN LARGE… · p. 15
  • A Pearson-distance alternative is available when rewards are centered under D, and its correlation can be estimated through sampling.The centering condition can be ensured by subtracting the expected canonicalised reward under D.Source: D APPROXIMATING STARC METRICS IN LARGE… · p. 16

E PROOFS OF MISCELLANEOUS CLAIMS

This appendix section proves claims about reward canonicalisation and EPIC equivalence, and presents proofs of STARC soundness, completeness, related metric results, and other theorems. The completeness arguments use geometric and linear-transformation bounds on angles to relate standardized rewards to STARC distance and establish a worst-case regret lower bound, treating the trivial-comparator case separately.

E PROOFS OF MISCELLANEOUS CLAIMS

  • For any policy, Proposition 9 establishes that c is a canonicalisation function: it is linear and identifies rewards exactly up to potential shaping and S1-redistribution.The proof notes that Vπ may be computed in an environment distinct from the training environment.Source: E PROOFS OF MISCELLANEOUS CLAIMS · pp. 16–17
  • For any weighted L2-norm, Proposition 10 establishes that a minimal canonicalisation function exists and is unique.The proof constructs it by projecting onto the orthogonal complement of rewards differing by potential shaping and S1-redistribution; it makes no claim about existence or uniqueness for other norms.Source: E PROOFS OF MISCELLANEOUS CLAIMS · p. 17
  • For any canonicalisation function, Proposition 11 establishes that the range of policy returns, maxπ J(π) − minπ J(π), is a norm on its image.The paper notes that this permits normalising rewards so this quantity equals 1.Source: E PROOFS OF MISCELLANEOUS CLAIMS · pp. 17–18
  • Proposition 12 expresses EPIC as a weighted L2 distance between normalized CEPIC rewards, provided the coverage distribution matches the distribution used to compute CEPIC.The paper says Gleave et al. (2020) also asserted this equivalence but did not provide a proof.Source: E PROOFS OF MISCELLANEOUS CLAIMS · pp. 18–19

F MAIN PROOFS

  • Appendix F divides the remaining proofs among STARC soundness, STARC completeness, results concerning EPIC and similar metrics, and remaining theorems.Source: F MAIN PROOFS · p. 19

F.1 SOUNDNESS

  • In the soundness proofs, reward L8-distance bounds differences in policy evaluation, and finite-dimensional norm equivalence extends the bound to any norm.Lemma 2 states that for any norm p, a positive constant Kp bounds policy-evaluation differences by Kp times p(R1, R2).Source: §F.1 · p. 20
  • Lemma 4 establishes that every linear map on the finite-dimensional reward space is bounded with respect to any norm.For linear c and norm n, a positive constant Kn satisfies n(c(v)) ≤ Kn · n(v) for all v.Source: §F.1 · p. 21
  • Theorem 1 establishes that every STARC metric is sound.The proof derives a bound using standardization and norm comparisons.Source: §F.1 · p. 22

F.2 COMPLETENESS

  • For the boundary of a bounded convex set containing the origin in its interior, Lemma 7 gives a positive lower bound on the angle between x and y − x for distinct boundary points.The proof constructs a bound from the set’s inner and outer radii.Source: §F.2 · pp. 22, 24
  • For any STARC metric, Lemma 8 bounds the angle between nonzero standardized rewards below by a positive constant times their STARC distance.The result applies when neither standardized reward is zero.Source: §F.2 · p. 24
  • Lemma 9 shows that any invertible linear transformation reduces the angle between vectors by at most a positive constant factor.The proof establishes the bound in two dimensions and extends it to general dimension.Source: §F.2 · pp. 27, 29
  • Theorem 2 establishes that every STARC metric is complete.The proof also requires distance zero when both rewards induce no variation in policy value.Source: §F.2 · p. 29
  • For nontrivial rewards, policies tied under J2 can incur worst-case regret under J1 bounded below in proportion to d(R1,R2).The proof represents J(π) as m(π)·R and uses projected level-set angles and an open sphere in Im(m); the bound scales with reward range and constants εℓ1ℓ2ℓ3(2/π).Source: §F.2 · pp. 30–31
  • When R2 is trivial but R1 is not, worst-case regret equals the J1 policy-value range, so the bound holds if L·d(R1,R0)≤1.The proof notes that J2 is constant across policies in this case.Source: §F.2 · p. 31

F.3 ISSUES WITH EPIC, AND SIMILAR METRICS

The section establishes a regret bound for EPIC-like metrics, then shows limits of their soundness and policy-order fidelity. Its counterexamples show that maximally high regret can coexist with arbitrarily small distance, and positive distance can persist between rewards with identical policy ordering across environments.

F.3 ISSUES WITH EPIC, AND SIMILAR METRICS

  • For any EPIC-like metric, a constant U bounds worst-case regret by the metric distance for policy pairs ordered under the comparison reward, independently of transition dynamics and initial-state distribution.Source: §F.3 · p. 32
  • Simple STARC metrics can assign arbitrarily small distance to reward pairs with maximally high regret, so no EPIC-like metric is sound.Source: §F.3 · pp. 32–33
  • EPIC-like metrics can give positive distance to reward functions that induce the same policy ordering under every transition function and initial-state distribution.Source: §F.3 · p. 33

F.4 OTHER PROOFS

The appendix proves key properties of STARC metrics, including their relation to policy ordering and the equivalence of sound, complete pseudometrics. It also details the small-MDP setup and rollout-regret calculation used in the experiments.

F.4 OTHER PROOFS

  • STARC metrics are pseudometrics, and zero distance holds exactly when two rewards induce the same policy ordering.The proof checks zero self-distance, symmetry, and the triangle inequality.Source: §F.4 · pp. 33–34
  • Any two sound and complete pseudometrics on the reward space are bilipschitz equivalent.The appendix also proves that one STARC metric is bounded above by another.Source: §F.4 · p. 34

G EXPERIMENTAL SETUP OF SMALL MDPS

  • The small-MDP experiments use 32 states, 4 actions, discount factor 0.95, and a uniform initial-state distribution.Transitions are made sparse by setting some sampled matrix entries to -20 before applying softmax.Source: §G.1 · p. 35

G.1 ENVIRONMENTS AND REWARDS

  • Each transition distribution is tested on 256 reward functions formed from 16 pairs and 16 interpolation steps per pair.Interpolation provides closer reward pairs and a wider range of distance and regret values.Source: §G.1 · p. 35

G.2 ROLLOUT REGRET

  • Rollout regret averages two cross-evaluations: using each reward's optimal policy under the other reward, normalized by the possible-return range.Optimal and anti-optimal deterministic policies are found with value iteration; returns are estimated from 32 episodes.Source: §G.2 · pp. 35–36

G.3 LIST OF METRICS

The appendix specifies the metric variants and reward-pair count used in the small-MDP experiments, and identifies Table 2 as the repository of full results.

G.3 LIST OF METRICS

  • The experiments combine six pseudo-canonicalisations, seven normalisations, and six distance norms, with VAL the only canonicalisation listed that removes both potential shaping and S1-redistribution.Minimal canonicalisation functions are excluded because they are prohibitively expensive to compute and are better suited to theoretical analysis.Source: §G.3 · p. 36

G.4 NUMBER OF REWARD PAIRS

  • The experiment uses 49,152 reward pairs, selected in advance as its stopping point.This corresponds to 96 CPU cores, two environments per core, 16 reward pairs per environment, and 16 interpolation steps.Source: §G.4 · p. 36

H FULL RESULTS OF SMALL MDP EXPERIMENTS

H.1 COMPARISON OF EXPERIMENTAL PERFORMANCE BASED ON CHOICE OF NORMS

The appendix presents the same experimental data reorganized to compare normalization and metric choices within each canonicalisation. It notes that these choices can make a noticeable difference to performance.

H.1 COMPARISON OF EXPERIMENTAL PERFORMANCE BASED ON CHOICE OF NORMS

  • The authors report that normalization and metric choices can make a noticeable difference to reward-metric performance.The appendix reorganizes the same data to make those choices easier to compare.Source: §H.1 · p. 41
  • Tables 3–8 organize correlations to regret by canonicalisation, with normalization functions in rows and metric functions in columns.The tables cover None, EPIC, DARD, MinimalPotential, VALPotential, and VAL canonicalisations.Source: Table 3 · p. 41, Table 4 · p. 41, Table 5 · p. 41

I EXPERIMENTAL SETUP OF REACHER ENVIRONMENT

The appendix describes the Reacher reward variants and the continuous-setting procedures for canonicalisation and norm estimation. The experiment uses seven reward functions and a discount rate of 0.99.

I EXPERIMENTAL SETUP OF REACHER ENVIRONMENT

I.1 REWARD FUNCTIONS

  • PotentialShaped adds potential-based shaping to GroundTruth, while SPrime uses GroundTruth for expected transitions and Random otherwise.PotentialShaped uses a randomly generated but fixed potential function; SPrime treats transitions within 1% or 0.01 as matching.Source: §I.1 · p. 42
  • SecondPeak adds a smaller secondary target, SemanticallyIdentical changes the target peak to a rounded Gaussian, and NegativeGroundReward negates GroundTruth.Random returns deterministic values generated from randomly sampled weights.Source: §I.1 · pp. 42–43

I.2 CANONICALISING AND NORMALISING IN CONTINUOUS SETTINGS

  • Continuous-setting norms are estimated by Monte Carlo sampling, with states and actions sampled uniformly and the next state set to the transition outcome.This removes impossible transitions and reduces the space from 22 dimensions to 12; for L8, the maximum sampled absolute value is used.Source: §I.2 · p. 43

Related papers

Loading 2309.15257v3…