PaperlayerView exact revision on arXiv

Source-linked AI summary

Goodhart's Law in Reinforcement Learning

Jacek Karwowski, Oliver Hayman, Xingjian Bai, Klaus Kiendlhofer, Charlie Griffin, Joar Skalse

arXiv:2310.09144v1cs.LG

TL;DR

Because real-world tasks are difficult to specify fully in a reward, this paper studies rewards as proxies for true objectives and asks how their optimization can produce Goodharting. It quantifies the effect, explains its geometry in MDPs, and develops optimization methods; the paper reports that Goodharting occurs across varied settings and that its methods provably avoid it.

  • Problem

    Real-world tasks are difficult to capture fully with rewards, while RL lacks rigorous answers about when and how to optimize a reward proxy for the true objective.

    Source: §1 · pp. 1–2

  • Method

    The paper quantifies Goodharting, explains it geometrically using MDP occupancy measures, and derives early-stopping and worst-case-reward optimization methods.

    Source: §2.2 · p. 4, §4 · p. 6, §5 · p. 8

  • Results

    Goodharting occurs across a wide range of environments and reward functions, and the proposed policy-optimization methods provably avoid it.

    Source: §1 · p. 2, §6 · pp. 9–10

  • Takeaways & Limitations

    The methods can yield less true reward than naïve optimization, so they are most applicable when avoiding Goodharting is essential.

    Source: §6 · p. 10

    The study focuses on Goodharting and does not comprehensively characterize other possible failure modes of maximizing misspecified rewards.

    Source: §6 · p. 10

Abstract

from arXiv · show

Implementing a reward function that perfectly captures a complex task in the real world is impractical. As a result, it is often appropriate to think of the reward function as a proxy for the true objective rather than as its definition. We study this phenomenon through the lens of Goodhart's law, which predicts that increasing optimisation of an imperfect proxy beyond some critical point decreases performance on the true objective. First, we propose a way to quantify the magnitude of this effect and show empirically that optimising an imperfect proxy reward often leads to the behaviour predicted by Goodhart's law for a wide range of environments and reward functions. We then provide a geometric explanation for why Goodhart's law occurs in Markov decision processes. We use these theoretical insights to propose an optimal early stopping method that provably avoids the aforementioned pitfall and derive theoretical regret bounds for this method. Moreover, we derive a training method that maximises worst-case reward, for the setting where there is uncertainty about the true reward function. Finally, we evaluate our early stopping method experimentally. Our results support a foundation for a theoretically-principled study of reinforcement learning under reward misspecification.

1 INTRODUCTION

Because real-world tasks are difficult to encode fully in rewards, the paper treats rewards as proxies and asks how optimizing them affects the true objective. It studies this through Goodhart’s law and situates its contribution against prior work on reward misspecification and Goodharting.

1 INTRODUCTION

  • Complex real-world tasks are difficult to capture in a reward, and misspecified rewards can produce undesirable behavior, making reward design a practical obstacle, especially in safety-critical applications.Learned rewards also face data-collection and interpretation challenges, and may fail under the distribution shift induced by policy optimization.Source: §1 · p. 1
  • The paper treats reward as a proxy for the true objective and argues that rigorous answers about when and how to optimize such proxies remain limited in RL.Its motivating questions include when a proxy is safe to maximize, how to maximize a misspecified proxy, and what failure modes to expect.Source: §1 · p. 2
  • Goodhart’s law predicts that proxy optimization can first improve the true objective, then worsen it once the proxy–objective correlation breaks down.The paper’s analogy is students improving test scores through strategies such as cheating without increasing subject understanding.Source: §1 · p. 2
  • The paper reports Goodharting across varied environments and reward pairs, explains its mechanism in RL, and derives policy-optimization methods that provably avoid it.It also evaluates the methods empirically.Source: §1 · p. 2

1.1 RELATED WORK

  • Unlike prior examples of RL susceptibility in certain situations, the paper presents Goodharting as robust across environments and uses that perspective to develop solution methods.Related work also studies reward hacking, but this paper analyzes it through Goodhart’s law.Source: §1.1 · p. 2
  • Prior language-model work found a Goodhart curve whose slope became less prominent with larger reward models and more training data; this paper targets varied RL environments and algorithm design.Gao et al. (2023) studied performance scaling in language models; this paper also aims to explain Goodhart’s law.Source: §1.1 · p. 3

2 PRELIMINARIES

The preliminaries formulate policy performance using MDP returns and occupancy measures, then use reward geometry and regularized optimization to quantify Goodharting. The section defines the normalized drop height as a measure of true-reward loss during proxy optimization.

2 PRELIMINARIES

  • An MDP specifies states, actions, transitions, an initial-state distribution, reward, and discount; policies induce trajectories and expected returns, which are maximized by optimal policies.The paper assumes finite state and action sets and primarily considers rewardless MDPs where the true reward is unknown.Source: §2 · p. 3

2.1 THE CONVEX PERSPECTIVE

  • A policy’s occupancy measure records its discounted state-action frequencies, and the set of possible measures is the convex hull of those for deterministic policies.Because policy return is the inner product of occupancy and reward, maximizing return becomes a linear optimization problem over the occupancy polytope.Source: §2.1 · p. 3

2.2 QUANTIFYING GOODHART’S LAW

  • The projected angle between rewards measures their difference in the MDP’s relevant geometry; angle zero means they induce the same ordering of policies.This projected angle is an instance of a STARC metric introduced by Skalse et al. (2023a).Source: §2.2 · p. 4
  • Optimization pressure is represented by the inverse of regularization strength; increasing pressure raises the regularized policy’s proxy return.The paper examines maximal causal entropy and Boltzmann rationality, using value iteration in both cases.Source: §2.2 · p. 4
  • Normalized drop height (NDH) measures the loss in true reward over proxy optimization, and is nonzero exactly when true and proxy rewards shift from initial correlation to anticorrelation.The paper uses NDH as its simplest measure after finding that more complex measures are strongly correlated with it in experiments.Source: §2.2 · p. 4

3 GOODHARTING IS PERVASIVE IN REINFORCEMENT LEARNING

The experiments test whether true-reward performance declines beyond a proxy-optimization threshold across varied environments, reward types, and optimization pressures. Goodharting appears in a substantial but non-universal share of sampled cases, while individual runs can also show monotonic improvement.

3 GOODHARTING IS PERVASIVE IN REINFORCEMENT LEARNING

  • The study tests whether true reward declines beyond a threshold while training on a proxy across environments, reward functions, and definitions of optimization pressure.The experiments vary state and action counts, terminal states, discounting, reward types, and reward sparsity.Source: §3 · p. 4, §3.2 · p. 5

3.1 ENVIRONMENT AND REWARD TYPES

  • The tested environments include Gridworld and Cliff variants, randomly sampled transition MDPs, and tree MDPs; reward functions are based on next-state rewards.The reward schemes include terminal, cliff, and path rewards.Source: §3.1 · pp. 4–5

3.2 ESTIMATING THE PREVALENCE OF GOODHARTING

  • The sampling procedure sparsifies and interpolates pairs of reward functions to create proxies at varying distances from the true reward.For each sampled MDP, a random fraction of reward values is zeroed before the rewards are linearly interpolated.Source: §3.2 · p. 5
  • A Goodhart drop occurred in 19.3% of 30,400 sampled experiments, indicating the behavior is common but not universal across the tested environments and reward functions.A drop is identified when true reward initially rises and then declines as optimization pressure increases.Source: §3.2 · pp. 5–6
  • Training myopic agents makes Goodharting less severe, according to additional empirical analyses reported in the appendix.Source: §3.2 · p. 6
  • In an illustrative run, true reward under proxy optimization either increased monotonically or rose and then fell, consistent with Goodhart’s law.The accompanying figure shows true rewards for policies trained on proxies of different distances.Source: Figure 2 · p. 5, §3.2 · p. 6

4 EXPLAINING GOODHART’S LAW IN REINFORCEMENT LEARNING

The paper explains Goodharting geometrically: proxy optimisation traces a path along the boundary of a convex policy polytope, where changes in direction can reduce true reward. The risk depends on reward alignment and on how the optimisation path meets the polytope boundary.

4 EXPLAINING GOODHART’S LAW IN REINFORCEMENT LEARNING

  • The paper characterises steepest ascent as a piecewise-linear path and shows that its directional reward component decreases along the path.This helps explain why additional optimisation pressure can make Goodharting more likely.Source: §4 · pp. 6–7
  • When proxy and true reward directions are less than π/2 apart, steepest ascent from an interior policy initially increases true reward, but a boundary projection can later decrease it.The path is piecewise linear along the boundary of the convex polytope, and steepest ascent continues along the proxy direction projected onto that boundary.Source: §4 · p. 6
  • At each new boundary, the optimisation path changes direction; true reward is more at risk when the path meets the boundary near a right angle and less at risk when the rewards are closely aligned.True reward may still increase after one boundary region is reached, before a later change in direction decreases it.Source: §4 · pp. 6–7
  • In a 2-state, 2-action example, optimising proxy R2 initially raised true reward before further optimisation past a critical point lowered it.The paper visualises this mechanism as the training run deflecting at the polytope border, where the optimal deterministic policy changes.Source: §4 · p. 7

5 PREVENTING GOODHARTING BEHAVIOUR

The paper derives a stopping rule for proxy optimisation under a bound on the proxy–true-reward angle, and also gives a regularised objective for maximising worst-case reward. In experiments, the stopping criterion prevented Goodharting in all tested cases, but incurred reward loss.

5 PREVENTING GOODHARTING BEHAVIOUR

  • With a bound θ on the angle between proxy and true reward, Theorem 1 identifies a stopping point that avoids Goodharting for every reward in Fθ.For concave policy optimisation, Corollary 1 makes this stopping point worst-case optimal in a specified sense.Source: §5 · p. 8
  • The paper also characterises a policy that maximises worst-case reward over possible true rewards within the angular bound, using a regularised objective.This objective provides an alternative to selecting a stopping point along a single optimisation curve.Source: §5 · p. 8
  • Pessimistic early stopping can lose true reward when the true reward is among those that would increase if optimisation continued.The paper derives an upper bound on this possible loss.Source: §5 · p. 8
  • Developing optimisation methods that refine an initial reward estimate as ambiguity becomes relevant remains future work, apart from initial exploration in Appendix F.Theorems 1 and 4 could be used to check when more information about the true reward is needed.Source: §5 · p. 8

5.1 EXPERIMENTAL EVALUATION OF EARLY STOPPING

  • Corollary 1's criterion yielded NDH = 0 in all tested cases, while early stopping lost 10%–44% of reward on average across environment sizes and types.The choice between MCE and Boltzmann Rationality had little effect, and smaller θ appeared associated with less reward loss.Source: Figure 5 · p. 9, §5.1 · p. 9

6 DISCUSSION

The discussion presents early stopping as a way to avoid Goodharting, while noting its assumptions, implementation needs, and trade-offs. Its geometric account links Goodharting to how proxy-driven ascent changes direction at the boundary of the occupancy-measure polytope.

6 DISCUSSION

  • Deployment requires occupancy-measure estimates, which may be costly and noisy, and an upper bound on the proxy–true reward angle that may be estimated from training data or expert evaluation.The occupancy measure can also be approximated by solving a transition-matrix equation.Source: §6 · p. 9
  • If the supplied angle bound θ does not upper-bound proxy–true reward mismatch, worst-case Goodharting may match naïve optimisation; nonconcavity or a loose bound can instead cause suboptimal stopping.The latter conditions still guarantee avoidance of Goodharting.Source: §6 · p. 9
  • Early stopping methods provably avoid Goodharting and are most applicable when avoiding it is essential, although they can yield less true reward than naïve optimisation.The discussion also describes them as worst-case optimal in a certain sense.Source: §6 · p. 10
  • The work does not comprehensively characterize proxy-optimization dynamics, other possible failure modes, or exactly when proxy differences make maximization harmful.The discussion identifies collecting more reward information over time and relaxing the methods while retaining guarantees as future directions.Source: §6 · p. 10

A A MORE DETAILED EXPLANATION OF GOODHART’S LAW

  • Goodharting occurs when the proxy-driven path initially raises true reward but later lowers it; this depends on boundary geometry and the angle between projected proxy and true rewards.A smaller projected reward angle makes Goodharting less likely.Source: A A MORE DETAILED EXPLANATION OF… · p. 15
  • As ascent proceeds, the boundary’s normal becomes more aligned with the projected proxy direction, making Goodharting more likely under continued optimisation pressure.The account connects this monotonic angle change to the increasing risk of a true-reward decline.Source: A A MORE DETAILED EXPLANATION OF… · p. 15

B PROOFS

The proofs establish structural properties of occupancy measures and reward alignment, then use them to justify an optimal-stopping objective and a regret bound. They also show that steepest ascent has a concavity property along its path.

B PROOFS

  • The occupancy-measure set Ω is the convex hull of deterministic-policy occupancy measures and lies in a space of dimension |S|(|A|−1).The proof establishes the convex-hull result by reducing stochastic-policy occupancy measures to deterministic ones and measures with a smaller k value.Source: B PROOFS · p. 16
  • A zero projected angle between true and proxy rewards holds exactly when they induce the same ordering of policies.The proof connects policy orderings to the linear return functions over Ω.Source: B PROOFS · pp. 16–17
  • Along steepest ascent, the directional reward of successive steps is nonincreasing.The proposition formalizes a concavity property of the ascent path.Source: B PROOFS · p. 17
  • The optimal-stopping objective maximizes proxy return minus a penalty for occupancy-measure size and its angular misalignment with the proxy; each local maximum on Ω is global.The penalty coefficient is κ = tan(θ)||MτR1||.Source: B PROOFS · p. 18
  • The regret bound combines the reward range over Ω with the ascent progress already made, scaled pessimistically by cos(θ).The reward range is the polytope’s diameter, while progress is measured by the distance between the initial and final occupancy measures.Source: B PROOFS · p. 18

C MEASURING GOODHARTING

The section defines measures of Goodharting using true reward as a function of optimisation pressure, and tests metrics designed to capture reward drops. The authors report strong agreement among the metrics and use normalised drop height as their simplest proxy measure.

C MEASURING GOODHARTING

  • The section defines f(λ) as true reward from a proxy-trained policy at optimisation pressure λ, f0 as the true-reward-trained policy’s true reward, and λ* as the pressure maximizing f.All rewards are assumed normalized.Source: C MEASURING GOODHARTING · p. 19
  • The metrics were independently designed to capture sudden reward drops as optimisation pressure rises, and the evaluation dataset contains 40,000 varied environments.The supplied examples include RandomMDP settings with terminal or uniform rewards.Source: C MEASURING GOODHARTING · p. 19
  • Manual checks found that all metrics strongly activate on intuitively high-Goodharting curves; Figure 7 shows each metric’s three highest-scoring training curves.The curves are selected from the dataset.Source: C MEASURING GOODHARTING · p. 19, Figure 7 · p. 20
  • The metrics are reported as highly correlated, and normalised drop height is used as the simplest Goodharting proxy.Figure 6 presents correlations for examples where a drop occurs beyond λ > 0.3, to avoid selecting adversarial examples.Source: C MEASURING GOODHARTING · p. 19, Figure 6 · p. 20

D EXPERIMENTAL EVALUATION OF THE EARLY STOPPING ALGORITHM

The evaluation includes example runs across environments and examines how Goodharting relates to reward lost through pessimistic stopping. It also reports a full breakdown of reward lost, though the supplied table caption gives no numerical results.

  • The authors present the NDH–reward-loss relationship and full numerical data for Figure 5a as sanity checks on the experiments.Source: D EXPERIMENTAL EVALUATION OF THE EARLY… · p. 21
  • As NDH increases, the potential reward gain from early stopping increases, with a fitted relationship between NDH and reward lost to pessimistic stopping.The caption reports a fitted slope of -1.863 and shows only cases with NDH > 0.Source: Figure 8 · p. 21
  • The evaluation reports a full breakdown of true reward lost by environment type and training method, with 320 datapoints missing because of numerical instability producing NaNs.Source: Table 1 · p. 21
  • Example runs show true reward across increasing optimisation pressures, with a second projection highlighting differences between optimal and pessimistic stopping points.The runs also plot how optimisation angle changes with pressure for each proxy reward.Source: Figure 9 · p. 23

E A SIMPLE EXAMPLE OF GOODHARTING

The example uses a two-state, two-action MDP to illustrate Goodharting with three sampled rewards. Its numerical setup uses a fixed discount and uniform initial-state distribution.

F ITERATIVE IMPROVEMENT

The iterative improvement algorithm alternates proxy-reward updates with policy optimisation under a bound on proxy error. It terminates at a policy optimal for the true reward, and the authors present it as a basis for practical training regimes that avoid Goodharting.

F ITERATIVE IMPROVEMENT

  • The authors propose using an expensive true-reward evaluation to estimate proxy error and choose when to stop proxy training before reward curves diverge.Source: F ITERATIVE IMPROVEMENT · p. 25
  • An oracle supplies increasingly accurate proxy rewards and error bounds; the iterative feedback algorithm is stated to avoid Goodharting and terminate at the true-reward-optimal policy.Source: F ITERATIVE IMPROVEMENT · p. 25
  • Algorithm 1 is proved to terminate at a policy optimal for the true reward.The proof states that the algorithm's inner loop maintains nondecreasing true reward and that termination occurs at the true-reward-optimal policy.Source: F ITERATIVE IMPROVEMENT · p. 25
  • The authors expect the algorithm to provide a theoretical basis for a practical training regime that avoids Goodharting.Source: F ITERATIVE IMPROVEMENT · p. 25

G.1 ADDITIONAL PLOTS FOR THE GOODHARTING PREVALENCE EXPERIMENT

Additional plots report that the operationalisation of optimisation pressure does not materially change results and examine how MDP parameters relate to NDH. The appendices also discuss projection computation and report compute resources.

G.1 ADDITIONAL PLOTS FOR THE GOODHARTING PREVALENCE EXPERIMENT

  • Figure 12 reports that changing the operationalisation of optimisation pressure does not significantly change results, while NDH values above zero are roughly exponentially distributed.Source: Figure 12 · p. 26

G.2 EXAMINING THE IMPACT OF KEY ENVIRONMENT PARAMETERS ON GOODHARTING

  • The parameter study varies one MDP dimension at a time because the original dataset has too few observations per dimension and confounded parameter correlations.Its base distribution samples state counts from 8–64, action counts from 2–16, and terminal states from 1–4.Source: §G.2 · p. 26
  • NDH is reported to rise with proxy–true reward distance and discount factor γ, while more actions appear to affect it and state count and transition determinism do not seem to.The authors caution that environment-type correlations may be spurious.Source: §G.2 · p. 26, Figure 15 · p. 27, Figure 13 · p. 27

H.1 COMPUTING THE PROJECTION MATRIX

  • The direct projection matrix maps rewards onto the hyperplane containing valid policies, but is too computationally expensive for environments with many states.An alternative using potential functions is described but not implemented; it aims to produce the same projection by making shaped rewards orthogonal to the relevant reward vectors.Source: §H.1 · p. 29

H.2 COMPUTE RESOURCES

  • The large-scale experiments used about 100 hours each on a 64-core c5a.16xlarge instance and an 8-core t2.2xlarge instance.The reported memory was 128 GB and 32 GB, respectively.Source: §H.2 · p. 29

I AN ADDITIONAL EXAMPLE OF THE PHASE SHIFT DYNAMICS

A 3-state, 2-action MDP exposes multiple changes in optimization direction; across five proxy rewards, the example shows concave Goodharting and illustrates how early stopping responds to a critical angle.

I AN ADDITIONAL EXAMPLE OF THE PHASE SHIFT DYNAMICS

Loading 2310.09144v1…