On Goodhart's law, with an application to value alignment

1 Introduction

• The paper models a measure as M = G + ξ and studies how correlation between the measure and true goal changes among increasingly extreme values of M.Its analysis asks whether optimizing M preserves a positive association with G.Source: §1 · p. 2
• Long-tailed discrepancies can produce strong Goodhart effects: power-law discrepancy tails thicker than the goal’s yield negative correlation between M and G, making optimization harmful.Normally distributed discrepancies produce only weak Goodhart, with correlation approaching zero extremely slowly.Source: §1 · p. 2
• When discrepancy tails depend on the goal, over-optimizing the measure can drive the goal to infinitely negative values if the thickest tails occur at infinitely negative goals.The paper calls this an infinitely harmful or infinite Goodhart effect.Source: §1 · p. 2
• The paper connects Goodhart’s law to alignment, where vague preferences are replaced by proxies, and warns that algorithmic deployment can trigger harmful effects through environment feedback loops.A simplified user–algorithm simulation illustrates strong Goodhart and exposes the limits of fixed-dataset testing.Source: §1 · p. 2, Related works · p. 3

Related works

• Related work places Goodhart’s law alongside overfitting, alignment, market impact, and long-tailed phenomena, while identifying this paper’s contribution as a mathematical link to discrepancy tails.The authors also state that their discussion does not fully explain overfitting.Source: Related works · p. 3

Structure of the paper

• The paper proceeds from a model and informal analysis to formal results, alignment consequences, practical examples, future research directions, and a conclusion.Its motivating scope includes metrics-based decisions across economics, management, public policy, education, and recommendation systems.Source: §1 · p. 1, Structure of the paper · p. 4

2 Model

• The model treats M as a proxy for an often unknown or unusable true score G over candidate choices ω.Examples include scientific quality proxied by citations and out-of-sample performance proxied by training-set measures.Source: §2 · p. 4
• Correlation between M and G is formalized over a probability distribution on candidates, including potentially sophisticated distributions of learned models.The framework permits random models produced by initialization and optimization procedures.Source: §2 · p. 4
• The discrepancy ξ = M − G is assumed independent of G except in the paper’s worst-case analysis, with ε measuring noise relative to goal variance.Under this baseline assumption, the unrestricted correlation is determined by the noise-to-signal ratio.Source: §2 · p. 5

2.1 When the measure becomes a target

• Selecting only the top α quantile of M creates selection bias, so the paper studies the conditional correlation ρα as α → 0.Weak Goodhart means near-zero correlation, strong Goodhart means negative correlation, and infinite Goodhart means the conditional expected goal tends to −∞.Source: §2.1 · p. 5
• The behavior of ρα depends on the tail distributions of both G and ξ, particularly their upper-tail probabilities at large values.This tail behavior determines whether optimizing the proxy remains useful or becomes harmful.Source: §2.1 · p. 5

2.2 Key finding

• Normally distributed discrepancies produce only weak Goodhart: correlation tends toward zero, but very slowly, as the proxy is maximized.The paper identifies large-deviation behavior of ξ as the governing factor.Source: §2.2 · p. 5
• Thicker-tailed discrepancies, especially power laws, can make M negatively correlated with G among top candidates, so over-optimizing M becomes detrimental.The paper notes that selecting too small a top fraction can be counter-productive, with the threshold depending on baseline correlation.Source: §2.2 · p. 5
• If discrepancy-tail thickness depends on G, over-optimizing M can be infinitely harmful, with Eα[G] tending to −∞.The paper connects this risk to feedback loops and heavy-tailed discrepancies in algorithms that modify their environments.Source: §2.2 · p. 6

3.1 Bounded goal, exponential discrepancy

• For exponential discrepancy, optimizing M initially improves G but eventually plateaus, leaving zero correlation without decreasing the goal—a weak Goodhart situation.The plateau can be as low as 1/2 for large ε, but remains of order 1 − O(ε) as ε → 0.Source: §3.1 · p. 6

3.2 Normal distributions

• For normally distributed G and ξ, the correlation among top M values tends to zero extremely slowly, remaining positive over a broad optimization range.The paper characterizes this as a very weak Goodhart’s law.Source: §3.2 · p. 7

3.3 Bounded goal, power law discrepancy

• With a bounded uniform goal and power-law discrepancy, optimizing M first raises expected G but eventually becomes counter-productive, returning toward the no-optimization value.The power-law model assumes β > 3, and the strong effect occurs for sufficiently small α, including a range roughly between ε and ε^(β−1).Source: §3.3 · pp. 8, 10, Figure 3 · p. 9
• In the same power-law setting, correlation declines from near 1, becomes negative, and eventually returns toward zero as optimization intensifies.For β ≈ 4, correlation among top candidates can reach −1/2 under roughly equally selective filtering.Source: Figure 4 · p. 9, §3.3 · p. 10

3.4 Power law goal, power law discrepancy

• When both goal and discrepancy have power-law tails, M remains excellent if the discrepancy tail is thinner than the goal’s, but becomes negatively correlated and strongly harmful when it is much thicker.The strong Goodhart effect in this setting appears only much later than α ≈ ε, around log2(α) ≈ 22 in the illustrated case.Source: §3.4 · pp. 10–11

3.5 Bounded goal, log-normal discrepancy

• For log-normal discrepancy, numerical computations suggest a double descent in expected G: it rises, falls, and rises again to exceed the post-initial-increase level.The authors could not analyze this setting theoretically, and question the numerical reliability and robustness for smaller ε.Source: §3.5 · p. 11, Figure 6 · p. 12

3.6 Worst-case Goodhart’s law

• Allowing discrepancy tails to depend on G yields settings where maximizing M drives the expected goal to −∞, even with centered conditional noise and fixed conditional variance ε^2.The construction makes the discrepancy tail thicker for increasingly negative goal values, so over-optimization favors those undesirable outcomes.Source: §3.6 · p. 12

4.1 Illustrative example

• Feedback from recommendations can alter user preferences, creating an interaction setting where strong Goodhart effects may arise.The illustrative model uses content recommendations, intrinsic preferences, and addiction-biased declared preferences.Source: §4.1 · p. 13
• With the measure underestimating addiction, the goal–measure correlation was 0.8, yet the highest measured outcome was associated with a remarkably low goal value.The measure used αM = 50 while the goal used αG = 5; Figure 7 depicts the correlation and discrepancy distribution.Source: §4.1 · p. 13, Figure 7 · p. 14
• The example is suggestive rather than compelling evidence, but it indicates that dangerous over-optimization cannot be excluded when feedback loops are present.The authors explicitly qualify the practical evidential strength of the example.Source: §4.1 · p. 13

4.2 On the hardness of robust alignment

• Fixed test measures can differ from deployed performance because algorithms modify their environments and thereby shift the distribution of future data.Deployed performance depends both on task accuracy and on the distributional shift induced by the algorithm.Source: §4.2 · p. 14
• Environment feedback can produce power-law discrepancies, opening the possibility that over-optimizing a measure becomes extremely counter-productive to the goal.This risk does not require the algorithm to learn from future data, although learning from adversarial inputs may aggravate it.Source: §4.2 · pp. 14–15
• The harmfulness of proxy optimization may depend more on which tail of the true goal is thicker than on the proxy itself.The positive discrepancy tail can correspond essentially to the negative tail of the goal distribution.Source: §4.2 · p. 15
• When the goal and discrepancy are probably dependent, the theorem-based discussion is speculative, so correlation alone is not strong evidence of safe proxy optimization.The authors frame robust alignment as especially difficult in complex environments with feedback loops.Source: §4.2 · p. 15

Understanding the log-normal discrepancy case

• For uniform goals with log-normal discrepancies, simulations suggest double descent, but the phenomenon and its onset lack a mathematical proof.Predicting overfitting and explaining double descent remain unresolved applications.Source: §5 · pp. 15–16

More complete analysis of Goodhart’s law

• The broader theory covers only five tail settings and mainly asymptotic regimes, leaving non-asymptotic Goodhart effects for future analysis.Other open directions include tail-skewed goals, biased training data, poisoning, and heavy-tailed learning.Source: §5 · p. 16

Diagnosing strong Goodhart’s law

• Estimating the discrepancy tail and testing its independence from the goal remain difficult because samples may be scarce or biased.The authors characterize this as an open question with practical consequences.Source: §5 · p. 16

Early stopping

• Early stopping near the ε-quantile may be safer under Goodhart risk, but robust stopping criteria are difficult because the true goal is unknown.The paper asks whether this observation is robust and how to identify the point where optimization becomes counter-productive.Source: §5 · pp. 16–17

Combining measures

• Averaging independent measures can reduce typical error, but remains vulnerable to the heaviest-tailed discrepancy; robust estimators may offer greater resilience.The paper presents measure aggregation as a practical way to make Goodharting all evaluators harder.Source: §5 · p. 17

Random tests

• Randomizing which measure is used may provide an effect similar to averaging while reducing computation when individual measures are costly.The robustness of this trade-off against heavy-tailed measures remains open.Source: §5 · p. 17

Quantifying the need to improve measures

• The paper asks how to choose investment in data, cleaning, design, and troubleshooting to improve measures with provable Goodhart-resilience guarantees.It also proposes adaptively selecting investment to maximize robustly.Source: §5 · p. 17

6 Conclusion

• Longer-tailed discrepancies produce stronger Goodhart effects, potentially making the true goal infinitely worse under extreme over-optimization.The paper distinguishes weak and strong Goodhart effects and identifies an infinitely bad worst case.Source: §6 · p. 17
• The paper argues that alignment is especially difficult in complex environments with feedback loops and calls for measures resilient to Goodhart’s law.This is presented as a motivation for further investigation rather than a completed solution.Source: §6 · p. 17
• “Goodharting” informally means hacking a proxy measure without necessarily satisfying the main goal’s desired targets.The term is attributed to informal usage among some AI researchers.Source: §6 · p. 17

A General facts

• The appendix’s general-facts analysis derives correlations among the top α values of M by conditioning on M ≥ mα under independent G and ξ.The joint density is restricted to the region where G + ξ exceeds the measure threshold.Source: A General facts · p. 22

B Bounded goal, exponential noise

• The derivation uses M = G + ξ together with the general conditional formulas and integration by parts.These steps produce the expressions used for the bounded-goal, exponential-noise case.Source: B Bounded goal, exponential noise · p. 23
• The section derives λ_α = e^−λm_α(e^λh_α −1) + λ(1 −h_α), with h_α = min(m_α, 1).This lemma supports the subsequent analysis of the expected goal as a function of the measure threshold.Source: B Bounded goal, exponential noise · p. 23
• The main result is a weak Goodhart law: correlation tends to zero, but extremely slowly.The theorem analysis identifies the limiting correlation as zero rather than negative.Source: B Bounded goal, exponential noise · p. 24
• For m_α ≥ 1, h_α = 1, enabling the asymptotic evaluation of the expected goal and correlation.The section then combines these expressions to prove the theorem and plot E_α[G].Source: B Bounded goal, exponential noise · pp. 23–24

C.1 Useful lemmas

• The second integral contribution is exponentially negligible in m under the stated inequalities, including γ < σ and κγ < σ.The proof uses lower integration bounds diverging with m and Gaussian-tail estimates.Source: §C.1 · pp. 25–26
• Terms involving the remainder are bounded relative to dominant integrals, while Taylor-expansion remainders satisfy |S_k(w)| ≤ C_k w^4.These bounds support replacing truncated integrals with simpler asymptotic expressions.Source: §C.1 · pp. 25–27
• The lemmas derive asymptotic approximations for integrals by splitting domains, changing variables, and isolating exponentially negligible contributions.The analysis separates integrals at √m or m/γ and uses substitutions such as h = v − κ²m.Source: §C.1 · pp. 24, 26
• After extending relevant integrals to the real line, odd Gaussian moments vanish, while the ℓ = 4 contribution remains O(1).This reduces the leading calculations to a small set of Gaussian moments up to exponentially small errors.Source: §C.1 · p. 27
• The resulting expansions compute the required I_k terms through O(m^-5), completing the auxiliary lemma calculations.The proof combines the preceding decompositions, substitutions, and moment estimates.Source: §C.1 · p. 27

C.2 Proof of Theorem 2

• The proof models M = G + ξ as a normal variable with variance 1 + ε² and relates threshold selection to Gaussian-tail integrals.It uses P[M ≥ m] and the threshold relation α = P[M ≥ m_α].Source: §C.2 · pp. 28–29

D.1 Preliminary lemmas

• Preliminary lemmas compute truncated moments and covariances needed for E_α[G], E_α[ξ], E_α[G²], E_α[ξ²], and E_α[Gξ].The proof applies the lemmas with different parameter substitutions for G and ξ.Source: §C.2 · pp. 28–29, §D.2 · p. 31

D.2 The regime mα ≥1 + η

• For m_α ≥ 1 + η, the selected expectations are assembled from the preceding lemmas using parameter choices tailored to each moment.The listed substitutions cover first and second moments of ξ and G and their cross-moment.Source: §D.2 · p. 31

D.3 The regime η ≤mα ≤1 + η

• For η ≤ m_α ≤ 1 + η, analogous formulas are derived by decomposing the remaining integrals and applying variants of the earlier lemmas.These formulas enable plotting E[G|M ≥ m_α] and the correlation as functions of m_α.Source: §D.3 · pp. 31–34

D.4 The limit α →0

• As α → 0, m_α scales as ηα^(-1/(β−1)), while the limiting variance and covariance expressions determine the asymptotic correlation.The section reports Var_α(G) → 1/12 and Cov_α(G, M) = −1/[12(β−2)].Source: §D.4 · pp. 34, 36, 37

D.5 What happens when α ≈ε

• As α approaches zero, the correlation begins positive and near 1, then becomes negative before converging toward zero.The turning point is studied by imposing m_α = 1 + η.Source: §D.5 · p. 37
• For β > 4, the limiting correlation satisfies ρ_α(ε) → −1/(β−2) as ε → 0.This is stated as the result of Lemma 27.Source: §D.5 · p. 38
• For 3 < β < 4, the limiting correlation instead satisfies ρ_α(ε) → −(4−β)/(2β−3) as ε → 0.The two β regimes differ because the relevant term has distinct limits depending on whether β exceeds or falls below 4.Source: §D.5 · pp. 38–40

E.1 Useful lemmas

• The supporting lemmas compute the expectations, variances, and covariances required to combine the two regime-specific limits.The section describes these calculations as the basis for the theorem.Source: §D.5 · pp. 38–40

E.2 Proving Theorem 5

• The selected mixed moments αE_α[G^cξ^b] are reduced to a common integral I(m, β−1−b, γ−c), with separate contributions from regions below and above g = m − η.The decomposition writes the quantity as A + B and combines the two parts asymptotically.Source: §E.2 · p. 41
• For γ − β > 1, the denominator requires more precise approximations, with cases explicitly given for γ − β ∈ (2, 3) and γ − β ∈ (3, 4).These approximations are derived from Equation 4.Source: §E.2 · pp. 42–43
• The proof divides the asymptotic analysis into six cases according to γ − β: (−∞, −2), (−2, −1), (−1, 0), (0, 1), (1, 2), and (2, +∞).Different cases determine which powers of m and η dominate.Source: §E.2 · p. 42
• The resulting approximations use leading powers such as m^(1−γ) or η^(β−1)m^(1−β), with further corrections when γ − β lies in intermediate ranges.Only the leading term of I is nonnegligible relative to the comparison terms in the stated asymptotic regime.Source: §E.2 · p. 42
• The proof verifies positivity of the constants and substitutes the asymptotic threshold and η expressions to obtain the theorem.The final step fits additional β and γ factors into the constants.Source: §E.2 · p. 44

• Discrete approximations of the integrals produced results similar to the computations using special.erf and integrate.quad.Source: F Uniform goal, log-normal noise · p. 44
• For ε = 1/16, the goal increases, then its increase slows, before accelerating rapidly again rather than decreasing.Source: F Uniform goal, log-normal noise · p. 44, Figure 8 · p. 44
• The ε = 1/16 curves become more similar to the normal distribution as ε decreases, accompanying the altered descent pattern.Source: F Uniform goal, log-normal noise · p. 44
• For larger ε, the double-descent phenomenon appears more pronounced, although computational approximation errors limit the observations.Source: F Uniform goal, log-normal noise · p. 44

• The construction assumes that G takes only negative values and has a probability density function p_G(g) = e^g.Source: G Worst-case Goodhart · p. 44
• The noise distribution is chosen as a power law whose tail is thicker for large negative values of g.Source: G Worst-case Goodhart · p. 45
• A point mass is added to the conditional noise distribution so its mean and variance can be set without changing the tail distribution.Source: G Worst-case Goodhart · p. 45
• The proof decomposes an integral into I(m), J(m), and K(m), then argues that I(m) and K(m) are negligible relative to J(m).Source: G Worst-case Goodhart · p. 47
• As α → 0, the threshold m_α → ∞ and the conditional expectation E_α[G] tends to −∞.Source: Figure 10 · p. 45, G Worst-case Goodhart · p. 47