Source-linked AI summary
Categorizing Variants of Goodhart's Law
David Manheim, Scott Garrabrant
TL;DR
The paper categorizes statistical misalignments arising when systems optimize metrics that diverge from true goals. It formalizes several Goodhart-like mechanisms and relates them to policy, machine learning, and AI alignment, including adversarial effects and unavoidable tail divergence.
Problem
Metric-based optimization can produce distinct statistical misalignments, but these mechanisms require clearer categorization across algorithmic and human systems.
Method
The paper formalizes regulators selecting states through metric thresholds and introduces specific cases involving ignored causes, causal structure, and adversarial responses.
Results
The analysis identifies adversarial misalignment through extremal effects, exacerbated regressional effects, or causal intervention, while inexact metrics necessarily diverge from goals in the tail.
Takeaways & Limitations
These dynamics apply to policy design, machine learning, and AI alignment, including limited-data errors, myopic goals, ignored causality, and perverse metric effects.
Takeaways & Limitations
The presentation assumes a single-dimensional metric, although multiple metrics and restrictions are said to follow similar dynamics.
Abstract
from arXiv · showhide
There are several distinct failure modes for overoptimization of systems on the basis of metrics. This occurs when a metric which can be used to improve a system is used to an extent that further optimization is ineffective or harmful, and is sometimes termed Goodhart's Law. This class of failure is often poorly understood, partly because terminology for discussing them is ambiguous, and partly because discussion using this ambiguous terminology ignores distinctions between different failure modes of this general type. This paper expands on an earlier discussion by Garrabrant, which notes there are "(at least) four different mechanisms" that relate to Goodhart's Law. This paper is intended to explore these mechanisms further, and specify more clearly how they occur. This discussion should be helpful in better understanding these types of failures in economic regulation, in public policy, in machine learning, and in Artificial Intelligence alignment. The importance of Goodhart effects depends on the amount of power directed towards optimizing the proxy, and so the increased optimization power offered by artificial intelligence makes it especially critical for that field.
Varieties of Goodhart-like Phenomena
Goodhart-like failures arise when optimizing a proxy causes it to stop tracking the regulator’s true goal. The paper distinguishes four mechanisms and formalizes proxy-based selection in state space.
- Varieties of Goodhart-like Phenomena: The paper distinguishes regressional, extremal, causal, and adversarial Goodhart as separate mechanisms that can also occur together.It introduces subcategories to clarify how these failure modes differ.
- Varieties of Goodhart-like Phenomena: A regulator selects permissible system states using a proxy metric because the true goal is incompletely known.The formal setup defines a state space, a true goal G(s), a proxy M(s), and a threshold c with M(s) ≥ c.
1 Regressional Goodhart
Regressional Goodhart occurs because optimizing an imperfect proxy selects both for the true goal and for proxy noise. As the proxy becomes extreme, the proxy and goal diverge in the tail.
- 1 Regressional Goodhart: Regressional Goodhart selects for both the true goal and the difference between the proxy and the goal.This phenomenon is also called “Tails come apart.”
- 1 Regressional Goodhart: Large proxy values tend to include large positive noise, making the corresponding goal predictably smaller than the proxy.Selecting states with M > c can still produce higher expected G than selecting without that threshold, despite the divergence.
- 1 Regressional Goodhart: An inexact metric necessarily diverges from the goal in the tail, so this effect cannot be avoided by merely choosing another imperfect measure.The passage describes regressional Goodhart as the simplest and most fundamental Goodhart effect.
2 Extremal Goodhart
Extremal Goodhart occurs when proxy-based selection moves the system into regions where the proxy–goal relationship changes or the learned model becomes inadequate. It can result from model insufficiency or regime change, without regulator intervention.
- 2 Extremal Goodhart: The formal setup treats the metric as a single-dimensional mapping used for decision-making and notes that multiple metrics follow similar dynamics.The presentation simplifies the metric structure for exposition.
- 2 Extremal Goodhart: Model insufficiency arises when a simplified or incompletely learned proxy relationship collapses outside the region where it was approximately accurate.Limited observations and overly simple models can fail when optimization reaches previously unobserved regions.
- 2 Extremal Goodhart: Extremal Goodhart occurs when proxy selection moves the system into regions where the proxy–goal relationship differs from the observed region.The relationship may be fundamentally different where the proxy takes extreme values.
- 2 Extremal Goodhart: Regime change can result from systematic measurement error or from a generating process that differs across regions.For example, wind-speed measurements may be biased above instrument design tolerances, changing their relationship with wind damage.
- 2 Extremal Goodhart: Regressional and extremal Goodhart require selection pressure but do not involve intervention by the regulator.Regulator intervention changes the state space and introduces causal Goodhart as an additional error type.
3 Causal Goodhart
Causal Goodhart occurs when a regulator’s intervention changes the causal relationship that made a proxy useful. Unlike earlier cases, it can arise without uncertainty or incorrect beliefs about the relationships.
- 3 Causal Goodhart: Causal Goodhart occurs when intervening on an indirect proxy–goal path changes the relationship between the measure and the goal.Maximizing the metric can make further intervention counterproductive, with moderate interventions sometimes outperforming extreme ones.
- 3 Causal Goodhart: Unlike regressional and extremal effects, causal Goodhart is induced by the regulator’s action rather than by uncertainty or an incorrect relationship model.The relevant issue is how intervention changes causal structure.
- 3 Causal Goodhart: The paper identifies three general cases of causal Goodhart in which the regulator intervenes using a correlation between a measure and the goal.The intervened node distinguishes the classes in the accompanying diagram.
Metric Manipulation
Metric manipulation changes the metric without necessarily improving the goal, potentially severing their relationship and making further optimization counterproductive.
- Shared Cause Intervention: Intervening on a shared cause can eliminate the metric’s causal relationship to the goal, leaving only combined error terms.Maximizing the shared cause restricts both variables while changing how they relate.
- Shared Cause Intervention: Correlated test scores can lose their original relationship after general test-taking skills are improved, because remaining correlation reflects other factors.The example illustrates how intervention on a shared cause can undermine a proxy relationship.
- Intermediary Intervention: Setting an intermediary to a specific value can make the goal and metric independent while increasing the metric without affecting the goal.This is the defining simple-model consequence of intermediary intervention.
- Metric Manipulation: Setting the metric directly can leave other variables unaffected, regardless of how the goal relates to upstream causes.The intervention targets the metric itself rather than the causal structure connecting it to the goal.
- Metric Manipulation: Changing grades or test scores without improving learning makes the metric useless or less useful for measuring the goal.This example shows direct metric manipulation in education.
- Causal Connections: Causal misunderstandings or regime changes can produce causal, extremal, or worsened correlational Goodhart effects depending on whether the regulator intervenes or selects.The failure mode depends on both the mistaken causal model and the type of optimization pressure applied.
Non-Causal Goodhart Effects in Causal Systems
When a regulator selects using an assumed causal model that differs from the true model, optimization can worsen regressional or extremal Goodhart effects but not causal Goodhart effects.
- Non-Causal Goodhart Effects in Causal Systems: The figures distinguish true causal paths from the regulator’s assumed paths using dashed and dotted lines, respectively.The mismatch represents the regulator’s mistaken understanding of the causal relationship.
- Non-Causal Goodhart Effects in Causal Systems: Selecting on the basis of an incorrect causal model can worsen regressional Goodhart effects or produce extremal Goodhart effects.These outcomes arise from selection under a mistaken model rather than intervention on the causal system.
- Non-Causal Goodhart Effects in Causal Systems: The described selection error does not produce causal Goodhart effects.The passage explicitly limits the resulting failure modes to regressional and extremal effects.
Ignored Intermediary
Ignoring an intermediary adds noise because the regulator treats a non-direct goal–metric relationship as direct, producing regressional or extremal Goodhart effects.
- Scope: If a teacher is treated as the regulator, the case differs from one in which the teacher is an agent.The distinction changes which multi-actor analysis applies.
- Ignored Intermediary: The ignored-intermediary error assumes the goal and metric are directly related even though an intermediary adds another source of noise.The intermediary creates an unmodeled causal step between the goal and metric.
- Ignored Intermediary: Without intervention, this causal mistake adds a term to the error and leads to regressional and extremal Goodhart effects.The error enters through the omitted intermediary rather than direct metric manipulation.
Ignored Goal Cause
An ignored goal cause occurs when the metric and goal depend on overlapping but nonidentical causes, so unmodeled causal structure introduces additional noise.
- Ignored Additional Cause: The metric may have multiple causes while the goal depends on only some of them, or the goal may have multiple causes while the metric captures only some.The proxy and goal therefore share only part of their causal structure.
- Ignored Additional Cause: Additional noise from an unmodeled cause worsens regressional Goodhart effects when that cause’s distribution matches the assumed relationship’s error distribution.The effect follows because the added cause contributes to the proxy error.
- Causal Structure: In the simple model, the goal is caused by the metric, while the more common case has a variable Y causing both the metric and the goal.The latter structure makes the metric and goal correlated through a shared cause.
4 Adversarial Goodhart
Adversarial Goodhart covers cases where other agents react to a regulator’s metric, either through misaligned goals or regulator-provided incentives. These reactions can produce extremal, regressional, causal, or metric-manipulation effects, including Cobra effects.
- Adversarial Misalignment: Adversarial misalignment arises when an agent with unrecognized goals acts independently in ways that adversely affect the regulator’s goal.The agent may apply selection pressure while anticipating the regulator’s metric-based selection.
- Adversarial Goodhart: Adversarial Goodhart occurs when other agents react to the regulator’s metric in ways that create Goodhart-like effects.The paper extends the earlier category of adversarial Goodhart into several specific cases.
- Adversarial Misalignment: Such misalignment can create extremal effects, worsen regressional effects, or change the effects of regulator optimization through causal intervention.The failure need not arise from only one Goodhart mechanism.
- Campbell’s Law: In Campbell’s Law cases, agents choose or manipulate metrics after observing the regulator’s metric, reducing its usefulness for achieving the original goal.The agent’s metric may be unhelpful on its own but become useful for hijacking the regulator’s selection.
- Cobra Effects: A normal Cobra effect occurs when incentives align an agent’s goal with the regulator’s metric, prompting actions that alter causal structure and worsen the targeted outcome.The paper’s example describes people breeding cobras to collect rewards, producing more cobras rather than fewer.
- Cobra Effects: Non-causal Cobra effects arise when agent selection pressure produces extremal effects or worsens regressional effects without causal intervention.The Cobra effect can also arise through shared causes, intermediary effects, or metric manipulation.
- Scope: The taxonomy is intended to clarify metric failures across algorithmic and human systems, including policy design, machine learning, and AI alignment.The paper highlights limited data, myopic goals, ignored causality, and perverse metric effects as relevant examples.