PaperlayerView exact revision on arXiv

Source-linked AI summary

Counterfactual harm

Jonathan G. Richens, Rory Beard, Daniel H. Thompson

arXiv:2204.12993v5cs.AIcs.LGstat.ML

TL;DR

Factual outcome statistics do not capture why outcomes occurred, and the paper proposes a causal-model-based statistical definition of harm and benefit. It argues that factual objectives cannot robustly avoid harmful actions in general, while counterfactual harm assessment can distinguish actions with the same outcome statistics.

  • Problem

    There is no formal statistical definition of harm, although ethical preferences can depend on why an outcome occurred rather than only on the outcome itself.

    Source: §1 · pp. 1–2

  • Method

    The paper defines harm and benefit counterfactually using causal models and compares outcomes against a specified default action.

    Source: §2.1 · p. 3, §3 · p. 5

  • Results

    Factual-objective agents are incapable of robustly avoiding harmful actions in general, according to the paper’s theoretical argument.

    Source: §8 · p. 10

  • Takeaways & Limitations

    Counterfactual comparisons can distinguish treatments with the same recovery rates when one can harm patients who would otherwise have survived.

    Source: §1 · p. 1

    The harm measure is comparative and depends on a chosen baseline; without specifying one, whether an action is harmful is ambiguous.

    Source: Appendices · p. 20

Abstract

from arXiv · show

To act safely and ethically in the real world, agents must be able to reason about harm and avoid harmful actions. However, to date there is no statistical method for measuring harm and factoring it into algorithmic decisions. In this paper we propose the first formal definition of harm and benefit using causal models. We show that any factual definition of harm must violate basic intuitions in certain scenarios, and show that standard machine learning algorithms that cannot perform counterfactual reasoning are guaranteed to pursue harmful policies following distributional shifts. We use our definition of harm to devise a framework for harm-averse decision making using counterfactual objective functions. We demonstrate this framework on the problem of identifying optimal drug doses using a dose-response model learned from randomized control trial data. We find that the standard method of selecting doses using treatment effects results in unnecessarily harmful doses, while our counterfactual approach allows us to identify doses that are significantly less harmful without sacrificing efficacy.

1 Introduction

Factual outcome statistics can miss ethically important differences in why outcomes occurred, motivating a statistical account of harm based on counterfactuals. The paper introduces such an account and uses it to frame harm-aware algorithmic decisions.

1 Introduction

  • Factual objectives depend on outcomes, while human preferences can also depend on why an outcome occurred.The paper names medicine, criminal justice, finance, and autonomous vehicles as consequential decision areas.Source: §1 · p. 1
  • Treatments with identical recovery rates differ ethically because treatment 2 can kill patients who would have survived untreated, whereas treatment 1 causes no such deaths.The treatments also have identical mortality rates, but the deaths have different causes.Source: §1 · p. 1
  • The paper proposes a statistical definition of harm and benefit, then develops counterfactual objectives for algorithmic decisions.Its roadmap includes definitions, harm-aware decision making, distributional shifts, and an application to drug-dose selection.Source: §1 · p. 2

2 Prelimiaries

The preliminaries establish structural causal models and counterfactual comparisons as the basis for the paper’s harm measure, then specify the decision setting and its assumptions.

2.1 Structural causal models

  • SCMs represent variables and their causal relations with a directed acyclic graph, structural functions, and noise variables, and support distributions under interventions and counterfactuals.The paper assumes causal sufficiency and knowledge of the SCM for its theoretical derivations.Source: §2.1 · pp. 2–3, §2.3 · p. 4

2.2 The counterfactual comparative account of harm

  • The counterfactual comparative account measures harm by comparing factual welfare with welfare under an alternative, and the paper generalizes the baseline to a specified default action or policy.For example, medicine may be compared with placebo and negligent decisions with a standard clinical policy.Source: §2.2 · p. 4
  • The paper connects counterintuitive harm scenarios to actual causality and argues that formal causal analysis can resolve them.It identifies this connection between harm research and actual causality as previously unnoted, to its knowledge.Source: §2.2 · p. 4

2.3 Decision theoretic setup

  • The decision setup divides environment variables into agent action, context, and action-descendant outcomes, and assumes no unobserved confounders and a known SCM.The framework describes single-agent, single-action cases, with extensions to sequential or multi-agent problems discussed in an appendix.Source: §2.3 · p. 4
  • With survival as utility, standard treatment selection maximizes expected outcome; in the example, treatments 1 and 2 have equal expected utility and CATE of 0.8.The example uses no treatment as the harm-comparison default.Source: §2.3 · p. 5

3 Counterfactual harm

The section defines counterfactual harm and benefit by comparing observed utility with utility under a default action. Expected harm can distinguish actions that factual treatment effects treat as equivalent.

3 Counterfactual harm

  • Counterfactual harm and benefit compare a person’s factual utility with the utility under a specified default action, using the SCM and the user’s utility.The max operation includes only counterfactual outcomes with higher utility for harm or lower utility for benefit.Source: §3 · p. 5
  • Expected harm and benefit are computed over the outcome distribution and decompose the expected-utility difference between an action and its default.The theorem states the trade-off between expected utility, benefit, and harm.Source: §3 · p. 5
  • For the same expected utility and CATE, treatment 1 causes zero counterfactual harm while treatment 2 causes harm with probability 0.1.The example identifies harm as the probability that a treated patient dies despite having recovered under no treatment.Source: §3 · p. 6

4 Harm in decision making

The section proposes penalizing expected counterfactual harm in action selection. Its investment example contrasts harm aversion with factual objectives such as expected return and risk aversion.

4 Harm in decision making

  • Expected-utility maximizers are indifferent to equal increases in benefit and harm, so they may accept greater harm for an equal increase in benefit.The proposed harm-averse adjustment weights the harm component more heavily, changing the trade-off to 1:(1 + λ).Source: §4 · p. 6
  • Harm-penalized utility subtracts λ times counterfactual harm from utility, with positive λ encoding harm aversion.At λ = 0 the agent is harm indifferent; for λ > 0 it may reduce expected utility to achieve a greater reduction in expected harm.Source: §4 · p. 6
  • Expected harm-penalized utility is a counterfactual objective requiring the SCM, unlike factual expected utility, which can be evaluated from data alone.The paper presents this objective as a way to balance expected benefits and harms.Source: §4 · p. 6
  • The harm-averse agent never reduces Alice’s expected return, while the risk-averse agent may reduce or cancel her investment and never chooses the return-increasing, zero-harm action.The risk-averse behavior follows from its inability to distinguish losses caused by the agent from exogenous market fluctuations.Source: §4 · p. 7

5 Counterfactual reasoning is necessary for harm aversion

The section argues that factual objectives cannot generally ensure harm aversion under outcome distribution shifts, while the counterfactual HPU avoids strictly harmful objectives. The results apply under stated outcome-dependence conditions.

5 Counterfactual reasoning is necessary for harm aversion

  • The expected HPU is not a strictly harmful objective for any positive λ, utility function, environment, or default action.Strictly harmful actions are more harmful without providing additional benefit.Source: §5 · p. 7
  • Distribution shifts in this analysis change the outcome distribution alone, through changes to outcome noise or its causal mechanism.The paper defines such shifts as changes to the exogenous noise distribution for Y and/or the mechanism producing Y.Source: §5 · pp. 7–8
  • Expected utility is strictly harmful under some outcome distribution shift when utility is outcome dependent for the default action and another action.The result assumes that utility does not always prefer dying after treatment to surviving without treatment.Source: §5 · p. 8
  • For any factual objective and users with outcome-dependent utility under the stated conditions, some outcome distribution shift makes its maximizing policy harmful.The result holds regardless of the factual objective and even if the agent retrains in the shifted environment.Source: §5 · p. 8

6 Illustration: Dose-response models

The dose-response example compares harm-averse dose selection with treatment-effect maximization and examines how a factual risk-averse objective behaves after a distribution shift. Lower-harm doses can preserve treatment effect, but risk aversion can become harmful after a shift.

6 Illustration: Dose-response models

  • The example uses an Aripiprazole dose-response model learned from randomized controlled trial meta-analysis data, with utility U(a, y) = y and zero dose as the harm default.The model predicts symptom-severity reduction using a generalized additive model with a cubic spline dose function.Source: §6 · p. 9
  • A dose below the treatment-effect optimum can reduce harm by 90% while reducing treatment effect by only 2%.For λ = 100, the optimal dose falls from 19.3 mg/day to 17.3 mg/day; lower-harm doses achieve a similar symptom-severity improvement.Source: §6 · p. 9
  • After a shift that adds dose-dependent outcome noise, the risk-averse objective is strictly harmful for every β > 0.In the shifted environment, expected utility is higher and expected harm lower at the utility-maximizing dose than at the risk-averse dose.Source: §6 · p. 10

7 Related work

The section relates the paper’s counterfactual harm approach to prior work on path-specific counterfactual fairness and objectives, then contrasts it with a later causal definition of harm.

7 Related work

  • Path-specific counterfactual fairness and path-specific objectives also use counterfactual contrasts with a baseline to impose ethical constraints on objectives.The paper discusses these works in Appendix L.3.Source: §7 · p. 10
  • A later causal definition labels an action harmful in deterministic settings when utility is below a default and another available action yields higher utility.The paper reports counterintuitive cases, including attributing harm to unharmful actions, and says the definition is intractable beyond simple, low-dimensional causal models.Source: §7 · p. 10

8 Conclusion

The conclusion challenges the assumption that human preferences can be represented solely by world states, independent of causal relations. It presents counterfactual harm as a way to reduce harm in a drug-dose example and argues that factual objectives cannot robustly avoid harm.

8 Conclusion

  • The paper proposes a statistical definition of harm and argues that factual-objective agents cannot robustly avoid harmful actions in general.Its dose-selection example demonstrates using the definition to meaningfully reduce harms caused by machine-learning algorithms.Source: §8 · p. 10
  • The conclusion argues that safe and ethical AI deployment depends on avoiding harm, and that advanced AI is limited without causal and counterfactual inference.The authors state that their results add weight to this view.Source: §8 · p. 10
  • The paper identifies as a core statistical-learning assumption that preferences depend only on world states, not the causal relations among them.This assumption allows factual objectives to be optimized without learning causal models or performing counterfactual inference.Source: §8 · p. 10

Appendices

The appendices show how counterfactual harm resolves omission, preemption, and preventing-worse examples by selecting a default action and, when needed, restricting causal paths. They distinguish total from path-specific harm and contrast the framework with other accounts.

Path-specific harm

  • Path-specific harm compares an action with a default while restricting the counterfactual outcome to selected causal paths; direct, indirect, and total harm select different paths.The construction replaces causal mechanisms so paths outside the selected subgraph retain factual parent values.Source: Appendices · p. 16

Omission problem

  • When Alice has no duty to give Bob clubs, not giving them causes no harm against that same default; an obligation to give them makes the omission harmful.With giving the clubs as the default, the example assigns harm of 1 to Alice’s omission.Source: Appendices · p. 17

Preemption problem

  • In the preemption example, blocking Alice’s effect on Eve isolates her direct harm to Bob, even though Eve would otherwise have robbed him.The counterfactual comparison gives Bob utility 1 rather than the factual utility 0.Source: Appendices · p. 17

Preventing worse

  • The preventing-worse cases motivate distinct queries: path-specific harm captures Alice’s robbery of $1, while total harm treats shooting Bob to prevent torture as non-harmful.The first case yields direct harm of 1; the second yields total harm of 0 relative to not shooting.Source: Appendices · pp. 18–19

Alternative accounts

  • Alternative accounts either label intrinsically bad outcomes or record losses without attributing them to a particular cause, unlike the comparative counterfactual account.The appendix notes that a loss-based account cannot distinguish losses caused by an assistant from those caused by market fluctuations.Source: Appendices · pp. 19–20

Default actions

  • The default action is part of the harm query, and its choice depends on the intended comparison rather than a universal baseline.The appendices illustrate different defaults for treatment, omission, and intrinsic-harm questions.Source: Appendices · pp. 20–21

E Comment on Beckers et. al.

This appendix examines harm definitions and harm-averse objectives, comparing the BCH and counterfactual definitions while formalizing safety conditions, action-specific harm under distribution shifts, and expected harm in heteroskedastic dose-response models.

Recovering the BCH definition from our definition

  • In single-action, single-outcome settings, BCH can be recovered by restricting the authors’ definition to deterministic models, action-independent utilities, and a default-utility cutoff.For multiple actions or outcomes, the definitions diverge because BCH permits arbitrary harmful contingencies; the authors say strict harm remains essentially equivalent under the stated constraints.Source: E Comment on Beckers et. al · p. 22

Example 1, drowning soldier (False positives)

  • BCH’s arbitrary contingencies can label an action harmful even when it could not change the bad outcome, as in the drowning-soldier example.The officer’s rescue attempt would not have saved the soldier, so the authors’ direct and total harm measures assign the decision not to rescue zero harm.Source: E Comment on Beckers et. al · p. 23

Example 2, Omission problem (False positives)

  • BCH can also treat failure to benefit as harm, as when Alice’s decision not to give Bob golf clubs is blamed for Eve’s robbery.The default utility needed to classify Eve’s robbery as harmful leads BCH to call Alice’s omission harmful; the authors instead use Alice’s choice not to give the clubs as the default action.Source: E Comment on Beckers et. al · pp. 23–24

Example 3, Cancelling harms and benefits

  • BCH can cancel one agent’s harm against another’s benefit, while recovering Alice’s $K theft as harm requires an arbitrary default utility of at least 1 + K.The authors’ path-specific harm measure assigns Alice’s theft direct harm K even when Eve reimburses Bob; the required BCH default grows with K.Source: E Comment on Beckers et. al · p. 24

Example 4, enforcing worst-case harm

  • BCH’s max-harm measure compares each case with its best counterfactual action, unlike standard medicine risk comparisons with control or placebo.This can classify a treatment as harmful relative to a more effective treatment even when it is not worse than no treatment; standard medicine comparisons instead assess risk against control or placebo.Source: E Comment on Beckers et. al · pp. 24–25

Practical limitations

  • BCH harm evaluation is intractable because it enumerates possible actions and contingencies, restricting its practical use in complex or continuous decision domains.The authors note that evaluating BCH harm is at least DP-complete and argue that even their dose-response model, with continuous action and outcome spaces, falls outside the described low-dimensional scope.Source: E Comment on Beckers et. al · p. 25

Discussion of our definition of harm in Becker’s et. al.

  • The authors argue that actual-cause criteria can produce wrong harm attributions, their default action is a comparative baseline rather than a moral judgment, and path-specific analysis is not a fix for but-for causation.They also compare path-specific analysis to established path-specific effects, where the baseline and causal path are inputs to the query.Source: E Comment on Beckers et. al · pp. 25–26

E Comment on Beckers et. al.

  • Agent 2 never selects action 2, choosing action 1 for λ < 0.003125 and action 3 otherwise.For action 1, the optimal K is 1/200λ.Source: E Comment on Beckers et. al · p. 27
  • Agent 3 chooses action 1 with K = 20 when λ < 11.93, and otherwise chooses action 2; action 3 is never selected.For action 1 with K > 1, the optimal K is 20 if λ < 12.002 and 0 otherwise.Source: E Comment on Beckers et. al · p. 27
  • The dose-response GAM is reparameterized as a heteroskedastic additive-noise model, enabling expected-harm calculation while the sample-noise term cancels.The expected-harm result applies to heteroskedastic additive-noise models with normally distributed outcomes whose mean and variance depend on action and context.Source: E Comment on Beckers et. al · pp. 28, 30, 31
  • For any utility function, environment, and default action, expected HPU is never a harmful objective when λ > 0.If another action has at least as much expected utility and strictly less expected harm, the HPU maximizer cannot select the more harmful action.Source: E Comment on Beckers et. al · p. 31
  • Any factual outcome distribution can be induced by a counterfactually independent model.The construction assigns mutually independent noise variables to each action and context.Source: E Comment on Beckers et. al · pp. 31–32
  • Under Lemma 2’s conditions, distribution shifts can change the expected harm of action a while leaving other actions’ expected harms unchanged.The shifts order action a’s expected harm as M− < M0 < M+, with all other actions’ expected harms equal across the three environments.Source: E Comment on Beckers et. al · p. 32

3. P(y|a′, x; M0) = P(y|a′, x; M+) = P(y|a′, x; M−) ∀a′ ∈A, including a, ¯a(x)

The section constructs outcome-distribution shifts that preserve factual outcome statistics while changing counterfactual harm. It uses these constructions to show that factual objectives cannot guarantee harm-averse decisions under the stated conditions.

3. P(y|a′, x; M0) = P(y|a′, x; M+) = P(y|a′, x; M−) ∀a′ ∈A, including a, ¯a(x)

  • Changing the joint counterfactual distribution can alter harm while leaving each action’s factual outcome distribution unchanged.The construction perturbs the exogenous-noise distribution while preserving the causal mechanism and factual outcome marginals.Source: §3 · p. 33
  • The context-dependent default action has zero expected harm in every environment.For a non-default action, the construction can produce nonzero harm under non-deterministic outcomes.Source: §3 · pp. 33–34
  • If the default action’s utility is outcome-dependent, a distributional shift can make its expected-utility-maximizing alternative harmful.Theorem 3 gives this result when the default and another action have overlapping utility ranges in some context.Source: §3 · p. 34
  • With overlapping utilities for the default and two other actions, any factual objective can be made harmful by an outcome-distribution shift.Theorem 4 establishes this even when the actions have the same expected utility and the shift preserves factual statistics.Source: §3 · pp. 34–35

L.1 Limitations

The framework’s current scope relies on observed variables and knowledge of the structural causal model, while the demonstration uses relatively simple dose-response models. The section also describes possible applications and a bound-based alternative when the model is unknown.

L.1 Limitations

  • The setup assumes all variables are observed for counterfactual computation, and the theoretical results assume knowledge of the data-generating structural causal model.The authors note that these assumptions may limit applicability, particularly where the model is unknown or poorly approximated.Source: §L.1 · p. 35
  • When the structural causal model is unavailable, a tight upper bound on harm can be derived from identifiable interventional distributions or specified assumptions.The authors say using this bound in place of counterfactual harm can ensure a chosen harm-benefit ratio.Source: §L.1 · pp. 35–36
  • The demonstration uses relatively simple dose-response analysis with GAM models, while applying the framework in complex domains without a known structural causal model remains future work.The authors suggest approximate models as a possible route to extending the framework.Source: §L.1 · p. 36
  • For medical imaging, the framework could estimate harm from multiple sclerosis by comparing a factual scan with counterfactual healthy images and their predicted QALYs.The example uses a deep structural causal model to generate counterfactual images.Source: §L.1 · p. 36
  • For sequential treatment decisions, a harm-averse Bellman equation can favor policies that maximize discounted cumulative HPU rather than reward alone.For λ > 0, the policy avoids actions whose harm-benefit ratio is below 1 : 1 + λ.Source: §L.1 · pp. 36–37

L.2 Relation to bigger picture AI safety and ethics

The framework addresses a failure mode of factual-objective algorithms, but it does not resolve established AI safety problems. Harm-averse agents can still produce harmful outcomes through other known vulnerabilities.

L.2 Relation to bigger picture AI safety and ethics

  • Harm aversion does not resolve robustness, risk-aversion, model-misspecification, or unsafe-exploration problems that can still lead to harmful outcomes.The paper frames factual-objective algorithms’ inability to robustly reason about harm as an additional failure mode, not a solution to those established problems.Source: §L.2 · p. 37

L.3 Related work

The section distinguishes harm from counterfactual fairness while noting that the measures can be used together. It also relates the framework to path-specific objectives, which constrain utility pathways but are themselves agnostic to harm.

L.3 Related work

  • Counterfactual fairness concerns unfair influence from a protected attribute, whereas harm measures whether an action worsened the user’s outcome; the measures can be applied together.The paper notes that an action can be strictly harmful while remaining fair.Source: §L.3 · p. 37
  • Path-specific objectives restrict utility to selected causal pathways but remain agnostic to harm, so they could be combined with path-specific harm.The authors describe this combination as a way to vary harm aversion by causal path.Source: §L.3 · p. 38

Related papers

Loading 2204.12993v5…