Path-Specific Objectives for Safer Agent Incentives

1 Introduction

• Recommendation systems can improve performance by making users’ preferences more predictable rather than catering to their tastes directly.Such incentives can indirectly achieve what designers asked for but did not intend (Everitt et al. 2021a).Source: §1 · p. 1
• When user preferences are hard to reward-specify and vulnerable to manipulation, the goal is to remove incentives to control them rather than specify the “right” preferences.The framework applies when some state is delicate and other state is tractable, under assumptions permitting causal-effect estimation.Source: §1 · p. 1
• Path-specific objectives maximize the causal effect of actions on reward along paths not mediated by delicate state, removing the instrumental control incentive over that state.The approach unifies and generalizes proposals on reward tampering (Uesato et al. 2020), online reward learning (Armstrong et al. 2020), and auto-induced distributional shift (Krueger, Maharaj, and Leike 2020).Source: §1 · p. 1
• The paper formalizes delicate state as a complement to reward specification, proposes path-specific objectives, and shows they generalize and unify prior work.The resulting agent does not intend to affect delicate state, unlike agents merely trying not to influence it or keep it constant (Halpern and Kleiman-Weiner 2018; Turner, Ratzlaff, and Tadepalli 2020; Krakovna et al. 2020).Source: §1 · p. 1

2 The Problem of Delicate State

• Delicate state is subtle and manipulable: it is hard to specify a reward for, while deliberate influence can bring harm.The approach removes incentives to control this state rather than trying to specify its reward directly.Source: §2 · p. 1

Subtlety

• A state may be subtle because it lacks an ethical ordering, is vague or unenforceable, is legally restricted, or is excluded for structural reasons.Simply avoiding explicit reward dependence on delicate state is insufficient because interactions between delicate and robust state can create instrumental incentives.Source: Subtlety · p. 2

Manipulability and Stability

• The paper treats influence as intentional when an agent has an incentive over a state and influences it, distinguishing manipulability from instability under natural behavior.The approach is most applicable where a system is safe under natural behavior but remains vulnerable to deliberate influence, such as preference manipulation or reward tampering.Source: Manipulability and Stability · p. 2
• Stability is hard to prove, though instability can be shown by identifying natural behavior that produces bad outcomes.Delicate states correlated with utility may also be systematically influenced without an incentive.Source: Manipulability and Stability · p. 2

3 Background on Causal Influence Diagrams

• CIDs are suited to delicate-state analysis because they represent causal interactions and provide graphical criteria for incentives to control variables.They combine influence diagrams and causality and have been used to identify incentives (Everitt et al. 2021a).Source: §3 · p. 2
• A causal influence diagram is a directed acyclic graph partitioned into structure, action, and utility nodes.Its graph represents agent decisions and goals alongside other variables (Everitt et al. 2021a).Source: §3 · p. 2
• Structural causal influence models pair a CID with structural functions and exogenous variables, while policies map observed action-node parents to decisions.Imposing a policy on a SCIM yields an SCM, where interventions can be represented with do(X = x) and potential responses.Source: §3 · p. 3
• An instrumental control incentive means an agent can gain utility by independently setting a variable; in a single-decision CID, it exists when a directed action-to-utility path passes through that variable.Figure 1 contrasts a graph with incentive over S and Z against one where Z does not influence reward and has no such incentive.Source: §3 · p. 3, Figure 1 · p. 3

4 General Delicate MDP CID

• A delicate MDP factors the state at each timestep into delicate state Z_t and robust state S_t.The paper applies causal-influence-diagram tools to a factored MDP with these two state components.Source: §4 · p. 3
• The CID includes Z_t, S_t, A_t, and R_t across timesteps, with actions observing current delicate state, robust state, and reward.Variables depend only on the most recent timestep, and reward discounting can be represented by scaling utility nodes.Source: §4 · p. 3
• Special cases remove influence arrows; reward-tampering work often assumes reward specification Z_t cannot directly influence the next robust state S_(t+1).This corresponds to modeling the reward specification as unable to influence the rest of the state.Source: §4 · p. 3

5 Path-specific Objectives

• PSO compares a candidate action with a default action, propagating the candidate’s effect along selected paths while imputing the agent’s expected future policy.The underlying path-specific effect compares an outcome under the default with one under the candidate action, with the changed value propagated only along the selected edge-subgraph.Source: §5 · pp. 3–4
• For every delicate MDP, the Figure 2b edge-subgraph admits no ICI over delicate state by removing directed paths from actions to that state.The proof identifies the removed arrows as those from actions and robust state to subsequent delicate state.Source: §5 · p. 4, Figure 2 · p. 4
• A PSO-optimal policy has no ICI over delicate state, although it may still systematically affect that state as a side effect of pursuing another objective.PSO optimization also optimizes the action’s total effect on utility in a modified model where delicate state takes its default-action counterfactual values.Source: §5 · p. 4
• For partially observable MDPs, split each time-step’s state into observed and unobserved variables and block paths from both forms of delicate state.Source: §5 · p. 4
• Except for one-time-step effects, path-specific effects are not experimentally identifiable, though further assumptions such as counterfactual independence can identify them in simulation.Approximations may be inaccurate, but the described approaches still remove ICI by producing models without directed paths from actions to delicate state.Source: §5 · p. 5

Estimating the Natural Distribution

• Estimating the natural distribution requires choosing a default action and future policy, and the imputed delicate-state value must not depend on descendants of the current action or current policy.Source: Estimating the Natural Distribution · p. 5
• A trustworthy-policy baseline is the most principled estimate; state-evolution rules and a fixed initial delicate state are alternatives with narrower conditions.The policy baseline needs a sufficiently accurate full-system simulation; a state baseline relies on knowing natural evolution, while a fixed initial state suits cases with little expected change.Source: Estimating the Natural Distribution · p. 5

• Table 2 schematically overviews prior work the framework generalizes, using the paper’s terminology; these proposals can be read as intervention distributions for estimating path-specific effects.Source: §6 · p. 5, Table 2 · p. 6
• Counterfactual Reward and Uninfluencability treat reward-function state as delicate, while counterfactual and frozen reward functions are used to avoid reward tampering.The proposals are interpreted as policy-baseline or fixed-state interventions for estimating path-specific effects.Source: §6 · pp. 5–6
• Auto-induced Distributional Shift is a restricted policy baseline: reusing intervention context every K steps weakens rather than removes control incentives, and divergence helps explain poor multi-timestep performance.Source: §6 · p. 6
• Taylor (2016) ignores influence through a state channel using a natural-decision distribution; this framework generalizes that formalization to multiple timesteps and alternative distribution strategies.Source: §6 · p. 6
• Approaches that demarcate delicate state need to show those parts are stable for their modifications to establish safe agents.Source: §6 · p. 6

Tabular Example: Barging

• In the tabular test, barging moves the person into the river to open a high-reward short path, while the desired behavior is to take the long path when the person blocks it.The long path gives a small reward; the short path yields a large reward only when the person is in the river.Source: Tabular Example: Barging · p. 6
• Normal agents barge and take the short path, while deterministic PSO agents take the long path and adapt if the person is accidentally pushed into the river.With a barging penalty of -11 in the oracle return, PSO maximizes performance and mitigates the ϵ-greedy handicap.Source: Figure 3 · p. 7, Tabular Example: Barging · p. 7

Content Recommendation

• The simulation treats user preferences as delicate and loyalties and sampled users as robust; recommendations affect clicks, loyalty, and users’ subsequent interests.Users become more interested in the topics shown to them, including when learning is absent.Source: Content Recommendation · p. 7
• Three PSO variants reduce preference drift similarly without meaningful accuracy harm or loyalty change; the policy-baseline performs marginally better than heuristic variants.The figure reports that PSO slows preference drift, while context-swapping eventually leads to more drift.Source: Content Recommendation · p. 8, Figure 5 · p. 8
• Preference drift can still occur without motivated control: the unstable user dynamics cause drift even under random behavior, and accurate recommendations can accelerate it.The authors cannot offer an oracle return because they do not understand what desirable user preferences would be.Source: Content Recommendation · p. 8

8 Related Work

• Path-specific effects measure impacts along selected causal pathways; prior policy optimization using them considered simpler causal graphs with stronger assumptions (Razieh, Kanki, and Shpitser 2018).The paper builds on Causal Influence Diagrams (Everitt et al. 2021a) and path-specific causal effects (Pearl 2001; Avin, Shpitser, and Pearl 2005).Source: §8 · p. 8
• The approach is orthogonal to better-specified rewards, preferences, and demonstrations, and differs from methods that aim to avoid changing parts of the environment.The passage names Leike et al. 2018, Christiano et al. 2017, Schaal 1997, Turner, Ratzlaff, and Tadepalli 2020, Krakovna et al. 2020, and Carroll et al. 2021.Source: §8 · pp. 8–9
• Reward tampering, online reward learning, and auto-induced distributional shift can be treated as special cases of delicate state and path-specific objectives.Taylor (2016) proposed optimizing an objective while ignoring effects along certain channels, using a counterfactual causal rule.Source: §8 · p. 9

9 Discussion and Limitations

• The approach complements reward- and demonstration-based safety work by removing incentives to control state that is easy to influence but hard to reward-specify.It separates state that can be rewarded from state where control incentives should be removed.Source: §9 · p. 9
• Safety requires stability: behavior without a systematic incentive to influence delicate state must itself remain safe.For manipulable state, stability cannot hold under every possible natural behavior, so establishing it is contingent, empirical, and a matter of degree.Source: §9 · p. 9
• Stability may be more plausible in isolable or competition-tested systems, while user-interesting content can also radicalize users.Studying incentivized side-effects is identified as future work, with PSO agents suggested as a tool for empirical exploration.Source: §9 · p. 9
• Implementation hinges on difficult graph and vertex choices, simulation-identifiable PSOs, partial state observation, and ethically deciding which state is delicate.Approximation of causal effects under other assumptions may be possible; observing psychological states such as beliefs can be difficult.Source: §9 · p. 9
• Recent work has begun making progress toward modelling user preferences in content recommendation.Source: §9 · p. 9