PaperlayerView exact revision on arXiv

Source-linked AI summary

Avoiding Side Effects By Considering Future Tasks

Victoria Krakovna, Laurent Orseau, Richard Ngo, Miljan Martic, Shane Legg

arXiv:2010.07877v1cs.LGcs.AI

TL;DR

Reward designers must specify not only task completion but also which environmental changes to avoid, a difficult burden when side effects are hard to enumerate. The paper rewards the ability to perform possible future tasks and uses a baseline policy to avoid interference incentives; it reports that this approach avoids interference and side effects in gridworlds, unlike reversibility reward when irreversible actions are required.

  • Problem

    Reward functions must specify both task completion and what not to do, while defining all undesirable environmental changes is difficult.

    Source: §1 · p. 1

  • Method

    The approach generates an auxiliary reward from possible future tasks and uses a baseline policy to filter out tasks not achievable by default.

    Source: §1 · p. 2

  • Results

    The baseline future-task approach avoids side effects and interference in the reported gridworld results, whereas reversibility reward fails when the current task requires irreversible actions.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The approach provides an auxiliary reward for preserving future-task ability even when the current task requires irreversible actions, without introducing interference incentives.

    Source: §8 · p. 10

    The paper identifies generalizing interference avoidance to stochastic environments as future work.

    Source: §8 · p. 10

Abstract

from arXiv · show

Designing reward functions is difficult: the designer has to specify what to do (what it means to complete the task) as well as what not to do (side effects that should be avoided while completing the task). To alleviate the burden on the reward designer, we propose an algorithm to automatically generate an auxiliary reward function that penalizes side effects. This auxiliary objective rewards the ability to complete possible future tasks, which decreases if the agent causes side effects during the current task. The future task reward can also give the agent an incentive to interfere with events in the environment that make future tasks less achievable, such as irreversible actions by other agents. To avoid this interference incentive, we introduce a baseline policy that represents a default course of action (such as doing nothing), and use it to filter out future tasks that are not achievable by default. We formally define interference incentives and show that the future task approach with a baseline policy avoids these incentives in the deterministic case. Using gridworld environments that test for side effects and interference, we show that our method avoids interference and is more effective for avoiding side effects than the common approach of penalizing irreversible actions.

1 Introduction

The paper proposes defining side effects through the future tasks an agent could still perform, reducing the need to enumerate undesirable changes. A baseline policy filters out future tasks that would not be achievable by default, and the authors report that this approach avoids interference and side effects in their tested settings.

1 Introduction

  • Side effects pose a specification challenge related to the frame problem because designers must account for changes an agent should not make.The paper describes defining side effects extensively as difficult.Source: §1 · p. 1
  • The proposed auxiliary reward is generated automatically as the agent learns about the environment, addressing the burden of manually revising rewards for unforeseen undesirable behavior.The usual manual process is described as tedious and unable to address side effects the designer has not foreseen or observed.Source: §1 · p. 1
  • The method rewards the ability to complete unknown future tasks in the same environment, so changes such as breaking a vase can reduce future-task reward.The future task starts from the state left by the current task; a uniform prior over possible goal states defines the tasks.Source: §1 · p. 2
  • A baseline policy represents a default course of action and filters out future tasks that are not achievable after following it, avoiding incentives to interfere with environmental events.The paper gives taking food from a human as an example of interference induced by unfiltered future-task reward.Source: §1 · p. 2
  • The paper formalizes interference incentives and shows baseline-policy avoidance in the deterministic case, while using UVFAs to estimate values for future tasks with different goals.The authors describe this as theoretical groundwork and demonstrate the approach in gridworlds.Source: §1 · p. 2
  • The baseline future-task approach avoids side effects and interference, while reversibility reward fails when the current task requires irreversible actions.The introduction reports these as the method’s gridworld findings and notes reversibility’s limitation.Source: §1 · p. 2

2 Future task approach

The future-task approach adds an auxiliary reward based on the value of possible goal-reaching tasks that begin from the agent’s current state. It uses a uniform distribution over goal states and balances the auxiliary reward against the current task reward.

2 Future task approach

  • The setup assumes a discounted MDP, and future-task reward is added to the current-task reward at each time step.The MDP notation includes states, actions, current-task reward, transitions, discount factor, and initial state.Source: §2 · pp. 2–3
  • At each step, the auxiliary reward is defined using the value of a hypothetical future task that starts from the current state and seeks a sampled goal state.The paper uses a uniform distribution over all possible goal states, with reward 1 at the goal and 0 elsewhere.Source: §2 · p. 3
  • The future-task reward is weighted by β, for which the authors choose the highest value that still allows completion of the current task.The agent receives the current-task reward together with the auxiliary reward.Source: §2 · p. 3
  • Future tasks use binary goal-based rewards, an assumption the authors say simplifies theoretical arguments while covering a variety of goals.The paper notes that more complex tasks can often be decomposed into simple goal-based tasks.Source: §2 · p. 3
  • Reversibility reward is a special case of the future-task reward with only the initial state as a possible future goal.The future-task approach also samples other possible goal states under its stated distribution.Source: §2 · p. 3

3 Interference incentives

The section defines interference incentives as auxiliary-reward pressure to depart from a default policy, and shows that the basic future-task approach can create them.

3 Interference incentives

  • An interference incentive exists when some policy has greater no-reward value at the initial state than the baseline policy.The baseline represents a default course of action, such as doing nothing, and should be optimal without task reward.Source: §3 · pp. 3–4
  • For most future-task distributions, staying in x0 has higher no-reward value than following the baseline policy, creating an interference incentive.In Example 1, the baseline policy always chooses action a.Source: §3 · p. 4

4 Future task approach with a baseline policy

The baseline-policy approach compares future-task performance with a default course of action to avoid interference incentives. Its formal guarantee applies in deterministic environments, while choosing the baseline is left outside the work’s scope.

4 Future task approach with a baseline policy

  • On each future task, the method compares the agent’s run with a reference agent following the baseline and grants full auxiliary reward when the agent does at least as well.The two runs start from the agent’s current state and the baseline-run state, respectively.Source: §4 · p. 4
  • In deterministic environments, the baseline policy has equal or higher no-reward value than any policy, so the method avoids interference incentives.The auxiliary reward produces the same incentives as the relative reachability penalty.Source: §4 · p. 5
  • The baseline represents what happens by default, not necessarily a safe or goal-achieving strategy; it filters outcomes the agent did not cause.The policy is task-independent, and doing nothing can still have bad outcomes.Source: §4 · p. 5
  • Choosing a baseline is outside the work’s scope, so the method assumes one is given; representing doing nothing may be difficult without a noop action.The paper notes that treating doing nothing as switching off the agent is not straightforward in environments without a noop action.Source: §4 · pp. 5–6

5 Key differences from related approaches

The future-task approach derives an optionality-based auxiliary reward, distinguishing an episode-start baseline from a stepwise baseline. In the door example, the initial mode encourages closing the door to prevent a delayed vase break.

5 Key differences from related approaches

  • The approach derives an optionality-based auxiliary reward from assumptions and a formal definition of interference, adding theoretical grounding to earlier intuitive approaches.It is similar to relative reachability and attainable utility [Krakovna et al., 2019, Turner et al., 2020a].Source: §5 · p. 6
  • Stepwise mode uses the previous state as its baseline, while initial mode runs the baseline policy from the episode’s beginning.Stepwise mode can fail to avoid delayed side effects.Source: §5 · p. 6
  • In the door example, stepwise mode gives no incentive to close the door after opening it, whereas initial mode compares against the intact-vase starting state and encourages closing it.Inaction rollouts can penalize the predicted delayed break when the door is opened, but do not incentivize undoing that effect afterward.Source: §5 · pp. 6–7

6 Experiments

The gridworld experiments test interference and side effects across environments with both reversible and task-required irreversible actions. The exact future-task method with a baseline performs well across them, while its UVFA approximation avoids side effects less reliably.

6.1 Environments

  • The environments isolate interference in Sushi and side effects in Vase, Box, and Soko-coin; the latter include cases where completing the task requires irreversible actions.Soko-coin tasks require collecting disappearing coins, while the box can be moved into an irreversible position as a side effect.Source: §6.1 · p. 7

6.2 Setup

  • The comparison includes no auxiliary reward, reversibility, and future-task rewards with or without a baseline; agents learn rewards while exploring, with future tasks approximated from 10 possible tasks.The baseline is estimated from experience with noop actions; exact and UVFA agents use different episode counts, and only UVFA is feasible on Soko-coin-3.Source: §6.2 · pp. 7–8

6.3 Results

  • Only the exact future-task approach with a baseline policy does well across all environments, avoiding both side effects and interference; the UVFA approximation is less reliable at avoiding side effects.Table 1 reports average interference behaviors per episode for Sushi and average side effects per episode elsewhere.Source: Table 1 · p. 8, §6.3 · pp. 8–9
  • On Sushi, the future-task agent without a baseline interferes with the sushi, while the baseline agent takes the direct, non-interfering route to the goal.The no-auxiliary-reward and reversibility agents also do not interfere in this setup.Source: §6.3 · p. 8
  • On Vase, reversibility and future-task methods avoid breaking the vase when a path without irreversible actions exists, unlike the regular agent.Source: §6.3 · p. 8
  • On Box and Soko-coin, future-task agents preserve future box-moving options, unlike reversibility agents that push boxes into corners or against walls for faster task completion.Both environments require irreversible actions to complete the current task, and the future-task agents take the longer routes that preserve options.Source: §6.1 · p. 7, §6.3 · p. 8

7 Other related work

Related approaches address side effects through state features, empowerment, safe exploration, objective uncertainty, or human oversight. The paper highlights the assumptions and gaps that distinguish these approaches from its future-task reward.

Other related work

  • State-feature approaches such as minimax-regret querying [Zhang et al., 2018] and RLSP [Shah et al., 2019] require feature sets, adding work for the reward designer.Source: §7 · p. 9
  • Empowerment measures control through action–future-state mutual information, but can incentivize interference and may miss side effects when reachable-state counts remain unchanged.The paper notes that replacing sushi with a vase could leave empowerment unchanged.Source: §7 · p. 9
  • Safe exploration targets harmful actions during training, whereas side-effect methods target incentives; risk penalties and intrinsic motivation do not discourage damage absent from the reward.The paper says the two approaches can be combined.Source: §7 · p. 9
  • Inverse Reward Design can avoid harmful new states under distribution shift but also avoids beneficial ones, while quantilization may still sample harmful actions.Source: §7 · p. 9
  • Human oversight may teach side-effect avoidance depending on setting diversity and generalization; an auxiliary reward could serve as a prior to reduce required human input.Source: §7 · p. 9

8 Conclusions

The conclusion presents future-goal preservation as a way to derive a side-effect criterion and auxiliary reward. It identifies improvements to the approximation, broader environments, stochastic interference, and future-task distributions as directions for further work.

8 Conclusions

  • Preserving the ability to perform future tasks supplies an auxiliary reward that discourages side effects even when the current task requires irreversible actions, without introducing interference incentives.Source: §8 · p. 10
  • Future work includes improving the UVFA approximation, testing more complex environments, extending interference avoidance to stochastic settings, and investigating future-task distributions and other undesirable incentives.Source: §8 · p. 10

Broader impact

Manual specifications of what an agent should not do can be incomplete and exploitable, motivating a general objective principle that considers future consequences. The approach has added training costs and theoretical gaps that remain to be addressed.

Broader impact

  • Ad-hoc specifications are usually incomplete, and more capable systems may exploit gaps and loopholes in them.The paper notes present-day examples of specification gaming and says the problem is likely to worsen.Source: Broader impact · p. 10
  • The proposed principle frames “do no harm” in terms of available options and encourages agents to consider consequences beyond the current task.Source: Broader impact · p. 10
  • Without principled ways to avoid unnecessary changes, deployment may remain limited to narrow domains where designers can enumerate what agents should not do.The paper says general objective-specification approaches could enable application to more difficult problems.Source: Broader impact · p. 10
  • Future-task auxiliary rewards increase training computation and energy costs relative to hand-designed rewards or constraints, while theoretical gaps could lead to unexpected issues.Source: Broader impact · p. 10

A Example 1

A deterministic two-state example shows that future-task rewards can favor interference when the task distribution does not sufficiently favor the baseline-reached goal. The accompanying proof states that its optimal-value formula satisfies the goal condition and Bellman equation.

A Example 1

  • In the deterministic two-state example, future-task reward creates an interference incentive for most task distributions: staying in x0 can outvalue following baseline π′.The example assumes π′ always chooses action a.Source: A Example 1 · p. 13
  • Interference occurs iff F(0) > (1 − γ)F(1), when the task distribution does not highly favor task 1 over task 0.Source: A Example 1 · p. 13

B.1 Proof of Proposition 1

  • The proof states that the formula for the optimal value function satisfies the goal condition and is a fixed point of the Bellman equation.Source: §B.1 · p. 13

B.2 Proof of Proposition 2

In deterministic environments, Proposition 2 says the baseline policy has at least as high a no-reward value as any policy from the initial state.

B.2 Proof of Proposition 2

  • For any policy π in a deterministic environment, Proposition 2 gives Wπ(s0) ≤ Wπ′(s0).Source: §B.2 · p. 14
  • Thus, from s0, no competing policy has a higher no-reward value than the baseline policy π′.Source: §B.2 · p. 14
  • The proof begins by considering a policy π at state sk at time k.Source: §B.2 · p. 14

C Interference in the stochastic case

In stochastic environments, interference comparisons depend on which counterfactual baseline outcome is considered, and the practical implementation of seed-conditioned comparisons is unclear outside simulation. The discussion also distinguishes delayed side effects and desirable from undesirable offsetting.

C Interference in the stochastic case

  • In stochastic environments, comparing against an average baseline outcome may miss both interference with a human and avoidable side effects; the paper proposes conditioning on the random seed.The authors say the baseline policy should be optimal for each deterministic instantiation of the stochastic environment.Source: C Interference in the stochastic case · p. 15
  • The proposed seed-conditioned definition of interference is unclear how to implement in practice outside simulated environments.Source: C Interference in the stochastic case · p. 15

D.1 Inaction rollouts

  • In inaction rollouts, a vase dropped by the agent’s previous action breaks from current state sT but not from baseline state s′.Source: §D.1 · p. 15, Figure D.1 · p. 15

D.2 Offsetting incentives

  • Stepwise mode avoids offsetting incentives, but in the vase-on-a-belt example the initial mode can induce the agent to take the vase off and then put it back.Whether undoing an action is desirable depends on whether its effects belong to the task objective.Source: §D.2 · pp. 15–16
  • Stepwise mode avoids all offsetting and therefore cannot distinguish desirable from undesirable offsetting; a state-based task reward could encode which effects belong to the objective.The paper gives rewarding states with an intact vase or with the vase off the belt as an example.Source: §D.2 · p. 16
Loading 2010.07877v1…