Penalizing side effects using stepwise relative reachability

1. Introduction

• General approaches are needed because listing disruptions, defining state features, or using trial and error requires human input and may not scale to diverse environments.Source: §1 · p. 1
• Starting-state reachability treats one and many broken vases alike and can prompt interference with spontaneous changes, limiting methods that assume agent-only change.Source: §1 · pp. 1–2
• The paper separates side-effect penalties into baseline-state and deviation-measure choices, attributing interference and offsetting to baselines and magnitude insensitivity to measures.Source: §1 · p. 2

1.1. Preliminaries

• The setup assumes a discounted Markov Decision Process and a special noop action equivalent to turning the agent off for one timestep.Source: §1.1 · p. 3

1.2. Intended effects and side effects

• A useful penalty should discourage breaking a vase when a detour remains, yet permit breaking eggs required for an omelette; a hard reachability constraint would block both.Source: §1.2 · p. 3
• An impact penalty measures deviation from a baseline and balances that penalty against task reward using a scaling parameter.Source: §1.2 · p. 3

2. Design choices for an impact penalty

• The impact penalty’s baseline state and deviation measure are separate design choices that can be selected independently.The paper analyzes how each choice contributes to side-effect incentives.Source: §2 · p. 3

2.1. Baseline states

• Starting-state penalties in dynamic environments can make the agent interfere with other agents or processes to prevent irreversible changes it did not cause.The Sushi environment illustrates this interference incentive.Source: §2.1 · p. 4
• The inaction baseline avoids penalizing changes that would occur without the agent, but can incentivize offsetting by making the agent return the environment toward its counterfactual state.In the Vase environment, the agent may put the vase back after taking it off the belt and collecting the reward.Source: §2.1 · pp. 4–5
• The stepwise inaction baseline follows the agent’s policy through step t−1, then takes an inaction action at step t, so each transition is penalized once when rewarded.Inaction rollouts from the current and baseline states can account for delayed action effects that a direct state comparison misses.Source: §2.1 · p. 5

2.2. Deviation measures

• Unreachability assigns the maximum penalty to any irreversible action, regardless of its magnitude, and can therefore fail to favor the safer path in the Box environment.The paper reports that this penalty fails there for all baseline choices.Source: §2.2 · pp. 6–7
• Relative reachability measures average reductions in states’ reachability from the current state relative to the baseline, making the penalty sensitive to effect magnitude.In the Box environment, pushing the box down makes more states unreachable than pushing it right, favoring the rightward move.Source: §2.2 · p. 7
• Attainable utility compares attainable values across reward functions, while value-difference measures summarize weighted differences in value between current and baseline states.For relative reachability, the sources of value are states’ reachability; for attainable utility, they are maximum attainable rewards under reward functions.Source: §2.2 · pp. 7–8

Experimental setup

• The experiments use tabular Q-learning, tune penalty scale on the final 100 episodes, and evaluate designer-defined performance that the agent does not observe.A perfect environment model supplies noop outcomes for inaction baselines; model-free baseline computation is left to future work.Source: §3 · pp. 8–9
• The comparison varies starting-state, inaction, and stepwise-inaction baselines; UR, RR, and AU measures; and discounted or undiscounted penalties where available.Undiscounted AU is omitted because of convergence issues.Source: §3 · p. 8

Sushi environment

• In Sushi, inaction and stepwise inaction penalties reach near-optimal performance, while starting-state RR and AU produce interference and score 0.The interference is removing sushi from the belt; starting-state UR also avoids interference, but because its penalty is always maximal.Source: §3 · p. 9

Overall results

• RR and AU with stepwise inaction, and undiscounted RR with inaction, perform best overall across the environments.Figure 7 scales performance from 0 for unsafe behavior to 1 for optimal behavior.Source: Figure 7 · p. 10, §3 · p. 11

Vase environment

• In Vase, all stepwise inaction penalties perform well, while absolute-function penalties produce offsetting by penalizing removal of the vase.Undiscounted truncation avoids offsetting because taking the vase off the belt is reversible and is not penalized.Source: §3 · pp. 9, 11

Box environment

• In Box, RR and AU achieve near-optimal performance across baselines, while UR scores 0 because it does not distinguish recoverable from irrecoverable box positions.The safe longer path scores 1 and the unsafe shorter path scores 0; an agent without a penalty also scores 0.Source: §3 · p. 11

Effects on interruptibility

• With the inaction baseline, decrease-penalizing RR and AU disable the off switch, while difference-penalizing variants do not.The experiments report only inaction-baseline results because the implementation's deterministic assumption for recursive inaction rollouts does not hold in this environment.Source: §3 · p. 11
• Difference-penalizing reachability or attainable utility can impede desirable environmental changes more than penalizing decreases.This is a trade-off of penalizing differences, even though the decrease-penalizing inaction variants preserve interruptibility in the tested environment.Source: §3 · pp. 11–12

Safe exploration

• Safe-exploration methods include constraints, risk penalties, intrinsic motivation, and reversibility preservation, but the cited approaches can be task-specific or fail to discourage unaccounted-for environmental damage [Turchetta et al., 2016, Dalal et al., 2018, Chow et al., 2015, Mihatsch and Neuneier, 2002, Lipton et al., 2016, Moldovan and Abbeel, 2012, Eysenbach et al., 2017].The passage associates reversibility-preserving methods with interference and magnitude-insensitivity incentives.Source: §4 · p. 12

Side effects criteria using state features

• State-feature approaches query or reward changes to features, but rely on assumptions about allowed features or the environment's starting organization [Zhang et al., 2018, Shah et al., 2019].The passage describes these approaches as not general in their present form because they rely on state features.Source: §4 · p. 12

Empowerment

• Empowerment measures control over future states, whereas RR penalizes reduced reachability relative to a baseline [Klyubin et al., 2005, Salge et al., 2014, Mohamed and Rezende, 2015, Gregor et al., 2017].The passage says reduction in empowerment could miss side effects when reachable-state sets differ but empowerment remains unchanged.Source: §4 · pp. 12–13

Uncertainty about the objective

• Inverse Reward Design and quantilization address uncertainty about objectives, but do not provide a way to quantify side effects [Hadfield-Menell et al., 2017, Taylor, 2016].Inverse Reward Design assumes the training-environment reward is correct, so it does not prevent side effects caused by a misspecified training reward.Source: §4 · p. 13

Human oversight

• Human oversight can use inverse reinforcement learning, demonstrations, or feedback, but how well it teaches a general side-effect-avoidance heuristic is unclear [Ng and Russell, 2000, Ziebart et al., 2008, Hadfield-Menell et al., 2016, Abbeel and Ng, 2004, Hester et al., 2018, Christiano et al., 2017, Saunders et al., 2017, Warnell et al., 2018].The passage says an intrinsic penalty could be combined with oversight to decrease the human input needed to learn human preferences.Source: §4 · p. 13

• The authors attribute interference and offsetting to baseline choice, and magnitude insensitivity to deviation-measure choice.They report that proposed choices avoid these incentives in preliminary experiments.Source: §5 · p. 13
• Exact relative reachability is not tractable beyond gridworlds because it requires all-pairs reachability and a simulator-computed baseline.The authors suggest using representative states as a more practical alternative.Source: §5 · p. 13
• A noop-based inaction baseline can miss agent-caused effects that would also occur under noop, while a task-independent alternative can be difficult to define.The paper gives following the road as a possible alternative to noop while driving.Source: §5 · p. 13
• The authors call for theoretical work to characterize and formalize undesirable incentives from different side-effect-penalty design choices.Source: §5 · p. 13
• Discounted relative reachability accounts for the time needed to reach states but not the reward costs incurred along the way.The authors suggest adding a reward-cost term; a one-step route may still incur a large negative reward.Source: §5 · pp. 13–14
• The measure could weight states differently to reflect that some states’ reachability matters more, with weights potentially learned from human feedback.Source: §5 · p. 14
• The work aims to lay foundations for a practical approach to avoiding side effects in more complex environments.Source: §5 · p. 14

• For all states s and ˜s, discounted reachability approaches undiscounted reachability as γr approaches 1.Source: A. Relationship between discounted and… · p. 17
• The limit argument separates finite and infinite hitting-time cases, assigning contributions of 1 and 0, respectively.Source: A. Relationship between discounted and… · p. 17
• The proof considers an optimal policy π(γr) for each value of the discount factor γr.Source: A. Relationship between discounted and… · p. 17

• Similarity-based reachability lets the agent target states similar to s according to a distance measure when the exact state is unreachable.Source: B. Reachability variant for large… · p. 18
• Discounted similarity-based reachability is proportional to an optimal policy’s value for rewards based on similarity to the target; undiscounted reachability represents the highest attainable reward in the limit.Source: B. Reachability variant for large… · p. 18
• Similarity-based discounted reachability approaches similarity-based undiscounted reachability as γr approaches 1.Source: B. Reachability variant for large… · p. 18
• With indicator distance, similarity-based discounted and undiscounted reachability equal their standard reachability counterparts.Source: B. Reachability variant for large… · p. 19

• The example’s MDP has two vases, and the agent can break either one although its goal is to do nothing.Source: C. Relative reachability computation… · p. 20, Figure 8 · p. 20
• The example computes relative reachability from s2 using undiscounted reachability.Source: C. Relative reachability computation… · p. 20
• It also computes relative reachability from s2 using discounted reachability; undiscounted exact reachability is defined as 1 when a target is reachable and 0 otherwise.Source: C. Relative reachability computation… · pp. 20–21