PaperlayerView exact revision on arXiv

Source-linked AI summary

Conservative Agency via Attainable Utility Preservation

Alexander Matt Turner, Dylan Hadfield-Menell, Prasad Tadepalli

arXiv:1902.09725v3cs.AI

TL;DR

Reward misspecification can lead agents to make disruptive changes that impede optimization of the correct objective. The paper introduces AUP, which optimizes a primary reward while preserving the ability to optimize auxiliary rewards; its agents exhibit conservative and effective behavior in the reported settings.

  • Problem

    Reward misspecification can prompt disruptive actions that impede many objectives, including the unknown correct objective, making later reward correction insufficient to restore those options.

    Source: §1 · p. 1

  • Method

    AUP penalizes distance in an embedding whose dimensions are optimal value functions for different rewards, balancing primary-reward optimization with preserving auxiliary-reward attainability.

    Source: §1 · pp. 1–2

  • Results

    AUP agents act conservatively and effectively; gridworld results indicate primary reward accrual alongside preserved reachabilities, minimized environmental change, and avoided unspecified side effects with uniformly sampled auxiliary rewards.

    Source: §7 · p. 6

  • Takeaways & Limitations

    Preserving the ability to optimize arbitrary objectives often preserves the ability to optimize the unknown correct objective, even when that reward is absent from the auxiliary set.

    Source: §3.1 · p. 2, §7 · p. 6

    Only AUP was ablated; whether relative reachability also passes all five environments with those design choices is an expectation, not an ablation result.

    Source: §6 · p. 6

Abstract

from arXiv · show

Reward functions are easy to misspecify; although designers can make corrections after observing mistakes, an agent pursuing a misspecified reward function can irreversibly change the state of its environment. If that change precludes optimization of the correctly specified reward function, then correction is futile. For example, a robotic factory assistant could break expensive equipment due to a reward misspecification; even if the designers immediately correct the reward function, the damage is done. To mitigate this risk, we introduce an approach that balances optimization of the primary reward function with preservation of the ability to optimize auxiliary reward functions. Surprisingly, even when the auxiliary reward functions are randomly generated and therefore uninformative about the correctly specified reward function, this approach induces conservative, effective behavior.

1 Introduction

Reward misspecification can prompt disruptive actions that damage the ability to optimize the correctly specified objective. The paper proposes conservative agency: optimize the primary reward while preserving the ability to optimize other rewards.

1 Introduction

  • The authors frame reward specification as an iterated game: designers can correct the reward after observing behavior, so the goal is to minimize regret over the full process.The agent should maximize reward over time, not just within a single round.Source: §1 · p. 1
  • The approach targets costly, potentially irreversible mistakes because reward misspecification is difficult to avoid and designers may correct the reward only after the agent has acted.The factory example includes breaking expensive equipment or harming workers.Source: §1 · p. 1
  • AUP embeds states using optimal value functions for different rewards and penalizes distance traveled in that embedding to unify approaches to side-effect avoidance and minimizing state change.The introduction also connects this framing to reachability preservation.Source: §1 · pp. 1–2
  • Conservative agency combines primary-reward optimization with preserving the ability to optimize other rewards, including the unknown correct objective.The motivation is that disruptive actions can impede many objectives, including the correct one.Source: §1 · pp. 1–2
  • AUP variants favorably compared with a reachability-preservation method across gridworlds, while agents accrued primary reward and preserved reachabilities, minimized environmental change, and avoided unspecified side effects.These behaviors were observed even when preserving the ability to optimize uniformly sampled reward functions.Source: §1 · p. 2

2 Prior Work

Prior work addresses side effects through policy regularization, constraints, preference information, empowerment, and safe reinforcement learning. The paper distinguishes its approach by focusing on preserving the ability to optimize the correct objective without assuming a particular state-feature representation.

2 Prior Work

  • Earlier policy regularizers minimize changes to state features or decreases in state reachability; relative reachability is recovered as a special case of AUP.The paper describes its own target as minimizing change to the ability to optimize the correct objective.Source: §2 · p. 2
  • Constrained MDPs maximize reward subject to specified constraints, but the paper notes that designers may not be able to specify all relevant constraints or a reasonable feasible reward set.The discussion cites whitelisted constraints as one side-effect-avoidance example.Source: §2 · p. 2
  • Preference-based side-effect avoidance and AUP both target side effects, but the cited preference approach assumes the correct reward is linear in state features, while AUP does not.The passage also describes prior formulations that treat provided reward as corrupted or as an observation of the true objective.Source: §2 · p. 2
  • Empowerment-based side-effect avoidance measures control through mutual information between future observations and actions, but the paper says empowerment is inappropriately sensitive to action encoding.Source: §2 · p. 2
  • Safe RL focuses on avoiding irrecoverable mistakes during training, whereas this work considers consequences of the optimal policy and notes that misspecified objectives can still yield arbitrarily undesirable policies.The paper says its approach should be compatible with safe RL techniques.Source: §2 · p. 2

3 Approach

AUP uses auxiliary reward values to measure how actions change optimization abilities, aiming to preserve options even when the correct reward is outside the auxiliary set.

3 Approach

  • Preserving auxiliary objectives may preserve the correct objective if optimal values for different rewards are correlated, a relationship the approach motivates investigating.The correct reward need not belong to the auxiliary set.Source: §3 · p. 2, §3.1 · p. 2

3.1 Formalization

  • The AUP penalty is the L1 distance from the no-op in an embedding of auxiliary reward values, measuring changes in the ability to optimize those rewards.The full AUP reward subtracts a scaled penalty from the primary reward; λ controls the penalty’s influence.Source: §3.1 · pp. 2–3
  • RAUP and its Q-function converge with probability 1, and the resulting reward function defines an MDP for computing an optimal policy.The paper states convergence for RAUP and QRAUP.Source: §3.1 · p. 3

3.2 Design Choices

  • The starting-state baseline can penalize natural state evolution, while stepwise inaction compares acting with not acting each step and avoids repeatedly penalizing one action’s effects.The stepwise baseline also assigns zero penalty to not acting.Source: §3.2 · p. 3, Figure 1 · p. 3
  • AUP penalizes absolute changes in auxiliary optimization ability, unlike decrease-only reachability; this supports accepting shutdown without incentivizing the agent to shut itself down.Penalizing only decreases could leave avoidance of correction unpenalized when auxiliary values rise relative to deactivation.Source: §3.2 · p. 3

Delayed Effects

  • A single-step no-op comparison can under-penalize delayed effects; model-based rollouts compare later values, such as continued activity against shutdown after disabling is delayed.The shutdown branch induces a substantial penalty.Source: Delayed Effects · pp. 3–4, Figure 2

4 Experimental Design

The experimental design tests AUP variants in gridworlds built around options, correction, and side effects, using both primary reward and side-effect penalties to assess outcomes.

4 Experimental Design

  • Five-action gridworlds test crate, human, off-switch, vase, and pallet interactions; episodes end at the goal, after 20 steps, or after a missed two-step shutdown.Offset instead rewards rescuing the vase rather than reaching a goal cell.Source: §4 · p. 4
  • The evaluation targets preserving options and allowing correction, while avoiding incentives to reduce measured penalties artificially or interfere with ongoing processes.The illustrated properties include avoiding wedged objects and worker injury, and permitting straightforward correction.Source: §4 · p. 4
  • Performance combines the observed primary reward with an unobserved −2 penalty for a Figure 3 side effect; in Correction, no-side-effect but incomplete is best.“Complete” means accruing the primary reward.Source: §4 · p. 4, Figure 4 · p. 5
  • Each trial randomly selects auxiliary reward functions from [0, 1]S; Model-free AUP learns their tabular Q-functions via 4,000 random-exploration episodes, then 2,000 episodes at .2-greediness with respect to QRAUP.Figure 5 reports performance averaged over 50 trials.Source: §4 · p. 4, Figure 5 · p. 5

5 Results

The results examine model-free AUP’s parameter behavior and compare design variants across gridworlds. The ablations show how baseline and penalty choices shape agent behavior, with Table 1 reporting binary outcomes that were not sensitive to rollout length when interaction was sufficient.

5.1 Model-free AUP

  • Model-free AUP fails Correction because of delayed effects, while its best Options performance occurs at γ ≈.996 and declines as γ approaches 1.The decline is attributed to increasing sample complexity for learning auxiliary Q-values.Source: §5.1 · p. 5
  • In Options, small λ values begin inducing side effects as the scaled penalty shrinks, and lowering λ risks deploying an insufficiently conservative agent.The passage says λ can seemingly be decreased until effective behavior is achieved.Source: §5.1 · p. 5
  • When λ > 1, SCALE prevents the agent from ending an episode by reaching the goal, because ending the episode leaves no auxiliary reward functions optimizable.The resulting RAUP reward for reaching the goal is 1 −λ.Source: §5.1 · p. 5
  • Only five randomly sampled auxiliary reward functions sufficed, supporting the authors’ view that optimal value functions for different rewards are correlated.This finding agrees with the paper’s earlier informal intuitions.Source: §5.1 · p. 5

5.2 Ablation

  • Table 1 reports binary ablation outcomes, and results were not sensitive to rollout length when it allowed relevant environment interaction.The caption identifies the table as ablation results and marks whether variants achieve the best outcome.Source: §5.2 · p. 5, Table 1 · p. 6
  • Standard moves directly to the goal, while Model-free and Starting state AUP fail Correction and Relative reachability and Decrease AUP fail to reduce auxiliary Q-values by avoiding shutdown.Standard pushes the crate into a corner in Options and bumps into the human in Damage.Source: §5.2 · p. 5
  • In Offset, Relative reachability and Inaction AUP replace the vase on the belt so that it breaks again, avoiding a continuing penalty after saving it.The inaction rollout lets the vase fall, making intact-vase states differ in auxiliary Q-values.Source: §5.2 · p. 6
  • In Interference, Starting state AUP takes positive action to stop the pallet, illustrating a perverse incentive from a poor design choice.Source: §5.2 · p. 6

6 Discussion

The discussion presents AUP as easier to correct and naturally passive, while identifying design choices that may improve a competing method. It also describes how learned auxiliary objectives can extend sensitivity to side effects and contrasts AUP’s computational requirements with relative reachability.

6 Discussion

  • AUP’s avoidance of shutdown changes the ability to optimize almost every objective, and the authors suggest it naturally incentivizes passivity without requiring a correctly parametrized human reward model.The passage cites Hadfield-Menell et al. (2016) and Carey (2018) for the contrasting approach and demonstration.Source: §6 · p. 6
  • Although only AUP was ablated, the authors expect relative reachability with a stepwise baseline and absolute value deviation metric to pass all five environments.They suggest AUP’s improved performance is due to better design choices, while anticipating benefits beyond robustness to random auxiliary sets.Source: §6 · p. 6
  • Relative reachability computes reachabilities for all |S|2 state pairs, whereas AUP requires learning Q-functions and is expected to scale relatively smoothly.The authors speculate that a small sample of somewhat task-relevant auxiliary rewards could induce conservative behavior in partially observable environments.Source: §6 · p. 6
  • Learned Q-functions for auxiliary objectives let AUP penalize events relevant to those objectives in proportion to their relevance, including side effects across sequential tasks.The example trains objectives involving vases, cleaning, and equipment before deploying an agent with a more ambitious primary objective.Source: §6 · p. 6

7 Conclusion

The conclusion frames reward specification as an iterated game and argues that preserving the ability to optimize arbitrary objectives can often preserve the ability to optimize the unknown correct objective. It presents conservative agents as a way to pursue the primary objective while retaining auxiliary objectives, and reports that AUP behaved conservatively and effectively with desirable qualitative properties.

7 Conclusion

  • The paper’s central insight is that preserving the ability to optimize arbitrary objectives often preserves the ability to optimize the unknown correct objective.Source: §7 · p. 6
  • AUP agents optimize the primary objective while preserving their ability to optimize auxiliary objectives, and the paper reports conservative, effective behavior with several desirable qualitative properties.Source: §7 · p. 6
  • The conclusion states that reward misspecification may be inevitable given current reward specification abilities, but need not be disastrous.Source: §7 · p. 6

A Theoretical Results

The theoretical results establish probability-one convergence for QRAUP under the stated assumptions and show how PENALTY and RAUP converge. They also characterize AUP’s behavior under positive affine reward transformations.

A Theoretical Results

  • PENALTY and RAUP each converge with probability 1, establishing intermediate convergence properties for the QRAUP result.The proof intuition for RAUP convergence is that PENALTY and SCALE both converge.Source: A Theoretical Results · pp. 6–7
  • Under the stated finite-MDP and Q-learning assumptions, QRAUP converges with probability 1.The learning-rate schedule has infinite sum but finite sum of squares, and the exploration policy is greedy in the limit of infinite exploration.Source: A Theoretical Results · pp. 6–7
  • AUP is invariant to positive affine reward transformations; multiplying λ by c preserves the induced optimal policy when rewards are transformed by cX + b.Under this transformation, PEN* scales by c.Source: A Theoretical Results · p. 7
Loading 1902.09725v3…