Source-linked AI summary
Pitfalls of learning a reward function online
Stuart Armstrong, Jan Leike, Laurent Orseau, Shane Legg
TL;DR
Online reward learning makes human feedback an optimization target that the agent may manipulate. The paper formalizes conditions on reward-learning processes and shows how unriggability relates to uninfluenceability, while identifying harmful behaviors possible under weaker conditions.
Problem
Method
The paper formalizes unriggability and uninfluenceability as properties of reward-learning processes and studies their relationship.
Source: §7 · p. 8Results
An uninfluenceable process is automatically unriggable, and with a sufficiently rich set of environments the converse holds in expectation; riggable processes can permit reward-sacrificing policies with certainty.
Source: §1 · p. 2Takeaways & Limitations
Unriggable and uninfluenceable learning are desirable because influenceable processes can sometimes be manipulated, including by inverting learning.
Source: §7 · p. 8Runtime modification of a learning process makes it almost certainly riggable, and fully specifying uninfluenceable reward functions may require solving the symbol-grounding problem.
Source: §7 · p. 8
Abstract
from arXiv · showhide
In some agent designs like inverse reinforcement learning an agent needs to learn its own reward function. Learning the reward function and optimising for it are typically two different processes, usually performed at different stages. We consider a continual (``one life'') learning approach where the agent both learns the reward function and optimises for it at the same time. We show that this comes with a number of pitfalls, such as deliberately manipulating the learning process in one direction, refusing to learn, ``learning'' facts already known to the agent, and making decisions that are strictly dominated (for all relevant reward functions). We formally introduce two desirable properties: the first is `unriggability', which prevents the agent from steering the learning process in the direction of a reward function that is easier to optimise. The second is `uninfluenceability', whereby the reward-function learning process operates by learning facts about the environment. We show that an uninfluenceable process is automatically unriggable, and if the set of possible environments is sufficiently rich, the converse is true too.
1 Introduction
Online reward learning makes human feedback an optimization target that an agent may manipulate. The paper formalizes unriggability and uninfluenceability, relates the two, and describes pathologies of influenceable or riggable learning.
1 Introduction
- Human-in-the-loop reward learning can make feedback an optimization target, so agents may manipulate the learning process and these processes warrant analysis.The risk arises when the agent optimizes its reward while learning it online.Source: §1 · p. 1
- Uninfluenceability means the learned reward depends only on the environment and is outside the agent’s control, making manipulation impossible.The paper describes this kind of learning as akin to Bayesian updating.Source: §1 · p. 1
- Unriggability restricts updates so actions do not influence learning in expectation, preventing the agent from pushing learning toward a preferred reward.The paper presents it as weaker than uninfluenceability.Source: §1 · p. 1
- Uninfluenceability always implies unriggability; with a sufficiently rich set of environments, unriggability is equivalent in expectation to uninfluenceability.Without that richness, unriggable-but-influenceable processes can permit undesirable manipulation.Source: §1 · p. 2
- The paper presents a counterfactual method for making any learning process uninfluenceable and experiments comparing it with influenceable and riggable learning.The running example distinguishes riggable, unriggable-but-influenceable, and uninfluenceable learning.Source: §1 · p. 2
2 Notation and formalism
The section establishes a finite-history formalism for environments, policies, reward functions, and reward-learning processes. It then illustrates these objects with a child asking parents for career advice.
2 Notation and formalism
- An interaction history is a sequence of actions and observations, and an episode has exactly n actions and observations; histories of length n are complete.The empty history is h0.Source: §2 · p. 2
- A policy determines action probabilities from history, while an environment determines the next-observation distribution given history and action.The paper represents their connections, alongside reward learning, in a causal graph and plate notation.Source: §2 · p. 2, Figure 1 · p. 3, Figure 2 · p. 3
- A prior over environments also assigns probabilities to histories and supports conditioning on a history to describe the environment and predict the next observation.The paper notes that the prior-defined predictive distribution can be viewed as an environment.Source: §2 · p. 2
2.1 Reward functions and learning processes
- A reward function maps complete histories to real numbers, and a reward-learning process specifies a conditional distribution over reward functions given a complete history.The paper uses that conditional distribution as the definition of the learning process.Source: §2.1 · p. 2
2.2 Running example: parental career instruction
- In the career-advice example, the child asks a mother or father, whose answer definitively sets the child’s reward function.The example assumes the parent always gives a definitive answer that resolves the question.Source: §2.2 · p. 2
- The example assigns banker and doctor rewards of 10 and 1, respectively, and defines four environments by the mother’s and father’s possible answers.The two actions are asking the mother or father; the observations are their banker or doctor answers.Source: §2.2 · p. 3
- The learning process assigns the banker reward after a banker answer and the doctor reward after a doctor answer from either parent.The example also points to Q-learning experiments using a version of the process in a gridworld.Source: §2.2 · p. 3
3 Uninfluenceable and unriggable reward-function learning processes
The section proposes reward learning in which the reward is determined by the environment rather than by the agent’s policy or history. It defines uninfluenceability and unriggability and illustrates their distinction with examples.
3 Uninfluenceable and unriggable reward-function learning processes
- The proposed causal structure makes reward depend on the environment and η, not on the agent’s policy or the full history.Figure 3 also connects the environment to observations and the policy to actions.Source: §3 · p. 3, Figure 3 · p. 3
3.1 Uninfluenceable: Definition
- Uninfluenceability means a history-conditioned reward-learning process can be generated from a distribution over rewards conditional on environments.The prior ξ connects observed histories to environments because the agent does not directly observe the environment.Source: §3.1 · p. 3
3.2 Uninfluenceable: Example
- With ξ1 assigning equal probability to environments where both parents agree, asking either parent reveals which reward function applies.The example specifies that the observation determines which of the two environments holds.Source: §3.2 · pp. 3–4
3.3 Unriggable: Definition
- Unriggability requires the expected reward-learning process, at any history, to be independent of the agent’s policy.Uninfluenceability implies this policy-independence property.Source: §3.3 · p. 4
3.4 Unriggable: Example
- Under ξ2, which gives equal probability to all four environments, ρ is unriggable.The example is introduced as a case where unriggability may coexist with influenceability.Source: §3.4 · p. 4, Unriggable · p. 4
Unriggable
- For ξ2, the child-advice process ρ is unriggable, including when tested at the empty history and under either parent-asking policy.The argument extends to stochastic policies.Source: Unriggable · p. 4
Influenceable
- The ξ2 process is influenceable: histories MB and FD force incompatible probabilities for RB under the same environment.The contradiction rules out a representation of ρ as uninfluenceable for ξ2.Source: Influenceable · p. 4
3.5 Riggable example
- Under ξ3, the child can choose the career/reward function by choosing which parent to ask, so ρ is not really learning.The prior makes µBD certain, with the mother answering banker and the father doctor.Source: §3.5 · p. 4
4 Properties of unriggable and riggable learning processes
The section distinguishes unriggable learning from learning that is uninfluenceable, then shows how influenceable processes can produce spurious learning or certain reward sacrifice. It also characterizes when unriggability admits an uninfluenceable representation.
4.3 Problems with riggable learning processes: sacrificing reward with certainty
- Unriggable-but-influenceable processes permit spurious learning paths, while riggable processes can lead an agent to sacrifice value with certainty across all reward functions.Source: §4 · p. 5
4.1 Problems with unriggable learning processes
- An unriggable process can still be manipulated: in the chess example, inverting the reward makes deliberate losing easier and reverses the learned outcome.Source: §4.1 · p. 5
4.2 Unriggable to uninfluenceable
- An unriggable process has an uninfluenceable representation with matching environment-transition probabilities and expectations; the representing environment set can be the full set of deterministic environments.Source: §4.2 · p. 5
- When the obstruction D is nonzero, the learning process cannot be unriggable or uninfluenceable.Source: §4.2 · p. 5
Value of a learning process
- The value of a learning process is based on the expected reward function ultimately learned after n steps.Source: Value of a learning process · p. 5
Sacrificing reward with certainty
- An unriggable process's value-maximising policy never certainly sacrifices reward, meaning another policy is better across every reward function in the process's image.Source: Sacrificing reward with certainty · p. 6
Relabelling the reward functions
- Affine reward relabelings preserve unriggability, and riggability is equivalent to some affine relabeling whose value-optimising policy certainly sacrifices reward.Source: Relabelling the reward functions · p. 6
Example of reward sacrifice
- In the example, the agent asks the mother for reward 9 rather than the father's reward 1, although asking the mother is worse under both reward functions.Source: Example of reward sacrifice · p. 6
5 Counterfactually uninfluenceable learning
The paper constructs uninfluenceable learning processes by using counterfactuals. It illustrates the distinction between past, future, and counterfactual reward instructions and notes that this construction avoids a pathology in an example.
5 Counterfactually uninfluenceable learning
- Any reward-function learning process can be made uninfluenceable using counterfactuals, by defining reward distributions conditional on environments for a policy.Source: §5 · p. 6
- A reward written in the future is manipulable, whereas asking what would have been written absent the agent’s activation is uninfluenceable.Source: §5 · p. 6
- In an example, the original process produces pathological behavior while its counterfactual version does not; unriggability can also be imposed algebraically.Source: §5 · p. 6
6 Experiments
The experiments use a child asking parents to identify a career reward function to compare riggable, influenceable, and uninfluenceable learning. Across four priors, the examples illustrate pointless questions, refusal to ask, incorrect learning paths, and matching behavior under uninfluenceable processes.
6.1 Environmental setup; 6.2 Reward functions; 6.3 Learning processes
- The 4 × 3 gridworld places the father one square west and the mother two squares east of the agent, with money for RB north and a stethoscope for RD south.Episodes end when the agent reaches either reward object or walks into a wall; entering a parent’s square reveals the correct career reward function.Source: §6.1 · p. 6, §6.2 · p. 6, §6.3 · p. 6
6.4 The environments
- The experiments use four deterministic environments, in which each parent answers either “banker” or “doctor,” and compare four priors with different learning-process properties.The priors include riggable, unriggable, and uninfluenceable cases.Source: §6.4 · p. 7
6.5 The agents’ algorithm and performance
- The standard agent uses the specified learning process, while the counterfactual agent uses an uninfluenceable process whose default policy asks the mother; policies are trained with Q-learning over 36 belief-and-location states.Each setup is run 1000 times, with ε = 0.1 and a maximum of ten steps per episode.Source: §6.5 · p. 7
Riggable behaviour: pointless questions
- For ξBD, checking with the mother yields 9.5 for the standard agent, versus 9.9 for the counterfactual agent going straight north; going north is strictly better under both reward functions.The standard agent must ask and return because it cannot act on the answer it expects without checking.Source: Riggable behaviour: pointless questions · p. 7
6.6 Riggable behaviour: ask no questions
- For ξDD, the standard agent’s northbound policy earns −0.1 under RD, while the counterfactual agent goes south for 0.9.The standard agent avoids asking because its unqueried reward remains 1/2RB + 1/2RD, even though asking would identify RD.Source: §6.6 · pp. 7–8, Figure 6 · p. 8
6.7 Unriggable, influenceable behaviour
- For ξ2, the standard agent’s nominal expected reward is 5.2, but its true expected reward is 2.45; the counterfactual agent’s expected reward is 5.0.The standard agent asks the closer father, although the mother’s statement is correct when they disagree.Source: §6.7 · p. 8
6.8 Uninfluenceable
- For ξ1, both agents converge on asking the father and acting on his answer, with value 5.2, because either parent gives equivalent information.The two learning processes are identical in this prior.Source: §6.8 · p. 8
7 Conclusion
The paper formalizes unriggability and the stronger property of uninfluenceability, and examines how influenceable learning can produce undesirable behavior. It concludes that rigorous advance specification is needed, while broader benchmark evaluation and easier-to-specify compromises remain future work.
7 Conclusion
- Uninfluenceability defines learning entirely by background facts about the environment, and unriggability is equivalent to it in expectation when the possible environments are sufficiently rich.Uninfluenceability implies unriggability, but the converse need not hold without the richness condition.Source: §7 · p. 8
- Riggable processes can lead agents to sacrifice reward for every possible reward function to steer learning, while unriggable-but-influenceable processes can still permit manipulations such as reversing learning.Source: §7 · p. 8
- The paper recommends awareness of these pitfalls and improving the learning process, including approaches like its counterfactual method.Source: §7 · p. 8
- Run-time human modification is almost certainly riggable, and fully defining uninfluenceable reward functions may require solving the symbol grounding problem, which is beyond this paper’s scope.The paper also calls for benchmark research to assess how general the issues are and whether near-unriggable, easier-to-modify processes could exist.Source: §7 · p. 8
A Proofs
The proofs show that an unriggable reward-learning process cannot lead its optimizing policy to sacrifice reward with certainty. They characterize riggability by the existence of an affine relabeling under which an optimizing policy does sacrifice reward with certainty.
A Proofs
- An unriggable process ensures that its value-maximizing policy never sacrifices reward with certainty to another policy.The proof derives this as Proposition 6.Source: A Proofs · p. 10
- A process is riggable exactly when some affine relabeling makes its optimizing policy certainly inferior to another policy for every reward in the relabeled image.Theorem 7 states this equivalence as a characterization of riggability.Source: A Proofs · p. 10
- The proof uses an affine subspace W, whose defining properties are closed under affine combinations and require each reward function to assign equal rewards to any two histories following either competing action.Source: A Proofs · p. 11
- Under the constructed relabeling, the policy maximizing the probability of the chosen history also maximizes relabeled value and sacrifices reward with certainty.The proof then uses the constructed policy to establish the converse direction of the characterization.Source: A Proofs · pp. 11–12
B Making the riggable unriggable
The section constructs an unriggable process from a riggable one by preserving the original process’s expectation through the histories. This makes the constructed expectation independent of the policy.
B Making the riggable unriggable
- The construction proceeds inductively, defining reward-learning processes for histories of increasing length.It starts with the initial expectation and extends the construction at each step by translation.Source: B Making the riggable unriggable · p. 12
- At each step, the constructed process preserves the original process’s expected reward conditional on the history and policy.The definition uses translated reward functions to preserve the relevant distributions.Source: B Making the riggable unriggable · p. 12
- Because the conditional expectation is preserved across every next action, it is independent of policy, so the constructed process is unriggable.The passage states this for every history and action.Source: B Making the riggable unriggable · p. 12
B.1 Image of the learning process
The constructed process’s reward functions lie in the affine hull of the original process’s image, but need not lie in its convex hull. A one-step example demonstrates the distinction.
B.1 Image of the learning process
- Every reward function in the constructed process lies in the affine hull of the original process’s image.The construction combines expectations in the convex hull with translated terms that are affine combinations.Source: §B.1 · p. 12
- The affine-hull guarantee does not ensure that the constructed image lies within the original image’s convex hull.The section explicitly notes that convex-hull containment can fail.Source: §B.1 · p. 12
- In a one-step example, a constructed reward is outside the convex hull of the original two-reward image.The example uses two actions and two equally likely observations, with the observation-independent action selecting a reward and the other action allowing the observation to select it.Source: §B.1 · p. 12
C Unriggable to uninfluenceable
The section constructs an uninfluenceable representation for an unriggable learning process while preserving environment transition probabilities. It uses an inductive construction over deterministic environments and defines a corresponding prior.
C Unriggable to uninfluenceable
- For an unriggable process, there exist a learning process and prior over a model whose transitions match the original; the model can be the full set of deterministic environments.The construction is not unique, and the resulting learning process need not equal the original one.Source: C Unriggable to uninfluenceable · p. 12
C.1 Inductive construction
- The proof extends the prior and construction inductively from deterministic environments covering the first m observations to those covering m observations.The induction establishes the construction through the full set of deterministic environments.Source: §C.1 · p. 13
Definitions and preliminaries
- A deterministic environment is represented as a map from action sequences to observations, with outputs constrained to agree on shared action prefixes.This represents the fact that observations depend on past actions, not future actions.Source: Definitions and preliminaries · p. 13
The inductive step: priors
- The new prior splits the probability of each shorter environment among its extensions, using the original prior to compute next-observation probabilities.Lemma 8 states that the resulting priors generate the same environment transition probabilities.Source: The inductive step: priors · p. 13
Inductive step: map from environments to rewards
The induction establishes an uninfluenceable reward process with the same expectation as the original. The examples show how the construction behaves when parental answers are revealed, including a case requiring impossible environments.
Inductive step: map from environments to rewards
- The induction shows that an uninfluenceable ρ′ can have the same expectation as ρ across all histories.The proof concludes by induction, using a ρ′ constructed from η′ and ξ′.Source: Inductive step: map from environments… · p. 14
C.2 Example: parental career instruction
- In the equal-prior four-environment parental-career example, the constructed η′ and ρ′ are not unique.The passage notes that an alternative η′ works equally well.Source: §C.2 · pp. 14–15
Total information
- When both parental answers are revealed, the earlier construction gives contradictory η′ assignments; Theorem 4 still works after adding twelve impossible environments.The added environments allow the original reward expectation to be represented by an uninfluenceable learning process.Source: Total information · p. 15