Reward-rational (implicit) choice: A unifying formalism for reward learning

Hong Jun Jeon; Smitha Milli; Anca D. Dragan

Source-linked AI summary

Reward-rational (implicit) choice: A unifying formalism for reward learning

Hong Jun Jeon, Smitha Milli, Anca D. Dragan

arXiv:2002.04833v4Feb 12, 2020cs.LGcs.AIcs.HCcs.RO

TL;DR

Because human feedback and behavior provide increasingly diverse evidence about intended rewards, robots need a common way to interpret it. The paper models these signals as approximately reward-rational choices from explicit or implicit options grounded in robot trajectories, and uses this view to unify prior work and develop meta-choice and active-feedback implications.

Problem
Reward functions are difficult to specify, while human demonstrations, feedback, and behavior provide diverse and partly untapped evidence about intended rewards.
Method
The paper models human feedback as an approximately reward-rational choice from an explicit or implicit option set, connected to robot trajectories through a grounding function.
Results
The formalism characterizes many diverse reward-learning information sources, supports combining and actively selecting feedback types, and motivates interpreting new feedback through the same recipe.
Takeaways & Limitations
The framework provides conceptual clarity for existing methods and a basis for multi-behavior-type reward learning, including information from the choice of feedback itself.
Takeaways & Limitations
The formalism does not apply to all feedback types, and misspecifying the human model may lead to a worse inferred reward function.

Abstract

from arXiv · show

It is often difficult to hand-specify what the correct reward function is for a task, so researchers have instead aimed to learn reward functions from human behavior or feedback. The types of behavior interpreted as evidence of the reward function have expanded greatly in recent years. We've gone from demonstrations, to comparisons, to reading into the information leaked when the human is pushing the robot away or turning it off. And surely, there is more to come. How will a robot make sense of all these diverse types of behavior? Our key insight is that different types of behavior can be interpreted in a single unifying formalism - as a reward-rational choice that the human is making, often implicitly. The formalism offers both a unifying lens with which to view past work, as well as a recipe for interpreting new sources of information that are yet to be uncovered. We provide two examples to showcase this: interpreting a new feedback type, and reading into how the choice of feedback itself leaks information about the reward.

1 Introduction

Reward learning increasingly treats human behavior and feedback as evidence about intended rewards, including information that people communicate purposefully or leak unintentionally. The paper proposes reward-rational implicit choice as a unifying formalism for interpreting these diverse sources.

Motivation: Human behavior provides reward information through demonstrations, trajectory comparisons, instructions, physical interventions, shutdowns, and world states.These behaviors may communicate preferences deliberately or leak them indirectly.
Motivation: The expanding range of feedback raises the problem of how robots can interpret both existing and yet-to-be-invented sources of reward information.The paper specifically highlights purposeful feedback and newly recognized leaks from human behavior.
Core insight: The proposed insight is to model human behavior as an approximately rational choice from an explicit or implicit set of options under the intended reward.The formalism uses the chosen option and its grounding into robot behavior to infer reward information.
Contribution: The paper surveys prior reward-learning work through this formalism and argues that many diverse information sources instantiate it.The resulting lens is intended to help compare existing methods and interpret new feedback types.

2 A formalism for reward learning

The formalism represents feedback as an approximately reward-maximizing choice from explicit or implicit options, connected to robot trajectories through a grounding function. Bayesian inference then updates beliefs over reward functions, with Boltzmann and constraint-based forms providing probabilistic and limiting interpretations.

2.1 Reward-rational implicit choice: Reward learning seeks to infer a trajectory-to-scalar reward function from human behavior.The reward is written as r : Ξ → R.
2.1 Reward-rational implicit choice: Different behaviors induce different option sets: comparisons expose alternatives explicitly, whereas demonstrations implicitly range over possible trajectories.Implicit option sets broaden applicability but are risky because the robot may specify them incorrectly.
2.1 Reward-rational implicit choice: The grounding function ψ maps human options to distributions over robot trajectories, resolving the mismatch between feedback objects and trajectory-defined rewards.Identity mappings apply to demonstrations and comparisons, while language, corrections, and proxy rewards require richer mappings.
2.1 Reward-rational implicit choice: The framework assumes finite fixed-horizon T trajectories.This assumption is stated in the paper’s trajectory definition.
2.1 Reward-rational implicit choice: A Boltzmann-rational human model assigns exponentially greater choice probability to options with higher grounded reward, with β controlling rationality.The model is presented for general groundings and simplified for deterministic mappings.
2.1 Reward-rational implicit choice: The paper derives the Boltzmann model as a maximum-entropy distribution for a satisficing agent making an ε-optimal choice in expectation.Larger ε corresponds to lower β and less optimal human behavior.
2.1 Reward-rational implicit choice: A reward-rational choice is feedback modeled as an approximately reward-maximizing selection from an explicit or implicit option set C under grounding ψ.This is the paper’s formal definition of the unifying feedback type.
2.2 Robot inference: Each feedback observation updates the robot’s belief over rewards through Bayesian inference, though exact inference is often intractable and requires approximations or MLE.As β → ∞, observed choices become constraints requiring the selected option to maximize reward.

3 Prior work from the perspective of the formalism

The paper instantiates reward-rational implicit choice across prior feedback types by specifying each type’s option set and grounding function. Comparisons, demonstrations, corrections, shutdowns, language, and proxy rewards thereby become related forms of reward evidence.

Overview: Prior work differs in how explicitly its choices and groundings are defined, requiring modifications for subtle cases such as shutdowns, initial states, reward, and punishment.The formalism organizes these cases around option sets and groundings.
Overview: The formalism’s probabilistic model and constraint simplification expose a common pattern across feedback types: exponentiated choice reward normalized over the option set.Table 1 organizes C and ψ, while Table 2 presents the probabilistic and constraint-based models.
Overview: An action-grounding variant maps choices to actions and evaluates them with optimal Q-values rather than trajectory rewards.The paper gives this as an alternative form of the formalism.
Trajectory comparisons: Trajectory comparisons use an explicit two-trajectory choice set and identity grounding; for linear rewards, the selected trajectory halves the feasible reward space.The comparison retains rewards under which the chosen trajectory scores higher than the alternative.
Gridworld illustration: In the gridworld illustration, comparisons remove rewards favoring the rug while providing no goal information because both trajectories reach the goal.Demonstrations constrain rug, goal, and dirt preferences more strongly; corrections are similar but provide no goal information.
Demonstrations: Demonstrations implicitly choose among all trajectories, use identity grounding, and generally constrain rewards more than pairwise comparisons.The paper relates this interpretation to IRL, Bayesian updates, MLE, and approximations to the partition function.
Corrections: Corrections combine an implicit correction set with non-identity grounding that propagates a configuration change through the trajectory while preserving endpoints.The propagated trajectory uses A^-1 with endpoint terms λ and γ.
Improvement: Improvement can be interpreted as a comparison in which the improved trajectory is preferred to the robot’s original trajectory.Prior implementations instead treated the improved trajectory as a demonstration with a single gradient step.

4 Discussion of implications

The formalism connects diverse human behaviors, supports combining and actively selecting feedback types, and offers a recipe for designing interpretations of new feedback. Its broader implication is conceptual clarity for multi-behavior reward learning, subject to applicability and modeling limits.

Implications: Modeling diverse behaviors as reward-rational choices lets robots compare purposeful feedback with information leaked through interventions or world states.The framework relates demonstrations, reward or punishment, initial states, corrections, and shutdowns within one lens.
Combining feedback: The formalism can combine feedback types by treating each as evidence under a joint likelihood and then applying Bayesian or approximate reward inference.The paper gives a case study combining proxy rewards, physical improvement, and comparisons.
Active selection: Active feedback selection chooses the type expected to maximize information gain, while allowing human-time costs to be incorporated.The selected type depends on the robot’s current reward belief and the choice model for that feedback type.
Active selection: In tested environments, demonstrations were optimal early when little was known, whereas comparisons became optimal later for fine-tuning.This matches manually designed procedures that train from demonstrations before comparison-based fine-tuning.
New feedback types: For new feedback, the recipe is to define the implicit option set and how its options ground to trajectories, then apply the reward-rational choice model.The paper illustrates this with selecting a length-k trajectory segment for credit assignment.
Conclusion: The authors present the formalism as conceptual clarity for existing and future methods and as a foundation for multi-behavior-type reward learning.They specifically connect it to combining and actively querying for feedback types.

Broader Impact

The paper places reward learning within applications where agents must adapt objectives to human preferences over time. It also highlights risks when changing rewards shifts control, mishandles stakeholders, or relies on a misspecified human model.

Agents in assistive robotics, autonomous driving, and recommender systems must balance multiple human-relevant objectives rather than rely on a fixed reward function.
Adapting rewards to end-user preferences can reduce system designers’ control, which may be unsuitable for safety-critical systems requiring predefined constraints.
Multiple stakeholders make ethical preference aggregation an open problem, while a misspecified human model can produce a worse reward than hand specification.

A Bounded rationality, maximum entropy, and Boltzmann-rational policies

The paper derives Boltzmann-rational policies as maximum-entropy models of bounded, satisficing humans. Satisficing error and rationality are linked one-to-one, while the framework illustrates how feedback can progressively constrain feasible rewards.

Bounded humans are modeled as satisficing: they choose options above a satisfactory threshold rather than maximizing reward.
The satisficing model sets expected reward to maxc∈C r(ψ(c)) − ϵ, where ϵ measures expected error and its maximum corresponds to always choosing the worst option.
Maximum entropy selects the choice distribution that adds no information beyond the satisficing constraint.
The unique solution is a Boltzmann distribution Pβ(f) ∝exp(β·r(ψ(c))), with β chosen to satisfy the satisficing constraint.
Satisficing error ϵ and rationality β have a one-to-one relationship, with β decreasing as ϵ increases.
Boltzmann-rationality is equivalent to bounded rationality under maximum entropy without adding other assumptions about human choice.
In the robot-arm case study, proxy feedback first shrinks feasible rewards, while later feedback and comparison remove rewards producing undesirable test-time behavior.

B A case study on combining feedback types

The case study progressively combines a proxy reward, trajectory improvement, and trajectory comparison to constrain rewards and improve generalization beyond training environments.

Combined Feedback: Starting with a proxy reward and adding improvement and comparison feedback narrows the reward space so behavior generalizes outside the training environments.The case study uses four training and test environments to illustrate this progression.
Cost Function and Features: The case study uses efficiency, table distance, and human distance features to represent the trajectory costs being learned.Efficiency is defined as the sum of squared configuration-space distances between consecutive waypoints; table and human features use modified distances.
Proxy Reward: A proxy reward matches desired behavior in two training environments but leaves many consistent rewards that produce different behaviors in test environments.The proxy constrains the feasible set from Θ to Fproxy, but does not guarantee optimal trajectories in environments 3 and 4.
Improvement: An improvement request removes rewards favoring inefficient trajectories that stay too close to the table, but does not yet guarantee optimal behavior everywhere.The feasible set shrinks from Fproxy to Fimprovement, though it remains too large for guaranteed generalization.
Trajectory Comparison: A trajectory comparison selects the more efficient orange trajectory over the alternative that prioritizes proximity to the table, further shrinking the feasible reward set.The comparison changes the feasible set from Fimprovement to Fcomparison.
Combined Feedback: After combining feedback types, every reward in Fcomparison produces the same optimal feature vector as the true reward across all four environments.The resulting feasible set is very small, and Figure 2 depicts only the optimal green trajectory remaining in each environment.

C Actively selecting which type of feedback to use

The paper applies reward-rational choice to actively select feedback types by expected information gain. In experiments, demonstrations help most early, while comparisons become more useful later, and combining them performs best overall.

Active Selection: The formalism selects the feedback type that greedily maximizes expected information gain under the robot’s current belief over rewards.The choice model uses the user’s possible choice within each feedback type and its probability under sampled rewards.
Results: Demonstrations are optimal early when little is known about the reward, whereas comparisons become optimal later for fine-tuning it.Figure 4 describes demonstrations as initially information-dense but quickly flattening, while comparisons obtain more information less efficiently.
Experiment Setup: The experiment compares active demonstrations, active comparisons, and their combination across eight 25x25 gridworld environments with four training and four test environments.Each environment uses a linear reward over three RGB features and ten start-goal pairs for querying.
Evaluation: The active-learning evaluation tracks feasible-set volume, diameter, maximum regret, and average regret over ten query iterations and 50 ground-truth rewards.Each metric serves as a proxy for the accuracy of the learned reward.
Results: Combining demonstrations and comparisons achieves lower volume, diameter, maximum regret, and average regret than demonstrations alone, in fewer iterations than comparisons alone.The comparison is reported across the active-learning metrics used in the experiment.

D Meta-choice: a new source of information

Meta-choice treats the human’s selection among available feedback types as an additional reward-rational choice, making feedback selection itself informative about the intended reward.

The meta-choice is the human’s reward-rational selection among available feedback types, such as corrections and turning the robot off.It extends the framework beyond interpreting only the content of a chosen feedback type.
Figure 5 shows that metareasoning can rule out more reward functions than a naive model that ignores unchosen feedback options.

D.1 Formalizing meta-choice

The formalization models feedback selection hierarchically: the human first chooses a feedback type, then chooses specific feedback within that type, with separate rationality parameters for each stage.

Treating feedback instances as independent choices is reasonable when the robot controls which feedback type the human considers, but breaks when multiple types are simultaneously available.
The framework models meta-choice as a first-stage choice among feedback types followed by a second-stage choice of the specific feedback provided.
The first-stage grounding maps a feedback type to the trajectory distribution produced by the human’s behavior and the second-stage grounding.
The human’s adaptation applies to purposefully communicated behavior types, not information sources such as the initial state.
Separate rationality parameters represent noise in selecting the feedback type and noise in executing the chosen feedback.

D.2 Comparing the literal interpretation to meta-choice

An experiment compares a naive learner with a metareasoning learner in a gridworld where the human can turn the robot off or provide corrections, finding that unchosen options provide additional reward information.

The gridworld varies whether the learner models meta-choice, the human’s meta-rationality β0, and the lava location that changes the rational feedback choice.Performance is measured by regret in holdout environments.
Meta-reasoning gains more information in both environments because the chosen feedback is interpreted relative to the feedback option that was available but not selected.Turning the robot off supports the inference that lava is bad, while choosing correction over off informs the importance of reaching the goal.

D.3 What happens when metarationality is misspecified?

Misspecifying the human’s metarationality can distort belief updates and regret, especially when the learner assumes metareasoning that the human does not exhibit; detecting low metarationality can favor dropping that assumption.

The experiments evaluate misspecification by comparing beliefs and expected regret across assumed metarationalities β0 when the human’s true value is β∗0.They average results over 50 randomly sampled reward functions and β0 values from 0.0 to 10.0.
The framework can be extended to model humans trading off reward against their own effort, especially when some feedback types are more difficult to provide.
When the human does not metareason, assuming metareasoning produces significant belief divergence and high expected regret.
Matching the learner’s β0 to the human’s β∗0 does not necessarily minimize expected regret because the optimal learner setting depends on β∗0.
If the human is poor at metareasoning, the experiments suggest it is safer to drop the metareasoning assumption; if metareasoning is present, it can improve learning.