PaperlayerView exact revision on arXiv

Source-linked AI summary

Human irrationality: both bad and good for reward inference

Lawrence Chan, Andrew Critch, Anca Dragan

arXiv:2111.06956v1cs.LG

TL;DR

Reward inference from human behavior often assumes rationality, despite the many forms of human irrationality and the open question of how they affect inference. This paper formalizes irrationality through changes to Bellman updates and studies their effects on reward learning. It finds that correctly modeled irrational behavior can improve inference over rational behavior, while treating irrational humans as noisy-rational can perform worse than using the prior.

  • Problem

    The paper asks how different forms of human irrationality affect reward inference and how inference changes when an irrational human is modeled as irrational rather than rational.

    Source: §I · p. 1

  • Method

    The authors operationalize irrationality as systematic deviations from the Bellman equation and use simulations to study reward inference under different human and robot models.

    Source: §I · p. 1, C. Contributions · p. 2

  • Results

    Across three domains, some correctly modeled irrational behaviors improve reward learning over rational behavior, while modeling irrationality as noisy-rational can perform worse than sticking with the prior.

    Source: 1) Summary · p. 7

  • Takeaways & Limitations

    The findings support accounting for human irrationality because reward-dependent behavioral variation can provide additional information for inference.

    Source: C. Contributions · p. 2

    The paper presents an analysis rather than a practical algorithm, and benefiting from irrationality still requires precise models of human irrationality.

    Source: 2) Limitations and Future Work · p. 8

Abstract

from arXiv · show

Assuming humans are (approximately) rational enables robots to infer reward functions by observing human behavior. But people exhibit a wide array of irrationalities, and our goal with this work is to better understand the effect they can have on reward inference. The challenge with studying this effect is that there are many types of irrationality, with varying degrees of mathematical formalization. We thus operationalize irrationality in the language of MDPs, by altering the Bellman optimality equation, and use this framework to study how these alterations would affect inference. We find that wrongly modeling a systematically irrational human as noisy-rational performs a lot worse than correctly capturing these biases -- so much so that it can be better to skip inference altogether and stick to the prior! More importantly, we show that an irrational human, when correctly modelled, can communicate more information about the reward than a perfectly rational human can. That is, if a robot has the correct model of a human's irrationality, it can make an even stronger inference than it ever could if the human were rational. Irrationality fundamentally helps rather than hinder reward inference, but it needs to be correctly accounted for.

I. INTRODUCTION

The paper asks how human irrationality and a robot’s model of it affect reward inference. It argues that correctly modeled irrational behavior can be more informative than rational behavior, while modeling biases as noisy rationality can perform poorly.

I. INTRODUCTION

  • The motivating gap is how much reward-learning performance changes when an irrational human is modeled as irrational rather than rational.The paper also asks whether correctly modeled irrationality limits the best inference performance achievable.Source: §I · p. 1

A. Irrationality can make human behavior more informative.

  • The figure contrasts rational behavior that is unchanged by speed preference with myopic behavior that varies with the human’s reward.This reward-dependent behavioral diversity can make the reward easier to distinguish.Source: Figure 1 · p. 1

C. Contributions

  • The paper formalizes irrationality by systematically altering Bellman-equation components, then studies reward inference across random MDPs, gridworld, and autonomous driving.The driving domain is drawn from prior literature.Source: §I · pp. 1–2

A. Irrationality can make human behavior more informative.

  • Some correctly modeled irrationalities improve reward inference over a correctly modeled rational human across all three environments.The paper explains this through increased mutual information between behavior and reward parameters.Source: A. Irrationality can make human… · p. 2

B. Unmodelled irrationality leads to remarkably poor reward inference.

  • Modeling an irrational human as rational or noisily rational can produce worse inference than skipping inference and using the prior.Modeling the type of irrationality correctly can outperform the standard noisy-rational assumption even without matching its exact parameters.Source: B. Unmodelled irrationality leads to… · p. 2

II. FRAMEWORK: BIASES AS DEVIATIONS FROM THE BELLMAN UPDATE

The framework studies simulated human behavior and reward inference by representing irrationality as deviations from rational planning in an MDP. It systematically modifies Bellman-update components and parameterizes the strength of each deviation.

A. Exploring biases through simulation

  • The authors simulate humans with different irrationalities, run reward inference, and measure the posterior against the known reward parameter.This simulation strategy addresses the difficulty of recruiting people with isolated biases and knowing their true rewards.Source: A. Exploring biases through simulation · p. 2

B. Background and formalism

  • The URMDP specifies states, actions, transition distributions, discounting, reward parameters, a prior, and a parameterized reward function.A human planner maps a reward parameter to a policy; irrationality means lower expected value than the optimal policy for at least one parameter.Source: B. Background and formalism · pp. 2–3

C. Types and degrees of irrationality

  • The framework creates complementary irrationality types by manipulating Bellman-update terms and operators, with parameters controlling their intensity.For example, discounting changes model myopia, while transition-function changes model optimism or illusion of control.Source: C. Types and degrees of irrationality · p. 3, Figure 2 · p. 3

1) Rational human:

  • The rational human uses value iteration with ground-truth models and a high discount factor.The irrationality models alter this Bellman update.Source: 1) Rational human · p. 3

2) Modifying the max operator: Boltzmann:

  • Boltzmann-rationality replaces maximization over actions with a Boltzmann operator, then assigns action probabilities proportional to exponentiated Q-values.As β approaches infinity, choices approach optimality; at β = 0, choices are uniform.Source: 2) Modifying the max operator: Boltzmann · p. 3

3) Modifying the transition function:

The framework models irrationalities by changing either transition beliefs or perceived rewards. These changes represent illusion of control, optimism or pessimism, and prospect-style sensitivity to gains and losses.

3) Modifying the transition function:

  • Transition-function modifications model illusion of control and optimism or pessimism by changing how humans represent possible outcomes.For illusion of control, the parameter n moves beliefs from uniform successor-state chances toward deterministic transitions; ω controls optimism or pessimism about outcome value.Source: 3) Modifying the transition function · p. 3

4) Modifying the reward:

  • Reward modifications replace r with f ◦ r, allowing models of loss aversion and scope insensitivity, including Prospect Bias.Its parameter c controls loss aversion: larger c emphasizes avoiding negative rewards, while smaller c emphasizes positive rewards.Source: 4) Modifying the reward · p. 3

5) Modifying the sum between reward and future value::

The framework also changes how rewards across time are aggregated and how future rewards are discounted. These modifications model extremal reward focus and several forms of myopia or non-exponential discounting.

5) Modifying the sum between reward and future value::

  • Extremal humans maximize expected maximum reward along a trajectory rather than expected cumulative reward.As α approaches 1, they emphasize the maximum over the full trajectory; as α approaches 0, they maximize immediate reward.Source: 5) Modifying the sum between reward… · p. 4

6) Modifying the discounting:

  • The framework models myopia by lowering γ or limiting value iteration to h steps, and models hyperbolic discounting with a separate parameter k.Lower γ and h = 1 focus on immediate reward, while h approaching infinity recovers rationality; k controls preference for now versus the future.Source: 6) Modifying the discounting · p. 4

III. EXPLORING THE EFFECTS OF KNOWN BIASES ON REWARD INFERENCE

In random MDPs, several correctly modeled irrationalities support better reward inference than rational behavior, and the advantage grows with more data. At T=30, every tested type except Prospect Bias improves inference over the fully rational planner.

A. Experimental Design: Exact Inference in Random MDPs.

  • The experiment varies planner type and degree, environment, ground-truth rewards, and demonstration lengths of T = 3, 15, or 30 state-actions.It uses exact Bayesian inference and evaluates the posterior with expected log loss; lower loss means higher likelihood assigned to the true reward.Source: A. Experimental Design: Exact… · p. 4

B. Results

  • With enough data, most irrationality types have parameter settings that outperform rational behavior, and the difference grows at T=30.The study reports that the degree of irrationality affects inference, with extreme settings often producing worse results.Source: B. Results · p. 4
  • At T=30, every irrationality except Prospect Bias improved inference over the fully rational planner; Optimism with ω = 3.16 performed best.Illusion of Control with n = 0.1 and Boltzmann with β = 10 followed Optimism.Source: B. Results · p. 4

IV. DOES THIS EFFECT GENERALIZE?

The inference advantage of some irrational behavior appears in both a structured gridworld and an autonomous-driving domain. In driving, shorter planning horizons produce more reward-dependent behavior and improve inference measures.

IV. DOES THIS EFFECT GENERALIZE?

  • The section tests whether the random-MDP finding generalizes, first in a structured gridworld and then in continuous-state, continuous-action driving.The driving evaluation uses approximate Bayesian IRL and CIOC, with reward similarity and normalized trajectory return as measures.Source: §IV · p. 4, A. Gridworld · p. 4, B. Autonomous Driving · p. 6

A. Gridworld

  • Gridworld results resemble those from random MDPs, suggesting that some irrationalities help inference across differently structured MDPs.The policies show that irrational behavior can distinguish reward parameters for which the rational human’s policies are identical.Source: A. Gridworld · pp. 4, 6

B. Autonomous Driving

  • In the merging task, shorter MPC horizons reduce Bayesian IRL posterior log loss and increase CIOC reward similarity and recovered-reward return.Horizons 3 and 5 yield more diverse behaviors across rewards than horizon 7.Source: B. Autonomous Driving · p. 6

C. Theoretical Analysis

The theoretical analysis uses mutual information to quantify how much a planner’s behavior reveals about reward parameters. It shows that some irrational planners can be arbitrarily more informative than rational behavior, while assuming the wrong model can substantially harm inference.

1) Informativeness as mutual information.:

  • Mutual information between policy and reward parameters measures planner informativeness and upper-bounds the improvement in log loss over the prior.Reward-maximizing planners need not maximize informativeness.Source: 1) Informativeness as mutual information · p. 6

2) Irrationalities exist that are arbitrarily better for inference than rationality.:

  • For some URMDPs, a deterministic irrational planner achieves I(θ, dIrrational(θ)) = log |Θ| while the rational planner achieves 0.The proposition applies to a family of URMDPs with state spaces of any size.Source: 2) Irrationalities exist that are… · p. 6

3) Boltzmann-rationality is (arbitrarily) more informative than full rationality.:

  • For some one-state, two-action MDPs with arbitrarily large |Θ|, Boltzmann-rationality achieves log |Θ| mutual information while rationality achieves 0.The paper presents this as evidence that a standard stochastic-choice model can outperform full rationality for reward inference.Source: 3) Boltzmann-rationality is… · p. 6

A. Assuming noisy rationality can lead to very poor inference

  • Using a default Boltzmann model when the human is irrational can greatly harm inference, including in the autonomous-driving environment.For a myopic human, a trajectory may reflect only the bottom reward, while a Boltzmann model interprets it as evidence about the relative rewards.Source: A. Assuming noisy rationality can lead… · p. 7, Figure 6 · p. 7

B. Approximate models of irrationality might be enough

Correctly modeling human irrationality can improve reward inference, while treating an irrational human as noisy-rational can perform so poorly that keeping the prior may be preferable. The paper also identifies limits to applying these findings and opportunities for future work.

B. Approximate models of irrationality might be enough

1) Summary:

  • When modeled correctly, irrational behavior can be more informative about reward than rational behavior, improving learning across three domains.The paper attributes this to irrational behavior varying more as a function of reward and increasing mutual information between reward and policy.Source: 1) Summary · p. 7

2) Limitations and Future Work:

  • Most results come from random MDPs and gridworlds; the authors call for replication in more realistic domains and algorithms that handle varied biases in larger domains.They also report confirming key results in a higher-dimensional, continuous autonomous-driving task.Source: 2) Limitations and Future Work · pp. 7–8
  • The work is an analysis, not a practical algorithm for improving reward learning.The authors say benefiting from irrationality still requires precise models of the human’s irrationality.Source: 2) Limitations and Future Work · p. 8
  • The authors propose training people to demonstrate more informatively, for example by acting myopically, and say real-user studies are needed to assess this opportunity.Source: 2) Limitations and Future Work · p. 8

Appendices

The appendices describe how policies are computed and show how irrational behavior can distinguish rewards that rational policies leave ambiguous. A gridworld example also illustrates why assuming the wrong human model can impair inference.

VII. ADDITIONAL DETAILS FOR RANDOM MDPS

  • Value iteration computes policies for planner-and-reward settings until successive value functions differ by less than 0.001 in the infinity norm.Source: §VII · p. 10

VIII. DETAILS FOR GRIDWORLD

  • The Frozen-Lake-derived gridworld is 5×5, with stochastic directional moves, terminal holes and rewards, a −10 hole penalty, and reward parameters in [0, 4].A move follows the chosen direction with probability 0.8 and one of two adjacent directions with probability 0.2.Source: §VIII · p. 10
  • Gridworld examples compare irrational policies that differ even when the rational human’s policies are identical, helping visualize improved inference.Source: §VIII · p. 10
  • In the example, a myopic agent with n ≤ 4 attends only to the bottom reward, whereas a Boltzmann model would take the trajectory only if that reward were not much below the top reward.Assuming Boltzmann-rationality for the myopic trajectory therefore leads to poor inference in this case.Source: §VIII · p. 10

IX. DETAILS FOR AUTONOMOUS DRIVING ENVIRONMENT

To test whether the findings matter in practice, the paper studies irrationality in a self-driving-car domain. It uses a simulated merging task and evaluates reward inference under model-predictive control, Bayesian IRL, and a maximum-likelihood method.

IX. DETAILS FOR AUTONOMOUS DRIVING ENVIRONMENT

  • The driving study uses a simulated merging scenario with point-mass car dynamics and a four-dimensional vehicle state controlled by steering and acceleration.The dynamics are discretized in time with dt = 0.1s.Source: §IX · p. 11
  • The car’s reward is modeled as a linear combination of features, and irrationality is represented by using shorter planning horizons in MPC.The planner executes the first action of a finite-horizon plan, then replans; inference is assumed to know the horizon exactly.Source: §IX · p. 11
  • Bayesian IRL uses a specified “fake” observation model because the demonstrations come from trajectory optimization rather than an inherently stochastic process.The paper considers a normal distribution in feature space for P(ξ|θ).Source: §IX · p. 11
  • For practical maximum-likelihood inference, the study uses CIOC with a Boltzmann model, a large β = 10^4, and optimization initialized at the true reward weights.Source: §IX · p. 11
  • CIOC estimates are evaluated by cosine similarity to the true reward weights and normalized trajectory return, while Bayesian IRL varies target-speed weights and planning horizons.The Bayesian conditions use θ_speed ∈ {0.5, 1, 2, 4} and horizons h = 3, 5, and 7; CIOC uses θ_speed = 1.Source: §IX · p. 11

X. PROOFS FOR SEC. IV-C

For deterministic planners, informativeness is bounded by both the reward space and the number of possible policies. Constructions show that rational behavior can reveal no reward information while alternative planners reveal all of it.

X. PROOFS FOR SEC. IV-C

  • For deterministic planners, informativeness is bounded by min(log |Θ|, |S| log |A|), from the entropy limits on rewards and possible policies.There are at most |A|^|S| deterministic policies.Source: §X · p. 11
  • A family of URMDPs can make rational planning reveal no information while a deterministic planner conveys min(log |Θ|, |S| log |A|) bits.The construction maps reward parameters injectively to policies, making the conditional entropy of θ given the policy zero.Source: §X · pp. 11–12
  • In one-state, two-action MDPs with arbitrarily large |Θ|, Boltzmann planning conveys log |Θ| bits while rational planning conveys none.The Boltzmann mapping is injective, so H(θ|dBoltz(θ)) = 0.Source: §X · p. 12

XI. MORE THEORY

The theory connects reward inference to transfer across environments and robot dynamics, where imitation alone may not suffice. It constructs a setting in which Boltzmann behavior reveals reward information that rational behavior does not.

XI. MORE THEORY

  • Inferring reward parameters matters when behavior must transfer to new environments or to a robot with different dynamics, since imitation alone may not suffice.The paper notes cases where robustness to dynamics changes requires identifying reward parameters.Source: §XI · p. 12
  • A two-state, two-action URMDP can require log |Θ| bits to compute optimal policies under new transitions, while rational behavior reveals zero bits and Boltzmann behavior reveals log |Θ| bits.The construction sets transition probabilities so the optimal action changes at thresholds in θ.Source: §XI · p. 12
  • In the constructed setting, rational planning selects the same optimal policy across reward parameters, whereas Boltzmann planning maps them injectively to policies.This makes the rational policy uninformative and the Boltzmann policy fully informative about θ.Source: §XI · p. 12
Loading 2111.06956v1…