Learning the Preferences of Ignorant, Inconsistent Agents

• Preference learning features in economics, cognitive science, and applied machine learning, including using past behavior to personalize social-network displays.Source: Introduction · p. 1
• Standard approaches assume near-optimal decisions with random action noise, but systematic deviations can lead to mistaken preference inferences.Source: Introduction · p. 1
• The paper models inaccurate beliefs and hyperbolic time inconsistency, then jointly infers preferences, beliefs, and biases from action sequences.Source: Introduction · p. 1
• Human judges and the model both explain choice variation through systematic factors such as false beliefs and strong temptations, rather than unexplainable error.Source: Introduction · p. 1
• When few actions leave preferences unidentified, humans offer multiple explanations, and the model captures their range and predicts their plausibility.Source: Introduction · pp. 1–2

Deviations from optimality

• The model considers inaccurate beliefs and hyperbolic discounting, with Naive and Sophisticated agents differing in how accurately they predict their future choices.Sophisticated agents model future choices accurately; Naive agents assume their future selves will assign the same discounted values as their present selves.Source: False beliefs and uncertainty · p. 2, Temporal inconsistency · p. 2

Formal model definition

• The model represents agents by beliefs, utility, type, discount strength, and choice noise, and infers their joint posterior from observed action sequences.The prior uses independent uniform distributions over bounded intervals for the agent components.Source: Inferring preferences · p. 3

Agents as probabilistic programs

• Probabilistic programs encode recursive decision-making and can be extended to other forms of sub-optimal planning; inference uses a discrete grid and exact enumeration with dynamic programming.The implementation uses WebPPL and automated inference over recursive functional programs.Source: Figure 2 · p. 3, Agents as probabilistic programs · p. 4

Model inferences

• A Naive episode is explained either by very high softmax noise or by Naive planning, which requires the Cafe to be more valuable than the Donut Store, but not too much more.If the Cafe’s utility were too high, the agent would not give in to the Donut Store’s temptation.Source: Example 1: Inference with full knowledge · p. 4, Figure 3 · p. 4
• When the agent may believe a closed Noodle Shop is open, observations support high shop utility if that belief is strong; otherwise, the shop’s utility does not affect their plausibility.The example adds uncertainty about the agent’s beliefs to the possible explanations of the Sophisticated episode.Source: Example 2: Inference with uncertainty · p. 4
• Inferences about discounting depend on Cafe utility: the data fit poorly when it is low, and fit best when that utility balances against discount strength.If Cafe utility is very high relative to discount strength, the agent could go there directly without risking the Donut Store’s temptation.Source: Example 2: Inference with uncertainty · p. 4, Figure 3 · p. 4
• 0.59 posterior probability favors the Vegetarian Cafe even though the agent chooses the Donut Store twice in three episodes; without time inconsistency, the model favors the Donut Store.The model without time inconsistency explains Episode 2 as noise, producing high-entropy predictions of future choices.Source: Figure 4 · p. 5, Example 3: Inference from multiple… · p. 5

Conclusion

• The experiments probe whether people infer preferences, biases, and false beliefs from actions, and the conclusion argues that realistic decision models support more accurate preference inference while preserving uncertainty.The authors identify realistic-domain applications and alternatives to human preference inferences as evaluation standards as priorities for future work.Source: Experiments with Human Subjects · p. 5, Conclusion · p. 6

Experiment 1: Inference with full knowledge

• In Experiment 1, subjects rated temptation explanations highest: giving in for the Naive agent and avoiding temptation for the Sophisticated agent.Subjects knew the agent had accurate knowledge and had seen prior episodes suggesting a Vegetarian Cafe preference.Source: Experiment 1: Inference with full… · p. 5, Figure 5 · p. 5

Experiment 2: Inference with uncertainty

• In Experiment 2, the model and subjects agreed on the three highest-scoring properties in both conditions, with false-belief and preference explanations ranking above time inconsistency.The model formalizes explanations including not knowing Donut Store D1 is open, falsely believing the Noodle Shop is open, and preferring D2 to D1.Source: Experiment 2: Inference with uncertainty · pp. 5–6, Figure 7 · p. 6
• Experiment 2’s explanations were mapped from human-invoked properties to model variables, and some explanations require a narrow range of utility values.The authors say these narrower ranges favor more specific explanations.Source: Experiment 2: Inference with uncertainty · p. 6, Figure 6

Experiment 3: Inference from multiple episodes

• In Experiment 3, 54% of subjects inferred a Vegetarian Cafe preference, compared with the model’s 59% posterior probability, despite more frequent Donut Store choices.The authors suggest subjects use their inference of sophisticated time-inconsistent planning to judge undiscounted preferences.Source: Experiment 3: Inference from multiple… · p. 6