Source-linked AI summary
Where Do You Think You're Going?: Inferring Beliefs about Dynamics from Behavior
Siddharth Reddy, Anca D. Dragan, Sergey Levine
TL;DR
Existing intent-inference methods commonly assume near-optimal demonstrations or explain deviations through noise or known biases, rather than users’ mistaken beliefs about dynamics. This paper estimates those internal beliefs from actions on known tasks and uses them to infer intent. Simulations and a 12-participant study show that the recovered model explains human actions better than real dynamics and supports assistance.
Problem
Inverse planning and IRL infer goals or rewards from demonstrations but generally assume near-optimal behavior or model deviations as noise or known biases, leaving internal dynamics mismatch as an alternative explanation for suboptimal actions.
Source: §1 · pp. 1–2Method
Results
In simulations and a study with 12 participants, the recovered dynamics explained user actions better than real dynamics and enabled assistance by transferring users’ control policy to real dynamics.
Source: §1 · p. 2Takeaways & Limitations
The approach provides a way to model intent when users’ actions are suboptimal in reality but near-optimal under their internal dynamics, with demonstrated utility for shared autonomy and reward inference.
Source: §1 · p. 2, §7 · p. 9Multiple internal dynamics models can explain the same demonstrations, making the correct model difficult to identify; the method uses multiple training tasks and a prior to address this ambiguity.
Source: §3.2 · p. 4
Abstract
from arXiv · showhide
Inferring intent from observed behavior has been studied extensively within the frameworks of Bayesian inverse planning and inverse reinforcement learning. These methods infer a goal or reward function that best explains the actions of the observed agent, typically a human demonstrator. Another agent can use this inferred intent to predict, imitate, or assist the human user. However, a central assumption in inverse reinforcement learning is that the demonstrator is close to optimal. While models of suboptimal behavior exist, they typically assume that suboptimal actions are the result of some type of random noise or a known cognitive bias, like temporal inconsistency. In this paper, we take an alternative approach, and model suboptimal behavior as the result of internal model misspecification: the reason that user actions might deviate from near-optimal actions is that the user has an incorrect set of beliefs about the rules -- the dynamics -- governing how actions affect the environment. Our insight is that while demonstrated actions may be suboptimal in the real world, they may actually be near-optimal with respect to the user's internal model of the dynamics. By estimating these internal beliefs from observed behavior, we arrive at a new method for inferring intent. We demonstrate in simulation and in a user study with 12 participants that this approach enables us to more accurately model human intent, and can be used in a variety of applications, including offering assistance in a shared autonomy framework and inferring human preferences.
1 Introduction
Intent inference commonly learns goals or rewards from demonstrations, but these approaches generally assume near-optimal behavior or model deviations as noise or known biases. This paper instead explains suboptimal actions through users’ mistaken beliefs about dynamics and estimates those beliefs to infer intent.
1 Introduction
- Intent inference supports predicting future behavior, imitation, and automated assistance; prior methods typically assume near-optimal actions or attribute deviations to noise, cognitive biases, or risk sensitivity.The introduction gives robotic teleoperation as an example where unfamiliar system and interface dynamics can make control challenging.Source: §1 · p. 1
- Suboptimal real-world actions may be near-optimal under a user’s internal dynamics model, so modeling that mismatch offers another route to explaining behavior and inferring intent.The paper situates internal models as tools people use for motor control and goal-directed decisions, and notes that simplified models can produce unintended outcomes.Source: §1 · p. 2
- The algorithm learns internal dynamics from demonstrations on known-reward tasks, then uses the learned model to infer intent on unknown tasks.It assumes actions are chosen with probability proportional to exponentiated soft Q-values and ties the internal dynamics to the soft Q function through the soft Bellman equation.Source: §1 · p. 2
- In a study with 12 participants playing Lunar Lander, the recovered dynamics explained actions better than real dynamics and enabled assistance by transferring users’ control policy to the real dynamics.The paper also reports simulation experiments that test recovery of internal dynamics, including in continuous-state MDPs that prior methods would otherwise find intractable.Source: §1 · p. 2
2 Background
Inverse reinforcement learning (IRL) aims to learn a user’s reward function from control demonstrations, and offers a starting point for inferring hidden properties from behavior. This paper uses the framework while shifting the learned hidden property from reward to internal dynamics.
2 Background
- IRL learns a user’s reward function from control demonstrations, providing a framework for extracting hidden properties from observed behavior.The passage distinguishes this goal from learning users’ beliefs about environmental dynamics.Source: §2 · p. 2
- The paper builds on maximum causal entropy IRL, where a demonstrator follows a policy maximizing entropy-regularized reward under specified dynamics.The background describes this setup for an MDP with a discrete action space.Source: §2 · p. 2
- Unlike prior IRL work that assumes real-world dynamics and fits rewards to demonstrations, this work assumes known-reward training tasks and fits internal dynamics that may differ from reality.The recovered dynamics are then used to infer intent, such as rewards, in new tasks.Source: §2 · p. 2
3 Internal Dynamics Model Estimation
The method infers a user’s implicit dynamics beliefs from actions on known tasks, treating those actions as near-optimal under the user’s unknown internal model. It fits that model through soft Q-learning, then regularizes it to address ambiguity among models that explain demonstrations equally well.
3 Internal Dynamics Model Estimation
- The model assumes stationary user beliefs, real dynamics known in advance or learned separately, and near-optimal actions with respect to the unknown internal model.Demonstrated state-action pairs indirectly encode the transitions users expected, which may differ from the transitions that occurred.Source: §3 · p. 3
3.1 Inverse Soft Q-Learning
- Inverse soft Q-learning fits internal dynamics and task-specific soft Q functions to explain demonstrations while tying them through the soft Bellman equation.Training tasks have known rewards and share the same internal dynamics; the constrained objective maximizes demonstration likelihood subject to zero soft Bellman error.Source: §3.1 · p. 3
- Learning the correct internal dynamics from demonstrations with unknown rewards is possible in principle, but the authors report that this relaxation usually makes learning too difficult.The exposition assumes discrete actions for simplicity and notes that continuous-action extensions can use existing sampling methods [25].Source: §3.1 · p. 3
- The optimization uses a penalty method, and continuous-state tasks use constraint sampling plus deterministic internal dynamics to handle intractable state integrals.The penalty cost combines action likelihood with soft Bellman error; the paper also uses tabular, structured linear, and shallow neural-network representations.Source: §3.1 · p. 4
3.2 Regularizing the Internal Dynamics Model
- Multiple internal dynamics models can explain demonstrations equally well, making the correct model difficult to identify; the paper proposes multiple training tasks and a prior favoring similarity to real dynamics.With only one task, a degenerate model can predict the reward-maximizing next state regardless of state or action; diverse task policies can rule out such solutions.Source: §3.2 · p. 4
- The action-intent prior models the action a user would take with perfect knowledge of real dynamics, then uses it to compute internal dynamics over reachable real-world states.The experiments examine multiple training tasks and the action-intent prior both together and separately; Figure 1 illustrates internal-to-real dynamics transfer for shared autonomy.Source: §3.2 · p. 4, Figure 1 · p. 5
4 Using Learned Internal Dynamics Models
The paper uses learned internal dynamics for two intent-inference applications: transferring a user’s policy to real-world dynamics in shared autonomy and learning rewards from misguided demonstrations. In both, behavior that is suboptimal under real dynamics can be understood using the user’s internal model.
4 Using Learned Internal Dynamics Models
- The framework applies learned internal dynamics to shared autonomy, where intent is the user’s desired next state, and reward learning, where intent is the user’s reward function.These applications use the learned model to infer either a desired next state or a reward function.Source: §4 · p. 5
4.1 Shared Autonomy via Internal-to-Real Dynamics Transfer
- Shared autonomy transfers the user’s policy by inferring a desired next state under internal dynamics and selecting a real-dynamics action that reaches it.The method is designed to assist without knowing the user’s reward function.Source: §4.1 · p. 5
4.2 Learning Rewards from Misguided User Demonstrations
- Standard IRL learns incorrect rewards when demonstrations are misguided because it treats the user’s internal dynamics as the real dynamics.Misguided behavior is suboptimal in the real world but near-optimal under the user’s internal dynamics.Source: §4.2 · p. 5
- In the grid-world experiment, incorporating learned internal dynamics into MaxCausalEnt IRL enables accurate reward learning from misguided demonstrations.The figure caption reports that the method learns accurate internal dynamics and enables reward learning from these demonstrations.Source: Figure 2 · p. 6, §4.2 · p. 6
5 User Study and Simulation Experiments
The experiments test whether the method recovers internal dynamics in simulation and can use them to assist real users. In a 12-participant Lunar Lander study, transfer assistance improved game outcomes, while simulations supported dynamics recovery and reward learning from misguided demonstrations.
5.2 User Study on the Lunar Lander Game
- The study tested internal-to-real transfer with 12 participants playing Lunar Lander, measuring success and crash rates with and without assistance.The task uses six discrete actions and nine distinct landing-site tasks.Source: §5 · p. 6, §5.2 · p. 8
5 User Study and Simulation Experiments
- The simulation experiments use a tabular grid world to check correctness and regularization, a continuous-state navigation task to test approximations, and grid-world reward learning.The user study tests whether the internal model can support assistance for real users.Source: §5 · p. 6
5.1 Simulation Experiments
- Simulations recovered internal dynamics accurately in tabular and continuous-state MDPs, and learned solutions did not significantly deviate from an exact algorithm.Accuracy increased with more training tasks, and the action intent prior helped when internal dynamics resembled real dynamics.Source: §5.1 · pp. 6–7
5.2 User Study on the Lunar Lander Game
- Unassisted users often crashed or flew out of bounds in the default environment, which they found difficult and unintuitive.The paper attributes this to unintuitive thruster controls and the game’s relatively fast pace.Source: Figure 3 · p. 7, §5.2 · p. 8
5.1 Simulation Experiments
- MaxCausalEnt IRL learned accurate rewards from misguided demonstrations when using learned internal dynamics, unlike its standard real-dynamics assumption.A SERD variant performed better than random but worse than the proposed method.Source: §5.1 · p. 7
5.2 User Study on the the Lunar Lander Game
- The subjective evaluations summarize responses from 12 participants on a 7-point Likert scale and use repeated-measures ANOVA to assess assistance effects.The supplied caption describes the measures and analysis but does not provide rating values.Source: Table 1 · p. 8
5.2 User Study on the Lunar Lander Game
- Users’ actions were better explained by a learned slower game-speed model, consistent with smaller forces and slower motion making their demonstrations closer to optimal.The authors note that the true internal dynamics of human users are unknown, so accuracy is supported through the assistance results rather than direct ground-truth comparison.Source: §5.2 · p. 8
- Assisted Lunar Lander users succeeded significantly more often and crashed significantly less often; assistance also reduced average speed by 30%.Users averaged 84 actions per minute with assistance versus 18 without it.Source: Figure 3 · p. 7, §5.2 · p. 8
6 Related Work
Prior work infers goals or rewards from actions, while this paper targets beliefs about world dynamics. Its approach differs from existing methods by allowing internal dynamics to differ from real dynamics and by handling continuous state spaces and dynamics parameters.
6 Related Work
- Inverse planning and IRL infer goals or rewards from observed actions, whereas this paper infers beliefs about world dynamics.The paper presents this as a different approach to intent inference.Source: §6 · p. 8
- Prior SERD work [28] includes demonstrator beliefs but constrains them to match real dynamics, while this paper learns potentially different internal dynamics.The proposed method adds multi-task training and an action intent prior as regularization techniques.Source: §6 · pp. 8–9
- Unlike SERD [28], which is intractable for continuous-state spaces, this method applies to continuous-state MDPs.The paper demonstrates its continuous-state applicability in experiments.Source: §6 · p. 9
- Golub et al.’s IME framework assumes linear-Gaussian dynamics, quadratic rewards, and sensory feedback delay, while this work imposes no functional-form restrictions and does not assume delay.The comparison is with an internal model estimation framework for brain-machine interface control.Source: §6 · p. 9
- Educational-game methods identify beliefs from discrete candidate models, whereas this algorithm learns continuous dynamics parameters.The cited work uses inferred beliefs to generate personalized hints and feedback.Source: §6 · p. 9
- The paper’s account of suboptimal behavior emphasizes incorrect dynamics beliefs rather than uncertainty or cognitive biases in planning and judgment.It also notes that resource-bounded users may plan with simplified models even when they know the real dynamics.Source: §6 · p. 9
7 Discussion
The paper reports that its method recovers dynamics models that explain human actions and supports applications in shared autonomy and inverse reinforcement learning. The current continuous-task dynamics parameterizations use at most seven parameters, while future work includes studying intuitive theories and intent inference applications.
7 Discussion
- Continuous-task dynamics parameterizations use at most seven parameters; increasing their number requires better regularization than the paper proposes.The soft Q function can use arbitrary neural-network parameterizations, unlike the smaller dynamics models.Source: §7 · p. 9
- The method recovers dynamics models that explain human actions and is useful for shared autonomy and inverse reinforcement learning.The evidence comes from simulation experiments and a small-scale user study.Source: §7 · p. 9
- Future work could estimate children’s intuitive theories without verbal judgments and develop adaptive prosthetic-limb brain-computer interfaces for difficult-to-specify control tasks.Source: §7 · p. 9
9 Appendix
The appendix tests internal-dynamics recovery and transfer in discrete and continuous simulations, then examines reward recovery from misguided demonstrations and assistance for human Lunar Lander users.
9.1.1 Grid World Navigation
- The grid world used 49 states and 49 possible target tasks, with simulated users behaving near-optimally under globally or locally scrambled action labels.The two corruption schemes differ in whether the action-label scrambling is consistent across states.Source: §9.1.1 · p. 13
- Grid-world models recovered the simulated users’ internal dynamics, especially when action scrambling was systematic across states.Figure 4 contrasts globally scrambled actions at left with locally scrambled actions at right.Source: §9.1.1 · p. 13, Figure 4 · p. 14
- Accuracy increased with more training tasks; action-intent regularization helped with globally scrambled actions but varied in effect for locally scrambled actions.The local corruption varies across states, whereas global corruption is consistent throughout the state space.Source: §9.1.1 · p. 14, Figure 4 · p. 14
9.1.2 2D Continuous Navigation
- In continuous navigation, the learned model captured beliefs that inertia was absent and actions controlled velocity; assistance produced more direct paths to the target.The unassisted user tended to oscillate around or miss the target.Source: §9.1.2 · p. 15, Figure 5 · p. 15
9.1.3 Learning Rewards from Misguided User Demonstrations
- Standard IRL failed on misguided demonstrations, while incorporating the learned internal dynamics yielded accurate reward estimates.The comparison used true reward from policies optimized for the learned rewards.Source: §9.1.3 · p. 16
9.2 User Study on the Lunar Lander Game
- Assistance significantly increased success and reduced crashes in the 12-participant Lunar Lander study (success: f(1, 11) = 109.58; crashes: f(1, 11) = 126.33; both p < 0.0001).The study compared assisted and unassisted conditions.Source: §9.2 · pp. 16–17, Figure 6 · p. 17