Source-linked AI summary
Trust-Aware Decision Making for Human-Robot Collaboration: Model Learning and Planning
Min Chen, Stefanos Nikolaidis, Harold Soh, David Hsu, Siddhartha Srinivasa
TL;DR
Human trust affects whether autonomous systems are used appropriately, but prior trust models did not close the loop with robot decision-making. This paper learns a trust-POMDP that infers and modulates latent human trust while selecting actions for long-term team performance. Experiments show that the policy calibrates trust by building it when low and intentionally reducing it when high, so maximizing trust alone can hurt performance when failures are possible.
Problem
Trust can be too low or too high, and prior work had not closed the loop between trust modeling and robot decision-making.
Method
The paper learns a POMDP with latent human trust, trust dynamics, and a human decision model from interaction data.
Results
The trust-POMDP calibrates trust to robot capabilities by building it when low and intentionally reducing it when high.
Takeaways & Limitations
Maximizing trust alone can be detrimental to performance when robotic failures are possible.
Takeaways & Limitations
The model represents trust as a single scalar and assumes static robot capabilities, although multidimensional trust and changing capabilities may be more accurate.
Abstract
from arXiv · showhide
Trust in autonomy is essential for effective human-robot collaboration and user adoption of autonomous systems such as robot assistants. This paper introduces a computational model which integrates trust into robot decision-making. Specifically, we learn from data a partially observable Markov decision process (POMDP) with human trust as a latent variable. The trust-POMDP model provides a principled approach for the robot to (i) infer the trust of a human teammate through interaction, (ii) reason about the effect of its own actions on human trust, and (iii) choose actions that maximize team performance over the long term. We validated the model through human subject experiments on a table-clearing task in simulation (201 participants) and with a real robot (20 participants). In our studies, the robot builds human trust by manipulating low-risk objects first. Interestingly, the robot sometimes fails intentionally in order to modulate human trust and achieve the best team performance. These results show that the trust-POMDP calibrates trust to improve human-robot team performance over the long term. Further, they highlight that maximizing trust alone does not always lead to the best performance.
1 INTRODUCTION
Trust must be calibrated to robot capabilities: low trust can cause unnecessary intervention, while excessive trust can lead to misuse. The trust-POMDP closes the loop between trust modeling and robot decisions, improving long-term collaboration through data-driven trust-aware action selection.
- Motivation: Low trust caused inexperienced participants to stop a capable robot from moving the wine glass, disrupting collaboration.Participants feared the glass might fall and break, despite the robot’s capability to handle it.
- Trust-POMDP model: The paper models human trust as a latent variable in a POMDP with learned trust dynamics and human decision models.The model is learned from interaction data and connects trust evolution to human actions.
- Trust-POMDP model: The trust-POMDP closes the loop between trust modeling and robot decision-making, enabling systematic trust inference and influence for long-term team performance.This extends prior work on trust elicitation and modeling by embedding trust in action selection.
- Validation: Compared with a myopic strategy, the trust-POMDP reduced participant interventions in simulated and real-robot table-clearing experiments.The validation included 201 simulation participants and 20 real-robot participants.
- Behavioral strategy: The strategy builds trust by moving low-risk objects first, then attempts the wine glass after estimated trust increases.The myopic strategy starts with the high-reward wine glass and triggers interventions from users with low initial trust.
- Adaptive failures: When trust is overly high and failures are possible, the robot can intentionally fail to reduce trust and improve long-term team performance.The paper presents intentional failure as an adaptive behavior enabled by the expanded computational framework.
2 RELATED WORK
Prior work has modeled trust, trust dynamics, and human adaptation, while POMDPs support decision-making under hidden human states. This paper links interaction history to human policy by incorporating trust dynamics into robot planning.
- Trust foundations: Trust research has examined ability, integrity, and benevolence, while automation trust differs because automated systems lack intentionality.In human-robot collaboration, the shared task objective motivates focusing on perceived robot ability.
- Trust measurement: Existing studies have used binary, continuous, and ordinal trust measures, including the Area Under Trust Curve for real-time interactive measurement.The cited measures capture trust at different levels of granularity and across interaction experience.
- Trust dynamics: Prior work has modeled temporal trust changes from task performance, transparency, failures, and online estimates of robot trustworthiness.These approaches motivate modeling trust as an evolving quantity during interaction.
- Decision-making under uncertainty: POMDPs provide a general framework for reasoning about hidden human mental states in sequential human-robot decision-making.Prior applications include communication, game AI, autonomous driving, and pedestrian interaction.
- Positioning: The paper incorporates human trust dynamics into planning to link the whole interaction history with the human policy.This extends prior work on online inference of human types, preferences, and mutual adaptation.
3 TRUST-POMDP
The trust-POMDP represents trust as a hidden state that summarizes interaction history and conditions human behavior. It combines belief-based inference with reward-based planning, balancing immediate task reward against information and trust modulation.
- 3.1 Human-robot team model: The team is modeled as an MDP with world states, robot and human actions, probabilistic transitions, and rewards for desirable team behavior.The robot must optimize expected total discounted reward while accounting for the human policy.
- 3.1 Human-robot team model: Because human behavior may depend on arbitrarily long interaction histories, directly optimizing the human policy can become extremely difficult.The framework addresses this history dependence through a compact trust representation.
- 3.2 Trust-dependent human behaviors: Trust is treated as a compact approximation of interaction history, allowing human behavior to be conditioned on inferred trust levels.The paper assumes trust can be represented as a single scalar random variable.
- 3.3 Trust dynamics: Trust evolves according to robot performance, which can depend on world states and both human and robot actions.Performance may represent task success or failure and is learned through interaction.
- 3.4 Maximizing team performance: The POMDP maintains a belief over the partially observed trust state while embedding trust dynamics and human behavior in its transition model.The augmented state combines fully observed world state x with partially observed trust θ.
- 3.4 Maximizing team performance: Robot actions support both exploitation of team reward and exploration that reveals or changes trust to improve future rewards.The resulting policy maps belief states and world states to robot actions and is solved with SARSOP.
4 LEARNING TRUST DYNAMICS AND HUMAN BEHAVIORAL POLICIES
The paper learns trust dynamics and human behavioral policies from table-clearing interaction data, embedding them in a trust-POMDP. The learned models link robot performance, human trust, and intervention behavior across objects with different risks.
- Model and data: The trust-POMDP nests learned trust dynamics and a human behavioral policy within a planning framework.The models are learned from data collected in an online AMT table-clearing experiment.
- Task and data: The table-clearing task involves a human and robot removing three bottles, a fish can, and a wine glass, with humans choosing whether to intervene or stay put.The task repeats until all objects are cleared.
- Task and data: Rewards distinguish robot success, robot failure, and human intervention, with penalties determined by each object’s consequences.Bottle failures incur no penalty, fish-can failures can spill contents, and breaking the glass has the highest penalty.
- Task and data: The data collection design samples pre-specified and random robot policies to cover policy variation without enumerating all O(K!) open-loop policies.Two sessions included an always-successful robot and a robot that failed with high probability.
- Trust dynamics: Trust is modeled as a linear Gaussian process driven by robot performance, including success, failure, or human intervention.Observed trust is measured with Muir’s questionnaire, while Bayesian inference estimates the dynamics parameters.
- Human behavioral policies: The trust-based human model fits collected behavior better than the trust-free model, with log-likelihoods of −153.37 and −156.40, respectively.The authors attribute relatively low likelihoods in both models to large variance among users.
- Human behavioral policies: Human intervention is more frequent for higher-risk objects, while the trust-based model predicts lower intervention as trust increases.The paper therefore identifies low-risk bottles as a useful starting point for building trust before higher-risk objects.
5 EXPERIMENTS
Experiments compared a trust-aware Trust-POMDP policy with a myopic policy in online AMT and real-robot table-clearing studies. The Trust-POMDP achieved higher accumulated rewards by building trust with low-risk objects and reducing interventions on higher-risk objects.
- Team performance: The online AMT accumulated reward was significantly larger under the trust-based condition than the myopic condition, F(1, 199) = 7.81, p = 0.006.
- Team performance: The real-robot accumulated reward was significantly larger under the trust-based condition than the myopic condition, F(1, 18) = 11.22, p = 0.004.
- Team performance: Intervention rates under the Trust-POMDP were 54% and 31% lower for cans and glass in AMT, and 100% and 71% lower for cans and glass with the real robot.
- Policy comparison: The Trust-POMDP started with low-risk bottles, built human trust, and attempted the high-risk glass later, unlike the myopic policy’s highest-to-lowest reward order.
- Human behavioral policy: Participants intervened less as trust increased, while object risk affected intervention: low-risk bottles were tolerated at low trust, whereas high-risk glass prompted intervention without sufficient trust.
6 ROBOT FAILURES
When robot failure is likely, the trust-POMDP reasons about intervention and actively modulates trust rather than simply maximizing it. It can probe initial trust, intentionally fail when trust is high, and achieve higher accumulated reward than trust maximization.
- Failure-aware policy: The trust-POMDP significantly outperforms the myopic policy by choosing whether to build trust with low-risk objects or directly attempt high-risk objects.The choice depends on whether human trust is sufficiently high.
- Failure-aware policy: With glass-failure probability 0.9, human intervention can avoid the large penalty associated with likely robot failure.The computed policy starts with the glass when intervention is most likely at the beginning of the task.
- Adaptive trust modulation: The robot uses a low-risk can pickup to estimate initial trust, then intentionally fails at bottles when trust is high before attempting the glass.If the human intervenes during the can attempt, the robot infers low trust and skips intentional failures.
- Adaptive trust modulation: The performance-maximizing policy reduces human trust to increase mean accumulated reward, whereas the trust-maximizing policy focuses on increasing trust.The policies produce different expected-trust and reward trajectories across likely human action sequences.
- Policy comparison: The trust-maximizing policy starts with the glass because early intervention is more likely and early failure causes a smaller trust decrease.The learned model predicts a steeper trust decline when failure occurs after trust has increased.
- Policy comparison: The mean accumulated reward was −1.36 for performance maximization versus −1.65 for trust maximization, a statistically significant difference (F(1, 19998) = 18.04, p < 0.001).This evaluation indicates that maximizing trust can be suboptimal when robotic failures are possible.
7 CONCLUSION
The paper presents a trust-POMDP that closes the loop between human trust modeling and robot decision-making. Experiments show that it calibrates trust to robot capabilities, improves performance over a trust-ignoring policy, and can intentionally reduce excessive trust when failures are likely.
- Conclusion: The trust-POMDP integrates human trust into robot decision-making and enables systematic inference and influence of trust for fluid collaboration.It closes the loop between trust models and robot decision-making.
- Conclusion: The policy improves performance over a myopic robot by increasing trust through low-risk objects when trust is low and reducing excessive trust through intentional failures.The policy calibrates human trust to the robot’s manipulation capabilities.
- Limitations: The current model treats trust as a single real-valued latent variable and assumes static robot capabilities.The paper also manually specifies the reward function, which may be difficult to specify in practice.