Source-linked AI summary
Establishing Appropriate Trust via Critical States
Sandy H. Huang, Kush Bhatia, Pieter Abbeel, Anca D. Dragan
TL;DR
Learned neural-network policies make it difficult for users to understand a robot’s capabilities and calibrate trust. The paper proposes showing users the policy’s critical states; its studies suggest this can support more appropriate trust and inform decisions about deployment and intervention.
Problem
Users need an accurate model of robot capabilities for appropriate trust, but learned black-box policies are difficult to assess because their actions cannot be examined in every possible state.
Source: §I · p. 1Method
The approach exposes users to the states the policy considers critical so they can assess its actions and decide whether to deploy it or take control.
Source: §I · pp. 1–2Results
The user studies suggest that observing critical-state behavior can support more appropriate trust than observing the robot over time, while some policy-quality preferences were not statistically significant.
Source: §I · p. 2, B. Analysis · p. 7Takeaways & Limitations
The authors describe showing critical states as a promising way to build trust and reveal whether a policy is trustworthy, for policies trained with maximum-entropy methods.
Source: §VI · p. 8The approach places substantial mental burden on users, assumes task expertise, and requires them to generalize from observed states to identify false-negative critical states.
Source: §VI · p. 8
Abstract
from arXiv · showhide
In order to effectively interact with or supervise a robot, humans need to have an accurate mental model of its capabilities and how it acts. Learned neural network policies make that particularly challenging. We propose an approach for helping end-users build a mental model of such policies. Our key observation is that for most tasks, the essence of the policy is captured in a few critical states: states in which it is very important to take a certain action. Our user studies show that if the robot shows a human what its understanding of the task's critical states is, then the human can make a more informed decision about whether to deploy the policy, and if she does deploy it, when she needs to take control from it at execution time.
I. INTRODUCTION
Accurate models of a robot’s capabilities matter for appropriate trust, yet learned black-box policies are difficult to assess across all possible states. The paper proposes showing users the policy’s critical states so they can judge whether to deploy it and when to take control.
I. INTRODUCTION
- Inaccurate trust can lead users either to interfere unnecessarily and miss robot capabilities or to expect correct behavior where the robot cannot handle a situation.Over-trust may lead to unexpected behavior, injuries, or damage.Source: §I · p. 1
- Assessing learned policies is difficult because they map raw inputs to action distributions, and their actions cannot be examined in every possible state.Performance summaries such as accident-free mileage do not tell users which states the policy still cannot handle.Source: §I · p. 1
- For many tasks, only a few states require a particular action because outcomes are similar across actions in most other states.For example, an autonomous car must slow when the vehicle ahead brakes, unlike when no vehicles are nearby.Source: §I · p. 1
- Showing a policy’s critical states is proposed as a way to help users judge whether to deploy it and when to take control.The studies suggest this can support more appropriate trust than observing the robot act over time.Source: §I · pp. 1–2
II. BACKGROUND
The background formalizes the robot’s decision problem as a Markov Decision Process and introduces policy values used to characterize critical states.
II. BACKGROUND
- A Markov Decision Process is specified by state and action spaces, transition probabilities, a reward function, and a discount factor.The discount factor γ lies in (0, 1].Source: A. Notation · p. 2
- A robot policy π maps each state to a probability distribution over actions.Source: A. Notation · p. 2
- A critical state is one where action-value Qπ varies greatly across actions, with only a few actions having high values.Most actions have mediocre or low values in such a state.Source: A. Notation · p. 2
B. Maximum-Entropy Reinforcement Learning
Maximum-entropy reinforcement learning combines return maximization with policy randomness. This can make critical states meaningful because the policy is random where actions have little impact and purposeful where they matter substantially.
B. Maximum-Entropy Reinforcement Learning
- Maximum-entropy reinforcement learning trains a policy to maximize return while also acting as randomly as possible.The tradeoff between return and entropy is set by α.Source: B. Maximum-Entropy Reinforcement… · p. 2
- Policies trained with maximum entropy can act randomly where actions have little impact on return and purposefully where actions have major impact.This behavior yields meaningful critical states.Source: B. Maximum-Entropy Reinforcement… · p. 2
- The paper trains neural-network policies with Soft Actor-Critic, a deep reinforcement-learning method based on maximum-entropy learning.The authors report that SAC produces policies with meaningful critical states in practice.Source: B. Maximum-Entropy Reinforcement… · p. 2
III. COMPUTING AND USING CRITICAL STATES
The paper identifies critical states using either a policy’s action uncertainty or its action-value function. These criteria require only policy or value-function outputs, so they can be applied to black-box policies.
A. Computation of Critical States
- A policy-based criterion marks states critical when its action-distribution entropy falls below a threshold.The criterion applies to continuous and discrete action spaces.Source: A. Computation of Critical States · p. 2
- A value-based criterion marks states where random actions perform much worse than optimal actions, using the learned action-value function.For continuous action spaces, the authors say to apply it after discretization; one-step rollouts can estimate action values from a learned value function.Source: A. Computation of Critical States · p. 2
- Action-value functions proved more reliable in practice because policies may exploit environment features, such as action clipping, to maximize entropy.The authors trained policies with SAC, which learns a policy and action-value function in parallel.Source: A. Computation of Critical States · p. 2
- Both approaches are implementation-agnostic and need only policy or action-value outputs, enabling application to black-box policies.In the Pong example, the query states s1–s4 are non-critical, while s5 and s6 are critical; the displayed sets vary in whether they include correct critical states or incorrect actions.Source: A. Computation of Critical States · p. 2, Figure 2 · p. 3
B. Using Critical States
Users inspect a policy’s actions in its critical states before deciding whether to deploy it. At execution time, those observations help distinguish states where they should trust the policy from missed critical states where they should intervene.
B. Using Critical States
- Before deployment, users inspect the policy’s actions in its critical states and may decline deployment after spotting false positives, false negatives, or incorrect actions.The paper assumes an expert user, while recognizing that users’ ground-truth critical-state sets may differ.Source: B. Using Critical States · p. 3
C. Justification of Critical States
- At execution time, users should trust the policy in critical states they inspected and approved, but take control in critical states the policy missed.For non-critical states, users do not care which action the policy chooses and refrain from taking control.Source: C. Justification of Critical States · p. 3
- Without observing the policy in its critical states beforehand, users cannot distinguish missed critical states from approved ones where intervention is unnecessary.The section frames intervention as relevant to execution-time supervision.Source: C. Justification of Critical States · p. 3, B. Using Critical States · p. 3
IV. USER STUDY: IMPACT OF CRITICAL STATES
The study tests how users respond to different errors in a policy’s critical states, first with constructed sets and then with actual trained policies. Its first experiment uses Pong to compare critical-state displays with a performance-statistic baseline.
IV. USER STUDY: IMPACT OF CRITICAL STATES
- The initial study constructs critical-state sets with at most one error to compare how false positives, false negatives, and incorrect actions affect trust.A later main study exposes users to critical states from trained policies.Source: §IV · p. 4
A. Experiment Design
- In Pong, participants first judged which states were critical, then saw a policy representation, and finally chose whether to take control at the same states.The ball heading toward the participant’s paddle close to it is critical; states where it heads back toward the opponent are non-critical.Source: A. Experiment Design · p. 4
- The study varied whether displayed critical states were correct, false-negative, weak-false-positive, false-positive, or incorrect-action, against a baseline stating the policy wins 95% of cases.Across six conditions, 72 participants were assigned between subjects; trust was measured with Likert questions and intervention choices.Source: A. Experiment Design · pp. 4–5
B. Analysis
Participants responded most strongly to incorrect actions in critical states, while false-positive and false-negative displays produced weaker responses. They often intervened even when a correct policy’s critical states suggested intervention was unnecessary.
B. Analysis
- Incorrect-action displays lowered all three subjective trust measures versus correct displays (Student’s t test, p < 0.0001), and increased intervention at s3 (p < 0.01) and s5 (p = 0.05).Participants also took control more often in incorrect-action than correct conditions at the weakly critical state s3.Source: B. Analysis · p. 5
- False-positive and false-negative displays did not lower subjective trust versus correct displays, although false positives showed a trend in the expected direction.The authors suggest Pong’s relative simplicity may have encouraged users to give policies the benefit of the doubt.Source: B. Analysis · p. 5
- Almost half of users who saw a correct critical-state set still chose to take control in a state where the policy had acted correctly.The authors note that participants therefore sometimes took control when it was not necessary.Source: B. Analysis · p. 5
- At the two critical query states, users tended to intervene less after seeing correct critical states than after hearing only the baseline performance statistic.Across conditions, users generally intervened at the two critical states and not the others.Source: B. Analysis · p. 5
- Users tended to intervene more after seeing false-negative critical states than after seeing correct ones; false-positive displays produced only slight intervention improvements.For s5 and s6, the false-positive and false-negative intervention trend had weak significance (p = 0.11).Source: B. Analysis · p. 5
V. USER STUDY: UTILITY OF CRITICAL STATES
The main user study compares showing policy critical states with rollout and random-state exposure to assess how well each helps users identify the better driving policy.
V. USER STUDY: UTILITY OF CRITICAL STATES
- The study tests whether critical-state exposure establishes more appropriate trust than a one-minute rollout or random-state exposure.It follows a previous study of how users respond to errors in critical states.Source: §V · p. 5
- Two neural-network driving policies were trained with SAC using four-component Gaussian mixture policies.Source: §V · p. 5
- The study presents ten critical states per policy, selected by clustering the top 10% most critical rollout states in last-hidden-layer feature space.It uses k-means++ and shows the most critical state from each of ten clusters.Source: §V · p. 5
A. Experimental Design
The experiment compares critical-state exposure with rollout and random-state baselines for two similarly performing policies, one of which fails in scenarios the other handles.
A. Experimental Design
- The driving simulator models highway passing with continuous steering actions, while the reward favors forward progress and penalizes risky or inefficient driving.The state includes the robot car’s lane, position, and heading, plus nearby cars’ relative positions, headings, and speeds; steering actions are discretized into 200 choices to compute critical states.Source: A. Experimental Design · pp. 5–6
- The figure shows query states and a subset of ten critical states; s1 and s2 are not critical, while the other shown query states are.Source: Figure 5 · p. 6
- The study compares critical-state exposure with a one-minute policy rollout and exposure to randomly selected states.The baselines approximate states users might encounter while observing or interacting with a robot over time.Source: A. Experimental Design · p. 6
- πA averages one crash per 700 timesteps and πB one per 640 timesteps, but πB fails in simple traffic scenarios that πA handles.The study notes that crashes can be inevitable because the agent can only steer while surrounding cars drive more slowly.Source: A. Experimental Design · p. 6
- The study measures trust with Likert responses and asks whether participants would take control at each query state.It uses a between-subjects design for exposure type and a within-subjects design for policy quality, with 60 participants across three conditions.Source: A. Experimental Design · p. 7
B. Analysis
Participants exposed to critical states more often distinguished the better policy through takeover behavior, while subjective ratings favored it as a nonsignificant trend.
B. Analysis
- Across five subjective questions, critical-state viewers tended to favor πA, while random-state viewers rated πA and πB similarly.The trend favoring πA in the critical-state condition was not statistically significant; the trust interaction was weak (F(2,57) = 2.37, p = 0.1).Source: B. Analysis · p. 7
- Across critical query states s3–s9, participants seeing critical states were more likely to trust a policy and not take control than those seeing either baseline exposure.Source: B. Analysis · p. 7
- Participants who saw critical states took control significantly more often for worse policy πB than πA; this difference was absent in both baseline conditions.Across the seven critical query states, exposure had a significant effect (F(2,57) = 5.57, p = 0.006), and the exposure-by-policy-quality interaction was significant (F(2,777) = 5.30, p = 0.005); the post-hoc comparison for πB had p = 0.001.Source: B. Analysis · pp. 7–8
- The authors conclude that showing critical states can help users calibrate trust between better and less-good policies.Source: B. Analysis · p. 8
VI. DISCUSSION AND FUTURE WORK
User studies suggest that showing a policy’s critical states can help build trust and reveal whether it is trustworthy. The approach is a step toward helping end-users decide whether to deploy a robot and when to take control.
VI. DISCUSSION AND FUTURE WORK
- User studies suggest that showing a policy’s critical states is promising for building trust and revealing whether the policy is trustworthy.The authors state that the approach applies to policies trained with a maximum-entropy-based approach.Source: §VI · p. 8
- Whether to trust a policy with incorrect critical states despite strong training-environment performance remains an open question.The authors’ hunch is that nonsensical critical states may signal difficulty generalizing to states outside the training distribution.Source: §VI · p. 8
- The approach places significant responsibility and mental burden on end-users, including judging states beyond those shown.It assumes task domain knowledge; one proposed remedy is to show additional states to help correct users’ understanding.Source: §VI · p. 8
- Showing critical states is a step toward helping end-users decide whether to deploy a robot and when to take control during deployment.Source: §VI · p. 8