Source-linked AI summary
Shared Autonomy via Hindsight Optimization
Shervin Javdani, Siddhartha S. Srinivasa, J. Andrew Bagnell
TL;DR
Shared autonomy must assist users without knowing their intended goal. The paper models this uncertainty with a POMDP, infers goal distributions using MaxEnt IOC, and uses hindsight optimization for tractable action selection. In a user study, the method enabled faster task completion with less control input, though users showed mixed preferences and the model assumes users do not adapt to assistance.
Problem
Shared autonomy systems often do not know the user’s intended goal and must infer it while providing assistance.
Method
The paper formulates shared autonomy as a POMDP, uses MaxEnt IOC to infer goal distributions from user inputs, and applies hindsight optimization to select assistance actions.
Results
The proposed system enabled users to complete tasks significantly faster and with less control input than predict-then-blend in a user study.
Takeaways & Limitations
Reasoning over an entire goal distribution supports efficient assistance even when confidence in a single goal is low.
Takeaways & Limitations
The model assumes users do not consider or adapt their strategy to the assistance, motivating extension as a stochastic game.
Abstract
from arXiv · showhide
In shared autonomy, user input and robot autonomy are combined to control a robot to achieve a goal. Often, the robot does not know a priori which goal the user wants to achieve, and must both predict the user's intended goal, and assist in achieving that goal. We formulate the problem of shared autonomy as a Partially Observable Markov Decision Process with uncertainty over the user's goal. We utilize maximum entropy inverse optimal control to estimate a distribution over the user's goal based on the history of inputs. Ideally, the robot assists the user by solving for an action which minimizes the expected cost-to-go for the (unknown) goal. As solving the POMDP to select the optimal action is intractable, we use hindsight optimization to approximate the solution. In a user study, we compare our method to a standard predict-then-blend approach. We find that our method enables users to accomplish tasks more quickly while utilizing less input. However, when asked to rate each system, users were mixed in their assessment, citing a tradeoff between maintaining control authority and accomplishing tasks quickly.
I. INTRODUCTION
Shared autonomy addresses uncertain user goals by combining teleoperation with assistance over a distribution of possible goals. The paper formalizes this as a POMDP and approximates action selection with hindsight optimization, then evaluates the approach against predict-then-blend.
- Shared autonomy combines teleoperation with autonomous assistance to address noisy, low-dimensional interfaces that can make operation tedious or goals impossible.
- Prior shared-autonomy methods commonly predict a single user goal and then assist for that goal, using confidence to regulate assistance.
- The proposed approach assists over an entire goal distribution, enabling assistance when confidence in any particular goal is low, especially in cluttered environments.
- The framework models shared autonomy as a POMDP with goal uncertainty and selects assistance actions by minimizing expected cost-to-go for the user and robot.
- Hindsight optimization, or QMDP, approximates the intractable POMDP solution while remaining computationally efficient and avoiding explicit information gathering.
- In a user study, the proposed system enabled faster task completion with less control input, although users showed mixed preferences and cited a control-authority–efficiency trade-off.
B. Assistance Methods
The paper models assistance under uncertain goals by combining goal inference, user and robot dynamics, and cost-sensitive action selection. Its approach reasons over multiple goals while approximating optimal POMDP actions with QMDP.
- Assistance Methods: Traditional assistance methods often assume a known goal or predict one most-likely goal before blending autonomy with user control.
- Assistance Methods: Recent methods plan over an entire goal distribution, but prior computation can rely on squared-distance costs that decompose over goals.
- Assistance Methods: POMDP-based assistance has also been used for pedestrian avoidance and cooperative games, where agents infer intentions and select assistance actions.
- Problem Statement: The proposed system assumes a discrete set of possible goals, infers the intended goal from user inputs, and selects actions minimizing expected goal-achievement cost.
- Problem Statement: The robot is modeled with continuous state and actions, deterministic dynamics, and a direct-teleoperation mapping from user inputs to robot actions.
- Problem Statement: MaxEnt IOC models the user as approximately optimizing a goal-specific cost function, while the POMDP represents uncertainty over the intended goal.
- Problem Statement: User inputs serve as observations for Bayesian inference over goals, and the robot cost can depend on those inputs to penalize actions that deviate from direct teleoperation.
- Hindsight Optimization: Because exact POMDP action selection is intractable, QMDP estimates cost-to-go assuming full observability next step and selects computationally efficient actions.
IV. MODELLING THE USER POLICY
The system models user inputs with MaxEnt IOC and uses input history to infer goal probabilities while accounting for assistance-induced feedback. The resulting prediction is kept dependent on user inputs rather than robot actions.
- User-policy model: The system defines robot-state and user-input sequences whose next state need not result from applying the recorded user input.This accommodates assisted interaction, where robot actions can alter the trajectory independently of user inputs.
- User-policy model: MaxEnt IOC models user behavior with a cost function whose trajectory probabilities decrease exponentially with cost.The cost function can be learned efficiently from user demonstrations.
- User-policy model: Dynamic programming computes the sequence-cost normalizer through soft-minimum value iteration, avoiding explicit enumeration of all sequences.Explicitly evaluating the partition function would require enumerating every sequence and calculating its cost.
- User-policy model: In the grasping example, user inputs shift both goal probabilities and the estimated value function toward the indicated direction.Without input, the robot moves toward the lowest-value position while the estimates remain unchanged.
- User-policy model: The model estimates goal probabilities from user inputs only, preventing positive feedback from robot actions that would otherwise reinforce already-likely goals.The goal posterior is computed for the partial sequence using Bayes’ rule.
V. HINDSIGHT OPTIMIZATION
Because selecting an optimal action in the goal-uncertain POMDP is intractable, the system uses QMDP, or hindsight optimization, to plan efficiently under the assumption that the goal becomes observable next.
- Hindsight optimization: QMDP approximates belief-state cost-to-go by assuming full observability will be obtained at the next time step.This produces deterministic goal-specific subproblems rather than solving the full POMDP.
- Hindsight optimization: The approximation does not gather information explicitly, but it can plan efficiently when information is expected from user interaction.The method assumes users provide inputs continuously, supplying information without explicit information-gathering actions.
- Hindsight optimization: The belief state represents uncertainty over goals as p(g|ξ0→t), and QMDP combines goal-specific action values using that belief.The action-value function conditions on robot state, robot action, and user input.
- Hindsight optimization: When the belief-state action maximization is unavailable directly, the system follows a first-order gradient approximation of the POMDP action-value.The paper presents two strategies for approximating the goal-specific action values.
1) Robot and user both act:
One approximation assumes the robot and user both act, while the implemented alternative assumes the robot takes over and completes the task optimally without further user input.
- Robot takes over: The robot-takes-over approximation assumes the user stops supplying inputs and uses Crob(s, a, u) = Crob(s, a, 0).This permits analytical value computation for many cost functions, such as constant-velocity motion toward the goal.
- Robot takes over: Unlike the standard QMDP alternative, robot takeover makes no assumptions about the user policy, improving robustness to modelling errors.The paper uses this method in its experiments.
VI. MULTI-GOAL MDP
For goals with multiple acceptable targets, the method derives goal-level value and action-value functions from target-level functions by selecting the target with the lowest cost-to-go.
- Multi-target assistance: A goal may contain multiple acceptable targets, such as several grasp poses for one object.Each target has corresponding robot and user cost functions.
- Multi-target assistance: The cost of a state-action pair is assigned using the target with the minimum cost-to-go after the resulting state.The resulting state x′ is the robot state after applying action a at x.
- Multi-target assistance: For deterministic transitions, the theorem states that goal value and action-value functions can be computed from target-level functions.The proof applies standard backward value iteration recursively.
- Multi-target assistance: The goal value equals the minimum target value, Vg(x) = minκ Vκ(x), under the deterministic-system construction.The paper derives this from the goal cost being bounded by the cost-to-go of each specific target.
B. Multi-Target Prediction
The method represents goals containing multiple acceptable targets with a distribution over target costs, then computes a soft-min goal value function from target value functions. This decomposition enables efficient goal-level prediction and assistance.
- Goals are modeled as distributions over multiple acceptable targets rather than the cost of a single target.
- The goal value function is computed from target value functions using a soft-min operation.For grasping, the goal value corresponds to the soft-min of 16 target value functions.
- The derivation marginalizes over targets and future trajectories to obtain the probability of the current input.
VII. USER STUDY
The user study compares the proposed policy with a conventional predict-then-blend system using shared prediction components and distinct assistance mechanisms. The policy uses specified cost functions, while blending relies on confidence-based arbitration.
- The study compares the proposed policy method with a conventional predict-then-blend approach called blend.
- Both systems use the same goal-prediction algorithm, with second-order approximations replacing difficult soft-min computations for efficiency.
- The policy requires separate user and robot cost functions, including a penalty for deviating from the user command.
- The policy’s distance-based cost is linear near a goal and constant beyond a distance threshold.The authors identify collision avoidance as a possible improvement to this baseline cost function.
- The baseline blend method uses predictor confidence to arbitrate between autonomy and user input.Confidence is based on distance to the nearest target and becomes zero beyond a threshold.
A. Hypotheses
The user study evaluates efficiency and satisfaction when participants grasp objects with two teleoperation systems. It measures completion time and user input, and surveys control, task success, speed, system preference, and willingness to use each system.
- Hypotheses: Efficiency is measured by total execution time and total user input, while satisfaction is assessed through a survey.
- Hypotheses: The hypotheses predict that policy users will grasp objects faster, use less control input, and show stronger preference for policy than blend users.
- Experiment setup: Ten robotics-experienced participants used both systems in a within-subjects design.
- Experiment setup: Participants grasped a canteen, block, and cup using two joysticks to control a robot arm.
- Experiment setup: Each system was tested once per object in randomized order, with system order counterbalanced across participants.
- Survey: The survey rated perceived control, whether the robot did what users wanted, task speed, willingness to use the system, and overall preference.
C. Results
The policy method improved task efficiency and reduced user input, but users did not consistently prefer it and often favored blending for its greater perceived control.
- Task performance: Users successfully used both systems, with two failures occurring under each method.Failures arose from attempting grasps too early or knocking objects over, and trials were reset and repeated.
- Task performance: Policy users grasped objects faster than with blending, supporting H1 (F(1, 9) = 12.98, p = 0.006).
- Task performance: Policy users grasped objects with less control input than with blending, supporting H2 (F(1, 9) = 7.76, p = 0.021).
- User preference: There was no evidence supporting H3 that users would agree more strongly on preferring policy.
- User preference: Users trended toward preferring blend: system-use ratings were Blend M = 4.90, SD = 1.58 and Policy M = 4.10, SD = 1.64, while preference was Blend M = 2.90, SD = 1.76.These differences were not statistically significant at α = 0.05.
- User preference: Users felt more in control during blending, although the methods differed less in whether the robot did what they wanted.Some users therefore viewed policy actions as aligned with their goals despite not feeling in control.
- User preference: Users sometimes compensated for policy actions because they produced unexpected or unplanned motion.The authors suggest user-specific policies as a possible way to alleviate this issue.
D. Examining trajectories
Users with different preferences adopted different interaction strategies: policy-preferring users ceded more work to the robot, while blend-preferring users often opposed autonomous actions.
- Examining trajectories: Policy-preferring users changed strategy to exploit assistance toward all goals, applying minimal input to guide the robot toward the correct goal.
- Examining trajectories: Blend-preferring users often opposed autonomous policy actions, suggesting the robot followed a strategy different from their own.
- Conclusion and future work: The framework assists over an entire goal distribution rather than predicting one goal before assisting, using MaxEnt IOC and Hindsight Optimization.
- Conclusion and future work: The user study found faster task completion with less control input, despite mixed preferences trending toward the simpler predict-then-blend approach.
- Conclusion and future work: The model assumes users do not consider or adapt their input strategy to assistance.The authors propose extending the model as a stochastic game to address this assumption.