PaperlayerView exact revision on arXiv

Source-linked AI summary

Shared Autonomy via Deep Reinforcement Learning

Siddharth Reddy, Anca D. Dragan, Sergey Levine

arXiv:1802.01744v2cs.LGcs.HCcs.RO

TL;DR

Prior shared-autonomy methods often require known dynamics, goal representations, or user policies, limiting use in real-world tasks. This paper learns model-free assistance from observations, user input, and task reward, and reports improved assistance in simulated, game-based, and real-quadrotor studies.

  • Problem

    Prior shared-autonomy approaches commonly assume known environment dynamics, user policies, and possible goal representations, although these assumptions may not hold in real-world tasks.

  • Method

    The paper uses human-in-the-loop deep reinforcement learning to map environmental observations and user input to action values, balancing user-following with assistance through value thresholding and action similarity.

  • Results

    The method improved user performance across Lunar Lander and quadrotor studies, with the real-quadrotor pilot study reporting more successes and fewer crashes with a copilot than without one.

  • Takeaways & Limitations

    The studies suggest that model-free deep reinforcement learning can provide assistive control without direct access to the user’s private information.

  • Takeaways & Limitations

    Inferring user intent in general will require memory, and users’ adaptation to the robot’s interface remains an open question.

Abstract

from arXiv · show

In shared autonomy, user input is combined with semi-autonomous control to achieve a common goal. The goal is often unknown ex-ante, so prior work enables agents to infer the goal from user input and assist with the task. Such methods tend to assume some combination of knowledge of the dynamics of the environment, the user's policy given their goal, and the set of possible goals the user might target, which limits their application to real-world scenarios. We propose a deep reinforcement learning framework for model-free shared autonomy that lifts these assumptions. We use human-in-the-loop reinforcement learning with neural network function approximation to learn an end-to-end mapping from environmental observation and user input to agent action values, with task reward as the only form of supervision. This approach poses the challenge of following user commands closely enough to provide the user with real-time action feedback and thereby ensure high-quality user input, but also deviating from the user's actions when they are suboptimal. We balance these two needs by discarding actions whose values fall below some threshold, then selecting the remaining action closest to the user's input. Controlled studies with users (n = 12) and synthetic pilots playing a video game, and a pilot study with users (n = 4) flying a real quadrotor, demonstrate the ability of our algorithm to assist users with real-time control tasks in which the agent cannot directly access the user's private information through observations, but receives a reward signal and user input that both depend on the user's intent. The agent learns to assist the user without access to this private information, implicitly inferring it from the user's input. This paper is a proof of concept that illustrates the potential for deep reinforcement learning to enable flexible and practical assistive systems.

I. INTRODUCTION

Shared autonomy combines user input with automated assistance, but prior approaches often rely on known dynamics, goal representations, and user policies. This paper proposes model-free deep reinforcement learning to learn assistance from observations, user input, and task reward, and evaluates it in simulated and real-time control tasks.

  • Prior shared-autonomy methods infer hidden user goals from input but commonly assume known dynamics, goal spaces, and goal-conditioned user policies.
  • These assumptions limit adaptability when dynamics are difficult to model, goals are hard to specify, or user input is systematically suboptimal.
  • The proposed model-free deep reinforcement learning algorithm maps environmental observations and user input to action values using task reward as the only supervision.
  • The method must preserve user feedback control while deviating from suboptimal commands, because ignoring input can reduce the quality of real-time user control.
  • The studies evaluate assistance with simulated pilots and users in Lunar Lander, plus users flying a real quadrotor.

II. RELATED WORK

The paper situates its approach among reinforcement-learning and shared-autonomy methods for teleoperation. Its model-free formulation targets settings where analytic environment and goal information required by related approaches is unavailable.

  • Shared-autonomy teleoperation methods such as parallel autonomy minimally adjust user input to produce safe trajectories, often using analytic environment descriptions.
  • Outer-loop stabilization requires knowledge of the user’s goal, whereas this paper addresses assistance without a dynamics model or goal representation.
  • The paper introduces its shared-autonomy problem after recapping reinforcement learning and the prior shared-autonomy formulation.
  • The paper builds on reinforcement learning, where Q-learning learns action values for policies that maximize expected future discounted return.

B. Shared Autonomy

Prior shared-autonomy formulations model the hidden user goal as partial observability in a POMDP. They assume the candidate goals, user policy, and environment dynamics are known to the agent.

  • The prior formulation augments the environment state with a user-known but agent-unknown goal, while the agent knows the candidate goal set.
  • The user’s goal-conditioned policy generates observed user input, and known environment dynamics define transitions that preserve the hidden goal.
  • The agent solves this partially observable problem using approximate methods such as hindsight optimization.
  • This formulation assumes the goal space, user policy, and transition dynamics are known ex-ante.

IV. MODEL-FREE SHARED AUTONOMY

The paper relaxes prior shared-autonomy assumptions by making dynamics, user policy, and goal space potentially unknown. It retains task-relevant reward structure through known general terms and observed user feedback.

  • The model-free method removes assumptions about known dynamics, the user’s private-information observation model, and the candidate goal set.
  • The formulation allows the transition model, user policy, and goal space to be unknown while retaining a reward that depends on private user information.
  • The reward is decomposed into known general terms and an unknown feedback term that is observed when the user reports success or failure.
  • Known reward terms can encode generally useful requirements such as avoiding collisions.
  • When the goal space and user policy are known, the goal can structure feedback and be inferred from the user policy using Bayesian inference.

B. Method Overview

The method concatenates environmental observations with user controls or inferred goals, then learns action values through end-to-end neural Q-learning. It can use raw actions, Bayesian goal inference, or supervised goal prediction depending on available information.

  • Method Overview: The agent concatenates environmental observations with user information and maps them to action values using neural-network Q-learning.When dynamics are unknown, user controls are incorporated as observations for the agent.
  • Method Overview: When the goal space is unknown, the method uses the user’s actions directly as the input representation.This setting is called raw action embedding.
  • Method Overview: When the user policy is known, Bayesian inference estimates the most probable goal from the user’s actions.Additional actions improve the goal estimate over time.
  • Method Overview: When a user-policy model is unavailable but the goal space is known, the policy learns to decode user actions and infer intent implicitly.The end-to-end network can discover relationships between controls and observations without explicit intent labels.

D. Q-Learning with User Control

Human-in-the-loop deep Q-learning addresses the need to preserve informative user feedback while reducing interaction demands. The implementation uses neural fitted Q-iteration with replay, a target network, and double Q-learning.

  • Q-Learning with User Control: Human-in-the-loop reinforcement learning must preserve informative user input while minimizing the number of environmental interactions.Ignoring user suggestions can degrade real-time control, while extensive interaction requirements burden human users.
  • Q-Learning with User Control: The method uses neural fitted Q-iteration with experience replay, a periodically updated target network, and double Q-learning.These components support off-policy value learning for human-in-the-loop training.
  • Q-Learning with User Control: A reactive policy was more effective for the reported tasks than integrating the user’s past actions with a recurrent policy.Past actions are acknowledged as potentially informative of intent, but the experiments favored a reactive policy.
  • Q-Learning with User Control: The algorithm initializes replay memory and Q-function parameters, stores observed transitions, trains on sampled minibatches, and periodically resets the target network.The training loop uses target values formed with double Q-learning.
  • Q-Learning with User Control: The learned state-action value function evaluates expected future return from the current environmental observation and user control input.Actions are selected using the approximate Q function rather than following a fixed user or agent policy exactly.

E. Control Sharing

The control-sharing policy balances user responsiveness with autonomous correction by choosing the user-nearest action among actions whose estimated values remain sufficiently close to optimal.

  • Control Sharing: The policy selects a feasible action closest to the user’s suggestion.An action is feasible when its value is not too much worse than the optimal action.
  • Control Sharing: The tolerance parameter α controls how much suboptimality in human suggestions the system accepts and therefore how much assistance it provides.Larger tolerance changes the trade-off between following the user and deviating toward higher-value actions.
  • Control Sharing: Subtracting the minimum action value from Q values prevents the feasible-action set from becoming empty when all action values are negative.The feasibility condition is also invariant to affine scaling of Q values.

V. SIMULATION EXPERIMENTS

The simulation experiments evaluate model-free shared autonomy with simulated pilots in Lunar Lander, where the landing goal is visible to the pilot but hidden from the agent’s state. The setup varies pilot behavior and measures team performance.

  • Simulation setup: The experiments analyze the method under different simulated pilots before testing real users.Simulation enables examination of tolerance, training, and testing across different input types.
  • Evaluation: The evaluation reports reward, success rate, and crash rate for simulated pilot-copilot teams on Lunar Lander.Table I reports reward standard errors over ten random seeds and the specified evaluation episodes.
  • Lunar Lander task: Lunar Lander requires piloting to a randomly selected landing site using two lateral thrusters and a main engine without crashing.Episodes end after crashing, leaving bounds, landing, remaining stationary, or timing out.
  • Lunar Lander task: The landing site is not directly available in the agent’s eight-dimensional state, while a human can see flags and provide a suggested control.Depending on the pilot policy, the suggestion may be an approximately optimal action or a signal about landing-site direction.
  • Implementation: The copilot uses a multilayer perceptron to approximate the Q function and an action-similarity function that counts agreeing action dimensions.The reward combines hard-coded general terms with terminal feedback for successful landing or failure.

A. Testing Unstructured Copilot Performance

In the Min-Assumptions setting, copilot assistance improves performance across simulated pilots, with the best tolerance level depending on the pilot’s behavior.

  • LAGGYPILOT models an otherwise optimal pilot with slow action changes, while NOISYPILOT models an otherwise optimal pilot that occasionally takes the wrong action.LAGGYPILOT repeats the previous action with probability p = 0.85; NOISYPILOT uses ϵ-greedy behavior with ϵ = 0.3 at test time.
  • SENSORPILOT signals the landing-site location through lateral-thruster actions while ignoring gravity and the main engine.
  • The pilot-copilot team outperforms both solo LAGGYPILOT and solo copilot on Lunar Lander.It crashes and goes out of bounds less often, uses less fuel, follows stabler trajectories, and finds the landing site more often.
  • NOISYPILOT and SENSORPILOT also benefit from assistance, although SENSORPILOT’s success rate does not substantially increase.
  • α = 0 is optimal for SENSORPILOT, whereas α ≈0.5 is optimal for LAGGYPILOT and NOISYPILOT.The tolerance hyperparameter is evaluated while shaping R_feedback to improve SENSORPILOT performance.

B. The Benefit of Structure when Structure Exists

When goal and user-policy structure is available, structured input decoding can outperform raw action embedding, while pilot-specific training and human assistance produce distinct benefits across evaluations.

  • B. The Benefit of Structure when Structure Exists: Bayesian goal inference and supervised goal prediction outperform raw action embedding when the goal space and user model are known.The study compares Bayesian goal inference, supervised goal prediction, and raw action embedding across pilot-tolerance values.
  • B. The Benefit of Structure when Structure Exists: Supervised goal prediction should replace raw action embedding when a convenient user-policy model is unavailable.
  • C. Adapting to Diverse Users: Copilots trained with different simulated pilots show evidence of user-specific adaptation, especially for SENSORPILOT.The SENSORPILOT-trained copilot acquires a relatively unique assistive policy, suggesting its goal-signaling behavior differs qualitatively from the other pilots.
  • C. Adapting to Diverse Users: A copilot trained without a pilot is just as effective at helping LAGGYPILOT as one trained with LAGGYPILOT in the loop.This supports pretraining without a human followed by fine-tuning with the human pilot.
  • VI. USER STUDY WITH A GAME AGENT: Human-copilot teams in Lunar Lander achieve significantly more successes and fewer crashes than solo human pilots.The game study used 30 episodes per condition and found a clear quantitative and qualitative benefit from combining a human pilot with a copilot.
  • VI. USER STUDY WITH A GAME AGENT: Users generally benefit from the copilot, particularly through crash avoidance, but perceive its behavior as somewhat inconsistent and overly aggressive.

VII. USER STUDY WITH A PHYSICAL ROBOT: QUADROTOR PERCHING

The physical-robot study evaluates shared autonomy for quadrotor perching under realistic control constraints, using simulation pretraining followed by human fine-tuning. With the copilot, users reach the landing pad more consistently, though the small sample makes the evidence mainly anecdotal.

  • The quadrotor perching task requires landing on a level square pad using 18 discrete movement, descent, hovering, and yaw actions.The state encodes vehicle position, velocity, angle, angular velocity, and the horizontal landing-pad offset.
  • The copilot approximates a Q function over environmental state and paired actions, combining general and user-generated reward signals.Action similarity counts the dimensions in which the copilot action and human action agree.
  • Pretraining uses an idealized point-mass simulation without external forces, execution noise, or sensor error, followed by fine-tuning on human-pilot data.
  • The real quadrotor’s target angle is randomly sampled during pretraining but is not directly available in the agent’s state.
  • With a copilot, human pilots consistently reach the landing pad, achieving significantly more successes and fewer crashes than without assistance.The study reports n = 4 participants; repeated-measures ANOVA results were f(1, 3) = 44.1045, p < 0.01 for success and f(1, 3) = 62.3151, p < 0.01 for crashes.
  • The physical-robot evidence is mainly anecdotal because the sample size is n = 4 participants.

VIII. DISCUSSION

The paper presents model-free reinforcement learning for shared autonomy under unknown dynamics, user policies, and goal representations, while identifying memory and user adaptation as open challenges.

  • The proposed algorithm helps human users with tasks without requiring known dynamics, user policies, or goal representations.
  • A behavioral policy lets users directly control the assistance level, while reward decomposition supports rapid learning of general behaviors and adaptation to individual users.
  • User studies with a virtual agent and a real robot suggest the method can improve user performance.
  • General intent inference may require memory, and modeling users’ adaptation to the robot interface could improve training and inform convergence guarantees.
Loading 1802.01744v2…