PaperlayerView exact revision on arXiv

Source-linked AI summary

Multimodal Probabilistic Model-Based Planning for Human-Robot Interaction

Edward Schmerling, Karen Leung, Wolf Vollprecht, Marco Pavone

arXiv:1710.09483v1cs.ROcs.LG

TL;DR

The paper addresses how robots can plan interactions when human behavior has multiple plausible futures and depends on the robot’s actions. It learns multimodal, history- and action-conditioned human response distributions from human-human data, then samples those responses to construct robot policies in traffic weaving. Human-in-the-loop simulation demonstrates the resulting real-time interaction policy.

  • Problem

    Human-robot interaction requires modeling multiple distinct human responses whose likelihoods depend on interaction history and candidate robot actions.

  • Method

    The paper learns a sampleable multimodal human-action model from human-human interactions using a CVAE and evaluates candidate robot action sequences through sampled responses.

  • Results

    The learned model supports real-time human-in-the-loop traffic-weaving policy validation, including awareness of multiple human responses and their action-dependent interaction modes.

  • Takeaways & Limitations

    The framework provides data-driven probabilistic reasoning over short-horizon human responses without assuming human motivations or a game-theoretic hierarchy.

  • Takeaways & Limitations

    The framework is designed for relatively short horizons and nominal operating conditions, remaining blind to situations not represented in its data.

Abstract

from arXiv · show

This paper presents a method for constructing human-robot interaction policies in settings where multimodality, i.e., the possibility of multiple highly distinct futures, plays a critical role in decision making. We are motivated in this work by the example of traffic weaving, e.g., at highway on-ramps/off-ramps, where entering and exiting cars must swap lanes in a short distance---a challenging negotiation even for experienced drivers due to the inherent multimodal uncertainty of who will pass whom. Our approach is to learn multimodal probability distributions over future human actions from a dataset of human-human exemplars and perform real-time robot policy construction in the resulting environment model through massively parallel sampling of human responses to candidate robot action sequences. Direct learning of these distributions is made possible by recent advances in the theory of conditional variational autoencoders (CVAEs), whereby we learn action distributions simultaneously conditioned on the present interaction history, as well as candidate future robot actions in order to take into account response dynamics. We demonstrate the efficacy of this approach with a human-in-the-loop simulation of a traffic weaving scenario.

I. INTRODUCTION

The paper frames multimodal, history-dependent human behavior as a central HRI challenge and proposes learning response distributions for model-based robot policy construction. Its framework uses CVAE-based generative modeling and massively parallel candidate-sequence evaluation for real-time interaction planning.

  • Motivation: Human behavior can follow multiple distinct action modes even under the same broad intent, with responses depending on evolving surroundings.Traffic weaving illustrates this ambiguity: a driver may accelerate to pass or brake to yield.
  • Validation: The framework learns human action distributions from human-human interaction data and validates the resulting robot policy in human-in-the-loop traffic weaving.The paper positions this as a data-driven alternative to assumptions about human preferences or game-theoretic reasoning.
  • Problem formulation: The framework conditions multimodal human-action predictions on interaction history and candidate future robot actions.This explicitly represents response dynamics rather than predicting human behavior independently of the robot’s planned actions.
  • Method: The paper uses a CVAE-based generative model whose mixture components represent different driving behaviors over a prediction horizon.Recurrent hidden states preserve temporal information while latent modes capture alternative behaviors.
  • Policy construction: The approach evaluates candidate robot action sequences by sampling human futures, enabling nearly 100 000 human futures to be simulated every 0.3 seconds.This massively parallel process supports real-time MPC-style policy construction in multimodal interactions.

II. PROBLEM FORMULATION

The interaction model represents a fully observed human-robot system in which human actions are stochastic responses to history and the robot’s next action. These dynamics define the random future human-action sequence used for planning over a finite horizon.

  • A. Interaction Dynamics: The human and robot are modeled with discrete-time state-space dynamics and joint state and control variables.The joint state combines human and robot states, while their controls specify the actions selected at each time step.
  • A. Interaction Dynamics: Interactions terminate when the joint state first reaches a terminal set T at final time T.The terminal condition formalizes when the modeled interaction ends.
  • A. Interaction Dynamics: At each nonterminal step, the human’s next action is drawn from a distribution conditioned on the full joint interaction history and the robot’s next action.This makes the human response explicitly dependent on both prior interaction evolution and the candidate robot control.
  • A. Interaction Dynamics: The human response is treated as a random variable with a probability density over possible next actions.The formulation distinguishes sampled action values from the corresponding random variable and its density.
  • A. Interaction Dynamics: Iteratively propagating the dynamics and sampling human responses lets the robot reason about a future human-action sequence over horizon N.The model supports forward reasoning over responses to a planned sequence of robot actions.
  • A. Interaction Dynamics: The formulation assumes both agents fully observe all past states and actions.This observability assumption supplies the history used to condition future human responses.

B. Robot Goal

The robot seeks a limited-lookahead policy that minimizes expected cost over a fixed horizon under the learned interaction dynamics. It repeatedly optimizes the horizon but executes only the first action in an MPC fashion.

  • B. Robot Goal: The robot policy selects a length-N action sequence as a function of the observed interaction history.The policy is designed as a limited-lookahead approximation to the expected-cost minimization problem.
  • B. Robot Goal: The objective sums discounted running costs and applies a terminal cost when the interaction ends before the planning horizon.The running cost depends on the current state, action, and next state, while the terminal cost handles early termination.
  • B. Robot Goal: The practical implementation uses N = 15 steps with a 0.1s interval and repeatedly resolves the optimization while executing only the first action.This receding-horizon procedure implements the policy in MPC fashion.
  • B. Robot Goal: The robot action at the current step is computed with nonzero computation time accounted for by treating it as already in progress.The optimization therefore selects the next action while the current action has been initiated previously.

C. Traffic Weaving Scenario

The traffic-weaving scenario models pairwise human-robot lane swapping with explicit vehicle dynamics, collision avoidance, and short-horizon urgency. The human model accounts for stochastic responses to candidate robot actions and interaction history.

  • The scenario considers two agents signaling intent to swap lanes on a two-lane highway.
  • The human-controlled car uses double-integrator longitudinal and lateral dynamics, while the robot uses a triple-integrator lateral model for continuous steering.
  • The robot incurs a large collision penalty and is encouraged to switch lanes before reaching the terminal set as urgency increases near the cutoff.
  • The human distribution conditions on interaction history and candidate robot future actions, allowing response dynamics and prior behavioral factors to be represented.
  • The phenomenological model requires significantly more data to fit, and its fixed-horizon robot cost objective is acknowledged as ad hoc.

III. HUMAN AGENT MODELING

The human agent model learns a conditional distribution over future human actions using a recurrent CVAE conditioned on interaction history and candidate robot actions. A latent variable captures structure in the interaction while neural networks are trained on observed interaction data.

  • The model predicts future human actions from joint interaction history and candidate future robot actions.
  • The CVAE introduces a latent variable z and represents the human-action density as a mixture over latent-conditioned distributions.
  • Recurrent encoder and decoder components manage the time-series structure of the interaction.
  • The conditional distributions are parameterized by neural networks fit to maximize likelihood on a dataset of observed interactions.

A. CVAE Architecture

The CVAE uses discrete latent variables for high-level human behavior modes and an autoregressive recurrent decoder for within-mode variation. Its mixture structure and sequence modeling jointly represent multimodal future action distributions.

  • A discrete latent space factors into Nz independent categorical elements, producing a mixture over latent behavior configurations.
  • Latent mixture components correspond to distinct human response modes such as accelerating versus decelerating or driving straight versus turning.
  • The autoregressive RNN decoder models additional multimodality within each high-level behavior by conditioning future actions on previously drawn actions.
  • The model optimizes the evidence-based lower bound using importance sampling over latent variables and trains with occasional predicted decoder inputs at a 10% rate.
  • Each decoder step uses a Gaussian mixture output layer to represent distributions over the next human action and variation within a latent behavior.

IV. ROBOT POLICY CONSTRUCTION

Robot policy construction exhaustively evaluates discrete robot action sequences by sampling human futures from the learned model. Massively parallel neural-network execution makes broad short-horizon evaluation practical.

  • The robot action space is discretized into eight actions applied across five 3-step windows over N = 15 future time steps.
  • The method evaluates candidate robot sequences by sampling human response futures and estimating their expected costs.
  • GPU-parallel neural-network computation enables evaluating all or a significant fraction of short-horizon action sequences.
  • Policy construction uses a two-stage process that first evaluates many sequences with few human samples and then reevaluates the most promising candidates.

V. TRAFFIC WEAVING CASE STUDY

The case study’s dataset and source code are publicly available, including network architecture details and hyperparameters.

  • The human-human traffic weaving dataset and source code for all reported results are available online.The release includes network architecture details and hyperparameters.

A. Data Collection

The study collected human-human traffic-weaving interactions in a driving simulator, with trajectories and scenario conditions designed to capture lane-swapping behavior.

  • 1105 human-human driving interaction trials were recorded from 19 pairs in a simulator.Drivers swapped lanes without verbal communication within 135 meters of straight road.
  • Each trial produced two learnable human interaction trajectories, yielding roughly 35 thousand exemplars when conditioning on history and partner future.The interactive scenario motivated collecting a targeted dataset rather than relying on existing datasets.
  • The experiment initialized one car at 29m/s (65mph), with the other car assigned a randomized speed difference.These randomized initial conditions formed the traffic-weaving scenarios.
  • The learned driver model predicts over a 1.5s horizon using interaction history and the other driver’s next 1.5s trajectory as a candidate plan.The CVAE mixture represents different times at which acceleration cessation may occur.

C. Robot Policy Construction

Robot policy construction evaluates discrete candidate futures with sampled human responses and cost-based scoring, using parallel computation for real-time planning.

  • The robot plans over five 0.3s action windows, choosing among four longitudinal and two lateral actions in each window.The prediction horizon is 1.5s and replanning occurs every 0.3s.
  • 4096 candidate robot action sequences are generated because each of four future windows has eight possible actions.The first action window is fixed from the previous planning iteration, leaving 8^4 sequences.
  • The running cost combines collision avoidance, control effort, lane-change incentive, and longitudinal disambiguation incentive.The collision term penalizes near-collision states, while the other terms encode effort, target-lane progress, and separating motion.
  • Sixteen human-response samples are averaged for every candidate, then the top 32 sequences receive 1024 sampled trajectories each.This two-stage process estimates expected cost more confidently for the strongest candidates.
  • The prediction-model sampling and scoring process takes approximately 0.25s when parallelized.This computation supports the targeted replanning rate.
  • Figure 4 compares 50 sampled human-action predictions across models using validation negative log-likelihood, where lower is better.The basic LSTM baseline has Eval NLL = -9.70 and is not pictured.

D. Human-in-the-Loop Example Trials

Human-in-the-loop trials integrated the learned policy with the simulator and illustrated how expected-cost reasoning adapts to different human responses.

  • The robot waits when that minimizes expected cost, despite modeling multiple possible human actions and ways to elicit interaction modes.The policy is aware of possible responses but selects the action sequence with the lowest expected cost.
  • When the human continues accelerating during a proactive lane-cutting attempt, the robot changes its behavior and brakes.In another trial, the robot nudges toward the lane divider when it expects the human to yield.
  • The framework learns sampleable human responses from human-human data and evaluates robot action sequences through finite-horizon MPC.It does not assume human motivations or game-theoretic interaction hierarchies.
Loading 1710.09483v1…