Source-linked AI summary
Enabling Robots to Communicate their Objectives
Sandy H. Huang, David Held, Pieter Abbeel, Anca D. Dragan
TL;DR
The paper addresses how to help users anticipate a robot’s behavior in novel situations by rapidly teaching them its objective function. It models approximate human inference to select informative robot behaviors, and finds that a Euclidean-based approximate model, especially with strategy coverage, improves teaching while leaving human extrapolation incompletely modeled.
Problem
Users need an accurate qualitative understanding of a robot’s objective function to anticipate its behavior in novel situations, but familiarization through observation can be lengthy.
Method
The robot models how people infer objectives from observed behavior and selects informative examples using candidate approximate-inference models.
Results
A deterministic Euclidean-based approximate-inference model performed best at teaching real users, and a coverage-augmented version outperformed passive familiarization.
Takeaways & Limitations
Teaching improves when the robot accounts for approximate human inference and covers the full range of strategies it can adopt.
Takeaways & Limitations
The coverage results suggest that an IRL-only model is insufficient to capture how people extrapolate from observed robot behavior.
Abstract
from arXiv · showhide
The overarching goal of this work is to efficiently enable end-users to correctly anticipate a robot's behavior in novel situations. Since a robot's behavior is often a direct result of its underlying objective function, our insight is that end-users need to have an accurate mental model of this objective function in order to understand and predict what the robot will do. While people naturally develop such a mental model over time through observing the robot act, this familiarization process may be lengthy. Our approach reduces this time by having the robot model how people infer objectives from observed behavior, and then it selects those behaviors that are maximally informative. The problem of computing a posterior over objectives from observed behavior is known as Inverse Reinforcement Learning (IRL), and has been applied to robots learning human objectives. We consider the problem where the roles of human and robot are swapped. Our main contribution is to recognize that unlike robots, humans will not be exact in their IRL inference. We thus introduce two factors to define candidate approximate-inference models for human learning in this setting, and analyze them in a user study in the autonomous driving domain. We show that certain approximate-inference models lead to the robot generating example behaviors that better enable users to anticipate what it will do in novel situations. Our results also suggest, however, that additional research is needed in modeling how humans extrapolate from examples of robot behavior.
I. INTRODUCTION
The paper aims to help people anticipate robots in novel situations by teaching them the objective functions underlying robot behavior. It selects informative behaviors using models of human inference, finding benefits from approximate-inference and strategy-coverage models while leaving extrapolation modeling open.
- Motivation: Users can anticipate robot behavior in novel situations when they understand the robot’s qualitative trade-offs, even without explicitly specifying its objective function.The objective function captures task-relevant trade-offs such as safety versus efficiency.
- Motivation: Because many environments reveal only part of a robot’s trade-offs, the robot should purposefully demonstrate behaviors that distinguish among objectives.A nearby car can make a lane merge more informative than an otherwise identical empty-lane scenario.
- Approach: The teaching approach models how people infer objectives from observed optimal behavior and selects examples that increase the probability of inferring the correct objective.This reverses the usual IRL direction, where machines infer objectives from human behavior.
- Results: In an autonomous-driving user study, one approximate-inference model significantly outperformed exact inference, while other approximate models performed comparably to exact inference.The result supports accounting for human inference approximations but cautions that the approximation model matters.
- Results: Teaching performance correlated with covering the robot’s full strategy space, and coverage-augmented teaching using an approximate user model produced the best results.The car should demonstrate both cutting off another vehicle and braking to merge behind it.
II. ALGORITHMIC TEACHING OF OBJECTIVE FUNCTIONS
The framework treats robot behaviors as evidence about an underlying objective parameter and chooses environments whose resulting trajectories most inform a human’s belief. It models human updating through Bayesian inference over candidate objectives, including approximate inference.
- Framework: The framework models how people infer a robot’s objective from behavior and uses that model to generate informative examples.The objective is represented as a feature-weighted function, with parameters θ determining the robot’s optimal trajectory in each environment.
- Framework: The robot searches for environments whose observed optimal trajectories maximize the human’s posterior probability on the correct objective parameters.The human begins with a prior over candidate parameters and updates beliefs as the robot acts.
- Framework: Bayesian updating reduces the modeling problem to specifying how likely a trajectory is when a person assumes a candidate objective generated it.The framework assumes the person knows the task-relevant features and the environment affects observed behavior.
- Approximate inference: The framework allows approximate human inference by retaining candidate objectives whose predicted optimal behavior is sufficiently close to the observed trajectory.This deterministic effect is more conservative than eliminating every objective for which the observation is not globally optimal.
C. Exact-Inference IRL as a Special Case
Exact-inference IRL treats observed behavior as optimal under candidate objectives, eliminating objectives inconsistent with that behavior. The paper relaxes this assumption with approximate models that tolerate near-optimal or probabilistically compatible trajectories.
- Inverse Reinforcement Learning extracts an objective function by assuming observed behavior optimizes one of several candidate objectives.
- Exact-inference teaching eliminates every objective whose optimal behavior does not match the observed trajectory.
- With a uniform prior, the exact model assigns equal probability to candidate objectives for which all observed trajectories are optimal.
- Approximate-inference models relax the assumption that people can perfectly identify global optima for every candidate objective.
- Deterministic effects conservatively retain candidates whose optimal trajectories are sufficiently close, whereas probabilistic effects reduce but never eliminate candidate probabilities.
- The models compare reward-based, Euclidean trajectory-based, and strategy-based notions of distance between observed and candidate-optimal trajectories.
TAILGATING
Driving environments are organized into strategy classes, and the teaching algorithm greedily selects examples that increase the inferred probability of the correct objective. Its design relies on candidate sampling, early stopping, and coverage properties of deterministic inference.
- Driving environments cluster into four classes with two trajectory strategies per class, including merging into the right lane or driving forward.
- The strategy-based metric treats trajectories in the same strategy cluster as equivalent for inference.
- The algorithm greedily selects environments to maximize the model’s posterior probability of the correct objective.
- It samples candidate objectives uniformly, selects up to ten examples, and stops when another example cannot improve the target probability.
- Under deterministic inference with a uniform prior, greedy selection is near-optimal because the objective is non-decreasing and submodular.
- Each added trajectory can eliminate candidates, while its elimination effect is independent of when it appears in the sequence.
F. Hyperparameter Selection
The study selects hyperparameters for approximate human-inference models and evaluates them in an autonomous-driving domain with varied environments and trajectory strategies.
- Hyperparameter selection: Too-large τ leaves many candidate objectives plausible, whereas too-small τ makes a few environments overly informative; analogous reasoning applies to λ.These parameters control how strongly observed trajectories eliminate or reweight candidate objectives.
- Hyperparameter selection: τ and λ are selected from {10^-5, 10^-4, . . . , 10^4, 10^5} to increase the posterior by at least 0.1 while maximizing unique examples.The selection reflects the expectation that human inference is approximate and benefits from multiple examples.
- Example domain: The driving domain contains 21,216 highway environments with three lanes and one varying non-autonomous car, grouped into four classes with two trajectory strategies per class.The autonomous car starts in the middle lane with fixed initial velocity, while the other car’s location and velocity vary.
- Example domain: The merging, braking, and tailgating classes each contrast distinct driving strategies, such as merging ahead versus behind or passing versus following.Other environments are grouped into an additional class not included among those three.
- Vehicle and reward model: The model uses reward features including distance to the other car and squared deviation from initial speed, with normalized environment-independent features.The vehicle is represented by a bicycle model with steering-change and acceleration controls, and γ = 1.
IV. ANALYSIS OF APPROXIMATE INFERENCE MODELS WITH IDEAL USERS
The ideal-user analysis compares six approximate-inference models, exact inference, and random example sequences. Matching the assumed learner model generally performs best, while Euclidean approximations provide robust performance across learner types.
- Model comparison: Six approximate-inference models differ in their distance metric and whether they deterministically eliminate or probabilistically reweight candidate objectives.All judge candidate objectives using the distance between observed and candidate-optimal trajectories.
- Evaluation: Each sequence is generated greedily to maximize the assumed learner’s posterior probability of the correct objective, alongside exact-inference and random baselines.The evaluation uses seven ideal users whose learning exactly matches one of the candidate models.
- Results: For every ideal user, the sequence generated under that user’s own model performs best by construction.The result reflects optimization of each sequence for its assumed learner model.
- Results: All sequences perform equally well for exact IRL, whereas Euclidean-distance sequences are robust across different approximate user models.The result suggests exact IRL may not capture the example-dependent performance of real users.
- Results: The random sequence is uninformative for nearly all ideal users except exact IRL, demonstrating the utility of algorithmic teaching.Exact IRL can succeed with random examples because enough examples eliminate incorrect objectives.
V. USER STUDY
The user study tests whether approximate-inference models help real people learn an autonomous car’s driving style rather than only idealized learners.
- User study: Participants observe simulated examples of an autonomous car and must anticipate its trajectory when riding in new situations.The study evaluates approximate-inference models with real users.
A. Experiment Design
The experiment compares exact and approximate teaching strategies with human participants, measuring how accurately and confidently they identify the car’s behavior. Results favor example quality and selected approximate models over quantity alone.
- Experiment design: The experiment varies inference type, deterministic versus probabilistic effects, and reward-, Euclidean-, or strategy-based distance metrics in a 2-by-3 approximate-model design.Exact inference is included as a comparison condition.
- Experiment design: Participants view selected training environments sequentially and identify the trajectory matching the target driving style in six test environments.Each test presents four trajectories, and participants report confidence from 1 to 7.
- Measures: The confidence score is positive when the participant is correct and negative when incorrect, rewarding accurate confidence and penalizing overconfidence.The study also records correctness and confidence as separate dependent variables.
- Analysis: r = 0.03: total example count does not correlate with performance, suggesting example quality matters more than quantity.The number of examples was identified as a possible confound because conditions generate different sequence lengths.
- Approximate models: The best approximate model uses Euclidean distance with deterministic effects, while determinism alone has no significant effect.Probabilistic models perform slightly worse on average, with the largest difference for Euclidean distance.
- Approximate models: approx* significantly outperforms exact-inference IRL (Welch’s t-test p = 0.025), supporting approximate-inference modeling with the caveat that not every approximation helps.The figure also evaluates coverage-augmented sequences and human performance.
VI. UTILITY OF ALGORITHMIC TEACHING
The study compares algorithmic teaching with passive or random exposure and finds that selecting informative behaviors can improve users’ anticipation of robot behavior. Performance depends more on example quality and strategy coverage than on the number of examples alone.
- Baseline Condition: Algorithmic teaching with the approximate-inference model outperformed the random baseline, although the difference was not statistically significant (Welch’s t-test p = 0.23).The random condition used eight environments to avoid disadvantaging it through fewer examples.
- Coverage: Users performed better on test strategies represented in training, while seeing only one side of an opposing strategy pair reduced performance.This pattern motivated measuring whether training examples covered contrasting strategy clusters.
- Coverage: The number of helpful environments strongly correlated with users’ confidence scores (Pearson r = 0.83, p = 1.4 × 10−11).Helpful environments counted positively when a strategy was shown and negatively when only its counterpart was shown.
- Coverage: The authors therefore introduced augmented algorithms that explicitly ensure coverage of possible strategies.Coverage was added to the teaching objective based on the observed relationship between strategy coverage and user performance.
B. Coverage-Augmented Algorithmic Teaching
Coverage augmentation extends approximate-inference teaching by selecting additional examples across trajectory strategies. The combined approach yields the strongest teaching performance and a significant advantage over random or exact-inference teaching, while the authors identify limitations in modeling human extrapolation and communication channels.
- B. Coverage-Augmented Algorithmic Teaching: Coverage augmentation adds strategy coverage to trajectory optimization after additional examples no longer substantially increase the inferred probability of the target objective.Extra examples are then selected using the approximate-inference model while covering distinct strategy clusters.
- B. Coverage-Augmented Algorithmic Teaching: The coverage study manipulated both strategy coverage and whether examples were generated with a user model or sampled uniformly.The experiment compared coverage-augmented and non-augmented conditions across approximate-model and random generation.
- B. Coverage-Augmented Algorithmic Teaching: Coverage had a marginal positive effect on confidence scores (F(1, 107) = 1.82, p = .07), with no interaction between coverage and model.The result suggests coverage helped regardless of whether a user model generated the examples.
- B. Coverage-Augmented Algorithmic Teaching: Coverage-approx* performed best and significantly outperformed the random baseline (Welch’s t-test p = 0.049), whereas coverage-random did not (p = 0.159).The comparison supports contributions from both coverage and the approximate-inference model.
- B. Coverage-Augmented Algorithmic Teaching: Coverage alone helped but was insufficient to outperform the baseline; the largest improvement occurred when coverage was combined with the approximate-inference model.The authors describe this combined improvement as significant while noting the qualification about multiple hypotheses.
- VII. DISCUSSION: The paper reports that physical behavior was used as the communication channel, while future work could add visualizations or language-based explanations.This identifies a scope boundary for the communication approach rather than a result of the coverage manipulation.