PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Reward Functions by Integrating Human Demonstrations and Preferences

Malayandi Palan, Nicholas C. Landolfi, Gleb Shevchuk, Dorsa Sadigh

arXiv:1906.08928v1cs.ROcs.AI

TL;DR

Learning robot reward functions is difficult because IRL depends on high-quality demonstrations, whereas preference-based learning is inefficient in high-dimensional control spaces. DemPref combines both feedback modes by using demonstrations to narrow the reward-function prior and improve active query generation. The method is more efficient than standard active preference learning, and users preferred it over IRL in a physical-robot study, although query-generation time affected usability.

  • Problem

    IRL struggles when high-quality demonstrations are difficult to obtain, while preference-based learning is inefficient in complex, high-dimensional robotics settings.

  • Method

    DemPref uses demonstrations to learn a prior over reward functions and to ground active preference-query generation, then combines both feedback modes.

  • Results

    Users rated DemPref more successful than IRL and preferred using it, while numerical experiments found it more efficient than standard preference-based learning.

  • Takeaways & Limitations

    Combining demonstrations with preferences can improve reward-learning efficiency while avoiding exclusive reliance on potentially low-quality demonstrations.

  • Takeaways & Limitations

    Users reported that generating each DemPref query took approximately 45 seconds and that some preference queries were difficult to interpret.

Abstract

from arXiv · show

Our goal is to accurately and efficiently learn reward functions for autonomous robots. Current approaches to this problem include inverse reinforcement learning (IRL), which uses expert demonstrations, and preference-based learning, which iteratively queries the user for her preferences between trajectories. In robotics however, IRL often struggles because it is difficult to get high-quality demonstrations; conversely, preference-based learning is very inefficient since it attempts to learn a continuous, high-dimensional function from binary feedback. We propose a new framework for reward learning, DemPref, that uses both demonstrations and preference queries to learn a reward function. Specifically, we (1) use the demonstrations to learn a coarse prior over the space of reward functions, to reduce the effective size of the space from which queries are generated; and (2) use the demonstrations to ground the (active) query generation process, to improve the quality of the generated queries. Our method alleviates the efficiency issues faced by standard preference-based learning methods and does not exclusively depend on (possibly low-quality) demonstrations. In numerical experiments, we find that DemPref is significantly more efficient than a standard active preference-based learning method. In a user study, we compare our method to a standard IRL method; we find that users rated the robot trained with DemPref as being more successful at learning their desired behavior, and preferred to use the DemPref system (over IRL) to train the robot.

I. INTRODUCTION

DemPref combines demonstrations and preference queries to learn personalized reward functions while addressing the limitations of relying on either feedback mode alone. Demonstrations provide structure for more efficient and better-grounded preference learning, and experiments support benefits over standard preference learning and IRL.

  • Demonstrations provide a prior over reward functions and improve the quality of actively generated preference queries.The prior reduces the possible reward-function space, while demonstrations ground query generation.
  • DemPref is designed to remain more robust to low-quality demonstrations than IRL and more efficient than standard preference-based learning.It does not rely exclusively on demonstrations and searches a smaller, demonstration-informed reward-function space.
  • DemPref unifies demonstrations and preference queries to learn personalized human reward functions.
  • In simulation, DemPref learns reward functions more efficiently than a standard preference-based learning algorithm.
  • In a physical-robot user study, users rated DemPref more successful than IRL and preferred using it to train robots.

II. BACKGROUND ON REWARD LEARNING

Reward-learning approaches trade off the quality and informativeness of demonstrations and preference feedback. IRL depends on difficult-to-obtain demonstrations, while preference-based methods face computational challenges in complex, high-dimensional control spaces.

  • Inverse Reinforcement Learning: IRL learns reward functions from expert demonstrations, but robot teleoperation is difficult because interfaces are noisy and often have fewer degrees of freedom than the robot.
  • Inverse Reinforcement Learning: IRL works best with diverse training environments, which can be costly or time-consuming to create for physical robotics tasks.
  • Inverse Reinforcement Learning: Demonstrations may not accurately express desired behavior because a person’s preferences can differ from their own actions.
  • Preference-Based Learning: Preference queries can better reflect human preferences and are easier to provide than full demonstrations, but each query is less informative.
  • Preference-Based Learning: Active preference-based methods seek informative queries but often require difficult optimization over complex, high-dimensional control spaces.

III. APPROACH OVERVIEW AND MODELLING DETAILS

DemPref combines demonstrations and preference queries to learn reward functions efficiently, using demonstrations to form an informative prior and guide active querying. The model assumes a linear reward over known features and uses a two-stage process to narrow and then refine the reward-function search.

  • Approach Overview: DemPref unifies demonstrations and preference queries, using their complementary strengths to learn reward functions accurately and efficiently.Preferences are described as more accurate but less informative than demonstrations.
  • Approach Overview: Demonstrations first construct a probabilistic prior over reward functions, after which active preference queries refine that prior toward the true reward function.The framework avoids using preferences initially and demonstrations at later stages to reduce the impact of their respective weaknesses.
  • Modelling Details: The robot is modeled as a fully observable dynamical system whose finite state-action trajectories are evaluated by the human’s reward function.The dynamics evolve through the system function f, and the goal is to learn the reward governing the human’s preferred behavior.
  • Modelling Details: The reward function is modeled as a linear combination of known state-control features, with the weight vector constrained by ||w||2 ≤1.Learning the human reward function therefore reduces to learning its weight vector.

IV. DEMPREF: A UNIFIED FRAMEWORK FOR REWARD LEARNING

DemPref assembles Bayesian IRL and active preference-based learning into a unified framework. It uses demonstrations to initialize a reward distribution, then generates informative trajectory comparisons and updates the distribution from noisy human preferences.

  • Unified Framework: DemPref combines Bayesian IRL initialization with active preference-based learning in one coherent reward-learning framework.The framework is presented as an organized integration of two previously distinct approaches.
  • Stage 1: Initializing a Reward Distribution from Demonstrations: Demonstrations are used to derive a Bayesian distribution over reward weights, assuming conditionally independent demonstrations and noisily reward-maximizing human behavior.The initial weight prior is uniform on the unit ball, and demonstration noise is modeled with a Boltzmann distribution.
  • Stage 2: Learning the Reward Function with Preference Queries: Active preference learning initializes from the demonstration-based prior, presents two generated trajectories, and updates the weight distribution after the human selects one.The update uses a model in which human choices are noisily optimal with respect to the true trajectory rewards.
  • Actively Generating Queries: Queries are generated by maximizing the minimum volume removed from the weight distribution across possible human preferences.This objective seeks trajectory pairs that are maximally informative in expectation, subject to feasibility constraints.

V. ITERATED CORRECTION: GENERATING BETTER QUERIES IN DEMPREF

Iterated Correction retains a human demonstration in preference queries, ensuring each query includes a trajectory reasonably reflective of the human’s reward. The method extends updates to rankings and supports multiple demonstrations through a replacement buffer.

  • Motivation: The method addresses non-convex query optimization, which can produce locally optimal but uninformative trajectories in complex control spaces.Such queries make human feedback less useful and can slow convergence.
  • Iterated Correction: Iterated Correction retains the demonstration during preference learning and replaces it when a generated trajectory is preferred.This grounds successive queries in an increasingly improved trajectory.
  • Ranking Updates: Ranking feedback preserves relative preferences among all queried trajectories instead of using only the human’s single favorite.The framework models rankings with a Plackett–Luce choice model.
  • Query Generation: Queries can present the stored trajectory alongside newly generated trajectories, after which the reward distribution is updated using the human’s choice.The stored trajectory is then replaced by the human’s most preferred trajectory.
  • Multiple Demonstrations: With multiple demonstrations, the method stores them in a buffer, samples one for each query, and replaces it with the human’s preferred trajectory after each iteration.This extends the single-demonstration procedure without changing its basic structure.

VI. SIMULATION EXPERIMENTS

The simulation study evaluates DemPref across Driver, Lunar Lander, and Fetch Reach domains using convergence toward a true reward-weight vector. Convergence is measured by expected cosine similarity between the learned distribution and the true vector.

  • Domains: The experiments cover Driver, Lunar Lander, and simulated and physical Fetch Reach settings.The domains represent driving, landing, and robotic reaching tasks with domain-specific reward features.
  • Evaluation: Convergence is measured by the expected cosine similarity between the true weight vector and the learned distribution over weight vectors.A value of m = 1 indicates perfect convergence to the true weight vector.
  • Demonstrations: Simulation demonstrations are generated with model predictive control by optimizing the true reward over a trajectory’s control sequence.The resulting demonstrations are used where an experiment requires them.

A. The Impact of Initializing with Demonstrations.

Initializing preference-based learning with demonstrations substantially accelerates convergence across the three simulated domains. However, additional low-quality demonstrations can concentrate the prior away from the true reward and slow learning.

  • Results and Analysis: Initializing with one demonstration significantly improves convergence on Driver, Lunar Lander, and Fetch Reach.The experiment varies initialization among zero, one, and three demonstrations over 25 queries.
  • Results and Analysis: In 10 queries with one demonstration, the standard method requires 30 queries on Driver, 35 on Lander, and 20 on Fetch Reach to attain the same m value.The comparison shows faster convergence from demonstration initialization across all three environments.
  • Results and Analysis: Three demonstrations improve convergence over one on Driver and Lunar Lander, but hurt performance on Fetch Reach.On Driver, the improvement occurs only early, before 10 preferences.
  • Results and Analysis: Lower-quality demonstrations can concentrate the prior farther from the true reward, causing preference-based learning to slow down.This explains why adding demonstrations does not uniformly improve convergence.
  • Results and Analysis: A single demonstration provides reliable convergence improvements regardless of domain complexity and is used in the remaining simulations and user study.The authors therefore use one demonstration as the practical default.

B. The Impact of Using Rankings instead of Preferences.

The ranking update outperforms the preference update for both three- and five-option queries. It can match the preference update’s performance with fewer options, reducing the queries and computation needed.

  • Results and Analysis: The ranking update significantly outperforms the preference update for both three and five presented options.The experiment compares the two updates without demonstrations or Iterated Correction.
  • Results and Analysis: With three options, the ranking update performs as well as the preference update with five options.This indicates that ranking feedback extracts more information per query in the tested settings.
  • Results and Analysis: Ranking feedback reduces the number of queries and associated computation needed by maximizing information extracted from each human response.The paper reports this as the explanation for the observed efficiency improvement.

C. The Impact of Iterated Correction.

The Iterated Correction (IC) method improves DemPref’s convergence, especially when the initial demonstration is low quality. Its benefit is significant in most tested settings and can substantially reduce the preference queries needed.

  • The experiment varies IC usage and demonstration quality across Driver and Lunar Lander using one demonstration and 25 preference queries.Low- and high-quality demonstrations were selected as the worst and best, respectively, from 100 generated demonstrations using the convergence metric.
  • The ranking-update algorithm significantly outperforms the standard preference-update algorithm for every nopt value on both domains.The ranking update with three options performs as well as the preference update with five options.
  • IC improves DemPref convergence significantly in both Driver cases and with the lower-quality Lunar Lander demonstration.In the remaining Lunar Lander case, IC still improves convergence, but the result is not statistically significant.
  • The convergence benefit of IC is greater with low-quality initial demonstrations in both domains.Matching IC’s performance after 10 queries requires ≫25 queries without IC on Driver and ∼20 on Lander for low-quality demonstrations.

VII. USER STUDY

The physical-robot user study compares IRL and DemPref under different training procedures and evaluates users’ judgments of the resulting robot behavior and systems. Users significantly favored DemPref for task success and future use, although they did not find it easier to use.

  • The study used a physical Fetch robot and tested generalization with different training and testing domain variants.Two test domains were counter-balanced to reduce dependence on a specific testing domain.
  • Participants compared Bayesian IRL with five demonstrations against DemPref with one demonstration and 15 preference queries.The information amounts were chosen so a simulated agent achieved similar convergence to the true reward under both systems.
  • The study recruited 15 participants, including six with prior robotics experience and none previously exposed to the system.
  • Users significantly preferred DemPref-trained robots for task accomplishment and preferred DemPref for future robot training over IRL.Both comparison results were significant with p = 0.0201, providing evidence for H1 and H2.
  • Users struggled to teleoperate the robot, and IRL sometimes overfit collision-avoidance preferences from training to test domains.In some test domains, the IRL robot did not attempt to reach the goal because avoiding the obstacle had greater learned value than reaching it.
  • DemPref was not rated easier to use than IRL, partly because preference queries took approximately 45 seconds to generate.Users explicitly identified query-generation time as a negative aspect of the DemPref experience.

APPENDIX

The appendix supplies supplementary videos, code, and instructions supporting the DemPref experiments. It documents demonstrations, preference queries, and experiment materials rather than presenting additional results.

  • Supplementary videos show keyboard teleoperation and DemPref preference queries generated for the robot.The preference-query videos illustrate choices between trajectories moving toward or away from the goal.
  • The project repository provides the DemPref implementation and associated experiment materials.
  • The appendix gives Conda-based setup instructions for running the DemPref code on an OSX system.It specifies creating and activating the dempref environment from the requirements file.

C. Additional Experimental Details

The additional experimental details document shared computational settings, simulation reward weights, and procedures for reproducing the experiments. They emphasize fixed hyperparameters and predefined domain-specific true reward functions.

  • The experimental system used eight cores and 4 GB of memory.
  • The experiments use constant hyperparameters across experiments, including β_D = 0.1, β_R = 5, and 50,000 Monte Carlo samples.Only minimal hyperparameter tuning was performed.
  • Simulation experiments use fixed true reward-function weight vectors for Driver, Lunar Lander, and Fetch Reach.The appendix lists one weight vector for each domain and states that no tuning was performed.
  • The appendix points to the provided code for experimental details not documented elsewhere.
Loading 1906.08928v1…