Learning Reward Functions from Diverse Sources of Human Feedback: Optimally Integrating Demonstrations and Preferences

Erdem Bıyık; Dylan P. Losey; Malayandi Palan; Nicholas C. Landolfi; Gleb Shevchuk; Dorsa Sadigh

Source-linked AI summary

Learning Reward Functions from Diverse Sources of Human Feedback: Optimally Integrating Demonstrations and Preferences

Erdem Bıyık, Dylan P. Losey, Malayandi Palan, Nicholas C. Landolfi, Gleb Shevchuk, Dorsa Sadigh

arXiv:2006.14091v2Jun 24, 2020cs.ROcs.AIcs.LG

TL;DR

The paper addresses how robots can learn human reward functions when demonstrations and preferences provide complementary but different information. It introduces a framework that initializes reward beliefs from demonstrations, refines them with active preference queries, and accounts for human answerability. Simulations and Fetch-robot user studies report that integrating the sources improves understanding of human goals over using either source alone.

Problem
Robots need to learn human reward functions from limited interaction data, but demonstrations and preferences provide different forms of information and prior work often treats them separately.
Method
DemPref initializes a high-level reward belief from demonstrations, actively refines it with preference queries, and selects data collection using information gain while accounting for human answerability.
Results
Combining demonstrations and preferences leads to a better understanding of human goals than relying on either source alone, across simulations and Fetch-robot user studies.
Takeaways & Limitations
Multiple human-feedback sources can be synthesized so robots learn efficiently when access to human data is limited.
Takeaways & Limitations
The volume-removal query method can select uninformative queries, including queries whose options are identical and provide no reward information.

Abstract

from arXiv · show

Reward functions are a common way to specify the objective of a robot. As designing reward functions can be extremely challenging, a more promising approach is to directly learn reward functions from human teachers. Importantly, data from human teachers can be collected either passively or actively in a variety of forms: passive data sources include demonstrations, (e.g., kinesthetic guidance), whereas preferences (e.g., comparative rankings) are actively elicited. Prior research has independently applied reward learning to these different data sources. However, there exist many domains where multiple sources are complementary and expressive. Motivated by this general problem, we present a framework to integrate multiple sources of information, which are either passively or actively collected from human users. In particular, we present an algorithm that first utilizes user demonstrations to initialize a belief about the reward function, and then actively probes the user with preference queries to zero-in on their true reward. This algorithm not only enables us combine multiple data sources, but it also informs the robot when it should leverage each type of information. Further, our approach accounts for the human's ability to provide data: yielding user-friendly preference queries which are also theoretically optimal. Our extensive simulated experiments and user studies on a Fetch mobile manipulator demonstrate the superiority and the usability of our integrated framework.

Introduction

The paper proposes integrating passive demonstrations with active preference queries to learn human reward functions efficiently while accounting for the human’s ability to provide data. Demonstrations provide broad behavioral information, while preferences refine uncertain, fine-grained aspects of the reward.

Demonstrations alone can fail to convey a user’s intent because humans cannot provide examples for every situation and robots may generalize poorly from limited demonstrations.The paper motivates preferences as an additional way to clarify behavior beyond demonstrated situations.
Demonstrations provide rich information about motion style and timing, whereas preferences provide explicit information about specific reward aspects.The two sources therefore offer complementary information about the human’s reward function.
The framework determines when to use each source and proves that passive data should precede active preference queries for efficient information gathering.It also introduces an optimal stopping condition that ends data collection when expected utility no longer outweighs its cost.
Combining demonstrations and preferences yields a better understanding of human goals than relying on either source alone.The framework uses demonstrations to initialize a high-level belief and preferences to iteratively reduce uncertainty.
The approach accounts for human answerability by replacing problematic volume-removal queries with information-gain queries that are informative and easier to answer.The proposed alternative has the same computational complexity as volume removal while minimizing uncertainty about the human’s answer.
The framework is evaluated in multiple simulations and two user studies using a 7-DoF Fetch robot arm.The paper compares DemPref with methods that use only a single source of human data.

Problem Formulation

The paper formalizes learning a human’s reward function from demonstrations and preference queries, then motivates DemPref’s ordering of passive demonstrations followed by active preferences. Demonstrations provide an informative but imprecise prior, while preferences refine uncertainty about specific reward aspects.

Problem Formulation: DemPref models the robot as a deterministic, fully observable Markov Decision Process with states, actions, dynamics, rewards, and a finite horizon.A trajectory is a finite sequence of state-action pairs, equivalently represented by an initial state and action sequence in the deterministic setting.
Problem Formulation: The human’s reward is represented as a linear combination of given features, so the robot learns the feature weights ω under the constraint ∥ω∥2 ≤1.Trajectory reward depends on cumulative feature counts, and the weights determine how the human values those features.
Problem Formulation: Demonstrations are passively collected trajectories provided through mechanisms such as kinesthetic teaching, teleoperation, or virtual reality.The demonstration dataset contains L human trajectories that convey information about the reward weights.
Problem Formulation: A preference query presents K robot trajectories, after which the human selects the trajectory that best matches their preferences.The robot can actively generate and play different trajectories while asking the human to indicate a favorite.
Problem Formulation: The objective is to learn the human’s reward parameters accurately and efficiently using the smallest combination of demonstrations and preference queries.The framework explicitly optimizes when to request demonstrations and when to request preferences.
DemPref: Learning Rewards from Demonstrations and Preferences: DemPref begins with high-level demonstrations and then shifts to fine-grained preference queries because demonstrations are informative but noisy, whereas preferences target ambiguous reward aspects.The paper states that this ordering is theoretically the most efficient and uses the approach to integrate both sources.
Initializing a Belief from Offline Demonstrations: Offline demonstrations initialize an informative but imprecise belief b0(ω)=P(ω|D) over the human’s reward weights.The belief is formed from passive demonstrations using a probabilistic model of how likely each demonstrated trajectory is under a candidate reward.
Initializing a Belief from Offline Demonstrations: Because passive demonstrations cannot directly investigate uncertain reward aspects, DemPref uses proactive questions next to obtain a fine-grained posterior.The demonstrations provide a high-level prior, while subsequent preference queries remove remaining uncertainty.

Updating the Belief with Proactive Queries

DemPref updates a reward-function belief initialized from demonstrations by actively selecting preference queries and incorporating human responses. The framework emphasizes information gain while accounting for query difficulty, revealing that volume removal can select uninformative or confusing questions.

Updating the Belief with Proactive Queries: DemPref updates its belief over reward weights after each preference response, starting from a prior initialized with human demonstrations.The human-response likelihood is modeled probabilistically, and responses are assumed conditionally independent given the query and reward weights.
Choosing Queries: Information gain balances expected information from a correct answer against the human’s ability to answer confidently, with an optimal stopping condition for costly data collection.The framework aims to minimize the number of questions while retaining accurate, fine-grained information about the reward.
Choosing Queries: The robot greedily selects each query to improve the next posterior because finding the optimal adaptive query sequence is generally NP-hard.This strategy considers only the immediate posterior update at each iteration.
Choosing Queries with Volume Removal: Volume removal can select difficult queries because it prioritizes options that are equally likely, even when they are uninformative about the reward.Such queries may contain identical options or nearly indistinguishable trajectories, making them difficult for humans to answer.
Choosing Queries with Volume Removal: In the two-option example, a query with equally good choices is hard to answer and provides no useful reward information, whereas a query with one matching option is easy for the human.Volume removal treats both query types as global solutions despite their different informativeness and difficulty.

Choosing Queries with Information Gain

The framework selects preference queries by maximizing information gain while accounting for both robot uncertainty and human answerability. Demonstrations are collected first because they initialize a belief that makes subsequent active queries more informative, with at least as much information as reversing the order.

Information gain favors queries where the robot is uncertain about the answer but the human can respond confidently, unlike volume removal.The objective incorporates both robot uncertainty over the response and human uncertainty when answering.
Information gain avoids the previous volume-removal failures while retaining equivalent computational complexity order.The comparison is made against the volume-removal objective.
Trivial queries are globally minimized under information gain, avoiding questions that provide no insight into the reward.Volume removal instead treats such uninformative queries as global solutions.
The approach ranks the easier query QB above QA because both are similarly informative to the robot, but QA is harder for the human to answer.Volume removal values QA and QB equally, whereas information gain accounts for the human’s uncertainty.
Demonstrations should precede preference queries because they provide high-level information that supports more incisive active questioning.Theorem 2 states that this order yields at least as much information as asking queries before collecting demonstrations.

Useful Tools & Extensions

The framework extends information-gain querying with user-aware stopping and implementation tools. It stops when no query’s information value covers its human cost, while several auxiliary tools can improve efficiency or applicability.

Optimal Stopping: Query costs can represent answering time, repeated similar questions, or poor interpretability, biasing selection toward informative low-cost questions.The cost is subtracted from the information-gain objective.
Optimal Stopping: The robot should stop asking questions when the best information-gain-minus-cost value is negative.Theorem 3 gives this condition as necessary and sufficient for stopping under information gain.
Optimal Stopping: The stopping result is specific to information gain and is not straightforward to extend to volume removal.
Other Potential Extensions: Batch optimization, iterated correction, and dynamically changing reward functions are potential extensions that are agnostic to volume-removal details.
Algorithm: DemPref’s algorithm collects demonstrations, initializes the reward belief, generates preference queries, and updates the belief after each response.The pseudocode includes a stopping check before eliciting the human’s answer.

Algorithm

DemPref combines offline demonstrations with active preference learning to infer human reward functions while modeling human choice and evaluating convergence across simulated and physical robot domains. Its experiments test initialization, query strategy, ordering, and stopping.

Algorithm: DemPref synthesizes high-level demonstrations with fine-grained preference queries to learn the human’s reward.
Algorithm: The robot generates preference questions while accounting for the human’s ability to respond, producing user-friendly and informative queries.
Experiments: The evaluation spans linear dynamics, driving, tossing, Lunar Lander, simulated Fetch, and physical Fetch environments.The experiments use trajectory features tailored to each domain and include a physical Fetch user-study setting.
Human Choice Models: Weak preference queries add an “About Equal” response and model a minimum perceivable difference between two trajectories.The experiments use two-option weak queries with δ = 1 when relevant.
Evaluation: The framework evaluates convergence of inferred reward parameters using an alignment metric relative to the true reward parameters.
Experiments: The experiments cover demonstration initialization, preference-only comparisons, information gain versus volume removal, data ordering, and optimal stopping.

Initializing with Demonstrations

Demonstration initialization substantially accelerates preference-based reward learning, but the benefit of adding more demonstrations depends on the domain. A single demonstration provides reliable convergence improvements across the tested environments.

Experimental Design: The experiment tests whether demonstrations accelerate learning and whether more demonstrations improve convergence.Simulations use Driver, Lunar Lander, and Fetch with 0, 1, or 3 demonstrations followed by preference queries.
Results: One demonstration reaches a convergence level in 10 preference queries that requires 30 on Driver, 35 on Lunar Lander, and 20 on Fetch without demonstrations.The result supports the hypothesis that demonstrations accelerate learning.
Results: Three demonstrations improve convergence over one on Driver and Lunar Lander, but hurt performance on Fetch.The Driver improvement occurs mainly before 10 preference queries.
Results: A single demonstration produces reliable convergence improvements regardless of domain complexity.

DemPref vs IRL

The study compares DemPref with Bayesian IRL using demonstrations and preference queries on simulated and physical Fetch tasks. Users judged DemPref-trained robots better at the task and preferred DemPref for future training, while ease-of-use did not differ significantly.

Experimental setup: The physical Fetch study used Bayesian IRL with five demonstrations and DemPref with one demonstration plus 15 proactive preference queries.Training and testing used different domain variants to reduce sensitivity to the training domain.
User study: Users significantly preferred the DemPref robot for task success and future training over the IRL system.Both comparisons were significant at p < 0.05 using Wilcoxon paired signed-rank tests.
Failure analysis: IRL sometimes overfit training-domain obstacle-avoidance preferences and failed to reach goals in a shifted test domain.The authors expect improved feature engineering and more diverse training domains could mitigate these issues, but rigorous evaluation is beyond the paper’s scope.
User study: Users did not rate DemPref as easier to use than IRL, with p = 0.297.Users cited query-generation time of approximately 45 seconds and difficulty evaluating some preference queries.

Information Gain vs Volume Removal

The paper evaluates information-gain preference queries against volume removal in simulations and user studies. Information gain improves learning rate, produces easier and more distinguishable queries, and yields trajectories users prefer.

Experimental design: The experiments combine simulated users in LDS, Driver, Tosser, and Fetch with real-user studies in Driver, Tosser, and physical Fetch.Real-user sessions included actively generated queries from both volume removal and information gain, with randomized ordering.
Simulation results: Information gain produces fewer wrong and “About Equal” responses than volume removal, especially early in learning.Weak preference queries also consistently reduce wrong answers in the simulated-user experiments.
Simulation results: Information gain significantly increases learning rate over volume removal across the evaluated tasks.The comparison includes strict and weak preference-query variants, with weak queries represented by dashed lines.
User studies: Users rated information-gain queries easier in all environments, with p < 0.005, and reported fewer indistinguishable options than with volume removal.The indistinguishable-query difference was significant for Driver at p < 0.05 and Tosser at p < 0.005.
User studies: Users significantly preferred trajectories learned with information gain over volume removal in both Driver and Tosser.The preference comparison used a one-sample t-test with p < 0.05 in both environments.

Optimal Stopping

The paper adds an optimal stopping rule that weighs expected information gain against the cost of querying human users. Simulations show that the rule terminates near the best cumulative active-learning reward across environments.

Query costs: The cost function favors queries where one feature difference is larger than the differences in the other features.This design is intended to improve query interpretability and may also improve learning efficiency.
Simulation results: Optimal stopping terminates querying with near-optimal cumulative active-learning rewards across LDS, Driver, Tosser, and Fetch.The reward is defined as cumulative information gain minus query cost.

Appendix

The appendix supplies derivations, theorem proofs, extensions, and additional experiments for the information-gain framework. It also establishes a formal stopping criterion based on the sign of the optimized objective.

Additional analyses: Additional appendix analyses cover unknown user-specific δ, continuous query optimization, “About Equal” responses, and query-independent costs.The appendix also includes the proof of Theorem 3.
Information-gain derivation: The appendix derives the mutual-information expression used for information-gain query selection.The derivation expands mutual information into entropy terms and estimates expectations using samples drawn from the prior.
Stopping criterion: Theorem 3 states that terminating is optimal exactly when the global solution to the stopping objective is negative.Continuing is not optimal when the best attainable expected information gain minus cost is negative.

Extension to User-Specific and Unknown δ

The extension learns human-model parameters alongside the reward when needed and derives information-gain objectives for unknown δ. Approximate optimization reduces computation, though joint parameter learning can reduce performance because the primary goal is learning the reward.

User-Specific Human Models: Unknown human-model parameters, including δ and temperature, are represented by ν and incorporated into the belief alongside reward parameters ω.The belief over (ω, ν) is updated while the primary learning target remains ω.
Computational Considerations: The extended objective is computationally heavier because it requires M samples of ω for each sampled value of ν.Its computation is nevertheless parallelizable.
Unknown δ: The information-gain derivation replaces ω with (ω, ν), producing a final optimization over queries that accounts for jointly modeled human parameters.The resulting objective uses samples over the extended parameter space.
Evaluation: An approximate, computationally faster optimization compares strict queries with weak queries under known and unknown δ using information gain across 100 simulated users.Users’ true δ values are sampled uniformly from [0, 2], without assuming prior knowledge beyond δ ≥0.

Comparison of Information Gain and Volume Removal without Query Space Discretization

Without query-space discretization, information gain outperforms volume removal, while weak preference queries learn faster than strict queries. Additional experiments show that “About Equal” responses improve learning, and optimal stopping achieves near-optimal cumulative rewards.

Continuous Query Optimization: Information gain outperforms volume removal for both strict and weak preference queries, while weak queries enable faster learning in continuous trajectory optimization.The comparison covers LDS, Driver, and Tosser tasks; Fetch is excluded because trajectory optimization is prohibitive in its larger action space.
Weak Preference Queries: Using information from “About Equal” responses improves learning for both volume removal and information gain with weak preference queries.The experiment compares retaining these responses against ignoring them and removing the corresponding queries.
Optimal Stopping: Optimal stopping achieves near-optimal cumulative active-learning rewards across all environments under query-independent costs.The stopping rule terminates the process while balancing query costs against learning rewards.