PaperlayerView exact revision on arXiv

Source-linked AI summary

Active Preference-Based Gaussian Process Regression for Reward Learning

Erdem Bıyık, Nicolas Huynh, Mykel J. Kochenderfer, Dorsa Sadigh

arXiv:2005.02575v2cs.ROcs.AIcs.LG

TL;DR

Robotics reward learning is challenging because demonstrations can be difficult to provide and existing approaches may require restrictive reward models or substantial data. This paper learns expressive rewards from pairwise human preferences using active Gaussian Process regression. In a user study, its method outperformed two alternatives on prediction accuracy and ratings of the robot’s learned behavior.

  • Problem

    Reward learning needs to capture preferred behavior even when demonstrations are difficult to provide, while avoiding restrictive reward models.

    Source: §II · p. 2, §III · p. 3

  • Method

    The method actively learns a Gaussian Process reward from pairwise trajectory preferences rather than demonstrations or reward-value labels.

    Source: §II · pp. 2–3

  • Results

    In the user study, ACTIVEGP achieved higher prediction accuracy (0.74±0.04) than ACTIVELINEAR (0.62±0.07) and RANDOMGP (0.62 ± 0.06), and higher user ratings (6.9 ± 0.6) than ACTIVELINEAR (3.4 ± 0.7) and RANDOMGP (5.1 ± 0.7).

    Source: C. Results and Discussion · p. 8

  • Takeaways & Limitations

    The reported results indicate that GP rewards can improve expressive power while remaining data-efficient for preference-based reward learning.

    Source: §II · p. 2

    For computational reasons, the method assumes the feature dimension d is small.

    Source: §III · p. 3

Abstract

from arXiv · show

Designing reward functions is a challenging problem in AI and robotics. Humans usually have a difficult time directly specifying all the desirable behaviors that a robot needs to optimize. One common approach is to learn reward functions from collected expert demonstrations. However, learning reward functions from demonstrations introduces many challenges: some methods require highly structured models, e.g. reward functions that are linear in some predefined set of features, while others adopt less structured reward functions that on the other hand require tremendous amount of data. In addition, humans tend to have a difficult time providing demonstrations on robots with high degrees of freedom, or even quantifying reward values for given demonstrations. To address these challenges, we present a preference-based learning approach, where as an alternative, the human feedback is only in the form of comparisons between trajectories. Furthermore, we do not assume highly constrained structures on the reward function. Instead, we model the reward function using a Gaussian Process (GP) and propose a mathematical formulation to actively find a GP using only human preferences. Our approach enables us to tackle both inflexibility and data-inefficiency problems within a preference-based learning framework. Our results in simulations and a user study suggest that our approach can efficiently learn expressive reward functions for robotics tasks.

II. RELATED WORK

Prior work learns rewards from demonstrations or preference comparisons, but demonstrations can be difficult to provide and active preference methods often rely on restrictive linear feature models. This work uses Gaussian Processes (GPs) to learn expressive rewards from pairwise preferences, within a finite-horizon deterministic MDP formulation.

II. RELATED WORK

  • Demonstrations can be difficult for high-degree-of-freedom robots, and autonomous-driving studies suggest people may prefer behavior unlike their own demonstrations [30] [33].The authors argue that reward learning should go beyond demonstrations to capture preferred behavior.Source: §II · p. 2
  • Prior active preference-learning methods commonly assume linear rewards over hand-coded features, limiting flexibility and requiring careful feature design.Earlier extensions include batch-active queries, rankings, general MDPs, and combinations with demonstrations, while multi-modal linear models remained limited to two modes [17] [21] [20].Source: §II · p. 2
  • The proposed active preference-based GP method learns expressive rewards without requiring reward-value labels or experts able to control the system desirably.The authors report improved expressive power while retaining data efficiency.Source: §II · pp. 2–3

III. PROBLEM FORMULATION

  • The problem is formulated as learning a trajectory reward from pairwise comparisons in a finite-horizon deterministic MDP.A trajectory comprises an initial position and the actions through the finite time horizon.Source: §III · p. 3
  • The reward is modeled as a GP over trajectory features, avoiding a required linear relationship between features and reward.The authors assume the feature dimension d is small for computational reasons.Source: §III · p. 3
  • Human choices are modeled noisily with a probit response model based on the difference between the two trajectory rewards.The user chooses between two demonstrated trajectories; the response noise is modeled with a normal distribution.Source: §III · p. 3

IV. METHODS

The method models reward as a Gaussian Process and updates that model from pairwise preference data. It approximates the posterior for inference, then uses reward predictions and uncertainty to support active query generation.

A. Gaussian Processes

  • Gaussian Processes model nonlinear reward functions and uncertainty without introducing parameters.The covariance matrix depends on the kernel; the method uses a variant of the radial basis function kernel.Source: A. Gaussian Processes · p. 3
  • The method fixes the reward at an arbitrary feature point to zero, resolving the ambiguity caused by preferences depending only on reward differences.This choice preserves preferences because shifting all rewards by a constant leaves the comparison likelihood unchanged.Source: A. Gaussian Processes · p. 3

B. Preference-based GP Regression

  • Preference data update the GP through a Bayesian posterior combining the user-response model and GP prior, but the resulting posterior is analytically intractable.The method uses Laplace approximation for computational efficiency; Expectation Propagation is described as more accurate but slower.Source: B. Preference-based GP Regression · p. 4
  • The posterior mode is used as an approximation to the posterior mean; following, the posterior covariance is (K^-1 + W)^-1, with W the negative Hessian of the log-likelihood.Source: B. Preference-based GP Regression · p. 4
  • The approximated GP supports inference of expected rewards and covariance at candidate trajectory features for active query generation.The method uses predicted reward means and covariance to help select informative preference queries.Source: B. Preference-based GP Regression · p. 4

C. Active Query Synthesis for Reward Learning

Because each preference query provides at most one bit, the paper selects queries by maximizing information gain for a GP reward model. The objective balances learning informative comparisons with questions users can answer confidently.

C. Active Query Synthesis for Reward Learning

  • Each preference query provides at most one bit, motivating active selection to maximize information gain for GP reward learning.The authors expect information gain to decrease as learning progresses.Source: C. Active Query Synthesis for Reward… · p. 4
  • The objective balances uncertainty in the GP's predicted query outcome, which favors faster convergence, against user certainty when answering.The first entropy term encourages informative outcomes; minimizing the second favors queries users find easier to answer.Source: C. Active Query Synthesis for Reward… · p. 5
  • The authors give an implementable objective using GP posterior means and binary entropy to synthesize informative preference queries.The objective's terms are computed using Eqn. (6).Source: C. Active Query Synthesis for Reward… · p. 5
  • Comparing a trajectory with itself is a global minimizer of the acquisition objective, with objective value zero.The paper contrasts this property with prior volume-removal methods that can produce same-trajectory queries.Source: C. Active Query Synthesis for Reward… · p. 5

V. SIMULATION EXPERIMENTS

The simulation experiments assess whether GP rewards improve expressiveness and whether active query generation improves data-efficiency.

V. SIMULATION EXPERIMENTS

  • The simulations compare GP and linear rewards to assess expressiveness.The experiments use two simulation domains.Source: §V · p. 5
  • The simulations compare active query generation with random querying to assess data-efficiency.Source: §V · p. 5

A. Simulation Environments

The evaluation uses Driver and Tosser simulations with simple trajectory features and simulated human preferences. The authors report that GP rewards capture nonlinearities without fine-tuning feature parameters.

A. Simulation Environments

  • The two domains are a 2D Driver simulation and a MuJoCo Tosser task, where users compare driving actions or throwing outcomes.Different users may disagree about whether a Tosser trajectory should target the green basket or drop the object.Source: A. Simulation Environments · p. 5
  • Driver features include distance to another car, speed, heading, and lane-center distance; Tosser features are maximum horizontal range and capsule flips.Source: A. Simulation Environments · p. 5
  • GPs capture nonlinearities, so the authors report no need to fine-tune feature parameters to learn the reward functions.Source: A. Simulation Environments · p. 5
  • Simulated users have noisy preferences generated from linear or degree-two polynomial reward functions with randomly sampled parameters, across five runs.The underlying reward-function parameters are sampled independently from a standard normal distribution.Source: A. Simulation Environments · p. 5

B. Baselines

In Driver and Tosser, active querying accelerates learning over random querying, while the evaluation compares GP and linear rewards and uses diverse test trajectories. The mini-golf illustration shows how the linear model's maximizers can miss useful interior outcomes.

C. Evaluation

B. Baselines

  • Baselines isolate model and query choices: ACTIVEGP uses a GP with active queries, ACTIVELINEAR uses a linear reward with, and RANDOMGP samples GP queries randomly.Source: B. Baselines · pp. 5–6

C. Evaluation

  • Active methods select expected-information-gain queries from uniformly sampled training trajectories, avoiding continuous optimization over all possible trajectories.The authors state that this reduces computation time relative to continuous optimization.Source: B. Baselines · p. 6
  • Poisson disk sampling creates diverse test sets to reduce the dominance of uninteresting random trajectories; too few samples raise variance, while too many can make random querying seem comparable.The test set is intended to represent the dynamically feasible region with equal importance.Source: C. Evaluation · pp. 6–7
  • In the mini-golf illustration, linear-reward maximizers lie near the shaded region's corners, while the GP can query a better interior trajectory.Source: Figure 6 · p. 7

VI. USER STUDIES

The Fetch mini-golf study compares reward models learned from preference queries, testing both prediction of user preferences and the quality of the resulting robot shots. ACTIVEGP outperformed ACTIVELINEAR and RANDOMGP on both measures, and the conclusion reports improved expressiveness and data-efficiency over baselines in simulations and user studies.

C. Results and Discussion

  • ACTIVEGP achieved 0.74±0.04 prediction accuracy, versus 0.62±0.07 for ACTIVELINEAR and 0.62 ± 0.06 for RANDOMGP (p < 0.05).The comparison used models trained with 15 queries.Source: Figure 7 · p. 8, C. Results and Discussion · p. 8
  • ACTIVEGP received 6.9 ± 0.6 for final robot performance, versus 3.4 ± 0.7 for ACTIVELINEAR and 5.1 ± 0.7 for RANDOMGP (p < 0.05).ACTIVELINEAR occasionally scored highly when users preferred an edge target, but generally failed to produce the desired behavior because of its lower expressive power.Source: Figure 7 · p. 8, C. Results and Discussion · p. 8

A. Experiment Setup

  • The Fetch mini-golf task varied shot speed and angle, where linear rewards cannot make middle-region targets optimal.The robot did not know the target locations, and target scores were known only to participants.Source: A. Experiment Setup · pp. 7–8

B. Subjects and Procedure

  • The study used 10 participants, who assigned scores to eight targets and answered 50 pairwise questions, including 15 for each learned model and five random test queries.Prediction accuracy was measured on the test queries; participants also rated three post-learning robot trajectories on a 9-point scale.Source: B. Subjects and Procedure · p. 8

VII. CONCLUSION

  • The authors conclude that simulations and user studies suggest ACTIVEGP is more successful than baselines in expressiveness and data-efficiency.The conclusion frames these findings as evidence that the method addresses limits in expressiveness, data-efficiency, and demonstrating or quantifying trajectories.Source: §VII · p. 8
  • The method is developed for pairwise comparisons, and GP regression becomes computationally heavy when the domain is high-dimensional.Feature design remains necessary; the authors note that Ψ(ξ) = ξ may be unrealistic for high-dimensional trajectory spaces.Source: §VII · p. 8

A. Active Query Generation Derivation

The derivation evaluates expected entropy terms for active query generation using posterior covariance and variance expressions. It simplifies the integrals with changes of variables and Gaussian convolutions.

A. Active Query Generation Derivation

Loading 2005.02575v2…