PaperlayerView exact revision on arXiv

Source-linked AI summary

Benchmarks and Algorithms for Offline Preference-Based Reward Learning

Daniel Shin, Anca D. Dragan, Daniel S. Brown

arXiv:2301.01392v1cs.LGcs.AI

TL;DR

Human-preference reward learning usually needs costly physical rollouts or accurate simulation, while related offline data may provide a safer alternative. The paper proposes OPRL, which actively queries preferences over offline trajectory segments, learns rewards, and performs offline RL; it finds that benchmark choice matters and that the combined approach can learn unseen tasks from data that never explicitly demonstrates them.

  • Problem

    Preference-based reward learning typically requires a high-fidelity simulator or expensive and potentially unsafe physical rollouts.

  • Method

    OPRL uses pool-based active learning over offline trajectory segments to obtain pairwise preferences, learn a reward distribution, and optimize a policy with offline RL.

  • Results

    Many offline RL benchmarks are unsuitable because trivial rewards can perform similarly to or better than true rewards, while curated and new tasks show that learned preferences with offline RL can produce unseen behaviors.

  • Takeaways & Limitations

    Offline datasets can support efficient preference-based reward inference and policy learning for tasks not explicitly demonstrated in the data.

  • Takeaways & Limitations

    The training process assumes no access to the MDP beyond the static offline trajectories, whose contents may be random, heterogeneous, or multi-task.

Abstract

from arXiv · show

Learning a reward function from human preferences is challenging as it typically requires having a high-fidelity simulator or using expensive and potentially unsafe actual physical rollouts in the environment. However, in many tasks the agent might have access to offline data from related tasks in the same target environment. While offline data is increasingly being used to aid policy optimization via offline RL, our observation is that it can be a surprisingly rich source of information for preference learning as well. We propose an approach that uses an offline dataset to craft preference queries via pool-based active learning, learns a distribution over reward functions, and optimizes a corresponding policy via offline RL. Crucially, our proposed approach does not require actual physical rollouts or an accurate simulator for either the reward learning or policy optimization steps. To test our approach, we first evaluate existing offline RL benchmarks for their suitability for offline reward learning. Surprisingly, for many offline RL domains, we find that simply using a trivial reward function results good policy performance, making these domains ill-suited for evaluating learned rewards. To address this, we identify a subset of existing offline RL benchmarks that are well suited for offline reward learning and also propose new offline apprenticeship learning benchmarks which allow for more open-ended behaviors. When evaluated on this curated set of domains, our empirical results suggest that combining offline RL with learned human preferences can enable an agent to learn to perform novel tasks that were not explicitly shown in the offline data.

1 Introduction

The paper introduces Offline Preference-Based Reward Learning (OPRL), which uses offline trajectory data to learn user-preferred rewards and policies without environmental interactions. It also shows that benchmark choice is critical: many offline RL domains reward high performance even with trivial rewards, whereas selected and newly proposed tasks expose the value of reward learning.

  • Offline Preference-Based Reward Learning: OPRL queries pairwise preferences over recorded trajectory segments, learns a reward function, and applies offline RL without environmental interactions.The approach avoids both physical rollouts and high-fidelity simulation for reward learning and policy optimization.
  • Benchmark motivation: Many standard offline RL benchmarks are ill-suited for reward learning because replacing rewards with zeros or constants can match or exceed true-reward performance.This means benchmark performance may primarily reflect data quality rather than learned reward quality.
  • Benchmark motivation: High offline data variability and multi-task data identify settings where reward learning is necessary for good policy learning.The authors use these settings to isolate the effects of reward learning more effectively.
  • Open-ended benchmarks: OPRL learns a counter-clockwise orbit from random point-to-point navigation data even though no complete orbit appears in the offline dataset.The method combines human preferences over trajectory snippets with offline RL to stitch incomplete segments into a novel behavior.
  • Contributions: The paper formalizes offline preference-based reward learning, evaluates existing benchmarks, proposes open-ended benchmarks, and empirically compares uncertainty and query-acquisition methods.It reports evidence that ensemble-based disagreement queries outperform other approaches.

2 Related Work

Prior preference-learning and safe apprenticeship methods commonly require environment rollouts, accurate simulators, dynamics models, or expert demonstrations. OPRL instead learns an explicit reward from preferences over offline data of potentially heterogeneous or random quality, then uses offline RL.

  • Preference-based reward learning: Online preference-based reward learning typically requires physical rollouts or accurate simulators, creating safety, efficiency, and sim-to-real concerns.These requirements can make repeated trial-and-error interaction unacceptable in real-world settings.
  • Safe apprenticeship learning: Safe apprenticeship methods address risk through assistance, tail-risk optimization, or performance bounds, but rely on accurate dynamics models or related assumptions.The cited approaches do not remove the need for a model-based description of the environment.
  • Offline apprenticeship learning: Prior offline apprenticeship methods often assume discrete actions, hand-crafted reward features, expert demonstrations, or explicit trajectory reward labels.OPRL instead targets fully offline reward learning without requiring expert demonstrations.
  • Offline imitation learning: Offline imitation methods such as DemoDICE and IQ-Learn use expert demonstrations, whereas OPRL learns an explicit reward from preferences over offline data of any quality, including random-policy data.This distinguishes OPRL from methods that imitate demonstrations directly or learn a parameterized Q-function instead of a reward.
  • Heterogeneous demonstrations: Like prior work using heterogeneous data, OPRL handles varied data but uses preference queries to learn a reward for offline RL rather than directly learning a scheduling algorithm.Its pipeline separates preference-based reward inference from policy optimization.

3 Problem Definition

The problem is to learn a policy from a static offline trajectory dataset when the true reward and the MDP are unavailable. The dataset may contain random, heterogeneous, or multi-task behavior, while a small number of pairwise preferences over trajectory snippets supplies reward information.

  • MDP formulation: The environment is modeled as an MDP with states, actions, reward, transition dynamics, an initial-state distribution, and a discount factor.The reward function maps state-action pairs to real values, while transitions specify the dynamics.
  • Offline setting: Unlike standard RL, the learner has no access to the true reward function or the MDP during training, only to a static dataset of trajectories.Each trajectory consists of contiguous state-action-next-state transition tuples.
  • Dataset assumptions: The offline dataset need not come from one expert or one task and may instead combine random data, multiple policies, or demonstrations for varied tasks.This setting differs from imitation learning based on expert demonstrations for a specific reward.
  • Learning objective: Given a small number of pairwise preferences over trajectory snippets, the goal is to find a policy maximizing expected discounted return under the unknown reward using only the offline trajectories.The objective is defined by the discounted return J(π) under the true but unobserved reward.

4 Offline Preference-Based Reward Learning

OPRL learns user preferences from trajectory snippets drawn from offline data, then uses the learned reward function for offline policy optimization. It selects informative queries actively using uncertainty estimates from ensembles or Bayesian dropout, without requiring environment rollouts or a dynamics model.

  • OPRL models preferences with the Bradley-Terry pairwise framework, approximating state rewards with a neural network and maximizing the probability of observed trajectory preferences.For a preference τi ≺τj, the model compares the summed predicted rewards of the two trajectory snippets.
  • The method represents reward uncertainty with either ensembles or Bayesian dropout to estimate which unlabeled trajectory pairs will most reduce uncertainty about the reward function.Ensembles use differently initialized reward networks, while Bayesian dropout repeatedly samples predictions with dropout applied during inference.
  • Unlike prior active preference-learning methods, OPRL avoids both on-policy environment rollouts and model-based query synthesis by sampling candidate pairs from offline data.Candidate-query value is estimated using disagreement or information gain before requesting a human label.
  • Query disagreement selects pairs with the highest variance in binary preference predictions, whereas information gain favors high overall outcome entropy and low entropy under individual reward hypotheses.Information gain is approximated from ensemble or dropout samples and is computed between reward parameters and the prospective human label.
  • Offline-dataset query search can be accelerated through parallel candidate evaluation, GPU batching of trajectory states, and anytime search for the next active query.These implementation choices make repeated value-of-information computation practical over many candidate snippets.
  • OPRL searches an offline dataset for trajectory-snippet pairs, actively queries the most valuable preference, updates the reward model, and then runs offline RL on the relabeled data.Algorithm 1 separates reward learning from policy learning while using the same offline dataset.

5 Experiments and Results

The experiments identify offline RL domains where reward learning matters, then evaluate OPRL with active preference queries on curated benchmarks and new open-ended tasks. OPRL performs well on several tasks, including behaviors absent from the offline data, while performance varies by domain and query method.

  • 5.1 Evaluating Offline RL Benchmarks: Offline RL benchmarks with expert-only data are poor tests of reward learning because trivial rewards often preserve expert-level performance, whereas diverse random or multitask data exposes reward sensitivity.The benchmark selection uses degradation above 20% to isolate domains where learning a shaped reward matters.
  • 5.2.1 Maze Navigation: Ensemble disagreement is the strongest active-query method overall, reaching near ground-truth Maze2D-Umaze performance after 15 preference queries but not matching ground truth on Maze2D-Medium.The authors attribute the harder Medium-Maze result to goal-oriented rewards, where trajectory comparisons may underrepresent the importance of the final destination.
  • 5.2.2 Flow Merge: Random queries achieve the best Flow Merge performance, while all query methods recover near-ground-truth performance.This domain evaluates traffic-flow maximization in a highway-merging scenario.
  • 5.2.4 Robotic Manipulation: OPRL improves robotic manipulation reward learning: ensemble disagreement and information gain outperform the ground-truth reward, while zero or constant rewards degrade performance.The results also report faster convergence for OPRL than for the ground-truth reward.
  • 5.3 New Offline Preference-Based Reward Learning Tasks: OPRL learns novel behaviors absent from the offline data, including constrained navigation and open-ended maze patterns, by combining preference-based reward learning with offline RL.The approach learns counter-clockwise orbits, balance, and windmill-like behaviors from datasets that do not contain complete successful examples.
  • 5.3.4 Pilot User Study: In a pilot user study, ensemble disagreement achieved 122.6 versus 134.6 for ground-truth reward performance, and pooled preferences produced human-aligned orbiting behavior.All users successfully taught an orbiting behavior, while pooled labels generally outperformed individual-user data.

6 Discussion and Future Work

OPRL combines offline preference-based reward learning with offline RL, using informative trajectory snippets from offline data instead of environment interaction. Its results indicate that a small number of preference queries can approach policies trained with extensive ground-truth reward data.

  • Using ensemble disagreement with only 10-15 preference queries produced policies with performance similar to policies trained using tens of thousands of ground-truth reward samples.
  • Across five tasks, ensemble disagreement achieved the best performance on three and outperformed random query selection on all five.
  • OPRL selects informative trajectory subsequences from random or multi-task offline datasets, showing that preference learning need not rely on expert demonstrations.
  • The reported IQM results aggregate performance using a statistic intended to be more robust to outliers than the mean.IQM discards the lowest and highest 25% of values before averaging the remaining 50%.

B Sensitivity Analysis on Number of Queries

The query-budget sensitivity analysis finds that ensemble disagreement is generally better than random querying, with 50 initial queries providing a favorable balance in the tested setting.

  • Number of initial queries: Ensemble disagreement outperformed random queries for every tested number of initial queries, while its advantage was smaller at 25 and 100 queries.Twenty-five queries may be insufficient to train the initial ensemble, whereas 100 queries may already suffice to train a good reward model.
  • Number of initial queries: More initial queries improved performance for random querying, whereas ensemble disagreement benefited from an intermediate initial-query budget.
  • Number of initial queries: Ensemble disagreement performed best with 50 initial queries, the default used in the sensitivity analysis.
  • Queries per round: Ensemble disagreement outperformed random querying across all tested numbers of queries per round, but its performance remained fairly consistent across those settings.Increasing the number of queries per round did not necessarily improve performance, partly because other hyperparameters were tuned for 10 queries per round.

C Sensitivity Analysis on Number of Ensemble Models and Number of Dropout Samples

The sensitivity analysis shows that uncertainty-estimation performance depends on ensemble or dropout sample counts, with disagreement and information-gain methods exhibiting different robustness patterns.

  • Number of ensemble models: Increasing ensemble models from 3 to 7 improved ensemble-disagreement performance, while increasing from 7 to 14 produced no improvement and a slight dip.
  • Number of ensemble models: Ensemble information gain improved slightly with more ensemble models and was less sensitive to ensemble size than ensemble disagreement.
  • Number of dropout samples: Increasing dropout samples improved dropout-disagreement performance, while dropout information gain worked best with 30 samples and degraded slightly beyond that.
  • Information gain: Information gain selects queries by estimating expected information about reward parameters from preference outcomes under an approximate posterior.The information gain is high when aggregate outcome entropy is high but individual posterior hypotheses have low entropy, indicating strong disagreement without a clear majority.
  • Information gain: Candidate information-gain queries are evaluated over trajectory pairs, and the pair with the highest estimated information gain is selected for human labeling.

E Evaluating Offline RL Benchmarks

Many D4RL offline RL benchmarks are poor tests of learned rewards because trivial or constant rewards can match or exceed true-reward performance. This occurs because offline RL policies remain close to dataset behavior, making reward quality irrelevant for expert-like datasets.

  • Average and zero reward masking performed competitively with true rewards on many D4RL benchmarks and often exceeded a purely random policy.
  • When all offline data consists of expert actions, offline RL can recover expert-like behavior regardless of the reward function.Behavioral-cloning methods avoid out-of-distribution actions, while other offline RL methods constrain policies near the behavior distribution.
  • With zero rewards, Advantage-Weighted Regression reduces exactly to behavioral cloning on the replay buffer.The value and advantage terms become zero, so policy learning reproduces observed dataset actions.
  • For non-zero constant rewards, Advantage-Weighted Regression can weight trajectories according to their lengths when termination times differ, allowing terminal-state information to affect learning.Positive constants favor longer trajectories, while negative constants favor shorter ones in the described setting.
  • The preferred uncertainty representation depends on the policy optimizer: dropout performed better than ensembles with CQL, whereas ensembles performed better with AWR.The authors identify interactions between query mechanisms and offline RL algorithms as an area for future work.

G Franka Kitchen Learning Curves

In FrankaKitchen, OPRL learns rewards from preferences that outperform ground-truth reward optimization and converge faster, while active query methods continue improving preference accuracy beyond five rounds.

  • G Franka Kitchen Learning Curves: OPRL with Ensemble Disagreement and Ensemble InfoGain achieves better FrankaKitchen performance than the ground-truth reward.The learned shaping rewards also converge faster according to the learning curves.
  • G Franka Kitchen Learning Curves: Dropout disagreement after 15 rounds reaches performance similar to the ground-truth reward in Kitchen-complete.
  • G Franka Kitchen Learning Curves: Active query methods continue improving held-out pairwise preference accuracy after round 5, unlike randomly queried T-REX.
  • G Franka Kitchen Learning Curves: Human-label evaluation in maze2d-medium-dense-v1 uses 100 query labels from six users to assess quantitative performance.

J Real human feedback experiments

Human-provided preferences generate a counter-clockwise orbit behavior.

  • J Real human feedback experiments: Human-provided preferences generate a counter-clockwise orbit.
  • J Real human feedback experiments: The figure presents an orbit behavior produced from human feedback.
  • J Real human feedback experiments: The demonstrated behavior is an orbit rather than a point-to-point trajectory.

K Reward Model, Hyperparameter, And Dataset Details

The experiments use specified ensemble, policy-learning, and evaluation settings, while benchmark tables test whether constant or zero rewards preserve offline RL performance.

  • K Reward Model, Hyperparameter, And Dataset Details: Active reward learning uses seven ensemble models or 30 dropout passes, with both settings tuned for speed and estimation accuracy.
  • K Reward Model, Hyperparameter, And Dataset Details: All reported results are averaged over three random seeds, and customized-behavior datasets are slated for release upon conference acceptance.
  • K Reward Model, Hyperparameter, And Dataset Details: Tables 12–14 compare true, average, and zero rewards across D4RL tasks for BCQ, BEAR, and CQL using a degradation percentage based on the best constant or random alternative.
Loading 2301.01392v1…