Source-linked AI summary
B-Pref: Benchmarking Preference-Based Reinforcement Learning
Kimin Lee, Laura Smith, Anca Dragan, Pieter Abbeel
TL;DR
Preference-based RL lacks a standard way to evaluate progress, particularly when teacher feedback is not perfectly rational. B-Pref introduces a benchmark with varied simulated teacher irrationalities and evaluates both performance and robustness, finding that existing methods often struggle with wrong labels.
Problem
Preference-based RL lacks a commonly adopted benchmark for quantifying progress and comparing algorithmic choices.
Method
B-Pref benchmarks preference-based RL across locomotion and manipulation tasks using simulated teachers with varied irrationalities and robustness metrics.
Results
Existing methods perform fairly efficiently with correct labels but often suffer poor performance when teachers provide wrong labels.
Takeaways & Limitations
B-Pref provides a common starting point for systematic study and motivates algorithms that are more robust to corrupted feedback.
Takeaways & Limitations
B-Pref focuses on proprioceptive inputs and dense rewards and does not explore robustness to new environments with different dynamics or initial states in depth.
Abstract
from arXiv · showhide
Reinforcement learning (RL) requires access to a reward function that incentivizes the right behavior, but these are notoriously hard to specify for complex tasks. Preference-based RL provides an alternative: learning policies using a teacher's preferences without pre-defined rewards, thus overcoming concerns associated with reward engineering. However, it is difficult to quantify the progress in preference-based RL due to the lack of a commonly adopted benchmark. In this paper, we introduce B-Pref: a benchmark specially designed for preference-based RL. A key challenge with such a benchmark is providing the ability to evaluate candidate algorithms quickly, which makes relying on real human input for evaluation prohibitive. At the same time, simulating human input as giving perfect preferences for the ground truth reward function is unrealistic. B-Pref alleviates this by simulating teachers with a wide array of irrationalities, and proposes metrics not solely for performance but also for robustness to these potential irrationalities. We showcase the utility of B-Pref by using it to analyze algorithmic design choices, such as selecting informative queries, for state-of-the-art preference-based RL algorithms. We hope that B-Pref can serve as a common starting point to study preference-based RL more systematically. Source code is available at https://github.com/rll-research/B-Pref.
1 Introduction
Preference-based RL replaces hand-engineered rewards with teacher feedback, addressing reward specification challenges but creating a need for standardized evaluation. B-Pref fills this gap with simulated teachers that include irrationalities and robustness-oriented metrics.
- Preference-based RL learns desired behavior from teacher preferences instead of relying on hand-engineered rewards.
- B-Pref addresses the lack of a standard benchmark needed to quantify progress and compare design choices in preference-based RL.
- The benchmark uses varied locomotion and robotic manipulation tasks with simulated teachers exhibiting diverse irrationalities.
- Existing methods perform efficiently with perfectly rational teachers but often perform poorly with more realistic irrational teachers.
- B-Pref supports systematic evaluation and comparison through an open-source benchmark and reference implementations.
2 Preliminaries
Preference-based RL trains agents from comparisons between behavior segments rather than environment rewards. Teachers may select one segment, mark segments equally preferable, or discard an uninformative query.
- The framework replaces the traditional reward signal with feedback indicating which observed behavior is preferred.
- Preference-based RL aims to learn desirable behavior using as few teacher queries as possible.
- A segment is a sequence of observations and actions, and teachers compare pairs of segments.
- Teacher responses can prefer the first segment, prefer the second, assign equal preference, or discard the query as incomparable.
3 B-Pref: Benchmarks environments for preference-based RL
B-Pref evaluates preference-based RL with simulated teachers whose feedback reflects rationality, noise, myopia, mistakes, skipping, and equal preferences. It measures both performance under query budgets and robustness to these irrationalities across four tasks.
- Simulated human teachers: The benchmark models stochastic preferences, with rationality controlling decisiveness and discounting representing myopic focus on recent behavior.
- Simulated human teachers: Teachers may make mistakes, skip queries when neither segment contains desired behavior, or mark similarly rewarding segments equally preferable.
- B-Pref evaluates true returns and normalized performance under feedback budgets while separately testing robustness to simulated teacher irrationalities.
- Individual irrationalities are evaluated separately, while the benchmark can also construct teachers combining multiple irrationalities.
- Tasks: The benchmark uses two locomotion tasks and two robotic manipulation tasks, focusing on proprioceptive inputs and dense rewards.
4 B-Pref: Algorithmic baselines for preference-based RL
The benchmark evaluates preference-based RL algorithms that alternate policy learning from learned rewards with reward learning from teacher feedback. It focuses on PEBBLE and PrefPPO, with PEBBLE adding pre-training, off-policy learning, and reward relabeling.
- Preference-based RL alternates agent learning with reward learning: policies maximize learned rewards while reward models fit teacher feedback.
- The reward model is trained as a preference predictor using supervised binary classification with cross-entropy loss.
- PrefPPO: PrefPPO uses PPO to update policies despite the learned reward being non-stationary during training.
- PEBBLE: PEBBLE improves the baseline framework through unsupervised pre-training, off-policy SAC, and relabeling past experiences after reward updates.
- Figure 2 reports IQM normalized returns with 95% confidence intervals across ten runs.
5 Using B-Pref to analyze algorithmic design decisions
B-Pref is used to compare preference-based RL methods and analyze sampling, feedback scheduling, and reward-learning design choices. Results show strong sensitivity to teacher irrationality and query-selection strategy.
- 5.2 Benchmarking prior methods: PEBBLE outperforms PrefPPO in most environments, with especially large gains on robotic manipulation tasks.
- 5.2 Benchmarking prior methods: Under corrupted-label teachers, existing methods often perform poorly, motivating techniques that improve robustness to irrational feedback.
- 5.3 Impact of design decisions in reward learning: Uncertainty-based sampling outperforms other schemes on Quadruped, while coverage-based sampling does not improve on uniform sampling and slows querying.
- 5.3 Impact of design decisions in reward learning: On the Equal teacher, uncertainty-based sampling achieves high returns despite receiving more uniform labels than other sampling schemes.
- 5.3 Impact of design decisions in reward learning: Rule-based increase and decay feedback schedules provide no major gains over uniform scheduling in most environments.
- 5.3 Impact of design decisions in reward learning: Learned reward functions reasonably align with ground-truth rewards in Sweep Into and Walker, although tanh bounding changes their scale.
6 Related work
Prior RL benchmarks target broad or specialized settings, while preference-based RL has lacked a benchmark for complex tasks and irrational teachers. B-Pref addresses this gap beyond earlier diagnostic suites.
- 6 Related work: Existing benchmarks cover general RL, generalization, safety, offline RL, and other specialized research purposes.
- 6 Related work: DERAIL benchmarks preference-based learning on simple diagnostic tasks, whereas B-Pref targets complex locomotion and robotic manipulation tasks.
- 6 Related work: B-Pref additionally models a wide array of teacher irrationalities and benchmarks state-of-the-art preference-based RL algorithms in depth.
- 6 Related work: Human-in-the-loop RL has used demonstrations, real-valued feedback, human preferences, and feedback-efficient reward learning to train agents.
7 Conclusion
B-Pref provides a benchmark for systematically evaluating preference-based RL, including performance under diverse teacher irrationalities. The authors report weaknesses of existing methods and identify scope limitations and safety concerns.
- 7 Conclusion: B-Pref evaluates preference-based RL across teacher irrationalities and analyzes state-of-the-art algorithms and design choices.The benchmark is intended as a common starting point for systematic study.
- 7 Conclusion: Existing methods often perform poorly when teachers provide wrong labels, while design-choice effects vary across task setups.The authors call for new active-learning and meta-learning algorithms.
- 7 Conclusion: B-Pref does not deeply evaluate robustness to new dynamics or initial states, visual observations, or sparse rewards.The benchmark focuses on proprioceptive inputs and dense-reward tasks.
- 7 Conclusion: Preference-based RL may allow malicious users to teach harmful behaviors, requiring careful consideration of safety issues.
A Preliminaries: Reinforcement learning algorithms
This section describes PPO and SAC as the underlying RL algorithms and presents the preference-based training pipeline, exploration procedure, and query-sampling schemes. It also notes a limitation of concatenating segment states for coverage sampling.
- A Preliminaries: Reinforcement learning algorithms: PPO uses clipped-ratio policy optimization with advantage estimates, and is described as more robust to reward non-stationarity from online learning.The policy is trained from recent on-policy experience.
- A Preliminaries: Reinforcement learning algorithms: SAC is an off-policy maximum-entropy actor-critic method that alternates soft policy evaluation and policy improvement.Its entropy term encourages exploration and robustness to noise, while replay improves sample efficiency but reduces robustness to changing rewards.
- A Preliminaries: Reinforcement learning algorithms: Unsupervised pre-training uses state entropy as an intrinsic reward to encourage broader state visitation before preference-based learning.The procedure is summarized as an exploration phase followed by policy and reward updates.
- A Preliminaries: Reinforcement learning algorithms: Preference-based RL alternates reward learning from queried segment preferences with policy learning using the learned reward and an RL objective.On-policy methods reset their buffer after updates, whereas off-policy methods can reuse stored transitions.
- A Preliminaries: Reinforcement learning algorithms: Query selection compares uniform, disagreement, entropy, coverage, and hybrid disagreement- or entropy-plus-coverage sampling schemes.Coverage selects representative centers by minimizing the largest distance to a nearest center.
- A Preliminaries: Reinforcement learning algorithms: Concatenating segment states for coverage is not permutation-invariant, although the authors expect the resulting issue to be insignificant because reordered pairs are unlikely.They identify permutation-invariant handling as a future direction.
D Experimental Details
The experiments use PEBBLE and PrefPPO with separately optimized hyperparameters, reward-model ensembles, and simulated teachers spanning several irrationality settings. Adaptive thresholds standardize skip and equal responses across environments.
- D Experimental Details: Table 1 lists PEBBLE hyperparameters, while Table 2 lists PrefPPO hyperparameters.
- D Experimental Details: The reward model is a three-layer, 256-unit-per-layer leaky-ReLU network trained with ADAM, while an ensemble of three models improves stability.Outputs are bounded with tanh and the initial learning rate is 0.0003.
- D Experimental Details: Experiments evaluate simulated teachers varying rationality, discounting, mistake probability, and adaptive skip or equal thresholds.The teacher configurations include β →∞ and β = 1, γ = 1 or 0.9, and ϵ = 0 or 0.1.
- D Experimental Details: Adaptive thresholds rescale skip or equal decisions using current policy performance, episode length, segment length, and ground-truth rewards.All experiments use ϵ_adapt = 0.1, and adaptive teachers may skip queries or provide equal labels.
E Additional experimental results
Additional experiments examine PEBBLE’s learned reward alignment and label smoothing under a corrupted-label teacher. The learned reward is reasonably aligned despite scale differences, while label smoothing provides only marginal gains.
- Reward analysis: The learned reward function is reasonably well-aligned with the ground truth reward across tested environments.Its scale differs because the output is bounded with tanh.
- Regularization for handling corrupted labels: Label smoothing is evaluated to improve robustness to corrupted labels, but its gains are marginal.The method uses the soft label y = 0.9 ∗y + 0.05 for cross-entropy computation.
- Reward analysis: Figure 6 compares the learned reward function with the ground truth reward using rollouts from a policy optimized by PEBBLE.
F Learning curves
The additional learning-curve results report performance across Walker, Quadruped, Button Press, and Sweep Into, with comparisons spanning PEBBLE, PrefPPO, and different query budgets. Aggregate metrics summarize mean, median, IQM, and optimality gap with confidence intervals.
- Learning curves: Learning curves compare PEBBLE and PrefPPO on Walker-walk and Quadruped-walk using ground truth reward measurements.The curves show means and standard deviations across ten runs, with asymptotic PPO and PrefPPO performance indicated by dotted lines.
- Learning curves: Learning curves compare PEBBLE and PrefPPO on Sweep Into and Button Press using success rate measurements.The curves show means and standard deviations across ten runs, with asymptotic PPO and PrefPPO performance indicated by dotted lines.
- Learning curves: Quadruped-walk learning curves evaluate PEBBLE with 2000 and 1000 queries using ground truth reward measurements.Each curve reports the mean and standard deviation across ten runs.
- Aggregate metrics: Aggregate metrics on Walker, Quadruped, Button Press, and Sweep Into report mean, median, IQM, and optimality gap with 95% confidence intervals.Higher mean, median, and IQM scores and lower optimality gap are better; intervals use percentile bootstrap with stratified sampling.
- Aggregate metrics: Aggregate metrics additionally compare PEBBLE and PrefPPO under 1000- and 2000-query Quadruped settings.The metrics use higher-is-better mean, median, and IQM scores and a lower-is-better optimality gap.