Source-linked AI summary
PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training
Kimin Lee, Laura Smith, Pieter Abbeel
TL;DR
Complex RL objectives are difficult to encode with engineered rewards, while human-in-the-loop methods require costly supervision. PEBBLE combines unsupervised exploration, preference-based reward learning, and off-policy relabeling to improve feedback efficiency. It learns more complex locomotion and manipulation behaviors and avoids reward exploitation, while replay-based learning remains sensitive to non-stationary rewards.
Problem
Engineered rewards are difficult to make sufficiently informative for complex objectives, and human-in-the-loop RL is costly because feedback is expensive to provide.
Method
PEBBLE uses unsupervised pre-training to diversify experience, learns a reward model from binary preferences between behavior clips, and relabels past experience for off-policy learning.
Results
PEBBLE improves sample- and feedback-efficiency, outperforms prior preference-based baselines on complex locomotion and manipulation tasks, and avoids reward exploitation.
Takeaways & Limitations
Preference-based learning with unsupervised exploration and experience relabeling can extend human-in-the-loop RL to higher-complexity behaviors and new behaviors difficult to specify with standard rewards.
Takeaways & Limitations
SAC’s replay-based sample efficiency makes it non-robust to non-stationary reward functions.
Abstract
from arXiv · showhide
Conveying complex objectives to reinforcement learning (RL) agents can often be difficult, involving meticulous design of reward functions that are sufficiently informative yet easy enough to provide. Human-in-the-loop RL methods allow practitioners to instead interactively teach agents through tailored feedback; however, such approaches have been challenging to scale since human feedback is very expensive. In this work, we aim to make this process more sample- and feedback-efficient. We present an off-policy, interactive RL algorithm that capitalizes on the strengths of both feedback and off-policy learning. Specifically, we learn a reward model by actively querying a teacher's preferences between two clips of behavior and use it to train an agent. To enable off-policy learning, we relabel all the agent's past experience when its reward model changes. We additionally show that pre-training our agents with unsupervised exploration substantially increases the mileage of its queries. We demonstrate that our approach is capable of learning tasks of higher complexity than previously considered by human-in-the-loop methods, including a variety of locomotion and robotic manipulation skills. We also show that our method is able to utilize real-time human feedback to effectively prevent reward exploitation and learn new behaviors that are difficult to specify with standard reward functions.
1. Introduction
PEBBLE addresses the difficulty and expense of specifying complex RL objectives by combining interactive preference feedback with unsupervised pre-training and off-policy experience relabeling. The method is reported to improve feedback efficiency, handle more complex locomotion and manipulation tasks, and avoid reward exploitation.
- Engineered reward functions can require extensive instrumentation, yet still fail to capture nuanced objectives and may be exploited by RL agents.Examples include accelerometers, thermal cameras, and motion capture for dense feedback.
- Human-in-the-loop RL offers tailored behavioral guidance through feedback, but its adoption is limited because practical, high-bandwidth human supervision is difficult to scale.
- PEBBLE combines unsupervised exploration, preference-based reward learning, and off-policy learning with relabeling of past experience.The agent first diversifies its experience, then learns a reward model from binary preferences between behavior clips and optimizes that inferred reward.
- Relabeling all past experience whenever the reward model changes reuses collected data to improve sample and feedback efficiency.
- PEBBLE improves sample- and feedback-efficiency in HiL RL and outperforms prior preference-based baselines on complex DMControl and Meta-world tasks.
- The method learns behaviors that are difficult to specify with typical rewards and can avoid reward exploitation, producing more desirable behaviors than an engineered-reward agent.
2. Related Work
Prior work addresses human feedback, reward learning, preference supervision, and unsupervised exploration, but each line of work has limitations in feedback scalability, human burden, or task coverage.
- Approaches requiring feedback at all times are difficult to scale because complex problems require substantial agent experience.
- Reward classifiers and regression models use human input to represent task success, but reliably providing unbounded real-valued utility judgments is challenging.
- Preference-based learning uses relative behavior judgments that are easier for humans to provide while remaining information-rich.
- Unsupervised RL pre-training develops behavioral priors and encourages exploration through intrinsic rewards such as prediction error and state novelty.
3. Preliminaries
The preliminaries define standard RL, Soft Actor-Critic, and preference-based reward learning. They motivate PEBBLE’s use of off-policy learning while identifying non-stationary rewards as a challenge for replay-based methods.
- Reinforcement learning: RL agents select actions from states, receive rewards, transition through the environment, and maximize expected discounted return.The return is the discounted sum of future rewards with discount factor γ ∈[0, 1).
- Soft Actor-Critic: Soft Actor-Critic is an off-policy maximum-entropy actor-critic method that alternates soft policy evaluation and policy improvement.Its objective combines reward and policy entropy, and it reuses past experience through a replay buffer.
- Soft Actor-Critic: SAC improves sample efficiency by reusing past experience but is not robust to non-stationary rewards.
- Reward learning from preferences: Preference-based reward learning represents each behavior segment as a sequence of observations and actions and trains a reward function to match human preferences between segment pairs.Preferences may favor either segment or express indifference; the Bradley-Terry model links preference probability to summed segment rewards.
4. PEBBLE
PEBBLE combines unsupervised exploration, preference-based reward learning, and off-policy RL with replay relabeling to reduce the feedback burden in interactive learning. It is evaluated on complex locomotion and robotic manipulation tasks, where it outperforms prior preference-based RL baselines.
- Unsupervised pre-training: Unsupervised pre-training uses intrinsic motivation to collect diverse experiences before preference-based task learning begins.The intrinsic reward is based on state entropy approximated through particle-based k-nearest-neighbor distances.
- Unsupervised pre-training: The intrinsic reward increases when states are farther from their k-th nearest neighbors, encouraging exploration of a wider range of states.In experiments, k-NN distances are computed against replay-buffer samples and normalized by a running standard-deviation estimate.
- PEBBLE algorithm: PEBBLE alternates teacher preference learning with off-policy agent learning, relabeling the replay buffer whenever the learned reward changes.The procedure first collects exploratory experience, queries preferences, updates the reward model, relabels past experience, and then updates the policy and Q-function.
- Selecting informative queries: PEBBLE samples preference queries using uniform or ensemble-based schemes because exact expected value of information is intractable.These schemes target queries likely to change the learned reward model without evaluating all trajectories induced by the updated policy.
- Using off-policy RL with non-stationary reward: Off-policy learning reuses replay-buffer experience, while relabeling that experience stabilizes training under the reward model’s non-stationarity.This addresses the poor sample efficiency of prior on-policy approaches and their resulting feedback inefficiency.
5. Experiments
Experiments evaluate PEBBLE’s sample and feedback efficiency, component contributions, benchmark performance, novel-behavior learning, and resistance to reward exploitation. Across locomotion and manipulation tasks, PEBBLE matches ground-truth-reward SAC while outperforming Preference PPO under comparable or larger feedback budgets.
- Setups: Experiments compare PEBBLE with Preference PPO and ground-truth-reward SAC/PPO on locomotion and robotic manipulation tasks.The evaluation uses DMControl and Meta-world tasks, with scripted preferences for quantitative benchmarking and human trainers for novel behaviors and reward-exploitation tests.
- Benchmark Tasks with Unobserved Rewards: With 1400 queries, PEBBLE reaches SAC’s performance on Cheetah-run, Walker-walk, and Quadruped-walk, while Preference PPO fails to match PPO despite more feedback.Figure 3 reports locomotion learning curves using 400, 700, or 1400 PEBBLE feedback pieces and 1400 or 2100 Preference PPO pieces.
- Benchmark Tasks with Unobserved Rewards: PEBBLE matches ground-truth-reward SAC and outperforms Preference PPO on every evaluated robotic manipulation task.The comparison covers six fundamental manipulation skills from Meta-world and uses success rate as the evaluation measure.
- Ablation Study: Relabeling improves robustness to reward-model changes, while unsupervised pre-training further improves sample efficiency and asymptotic performance on Quadruped-walk.The ablation applies the techniques incrementally with 1400 queries; pre-training exposes diverse behaviors that can induce a better-shaped reward.
- Ablation Study: Uncertainty-based query sampling outperforms uniform sampling on complex Quadruped-walk, whereas longer feedback segments provide more meaningful signal than step-wise feedback.The sampling advantage does not appear on relatively simple Walker and Cheetah environments, and longer segments provide additional reward-learning context.
- Human Experiments: Human feedback trains novel behaviors with 50–200 queries and changes reward-exploiting one-legged Walker locomotion into a more natural two-legged gait.Demonstrated behaviors include Cart pole swinging, Quadruped leg waving, and Hopper backflips; trainers also guide variations of the same behavior.
6. Discussion
PEBBLE combines unsupervised pre-training with off-policy learning to improve feedback efficiency and extend human-in-the-loop RL to more complex behaviors. It also supports novel behaviors and avoidance of reward exploitation relative to engineered rewards.
- PEBBLE combines unsupervised pre-training and off-policy learning to improve sample- and feedback-efficiency in human-in-the-loop RL.
- The framework applies to higher-complexity tasks, including locomotion and robotic manipulation skills.
- PEBBLE learns novel behaviors and avoids reward exploitation, producing more desirable behaviors than an agent trained with an engineered reward function.
A. State Entropy Estimator
PEBBLE uses state entropy as an intrinsic reward and approximates it with a particle-based k-nearest-neighbors entropy estimator because the underlying density is difficult to compute in high dimensions.
- The method approximates state entropy with a simplified particle-based entropy estimator.
- A particle-based k-nearest-neighbors estimator is employed because estimating the state density directly is difficult with high-dimensional data.
- The estimator uses the k-nearest neighbor of each sample, a bias-correction term, the digamma function, the gamma function, and the state dimension.
B. Experimental Details
Experiments use SAC-based PEBBLE with task-specific feedback-query budgets and a three-layer ensemble reward model. Evaluation covers DMControl locomotion and randomized Meta-world manipulation settings.
- PEBBLE uses the publicly released SAC implementation, with hyperparameters listed in Table 1.
- DMControl experiments use 50-step segments and feedback every 20K timesteps, with query-session sizes varying by environment and feedback budget.
- Meta-world experiments use 10-step segments, K = 2400, and feedback budgets of 2500, 5000, and 10000.
- Preference PPO baselines use the publicly released PPO implementation and feedback-query schedules specified separately for DMControl and Meta-world.
- The reward model is a three-layer neural network with 256 hidden units per layer, trained as an ensemble of three models with tanh-bounded outputs.
- Meta-world evaluation randomizes reset and goal positions to create a more realistic single-task manipulation setting.
C. Effects of Sampling Schemes
Uncertainty-based sampling improves PEBBLE’s performance on Quadruped relative to uniform sampling, but provides little additional benefit on simpler locomotion and manipulation tasks. Figure 8 reports means and standard deviations across ten runs.
- On Quadruped, uncertainty-based sampling using ensemble disagreement or entropy outperforms naive uniform sampling.
- On simpler environments such as Walker and Cheetah, uncertainty-based sampling provides no extra gains.
- Robotic manipulation tasks likewise show little performance difference between sampling schemes when the tasks are simpler.
- Figure 8 shows PEBBLE learning curves with 1400 feedback pieces across sampling schemes, reporting means and standard deviations over ten runs.
D. Examples of Selected Queries
This section presents examples of selected queries used to teach the Cart, Quadruped, and Hopper agents, alongside Meta-world learning curves under different sampling schemes.
- Figures 10–12 show selected-query examples for teaching the Cart, Quadruped, and Hopper agents.The accompanying text identifies these figures as examples from the selected queries used to teach the agents.
- Figure 9 compares PEBBLE learning curves across various sampling schemes on Meta-world tasks.
- The curves in Figure 9 report means and standard deviations across ten runs.Solid lines represent means, while shaded regions represent standard deviations.