PaperlayerView exact revision on arXiv

Source-linked AI summary

Few-Shot Preference Learning for Human-in-the-Loop RL

Joey Hejna, Dorsa Sadigh

arXiv:2212.03363v1cs.ROcs.AIcs.LG

TL;DR

Reward learning for robotics must capture human intent, but prior preference-based methods either restrict reward models or demand impractical numbers of queries. This paper pre-trains preference models on prior tasks and adapts them with a handful of human queries. It reports a 20× feedback reduction on Meta-World benchmarks and demonstrates training with real human feedback.

  • Problem

    Robot reward functions are difficult to design to capture human intent, while preference-learning methods can require impractical query counts or restrict reward-model expressiveness.

    Source: §1 · pp. 1–2

  • Method

    The method meta-learns preference reward models from prior-task data, then adapts them to new tasks using a handful of human comparisons.

    Source: §1 · p. 2, §3 · p. 4

  • Results

    20× less feedback than PEBBLE’s maximum budget still solves almost all Meta-World tasks with near-100% success, and the paper reports training policies from real human feedback.

    Source: §1 · p. 2, §4.1 · p. 6

  • Takeaways & Limitations

    The multi-task approach is presented as a way to use prior-task data to improve query efficiency and train manipulation policies from real human feedback.

    Source: §5 · p. 8

    Reward adaptation under larger distribution shifts remains uncertain, and training from scratch may perform better for significantly out-of-distribution tasks.

    Source: §5 · p. 8

Abstract

from arXiv · show

While reinforcement learning (RL) has become a more popular approach for robotics, designing sufficiently informative reward functions for complex tasks has proven to be extremely difficult due their inability to capture human intent and policy exploitation. Preference based RL algorithms seek to overcome these challenges by directly learning reward functions from human feedback. Unfortunately, prior work either requires an unreasonable number of queries implausible for any human to answer or overly restricts the class of reward functions to guarantee the elicitation of the most informative queries, resulting in models that are insufficiently expressive for realistic robotics tasks. Contrary to most works that focus on query selection to \emph{minimize} the amount of data required for learning reward functions, we take an opposite approach: \emph{expanding} the pool of available data by viewing human-in-the-loop RL through the more flexible lens of multi-task learning. Motivated by the success of meta-learning, we pre-train preference models on prior task data and quickly adapt them for new tasks using only a handful of queries. Empirically, we reduce the amount of online feedback needed to train manipulation policies in Meta-World by 20$\times$, and demonstrate the effectiveness of our method on a real Franka Panda Robot. Moreover, this reduction in query-complexity allows us to train robot policies from actual human users. Videos of our results and code can be found at https://sites.google.com/view/few-shot-preference-rl/home.

1 Introduction

Reward design makes robot RL difficult: sparse rewards provide little learning signal, while dense rewards can misalign with human intent. The paper proposes pre-training reward models across tasks and adapting them with a handful of human queries.

1 Introduction

  • Sparse rewards hamper exploration, while dense hand-designed rewards can be difficult to specify and exploited in ways that diverge from human intent.These challenges become more pronounced across realistic multi-task settings with large state and action spaces.Source: §1 · p. 1
  • Prior preference-learning methods either constrain reward models to simplify query selection or require thousands of queries, motivating a multi-task approach that expands the data available for learning.The paper proposes pre-training on prior tasks and adapting reward models to new preferences with only a handful of queries.Source: §1 · p. 2
  • 20× less feedback than PEBBLE’s maximum budget lets the method solve almost all Meta-World tasks with near-100% success, and the work also reports learning policies from real human feedback.The introduction identifies real-world Franka Panda experiments as well.Source: §1 · p. 2

2 Related Work

Related work spans reward learning from demonstrations and other human signals, preference-based learning, and meta-learning. The paper distinguishes its approach by using prior-task data to pre-train preference reward models for few-shot adaptation.

2 Related Work

  • Preference-learning methods use pairwise comparisons, while other reward-learning approaches draw on demonstrations, corrections, language, scalar scores, or rankings.Demonstrations may be expensive or misaligned with human preferences, and stronger signals such as scores or multiple rankings can be harder for users to provide.Source: §2 · p. 2
  • Active preference-learning methods often assume linearity in predefined features, while deep preference-learning methods can require feedback volumes impractical for human users.The cited discussion notes PEBBLE’s requirement of 25k queries for drawer opening.Source: §2 · pp. 2–3
  • Unlike PEBBLE’s single-task setting, this work draws on diverse data from multiple prior tasks to improve query efficiency through pre-training.The overview depicts prior-task trajectory comparisons training a reward model, followed by human-feedback adaptation for a new task.Source: §2 · p. 3, Figure 1 · p. 3

3 Few-Shot Preference Learning for RL

The method pre-trains a preference reward model across prior tasks, then adapts it to a new task from a handful of human comparisons. It uses that reward to train a policy through standard RL updates.

3 Few-Shot Preference Learning for RL

  • For a new task, the goal is to learn a policy from human feedback using as few queries as possible, without restricting prior-task data to a particular form.The prior data must contain enough information to estimate each task’s reward.Source: §3 · p. 3
  • The method learns rewards from labeled pairwise comparisons of partial trajectory segments, using a preference predictor and binary cross-entropy loss.The loss raises predicted rewards for preferred segments relative to unpreferred ones.Source: §3 · p. 4
  • MAML pre-trains reward-model parameters on prior-task preference datasets so they can adapt to a new task after only a few gradient steps.Prior-task data may come from offline datasets, simulated policies, or actual humans.Source: §3 · p. 4
  • Offline pre-training comparisons can be generated from known rewards, recovered through inverse RL, or derived from behavior; experiments use artificial reward labels.For behavior-based comparisons, trajectories from the task’s policy are preferred over those from another task’s policy.Source: §3 · pp. 4–5
  • During online learning, an ensemble selects comparisons by maximizing disagreement, then the reward model resets to its pre-trained weights and adapts on new feedback.The updated reward relabels transitions before SAC performs its actor-critic update.Source: §3 · p. 5
  • The approach’s key algorithmic changes are pre-training rewards from prior data and resetting the reward model for new-task adaptation, without structural assumptions on reward functions.The paper presents this as a multi-task preference-based RL setting.Source: §3 · p. 5

4 Experiments

The experiments test query efficiency, learning from human preferences, and transfer to real-robot tasks. Few-Shot uses 20× fewer feedback queries while achieving near-100% success on almost all Meta-World tasks, and it outperforms PEBBLE in human-feedback and real-robot evaluations.

4.1 How query-efficient is few-shot preference-based RL?

  • The 4.1 evaluation uses Meta-World tasks, including sequential manipulation problems, and compares Few-Shot with SAC, PEBBLE, and an initialization-only baseline.The initialization baseline uses pretrained reward-model weights without re-adaptation after new feedback.Source: §4.1 · p. 6
  • 20× fewer feedback queries than PEBBLE’s maximum budget still solves almost all tasks near 100% success; all but Button Press match SAC asymptotically, while PEBBLE fails at that reduced budget.The evaluation uses 10 prior tasks and six unseen Meta-World tasks.Source: §4.1 · p. 6

4.2 Can few-shot preference learning be used with humans?

  • Human-feedback tests across two Meta-World and two DM Control environments found that Few-Shot outperformed PEBBLE, and users found its Meta-World queries easier to answer.The procedure let users skip difficult queries and increased uniform queries, but users still made potentially inconsistent labeling errors.Source: §4.2 · pp. 6–7, Figure 3 · p. 7

4.3 Can Few-Shot preference-based RL be used in the real world?

  • On Franka Panda reach and block-pushing tasks, Few-Shot consistently outperformed PEBBLE despite the sim-to-real gap; Table 1 reports goal-distance error in meters, with lower values better.The tasks use unseen goal locations, and results are averaged across multiple initial configurations.Source: Table 1 · p. 7, §4.3 · p. 7

5 Discussion

The paper shifts human-in-the-loop RL from single-task to multi-task learning, using prior-task data to improve preference-learning query efficiency. It also identifies limits in feedback needs, pre-training transfer, query difficulty, and human consistency.

Query Complexity

  • The most complex tasks still require more feedback than desired, motivating expansion of the pre-training data set.Source: §5 · p. 8

Pre-training Methods

  • Pre-training uses artificial query data for generality, while other objectives such as direct reward regression could further boost performance.Source: §5 · p. 8

Pre-training Data

  • Reward adaptation under large shifts from sub-optimal users, new tasks, or sim-to-real transfer remains uncertain; out-of-distribution tasks may favor training from scratch.Pre-training can also over-regularize the reward model, as in Door Close.Source: §5 · p. 8

Query Difficulty

  • Active preference queries can be too difficult for users, so future query design should consider answerability as well as theoretical information content.Source: §5 · p. 8

User Inconsistency

  • Human users may label queries inconsistently, with differing styles making this challenge worse in crowdsourced data.The authors suggest additionally modeling human users as future work.Source: §5 · p. 8

Conclusion

  • The method reduces human queries for complex manipulation policies by 20× on standard benchmarks and is effective with real-human feedback.The conclusion presents the multi-task shift as a way to use additional data sources to boost preference-based RL query efficiency.Source: §5 · p. 8

A.1 Ablations

The ablations show that few-shot preference learning can match PEBBLE with substantially less feedback, and that its advantage persists when performance is plotted against feedback rather than training steps.

A.1 Ablations

  • With 20× less feedback than PEBBLE, the few-shot method remains at par; halving its own feedback nearly preserves performance in Window Close, Door Unlock, and Sweep-Into.The figure compares PEBBLE using the feedback amount from Lee et al. with the few-shot method using half its Figure 2 amount.Source: §A.1 · p. 13, Figure 6 · p. 13
  • Disagreement sampling improves performance in some environments, particularly Drawer Open, but makes no difference in others.Source: §A.1 · p. 13

A.2 Plots of Feedback versus Performance

  • Plotting performance against total feedback preserves the overall trend: the few-shot method outperforms baselines for the feedback provided.The original environment-step plots also show convergence after feedback stops and allow SAC to appear on the same graph.Source: §A.2 · p. 13

A.3 Locomotion Experiments

In Cheetah Velocity, the few-shot method learns a reward for an unseen temporal target using data from other velocities and needs far fewer queries than PEBBLE to converge.

A.3 Locomotion Experiments

  • The Cheetah Velocity task uses an unseen target speed of 1.5 m/s, with 10 other velocities used for pre-training.The task tests a temporal goal that cannot be represented by an environment configuration.Source: §A.3 · p. 13
  • Around 100 queries suffice for the few-shot method to converge across feedback schedules, while PEBBLE does not reach comparable performance even with 1,000 queries.The comparison runs for 500,000 environment steps, and the method learns the ground-truth reward quickly.Source: §A.3 · p. 14, Figure 10 · p. 15
  • The initial baseline performs similarly, but has slightly worse asymptotic convergence in two of the three feedback schedules.The authors report minimal hyperparameter tuning in these environments.Source: §A.3 · p. 14

A.4 Comparison with Example Based Methods

The comparison finds that preference-based learning can outperform example-based RCE on temporal and sparse tasks, including when RCE has expert data.

A.4 Comparison with Example Based Methods

  • Example-based methods often assume a goal-state reward and optimize sparse-like rewards, which can hinder temporal objectives and longer-horizon optimization.For temporal tasks such as driving, examples may not capture intermediate safety and comfort goals.Source: §A.4 · p. 15
  • With 200 examples or feedback items, Few-Shot outperforms RCE in Cheetah Velocity, even when RCE uses an expert trajectory, and converges sooner.Figure 11 covers Cheetah Velocity and Point Maze; the Cheetah expert examples come from a pretrained oracle policy.Source: §A.4 · p. 15, Figure 11 · p. 16
  • RCE performance collapses when trained on relabeled target-velocity states without difficult-to-specify joint information.Source: §A.4 · pp. 15–16
  • In sparse Point Mass Barrier, RCE cannot overcome exploration difficulty and the long horizon, while preference learning nearly matches oracle SAC.The authors identify temporal behavior that is hard to specify with examples and extremely sparse tasks as settings where preference-based learning excels.Source: §A.4 · p. 16

A.5 Human Feedback

Across four DM Control Reacher users, Few-Shot outperformed PEBBLE, though feedback accuracy constrained its robustness. Users’ preference accuracy tracked final reward, with performance declining below 75% agreement with ground truth.

A.5 Human Feedback

  • Across all four users, policies trained with Few-Shot outperformed those trained with PEBBLE.Each user supplied 48 preferences per policy.Source: §A.5 · p. 16
  • Users did not find feedback significantly more difficult with Few-Shot than with PEBBLE on this task.This differed from the MetaWorld tasks.Source: §A.5 · p. 16
  • Performance was relatively robust until preference agreement with ground truth fell below 75%, after which it began to decline.The authors identify inaccurate user feedback as a limitation for reward adaptation.Source: §A.5 · p. 16

A.6 Franka Panda Experiments

The appendix provides Franka Panda learning curves and details the environments, datasets, and evaluation settings used across Meta-World, DM Control, Franka Panda, and locomotion experiments.

B Experiment Details

  • The experiment-details section explains that environment, evaluation, and dataset-generation specifics are included for the few-shot preference-based RL evaluation.This is an overview of the section’s scope.Source: B Experiment Details · p. 16

A.6 Franka Panda Experiments

  • Figure 13 presents learning curves for Franka Panda models omitted from the main paper for space constraints.The supplied text does not report specific outcomes from those curves.Source: §A.6 · p. 16, Figure 13 · p. 17

B.1 Meta-World

  • Meta-World pretraining data cover 10 prior tasks, each with 25 parametric variations; for each variation, 6,000 queries are sampled uniformly and labeled with ground-truth reward.Source: §B.1 · pp. 17–18

B.2 DM Control

  • DM Control pretraining uses 25,000 random timesteps across 16 Point Mass goals and 12 Reacher goals, generating 4,000 artificial queries per task.Evaluation uses unseen goals for Point Mass and Reacher, and a specified goal for Point Mass Barrier.Source: §B.2 · p. 18

B.3 Franka Panda

  • The Franka Panda experiments use end-effector control for reach and block pushing, with unknown goal locations and distance-based rewards.The block’s position is observed, but its goal location is not.Source: §B.3 · p. 18
  • Policies transfer from simulation to a real Franka Panda, with success defined within 2.5 cm for reach and 5 cm for block pushing.The setup uses two RealSense cameras and ArUco tracking; results are reported in final meters to the goal.Source: §B.3 · p. 19, Figure 14 · p. 19

B.4 Locomotion

  • Locomotion varies the target speed because its reward depends on behavior over time; the unseen 1.5 m/s velocity is used for evaluation.Ten training velocities range from 0.25 to 2.75 m/s in 0.25 m/s increments.Source: §B.4 · p. 19

B.5 Human Experiments

Human experiments compare Few-Shot and PEBBLE with users answering queries during policy training; the appendix also reports implementation settings and illustrative query comparisons.

B.5 Human Experiments

  • Three additional users trained two PEBBLE and two Few-Shot policies per environment, with queries answered intermittently during parallel training.Total query-answering time ranged from about 22 minutes for Point Mass to about one hour for Door Close; users skipped queries when uncertain.Source: §B.5 · pp. 19–20
  • In harder environments, users skipped fewer queries and were more accurate with Few-Shot; visualizations suggest its queries were easier to distinguish, likely due to pretraining.The human study also tracked skipped queries and preference accuracy relative to ground-truth rewards.Source: §B.5 · p. 20

C Hyperparameters

  • The appendix specifies separate random-seed counts for Meta-World, human-feedback, and real-world experiments, and tabulates pretraining, algorithm, and feedback-schedule settings.Meta-World artificial-feedback experiments use five seeds per method; human-feedback experiments use two.Source: C Hyperparameters · p. 20, Table 2 · p. 20, Table 3 · p. 21
  • Pretraining uses MAML with Adam and learned inner learning rates; online adaptation allows up to 40 MAML steps before further Adam training if accuracy remains below 95%.All methods use a constant feedback schedule and omit prior unsupervised exploration.Source: C Hyperparameters · p. 20

D Additional Visualizations

Loading 2212.03363v1…