PaperlayerView exact revision on arXiv

Source-linked AI summary

Preference Transformer: Modeling Human Preferences using Transformers for RL

Changyeon Kim, Jongjin Park, Jinwoo Shin, Honglak Lee, Pieter Abbeel, Kimin Lee

arXiv:2303.00957v1cs.LGcs.AIcs.RO

TL;DR

Preference-based RL can require substantial human feedback to learn rewards aligned with human intent. Preference Transformer models preferences as weighted non-Markovian rewards using causal and bidirectional self-attention, and outperforms current preference-modeling methods across complex control benchmarks.

  • Problem

    Preference-based RL can require substantial human feedback to learn a reward function aligned with human intent, while common models assume Markovian rewards and equal weighting across a trajectory.

    Source: §1 · p. 1, §4.1 · p. 4

  • Method

    Preference Transformer uses a transformer architecture to model preferences as a weighted sum of non-Markovian rewards, generating rewards and importance weights from trajectory segments.

    Source: §1 · p. 2, §4.1 · p. 4, §4.2 · p. 5

  • Results

    Preference Transformer significantly outperforms current preference-modeling methods on complex navigation, locomotion, and robotic manipulation tasks from offline RL benchmarks.

    Source: §6 · p. 9

  • Takeaways & Limitations

    The learned preference attention captures events critical to human decisions, and the method learns a double backflip behavior that a Markovian reward agent fails to learn.

    Source: §6 · p. 9, §5.5 · p. 9

    The human feedback was collected from domain-expert authors; evaluation using public crowdsourcing platforms remains future work.

    Source: ETHICS STATEMENT · p. 10

Abstract

from arXiv · show

Preference-based reinforcement learning (RL) provides a framework to train agents using human preferences between two behaviors. However, preference-based RL has been challenging to scale since it requires a large amount of human feedback to learn a reward function aligned with human intent. In this paper, we present Preference Transformer, a neural architecture that models human preferences using transformers. Unlike prior approaches assuming human judgment is based on the Markovian rewards which contribute to the decision equally, we introduce a new preference model based on the weighted sum of non-Markovian rewards. We then design the proposed preference model using a transformer architecture that stacks causal and bidirectional self-attention layers. We demonstrate that Preference Transformer can solve a variety of control tasks using real human preferences, while prior approaches fail to work. We also show that Preference Transformer can induce a well-specified reward and attend to critical events in the trajectory by automatically capturing the temporal dependencies in human decision-making. Code is available on the project website: https://sites.google.com/view/preference-transformer.

1 INTRODUCTION

Reward engineering can require substantial task knowledge, while preference-based RL learns rewards from human comparisons but remains difficult to scale. The paper proposes Preference Transformer to model preferences using weighted non-Markovian rewards and reports solving control tasks from real human preferences.

1 INTRODUCTION

  • Reward engineering requires extensive instrumentation and human effort to combine objectives such as velocity, energy use, and torso verticality into one scalar.Source: §1 · p. 1
  • Preference-based RL uses relative judgments between trajectories to learn rewards, but existing approaches still require substantial human feedback to scale across applications.Source: §1 · p. 1
  • Prior models commonly assume Markovian rewards and equally weighted rewards across a trajectory, assumptions that may not reflect human judgments.Source: §1 · p. 1
  • Earlier events can influence judgments of later events, and humans’ sensitivity to remarkable moments makes within-trajectory credit assignment relevant.Source: §1 · p. 2
  • Preference Transformer models preferences with a weighted sum of non-Markovian rewards, using stacked causal and bidirectional self-attention to produce rewards and importance weights.Source: §1 · p. 2
  • Preference Transformer solves complex navigation, locomotion, and robotic manipulation tasks by learning rewards from real human preferences, and its learned rewards capture critical trajectory events.The authors also analyze the learned reward function and importance weights.Source: §1 · p. 2

2 RELATED WORK

Related work covers preference-based reward learning, transformer applications in reinforcement and imitation learning, and non-Markovian reward learning. Preference Transformer brings transformer-based modeling of temporal dependencies in human decisions to preference-based control learning.

2 RELATED WORK

  • Preference-based RL learns rewards from human preferences; prior work has pursued feedback efficiency through pre-training, data augmentation, exploration, and meta-learning.Source: §2 · p. 2
  • Unlike transformer reward models for language tasks, this work uses transformers with a new preference model for control tasks that captures temporal dependencies in human decisions.Source: §2 · p. 3
  • Transformers have been applied to reinforcement and imitation learning, including sequential formulations of offline RL and one-shot, multi-domain, and multi-task imitation.Source: §2 · p. 3
  • Prior non-Markovian reward learning includes LSTM-based return decomposition and methods that distribute episodic return uniformly across timesteps.Source: §2 · p. 3

3 PRELIMINARIES

The preliminaries define the discounted-return objective for reinforcement learning and the human preference data used to learn rewards. They then describe the standard preference-prediction and policy-optimization steps.

3 PRELIMINARIES

  • An RL agent receives a state, selects an action, receives a reward, and seeks a policy that maximizes expected discounted cumulative reward.The discount factor is γ.Source: §3 · p. 3
  • A preference dataset stores pairs of trajectory segments and a label indicating whether the first or second is preferred, or whether they are equally preferable.Source: §3 · p. 3
  • Prior work predicts preferences using the Bradley–Terry model, updates the reward by cross-entropy against human labels, and then optimizes a policy using the learned reward.Source: §3 · p. 3

4 PREFERENCE TRANSFORMER

Preference Transformer replaces equal-weight Markovian preference scoring with a weighted non-Markovian reward model, then implements it with causal and bidirectional transformer attention. It is trained on human preference labels and supplies learned rewards during RL.

4.1 PREFERENCE MODELING

  • The predictor replaces equal-weight Markovian returns with preferences modeled over weighted sums of non-Markovian rewards, enabling within-segment credit assignment.The formulation generalizes the conventional model when rewards are state-action dependent and all importance weights equal 1.Source: §4.1 · p. 4

4.2 ARCHITECTURE

  • A causal GPT backbone builds prefix-dependent embeddings, then bidirectional preference attention produces reward and weight outputs whose average represents the segment’s weighted reward sum.The preference attention layer maps hidden embeddings to reward values and importance weights.Source: §4.2 · p. 5

4.3 TRAINING AND INFERENCE

  • Preference Transformer is trained with cross-entropy against human preference labels, then supplies learned rewards using the preceding H transitions during RL.At timestep t, inference uses the t-th reward value from the preference attention layer.Source: §4.3 · p. 5

5 EXPERIMENTS

The experiments evaluate Preference Transformer (PT) on offline control tasks with real and scripted preferences, testing its reward learning, human alignment, and ability to learn complex behaviors.

5.1 SETUPS

  • Using real human feedback, experiments compare PT, MLP-based MR, and LSTM-based NMR with IQL across AntMaze, locomotion, and robotic manipulation tasks.The study uses offline datasets, IQL with Markovian policy and value functions, and reports means and standard deviations across eight runs.Source: §5.1 · pp. 6–7, Table 1 · p. 6

5.2 BENCHMARK TASKS WITH REAL HUMAN TEACHERS

  • PT consistently outperforms preference-modeling baselines on almost all real-human-preference tasks and nearly matches task-reward IQL, while MR and NMR fail on hard tasks.The largest gap is on complex AntMaze tasks, where long-term behavior context and critical events matter.Source: Table 1 · p. 6, §5.2 · p. 7

5.3 REWARD AND WEIGHT ANALYSIS

  • In AntMaze trajectories, learned rewards rise as the agent approaches the goal and remain low when it struggles or flips, while importance-weight spikes mark turning or flipping events.These patterns appear in successful and failure trajectory segments.Source: Figure 3 · p. 7, §5.3 · p. 8

5.4 BENCHMARK TASKS WITH SCRIPTED TEACHERS

  • PT performs strongly with scripted as well as human teachers, but scripted teachers often disagree with humans and can yield misleading evaluations.The scripted teacher misses behavioral context such as direction; disagreement is high across tasks.Source: §5.4 · pp. 8–9, Figure 6 · p. 9

5.5 LEARNING COMPLEX NOVEL BEHAVIORS

  • With 300 human-feedback queries, Hopper learns a double backflip, whereas an agent using a Markovian reward fails to learn it.The task requires capturing non-Markovian context, including the number of rotations.Source: §5.5 · p. 9, Figure 7 · p. 9

6 DISCUSSION

Preference Transformer models human preferences with weighted non-Markovian rewards and a transformer architecture. It outperforms current preference-modeling methods across several control domains and its attention captures events critical to human decisions.

6 DISCUSSION

  • Preference Transformer significantly outperforms current preference-modeling methods on complex navigation, locomotion, and robotic manipulation tasks.The experiments use tasks from offline RL benchmarks.Source: §6 · p. 9
  • A preference attention layer uses self-attention to infer each reward's importance within a weighted sum of non-Markovian rewards.The framework models temporal dependencies in human decisions.Source: §6 · p. 9
  • The learned preference attention captures events critical to human decisions.The paper proposes this as a way to scale preference-based RL and other human-in-the-loop learning to various applications.Source: §6 · p. 9

FUTURE DIRECTIONS

The paper identifies several future directions: using importance weights in reward learning or RL, and combining Preference Transformer with other preference models.

FUTURE DIRECTIONS

  • Importance weights could guide query sampling to improve feedback efficiency in preference-based reward learning.The cited examples include query sampling methods from Sadigh et al. (2017) and Lee et al. (2021c).Source: FUTURE DIRECTIONS · p. 10
  • Importance weights could also be used for stabilizing Q-learning through weighted updates.The paper cites Kumar et al. (2020) and Lee et al. (2021a) as examples.Source: FUTURE DIRECTIONS · p. 10
  • Combining Preference Transformer with regret-based preference models is an open direction.The paper notes that Knox et al. (2022) use segment regret in simple grid worlds and make assumptions including generated successor features.Source: FUTURE DIRECTIONS · p. 10

ETHICS STATEMENT

The authors collected feedback from robotics and RL experts and leave evaluation with public crowd-sourcing platforms to future work. The described tasks span navigation, locomotion, and robotic manipulation benchmarks.

ETHICS STATEMENT

  • The feedback dataset was labeled by authors familiar with robotics, RL, and the target tasks; evaluation using public crowd-sourcing platforms remains future work.The authors expect their collection strategy to provide high-quality, clean labels and to be closer to practice.Source: ETHICS STATEMENT · p. 10

A TASKS AND DATASETS

  • AntMaze tests a Mujoco Ant reaching goals in medium or large mazes, using diverse or play datasets and sparse goal rewards.Diverse data randomizes start and goal locations; play data uses hand-picked goals.Source: A TASKS AND DATASETS · p. 15
  • Gym-Mujoco locomotion tasks require Walker2d or Hopper robots to move forward while minimizing energy cost, using medium-expert or medium-replay datasets.Rewards combine torso forward velocity, control penalty, and survival bonus.Source: A TASKS AND DATASETS · p. 15
  • Robosuite experiments use simulated Panda robots for cube lifting and can placement, with data from proficient or mixed-proficiency teleoperators.The tasks use sparse rewards.Source: A TASKS AND DATASETS · p. 15

B EXPERIMENTAL DETAILS

The experimental details describe Preference Transformer and baseline configurations, evaluation procedures, and human-feedback collection across control tasks. Human teachers compare short video segments using task-specific guidance.

B EXPERIMENTAL DETAILS

  • The learned reward is normalized to improve IQL training stability, and training uses eight random seeds.Reward learning takes under 10 minutes in AntMaze with 1,000 human feedback; IQL training takes about an hour on the stated hardware.Source: B EXPERIMENTAL DETAILS · p. 15
  • Preference Transformer uses one causal transformer layer with four attention heads, followed by a bidirectional self-attention layer with one head.The model is trained with AdamW, a learning rate of 1×10^-4, and batch size 256.Source: B EXPERIMENTAL DETAILS · p. 15
  • Evaluation uses 100 AntMaze rollouts, 10 Gym-Mujoco rollouts, and 50 Robosuite rollouts; metrics are normalized scores for the first two and success rate for Robosuite.The normalized score follows the D4RL measure.Source: B EXPERIMENTAL DETAILS · pp. 15–16
  • The Markovian baseline is a two-layer MLP, while the non-Markovian baseline reimplements CSC Instance Space LSTM.The LSTM hidden dimensions are doubled to match the non-Markovian reward model and Preference Transformer.Source: B EXPERIMENTAL DETAILS · p. 16
  • For Hopper backflips, three reward-learning stages with 100 human-feedback queries each were enough to perform double backflips.The setup pre-trains the policy for 10,000 timesteps and repeats reward learning and replay-buffer relabeling every 10,000 timesteps.Source: B EXPERIMENTAL DETAILS · p. 16

C HUMAN PREFERENCES

  • Human teachers compare three-second video segments and may mark two segments equally preferred when they cannot decide.Instructions prioritize task-specific behavior, including reaching AntMaze goals, stable locomotion, or grasping and moving the Robosuite object.Source: C HUMAN PREFERENCES · pp. 16–17

D FULL EXPERIMENTAL RESULTS WITH SCRIPTED TEACHER

The section reports reward-model comparisons, examples of human–scripted teacher disagreements, and videos showing learned importance weights aligned with human intent.

D FULL EXPERIMENTAL RESULTS WITH SCRIPTED TEACHER

  • The experiments compare Preference Transformer, Markovian-reward models, and non-Markovian-reward models across AntMaze, locomotion, and manipulation tasks using scripted and human preferences.The table caption reports averages and standard deviations over eight runs but supplies no cell values.Source: Table 3 · p. 17

E ADDITIONAL EXAMPLES OF QUERIES

  • Human judgments reject a flipped ant trajectory despite its goal proximity, while the scripted teacher favors it; another example highlights falling as context the hand-designed reward misses.The examples illustrate humans balancing multiple objectives where scripted teachers do not.Source: E ADDITIONAL EXAMPLES OF QUERIES · p. 18, Figure 8 · p. 18

F DESCRIPTION OF THE VIDEO EXAMPLES

  • Learned importance weights align with human intent, and the supplementary videos visualize higher-weight frames with brighter colors.The weights appear as a color map on the video rim.Source: F DESCRIPTION OF THE VIDEO EXAMPLES · p. 18

G EXPERIMENTAL RESULTS USING NON-MARKOVIAN MODELS

With non-Markovian policy and value functions, Preference Transformer outperforms the LSTM-based non-Markovian reward model, although overall performance declines relative to the Markovian setup.

G EXPERIMENTAL RESULTS USING NON-MARKOVIAN MODELS

  • Preference Transformer outperforms NMR with non-Markovian policy and value functions, while all methods perform worse overall than with Markovian policy and value functions.The authors expect the decline because IQL was developed and its hyperparameters tuned for a Markovian setup.Source: G EXPERIMENTAL RESULTS USING… · p. 19
  • The comparison uses non-MDP IQL on AntMaze and Gym-Mujoco locomotion tasks, with reward models trained on the same real-human preference dataset.The table caption reports averages and standard deviations over eight runs but provides no values.Source: Table 4 · p. 19
  • For the non-Markovian setup, the previous timestep’s predicted reward is supplied as an additional input.Source: G EXPERIMENTAL RESULTS USING… · p. 19

Related papers

Loading 2303.00957v1…