PaperlayerView exact revision on arXiv

Source-linked AI summary

Weak Human Preference Supervision For Deep Reinforcement Learning

Zehong Cao, KaiChiu Wong, Chin-Teng Lin

arXiv:2007.12904v2cs.AI

TL;DR

Reward learning from human preferences can address complex RL tasks, but fixed trajectory choices are not dynamic and require thousands of human judgments. The paper introduces graded preference scaling and a supervised estimator to predict some preferences. In MuJoCo experiments, the framework achieves higher cumulative rewards than fixed preferences, while the estimator reduces human input by up to 30%.

Abstract

from arXiv · show

The current reward learning from human preferences could be used to resolve complex reinforcement learning (RL) tasks without access to a reward function by defining a single fixed preference between pairs of trajectory segments. However, the judgement of preferences between trajectories is not dynamic and still requires human input over thousands of iterations. In this study, we proposed a weak human preference supervision framework, for which we developed a human preference scaling model that naturally reflects the human perception of the degree of weak choices between trajectories and established a human-demonstration estimator via supervised learning to generate the predicted preferences for reducing the number of human inputs. The proposed weak human preference supervision framework can effectively solve complex RL tasks and achieve higher cumulative rewards in simulated robot locomotion -- MuJoCo games -- relative to the single fixed human preferences. Furthermore, our established human-demonstration estimator requires human feedback only for less than 0.01\% of the agent's interactions with the environment and significantly reduces the cost of human inputs by up to 30\% compared with the existing approaches. To present the flexibility of our approach, we released a video (https://youtu.be/jQPe1OILT0M) showing comparisons of the behaviours of agents trained on different types of human input. We believe that our naturally inspired human preferences with weakly supervised learning are beneficial for precise reward learning and can be applied to state-of-the-art RL systems, such as human-autonomy teaming systems.

I. INTRODUCTION

Reward functions can be difficult to design for complex tasks, while fixed trajectory preferences require repeated human judgments and may not reflect natural intent. The paper proposes weak, graded preferences and a supervised estimator to reduce human input.

I. INTRODUCTION

  • Designing an effective reward function for complicated scenarios is challenging, and overly simple designs may produce behavior misaligned with expectations.The introduction also notes challenges with inverse reinforcement learning and imitation learning, including unavailable human demonstrations when robot movements exceed human movement.Source: §I · p. 1
  • Fixed trajectory comparisons offer only left, right, or equal choices and require human input over thousands of iterations, limiting how naturally preferences can be expressed.The paper represents these fixed choices as preference values 1, 0, and 0.5, respectively.Source: §I · p. 1
  • The framework combines a scaling model for dynamic, weak human preferences with a supervised human-demonstration estimator to predict preference values from initial human inputs.The authors report higher cumulative rewards than fixed preferences in five MuJoCo experiments and up to 30% fewer human inputs without significantly sacrificing reward values.Source: §I · pp. 1–2

A. Preference Learning

The related work describes preference learning, pairwise comparisons, and the feedback pathway that integrates human judgments into reinforcement learning. It distinguishes absolute, equal, and weak preferences within that context.

A. Preference Learning

  • Prior work links preference learning to better actions and robotic movement, while DemPref requires collecting many purely human-demonstrated actions before training.The paper also describes a model that adjusts the attitude and importance of relative criteria without improving preference coverage.Source: A. Preference Learning · p. 2

B. Learning by Pairwise Comparisons

  • Pairwise comparisons let humans rank two options, with notation distinguishing absolute preference, indifference, and weak preference.The cited survey contrasts two-option feedback in preference learning with the single feedback option accepted in RL.Source: B. Learning by Pairwise Comparisons · p. 2

C. RL from Human Preferences

  • RL from human preferences connects an agent, a preference interface that requests judgments, and a reward predictor that uses those judgments to train policies.The predictor estimates relative rewards from the stored preferences, which are returned to the agent for policy training.Source: C. RL from Human Preferences · p. 2

D. Motivations

The paper motivates weak preferences and preference estimation as responses to fixed-choice limitations and extensive labeling demands. It describes a synthetic scaling procedure that maps segment rewards to graded preference values.

D. Motivations

  • Christiano’s setup omits weak-preference conditions and requires at least 5,500 human inputs, motivating coverage of weak choices and supervised estimation to reduce labeling effort.The authors also suggest that fatigue may increase preference-judgment errors over long sessions.Source: D. Motivations · p. 2

A. Settings and Goal

  • The agent maximizes cumulative predicted rewards while the interface queues trajectory clips, asks humans to compare randomly selected pairs, and passes judgments to the reward predictor.The study modifies the fixed left, right, or equal input design to support weak-preference conditions.Source: A. Settings and Goal · p. 3

B. Weak Human Preferences: Synthetic Preference Scaling

  • Synthetic preference scaling assumes the preferred segment has higher task reward and maps relative segment rewards to values from 0.0 to 1.0, with 0.5 denoting equal preference.The method normalizes reward values after removing the lowest and highest 10% to limit the influence of outliers.Source: B. Weak Human Preferences: Synthetic… · p. 3

C. Human Preference Scaling for Deep RL

The preference-learning loop updates the policy and reward model using human comparisons of sampled trajectory segments. Its three processes pass trajectories, preferences, and model parameters asynchronously.

C. Human Preference Scaling for Deep RL

  • The agent trains a policy to maximize the sum of predicted rewards from its observation-action pairs.The reward function is estimated by a deep neural network.Source: C. Human Preference Scaling for Deep RL · p. 3
  • The system randomly selects a pair of trajectory segments and sends them to the preference interface for human comparison.The segments are selected from trajectories generated by the current policy.Source: C. Human Preference Scaling for Deep RL · p. 3
  • A human’s preference scaling is linked to the segment pair, updates the reward model, and returns model parameters to policy training asynchronously.The paper describes a handoff from policy training to segment comparison, then preference updates and back to policy training.Source: C. Human Preference Scaling for Deep RL · p. 4

Preference Elicitation:

The paper replaces fixed left/right/equal labels with graded preference scores that distinguish how strongly a person favors one trajectory segment.

Preference Elicitation:

  • Fixed labels encode left, right, or equal preference but not how much better one segment is.The stored label z is one of 0, 0.5, or 1.Source: Preference Elicitation · p. 4
  • A score from 0.0 to 1.0 represents dynamic preference, with 0.0 and 1.0 marking absolute preferences and 0.5 indicating no difference.Values between 0.0 and 0.5 weakly favor σ1; values between 0.5 and 1.0 weakly favor σ2.Source: Preference Elicitation · p. 4
  • The graded scale communicates how much better the chosen segment is, rather than only which segment is preferred.Any value between 0.0 and 1.0 can express the human’s preference strength.Source: Preference Elicitation · p. 4

Fitting the Reward Function:

The reward estimate serves as a latent explanation of human judgments: preference probabilities depend exponentially on summed predicted rewards, and the estimate is fitted against human labels.

Fitting the Reward Function:

  • The estimated reward is treated as a latent factor explaining human judgments, with preference probability depending exponentially on reward summed over a clip.This follows the design process attributed to Christiano.Source: Fitting the Reward Function · p. 4
  • The model converts the summed predicted rewards for two segments into a probability of preferring one segment over the other.The displayed expression gives the probability for σ1 over σ2.Source: Fitting the Reward Function · p. 4
  • The reward estimate is trained by minimizing cross-entropy between predicted preferences and human-input labels.The displayed loss is identified as cross-entropy in the accompanying prose.Source: Fitting the Reward Function · p. 4

Optimising the Policy:

The policy is optimized with PPO, while a supervised estimator predicts some preference inputs from prior human judgments to reduce the feedback burden.

Optimising the Policy:

  • PPO is used for simulated robotics tasks because the estimated reward may be non-stationary and policy-gradient methods are described as robust to reward changes.The study uses the same parameter settings as Christiano’s work.Source: Optimising the Policy · p. 4

D. Human Preference Supervision: Human-Demonstration Estimator for Deep RL

  • The human-demonstration estimator uses linear regression or RBF-kernel SVR on initial preferences, with 50/50 or 70/30 train/test splits, replacing 30–50% of human inputs.The estimator predicts preference scalings from the collected initial preferences; Fig. 2 identifies the human-demonstration model as panel B.Source: D. Human Preference Supervision:… · p. 4, Figure 2 · p. 5
  • For linear regression, the stated objective is ordinary least squares over a preferred segment and its estimated preference scaling.The passage identifies the estimated scaling value and coefficient; SVR instead constrains absolute error by a tunable maximum error ε.Source: D. Human Preference Supervision:… · pp. 4–5
  • Estimator performance is measured by mean squared error between estimated and ground-truth preference scaling values.MSE is the average squared difference between those values.Source: D. Human Preference Supervision:… · p. 5

IV. EXPERIMENT

The experiments compare four RL approaches across five MuJoCo scenarios. They use PPO training settings with and without human feedback.

IV. EXPERIMENT

  • The models were implemented with TensorFlow under OpenAI Gym, and results were consolidated with TensorBoard.Source: §IV · p. 5

A. Robotic Control Scenarios

B. Parameter Settings

  • PPO settings specify collected experience and policy updates, including horizon, mini-batches, epochs, discount factor γ, and GAE parameter λ.Source: B. Parameter Settings · p. 5
  • Experiments train each scenario for 3 × 10^7 timesteps over 10 iterations under PPO, with or without human feedback.Source: B. Parameter Settings · p. 6

C. Baselines

Across five MuJoCo scenarios, RLHPS generally achieves higher cumulative rewards than fixed-preference RLHP and PPO, with Hopper an exception. The section also describes the two baselines and the human-preference input setup.

Traditional RL: / RL from Human Preferences (RLHP):

V. RESULTS

  • The five-scenario comparison includes PPO, RLHP, RLHPS, and RLHPS with Demo; human preference methods use 700–1,400 labels, less than 0.01% of training timesteps.Source: §V · p. 6

A. RL from Human Preference Scaling (RLHPS)

  • RLHPS cumulative rewards exceed RLHP and PPO in every scenario except Hopper, where algorithms are similar after 2 × 10^7 timesteps; 1,400 labels generally yield higher rewards than 700.Source: A. RL from Human Preference Scaling… · p. 7

Cumulative Reward Values:

  • For Walker with 1,400 inputs, RLHPS gains approximately 1,500 reward over PPO and approximately 1,000 over RLHP; in Swimmer, rewards are approximately 350, 300, and 150 for RLHPS, RLHP, and PPO.Source: Cumulative Reward Values · p. 7
  • Cheetah RLHPS reaches approximately 4,000 reward versus 3,000–3,500 for RLHP and PPO; in Ant, scale-based preferences reach positive rewards faster than fixed preferences or PPO.Source: Cumulative Reward Values · p. 7

Instant Reward Distributions:

With 30% of preference labels estimated, RLHPS with Demo achieves similar or superior cumulative rewards to the compared RLHPS variants, while reducing human inputs. The section also contrasts initial and final RLHPS reward distributions and reports variation across scenarios.

Instant Reward Distributions:

B. RL from Human Preference Scaling with Demonstrations (RLHPS with Demo)

C. Observation of Testing Behaviours

Across MuJoCo tests, RLHPS variants are reported to improve cumulative rewards and reduce human-input costs, though observed behaviors include Walker and Ant exceptions. The authors also bound label substitution under current settings and identify frame selection and reward normalization as limitations.

C. Observation of Testing Behaviours

  • RLHPS variants generally show more goal-appropriate behavior than RLHP or PPO, with slow Walker progress and slower Ant learning for RLHPS with Demo as exceptions.The paper released a video comparing agents trained with different types of human input.Source: C. Observation of Testing Behaviours · p. 9

D. Limitations

  • Under the current settings, replacing up to 30% of human preference inputs with estimated inputs maintains training performance, while predicting half dramatically impacts it.The authors characterize this reduction as limited.Source: D. Limitations · p. 9
  • Random frame selection may affect preferences for near-equal options; the authors suggest selecting diverse-reward or uncertain segments and further investigating reward normalization.They note that reward distributions could affect preference levels for the estimator.Source: D. Limitations · p. 9

VI. CONCLUSIONS

  • The framework combines preference-level scaling with an estimator that predicts labels, and the authors see it as a possible route to learning natural human preferences with fewer inputs.They frame the work around deep-RL robotic control and broader robotics challenges including automated driving systems.Source: §VI · pp. 9–10
  • RLHPS and RLHPS with Demo achieve higher cumulative reward values and reduce human-input costs by up to 30% compared with PPO and RLHP.The conclusion attributes the input reduction to replacing some human preferences with estimator-generated labels.Source: §VI · p. 10
Loading 2007.12904v2…