Source-linked AI summary
Interactive Learning from Policy-Dependent Human Feedback
James MacGlashan, Mark K Ho, Robert Loftin, Bei Peng, Guan Wang, David Roberts, Matthew E. Taylor, Michael L. Littman
TL;DR
The paper investigates whether human feedback depends on the learner’s current policy, rather than only on the behavior being taught. It introduces COACH to learn from such policy-dependent feedback and reports convergence to a local optimum plus successful learning of multiple behaviors on a physical robot.
Problem
Existing human-centered reinforcement-learning algorithms commonly assume feedback depends only on action quality, whereas the paper investigates whether feedback also depends on the learner’s current policy.
Method
COACH uses the advantage function as a model of human feedback and directly modifies the agent’s policy from feedback without a critic component.
Results
COACH converges to a local optimum with policy-dependent feedback and successfully learns five behaviors on a physical robot.
Takeaways & Limitations
Policy-dependent feedback can support useful training strategies, including differential feedback, diminishing returns, and compositional robotic training.
Takeaways & Limitations
The convergence analysis assumes that ρ is the discounted expected reward from a fixed start-state distribution.
Abstract
from arXiv · showhide
This paper investigates the problem of interactively learning behaviors communicated by a human teacher using positive and negative feedback. Much previous work on this problem has made the assumption that people provide feedback for decisions that is dependent on the behavior they are teaching and is independent from the learner's current policy. We present empirical results that show this assumption to be false -- whether human trainers give a positive or negative feedback for a decision is influenced by the learner's current policy. Based on this insight, we introduce {\em Convergent Actor-Critic by Humans} (COACH), an algorithm for learning from policy-dependent feedback that converges to a local optimum. Finally, we demonstrate that COACH can successfully learn multiple behaviors on a physical robot.
1. Introduction
Programming robots is difficult because real-world behavior is rich and unpredictable, while desired behaviors are often hard to articulate. Human feedback offers a way for non-programmers to teach complex behavior without programming.
- Real-world complexity and poorly articulated expectations make programming robots difficult for end-users.
- Reinforcement learning from human trainer feedback can teach complex behavior using simple positive and negative signals.
- Animals demonstrate that simple training signals can support complex tasks including guiding, detection, and multi-stage problem solving.
- Existing human-feedback algorithms often interpret feedback as policy guidance but assume it is independent of the learner’s current policy.
- The paper reports that this assumption is incorrect, introduces COACH for policy-dependent feedback, and demonstrates convergence to a local optimum.
2. Background
The paper frames decision-making with Markov decision processes and explains how reinforcement-learning agents learn policies from observations. Actor-critic methods separate action selection from value estimation.
- An MDP represents decision-making with states, actions, transitions, rewards, and a discount factor.
- A stochastic policy assigns an action probability distribution to each state, and the optimal policy maximizes expected future discounted reward.
- The value function evaluates expected future discounted reward from a state, while the action-value function conditions on taking a specific action first.
- Value functions can be recursively defined through the Bellman equation, with optimal-policy values commonly denoted V* and Q*.
- In reinforcement learning, agents lack direct access to transition and reward functions and must learn policies from environmental observations.
- Actor-critic algorithms use an actor to select actions through a parameterized policy and a critic to estimate its value function.
3. Human-centered Reinforcement Learning
Human-centered reinforcement learning replaces a stationary environmental reward with feedback from a trainer teaching a target policy. The paper motivates interpreting this feedback as commentary on behavior rather than reward to maximize.
- Human-centered reinforcement learning places an agent in an MDP where rewards are generated by a human trainer rather than a stationary reward function.
- The trainer communicates a target policy through numeric feedback as the agent acts, and the agent aims to learn that policy.
- Treating human feedback as a stationary reward can work in some cases but requires feedback suitable for maximization and has limiting complications.
- Human feedback is better interpreted as commentary on behavior, with positive and negative signals roughly meaning that an action was good or bad.
4. Related Work
Prior work includes methods that estimate feedback-derived rewards, infer probabilistic target policies, combine human and environmental rewards, or learn from demonstrations. These approaches differ in how they represent and obtain guidance.
- TAMER estimates a target reward function by treating interactive trainer feedback as exemplars while the learner acts.
- SABL and Policy Shaping treat feedback as discrete probabilistic evidence about the trainer’s target parameterized policy.
- SABL additionally models learnable parameters for how often trainers provide explicit positive or negative feedback.
- Some studies combine human feedback with environmental rewards so agents can continue learning without an active trainer, although rewards must be programmatically defined.
- Learning from demonstration is complementary to human-feedback approaches because demonstrations are not always possible or convenient.
- Behavior-analysis reinforcement schedules emphasize diminishing returns, differential feedback, and policy shaping as forms of policy-dependent feedback.
5. Policy-dependent Feedback
Human feedback depends not only on action quality but also on the learner’s current policy. An AMT study found more positive feedback for improving behavior and negative feedback otherwise.
- Policy-dependent Feedback: Existing HCRL algorithms commonly assume feedback depends only on action-selection quality, but feedback may also depend on the learner’s current policy.The paper distinguishes policy-independent feedback from policy-dependent feedback, where the same action can receive different feedback under different policies.
- Policy-dependent Feedback: Prior observations that trainers taper feedback as learning progresses are consistent with policy-dependent feedback.As learner performance improves, opportunities to improve over the current policy decrease, so trainer feedback can diminish.
- Experimental Design: Participants trained a virtual dog in a grid world to reach a yellow goal quickly while avoiding green cells, using feedback from −50 to 50 after every action.The dog began with behavior described as bad, alright, or good, and feedback was entered through a continuous labeled slider.
- Experimental Design: The final episode used identical alright behavior across conditions, while prior episode trajectories differed as improving, steady, or degrading.The first step of the final episode was designed to reveal whether feedback depended on the learner’s policy rather than only the observed behavior.
- Empirical Results: Improving behavior received positive feedback, whereas steady and degrading conditions received negative feedback, with the improving condition differing significantly from both.Improving: Mean = 9.8, Median = 24; steady: Mean = −18.3, Median = −23.5; degrading: Mean = −10.8, Median = −18.0.
6. Convergent Actor-Critic by Humans
COACH models human feedback with policy-sensitive advantage information and directly updates the actor without a critic. Real-time COACH extends it for delayed, sparse, and variable-magnitude feedback.
- COACH: COACH is an actor-critic-based algorithm designed to learn from policy-dependent human feedback.Its formulation uses the advantage function as a model of human feedback and directly modifies policy parameters.
- The Advantage Function and Feedback: The advantage function measures how much better or worse an action is than the agent’s performance under its current policy.Choosing actions that maximize the advantage function improves a suboptimal policy, while policy-gradient methods can use it for gradual improvement.
- The Advantage Function and Feedback: Advantage-based feedback produces differential feedback, diminishing returns, and policy shaping because feedback reflects improvement over current behavior.When the policy is optimal, advantage-function feedback is zero or negative.
- Convergence and Update Rule: COACH updates policy parameters using trainer feedback multiplied by the policy-gradient term, yielding convergence when feedback follows Qπ or Aπ.The paper states that these policy-dependent feedback forms locally maximize discounted expected reward.
- Real-time COACH: Real-time COACH addresses practical feedback by aggregating repeated signals, applying eligibility traces to earlier transitions, and accommodating delayed feedback.Multiple traces with different decay rates let trainers control how far feedback propagates; delayed feedback is associated with events from d steps earlier.
7. Comparison of Update Rules
The comparison evaluates Q-learning, TAMER, and COACH under feedback strategies matched and mismatched to their update-rule assumptions. Performance depends strongly on the feedback type: Q-learning suits task rewards, TAMER suits action feedback, and COACH suits policy-dependent improvement feedback.
- Learning algorithms: Q-learning maintains Q(s, a) for delayed reward maximization, while TAMER predicts immediate human reward for state–action pairs.Q-learning selects argmax_a Q(s, a); TAMER applies each feedback signal to the last action in this simplified implementation.
- Learning algorithms: COACH uses a softmax policy parameterized by θ(s, a), with a single λ = 0 trace and learning rate α = 0.05.The parameters are initially zero, and the trace setting determines whether feedback influences earlier decisions.
- Feedback strategies: The three feedback strategies are sparse task rewards, policy-independent action feedback, and improvement feedback defined by the learner’s current-policy advantage function.Each strategy is intended to match a different algorithmic assumption: Q-learning, TAMER, or COACH, respectively.
- Task feedback: Under task feedback, TAMER can fail and COACH performs poorly with λ = 0, whereas λ = 0.9 makes COACH converge to reasonable behavior, though slower than Q-learning.The result supports using traces to address delayed feedback.
- Action feedback: Under action feedback, Q-learning performs poorly because the strategy induces positive behavior cycles, while TAMER and COACH perform well.TAMER performs slightly better because this feedback matches the type it was designed to handle.
- Improvement feedback: Under improvement feedback, COACH performs best; Q-learning often flounders, while 29% of TAMER trials fail to improve and time out.Q-learning also fails to do well in 14% of trials after 100 training episodes.
8. Robotics Case Study
Real-time COACH learned five behaviors on a TurtleBot under fast, noisy, partially observed conditions, while TAMER struggled with compositional training and forgetting. The study also shows that COACH’s use of feedback traces and policy-dependent strategies supports robust behavior learning.
- Experimental setup: The robot acted every 33ms using five discretized actions and RGB Kinect input, while a Wii controller delivered numeric feedback.The platform introduced fast decision cycles, noisy non-Markov visual observations, and perception hidden from the trainer.
- Experimental setup: COACH successfully learned all five TurtleBot behaviors, each in less than two minutes including verification.The behaviors were push–pull, hide, ball following, alternate, and cylinder navigation.
- Training methods: COACH combined differential feedback and diminishing returns with compositional training to reinforce only behaviors needing adjustment and combine subbehaviors.Cylinder navigation used the ball as a lure before reinforcing navigation toward the cylinder without the ball present.
- Results and discussion: TAMER learned only with flat training, forgetting earlier behavior after new decisions and failing on compositional and lure-trained behaviors.For alternate, this forgetting caused the robot to drift off course when returning to the ball.
- Results and discussion: A smaller positive reward caused TAMER to unlearn staying, whereas COACH strengthened the behavior with any positive reward.The difference follows from TAMER treating feedback as reward-function exemplars rather than preserving positive behavioral reinforcement.
9. Conclusion
The paper shows that human numeric feedback depends on the agent’s current policy and introduces COACH to learn from that feedback. COACH converges to a local optimum, handles multiple feedback strategies, and learns behaviors on a physical robot.
- Human trainers’ numeric feedback is influenced by the agent’s current policy, contrary to the assumption that feedback is policy-independent.
- COACH is an algorithm designed to learn from policy-dependent feedback and converge to a local optimum.
- COACH learns robustly across multiple feedback strategies and supports robotics training with advanced methods.
- Because COACH uses the actor-critic paradigm, future work could combine it with learning from demonstrations and environmental rewards.