Source-linked AI summary
Using Natural Language for Reward Shaping in Reinforcement Learning
Prasoon Goyal, Scott Niekum, Raymond J. Mooney
TL;DR
Reinforcement learning often requires reward shaping, but designing appropriate intermediate rewards is difficult. The paper introduces LEARN, which uses natural-language instructions and agent actions to generate shaping rewards that integrate with standard RL algorithms. Across 15 Montezuma’s Revenge tasks, language-based rewards produced a 60% relative improvement in completed episodes after the same 500,000 timesteps.
Problem
Designing dense intermediate rewards is difficult, while sparse rewards often make reinforcement learning slow and difficult to learn from.
Method
LEARN predicts whether an agent’s trajectory matches a natural-language instruction and uses that prediction as an intermediate reward for reinforcement learning.
Results
60% relative improvement: after 500,000 timesteps, Ext+Lang completed 1529.43 episodes on average versus 903.12 for ExtOnly.
Takeaways & Limitations
Language-based rewards can train faster and learn a better policy in sparse-reward settings, while potentially allowing non-experts to specify tasks.
Takeaways & Limitations
The current approach loses temporal information by aggregating past actions into an action-frequency vector and cannot use descriptions referring to objects in the state.
Abstract
from arXiv · showhide
Recent reinforcement learning (RL) approaches have shown strong performance in complex domains such as Atari games, but are often highly sample inefficient. A common approach to reduce interaction time with the environment is to use reward shaping, which involves carefully designing reward functions that provide the agent intermediate rewards for progress towards the goal. However, designing appropriate shaping rewards is known to be difficult as well as time-consuming. In this work, we address this problem by using natural language instructions to perform reward shaping. We propose the LanguagE-Action Reward Network (LEARN), a framework that maps free-form natural language instructions to intermediate rewards based on actions taken by the agent. These intermediate language-based rewards can seamlessly be integrated into any standard reinforcement learning algorithm. We experiment with Montezuma's Revenge from the Atari Learning Environment, a popular benchmark in RL. Our experiments on a diverse set of 15 tasks demonstrate that, for the same number of interactions with the environment, language-based rewards lead to successful completion of the task 60% more often on average, compared to learning without language.
1 Introduction
The paper addresses the difficulty of specifying dense reward functions by using natural-language instructions to provide intermediate rewards for reinforcement learning. Its framework predicts whether an agent’s trajectory follows an instruction, enabling faster learning in sparse-reward settings.
- Motivation: Sparse rewards can make reinforcement learning difficult and slow, whereas dense rewards are harder to specify.The motivating example is navigating Montezuma’s Revenge by jumping over a skull while moving left.
- Motivation: Reward shaping provides intermediate rewards for progress toward a goal, but designing those rewards is particularly hard for non-experts.The paper frames natural language as an easier way to specify such intermediate signals.
- Proposed direction: Natural-language instructions are proposed as intermediate rewards that can accelerate learning and help non-experts teach agents new skills.The example instruction is “Jump over the skull while going to the left.”
- Framework: The framework takes arbitrary natural-language instructions and the agent’s executed trajectory, then predicts whether the agent is following the instruction.That prediction is used as an intermediate reward to guide exploration.
- Challenges: Natural-language reward shaping must address symbol grounding, incomplete instructions, and ambiguity or variation in how behaviors are described.Examples include grounding “snake” to image pixels and “jump” to an action, and handling instructions at different abstraction levels.
- Integration: The approach modifies the reward function, allowing language-based rewards to be used with any standard reinforcement learning algorithm.The evaluation uses Montezuma’s Revenge and reports faster learning with robustness to instruction variation.
2 Overview of the Approach
The approach extends reinforcement learning with a language command and uses a two-phase framework: train LEARN, then use its predictions as shaping rewards during policy learning. Because only the reward function is modified, the language-aided stage is agnostic to the particular reinforcement learning algorithm.
- Language-augmented MDP: The language-augmented MDP adds a language command l describing the intended behavior to the standard MDP tuple.The standard tuple contains states, actions, transitions, rewards, and a discount factor.
- LEARN: The first phase trains LEARN on paired trajectory-language data to predict whether a language description matches the trajectory’s actions.Natural-language instructions are collected for trajectories in the environment.
- Framework: The framework augments the standard agent-environment reinforcement learning loop with a LanguagE-Action Reward Network module.The language-based prediction is incorporated as an additional reward rather than as a distribution over actions.
- Language-aided RL: The second phase uses LEARN to predict whether the agent is making progress and treats that prediction as a shaping reward.The reward is computed from the trajectory executed so far and the language instruction.
3 LanguagE-Action Reward Network
LEARN learns whether action trajectories match natural-language instructions and converts that judgment into a two-class prediction. It is trained on trajectory-language pairs, using action-frequency representations and language encoders, with human-annotated gameplay clips.
- Model: LEARN represents trajectory segments as action-frequency vectors and pairs them with language descriptions.Each vector records the fraction of timesteps spent on each available action.
- Training data: Positive examples pair a trajectory-derived vector with its original instruction, while negative examples pair mismatched instructions or random vectors with language.
- Neural network architecture: The network concatenates encoded action and language vectors, then predicts whether they are RELATED or UNRELATED.The action representation and language representation are processed separately before classification.
- Language encoder: Three language encoders trade off prior linguistic knowledge against flexibility: InferSent, GloVe+RNN, and RNNOnly.InferSent is least flexible, while RNNOnly begins without linguistic knowledge.
4 Using Language-based Rewards in RL
LEARN uses the agent’s partial action history and a language instruction to produce a potential-based shaping reward. The resulting reward can be added to the environment reward while preserving compatibility with standard RL algorithms.
- Reward generation: At each timestep, LEARN evaluates the action-frequency vector formed from the agent’s previous actions together with the associated language instruction.The vector records action frequencies in the history through timestep t−1.
- Reward interpretation: Higher LEARN-relatedness probabilities indicate trajectories whose actions better match the instruction.For example, an instruction involving jumping left favors trajectories with frequent JUMP and LEFT actions.
- Integration with RL: Because language reward modifies only the reward function, the approach remains agnostic to the particular RL algorithm.The language reward is combined with the environment reward to guide exploration toward relevant actions.
- Reward definition: The potential function is φ(f_t) = p_R(f_t) − p_U(f_t), and the language reward is R_lang(f_t) = γ · φ(f_t) − φ(f_t−1).
5 Experimental Evaluation
The evaluation tests language-based shaping on 15 sparse-reward Montezuma’s Revenge tasks against an extrinsic-reward-only baseline. Across tasks, Ext+Lang learns faster and generally achieves stronger final policies, while performance depends on how accurately LEARN grounds instructions to actions.
- Experimental setup: The benchmark uses 15 tasks requiring navigation from fixed starts to fixed goals, with +1 extrinsic reward only upon reaching each goal.
- Experimental setup: ExtOnly uses environment reward alone, whereas Ext+Lang adds potential-based language reward at every step.
- Metrics: AUC measures learning speed from successful-episode curves, while Final Policy evaluates performance after 500,000 training steps.
- Results: Ext+Lang reaches 903.12 average successful episodes after 358,464 timesteps, a 30% speed-up relative to ExtOnly.
- Statistical significance: Ext+Lang significantly improves AUC on 11 of 15 tasks, deteriorates on 1, and shows no significant difference on 3.
- Language-reward analysis: Language rewards correlate with relevant actions in analyzed tasks, but task 14 is a failure case where poor grounding causes statistically significant AUC deterioration.Descriptions involving “jump” produce noisy groundings because related actions co-occur and humans use similar terms for them.
- Language-reward analysis: LEARN grounds action names to environment actions despite receiving ordinal action representations rather than names such as JUMP or LEFT.
6 Related Work
The paper situates its approach among methods using natural language to aid reinforcement learning, distinguishing LEARN through flexible language grounding and reward-based policy optimization.
- Methods using language to aid RL: Prior work using language to aid RL includes object-oriented reward functions, contextual-bandit goal descriptions, rule-based action guidance, action-value features, and adversarial reward learning.These approaches differ in how language is represented and integrated into learning.
- Limitations of prior methods: Object-oriented approaches rely on predefined objects, properties, spatial relations, or simple language features, making scaling to complex environments and instructions difficult.
- Empirical comparison: Figure 5 compares different reward functions for selected tasks.
- Relation to planning: High-level planning methods use language to generate subgoals, whereas this work targets lower-level exploration speed through language-based rewards.
- Positioning: Unlike related work that improves language instruction-following with RL, this approach uses instructions to improve RL performance.
7 Conclusions and Future Work
The paper concludes that LEARN provides language-based rewards for faster learning in sparse-reward settings and identifies extensions addressing temporal information, state references, and multi-step instructions.
- Conclusions: LEARN predicts whether trajectory actions match language descriptions and uses its outputs as intermediate rewards for RL.
- Conclusions: Language-based rewards can train faster and learn a better policy in sparse-reward settings.
- Conclusions: The approach may allow non-experts to specify tasks for RL agents using natural language.
- Future work: Future extensions include preserving temporal action order, incorporating states into rewards, and handling multi-step instructions.
A Example Annotations
The example annotations illustrate substantial variation and noise in crowdsourced language descriptions, while the method still extracts useful information from them.
- Annotation examples: 20 randomly selected Amazon Mechanical Turk annotations show variation in description length and vocabulary.
- Annotation noise: Several annotations contain spelling errors, ill-formed language, or descriptions that are not very informative.
- Annotation noise: The annotations were not manually corrected or filtered for these issues because doing so requires significant effort.
- Method robustness: The method is reported to extract useful information despite noise in the annotations.
B Policy Invariance
The policy-invariance analysis shows that action-frequency-vector shaping can preserve an optimal policy when expressed as potential-based shaping in an augmented MDP, while other optimal policies may also exist.
- Theorem: The theorem states that action-frequency-vector reward shaping does not change the optimal policy.
- Proof construction: The proof augments each state with a vector recording the counts of each action and defines transitions and rewards only for consistent count updates.
- Proof conclusion: The derived optimal action-value function satisfies the Bellman optimality equation for the augmented MDP.
- Caveat: The augmented MDP may admit other optimal policies that depend on the language instruction.
- Policy invariance: The augmented-state formulation makes the proposed shaping reward a function only of the augmented state, enabling potential-based policy invariance.
C Sensitivity Analysis
LEARN’s language-based rewards remain effective when noise is added to its predictions. On Task 8, performance improves over the baseline even with substantial noise, indicating robust predictions.
- Gaussian noise N(0, σ) was added to LEARN’s potential function, with σ varied from 0.01 to 1.0.The perturbation tests sensitivity to noise in LEARN’s output.
- Language-based rewards improve over the baseline on Task 8 even with significant prediction noise.The results shown in Figure 6 compare noisy LEARN outputs against the baseline.
- The authors interpret these results as evidence that LEARN’s predictions are fairly robust.
D Amazon Mechanical Turk interface
The paper uses an Amazon Mechanical Turk interface to collect natural-language descriptions. Figure 6 separately summarizes how prediction noise affects successful episodes across tasks.
- Amazon Mechanical Turk interface: Figure 7 shows the interface used on Amazon Mechanical Turk for collecting natural-language descriptions.
- Sensitivity analysis: Figure 6 plots mean successful episodes averaged over all tasks, with shaded regions showing 95% confidence intervals.The figure examines the effect of adding noise to LEARN’s predictions.