PaperlayerView exact revision on arXiv

Source-linked AI summary

From Language to Goals: Inverse Reinforcement Learning for Vision-Based Instruction Following

Justin Fu, Anoop Korattikara, Sergey Levine, Sergio Guadarrama

arXiv:1902.07742v1cs.LGstat.ML

TL;DR

Because reward functions are difficult to engineer and language-conditioned policies combine language interpretation with physical planning, this paper learns neural rewards from language using inverse reinforcement learning. In test environments, reward learning outperforms policy learning, and learned rewards generalize to novel tasks and house layouts.

  • Problem

    Reward engineering makes RL objectives difficult to specify, while language-conditioned policies must jointly interpret commands and plan through environment dynamics.

    Source: §1 · p. 1

  • Method

    LC-RL uses inverse reinforcement learning to ground free-form language commands as neural reward functions, leaving task planning to reinforcement learning.

    Source: §1 · p. 2, §7 · p. 9

  • Results

    Reward learning outperforms policy learning in test environments, with LC-RL and Reward Regression generalizing to novel tasks and house layouts.

    Source: §6.3 · p. 8, §7 · p. 9

  • Takeaways & Limitations

    The approach lets an agent learn and interact within a test environment rather than relying on zero-shot transfer of a policy.

    Source: §7 · p. 9

    Training requires tractable domains with known dynamics and access to ground-truth states, although learned rewards can be used in environments with unknown dynamics.

    Source: §5.1 · p. 4, §7 · p. 9

Abstract

from arXiv · show

Reinforcement learning is a promising framework for solving control problems, but its use in practical situations is hampered by the fact that reward functions are often difficult to engineer. Specifying goals and tasks for autonomous machines, such as robots, is a significant challenge: conventionally, reward functions and goal states have been used to communicate objectives. But people can communicate objectives to each other simply by describing or demonstrating them. How can we build learning algorithms that will allow us to tell machines what we want them to do? In this work, we investigate the problem of grounding language commands as reward functions using inverse reinforcement learning, and argue that language-conditioned rewards are more transferable than language-conditioned policies to new environments. We propose language-conditioned reward learning (LC-RL), which grounds language commands as a reward function represented by a deep neural network. We demonstrate that our model learns rewards that transfer to novel tasks and environments on realistic, high-dimensional visual environments with natural language commands, whereas directly learning a language-conditioned policy leads to poor performance.

1 INTRODUCTION

Reward functions can be difficult to engineer, especially for complex image-based observations, while language offers a natural way to communicate goals. The paper proposes grounding language commands as rewards so reinforcement learning can plan in new environments rather than relying on zero-shot transfer of language-conditioned policies.

  • Reward functions are often difficult to engineer, particularly for tasks with complex image-based observations.Source: §1 · p. 1
  • Language-conditioned policies must jointly interpret commands and plan through potentially complicated environment dynamics.Source: §1 · pp. 1–2
  • The paper proposes using inverse reinforcement learning to ground free-form language commands as reward functions represented by a neural network.Source: §1 · p. 2
  • A learned reward can be evaluated without test-time knowledge of environment dynamics, letting an agent plan through reinforcement learning instead of relying on zero-shot policy transfer.Source: §1 · p. 2

2 RELATED WORK

Prior language-grounding approaches include language-conditioned policies, hand-designed task-completion rewards, semantic parsing, and language-conditioned reward learning. The paper distinguishes its approach through deep, differentiable reward functions for image observations and reports better generalization to new environments for reward-based than policy-based approaches.

  • Policy-based language grounding includes behavioral cloning from language-action pairs and task-completion rewards that require hand-designed detectors.Source: §2 · p. 2
  • Semantic parsing converts language into an executable form corresponding to actions within an environment.Source: §2 · p. 2
  • MacGlashan et al. (2015) use a hand-designed symbolic reward class, whereas this work uses differentiable function approximators that can handle raw images.Source: §2 · p. 3
  • The authors report that policy-based approaches generalize worse to new environments because they rely on zero-shot generalization at test time.Source: §2 · p. 3

3 BACKGROUND

The background frames inverse reinforcement learning as inferring rewards from expert demonstrations under a maximum-entropy reinforcement-learning model. In this formulation, trajectories are more probable when their returns are higher, and reward learning fits a trajectory distribution to demonstrations.

  • The model is an entropy-regularized MDP with state and action spaces, transition dynamics, reward, discount factor, and initial-state distribution.Source: §3 · p. 3
  • MaxEnt reinforcement learning seeks an optimal policy that maximizes expected trajectory return plus trajectory entropy.Source: §3 · p. 3
  • Inverse reinforcement learning infers a reward function from expert demonstrations assumed to come from an optimal policy.Source: §3 · p. 3
  • In MaxEnt IRL, trajectory probabilities are proportional to exponentiated returns, so learning rewards amounts to fitting an energy-based trajectory model by maximum likelihood.Source: §3 · p. 3

4 MULTI-TASK IRL

Language-conditioned IRL must learn rewards that generalize across tasks, which may differ in their state spaces, dynamics, and rewards. The paper shares a reward model across tasks and uses language commands in place of task-specific context identifiers.

  • The multi-task objective targets reward functions that generalize to new tasks, which may differ in state spaces, dynamics, or rewards.Source: §4 · p. 3
  • Each training iteration samples a task and demonstrations, computes optimal action values and state-action marginals, then updates the reward using their difference.Source: §4 · p. 4
  • The method requires tasks to share observation and action spaces and uses rewards that depend on observations rather than states.Source: §4 · p. 4
  • The shared reward model takes language commands as proxies for task context, and training samples environments across stochastic-gradient updates.Source: §4 · p. 4

5 LANGUAGE-CONDITIONED REWARD LEARNING (LC-RL)

LC-RL learns language-conditioned rewards with maximum causal entropy IRL, using dynamics-aware training to produce rewards evaluable without dynamics knowledge. Its architecture combines language and panoramic image embeddings, with global pooling outperforming a fully connected CNN layer.

5 LANGUAGE-CONDITIONED REWARD LEARNING (LC-RL)

  • LC-RL adapts maximum causal entropy IRL to multiple tasks, representing language-conditioned rewards with convolutional neural networks and allowing model-free RL at test time without dynamics knowledge.Training uses dynamic programming methods that require dynamics knowledge.Source: §5 · p. 4

5.1 COMPUTING EXACT IRL GRADIENT UPDATES

  • Training updates depend on ground-truth states and use known dynamics, but the reward itself depends only on observation, action, and language, so it can be evaluated in environments with unknown dynamics.The authors describe training in mapped environments and using the resulting reward in unknown environments.Source: §5.1 · p. 4

5.2 ARCHITECTURE

  • The architecture encodes the command with an LSTM and the four cardinal-view images with shared-weight CNNs, then element-wise multiplies the embeddings and passes them through a fully connected network to produce reward.Figure 2 describes a panoramic semantic image and a sequence of one-hot word vectors as inputs, with a scalar reward as output.Source: §5.2 · p. 5, Figure 2 · p. 5
  • Global pooling performed significantly better than a fully connected layer at the end of the CNN, while element-wise multiplication and concatenation showed no appreciable performance difference.The max global-pooling architecture selected scene objects and let the language embedding modulate which features to attend to.Source: §5.2 · p. 5

6 EVALUATION

The evaluation tests language-grounded reward learning in simulated indoor environments across navigation and pick-and-place tasks, including novel tasks and houses. Learned rewards generalize to both and outperform the policy-based approach, while reward regression benefits from oracle supervision.

6.1 ENVIRONMENT

  • The study evaluates NAV and PICK tasks in simulated indoor scenes, using panoramic semantic observations and four actions including object interaction.States combine grid coordinates and agent orientation; the environments are built on SUNCG scenes.Source: §6.1 · pp. 5–6, §6 · p. 5
  • Commands use a preset grammar—“go to X” for navigation and “move X to Y” for pick-and-place—so they specify goals rather than action sequences.Within a PICK task, the agent interacts with one object that can occupy two locations; objects and locations vary across environments.Source: §6.1 · pp. 6–7

6.2 EXPERIMENTAL PROCEDURE

  • The task split tests novel object-location combinations in training houses and then novel tasks in entirely unseen houses.The dataset contains 1,413 tasks across 14 objects and 76 house layouts; 1,004 tasks are for training, 236 for task testing, and 173 for house testing.Source: §6.2 · p. 7
  • LC-RL learns a shared reward from language-demonstration pairs using 10 demonstrations per task; Reward Regression provides an oracle comparison using ground-truth rewards.The evaluation also compares against GAIL-based reward baselines and an optimal behavioral-cloning baseline.Source: §6.2 · p. 7

6.3 EXPERIMENTAL RESULTS

  • Both LC-RL and Reward Regression generalize to novel tasks and house layouts, and both significantly outperform the policy-based approach.Reward Regression performs better than LC-RL because it uses oracle ground-truth supervision.Source: Table 1 · p. 8, §6.3 · p. 8
  • DQN with epsilon-greedy exploration alone performs poorly against the exact solver and comparably to cloning; value-function shaping improves results but requires ground-truth dynamics.The authors report that regression-learned rewards appear easier to re-optimize than IRL-learned rewards.Source: Table 2 · p. 8, §6.3 · p. 9
  • Learned rewards sometimes favor reaching the goal without placing a PICK object down, and commands can be ambiguous when locations or objects have duplicates.The evaluation does not allow agents to ask for clarification.Source: §6.3 · p. 8
  • The cloning baseline performs poorly on training and test environments; the authors suspect learning across environments is harder with goal-level commands than step-by-step instructions.The task requires mapping language to house layouts rather than following actions specified in the command.Source: §6.3 · pp. 8–9

7 CONCLUSION

The conclusion presents LC-RL as a scalable neural method for language-conditioned reward learning. It reports better test-environment performance than policy learning, while noting that training requires known dynamics.

7 CONCLUSION

  • LC-RL is an algorithm for scalable training of neural language-conditioned reward functions.Source: §7 · p. 9
  • Training is restricted to tractable domains with known dynamics, but the learned reward can be used with standard RL methods in environments with unknown dynamics.Source: §7 · p. 9
  • In test environments, reward learning outperforms policy learning because agents can learn and interact there instead of relying on zero-shot policy transfer.Source: §7 · p. 9

A EXPERIMENT HYPERPARAMETERS

The hyperparameter appendix describes the demonstration and optimization setup, neural architecture sweep, and image-size ablation.

A EXPERIMENT HYPERPARAMETERS

  • The environment gives 30 time-steps per task and a reward of 10 for successful completion; the MDP solvers use γ = 0.99.Source: A EXPERIMENT HYPERPARAMETERS · p. 12
  • IRL training uses 10 demonstrations per environment and Adam with a learning rate of 5 ∗10−4.Source: A EXPERIMENT HYPERPARAMETERS · p. 12
  • The CNN uses 5x5 and 3x3 convolutions with 16 and 32 filters, respectively, followed by 32- and 1-unit fully connected layers.Embedding size is 32; increasing filter or embedding sizes did not yield significant performance differences.Source: A EXPERIMENT HYPERPARAMETERS · p. 12
  • The architecture sweep compares pooling, fully connected, and FiLM image embeddings, plus multiplicative versus concatenated language-image fusion.Train and test accuracies are averaged over 3 seeds.Source: A EXPERIMENT HYPERPARAMETERS · p. 12
  • Image sizes from 32 by 24 to 64 by 64 had negligible impact, so the smaller size was selected for computational reasons.Source: A EXPERIMENT HYPERPARAMETERS · p. 12

B LEARNED REWARDS AND QUALITATIVE RESULTS

The qualitative examples pair learned rewards with birds-eye rollouts, including a value function that shifts its guidance from acquiring the fruit bowl to delivering it to the bathroom.

B LEARNED REWARDS AND QUALITATIVE RESULTS

  • Before acquisition, the value directs the agent to the fruit bowl; afterward, it directs the agent to the bathroom.The green-outlined tiles mark pickup locations within 1 meter of the bowl, and the red-outlined tile marks the bathroom drop-off area.Source: B LEARNED REWARDS AND QUALITATIVE… · p. 12
  • Figure 6 pairs learned rewards with a birds-eye rollout for “go to fruit bowl.”Source: Figure 6 · p. 13
  • Figure 7 pairs learned rewards with a birds-eye rollout for “move pan to living room.”Source: Figure 7 · p. 14

C OBSERVATION CACHING

Repeated observations make reward evaluation wasteful, so the method caches repeated computations to reduce runtime.

C OBSERVATION CACHING

  • 10-100 times speedups on reward computation result from caching repeated computation, depending on environment structure.The example reuses living-room image computations across states that differ only in the cup’s location.Source: C OBSERVATION CACHING · p. 14
  • Evaluating and back-propagating rewards across all states and actions is the main computational bottleneck, rather than Q-iteration.Q-iteration still has a cubic runtime dependency on state-space size.Source: C OBSERVATION CACHING · p. 14
  • Caching helps because state differences such as the cup’s location may leave living-room observations unchanged.Source: C OBSERVATION CACHING · p. 14

Related papers

Loading 1902.07742v1…