PaperlayerView exact revision on arXiv

Source-linked AI summary

Mapping Instructions and Visual Observations to Actions with Reinforcement Learning

Dipendra Misra, John Langford, Yoav Artzi

arXiv:1704.08795v2cs.CL

TL;DR

The paper tackles instruction execution from raw visual observations and text without structured environment representations or separately trained pipelines. It learns a neural policy that uses reinforcement learning with reward shaping in a contextual-bandit setting, and reports lower execution error than supervised and common reinforcement-learning baselines. The approach is evaluated in a simulated Blocks environment, with training relying on state access unavailable at test time.

  • Problem

    Existing instruction-following methods assume structured environment representations or separately trained language and visual models, whereas this task requires acting from raw visual observations and text.

  • Method

    A single neural network policy jointly processes instructions and RGB observations, using reward shaping from demonstrations or goal states to train with policy gradients in a contextual-bandit setting.

  • Results

    24% lower execution error than standard supervised learning and 34-39% lower error than common reinforcement-learning techniques were reported in the Blocks environment.

  • Takeaways & Limitations

    Reward shaping combines supervised-like training signals with reinforcement-based exploration for instruction execution in a few-samples regime.

  • Takeaways & Limitations

    Training assumes access to an instrumented environment that provides world state for computing rewards, while agent contexts contain no physical information beyond observed images.

Abstract

from arXiv · show

We propose to directly map raw visual observations and text input to actions for instruction execution. While existing approaches assume access to structured environment representations or use a pipeline of separately trained models, we learn a single model to jointly reason about linguistic and visual input. We use reinforcement learning in a contextual bandit setting to train a neural network agent. To guide the agent's exploration, we use reward shaping with different forms of supervision. Our approach does not require intermediate representations, planning procedures, or training different models. We evaluate in a simulated environment, and show significant improvements over supervised learning and common reinforcement learning variants.

1 Introduction

The paper addresses instruction execution from raw RGB observations and language by learning one model that directly selects actions. It combines reinforcement learning with reward shaping and evaluates complete action sequences in an interactive Blocks simulator.

  • Existing instruction-following approaches rely on structured environment representations or separately trained language and visual models.
  • The proposed agent directly maps instruction text and raw RGB observations to actions without intermediate representations, planning procedures, or multiple trained models.
  • The agent executes instructions sequentially, selecting actions from the current visual state and instruction until it predicts completion.
  • Reward shaping incorporates demonstrations or goal-state annotations to guide exploration and support contextual-bandit learning.The approach varies supervision by using complete demonstrations for some training examples or goal-state information.
  • 24% lower execution error than standard supervised learning and 34-39% lower error than common reinforcement-learning techniques were reported in the Blocks environment.The task requires predicting complete action sequences, with 81 possible actions per step and 15.4 steps required on average.

2 Technical Overview

The technical formulation represents instruction execution as state transitions produced by sequential actions, with visual observations and training-time state access. Policy-gradient learning uses reward shaping from demonstrations or goal states, and evaluation measures final-state distance.

  • An instruction is executed as a sequence of state-action pairs ending with the special STOP action.Actions transform states through a transition function, and execution ends when STOP is selected.
  • In Blocks, each action moves one of 20 blocks north, south, east, or west, yielding 81 possible actions including STOP.Blocks cannot move over or through other blocks.
  • Training uses policy gradients with reward shaping derived from demonstrations, while an instrumented environment supplies world state for training rewards only.The paper also experiments with training from goal states rather than complete demonstrations.
  • Execution error is measured as the distance between the final execution state and the specified goal state.

3 Related Work

The paper situates its contribution among language-understanding, reinforcement-learning, robotics, and language-vision research. Unlike many related instruction-following systems, it predicts the next action from an instruction and world image.

  • Prior instruction-following work commonly uses structured environment representations with semantic parsing, alignment models, reinforcement learning, or neural networks.
  • The paper addresses next-action prediction conditioned jointly on a world image and a natural-language instruction.
  • Its setting differs from game-based neural reinforcement learning because the data are limited and contexts are constrained by the finite training instruction set.
  • The work is related to robotics policy learning through its instrumented training environment and extends that setting to tasks specified with natural language.

4 Model

The model is a neural policy that combines language, current and recent visual observations, and action history to choose each next action. LSTM and CNN encoders feed a task-specific perceptron whose action probabilities factor into direction and block choices.

  • At each step, the policy observes the instruction and current RGB image, predicts an action, and transitions to the next state until STOP is predicted.
  • The agent context contains the instruction, current image, K previous-state images, and previous action.Previous observations and action history help distinguish execution stages and avoid repeating failed actions.
  • An LSTM encodes the instruction, while a CNN embeds the current and previous images concatenated along the channel dimension.
  • The visual, textual, and previous-action representations are concatenated into the agent-context vector used for action selection.
  • A task-specific feedforward perceptron decomposes each block-world action into block and direction probabilities.The overall action probability is the product of the component probabilities.

5 Learning

The agent learns a neural-network policy with policy gradients in a contextual bandit setting, using rollouts and shaped rewards to train from raw instruction-execution contexts.

  • Policy gradient estimates the parameters θ of a probabilistic neural-network agent policy from sampled actions and rewards.
  • The original problem reward is sparse: it rewards successful STOP, penalizes incorrect STOP and failed actions, and gives negative feedback otherwise.
  • Sparse rewards make random exploration unlikely to encounter goal states, so the agent may receive no positive reward and fail to learn.
  • The training objective optimizes immediate reward in a contextual bandit setting rather than total expected reward.The setting offers tighter sample-complexity bounds than general reinforcement learning.
  • Entropy regularization delays premature convergence but does not eliminate it.
  • The agent performs bounded rollouts for each training example, updating its context, sampling actions, executing them, and approximating the gradient at every step.Rollouts stop at the episode limit or when the policy selects STOP.

6 Reward Shaping

Reward shaping augments the sparse task reward with supervision-derived information, combining goal-directed distance and demonstration-based action guidance while preserving policy ordering under safe potentials.

  • Reward shaping adds information to the problem reward, allowing demonstrations and exploration to be combined within reinforcement learning.
  • Arbitrary shaping can change policy optimality, whereas potential-based terms provide sufficient conditions for preserving the original ordering of policies.
  • The method introduces two shaping terms, whose sum is added to the problem reward.
  • Distance-based Shaping (F1): Distance-based shaping rewards movement toward the goal using a safe potential-based term proportional to negative distance.In the block world, distance sums Euclidean distances between corresponding block positions.
  • Trajectory-based Shaping (F2): Trajectory-based shaping uses nearby demonstration states to encourage the action taken in the corresponding demonstration.It requires access to the previous state and action and can address cases where direct distance reduction is suboptimal.

7 Experimental Setup

The experiments use a simulated block-moving environment with RGB observations, natural-language instructions, and a large action space, evaluated against gold goal states on a corpus with train, development, and test splits.

  • The environment requires moving blocks on a plane through complete action sequences, rather than predicting only source and target positions.
  • Up to 20 logo- or digit-marked blocks can be moved in four directions, with deterministic transitions and observations from above.
  • The setup uses a relatively large action space and avoids decomposing the task into source-target prediction followed by a planner.
  • The corpus contains single- and multi-block instructions, but multi-block instructions are beyond the paper’s scope and demonstrations are generated using shortest paths.Shortest-path demonstrations can introduce noise for trajectory-specific instructions and from state-space discretization.
  • The corpus includes 11,871 training, 1,719 development, and 3,177 testing instructions.
  • Table 1 compares corpus statistics for the block environment with the SAIL navigation domain.
  • Task completion error is the sum of Euclidean distances between final and gold block positions, normalized by block size.

8 Results

The proposed contextual-bandit approach with reward shaping outperforms baselines, benefits from limited demonstrations, and still exhibits stopping and language-vocabulary errors.

  • Development results: The proposed approach significantly outperforms supervised learning, REINFORCE, DQN, RANDOM, and STOP baselines on development results.The authors attribute weaker supervised performance to execution errors and weaker general reinforcement learning performance to limited-data sample complexity.
  • Ablations: Removing supervised initialization or previous-action information increases error, while removing initialization and trajectory shaping raises error to 5.45.After two epochs, error is 3.94 with initialization versus 6.01 without; replacing the shaping design with a negative distance reward fails to converge.
  • Supervision ablation: Most of the benefit from complete-trajectory supervision is obtained with only 12.5% of training examples.Demonstrations are used for both supervised initialization and the trajectory-based shaping term.
  • Test results: Test results show the proposed approach outperforming all baselines, although it remains below the demonstrations upper bound.Ensembles are used to reduce the risk of development-set overfitting.
  • Execution behavior: The agent often reaches the goal but fails to stop, generating 28.7 steps on average and using the 40-step maximum in 55.2% of development cases.Development demonstrations average 15.2 steps, highlighting difficulty learning behavior at the rarely observed absorbing STOP state.
  • Planner comparison: A supervised variant with a perfect planner achieves 2.78 mean development error, nearly two points better than supervised learning with the proposed approach.This variant uses raw image input but removes the sequence-of-decisions setting.
  • Language analysis: Mean development error increases from 3.49 with no unknown words to 3.97 with one and 4.19 with two unknown words.No meaningful correlation is observed with new bigrams, trigrams, or instruction length.

9 Conclusions

The paper addresses instruction execution from raw visual observations in a few-samples regime by combining reinforcement learning with reward shaping and supervised signals.

  • Conclusion: The approach combines reward shaping with supervised-like signals and reinforcement-based exploration in a contextual bandit setting.The conclusion presents this combination as designed for learning with few samples.

A Reward Shaping Theorems

The paper defines potential-based shaping terms whose addition preserves policy ordering under the stated theorem, while evaluating their effectiveness empirically in a contextual bandit setting.

  • Scope: The paper shows empirically that safe shaping terms are effective in a contextual bandit setting, although the theorem itself applies to total-reward optimization.This distinction separates the formal guarantee from the reported contextual-bandit evidence.
  • Shaping terms: The distance-based term F1 follows the state-potential theorem, while the trajectory-based term F2 follows a look-back advice-shaping construction.Both definitions omit the discounting factor because γ is set to 1.0.
  • Definitions: A shaping term is potential-based when it equals the discounted difference between potential functions of consecutive states and actions.The state-only form uses φ:S→R, while the look-back form uses φ:S×A→R.
  • Theorems: Adding a potential-based shaping term to the reward does not modify the total order of policies.The theorem is stated for the total-reward objective.

B Evaluation Systems

The evaluation compares fixed, random, supervised, and deep reinforcement-learning systems, using a recurrent visual-language policy and specified training controls.

  • Baselines: STOP immediately terminates execution, while RANDOM samples actions uniformly until STOP or the execution horizon.The horizon is J actions.
  • Baselines: SUPERVISED trains on instruction-state-action triplets by maximizing the demonstrated-action log-likelihood.Agent contexts are generated using annotated previous actions.
  • Baselines: DQN replaces the policy-gradient approach with a Q-network while retaining the described architecture.The supplied passage also notes an oracle-planner variant trained to predict moved blocks and target positions.
  • Architecture: The visual encoder uses 120x120 RGB images and a four-layer CNN, while the language component uses a one-layer RNN with 150-dimensional embeddings and 250 LSTM units.The action embedding dimension is 56.
  • Architecture: The action output represents four directions plus STOP, with K=4 previous images and horizon J=40.The direction/output matrix covers north, south, east, west, and STOP.
  • Training: Models use specified Gaussian or truncated-normal initialization, validation distance for stopping, development performance for model selection, and fixed reward-shaping penalties.The reported entropy coefficient is λ=0.1, with learning rates of 0.001 for supervised learning and 0.00025 for policy gradient.

D Dataset Comparisons

The paper situates Blocks among instruction-following corpora and reports design choices, empirical observations, and scope constraints for the agent.

  • Dataset comparisons: Blocks provides the largest training set and a relatively complex environment among the reviewed instruction-following datasets.The comparison includes Blocks, SAIL, Matuszek, and Misra corpora.
  • Dataset comparisons: SAIL offers partial observability, whereas Blocks is more complex along dimensions involving instructions, trajectories, and action spaces.Partial observability means the agent observes only what is around it.
  • Model choices: Mean pooling of LSTM hidden states performed better than using only the final hidden state, while the relevant scenarios remain a future research question.The paper reports this comparison empirically and notes a similar observation by Narasimhan et al. (2015).
  • Agent behavior: The agent rarely moves obstacles or other blocks, instead preferring to move blocks around obstacles; this behavior appears even with limited demonstrations.The behavior is learned from the data rather than enforced by an explicit rule.
  • Model limitations: The agent lacks an explicit mechanism for marking completed instruction portions, and attention did not improve performance in the tested domain.Designing an architecture that supports such tracking is identified as future work.
  • Scope boundaries: Multi-instruction paragraphs remain an open problem because they may merge placements, lack trajectories, and require resolving block order.The authors also expect cascading errors when executing segmented instruction sequences.
Loading 1704.08795v2…