PaperlayerView exact revision on arXiv

Source-linked AI summary

Mapping Instructions to Actions in 3D Environments with Visual Goal Prediction

Dipendra Misra, Andrew Bennett, Valts Blukis, Eyvind Niklasson, Max Shatkhin, Yoav Artzi

arXiv:1809.00786v2cs.CL

TL;DR

Instruction following requires mapping language and observations to actions despite intertwined goal, perception, and planning challenges. The paper separates visual goal prediction from action generation using LINGUNET and an RNN, and introduces LANI and CHAI for evaluation. Decomposition significantly improves LANI performance, shows similar trends on CHAI with weaker results, and exposes substantial remaining difficulty.

  • Problem

    Existing direct language-to-action models combine language, perception, and planning, while goal selection need not be recomputed at every action step.

  • Method

    The model predicts visual goals with language-conditioned LINGUNET, generates actions with an RNN, and trains the stages with supervised learning and policy gradient from demonstrations.

  • Results

    Decomposition significantly improves LANI instruction execution and shows similar trends on CHAI, where overall results are weaker.

  • Takeaways & Limitations

    LANI and CHAI provide benchmarks spanning landmark navigation, household manipulation, and instructions with multiple intermediate goals.

  • Takeaways & Limitations

    Action generation depends completely on the predicted goal, limiting handling of execution constraints and making cascading errors possible.

Abstract

from arXiv · show

We propose to decompose instruction execution to goal prediction and action generation. We design a model that maps raw visual observations to goals using LINGUNET, a language-conditioned image generation network, and then generates the actions required to complete them. Our model is trained from demonstration only without external resources. To evaluate our approach, we introduce two benchmarks for instruction following: LANI, a navigation task; and CHAI, where an agent executes household instructions. Our evaluation demonstrates the advantages of our model decomposition, and illustrates the challenges posed by our new benchmarks.

1 Introduction

The paper separates instruction execution into predicting visual goals and generating actions, then evaluates this decomposition on two new 3D instruction-following benchmarks. The approach uses language-conditioned visual goal prediction and reports stronger navigation results, while both tasks remain challenging.

  • Direct language-to-action models combine language, perception, and planning in a single model, reducing engineering but motivating explicit decomposition.
  • The proposed model predicts a goal from the initial observation, then generates the actions required to complete it.
  • LINGUNET predicts a probability distribution over goal locations or objects in the agent’s visual observation, followed by an RNN action generator.
  • The decomposition supports supervised learning for goal prediction and policy-gradient learning for exploratory action generation, while retaining demonstration-only training without external resources.
  • LANI contains 6,000 navigation sequences, while CHAI contains 1,596 household instruction sequences combining navigation and manipulation.
  • Decomposition significantly improves LANI instruction execution, shows similar trends on CHAI, and leaves a large gap to human-level performance across both tasks.

2 Technical Overview

The technical formulation maps language and visual context to a goal distribution and then to discrete actions. Training separates supervised goal prediction from policy-gradient action learning, using world-state access only for training rewards.

  • Task: Instructions are token sequences, and the agent executes them by generating discrete actions until producing the special action STOP.
  • Task: LANI actions move or rotate the agent, whereas CHAI additionally supports object interaction and state variables such as open or closed containers.
  • Model: The agent observes its pose and an RGB image rather than the world state, defining the context used to select each action.
  • Model: Goal prediction represents likely target locations or objects as a probability distribution over visual observations.
  • Learning: Goal prediction uses supervised learning, while action generation uses oracle goals with policy gradient in a contextual bandit setting.
  • Learning and Evaluation: World-state access is assumed to compute training rewards, while evaluation measures task-completion accuracy and final-state distance to the goal.

3 Related Work

The paper builds on symbolic instruction mapping, direct visual-to-action learning, referring-expression interpretation, and prior language datasets. Its benchmarks extend existing environments with larger state spaces, interactive settings, and more complex manipulation.

  • Earlier instruction-following work used intermediate symbolic representations, while newer approaches map raw visual observations directly to actions.
  • The proposed goal predictor resembles referring-expression interpretation but targets visual goals for both navigation and manipulation within instruction following.
  • LANI adapts the Map Task’s landmark navigation structure while adding scalable data collection and an interactive navigation environment.
  • LANI has a significantly larger state space than prior navigation environments, while CHALET is larger and supports more complex manipulation than comparable environments.

4 Model

The model explicitly separates predicting an instruction-conditioned visual goal from generating actions toward it. LINGUNET produces a goal distribution over the initial panorama, whose inferred location is projected into each current view and supplied to an action-generating RNN.

  • Model overview: The policy decomposes instruction following into goal prediction from the initial panorama and recurrent action generation conditioned on that goal.The action probability is modeled as P(l_g | x̄, I_P) followed by P(a_t | l_g, (I_1,p_1), …, (I_t,p_t)).
  • Action generation: At each time step, the predicted goal is projected into the current observation as a binary mask and passed to an RNN that generates the next action distribution.The mask marks image elements corresponding to the goal location, without distinguishing visible from occluded locations.
  • Goal prediction: LINGUNET predicts a probability distribution over panorama feature-map pixels by conditioning U-NET-style reconstruction on the natural-language instruction.The goal distribution P_g is generated from text-conditioned feature maps and includes an additional out-of-sight bias position.
  • Architecture: The architecture encodes the instruction with an LSTM and the initial panorama with a CNN augmented by positional embeddings before applying language-conditioned convolutions and deconvolutions.Text-derived 1 × 1 kernels create text-conditioned feature maps that are decoded into the goal representation.
  • Goal prediction: The model selects the maximum-probability goal pixel and uses backward camera projection to recover its corresponding 3D environment location l_g.Projection uses camera parameters and the agent pose at the beginning of execution.

5 Learning

The decomposition permits different training procedures for the two subproblems: supervised learning for visual goal prediction and policy-gradient learning for exploratory action generation. Goal prediction uses gold goal distributions, while action generation maximizes reward with entropy regularization.

  • Learning strategy: The model uses supervised learning for goal prediction and contextual-bandit policy gradients for action generation.This choice separates language understanding, which has limited data, from action exploration, where autonomous exploration is useful.
  • Goal prediction: Goal prediction minimizes cross-entropy against a deterministic gold distribution at the goal pixel or an out-of-sight position.The goal is the agent’s location in the goal state, and parameters are updated with Adam.
  • Action generation: Action generation maximizes expected immediate reward while exploring the environment.The reward combines successful completion, penalties for incorrect completion or collision, and shaping based on movement toward or away from the goal.
  • Action generation: The action objective adds entropy regularization to the example-specific reward, with λ controlling the regularization coefficient.The objective is optimized using policy-gradient estimates based on sampled actions and gold goal locations.

6 Tasks and Data

The paper introduces LANI for 3D landmark navigation and CHAI for navigation plus household manipulation. Their crowdsourced instructions support evaluation of both complete instruction sequences and individual instructions, with CHAI presenting the more complex setting.

  • 6.1 LANI: LANI task completion requires stopping within an aerial distance of 5 from the goal, alongside stop distance error as an evaluation metric.Performance is evaluated using task completion accuracy and stop distance error.
  • 6.1 LANI: LANI instructions are crowdsourced from reference paths near landmarks, with workers also aligning instruction segments to agent behavior.The resulting data supports evaluation of complete instruction paragraphs and single instructions in isolation.
  • 6.2 CHAI: CHAI combines navigation and simple manipulation in simulated houses, including moving objects, picking and placing, and opening or closing containers.The corpus uses five houses with up to six rooms, each containing approximately 30 objects per room on average.
  • 6.2 CHAI: CHAI instructions are collected from 36 household scenarios and automatically segmented into sentences before workers continue the instruction sequence without seeing the original scenario.The data supports studying complete paragraphs and individual instructions, with 70%/15%/15% train, test, and development splits.

7 Experimental Setup

The CHAI adaptation adds intermediate-goal prediction and uses separate goal types to handle navigation and interaction sequences. Evaluation uses development data and compares against several behavioral baselines.

  • Method Adaptations for CHAI: CHAI execution predicts a sequence of intermediate goals, including NAVIGATION and INTERACTION, before applying the model separately to each goal type.For example, “pick up the red book and go to the kitchen” yields INTERACTION followed by NAVIGATION goals.
  • Method Adaptations for CHAI: The CHAI adaptation replaces backward camera projection with ray casting to identify INTERACTION goals involving objects away from the ground.
  • Evaluation: Development performance is reported in Table 3, while the specified action space reduces the number of actions required per instruction.
  • Baselines: The experiments compare the approach with STOP, RANDOMWALK, MOSTFREQUENT, MISRA17, and CHAPLOT18 baselines, alongside goal-prediction comparisons.

8 Results

The model improves navigation-related instruction following, especially on LANI, but CHAI remains difficult, particularly for manipulation. Human performance is imperfect and still substantially exceeds the approach, while goal prediction and decomposition ablations clarify key challenges.

  • Overall performance: On LANI, the approach improves task completion accuracy over CHAPLOT18 by 5%, while both methods outperform MISRA17.
  • Overall performance: On CHAI, CHAPLOT18 and MISRA17 fail to learn, while the approach improves stop distance; all models perform poorly, especially on manipulation.
  • CHAI navigation: 3.24 stop distance (SD) and a 17% reduction of error are achieved on navigation-only CHAI instructions, compared with an 8% reduction over the entire corpus.
  • Human evaluation: Human followers achieve 63% successful task completion on LANI and 100% manipulation accuracy on CHAI, but the gap to model performance remains large.
  • Human evaluation: Human followers receive a mean rating of 4.38 versus 3.78 for the approach in a 50-example LANI evaluation.
  • Ablations: Ablations show the largest benefit from decomposing learning and using supervised learning for the language problem, with especially ineffective CHAI behavior after ablation.
  • Oracle goals: Oracle goals significantly improve navigation on both tasks, but the model still fails to learn reasonable CHAI manipulation behavior.
  • Goal prediction: The approach outperforms Janner et al. (2018) on goal prediction, while temporal coordination and co-reference remain challenging.

9 Discussion

The discussion presents explicit goal–action separation as interpretable and avoids hand-designed symbolic representations, while identifying cascading and trajectory-constraint limitations.

  • The model separates goal prediction from action generation, producing an interpretable goal representation without manually designed logical ontologies.
  • Action generation depends completely on the predicted goal and is not otherwise exposed to the instruction.
  • Cascading errors can arise because incorrect goal predictions directly affect subsequent action generation.
  • The model is unlikely to reason successfully about instructions that constrain the intermediate execution trajectory.It may reach the final goal correctly while failing to account for how the goal is achieved.
  • LANI uses a shaped reward encouraging movement and turning toward the goal, while CHAI additionally rewards intermediate-goal interactions.CHAI updates the goal when an intermediate goal is accomplished.
  • Compared baselines include Misra et al.’s CNN–LSTM policy-gradient model and Chaplot et al.’s gated-attention architecture.
  • The proposed training setup jointly trains randomly initialized goal prediction and action generation models.

D Hyperparameters

The experiments tune models with held-out data and use fixed architectural, panorama, optimization, and training settings; Tables 7 and 8 summarize resource and CHAI error-analysis information.

  • LANI reserves 5% of training data for hyperparameter tuning, whereas CHAI uses its development set.
  • Models train for 20 epochs, with the optimal stopping epoch selected using the tuning set.
  • The models use 32-dimensional word and time embeddings and single-layer 256-unit LSTMs.
  • Table 7 compares LANI and CHAI with existing natural-language instruction corpora, while Table 8 reports CHAI mean goal-prediction error by analysis category.Table 8 also reports p-values from two-sided t-tests comparing row means.
  • Both benchmarks use 60° cameras and panoramas assembled from six 128×128 RGB images.
  • Optimization uses a learning rate of 0.00025 and entropy coefficient λ of 0.05, with LINGUNET dropout probability 0.5.

E CHAI Error Analysis

CHAI error analysis compares goal-prediction error across instruction-analysis categories, while example predictions show reasonable distributions even when tasks fail and expose strict LANI evaluation.

  • Predicted probability distributions were reasonable in many cases where the agent failed to complete the task successfully.
  • The LANI evaluation metric was often too strict, particularly for ambiguous instructions.Figure 7 marks LANI success or failure according to the task-completion metric.
  • Figure 7 displays predicted goal-probability maps over observed panoramas, with the top examples from LANI and the bottom examples from CHAI.A white arrow marks the agent’s forward direction.
Loading 1809.00786v2…