PaperlayerView exact revision on arXiv

Source-linked AI summary

ALFWorld: Aligning Text and Embodied Environments for Interactive Learning

Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Côté, Yonatan Bisk, Adam Trischler, Matthew Hausknecht

arXiv:2010.03768v2cs.CLcs.AIcs.CVcs.LGcs.RO

TL;DR

Embodied agents need infrastructure that supports both abstract language reasoning and concrete visual execution. ALFWorld aligns TextWorld with ALFRED, and BUTLER transfers TextWorld-learned policies to embodied tasks, improving generalization while training faster than embodied-from-scratch approaches.

  • Problem

    Existing work lacks infrastructure for embodied agents to reason abstractly in language and execute concretely in visual environments.

  • Method

    ALFWorld aligns interactive TextWorld and ALFRED environments, while BUTLER learns high-level text policies before using them to generate embodied actions.

  • Results

    BUTLER generalizes zero-shot to unseen embodied tasks, trains 7× faster, and performs better than training from scratch in the embodied world.

  • Takeaways & Limitations

    Reasoning in textual space can provide priors that support embodied generalization, and ALFWorld's modular pipeline permits independent improvements to language, planning, navigation, and vision.

  • Takeaways & Limitations

    Transfer is constrained by domain gaps between text and embodied worlds, including physical-size constraints, visual misdetections, and collision-related navigation failures.

Abstract

from arXiv · show

Given a simple request like Put a washed apple in the kitchen fridge, humans can reason in purely abstract terms by imagining action sequences and scoring their likelihood of success, prototypicality, and efficiency, all without moving a muscle. Once we see the kitchen in question, we can update our abstract plans to fit the scene. Embodied agents require the same abilities, but existing work does not yet provide the infrastructure necessary for both reasoning abstractly and executing concretely. We address this limitation by introducing ALFWorld, a simulator that enables agents to learn abstract, text based policies in TextWorld (Côté et al., 2018) and then execute goals from the ALFRED benchmark (Shridhar et al., 2020) in a rich visual environment. ALFWorld enables the creation of a new BUTLER agent whose abstract knowledge, learned in TextWorld, corresponds directly to concrete, visually grounded actions. In turn, as we demonstrate empirically, this fosters better agent generalization than training only in the visually grounded environment. BUTLER's simple, modular design factors the problem to allow researchers to focus on models for improving every piece of the pipeline (language understanding, planning, navigation, and visual scene understanding).

1 INTRODUCTION

ALFWorld aligns abstract text interaction with physically embodied simulation, enabling agents to learn language-based policies and transfer them to visual tasks. BUTLER uses this framework to generalize from TextWorld to unseen embodied settings, with faster training and better performance than embodied-from-scratch training.

  • The framework tests whether abstract reasoning exposes semantic priors that help agents complete embodied tasks in novel environments.
  • ALFWorld provides parallel interactive views of the same world: TextWorld supplies textual observations and high-level actions, while ALFRED renders images and executes low-level physical actions.
  • TextWorld supplies programmatic linguistic feedback during exploration, enabling embodied agents to learn complex multi-step policies expressed in language through interaction.
  • BUTLER first learns abstract tasks in TextWorld with imitation learning, then transfers its policies to embodied ALFRED tasks through high-level text subgoals.
  • 7× faster training and better performance distinguish TextWorld-first training from training from scratch in the embodied world.

2 ALIGNING ALFRED AND TEXTWORLD

ALFWorld aligns diverse ALFRED household tasks with interactive TextWorld counterparts through a shared latent world structure. The resulting framework supports interaction and evaluation across language and embodied modalities.

  • ALFRED evaluates household-task completion from natural-language goals and egocentric visual observations using low-level navigation and manipulation actions.
  • The benchmark omits crowdsourced step-by-step instructions and focuses on goal descriptions specifying only what must be achieved.
  • ALFWorld adopts six ALFRED task types involving object discovery, receptacle search, object interaction, and placement in a designated location.
  • Its 120 rooms and varied object, receptacle, location, quantity, and appearance configurations provide seen and unseen evaluation settings.
  • TextWorld analogs are generated from ALFRED scenes using a shared latent structure and PDDL-defined dynamics.
  • High-level actions such as heat, cool, clean, and goto correspond to multiple low-level embodied actions.
  • Because both modalities are interactive, agents can train and be evaluated in either the language or embodied world on held-out tasks.

3 INTRODUCING BUTLER: AN EMBODIED MULTI-TASK AGENT

BUTLER is a modular agent that learns abstract high-level policies in text before executing them in embodied environments. Its components translate between textual observations, language actions, and low-level physical actions.

  • Architecture: BUTLER separates embodied task solving into a text agent, a visual state estimator, and a low-level controller.BUTLER::BRAIN generates high-level text actions, BUTLER::VISION converts visual frames into textual observations, and BUTLER::BODY executes those actions physically.
  • BUTLER::BRAIN: BUTLER::BRAIN encodes initial and current observations with the goal, then generates high-level action sentences token by token.A recurrent aggregator and observation queue provide history because the games are partially observable.
  • BUTLER::BRAIN: During evaluation, BUTLER::BRAIN uses beam search to propose alternative action sentences after failed actions.This addresses cases where failures provide insufficient feedback and a deterministic actor might repeat the same incorrect action.
  • BUTLER::VISION: BUTLER::VISION uses a separately trained Mask R-CNN detector to convert each visual frame into a templated textual observation.Detected objects receive class-and-instance identifiers, while successful and unsuccessful actions produce affirmative responses or “Nothing happens.”
  • BUTLER::BODY: BUTLER::BODY converts each high-level text action into executable low-level manipulation or navigation actions.Manipulation uses ALFRED API actions and pixel masks, while navigation uses A* over a pre-built grid map and executes motion primitives open-loop.

4 EXPERIMENTS

The experiments test interactive language training, transfer from TextWorld to embodied tasks, human-goal generalization, and alternative training strategies. Results support interactive TextWorld training and reveal domain gaps that hinder embodied transfer.

  • Importance of Interactive Language: The experiments compare interactive TextWorld training with static expert demonstrations using DAgger and Behavior Cloning.The agents use the same expert policies and matched episode counts for a fair comparison.
  • Importance of Interactive Language: BUTLER improves over Seq2Seq on all seen task types and five of seven unseen task types.The result supports interactive TextWorld training as a component of generalization to unseen embodied tasks.
  • Transferring to Embodied Tasks: Zero-shot transfer must address physical constraints absent from TextWorld, such as object and receptacle size compatibility.TextWorld may permit placing an object into any receptacle even when the embodied environment makes the placement physically impossible.
  • Transferring to Embodied Tasks: Even with perfect state estimation, object detection, and navigation, BUTLER-ORACLE shows a notable performance drop from TextWorld to embodied tasks.The paper attributes the gap to modality differences, detector misdetections, and collision-related navigation failures.
  • Generalizing to Human-Annotated Goals: Human-annotated goals contain 66 unseen verbs and 189 unseen nouns, yet BUTLER achieves non-trivial goal completion on some categories.Pick-and-place tasks are described as relatively generalizable to human language.
  • To Pretrain or Not to Pretrain in TextWorld?: The training-strategy comparison evaluates EMBODIED-ONLY, TW-ONLY, and HYBRID agents after 50K episodes in embodied scenes with oracle estimation and control.TW-ONLY uses pure TextWorld training and zero-shot transfer; HYBRID uses TextWorld 75% of the time and embodied training 25% of the time.

5 ABLATIONS

The ablations examine training strategies, modalities, decoding, observation history, and training-game scale across TextWorld and embodied settings. Interactive training, beam search, textual observations, recurrent history aggregation, and more training games support generalization, while visual models and fewer games overfit.

  • Training strategy: Reinforcement learning struggled because token-by-token generation makes valid actions extremely unlikely.For sequence length 10, the probability of randomly generating a grammatically correct and contextually valid action is 7.02e-44.
  • Training strategy: DAgger with a rule-based expert replaced unsuccessful reinforcement learning for training BUTLER::BRAIN over 100K episodes.The agent was trained by interacting with the set of training games.
  • Decoding and supervision: Beam-search ablation reduced seen All Tasks performance by 21%, while removing DAgger and using demonstrations reduced it by 30%.The results identify online interaction through DAgger and beam search as important for recovering from mistakes and sub-optimal behavior.
  • Unimodal baselines: Visual baselines overfit seen environments and generalize poorly to unfamiliar environments, whereas text-space policies transfer without learning visually robust state representations.The ACTION-ONLY baseline achieved zero performance, indicating that memorizing action sequences is infeasible.
  • Model components: Keeping the initial receptacle observation improves receptacle-word generation for unseen tasks, and recurrent aggregation helps history-based decisions.The initial observation may be unnecessary in seen environments, while history is useful for tracking object locations.

6 RELATED WORK

ALFWorld extends interactive environment research by aligning textual interaction with visual and physical embodiment. It differs from prior text-only, vision-language, embodied-language, and language-mediated policy approaches in the degree and role of interaction.

  • Interactive environments: ALFWorld extends interactive environments with fully interactive alignment between textual interactions, photo-realistic renderings, and physical interactions.This addresses the broader goal of grounding language learning in embodied settings.
  • Interactive text-only environments: TextWorld and Jericho support textual interaction but are not grounded in visual or physical modalities.ALFWorld builds on this text-based environment tradition while adding aligned embodiment.
  • Vision and language: Vision-language representation-learning systems generally lack embodied or sequential decision making.The cited examples include MAttNet, CMN, VQA, CLEVR, and ViLBERT.
  • Embodied language learning: Embodied-language environments communicate instructions, goals, or queries but do not provide fully interactive textual modalities.The examples include BabyAI, Room2Room, ALFRED, InteractiveQA, EmbodiedQA, and NetHack.
  • Language for state and action representation: Prior work also uses language as an intermediate state, representation, instructor, or abstraction for transferring or executing policies.These approaches include language-mediated policies in VizDoom and other environments, natural-language instruction of low-level executors, and language-based abstraction.
  • World models: Using TextWorld as a game engine relates ALFWorld to inverse graphics, inverse dynamics, and proposals that language can substitute for sensorimotor simulation.These connections frame abstract world representations as tools for reasoning and future prediction.

7 CONCLUSION

The paper introduces ALFWorld as an aligned interactive text and embodied environment and uses TextWorld pre-training to support zero-shot transfer to ALFRED tasks. It concludes that textual reasoning improves unseen-task generalization and training speed relative to vision, while leaving modular components open to replacement.

  • Conclusion: ALFWorld provides an interactive text environment aligned with embodied worlds.Agents can explore, interact, and learn abstract policies in the textual environment.
  • Conclusion: Pre-training BUTLER in TextWorld enables zero-shot generalization to embodied tasks in the ALFRED dataset.The conclusion presents this as the central transfer result of the framework.
  • Conclusion: Textual-space reasoning yields better generalization to unseen tasks and faster training than other modalities such as vision.The paper states this as an empirical conclusion rather than a universal claim about all modalities.
  • Future work: BUTLER’s modular components can be independently upgraded, including replacing the template-based state estimator and A* navigator with learned modules.The proposed future direction includes end-to-end training of the full pipeline.
  • Future work: Future textual dynamics models could support text engines for new domains without symbolic state descriptions such as PDDL.This is presented as an avenue for extending the framework beyond its current components.

A.1 OBSERVATION QUEUE

BUTLER encodes current text observations and goals, aggregates their interactions with attention, incorporates history recurrently, and decodes actions using generation and source pointing.

  • A.1 OBSERVATION QUEUE: An observation queue preserves the initial room-layout observation while concatenating recently cached observations for BUTLER::BRAIN.The initial observation remains visible at every game step, regardless of queue length.
  • A.1 OBSERVATION QUEUE: A shared transformer encoder processes each text observation and goal, producing token representations with hidden size H = 64.The embeddings are initialized from fixed pre-trained 768-dimensional BERT embeddings.
  • A.1 OBSERVATION QUEUE: Tri-linear similarity computes observation-goal token interactions, which are normalized along both dimensions to aggregate the two representations.The similarity matrix has dimensions determined by the observation and goal token counts.
  • A.1 OBSERVATION QUEUE: A GRU incorporates history after mean pooling the aggregated token representation into the recurrent state.The recurrent representation has dimensionality H, and the previous GRU output supplies the prior state.
  • A.1 OBSERVATION QUEUE: The decoder combines observation-goal and recurrent representations, then uses attention and a pointer softmax to generate or copy action tokens.The pointer mechanism interpolates vocabulary generation with pointing to source-text tokens.

B TRAINING AND IMPLEMENTATION DETAILS

The implementation combines imitation-learning supervision, recurrent training, beam-search recovery, symbolic TextWorld state updates, contextual text generation, and visual detection for embodied execution.

  • B TRAINING AND IMPLEMENTATION DETAILS: The system uses Adam optimization, DAgger transitions from a rule-based expert, replay-buffer training, and a maximum episode length of 50 steps.DAgger collects batches of 10 transitions, while the replay buffer stores up to 500K episodes.
  • B TRAINING AND IMPLEMENTATION DETAILS: Recurrent training samples sequences of four transitions, using the first two to estimate recurrent states and the last two to update parameters.The agent is updated after every five data-collection steps from batches of 64 data points.
  • B TRAINING AND IMPLEMENTATION DETAILS: BUTLER::BRAIN generates actions token by token, stopping at [EOS] or a maximum token length of 20.This decoder produces action sequences rather than selecting only from a fixed command list.
  • B TRAINING AND IMPLEMENTATION DETAILS: Beam-search recovery uses width 10, tests the top five ranked outputs in order, and is applied during evaluation but not training.The heuristic is helpful in most cases but is not guaranteed to succeed.
  • B TRAINING AND IMPLEMENTATION DETAILS: TextWorld maintains predicate-based game states with a classical planner and realizes context-sensitive textual observations from state and last action.Applying an action replaces predicates according to PDDL-defined production rules, while the text generator samples and realizes templates.
  • B TRAINING AND IMPLEMENTATION DETAILS: A Mask R-CNN detector is fine-tuned on 50K simulator-generated images and instance-segmentation labels from ALFRED demonstrations.The rule-based expert decomposes tasks into subgoals executed sequentially by a closed-loop controller.

F BENEFITS OF TRAINING IN TEXTWORLD OVER EMBODIED WORLD

Training in TextWorld provides an abstract counterpart to embodied interaction, with aligned observations and actions supporting evaluation across textual and visual task environments.

  • F BENEFITS OF TRAINING IN TEXTWORLD OVER EMBODIED WORLD: TextWorld pre-training yields higher expert goal-success rates across environments through stronger navigation and manipulation subroutines.The abstract textual space avoids object misdetections and slow execution speed.
  • F BENEFITS OF TRAINING IN TEXTWORLD OVER EMBODIED WORLD: Failed actions and actions without state changes produce the textual observation “Nothing happens.”Object identifiers come from detections or ground-truth instances, while receptacle identifiers come from the initial observation.
  • F BENEFITS OF TRAINING IN TEXTWORLD OVER EMBODIED WORLD: ALFWorld represents high-level text actions together with their observation templates for aligned interaction.These templates define the textual interface used by the environment.
  • F BENEFITS OF TRAINING IN TEXTWORLD OVER EMBODIED WORLD: Training games use templated goal descriptions sampled from task-specific object, receptacle, and lamp classes.The two corresponding templates for each task are sampled with equal probability.

H.2 HUMAN ANNOTATED GOALS

Human-annotated evaluation goals contain substantial lexical novelty, while action-generation design addresses failures of fixed candidate-command approaches during embodied transfer.

  • H.2 HUMAN ANNOTATED GOALS: Human evaluation goals contain 66 unseen verbs and 189 unseen nouns relative to the templated training instructions.The unseen vocabulary includes action terms and environment-object terms not present in training templates.
  • H.2 HUMAN ANNOTATED GOALS: Candidate commands from the TextWorld Engine caused rapid overfitting without learning affordances, commonsense, or pre-conditions.Agents trained with those candidates achieved zero performance on embodied transfer.
  • H.2 HUMAN ANNOTATED GOALS: Populating command templates was infeasible because some scenarios produced thousands of candidate actions per game step.In the embodied setting, candidate generation would otherwise require handcrafted heuristics with strong priors and commonsense knowledge.

J ALFRED TASK DESCRIPTIONS

ALFWorld describes six task types that require agents to locate objects, manipulate them, interact with fixtures, and place them correctly. Walkthroughs illustrate these tasks through text-based observations and action sequences.

  • Pick & Place requires finding an object, picking it up, locating the correct destination, and putting it down.
  • Examine in Light requires finding an object, locating a light source, and turning it on while holding the object.
  • Clean & Place, Heat & Place, and Cool & Place add sink, microwave, or refrigerator interactions before placing the object.These tasks require washing, heating, or cooling the held object before moving it to its final location.
  • Pick Two & Place requires completing a pick-and-place sequence twice and returning the second object to the previous destination.
  • Text walkthroughs show agents using room observations to navigate, retrieve objects, operate fixtures, and complete goals.Examples include taking an alarm clock and using a desk lamp, cleaning a cloth at a sink, and placing a remote control on an armchair.
Loading 2010.03768v2…