PaperlayerView exact revision on arXiv

Source-linked AI summary

Large Language Models as Generalizable Policies for Embodied Tasks

Andrew Szot, Max Schwarzer, Harsh Agrawal, Bogdan Mazoure, Walter Talbott, Katherine Metcalf, Natalie Mackraz, Devon Hjelm, Alexander Toshev

arXiv:2310.17722v2cs.LGcs.AIcs.CL

TL;DR

Embodied agents have not generalized to many new tasks as well as LLMs in other domains. LLaRP adapts a frozen pre-trained LLM into an online-RL vision-language policy, reaching 42% success on over 1,000 unseen tasks versus 25% for an LSTM-based policy and 22% for zero-shot LLM applications.

  • Problem

    Embodied agents’ generalization to many new tasks has not matched that seen in other domains, while approaches using static expert datasets require prohibitively large and expensive diverse data.

    Source: §1 · p. 1

  • Method

    LLaRP adapts a pre-trained frozen LLM into a vision-language policy with learned input and output adapters, trained through online reinforcement learning.

    Source: §1 · pp. 1–2

  • Results

    On over 1,000 unseen tasks, LLaRP achieves 42% success, compared with 25% for an LSTM-based policy and 22% for zero-shot LLM applications.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Language Rearrangement provides 150,000 distinct instructions with automatically generated rewards and evaluation splits stressing paraphrastic robustness and behavior generalization.

    Source: §1 · p. 2

    LLMs are significantly larger than typical RL models, and training the action decoder may hinder the policy’s ability to leverage the LLM’s world knowledge.

    Source: §6 · p. 9

Abstract

from arXiv · show

We show that large language models (LLMs) can be adapted to be generalizable policies for embodied visual tasks. Our approach, called Large LAnguage model Reinforcement Learning Policy (LLaRP), adapts a pre-trained frozen LLM to take as input text instructions and visual egocentric observations and output actions directly in the environment. Using reinforcement learning, we train LLaRP to see and act solely through environmental interactions. We show that LLaRP is robust to complex paraphrasings of task instructions and can generalize to new tasks that require novel optimal behavior. In particular, on 1,000 unseen tasks it achieves 42% success rate, 1.7x the success rate of other common learned baselines or zero-shot applications of LLMs. Finally, to aid the community in studying language conditioned, massively multi-task, embodied AI problems we release a novel benchmark, Language Rearrangement, consisting of 150,000 training and 1,000 testing tasks for language-conditioned rearrangement. Video examples of LLaRP in unseen Language Rearrangement instructions are at https://llm-rl.github.io.

1 INTRODUCTION

The introduction frames a gap in embodied-task generalization and presents LLaRP, an online-RL vision-language policy. It highlights generalization across paraphrased instructions and novel behaviors, training efficiencies, and a large rearrangement benchmark.

1 INTRODUCTION

  • LLMs have demonstrated capabilities beyond language, including visual understanding, reasoning, and robot control, often emerging zero-shot across non-language input and output spaces.The introduction motivates testing whether these broad capabilities can transfer to embodied decision-making.Source: §1 · p. 1
  • LLaRP uses a pre-trained frozen LLM as a vision-language policy with learned input and output adapters, trained through online reinforcement learning.The policy takes language and visual observations and outputs actions in the environment.Source: §1 · pp. 1–2
  • Evaluation covers paraphrastic robustness to changed wording or object references and behavior generalization to tasks requiring novel outcomes, including new object combinations and logical conditions.The figure also describes generalization to unseen objects and scenes, spatial descriptions, and variable numbers of rearrangements.Source: §1 · p. 2, Figure 1 · p. 2
  • On over 1,000 unseen tasks, LLaRP achieves 42% success, versus 25% for an LSTM-based policy and 22% for zero-shot LLM applications.The paper also reports that LLaRP outperforms baselines on novel instructions and unseen behaviors, and gives a non-trivial boost over a Transformer baseline on Atari.Source: §1 · p. 2
  • LLM-based models show better sample efficiency than other common architectures in basic PPO and continual learning, and LLaRP needs less supervision than commonly used imitation learning.The paper attributes these training-efficiency findings to access to world knowledge encoded in the LLM.Source: §1 · p. 2
  • Language Rearrangement supplies 150,000 distinct instructions with automatically generated rewards and evaluation splits that stress-test paraphrastic and behavior generalization.The benchmark is introduced to support research on generalization in Embodied AI.Source: §1 · p. 2

2 RELATED WORK

Prior work applies LLMs to text-based decision-making and embodied vision-language systems, but the approaches described here differ in their use of online reinforcement learning, observations, or expert data.

2 RELATED WORK

  • Prior LLM decision-making policies often use text-based states and actions, while other work fine-tunes LLMs for text decisions or uses them to supply rewards and high-level goals.Code as Policies relies on perception modules unavailable in the described setting.Source: §2 · p. 2
  • LLaRP’s architecture feeds encoded task instructions and egocentric RGB history to a pre-trained LLM, then projects hidden outputs to action and value predictions.Online RL trains only the action-output module and observation-encoder MLP; the other components remain frozen.Source: Figure 2 · p. 3
  • Many language-conditioned policies are trained by imitation on expert trajectory-language pairs, whereas Perceiver-actor requires offline data; other work augments those datasets or studies open-vocabulary manipulation.The cited examples include pick-and-place rearrangement and object manipulation in navigation settings.Source: §2 · p. 3
  • Interactive embodied VLMs such as PaLM-E and RT-2 use high-quality expert data, while many works adding 3D spatial information operate in static settings.The passage contrasts those settings with interactive decision-making in visual embodied environments.Source: §2 · p. 3
  • The authors state that prior work had not demonstrated improved generalization from LLMs used as vision-language policies in online RL; Carta et al. used textual observations and actions instead.The paper evaluates both linguistic and target-behavior variation and describes Language Rearrangement as requiring manipulation, navigation, and exploration.Source: §2 · p. 3

3 METHOD

LLaRP adapts a pretrained LLM into a goal-conditioned policy that maps language instructions and egocentric visual observations to actions, targeting generalization to unseen instruction distributions. It uses learned adapters and interactive reinforcement learning while keeping the LLM backbone and visual encoder frozen.

3 METHOD

  • The policy seeks to maximize discounted reward for natural-language goals and generalize to instruction distributions not seen during training.The formulation treats egocentric visual observations as partial views of the scene and conditions rewards on the goal.Source: §3 · p. 4

3.1 LARGE LANGUAGE MODEL REINFORCEMENT LEARNING POLICY ARCHITECTURE

  • LLaRP encodes language goals and visual observations as token embeddings, processes them with an LLM backbone, then decodes outputs into environment actions.A learnable observation encoder projects visual features to the language-model embedding dimension, while the action module predicts an action distribution and value estimate.Source: §3.1 · p. 4

3.2 LARGE LANGUAGE MODEL REINFORCEMENT LEARNING POLICY TRAINING

  • LLaRP is trained through interactive rollouts and reinforcement-learning updates, with both the LLM backbone and visual encoder frozen during training.The policy collects experience by generating actions autoregressively and uses DD-PPO updates on minibatches of collected data.Source: §3.2 · p. 4
  • Table 1 divides unseen-instruction evaluation datasets into paraphrastic robustness, which tests linguistic variation, and behavior generalization, which tests new behavior types.Red text marks the instruction concept distinct from the training distribution.Source: Table 1 · p. 5

4 LANGUAGE REARRANGEMENT PROBLEM

Language Rearrangement is a large language-conditioned benchmark for household object rearrangement, with training and testing tasks designed to assess generalization. Its evaluations distinguish new phrasings of familiar behavior from instructions requiring behavior not seen in training.

4 LANGUAGE REARRANGEMENT PROBLEM

  • Language Rearrangement defines 150,000 training and 1,000 testing tasks involving household activities and varied instructions that require object interactions, search, and logical reasoning.Tasks include unseen instruction concepts and may involve multiple object interactions such as picking, placing, opening, and closing.Source: §4 · p. 5

4.1 TASK DEFINITION

  • Tasks specify household goal states in natural language; instructions are generated from templates, rephrasings, scene-grounded entities, and feasibility checks, with sparse rewards for task or subtask completion.Some instructions require exploration, and task conditions vary in disclosed object locations, object counts, and logical operators.Source: §4.1 · p. 5
  • The high-level policy selects among 70 low-level skills, while the agent relies on onboard sensing without prebuilt maps, object models, or other privileged information.Skills include object-specific picking, receptacle placement, navigation, and opening or closing receptacles.Source: §4.1 · pp. 5–6

4.2 GENERALIZATION TEST DATASETS

  • Evaluation uses unseen houses and separates paraphrastic robustness—new wording for familiar goals—from behavior generalization, which requires a behavior absent from training.Behavior generalization includes new logical expressions, such as finding all objects of a type.Source: §4.2 · p. 6

5 EXPERIMENTS

On Language Rearrangement, LLaRP generalizes better than baselines on most unseen task groups and learns efficiently. Its strongest results span aggregate zero-shot generalization, novel behaviors, and faster learning across training settings.

5.2 EMPIRICAL ANALYSIS

  • 45% vs 28%: LLaRP exceeds LSTM-Flan on Behavior Generalization, which tests tasks requiring novel optimal behaviors.The paper also reports gains on paraphrastic robustness and argues that LLMs help with novel behaviors.Source: §5.2 · p. 7
  • LLaRP is 7% and 12% better than the next-best model on rephrased instructions and novel objects, with substantial boosts on more complex novel tasks.Examples of the complex tasks include context, conditional statements, and multiple chained rearrangements.Source: §5.2 · p. 7
  • 42% vs 25%: LLaRP achieves almost 1.7x the next-best baseline’s combined zero-shot success rate across generalization types.The comparison aggregates results across generalization datasets.Source: Table 2 · p. 7, §5.2 · p. 7
  • 50M–100M steps: LLaRP converges before LSTM-Flan, while LLaRP-Scratch takes 10x more samples than LLaRP to converge.The stated comparison is 50M versus 500M environment samples for LLaRP and LLaRP-Scratch, respectively.Source: Figure 4 · p. 8, §5.2 · p. 8
  • For any matched number of episodes or demonstrations, RL outperforms IL, with a larger margin in the low-data regime.RL requires a reward definition, whereas IL requires full demonstration trajectories.Source: Figure 4 · p. 8, §5.2 · p. 9
  • 3x more efficient: LLaRP reaches near-perfect performance on continual Multiple Rearrangements learning in 500k steps, versus 1.5M for LSTM-Flan.This evaluates learning on downstream tasks beyond the training tasks.Source: §5.2 · p. 8
  • 4% boost: LLaMA-13B gives higher total evaluation performance than LLaMA-7B in LLaRP.The figure compares average success rates across evaluation datasets.Source: Figure 5 · p. 8, §5.2 · p. 9
  • 43 of 55 games: LLaRP achieves a higher average Atari score than LLaRP-Scratch, with mean, median, and interquartile-mean performance 3x–4x higher when aggregated.The comparison uses two seeds per game.Source: Figure 7 · p. 9, §5.2 · p. 9

6 CONCLUSION

The conclusion presents LLaRP as a reinforcement-learning scheme for pretrained LLMs and introduces a rearrangement dataset with 150,000 training instructions and 10 evaluation datasets. It identifies LLM size and the action decoder as limitations for future work.

6 CONCLUSION

  • LLaRP outperforms nonpretrained transformer- and LSTM-based models on sample efficiency and generalization, while its larger LLM size and action decoder remain limitations.The authors suggest future work could let LLaRP interact through the LLM language head without an action decoder.Source: §6 · p. 9

B.1 PLANNING DOMAIN DEFINITION LANGUAGE (PDDL) DETAILS

  • Language Rearrangement episodes use PDDL entities, hierarchical types, and predicates to represent objects, receptacles, and simulator-state conditions.Predicate expressions combine predicates, entities, types, logical connectives, and quantifiers to define instruction success criteria.Source: §B.1 · pp. 17–18

B.2 INSTRUCTION GENERATION DETAILS

  • The instruction-generation pipeline samples templates, substitutes constrained entities, and selects compatible scenes to create grounded goals and language instructions.Each template pairs an ungrounded goal condition with placeholder constraints and 11 language templates expressing that goal.Source: §B.2 · p. 18

B.3 SKILL DETAILS

  • The policy selects among 70 parameterized navigation, pick, place, open, and close skills; invalid actions are treated as no-ops.A skill executes only when valid in the current state.Source: §B.3 · p. 19

B.4 ADDITIONAL TASK DETAILS

Language Rearrangement uses high-level skills in Habitat 2.0 and defines success through goal satisfaction within a step limit. Its reward combines task completion, subgoal progress, invalid-action cost, and a per-step penalty; evaluation datasets test language and behavior generalization.

B.4 ADDITIONAL TASK DETAILS

  • Episodes use kinematic skill-state updates in Habitat 2.0, and success requires satisfying the goal within 32 high-level policy steps.The evaluation includes held-out object categories, including mug, orange, lid, toy airplane, and wrench.Source: §B.4 · pp. 19–20
  • The reward combines 10 for success, 5 for a subgoal, −0.1 for an invalid action, and a −0.01 step penalty.Subgoals come from an oracle planner's optimal action sequence, which the agent cannot directly imitate because it uses oracle state information.Source: §B.4 · p. 20

B.5 ADDITIONAL DATASET DETAILS

  • Paraphrastic-robustness evaluations cover rephrasing, visual referring expressions, spatial relationships, contextual descriptions, and irrelevant instruction text.Referring expressions use visual descriptions rather than entity names, which are the only object references seen during training.Source: §B.5 · p. 20
  • Behavior-generalization evaluations test rearranging three objects after training on two, novel objects, “for all” instructions, and conditional behavior tied to whether the fridge is open.Conditional tasks also require leaving the object that does not need moving undisturbed.Source: §B.5 · pp. 20–21

B.6 DATASET ANALYSIS

The dataset analysis finds that evaluation instructions and scenes differ from training, with additional variability in instruction length and vocabulary. The authors use instruction embeddings and token counts to characterize these differences.

B.6 DATASET ANALYSIS

  • Evaluation datasets use different instruction templates and unseen scenes, and all episodes use unseen object positions.The unseen scenes come from the ReplicaCAD evaluation split and differ in kitchen and furniture locations.Source: §B.6 · p. 21
  • Training instructions show minimal embedding overlap with evaluation instructions in a t-SNE visualization of Flan-T5-XL embeddings.The visualization uses 100 randomly selected instructions from each dataset.Source: §B.6 · p. 21
  • Conditional-instruction prompts reach approximately twice the median task length, while instruction word frequencies are long-tailed across datasets.Common words include receptacle and relative-location nouns, action verbs, and articles.Source: §B.6 · pp. 21–22

B.7 COMPARING LANGUAGE REARRANGEMENT TO OTHER BENCHMARKS

Language Rearrangement is compared with embodied AI and interactive instruction-following benchmarks. The paper highlights its instruction scale, template diversity, dense rewards, and use of kinematic rather than dynamically simulated low-level skills.

B.7 COMPARING LANGUAGE REARRANGEMENT TO OTHER BENCHMARKS

  • Language Rearrangement has almost 19x more instructions than ALFRED and 8x more instruction templates than CALVIN, and provides dense rewards where the compared tasks use sparse success rewards.The benchmark comparison is presented in Table 4.Source: §B.7 · p. 22, Table 4 · p. 22
  • Although Habitat 2.0 supports dynamic physics, the benchmark evaluates skills through kinematic state updates rather than dynamically simulated low-level execution.Other cited tasks that simulate full robot control are constrained to tabletop robot-arm settings.Source: §B.7 · p. 22

C METHOD DETAILS

This appendix details LLaRP’s training setup, model variants, and baselines, along with modifications for scaling LLaMA and training on Atari.

C METHOD DETAILS

  • Training used a full node of 8 A100-80GB GPUs, data parallelism, and DD-PPO to handle straggler environment workers.Hyperparameters for reinforcement-learning methods are summarized in Table 3; Table 5 collects zero-shot Language Rearrangement results across baselines and settings.Source: C METHOD DETAILS · pp. 22–23, Table 5 · p. 23

C.1 LLARP DETAILS

  • LLaRP uses RGB observations with a frozen VC1 encoder, projects the image [CLS] embedding to the LLM token dimension, and maps outputs to actions using a two-layer MLP with ReLU, LayerNorm, and 512 hidden units.By default, LLaRP uses base LLaMA-7B V1, with weights converted to bfloat16; its Language Rearrangement context window is 32 steps. Training throughput is 700–800 steps per second on a full compute node of 8 GPUs, including inference, updates, and environment stepping.Source: §C.1 · p. 23

C.2 LLARP-SCRATCH DETAILS

  • LLaRP-Scratch uses a roughly 2-billion-parameter transformer decoder whose parameters are all updated during training.Training throughput is reported at 800–900 steps per second on a full compute node.Source: §C.2 · p. 23

C.3 LSTM-FLAN/LLAMA DETAILS

  • LSTM-Flan fine-tunes the Flan-T5-XL encoder to represent instructions, whereas LSTM-LLaMA uses frozen LLaMA-7B weights and a Perceiver instruction representation.Both methods use VC1 image features; LSTM-Flan concatenates the projected image embedding with the language representation before the LSTM.Source: §C.3 · p. 24

C.4 ZERO-SHOT BASELINE DETAILS

  • Zero-shot baselines include language-only ZS-LLaMA, multimodal ZS-Flamingo, and feedback-driven ZS-ChatGPT, with prompts supplying action rules and examples.ZS-LLaMA plans from language alone; ZS-ChatGPT can revise its plan using natural-language environment feedback.Source: §C.4 · pp. 24–25

C.5 MODIFICATIONS TO SCALE AND TRAIN LLAMA

  • LLaRP variants with larger or trainable LLaMA models require model parallelism across four GPUs and training scaled to four eight-GPU nodes.The batch size is matched across the scaled training setup.Source: §C.5 · p. 27

C.6 MODIFICATIONS FOR ATARI

  • For Atari, a trainable visual encoder was needed to reliably solve Pong within 100M steps; Huber value loss, gradient-norm clipping at 0.5, and reward clipping addressed training instability.A per-device batch size of 64 with context length 32 exposes 16,384 states across eight GPUs.Source: §C.6 · p. 27

D.1 TRANSFORMER POLICY EXPERIMENTS

The section examines stability and efficiency in transformer-based online RL. Frozen pretrained LLM weights and larger batches support training, though sample-efficiency patterns differ by comparison.

Transformer Policy Experiments

  • The experiments address the challenge of stabilizing transformer-based policies that attend to previous observations in online RL.The section contrasts these policies with the more typical use of RNN-based policies when history matters.Source: §D.1 · p. 27
  • The authors report training billion-parameter transformer policies with PPO on high-dimensional visual tasks without Parisotto et al.’s transformer modifications.They describe those modifications as incompatible with LLM architectures.Source: §D.1 · pp. 27–28

Effect of Batch Size on Stability

  • At lower batch sizes, transformer policies had more unstable learning curves than same-sized RNN policies, and were less sample efficient at every tested batch size.The sample-efficiency comparison reverses for LLaRP versus RNN approaches in Language Rearrangement.Source: §D.1 · p. 28

Effect of Pretrained LLM Weights

  • Under the same architecture, frozen pretrained LLM weights converged in under 100M steps, while LLaRP-Scratch required 500M samples and showed unstable performance dips.The comparison concerns training performance for LLaRP and LLaRP-Scratch.Source: §D.1 · p. 28

D.2 ADDITIONAL LANGUAGE REARRANGEMENT EXPERIMENTS AND ANALYSES

The analyses compare Language Rearrangement efficiency and training choices, then test a harder setting with strict action-failure and termination rules.

D.2 ADDITIONAL LANGUAGE REARRANGEMENT EXPERIMENTS AND ANALYSES

  • LLaRP was over 1.5x as efficient as the best-performing baseline on zero-shot Language Rearrangement evaluation.Efficiency averages a per-episode score based on steps to completion, assigning unsuccessful episodes the maximum step count; Figure 11 rescales values to [0, 1].Source: Figure 11 · p. 28, §D.2 · pp. 28–29

D.3 HARDER TASK VARIANT

  • In the harder task setting, LLaRP greatly outperformed LSTM-Flan, though both policies lost performance and remained unconverged after 700M training steps.Invalid actions ended episodes, and a termination action was required after task completion.Source: Figure 12 · p. 29, §D.3 · p. 29

D.2 ADDITIONAL LANGUAGE REARRANGEMENT EXPERIMENTS AND ANALYSES

  • Fine-tuning the Flan-T5 encoder helped LSTM-Flan learn harder instructions and converge faster; with frozen weights, it learned only easy instructions even after 200M steps.Figure 10b compares freezing versus unfreezing the encoder.Source: §D.2 · p. 29
  • LLaRP full fine-tuning had poor training performance and was run for only 15M steps because of slow training, leaving longer training and larger batches as possible improvements.Table 5 includes the full-finetuning variant among its reported settings.Source: §D.2 · pp. 28–29

E QUALITATIVE RESULTS

Qualitative examples show LLaRP exploring and acting on inferred object and task relationships, alongside recurring failure modes across difficult instruction types.

Failure modes

  • Reported failures include aimless navigation for context or referring-expression instructions, wrong-object actions, incomplete rearrangements, and missed second objects.The authors also report failure to check whether the fridge is open before a conditional rearrangement.Source: E QUALITATIVE RESULTS · pp. 29–30

Qualitative successes

  • Successful trajectories include inferring receptacles and tools from instruction context, handling a previously unseen wrench, and searching for multiple fruit objects.Examples also show mapping “yellow curved fruit” to a banana and moving it to the blue table.Source: Figure 13 · p. 31

Qualitative results

Loading 2310.17722v2…