PaperlayerView exact revision on arXiv

Source-linked AI summary

Grounding Large Language Models in Interactive Environments with Online Reinforcement Learning

Thomas Carta, Clément Romac, Thomas Wolf, Sylvain Lamprier, Olivier Sigaud, Pierre-Yves Oudeyer

arXiv:2302.02662v5cs.LG

TL;DR

LLM knowledge about physical rules can be misaligned with interactive environments, motivating functional grounding through interaction. GLAM uses an LLM as a policy and updates it with online RL in BabyAI-Text; experiments report improved task performance, sample efficiency, and zero-shot generalization. The approach is not designed for real-world deployment.

  • Problem

    LLM knowledge may be misaligned with interactive environments because language-model training does not directly incentivize environmental problem solving or learning from interaction data.

  • Method

    GLAM functionally grounds an LLM policy by updating it with online RL using observations and rewards in the textual BabyAI-Text environment.

  • Results

    GLAM improved performance over zero-shot LLM use, supervised finetuning, and RL finetuning of non-pretrained LLMs, while boosting sample efficiency and zero-shot generalization.

  • Takeaways & Limitations

    Using pretrained LLMs as agent policies may move RL beyond the Tabula-Rasa setting toward more sample-efficient agents.

  • Takeaways & Limitations

    GLAM is not designed for real-world deployment, so the authors do not recommend using it in applicative real-world contexts.

Abstract

from arXiv · show

Recent works successfully leveraged Large Language Models' (LLM) abilities to capture abstract knowledge about world's physics to solve decision-making problems. Yet, the alignment between LLMs' knowledge and the environment can be wrong and limit functional competence due to lack of grounding. In this paper, we study an approach (named GLAM) to achieve this alignment through functional grounding: we consider an agent using an LLM as a policy that is progressively updated as the agent interacts with the environment, leveraging online Reinforcement Learning to improve its performance to solve goals. Using an interactive textual environment designed to study higher-level forms of functional grounding, and a set of spatial and navigation tasks, we study several scientific questions: 1) Can LLMs boost sample efficiency for online learning of various RL tasks? 2) How can it boost different forms of generalization? 3) What is the impact of online learning? We study these questions by functionally grounding several variants (size, architecture) of FLAN-T5.

1 Introduction

LLMs can encode abstract knowledge about physical rules, but their internal knowledge may not align with interactive environments. The paper studies functional grounding through incremental online RL, asking about sample efficiency, generalization, and online interventions.

  • LLMs capture aspects of physical rules, including space, colors, and affordances, yet this knowledge may remain misaligned with environmental dynamics.
  • Functional grounding connects an agent’s internal symbols to environmental dynamics so they can support appropriate action.
  • The paper proposes incrementally updating an LLM policy from goals, observations, and interaction outcomes in textual environments.
  • The experiments investigate sample efficiency, generalization to new objects and tasks, and the impact of online RL versus offline Behavioral Cloning.
  • GLAM uses an LLM as an agent policy in BabyAI-Text and applies online PPO to ground it with environment rewards.

2 Related work

Prior work used language models as planners, pretrained policies, or RL-finetuned generators, but these approaches generally did not directly ground LLM policies through interactive environmental feedback. GLAM addresses this gap in textual, language-conditioned RL.

  • Language-conditioned RL trains instruction-following agents to execute actions in interactive environments that fulfill language instructions.
  • Textual RL environments expose agents to complex dynamics through text, but GLAM’s setting trades large action spaces for systematic experiments with computationally expensive LLMs.
  • Prior robotics and textual systems used LLMs as high-level planners rather than directly as grounded policies for low-level actions.
  • Behavioral Cloning from expert trajectories can underperform a smaller randomly initialized RL agent, supporting the importance of direct environmental interaction for grounding.
  • Unlike RLHF, GLAM uses sparse task-conditioned rewards from an external environment rather than a learned human-preference reward model.

3 GLAM: Grounding LLMs with online RL

GLAM treats an LLM as the policy in a textual, goal-conditioned environment and grounds it through online PPO. The method converts language-model action likelihoods into a policy distribution, adds value estimation, and updates the model from interaction rewards.

  • GLAM uses an LLM as an agent policy, updating it with observations and rewards from an interactive environment toward language-specified goals.
  • 3.1 Problem statement: BabyAI-Text provides textual partial observations, language goals, six discrete text commands, and sparse scalar rewards in procedurally generated navigation tasks.
  • 3.2 LLMs as policies in interactive environments: The policy prompt combines the task description, current textual observation, and possible actions without intensive prompt engineering.
  • 3.2 LLMs as policies in interactive environments: For each action, GLAM computes token-level conditional likelihoods, sums their log probabilities, and applies softmax normalization to sample an action.
  • 3.3 Online RL finetuning: PPO learns both the policy and a value function, with the action policy given by the LLM likelihood and value estimation supplied by an added MLP head.
  • 3.4 Computational considerations: Computing every action’s token likelihood requires repeated LLM forward passes, making online RL computationally expensive for large models.

4 Experiments

The experiments evaluate GLAM with Flan-T5 agents in BabyAI-Text across multi-task learning, sample-efficiency, robustness, and generalization tests. GFlan-T5 learns substantially faster than the comparison agents and shows selective robustness to larger action spaces, distractors, novel symbols, synonyms, and languages.

  • Experimental setup: GFlan-T5 is evaluated against NPAE-Flan-T5, DRRN, and Symbolic-PPO on five spatial and navigation tasks in BabyAI-Text.Tasks include navigation, object manipulation, temporal sequences, and door unlocking, with procedurally generated rooms containing eight distractors.
  • Q1: Sample efficiency: GFlan-T5 reaches 0.8 average success after 250,000 steps and 0.9 after approximately 600,000 steps, while DRRN and NPAE-Flan-T5 remain below 0.2 after 1.5 million steps.Symbolic-PPO almost reaches 0.4 after 1.5 million steps, despite using easier symbolic observations.
  • Q1: Action-space and distractor robustness: GFlan-T5’s success rate decreases by 14% from 4 to 16 distractors, compared with a 38% decrease for Symbolic-PPO.The authors hypothesize that GFlan-T5 quickly focuses on relevant aspects of the environment.
  • Q2/Q3: Generalization: On a new task composition, GFlan-T5 reaches 0.12 success, outperforming Flan-T5 at 0.07 and Random at 0.05, although all agents perform poorly.The tested sequence was not mastered during training, limiting performance for every agent.
  • Q2/Q3: Generalization: For synonym substitution, GFlan-T5 achieves 0.12 versus 0.01 for Flan-T5 but suffers an 87% performance drop from the original setting.The authors attribute the drop to possible overfitting to the actions’ vocabulary; French-language generalization also remains weak at 0.02 success.
  • Q4: Online interventions: GFlan-T5 outperforms all behavioral-cloning baselines on Go To and invented-noun-and-adjective tasks, with performance changing from 0.82 to 0.74 under invented words.The comparison uses online-trained GFlan-T5 and behavioral-cloning agents trained from GFlan-T5 or procedural-bot trajectories.

5 Conclusion

The paper proposes GLAM for functionally grounding LLMs through online RL in interactive textual environments, improving performance, sample efficiency, and zero-shot generalization. The study remains limited to textual environments and is computationally inefficient.

  • GLAM aligns LLM internal symbols with external dynamics through online reinforcement learning in interactive textual environments.The method requires almost no environment-specific modifications to the LLM.
  • GLAM improves task performance compared with zero-shot LLM use, supervised finetuning, and RL finetuning of non-pretrained LLMs.
  • GLAM boosts sample efficiency and zero-shot generalization to new objects and several new tasks.
  • The experiments are limited to a textual environment, and the method is computationally inefficient.
  • The results suggest that using LLMs as agent policies may help create more sample-efficient RL agents than tabula-rasa approaches.
  • The approach is not designed for real-world deployment, so the authors do not recommend using it in applicative contexts.

Appendices

The appendices document the environments, supplementary experiments, qualitative analyses, implementation details, and vocabulary substitutions used throughout the study.

  • Environments: Appendix A presents the BabyAI and BabyAI-Text environments.
  • Additional results: Appendix B reports per-task success rates, averaging procedures, observation-structure effects, pretraining, model size, and additional ablations.
  • Qualitative analysis: Appendix C analyzes GFlan-T5 during mixed-task training by plotting action-distribution evolution for 11 prompts.
  • Complementary tests: Appendix D provides complementary tests for Q2 and Q3 and analyzes grounding of the temporal symbols “then” and “after.”
  • Implementation details: Appendices E–H describe the distributed setup, PPO and Behavioral Cloning hyperparameters, confidence intervals, and vocabulary substitutions.

A.1 BabyAI

BabyAI is a procedurally generated, language-conditioned MiniGrid environment where agents complete varied object-interaction tasks under a step limit. BabyAI-Text preserves the underlying tasks, rewards, and visualization tools while expressing observations textually.

  • BabyAI gives agents a limited number of steps to complete language goals in procedurally generated gridworld environments.
  • The gridworld contains colored boxes, balls, doors, and keys placed in connected 8 × 8 rooms, with some objects acting as distractors.
  • Agents use six primitive actions: turn left, turn right, go forward, toggle, pick up, and drop.
  • BabyAI-Text maps BabyAI’s partial symbolic observations to template-based linguistic descriptions while retaining the inner MiniGrid environment, tasks, rewards, and visualization tools.
  • The textual representation specifies object colors, object types, and relative positions using steps right, left, and forward from the agent.

B.1 Per-task success rate

Per-task analyses show that functional grounding helps GFlan-T5 master several tasks, while OpenDoor remains unresolved. Comparisons also examine observation structure and pretrained-network configurations affecting learning.

  • Per-task performance: GFlan-T5 masters GoTo and PickUp while improving PutNextTo and PickupThen/AfterPickup after 1.5 million training steps.The evaluation uses 1000 test episodes of a mixed setup and four random seeds.
  • Per-task performance: OpenDoor remains without a robust strategy because the agent must find the correct key and discover that toggle opens the door.
  • Textual vs symbolic representation: Symbolic-PPO converges faster than DRRN on Go To Red Ball, suggesting symbolic observations make this trivial task easier.Symbolic-PPO receives symbolic observations, whereas DRRN receives textual observations.
  • Impact of pretraining: The pretraining comparison evaluates GFlan-T5, AFlan-T5, NPAE-Flan-T5, NPE-Flan-T5, and NPA-Flan-T5 with different pretrained-weight and action-probability configurations.
  • Impact of pretraining: GFlan-T5 performs best, while AFlan-T5 surpasses non-pretrained networks after 250000 frames.
  • Impact of pretraining: Action heads are crucial for non-pretrained networks: NPE fails to learn within the allotted steps, whereas NPAE and NPA have similar learning curves.

B.5 Impact of the size of the LLM

The number of LLM parameters strongly affects online RL learning, while GFlan-T5 remains comparatively robust to larger action spaces and more distractors.

  • LLM size: 80 million, 780 million, and 3 billion parameters are compared for GFlan-T5 on Go to <object> over 400,000 training steps.The experiments use one room and eight distractors, with results averaged over two seeds.
  • LLM size: Pretraining prior knowledge appears impactful only when the network is large enough, with small and large models showing different learning properties.The authors relate this size-dependent behavior to emergent abilities.
  • Action-space robustness: GFlan-T5 efficiently handles restricted, canonical, and augmented action spaces, with initial biases especially helpful when many actions are available.The action spaces contain 3, 6, and 9 actions, respectively, with only 3 useful actions in the augmented setting.
  • Action-space robustness: A 0.35 success-rate gap over the second-best agent after 50,000 steps appears in the augmented setting, versus almost no gap in the restricted setting.This supports rapid discarding of useless actions early in finetuning.
  • Distractor robustness: NPAE-Flan-T5 is also unaffected by distractor changes, but the authors attribute this to its inability to learn the task within 400,000 steps.This caveat limits the interpretation of its apparent robustness.

B.7 Robustness to domain-specific vocabulary

GFlan-T5 adapts to a domain-specific vocabulary that reverses the meanings of “turn left” and “turn right,” despite poorer initial performance.

  • Vocabulary reversal: Flipping the meanings of “turn left” and “turn right” initially lowers performance because GFlan-T5 must relearn the rotation mapping.The comparison uses average success rate over training with two seeds.
  • Vocabulary reversal: GFlan-T5 converges at a similar speed in the flipped and non-flipped environments.The result suggests adaptation to domain-specific vocabularies under online grounding.

C Evolution of actions distribution on evaluation prompts

Action-probability trajectories reveal which skills GFlan-T5 acquires quickly and where grounding remains uncertain, especially for spatial relations, temporal order, and unseen task compositions.

  • Evaluation setup: Figure 18 tracks action probabilities across the test prompts to visualize skill acquisition during online RL grounding.The evaluation uses 11 prompts listed in Table 2 and tests the LLM at each update.
  • Navigation prompts: Prompts 0 and 1 test simple navigation, and the optimal behavior is learned in fewer than 100 updates.Navigation actions already have high probability from the beginning, even when the initial bias is wrong and strong.
  • Pick Up prompts: For Pick Up tasks, both the pick up action and navigation actions have high initial probability.This reflects useful pretrained biases before online updates.
  • Spatial grounding: Prompts 6 and 7 expose difficulty grounding that an object one step forward can be picked up or dropped without additional movement.For prompt 7, go forward and pick up or drop retain almost equal probability at the end of training.
  • Temporal constructions: Temporal order is handled better when the specified order matches the required action order than when “after” reverses that order.Prompt 9 favors the wrong direction and shows more hesitation than prompt 8.
  • Open <door> prompts: The Open <door> prompts remain difficult because the agent must infer that a same-color key is needed to open a locked door.Within the training budget, the agent fails to associate the key with the task.
  • Generalization prompt: The unseen composed task Pick up <object A> then/after Pick up <object B> tests whether the agent can drop a carried object before picking up another.The drop action is optimal because it is the only action that completes the instruction in the described state.

D.1 Recapitulating results table

The complementary tests examine generalization to unseen words, multilingual action commands, and temporal components, reporting both robustness and clear scope boundaries.

  • Results overview: The numerical results in Figure 4 are summarized with confidence intervals calculated according to Appendix G.The section also organizes complementary tests for the generalization questions.
  • Unseen vocabulary: A success rate of 0.87 indicates that GFlan-T5 is unaffected by unseen in-vocabulary objects used as task goals.These objects appeared during training as distractors but were excluded from training goals.
  • Unseen vocabulary: GFlan-T5 is likewise unaffected by unseen out-of-vocabulary adjectives, with a reported success rate of 0.87.The adjectives are replaced with predefined synonyms outside the BabyAI-Text vocabulary.
  • Language generalization: Keeping the environment in English while translating actions into French yields a 0.15 success rate, above the 0.02 rate in the fully French environment.The authors state that generalization depends on how many grounded words are modified.
  • Temporal components: GFlan-T5 grounds “then” and “after” better than the original Flan-T5 agent on tasks where action order determines completion.The temporal-symbol tests are summarized in Table 6.

E Distributed experimental setup

The experiments use distributed environment interaction and parallel LLM instances to make online RL finetuning practical. Evaluation generalization tests use repeated seeds and episode-level success-rate confidence intervals, with additional substitution-based test settings.

  • Distributed data collection: 32 BabyAI-Text environments run synchronously in parallel, producing 32 states and requiring 32 actions at every step.The environments run on CPUs in a distributed data-collection setup.
  • Distributed LLM inference: Four parallel LLM instances handle experiment requests through a client-server architecture with a master dispatching calls and gathering results.The RL script acts as the client, while one server hosts each LLM instance.
  • Distributed optimization: PPO updates distribute forward and backward passes across LLM instances, then gather gradients to update each LLM and its value head.Lamorel also supports a custom computational graph connected to the LLM.
  • Hardware allocation: Flan-T5 780M uses two Nvidia A100 80GB GPUs per instance, requiring eight GPUs across four instances; smaller and larger variants use different per-instance allocations.Flan-T5 80M uses one Nvidia V100 32GB per instance, while Flan-T5 3B uses four Nvidia A100 80GB per instance.
  • Evaluation and uncertainty: Generalization results are reported over two seeds and 1000 episodes per test, with confidence intervals derived from seed variability and episode success rates.The setup models agent success with Bernoulli variables and uses normal approximations for the reported intervals.
  • Generalization test construction: The study tests generalization through word substitutions, including out-of-vocabulary nouns and adjectives, selected synonyms, and French translations.Synonym actions are retained only when Flan-T5-Large judges the original and substituted actions synonymous.
Loading 2302.02662v5…