PaperlayerView exact revision on arXiv

Source-linked AI summary

Survey on Large Language Model-Enhanced Reinforcement Learning: Concept, Taxonomy, and Methods

Yuji Cao, Huan Zhao, Yuheng Cheng, Ting Shu, Yue Chen, Guolong Liu, Gaoqi Liang, Junhua Zhao, Jinyue Yan, Yun Li

arXiv:2404.00282v3cs.LGcs.AIcs.CLcs.RO

TL;DR

RL continues to face challenges in reward design, generalization, and multi-task learning, motivating the use of LLMs' pretrained knowledge and capabilities. This survey reviews LLM-enhanced RL, defines its scope, and proposes a four-role taxonomy; it concludes that LLMs can support RL through information processing, reward design, decision-making, and generation, while reward-design methods still require prompt and instruction modification.

  • Problem

    RL faces persistent challenges in reward-function design, generalization, and multi-task learning, while the growing LLM-enhanced RL literature lacks a comprehensive review and unified framework.

  • Method

    The survey defines LLM-enhanced RL and organizes LLM functionality within classical agent-environment interaction into four roles: information processor, reward designer, decision-maker, and generator.

  • Results

    The survey synthesizes methods showing how LLMs provide observational representations, generate rewards, support decisions, and produce task-relevant artifacts across LLM-enhanced RL.

  • Takeaways & Limitations

    LLM-enhanced RL offers a framework for leveraging pretrained knowledge to address RL challenges, but reward-design approaches still call for automated self-evolving frameworks without human intervention.

Abstract

from arXiv · show

With extensive pre-trained knowledge and high-level general capabilities, large language models (LLMs) emerge as a promising avenue to augment reinforcement learning (RL) in aspects such as multi-task learning, sample efficiency, and high-level task planning. In this survey, we provide a comprehensive review of the existing literature in LLM-enhanced RL and summarize its characteristics compared to conventional RL methods, aiming to clarify the research scope and directions for future studies. Utilizing the classical agent-environment interaction paradigm, we propose a structured taxonomy to systematically categorize LLMs' functionalities in RL, including four roles: information processor, reward designer, decision-maker, and generator. For each role, we summarize the methodologies, analyze the specific RL challenges that are mitigated, and provide insights into future directions. Lastly, a comparative analysis of each role, potential applications, prospective opportunities, and challenges of the LLM-enhanced RL are discussed. By proposing this taxonomy, we aim to provide a framework for researchers to effectively leverage LLMs in the RL field, potentially accelerating RL applications in complex applications such as robotics, autonomous driving, and energy systems.

I. INTRODUCTION

LLM-enhanced RL has emerged to address persistent RL challenges, but the field lacks a comprehensive review and unified framework. This survey defines the paradigm and organizes LLM functionality into four roles while reviewing methods and future directions.

  • Motivation: LLMs offer reasoning, natural language understanding, and task-planning capabilities that can address RL challenges including sample inefficiency, reward design, generalization, and language understanding.Proposed uses include context-informed predictions, nuanced rewards, language-based feedback, and instruction translation.
  • Research gap: The rapidly developing LLM-enhanced RL literature lacks a comprehensive review and unified framework for organizing its integration methods.
  • Contributions: The survey defines LLM-enhanced RL as a paradigm integrating LLMs with classical agent-environment interaction to provide multifaceted enhancements.
  • Contributions: Its taxonomy classifies LLMs as information processors, reward designers, decision-makers, and generators within the classical RL paradigm.
  • Contributions: The survey reviews algorithmic characteristics for each role and analyzes applications, opportunities, challenges, and future research directions.

A. Background of Reinforcement Learning

Classical RL trains an agent to maximize cumulative rewards through trial-and-error interaction with an environment. Its practical challenges include generalization, reward design, compounding planning errors, multi-task learning, and multimodal or language-based interaction.

  • 1) Classical Reinforcement Learning: In the classical RL paradigm, an agent observes states, selects actions through policy π, and optimizes cumulative discounted rewards in an MDP.The MDP is formalized by ⟨S, A, T, R, γ⟩.
  • 2) Challenges of Reinforcement Learning: Generalization remains difficult because agents trained in specific or simulated settings may struggle in novel or dynamic environments.Successful transfer requires adapting learned task solutions to underlying principles across situations.
  • 2) Challenges of Reinforcement Learning: Reward functions are difficult to design for sparse and complex tasks, and reward shaping can bias agents toward suboptimal policies or scenario-specific behavior.Designed rewards may require extensive manual trial-and-error and can produce unintended behavior.
  • 2) Challenges of Reinforcement Learning: Model-based RL accumulates prediction errors over longer planning horizons, causing deviations from optimal trajectories in complex, high-dimensional environments.
  • 2) Challenges of Reinforcement Learning: Multi-task RL must manage unequal task difficulty, task interference, parameter sharing, sample efficiency, and knowledge transfer without negative interference.
  • Multimodal RL: Language-conditional RL incorporates natural language into task descriptions or state and action spaces, whereas language-assisted RL uses language to communicate domain knowledge or structure policies.

B. Background of Large Language Models

LLMs are large Transformer-based models whose scale and training data support in-context learning, instruction following, reasoning, and generalization. The paper frames LLM-enhanced RL as using these capabilities to assist RL with multimodal information and environmental adaptation.

  • LLM foundations: LLMs typically contain billions of parameters and are trained on massive text datasets, enabling broad language generation, knowledge representation, reasoning, and generalization capabilities.Examples include GPT-3 and LLaMA.
  • LLM capabilities: Scaling model parameters and training data can produce emergent abilities such as in-context learning, reasoning, and generalization.These abilities are associated with scaling laws.
  • LLM capabilities: In-context learning uses task-related examples or knowledge in prompts without explicit retraining or gradient updates.The prompt context helps models understand situations and execute instructions.
  • LLM capabilities: Instruction following enables LLMs to comprehend natural-language descriptions of new tasks and generalize across previously unseen tasks.The passage attributes this capability to diverse instruction-formatted datasets and instruction tuning.
  • LLM capabilities: CoT, ToT, and GoT prompting structure complex problem solving into sequential or hierarchical reasoning steps, while planning prompts can produce action sequences.These strategies make reasoning pathways more articulated and understandable.
  • LLM-enhanced RL: LLM-enhanced RL uses multimodal processing, generation, reasoning, and other high-level cognitive capabilities of pretrained knowledge-rich models to assist RL.The framework addresses agents operating with multimodal information and supports adaptation through environmental interaction.

B. Framework

The framework embeds LLMs within classical agent-environment interactions, where they process information, redesign rewards, assist decisions, and generate or interpret task-related content. These functions correspond to four roles and target multimodal understanding, generalization, sample efficiency, long-horizon planning, and sparse rewards.

  • Framework: LLMs operate alongside trial-and-error agent-environment interactions by processing state information, redesigning rewards, assisting action selection, and interpreting policies.The framework preserves the classical RL interaction paradigm while adding multiple LLM functions.
  • Taxonomy: The taxonomy identifies four LLM roles in RL: information processor, reward designer, decision-maker, and generator.The roles organize how LLM functionalities integrate into the classical agent-environment paradigm.
  • Framework characteristics: LLMs improve multimodal information understanding by helping agents learn from tasks and environments described through natural language and vision data.This addresses the difficulty of jointly comprehending complex multimodal information and optimizing control policies.
  • Framework characteristics: Pretrained knowledge and sequence modeling support multi-task learning, task-variance accommodation, and knowledge transfer across multiple tasks.The framework presents these capabilities as supporting generalization.
  • Framework characteristics: LLMs can improve sample efficiency by using prior knowledge or generating simulated data, reducing the samples required by exploratory RL.The passage presents sample-efficiency improvement as a characteristic of the enhanced paradigm.
  • Framework characteristics: LLMs can decompose complex tasks into subtasks, helping agents plan over long temporal horizons and handle multi-step tasks.The stated motivation is the credit-assignment difficulty that increases with trajectory length.
  • Framework characteristics: LLMs contribute context- and knowledge-based reward shaping or reward-function design to guide policy learning in sparse-reward environments.The reward-designer role includes implicit reward values and explicit executable reward-function code.
  • Generator: As generators, LLMs can produce abstract world models that guide exploration, while RL gameplay verifies and corrects those models.In Minecraft item crafting, this combination achieved an order of magnitude improvement in sample efficiency over traditional methods.

IV. LLM AS INFORMATION PROCESSOR

As an information processor, an LLM or VLM decouples multimodal representation or language processing from downstream RL policy learning. Frozen models provide reusable representations, while fine-tuning and contrastive learning target invariance and adaptation to environmental variation.

  • Motivation: End-to-end multimodal RL forces agents to learn feature comprehension and control jointly, while reward-only feature learning can narrow utility and impair generalization.This motivates decoupling representation learning from policy learning.
  • Information processor: Pretrained LLMs or VLMs can extract observation representations or translate natural language into formal specifications for downstream RL networks.This information-processor role reduces the burden of learning multimodal features and task interpretation simultaneously.
  • Feature Representation Extractor: Frozen pretrained models directly extract observation embeddings, whereas fine-tuned models use contrastive learning to adapt representations to new environments.The two categories differ according to whether the pretrained model is trained simultaneously with RL.
  • Frozen Pre-trained Model: History Compression via Language Models uses a frozen language Transformer and FrozenHopfield to compress observation history before combining it with a learned current-observation encoding.The method approximates the underlying MDP in partially observed settings using compressed history representations.
  • Frozen Pre-trained Model: Semantic HELM stores human-readable visual-history semantics in a CLIP and language-model embedding database, retrieving top-k embeddings for actor-critic input.With PPO and Transformer XL, the extracted semantics gave a memory-less agent scores comparable to memory-based methods on partially observable environments.
  • Frozen Pre-trained Model: Frozen pretrained representations cannot dynamically adjust to the task and environment, limiting their adaptability despite helping agents cope with partial observability.The limitation is stated directly after the Semantic HELM experiment.
  • Fine-tuning Pre-trained Model: Contrastive learning matches a query more closely to a positive key than to negative keys, and maximizing alignment across observation changes learns invariant representations.InfoNCE implements this objective as a softmax-style classification loss.
  • Fine-tuning Pre-trained Model: Visual prompt-based contrastive learning increased success rates on unseen scenarios by 18-20% and improved generalization under environmental variation.ReCoRe further enforces intervention invariance with auxiliary tasks such as depth prediction and image denoising.

B. Language Translator

LLMs can translate informal natural-language instructions and environmental information into formal, task-specific representations that reduce the burden on RL policy learning. The survey identifies feature extraction, language translation, and multimodal generalization as key directions and challenges.

  • B. Language Translator: LLMs translate diverse natural-language instructions into formal task-specific information, such as feature representations or task-specific languages, for RL agents.
  • B. Language Translator: Language translation can prevent policies from directly processing natural-language instructions, supporting more efficient policy learning.
  • B. Language Translator: LLMs can also convert natural-language environmental information into formal domain-specific language specifying task objectives, rewards, policies, or unified MDP knowledge.
  • B. Language Translator: Separating information processing from control lets LLMs extract representations or handle language-based information before RL policy learning.
  • C. Summarization and Outlook: Multimodal translation remains challenging because modalities may conflict, while existing methods often align modalities without incorporating task information.
  • C. Summarization and Outlook: Future work targets computationally efficient, more generalizable, task-compositional feature extraction and broader, more accurate, multimodal language translation.

V. LLM AS REWARD DESIGNER

LLMs address difficult reward-design problems by using language understanding and pretrained knowledge to provide implicit rewards or generate explicit reward-function code. The survey organizes these approaches as implicit and explicit reward models.

  • V. LLM AS REWARD DESIGNER: Reward design is difficult because expressing desired behavior and constructing dense learning signals often requires expert demonstrations or meticulous manual crafting.
  • V. LLM AS REWARD DESIGNER: LLMs can design or shape reward functions from natural-language instructions and environment information using pretrained common sense, code generation, and in-context learning.
  • A. Implicit Reward Model: Implicit reward models provide auxiliary or overall reward values through direct language prompting or alignment scoring between visual observations and instructions.
  • B. Explicit Reward Model: Explicit reward models generate executable reward-function code, with potential self-refinement through evaluation loops.

2) Alignment Scoring:

Alignment scoring uses vision-language similarity to derive rewards from task descriptions and visual observations, while explicit code generation makes reward logic inspectable and optimizable. The survey highlights self-refinement, alignment quality, transferability, and bias as central concerns.

  • 2) Alignment Scoring:: Alignment-scoring methods compute goal probabilities from cosine similarity between visual-state and language-description embeddings, then threshold the probability into a binary reward.
  • 2) Alignment Scoring:: CLIP-based reward models can specify tasks with a single-sentence text prompt and minimal prompt engineering.
  • 2) Alignment Scoring:: R3M alignment scores and VLM-generated preference labels are used to construct reward signals from visual demonstrations or behavioral preferences.
  • B. Explicit Reward Model: Explicit reward code exposes the calculation logic, allowing humans to evaluate and optimize the generated reward function.
  • B. Explicit Reward Model: 90% of designed tasks were reliably solved versus 50% for the baseline when LLM-generated lower-level reward parameters connected high-level instructions to robot actions.
  • B. Explicit Reward Model: Self-refinement systems use evaluation results and reflection to iteratively select or optimize reward-code candidates.
  • C. Summarization and Outlook: LLM reward designers may transfer model biases into reward functions, causing suboptimal or harmful behaviors and motivating regularization or human intervention.
  • C. Summarization and Outlook: Future work should improve alignment with human intentions, cross-task transfer, specialized-task knowledge, and adaptability beyond manually designed motion templates.

VI. LLM AS DECISION-MAKER

LLMs serve as decision-makers through action-making and action-guiding: they model offline RL trajectories for action generation or constrain action selection with candidates and expert actions. Pretrained representations improve generalization and data efficiency, particularly in sparse-reward and long-horizon tasks.

  • VI. LLM AS DECISION-MAKER: The survey divides LLM decision-making into action-making, which generates actions, and action-guiding, which instructs action selection.
  • A. Action-Making: Action-making treats offline RL as sequence modeling, using return-to-go, states, and actions to predict future actions.
  • B. Action-Guiding: Action-guiding generates reduced action candidates or expert actions to improve sample and exploration efficiency in large action spaces.
  • A. Action-Making: Pretrained LLMs can be fine-tuned for action generation or combined with goals, instructions, trajectories, and task-specific smaller models for adaptation.
  • A. Action-Making: 43.6% improvement in out-of-distribution task completion rates was reported with less training data, such as 500 versus 10K samples, compared with basic DT.
  • A. Action-Making: 50% performance gain was demonstrated in sparse-reward Kitchen and Reacher2d environments, while AntMaze scores were five times higher than non-pretrained counterparts.
  • A. Action-Making: Across offline RL benchmarks, general language models outperformed Decision Transformer and reduced D4RL training time by 3-6x.

B. Action-Guiding

As an action-guider, an LLM proposes action candidates or expert actions to improve exploration and sample efficiency in large-action, language-involved, and human-collaborative environments. This guidance can align policies with human preferences and reduce training iterations, but inherits LLM biases and incurs substantial computational and data costs.

  • Action-guiding: LLMs guide action selection by generating reasonable action candidates or expert actions, improving sample and exploration efficiency in enormous action spaces.The role addresses action-space and natural-language challenges by instructing the RL agent’s action selection.
  • Action-guiding: In text-based games, LLMs reduce large action spaces to task-relevant candidate sets that are likely to yield high rewards across multiple tasks.This targets exploration inefficiency when only a tiny fraction of actions are accessible.
  • Action-guiding: LLMs produce expert actions from human behavior and general knowledge to regularize RL agents when expert demonstrations are unavailable.This approach addresses inefficient learning and undesirable equilibria in complex human-AI collaboration tasks.
  • Action-guiding: Training iterations decreased by a factor of 1 to 9 when an LLM-based teacher’s policy difference was included in the RL learning loss.The experiments were conducted on simulation platforms, while instructRL produced policies aligned with human preferences.
  • Action-making: Direct action generation requires huge computational resources and fine-tuning data, motivating cost-effective adaptations such as LoRA.The survey identifies efficient exploitation of LLMs for direct decision-making as a longer-term goal.
  • Action-guiding: Action-guiding agents inherit the instructor LLM’s biases and limitations, while the LLM cannot inherently interrogate or intervene in the environment.Future work emphasizes filtering or correcting bias and enabling active interaction to improve actuality and reasoning.

VII. LLM AS GENERATOR

As generators, LLMs support model-based RL by simulating environmental dynamics and explainable RL by interpreting policies. They can generate trajectories or latent representations for learning, while policy interpretations expose agent behavior, but explanation quality and environmental grounding remain important limitations.

  • World Model Simulator: World model simulators generate trajectories autoregressively or learn latent representations of future worlds to support policy learning in real environments.Real-world knowledge and data construct the simulator, which models dynamics and generates trajectories.
  • World Model Simulator: LLMs serve as world model simulators that use real-world data and knowledge to model dynamics, generate simulated worlds, and assist policy learning.The generator role also includes policy interpretation for potentially explainable RL.
  • World Model Simulator: IRIS outperformed humans in 10 out of 26 Atari 100k games using the equivalent of two hours of gameplay.The agent used a discrete autoencoder and an autoregressive Transformer to learn an Atari world model.
  • World Model Simulator: A video-based intrinsic bonus improved vision-based RL on manipulation and locomotion tasks by transferring pre-trained representations from unseen domains.The framework learned world-model dynamics from action-free video representations and encouraged diverse behaviors.
  • Policy Interpreter: LLMs can generate readable policy interpretations from state-action descriptions or trajectory histories for human understanding.Policy interpretation uses prompts containing states, actions, or trajectory information.
  • Policy Interpreter: S2E jointly embeds state-action pairs and concept-based explanations, using explanations for reward shaping during training and insights at deployment.Another approach distills policies into decision trees before prompting an LLM to produce natural-language explanations.

C. Summarization and Outlook

The survey compares four LLM roles in RL and examines their applications, opportunities, and challenges. In the generator role, LLMs simulate environments and explain policies, while grounding, computational efficiency, reward fidelity, bias, and explanation accuracy constrain deployment.

  • Challenges: World model simulators face grounding and functional-competence problems when abstract knowledge does not match specific environments, while policy interpreters depend on understanding features and implicit policy logic.The survey identifies both issues as directions requiring further work.
  • Comparative Analysis: The survey’s comparative analysis organizes LLM-enhanced RL around information processing, reward design, decision-making, and generation.The comparison is intended to clarify the strengths and limitations of each approach.
  • Comparative Analysis: Information processors translate complex multimodal inputs into formats usable by RL agents, but may be computationally inefficient or lack specialized domain knowledge.Their main benefit is improved understanding and interaction with complex environments.
  • Comparative Analysis: Reward designers provide intuitive and flexible rewards that can better align RL objectives with human intentions, but may miss task-specific nuances and long-term goals.This limitation is especially relevant in highly specialized domains.
  • Comparative Analysis: Decision-makers directly generate or guide actions to improve sample efficiency and exploration, but computational overhead and inherited training-data bias can hinder real-time use.The role is particularly suited to tasks requiring complex reasoning or long-term planning.
  • Comparative Analysis: Generators simulate complex environments for model-based RL and explain policies, but must align simulations with real-world dynamics and maintain explanation relevance and accuracy.These capabilities target sample efficiency and RL transparency.

B. Applications of LLM-Enhanced RL

LLM-enhanced RL is positioned for applications requiring multimodal understanding, multi-objective reasoning, and generation, including robotics, autonomous driving, energy management, and healthcare recommendation. The survey also identifies under-explored RL settings and enhancements to LLM knowledge, modules, tools, and collaboration.

  • Applications: Robotics applications may use LLM-enhanced RL to improve human-robot interaction, understand human needs and behavior, and strengthen decision-making and planning.
  • Applications: Autonomous driving may use LLMs to process multimodal sensor information and language instructions or design rewards combining safety, efficiency, and passenger comfort.
  • Applications: Energy-management applications may use reward design for economy, safety, and low-carbon objectives while improving sample efficiency in systems with renewable-resource uncertainty.
  • Applications: Healthcare recommendation may use LLM domain knowledge to analyze patient data and medical histories, accelerating RL learning and supporting more accurate recommendations.
  • Opportunities: Specialized RL directions remain under-explored, including multi-agent, safe, transfer, explainable, multi-task, in-context, and human-centric RL.For multi-task RL, the four LLM roles support entity processing, instruction alignment, action generation, and task-specific trajectories or representations.
  • Opportunities: Future LLM enhancements include retrieval-augmented generation, continual learning, planning, memory and action modules, external tools, and collaboration among multiple LLM agents.These directions target domain knowledge, retained learning, reasoning, real-time processing, and role-specialized cooperation.

D. Challenges of LLM-Enhanced RL

LLM-enhanced RL is constrained by model reliability, adaptation to unfamiliar environments, computational cost, and practical deployment concerns. The survey organizes the paradigm around four LLM roles that address information extraction, reward design, decision-making, and generation, while identifying directions for improving each role.

  • Challenges: Underlying LLM biases and hallucinations can compromise multimodal interpretation and RL-agent reliability.The survey calls for robust evaluation frameworks to assess these effects systematically.
  • Challenges: LLMs may struggle with novel RL environments that are poorly represented in their training data.Proposed responses include retrieval-augmented generation and methods that ground or expand task-related knowledge.
  • Challenges: LLM inference adds computational overhead and latency that can slow RL learning, especially during online interaction.Prompt pruning and other cost-reduction methods are identified as potential remedies.
  • Challenges: Practical deployment raises ethical, legal, privacy, intellectual-property, and accountability concerns.The survey points to responsible-AI frameworks, including differential privacy for data protection.
  • Taxonomy: The taxonomy assigns LLMs four RL roles: information processor, reward designer, decision-maker, and generator.These roles respectively support representation extraction, reward construction, action guidance, and world-model or policy interpretation functions.
  • Future directions: Future work targets stronger multimodal information extraction, automated reward self-evolution, cheaper online decision-making, and human-guided world models and policy interpreters.These directions correspond to the survey’s role-specific recommendations.
Loading 2404.00282v3…