PaperlayerView exact revision on arXiv

Source-linked AI summary

Survey on Large Language Model-Enhanced Reinforcement Learning: Concept, Taxonomy, and Methods

Yuji Cao, Huan Zhao, Yuheng Cheng, Ting Shu, Yue Chen, Guolong Liu, Gaoqi Liang, Junhua Zhao, Jinyue Yan, Yun Li

arXiv:2404.00282v3cs.LGcs.AIcs.CLcs.RO

TL;DR

Designing high-performing RL rewards is difficult, especially when desired behaviors are hard to specify. This survey reviews LLM-enhanced RL through a four-role taxonomy and identifies the roles of information processor, reward designer, decision-maker, and generator.

  • Problem

    High-performing reward functions are difficult to design because desired behaviors can be hard to specify and dense learning signals often require expert input or meticulous manual crafting.

    Source: §V · p. 8

  • Method

    The survey reviews LLM-enhanced RL using the classical agent-environment paradigm and organizes LLM functionalities into a structured taxonomy.

    Source: §IX · p. 17

  • Results

    The taxonomy identifies four LLM roles in RL: information processor, reward designer, decision-maker, and generator.

    Source: §IX · p. 17

  • Takeaways & Limitations

    The taxonomy provides researchers a framework for leveraging LLMs in RL across these roles.

    Source: §IX · p. 17

    Reward-design methods still require repeated prompt and instruction modifications, motivating automated self-evolving frameworks without human intervention.

    Source: §IX · p. 18

Abstract

from arXiv · show

With extensive pre-trained knowledge and high-level general capabilities, large language models (LLMs) emerge as a promising avenue to augment reinforcement learning (RL) in aspects such as multi-task learning, sample efficiency, and high-level task planning. In this survey, we provide a comprehensive review of the existing literature in LLM-enhanced RL and summarize its characteristics compared to conventional RL methods, aiming to clarify the research scope and directions for future studies. Utilizing the classical agent-environment interaction paradigm, we propose a structured taxonomy to systematically categorize LLMs' functionalities in RL, including four roles: information processor, reward designer, decision-maker, and generator. For each role, we summarize the methodologies, analyze the specific RL challenges that are mitigated, and provide insights into future directions. Lastly, a comparative analysis of each role, potential applications, prospective opportunities, and challenges of the LLM-enhanced RL are discussed. By proposing this taxonomy, we aim to provide a framework for researchers to effectively leverage LLMs in the RL field, potentially accelerating RL applications in complex applications such as robotics, autonomous driving, and energy systems.

I. INTRODUCTION … II. BACKGROUND

LLM capabilities offer ways to address persistent RL challenges, while this survey defines LLM-enhanced RL and organizes its methods, contributions, and research directions in a unified taxonomy.

I. INTRODUCTION

  • Deep RL expanded RL’s reach, but language- and vision-involved tasks add challenges in jointly learning features and policies and coping with complex state-action spaces.The introduction traces RL’s progression from traditional methods to deep RL, citing DQN’s Atari performance and applications in games and energy management,, [6],.Source: §I · p. 1
  • The introduction identifies sample inefficiency, reward design, generalization, and natural-language understanding as unresolved challenges for RL in language- and vision-related tasks.These include extensive interaction needs, the difficulty of aligning rewards with high-level objectives, poor transfer to intervention-affected states, and difficulty handling linguistic nuance.Source: §I · pp. 1–2
  • LLMs’ language understanding, reasoning, planning, and pretrained knowledge may help address RL’s sample-efficiency, reward-design, generalization, and language-understanding challenges.Prior work used LLMs to suggest context-informed predictions, construct reward functions, provide language-based feedback, and translate instructions into task-specific language.Source: §I · p. 2

A. Contributions

  • The survey addresses the lack of a comprehensive review and unified framework by defining LLM-enhanced RL and proposing a taxonomy of LLM functionalities.It presents a framework for integrating LLMs into classical agent-environment interaction and categorizes their roles as information processors, reward designers, decision-makers, and generators.Source: §I · p. 2, A. Contributions · p. 2
  • The survey reviews algorithmic characteristics of each LLM role and uses them to analyze future applications, opportunities, challenges, and a potential research roadmap.Source: A. Contributions · p. 2

B. Text Organization

  • The paper’s remaining sections move from RL and LLM foundations to the paradigm framework, four role-based analyses, applications and challenges, and a conclusion.Source: B. Text Organization · p. 2

II. BACKGROUND

  • The background section covers classical RL and its challenges, multimodal language and vision in RL, and LLM capabilities relevant to RL.Source: §II · p. 2

A. Background of Reinforcement Learning … 3) Multimodal Reinforcement Learning:

The background defines classical RL as policy optimization in an MDP, outlines persistent challenges, and describes how visual and natural-language data enter RL problems. It distinguishes language-conditional from language-assisted RL and notes the additional demands of multimodal learning.

1) Classical Reinforcement Learning:

  • Classical RL optimizes a policy through agent-environment interaction to maximize cumulative discounted rewards, formalized as an MDP over states, actions, transitions, rewards, and a discount factor.The agent selects actions from observed states and learns through trial and error.Source: 1) Classical Reinforcement Learning · p. 3, Figure 1 · p. 3

2) Challenges of Reinforcement Learning:

  • RL generalization remains difficult because policies trained in specific or simulated settings may fail in novel or dynamic environments.Real-world environments are rarely static or perfectly predictable, so learned behavior must transfer to unseen conditions.Source: 2) Challenges of Reinforcement Learning · p. 3
  • Reward design is difficult in sparse or complex tasks: shaping can bias policies, while high-performing rewards often require extensive trial and error,, [53],,.Designed rewards may be sub-optimal or produce unintended behavior.Source: 2) Challenges of Reinforcement Learning · p. 3
  • In model-based RL, prediction errors accumulate over longer planning horizons, especially in high-dimensional environments [55],.The passage suggests LLM predictive capabilities and sequential understanding could help mitigate these errors.Source: 2) Challenges of Reinforcement Learning · p. 3
  • Multi-task RL must address negative transfer, task interference, parameter-sharing tradeoffs, limited sample efficiency, and knowledge transfer without interference, [58],,,.Differences in task difficulty can also let simpler tasks overshadow learning on more complex tasks.Source: 2) Challenges of Reinforcement Learning · p. 3

3) Multimodal Reinforcement Learning:

  • RL increasingly incorporates visual observations and natural-language tasks, but multimodal agents must jointly process complex inputs and optimize control.Visual observations occur in applications such as robotics and video-game control; natural language understanding [63] and visual-based reward design remain issues.Source: 3) Multimodal Reinforcement Learning · pp. 3–4
  • Language-conditional RL makes language part of the problem, either as task instructions or within the observation or action space [17].Examples include instruction following, text games, dialogue systems, and question answering.Source: 3) Multimodal Reinforcement Learning · p. 3
  • Language-assisted RL uses language to support learning without making it part of the problem formulation, by conveying domain knowledge or structuring policies.Wikis and manuals can provide task-related knowledge, while verbal guidance such as “avoid hitting the wall” can encourage generalizable abstractions.Source: 3) Multimodal Reinforcement Learning · pp. 3–4

B. Background of Large Language Models · III. LARGE LANGUAGE MODEL-ENHANCED REINFORCEMENT LEARNING · A. Definition

LLM-enhanced RL uses pre-trained LLMs’ multimodal processing, generation, reasoning, and other cognitive capabilities to assist RL. The survey frames this approach against conventional RL’s challenges and explains the capabilities that make LLMs useful for it.

B. Background of Large Language Models

  • LLMs are typically Transformer-based models with billions of parameters trained on massive, terabyte-scale text data, such as GPT-3 and LLaMA.Their parameters and internet-scale training data support language generation, knowledge representation, logical reasoning, and generalization to novel tasks.Source: B. Background of Large Language Models · p. 4
  • Scaling Laws describe how increasing model size and training data tends to produce emergent abilities absent in smaller models, including in-context learning, reasoning, and generalization,,.Source: B. Background of Large Language Models · p. 4
  • In-context learning lets LLMs perform tasks without retraining or gradient updates by using prompt examples or related task information as context.Source: B. Background of Large Language Models · p. 4
  • Instruction tuning on task-specific natural-language datasets helps LLMs follow instructions for unseen tasks and generalize across tasks without explicit examples [71],, [73].Source: B. Background of Large Language Models · p. 4

A. Definition

  • Conventional RL struggles to interpret complex multimodal data while optimizing control in changing environments, and also faces sample inefficiency, reward-design difficulty, and generalization challenges.The cited examples include robots following natural-language navigation instructions and visual games with natural-language tasks,,.Source: A. Definition · p. 4
  • LLM-enhanced RL uses pre-trained, knowledge-inherent LLMs’ multimodal processing, generation, reasoning, and other high-level cognitive capabilities to assist the RL paradigm.The authors introduce this concept because existing studies vary and lack a standard concept specifying their systematic methodology.Source: A. Definition · p. 4
  • Unlike traditional model-based RL, LLM-enhanced RL starts with pre-trained reasoning and planning capabilities and can transfer knowledge across domains to adapt to unseen environments.Its knowledge-rich pre-training also addresses a limitation of pre-trained models: inability to interact with environments to expand and ground their knowledge.Source: A. Definition · p. 4
  • Environmental interaction generates task-specific data, grounds LLMs in particular domains through in-context learning, and supports adaptation to dynamic changes through continuous learning.Source: A. Definition · p. 5

B. Framework · C. Characteristics · D. Taxonomy

The framework places LLMs within the classical agent–environment interaction loop to process information, redesign rewards, assist action selection, and interpret policies. The survey organizes these functions into four roles and highlights their potential to support multimodal understanding, generalization, sample efficiency, long-horizon planning, and reward design.

B. Framework

  • The framework maps LLM functions in the agent–environment loop to four roles: information processor, reward designer, generator, and decision-maker.During trial-and-error learning, LLMs process state information, redesign rewards, assist action selection, and interpret policies.Source: B. Framework · p. 5

C. Characteristics

  • LLMs support multimodal understanding and multi-task generalization by processing language and vision data and transferring pretrained knowledge across tasks.Their sequence-modeling capacity can accommodate task variation and assist agents handling multiple tasks.Source: C. Characteristics · p. 5
  • LLMs can improve sample efficiency through simulated data or prior knowledge, and decompose complex tasks into subtasks for longer-horizon planning.The survey notes this planning role for complex, multi-step tasks such as Minecraft.Source: C. Characteristics · p. 5
  • LLMs use contextual understanding and domain knowledge to shape rewards or design reward functions, helping guide learning in sparse-reward environments.Reward generation can address cases where high-performance reward functions are difficult to define.Source: C. Characteristics · p. 5

D. Taxonomy

  • As information processors, LLMs extract useful features or translate language-based observations and instructions into formal task languages; robot instruction following illustrates this role.These transformations can reduce the agent’s burden of understanding multimodal information and simplify learning from varied natural-language instructions.Source: D. Taxonomy · pp. 5–6
  • As reward designers, LLMs provide implicit reward values or generate explicit executable reward-function code; robotic dexterous-manipulation reward design is one example.The implicit model may be trained or prompted, while the explicit model specifies the calculation of reward scalars from environment specifications and language-based goals.Source: D. Taxonomy · p. 6
  • As decision-makers, LLMs generate reward-conditioned actions for offline RL or guide action selection with candidate sets and expert actions to improve exploration and sample efficiency.Expert actions can regularize policy learning so their prior knowledge is distilled into the RL agent.Source: D. Taxonomy · p. 6
  • As generators, LLMs can produce trajectories for model-based RL and policy explanations; Minecraft world-model guidance achieved an order-of-magnitude sample-efficiency improvement over traditional methods.The agent verifies and corrects hypothesized subgoal sequences through gameplay, combining LLM knowledge with grounded experience.Source: D. Taxonomy · p. 6

IV. LLM AS INFORMATION PROCESSOR · A. Feature Representation Extractor · 1) Frozen Pre-trained Model:

Pre-trained language and vision-language models can separate feature extraction from policy learning, supplying representations or task specifications that support multi-task learning and generalization. Frozen models can compress history for partially observed RL, though their representations cannot adapt dynamically to each task and environment.

IV. LLM AS INFORMATION PROCESSOR

  • End-to-end RL must learn to understand language or visual inputs while controlling, and reward-only feature learning can narrow representations and hamper generalization.Separately trained encoders have consequently gained popularity,, [86].Source: §IV · p. 6
  • Pre-trained LLMs or VLMs can extract observation representations or translate natural language into formal specifications, enabling multi-task execution, greater sample efficiency, and zero-shot generalization across diverse sparse-reward environments.Their representation ability and prior knowledge support this decoupled information-processing role.Source: §IV · p. 6

A. Feature Representation Extractor

  • Pre-trained models can provide scaffold embeddings for downstream networks; they may be frozen for observation extraction or fine-tuned with contrastive loss for adaptation to new environments.The fine-tuned representation is described as invariant to appearance variation and is fed to an actor-critic network.Source: A. Feature Representation Extractor · p. 6, 1) Frozen Pre-trained Model · p. 7

1) Frozen Pre-trained Model:

  • HELM uses a frozen language Transformer and FrozenHopfield to compress observation history, then combines it with a learned CNN encoding of the current observation to address partial observability.The compressed history approximates the underlying MDP and supports policy optimization.Source: 1) Frozen Pre-trained Model · p. 6
  • Semantic HELM builds a human-readable memory from prompt-augmented CLIP and language-model tokens, retrieving top-k embeddings for actor-critic input to support policy optimization in partially observable environments.It associates visual inputs with language tokens to summarize past visual observations.Source: 1) Frozen Pre-trained Model · pp. 6–7

2) Fine-tuning Pre-trained Model: … C. Summarization and Outlook

The section describes how fine-tuned representations and language translation can help RL agents handle changing observations and natural-language information. It also identifies multimodal information processing and improved feature extraction and translation as areas for further work.

2) Fine-tuning Pre-trained Model:

  • Invariant representations keep policy/value inputs stable under appearance shifts; contrastive learning aligns transformed views against negatives using InfoNCE [91].This state abstraction is intended to support robust performance and generalization in unseen environments.Source: 2) Fine-tuning Pre-trained Model · p. 7
  • Visual-prompt contrastive training with a pre-trained VLM on expert demonstrations improves unseen-scenario success by 18-20%; ReCoRe adds depth-prediction or denoising regularization.The demonstrations provide domain factors such as camera settings and stride length; the auxiliary task explicitly enforces invariance to environment changes.Source: 2) Fine-tuning Pre-trained Model · p. 7

B. Language Translator

1) Instruction Information Translation:

  • For instruction-following, an LLM can translate natural language into a unique task language or game information and metadata for generating training games with Inform7 [94].The inside-out scheme prevents direct policy exposure to natural-language instructions and supports efficient policy learning.Source: 1) Instruction Information Translation · p. 8

2) Environment Information Translation:

  • For environment information, LLMs can ground language in task components,,, or translate it into RLang, which expresses MDP elements and solutions.Training on RLang lets agents use grounded information rather than learning tabula rasa.Source: 2) Environment Information Translation · p. 8

C. Summarization and Outlook

  • Multimodal RL must address modality contradictions and fusion; weighting, CLIP-based virtual modalities, and attention-based contrastive learning are proposed approaches –[103].Existing multimodal methods often focus on modality alignment without considering task information.Source: C. Summarization and Outlook · p. 8
  • Future feature-extractor work should improve computational efficiency and representation generalization, then exploit task compositionality and adapt extraction to diverse control tasks.The paper frames efficiency and generalization as short-term goals, and compositionality and adaptive extraction as long-term goals.Source: C. Summarization and Outlook · p. 8
  • Future language-translator work should expand task coverage and improve translation efficiency and accuracy, then develop multimodal translation integrated with RL algorithms.These are described respectively as short-term and long-term objectives toward a more general translator that helps agent learning.Source: C. Summarization and Outlook · p. 8

V. LLM AS REWARD DESIGNER · A. Implicit Reward Model · 1) Direct Prompting:

LLMs can ease difficult reward design by using language and environment information to provide implicit rewards or generate explicit reward code. The section focuses on implicit rewards, including direct prompting approaches that supply task feedback or auxiliary learning signals.

V. LLM AS REWARD DESIGNER

  • Reward functions are difficult to design because expressing desired behavior and producing dense learning signals often require expert demonstrations, manual subgoal decomposition, or auxiliary objectives.These approaches also require expert input and meticulous manual crafting.Source: §V · p. 8
  • LLMs’ pretrained knowledge, code generation, and in-context learning may help design or shape rewards from natural-language instructions and environment information.The survey distinguishes implicit rewards from executable reward-function code.Source: §V · p. 8

A. Implicit Reward Model

  • An implicit reward model uses task objectives and observations to provide auxiliary or overall reward values, either through direct prompting or language–visual alignment scoring.Fig. 4(i) illustrates these two approaches.Source: A. Implicit Reward Model · p. 8, Figure 4 · p. 9

1) Direct Prompting:

  • Reference uses examples of desirable behavior and preference descriptions to prompt an LLM as a proxy reward function, while Read and Reward extracts manual information for task evaluation.Read and Reward combines question-and-answer extraction for retrieval and summarization with a reasoning module; experiments report improved performance and training speed for RL algorithms using the framework.Source: 1) Direct Prompting · p. 9
  • Carta et al. use question generation and QA to extract auxiliary objectives from a language goal, rewarding trajectories that reconstruct partial information about that goal.The intrinsic reward encourages the agent to produce trajectories useful for reconstructing the global goal.Source: 1) Direct Prompting · p. 9
  • Lafite-RL provides interactive rewards based on an LLM’s real-time understanding of agent behavior, using separate prompts for scenario understanding and evaluation criteria.The framework is described as accelerating RL while reducing human effort during agent–environment interaction.Source: 1) Direct Prompting · p. 9
  • LAMP [110] augments masked world-model pretraining with language-modulated rewards assigned by an R3M score predictor.Its algorithm uses DistilBERT for language preprocessing, prompt and synonym buffers, and randomized scenes with ShapeNet objects from Ego4D and RLBench data.Source: 1) Direct Prompting · p. 9

2) Alignment Scoring: · B. Explicit Reward Model · C. Summarization and Outlook

LLMs and vision-language models support reward design through alignment scores, learned preferences, and executable reward code. The survey also identifies bias risks and future needs for better alignment, generalization, and task-specific reward-code generation.

2) Alignment Scoring:

  • CLIP provides zero-shot rewards from language-goal and visual-state similarity, requiring only a single-sentence task prompt with minimal prompt engineering.The similarity-derived goal probability is thresholded to produce a binary reward.Source: 2) Alignment Scoring · p. 9
  • LAMP [110] uses frozen VLM alignment scores as noisy shaped exploration rewards to warm-start sample-efficient learning, while similarity-based expert-demonstration rewards mitigate mis-generalization.LAMP combines the generated rewards with standard novelty-seeking exploration rewards for language-conditioned policies.Source: 2) Alignment Scoring · p. 9
  • Preference labels generated by a VLM led to better performance than using the VLM as a raw reward-score function [116].The evaluation used vision-based manipulation tasks.Source: 2) Alignment Scoring · p. 10

B. Explicit Reward Model

  • 90% of designed tasks were tackled using LLM-defined low-level reward parameters, compared with 50% for the baseline [117].The parameters are derived from high-level instructions to bridge language goals and low-level robot actions.Source: B. Explicit Reward Model · p. 10
  • Eureka iteratively selects and reflects on reward-code candidates, achieving human-level reward design and solving dexterous manipulation tasks previously infeasible with manual engineering.Its candidates are trained and scored in the environment before the best code is selected for reflection and refinement.Source: B. Explicit Reward Model · p. 10
  • Text2Reward generates dense executable reward programs and refines them using human feedback when policies fail to reach goals amid RL sensitivity and language ambiguity.The policy is executed in the environment so people can assess behavior and guide reward refinement.Source: B. Explicit Reward Model · p. 10

C. Summarization and Outlook

  • LLM reward biases can cause overoptimization or harmful behavior; proposed mitigations include reward regularization, human intervention, and ensembles of reward functions [121] [124] [125].Task-specific biases remain understudied, and the cited mitigation approaches target different risks.Source: C. Summarization and Outlook · p. 10
  • Future work on implicit rewards should reduce instruction ambiguity and improve alignment with human intentions, then study transfer across tasks and complex visual environments.The survey highlights generalization and transferability as longer-term directions.Source: C. Summarization and Outlook · pp. 10–11

VI. LLM AS DECISION-MAKER · A. Action-Making

The section frames LLMs as decision-makers that either make actions or guide action selection, and reviews action-making methods that use pretrained language models in offline RL. It highlights reported gains in generalization, sample efficiency, sparse-reward learning, and complex-task performance.

VI. LLM AS DECISION-MAKER

  • The taxonomy distinguishes action-making, which predicts actions from trajectories, from action-guiding, which proposes reduced action sets or expert actions to regularize RL.The survey motivates LLM decision-making by their performance in language understanding, instruction following, vision-language navigation, planning, and sequential reasoning, [126],,.Source: §VI · p. 11, Figure 5 · p. 11

A. Action-Making

  • Action-making approaches either fine-tune a pretrained LLM to generate actions or condition it on goals or instructions and trajectories; a smaller task-specific model can support rapid adaptation.One scaffold method adds goals alongside observations as LLM inputs.Source: A. Action-Making · p. 11
  • Pre-trained LLMs improved novel-task completion by 43.6% using 500 vs 10K training data and yielded a 50% performance gain in sparse-reward Kitchen and Reacher2d [130].These results are reported against basic Decision Transformer and in sparse-reward environments, respectively.Source: A. Action-Making · p. 11
  • In AntMaze, pre-trained representations achieved five times higher scores than nonpretrained counterparts.The passage attributes the advantage on this long-horizon navigation task to future information encoded in pretrained representations.Source: A. Action-Making · p. 11
  • Fine-tuned general language models outperformed Decision Transformer and reduced D4RL training time by 3–6x on control and game tasks.The study investigated transferability of general language models to specific offline RL tasks.Source: A. Action-Making · p. 11
  • A policy trained with textual captions describing next subgoals consistently outperformed its caption-free baseline [134].The approach interleaves generated captions and actions to unify language reasoning with action in one policy.Source: A. Action-Making · p. 11
  • The method fine-tuned an LLM using trajectory data from the D4RL dataset and achieved state-of-the-art performance on sparse-reward tasks with limited data samples.Source: A. Action-Making · p. 12
  • Co-fine-tuning vision-language models on robotic trajectories and Internet-scale vision-language data improved generalization and supported complex multistage semantic reasoning.The framework represented actions as natural-language tokens, and chain-of-thought reasoning helped agents solve complex tasks.Source: A. Action-Making · p. 12

B. Action-Guiding … C. Summarization and Outlook

The section describes LLM action-guiding through candidate actions and expert actions, which can improve exploration or sample efficiency. Its outlook highlights safety concerns and challenges involving cost, bias, and real-world interaction.

B. Action-Guiding

  • LLMs guide RL action selection by generating candidate actions or expert actions, addressing exploration and sample-efficiency challenges.The section also summarizes action-making as predicting future actions with pretrained models, action-guiding as proposing candidates, and expert actions as supplying demonstrations.Source: B. Action-Guiding · p. 12, C. Summarization and Outlook · p. 12

1) Action Candidates:

  • More than 20% higher scores were achieved across 28 Jericho text games using GPT-2-generated candidates and a separate network to calculate their Q-values [136, 137].The candidates excluded non-useful actions and sped exploration.Source: 1) Action Candidates · p. 12
  • SayCan selects the candidate with the largest critic value after an embodied LLM creates a step-by-step plan and task-specific action candidates.In an office kitchen, it completed long-horizon, abstract natural-language instructions from 101 real-world robotic tasks.Source: 1) Action Candidates · p. 12

2) Expert Actions:

  • InstructRL uses an LLM-derived prior policy to align RL policies with human preferences, while Zhou et al. [139] reduced training iterations by a factor of 1 to 9.InstructRL adds an auxiliary action-probability term conditioned on trajectory and instructions; Zhou et al. include the student–teacher policy difference in the learning loss.Source: 2) Expert Actions · p. 12
  • Over 85% success rates were reported for long-horizon robotics tasks using high-level language plans to guide a single RL policy across states.The approach breaks tasks into stages and outperformed classical, language-based, and end-to-end approaches across different benchmarks.Source: 2) Expert Actions · p. 12

C. Summarization and Outlook

  • Safety remains a concern for costly or risky decision-making tasks; proposed approaches include dynamically relabeling rewards and testing, human oversight, or memory-based safety mechanisms [145].Offline RL methods have also used pessimistic estimations and stationary distribution correction [143] with fixed constraint thresholds.Source: C. Summarization and Outlook · pp. 12–13
  • Future work should reduce action-making’s computational and fine-tuning costs, while addressing action-guiding bias and enabling LLMs to interrogate and intervene in real environments.Suggested directions include LoRA and hybrid models; filtering or correcting biased information; and using real-world interactions to improve the LLM’s actuality and reasoning.Source: C. Summarization and Outlook · p. 13

VII. LLM AS GENERATOR … B. Policy Interpreter

The survey groups LLMs as generators into world model simulators that support imagined rollouts and policy learning, and policy interpreters that explain agent behavior. It reviews trajectory rollout and dynamics-representation methods, then describes policy-level explanation approaches and the limited scope of current LLM use in explainable RL.

VII. LLM AS GENERATOR

  • LLMs can simulate environmental dynamics by predicting states and rewards, supporting imagined rollouts for risky real-world RL and improving sample efficiency in model-based learning.They can also interpret policies to help end-users understand decisions and inform reward design.Source: §VII · p. 13

A. World Model Simulator

  • A world model simulator uses real-world data and knowledge to learn dynamics representations, generate trajectories, and assist policy learning; its LLM roles include trajectory rollout and representation learning.The trajectory rolloutor autoregressively generates trajectories for agent learning and planning; the dynamics representation learner predicts latent world representations.Source: A. World Model Simulator · p. 13, Figure 6 · p. 13

1) Trajectory Rolloutor:

  • IRIS exceeds human performance in 10 of 26 Atari games using two hours of gameplay; TWM trains a state-of-the-art Atari 100k policy, while TransDreamer outperforms Dreamer with long-range memory.IRIS uses a discrete autoencoder and autoregressive Transformer; TWM produces meaningful generated experiences, and TransDreamer applies Transformer dynamics predictions to 2D and 3D visual RL.Source: 1) Trajectory Rolloutor · p. 13
  • A framework trained on action-free video representations adds a video-based intrinsic bonus to encourage diverse exploration and improves vision-based manipulation and locomotion through transfer from unseen domains.The bonus guides exploration while the pre-trained representations are transferred from unseen domains.Source: 1) Trajectory Rolloutor · p. 14

2) Dynamics Representation Learner:

  • A visual model decouples representation and dynamics learning to achieve state-of-the-art visual robotics results; Dynalang predicts future text and image representations, outperforming methods given only task descriptions on grounded-language tasks.Dynalang trains a multimodal world model, imagines rollouts, and trains its policy to maximize imagined rewards.Source: 2) Dynamics Representation Learner · p. 14
  • LanGWM learns language-grounded visual features by masking bounding boxes and predicting them from language descriptions, yielding state-of-the-art results on out-of-distribution tests.The approach uses language to improve generalization of visual features in world-model learning.Source: 2) Dynamics Representation Learner · p. 14

B. Policy Interpreter

  • Explainable RL covers feature importance, learning process and MDP, and policy-level categories, but current LLM use is limited to policy interpretation and the area needs more attention.The survey identifies policy interpretation as the current extent of LLM use in explainable RL.Source: B. Policy Interpreter · p. 14
  • LLMs generate readable explanations from state-action or trajectory prompts; S2E informs reward shaping, [162] verbalizes decision-tree paths, and enables interactive queries about sub-rewards and motion primitives.S2E also provides insights to end-users at deployment; decomposes rewards by object properties and uses high-level actions at precise 3D positions.Source: B. Policy Interpreter · p. 14

C. Summarization and Outlook · VIII. DISCUSSION · A. Comparison of Different LLM-Enhanced RL Approaches

The survey compares the strengths and limitations of four LLM roles in RL and discusses real-world applications and future opportunities, including multimodal environments. It also outlines role-specific challenges and directions for improving world models and policy explanations.

A. Comparison of Different LLM-Enhanced RL Approaches

  • As generators, LLMs can simulate environments for model-based RL and explain policies, supporting sample efficiency and transparency, though simulation alignment and explanation accuracy remain challenges.They can generate trajectories or latent world representations, while explanations can support end-user understanding and reward shaping; explainable-RL uses remain limited.Source: C. Summarization and Outlook · p. 14, A. Comparison of Different… · p. 15

C. Summarization and Outlook

  • The outlook identifies environment misalignment and observational-only world models as barriers, while noting limited research on generators’ multi-task potential.It proposes better environment alignment in the short term and language instructions or an adapter for flexible world models and multi-task learning in the long term.Source: C. Summarization and Outlook · p. 15
  • Policy explanations depend on understanding feature representations and implicit policy logic, and improving that understanding with domain knowledge or examples remains a major issue.Suggested directions include interpreting observation-policy correlations and using MDP context to answer nuanced “why” and “why-not” questions.Source: C. Summarization and Outlook · p. 15

VIII. DISCUSSION

  • The discussion reviews the LLM-enhanced RL framework, including multimodal-model integration, before analyzing role comparisons, potential applications, opportunities, and challenges.Source: §VIII · p. 15

A. Comparison of Different LLM-Enhanced RL Approaches

  • The comparison frames LLM-enhanced RL through four roles, each with distinct strengths and limitations.The discussion also considers real-world applications and future opportunities, particularly in multimodal information environments.Source: A. Comparison of Different… · p. 15, §VIII · p. 15
  • As information processors, LLMs translate complex multimodal inputs into agent-usable formats, but can incur computational costs and lack specialized domain knowledge.Source: A. Comparison of Different… · p. 15
  • As reward designers, LLMs offer intuitive, flexible rewards that can align RL objectives with human intentions, but task-specific nuance and long-term goals remain difficult to capture.Source: A. Comparison of Different… · p. 15
  • As decision-makers, LLMs can generate or guide actions to support sample efficiency, exploration, reasoning, and planning, but overhead and inherited biases can be limiting.Source: A. Comparison of Different… · p. 15

B. Applications of LLM-Enhanced RL · C. Opportunities for LLM-Enhanced RL

The survey identifies robotics, autonomous driving, energy management, and healthcare recommendation as potential applications of LLM-enhanced RL. It also outlines opportunities to extend RL into underexplored settings and strengthen LLMs through model- and agent-level capabilities.

B. Applications of LLM-Enhanced RL

  • In robotics, LLM-enhanced RL could improve human-robot interaction, help robots understand human needs and behavioral logic, and strengthen decision-making and planning.These opportunities draw on LLMs’ natural-language understanding and general logical reasoning abilities.Source: B. Applications of LLM-Enhanced RL · pp. 15–16
  • For autonomous driving, LLMs could process multimodal sensor data and natural-language instructions or design rewards spanning safety, efficiency, and passenger comfort.Source: B. Applications of LLM-Enhanced RL · p. 16
  • In energy management, LLM-enhanced RL could design rewards for economy, safety, and low carbon while improving sample efficiency for new energy systems.Source: B. Applications of LLM-Enhanced RL · p. 16
  • In healthcare recommendation, LLMs could use domain knowledge to analyze patient data and histories, accelerating RL learning and improving diagnostic and treatment recommendations.Source: B. Applications of LLM-Enhanced RL · p. 16

C. Opportunities for LLM-Enhanced RL

  • Beyond general RL, [134],, opportunities include multi-agent, safe, transfer, explainable, multi-task, in-context, and human-centric RL.LLM-based multi-agent RL remains largely unexplored, despite work on multi-LLM-agent collaboration [165], [166]; LLMs could facilitate agent communication and collaboration [168]. They could also support safety-oriented cost functions and transfer through task or environment similarities.Source: C. Opportunities for LLM-Enhanced RL · p. 16
  • LLMs could support multi-task RL through entity representation, instruction alignment, action-set and expert-action generation, and task-specific trajectories; in-context and human-centric RL offer further opportunities.In-context RL uses a query state and offline dataset for sample-efficient decision-making without extensive trial and error; LLMs could mediate human-AI communication in healthcare and robotics [170].Source: C. Opportunities for LLM-Enhanced RL · p. 16
  • Model-level opportunities include retrieval-augmented generation [171] and continual learning to provide domain knowledge and adapt to tasks while retaining pre-trained capabilities.Source: C. Opportunities for LLM-Enhanced RL · p. 16
  • Agent-level opportunities include planning, memory, and action modules, plus collaboration among specialized LLM agents to guide RL.CoT and MCTS could aid long-term planning, memory retrieval could support adaptive learning from experience [176], and tools such as mathematical solvers [177] and internet browsers [178] could assist complex computation and real-time information processing. Multiple agents could divide short-term problem-solving, long-term planning, and coordination roles [179].Source: C. Opportunities for LLM-Enhanced RL · pp. 16–17

D. Challenges of LLM-Enhanced RL · IX. CONCLUSION

The survey organizes LLM-enhanced RL around four roles in the agent–environment paradigm and reviews their methods, applications, challenges, and future directions. It also identifies model limitations, adaptation, computational demands, and responsible real-world deployment as key challenges.

D. Challenges of LLM-Enhanced RL

  • LLM biases and hallucinations can undermine input interpretation and agent reliability, motivating context-specific capability evaluation and uncertainty quantification [180].The survey calls for robust evaluation frameworks to characterize capability boundaries in specified RL contexts and uncertainty methods to flag unreliable responses.Source: D. Challenges of LLM-Enhanced RL · p. 17
  • LLMs may not adapt to novel RL environments, so methods must expand task knowledge and ground it in specific domains.Options include RAG with external knowledge [171], synthetic-data augmentation, continual learning, value functions for instruction utility, and expert trajectories for in-context learning.Source: D. Challenges of LLM-Enhanced RL · p. 17
  • LLM inference overhead slows RL learning, while prompt pruning [183], MoE [184], SSM, caching [186], and asynchronous processing [187] offer efficiency strategies.These approaches target input length, conditional computation, linear-time sequence modeling, reuse of intermediate computations, and parallel execution, respectively.Source: D. Challenges of LLM-Enhanced RL · p. 17
  • Responsible deployment must address privacy, intellectual property, accountability, transparency, robustness, and ethical and legal standards.Proposed approaches include differential privacy [188], interpretable systems, adversarial training [190], and integrating societal values and ethical standards [191].Source: D. Challenges of LLM-Enhanced RL · p. 17

IX. CONCLUSION

  • As information processors, LLMs extract observational representations and formal specifications to improve sample efficiency, with future work targeting goal-based and multimodal information.Source: §IX · pp. 17–18
  • As reward designers, LLMs can generate high-performing rewards for intricate or unquantifiable tasks, but prompt revisions motivate automated self-evolving frameworks.Source: §IX · p. 18
  • As decision-makers, LLMs provide direct actions or indirect advice to improve exploration, while reducing online computational overhead remains an important direction.Source: §IX · p. 18
  • As generators, LLMs serve as high-fidelity world models and language-based policy interpreters, with human instructions proposed to improve their accuracy and generalizability.Source: §IX · p. 18

Related papers

Loading 2404.00282v3…