PaperlayerView exact revision on arXiv

Source-linked AI summary

Large Language Models for Robotics: A Survey

Fanlong Zeng, Wensheng Gan, Zezheng Huai, Lichao Sun, Hechang Chen, Yongheng Wang, Ning Liu, Philip S. Yu

arXiv:2311.07226v2cs.ROcs.AI

TL;DR

Robotics seeks to develop systems that can understand, plan, and act through multimodal interaction, while LLM integration offers stronger language capabilities but introduces unresolved challenges. This survey synthesizes LLM-based robotics across core modules, representative models, applications, coordination methods, and societal implications, concluding that the field has substantial potential alongside constraints in data, training environments, safety, and security.

  • Problem

    Robots require multimodal perception, reasoning, planning, and interaction capabilities, while achieving dexterous embodied intelligence remains complex.

  • Method

    The survey reviews LLM development, representative LLM-based robots, techniques across perception, decision-making, control, and interaction, cross-module coordination, applications, and challenges.

  • Results

    The review identifies LLM-based robotics as a promising path toward embodied intelligence and summarizes its contributions to robot control, perception, decision-making, planning, and interaction.

  • Takeaways & Limitations

    LLM-based robotics provides new paradigms for robot control, path planning, semantic understanding, and more natural human-robot interaction.

  • Takeaways & Limitations

    The field remains constrained by multimodal data collection, imperfect transfer across training environments, and risks involving harmful actions and private-data leakage.

Abstract

from arXiv · show

The human ability to learn, generalize, and control complex manipulation tasks through multi-modality feedback suggests a unique capability, which we refer to as dexterity intelligence. Understanding and assessing this intelligence is a complex task. Amidst the swift progress and extensive proliferation of large language models (LLMs), their applications in the field of robotics have garnered increasing attention. LLMs possess the ability to process and generate natural language, facilitating efficient interaction and collaboration with robots. Researchers and engineers in the field of robotics have recognized the immense potential of LLMs in enhancing robot intelligence, human-robot interaction, and autonomy. Therefore, this comprehensive review aims to summarize the applications of LLMs in robotics, delving into their impact and contributions to key areas such as robot control, perception, decision-making, and planning. This survey first provides an overview of the background and development of LLMs for robotics, followed by a discussion of their benefits and recent advancements in LLM-based robotic models. It then explores various techniques, employed in perception, decision-making, control, and interaction, as well as cross-module coordination in practical tasks. Finally, we review current applications of LLMs in robotics and outline potential challenges they may face in the near future. Embodied intelligence represents the future of intelligent systems, and LLM-based robotics is one of the most promising yet challenging paths toward achieving it.

1. Introduction

This introduction presents LLM-based robotics as a promising route toward embodied intelligence by strengthening natural-language interaction and integrating perception, decision-making, control, and interaction. It also identifies substantial challenges involving computation, safety, dialogue, robot form, datasets, and societal effects.

  • LLMs can give robots stronger natural-language understanding and generation, supporting more intelligent and human-like conversations and interactions.
  • Recent work explores LLMs for understanding user intentions, generating fluent responses, and combining language models with knowledge graphs or sentiment analysis.
  • LLM-based robotics faces challenges including substantial computing and data requirements, inaccurate or harmful outputs, dialogue consistency, and uncertainty about suitable robot forms.
  • The survey reviews LLM applications across robot perception, decision-making, control, path planning, interaction, and cross-module coordination.
  • The survey discusses potential future challenges and the possible effects of LLM-integrated robots on human society, including labor replacement and questions about robot consciousness.

2. Language Model Overview

This section introduces LLM development and derivative models used in robotics, including VLM, VLA, VLN, and Agent architectures. It also surveys representative models, companies, and historical trajectories.

  • LLMs are presented as the brains of robotics, with an overview of their development and robotics-oriented derivatives.
  • Evolution of Language Models: LLM development is divided into Statistical, Neural, Pre-trained, and LLM periods.
  • Derivative Models of LLM: VLMs jointly process and semantically align visual and linguistic information, supporting captioning, visual question answering, and medical image diagnosis.
  • Derivative Models of LLM: VLA models combine visual perception, language comprehension, and physical actuation to convert visual information and instructions into executable action sequences.
  • Derivative Models of LLM: The survey identifies VLM, VLA, VLN, and Agent as critical LLM derivatives and compiles their historical development in Tables 2–5.
  • The section also presents prominent LLM companies and representative robotics models, including Gemini, Claude, DeepSeek, Llama, and listed VLA, VLN, and agent systems.

3. Robotics Based on LLMs

This section frames LLM-based robotics around agents, embodied robots, and multi-agent coordination. It covers brain-like agent structures, robot collaboration, long-horizon manipulation, and humanoid platforms.

  • LLM-based robotics uses LLMs as brains within systems that combine perception, decision-making, control, and action.
  • Agents: An LLM-based agent perceives its environment, makes decisions, and executes actions toward goals, while evolving from an intelligent assistant toward an autonomous agent.
  • Multi-Agent: Multi-agent robotics includes collaboration among isomorphic robots and heterogeneous robots, with examples spanning multi-arm manipulation, quadrotors, robotic dogs, and robotic arms.
  • Multi-Agent: RoCo combines LLM-based collaboration and task decomposition for multi-arm motion and introduces RoCoBench, a 6-task benchmark for multi-robot manipulation.
  • Long-Horizon Manipulation: Long-horizon manipulation requires prolonged reasoning, precise execution, and error recovery, while existing approaches face error accumulation and limited execution verification.
  • Long-Horizon Manipulation: Agentic Robot addresses component coordination through a Standardized Action Procedure spanning planning, execution, and verification.
  • Humanoid Robots: The section surveys humanoid robotics companies and platforms including Unitree, Boston Dynamics, Figure AI, and Tesla Optimus.

4. Related Technologies

The section organizes robotics technologies into perception, decision-making, control, and interaction, emphasizing decision-making as the core connection between perception and control.

  • Robotics technologies are divided into perception, decision-making, control, and interaction.
  • Decision-making is emphasized as the core component of LLM-based robotics and as the link between perception and control.

4.1. Perception

Robot perception increasingly uses multimodal models to process visual and linguistic inputs, distinguish 2D from 3D scenes, and support context-dependent understanding. Indirect perception extends these outputs beyond immediate action to planning, decision-making, and multi-step reasoning.

  • 4.1.1. Vision-Language Model: Multimodal perception combines vision and language because conventional language and visual models accept only domain-specific inputs.Visual-language models process images and text simultaneously.
  • 4.1.1. Vision-Language Model: Visual-language models have progressed through self-supervised pretraining, contrastive pretraining, and large multimodal models.Their use also requires distinguishing 2D scenes from 3D scenes.
  • 4.1.1. Vision-Language Model: VLMs can associate images with pertinent text and support navigation by extracting landmarks for topological maps.CLIP is used in LM-Nav to predict text from natural-language instructions.
  • 4.1.2. Scene Understanding: Scene understanding models represent relationships among objects, structures, and semantics to support context-dependent robot decisions.Swin3D provides scalable sparse-voxel attention, while BIP3D combines pretrained 2D features with explicit 3D position encoding.
  • 4.1.3. Indirect Perception: Indirect perception uses perceptual outputs to inform planning, decision-making, and multi-step reasoning rather than only immediate action.GRUtopia uses LLM-driven NPCs to provide environmental knowledge and action feedback.

4.2. Decision-making

Decision-making connects perception to control by mapping observations to actions, while LLMs contribute knowledge, memory, reasoning, planning, and coordination. The surveyed work spans task planning, generalist policies, low-level trajectory generation, and multi-robot execution.

  • 4.2. Decision-making: Decision-making analyzes perception inputs and generates appropriate actions, connecting the preceding and following robot modules.A policy maps observations to actions through a defined relationship.
  • 4.2. Decision-making: LLMs can serve as robot brains by integrating knowledge, memory, and reasoning for intelligent task planning and execution.Their capabilities include language understanding, generation, reasoning, and few-shot generalization.
  • 4.2. Decision-making: Robot memory can reuse past strategies for new tasks, but growing memory sequences require effective storage, indexing, management, and retrieval.The discussion identifies catastrophic forgetfulness as a risk of increasing memory burden.
  • 4.2.1. Planning: Planning decomposes abstract goals into sequential subtasks or tree structures, with chain-of-thought processing one subtask at a time.FLARE is described as generating environment-aware, executable plans with minimal language-based action examples.
  • 4.2.1. Decision Policy: GPT-4 directly predicted dense end-effector pose sequences from object detection and segmentation, without examples, motion primitives, or external trajectory optimizers.The study also reports failure detection and trajectory replanning for common manipulation tasks.
  • 4.2.2. Multi-Robot Collaboration: Long-horizon heterogeneous multi-robot planning remains challenging because coordination demands increase while context windows impose limits.The surveyed direction emphasizes token-efficient planning and centralized task assignment based on robot profiles.
  • 4.2.2. Multi-Robot Collaboration: Roco combines LLM-based communication and reasoning with subtask plans and waypoint paths, while EMOS reports more efficient decentralized parallel execution.Robot-specific agents can execute allocated actions through functional calls and control libraries.

4.3. Control

Robot control research addresses how language-conditioned behavior is learned and how parsed commands become executable actions. VLA models integrate visual perception, language understanding, and action generation, while reasoning and human intervention target action quality and dexterity.

  • 4.3. Control: Robot learning of natural-language action policy centers on learning language-conditioned behavior and executing actions after parsing commands.The survey presents these as two core control questions.
  • 4.3.1. Language-conditioned Behavior: Execution methods can evaluate whether a parsed skill is executable in the current state using temporal-difference reinforcement learning and affordance functions.LM-Nav is introduced in the surrounding discussion of language-conditioned execution.
  • 4.3.2. Vision-Language-Action Models: VLA is an end-to-end model that integrates visual perception, language understanding, and action generation to output executable robotic commands.It connects environmental stimuli and task requirements to physical responses.
  • 4.3.2. Vision-Language-Action Models: OpenVLA integrates visual perception, natural-language comprehension, and action generation to execute complex tasks from language instructions.Its architecture includes a hybrid visual encoder combining DINOv2 spatial features.
  • 4.3.2. Vision-Language-Action Models: FAST achieves a 10× improvement in token efficiency over conventional action tokenization methods by compressing action sequences with DCT and BPE.This method is presented as an optimized variant of π0.
  • 4.3.3. Human Intervention: ECoT adds structured chain-of-thought reasoning before action, while SERL combines demonstrations, human intervention, and efficient reinforcement learning for dexterous manipulation.The passage reports near-perfect success rates and fast cycle times for SERL.

4.4. Interaction

Interaction enables robots to engage with humans and environments, but capabilities learned in virtual or simulated settings often lose accuracy in the real world. LLM- and MLLM-based agents support planning, multimodal decision-making, memory, and natural-language interaction.

  • 4.4. Interaction: Interaction connects robots with both environments and humans, yet models transferred from virtual or simulated settings tend to be less accurate in the real world.The passage identifies real-world transfer as a central interaction challenge.
  • 4.4.1. Embodied Agents: LLMs and MLLMs used as cognitive components improve planning and reasoning efficiency for open-world embodied agents such as Minecraft systems.These systems commonly pursue material collection, tool-crafting, and the Obtain Diamond task.
  • 4.4.1. Embodied Agents: JARVIS-1 translates multimodal inputs into actionable decisions and uses multimodal memory to retain observations and environmental feedback.It is designed as a multitask agent for Minecraft navigation.
  • 4.4.2. Human-Robot Interaction: Language user interfaces let people interact with robots through natural language, avoiding the rigid instructions or intricate algorithms associated with traditional approaches.ChatGPT is given as a representative LUI product.
  • 4.4.2. Human-Robot Interaction: Because natural language is vague and ambiguous, future robots should use LLMs to ask clarifying questions when user intent is uncertain.The proposed interaction behavior gives robots greater initiative in dialogue.

5. Cross-Module Coordination

Cross-module coordination combines perception, reasoning, policy generation, action monitoring, and causal reasoning to address practical robotic tasks. The survey highlights affordance grounding, physical grounding, navigation, and failure reasoning as representative coordination problems.

  • 5.1. Affordance Grounding: Affordance grounding links objects to possible uses by integrating perception and reasoning, supporting interaction understanding, visual navigation, and manipulation.It requires scene understanding spanning object detection, localization, recognition, layout, 3D shape, physics, and functionality.
  • 5.2. Physical Grounding: Physical grounding maps perceived object or language information to specific real-world entities or physical states through perception and spatial reasoning.Current VLMs remain limited in understanding physical concepts such as material and fragility.
  • 5.3. Failure Reasoning: Failure reasoning integrates perception, action monitoring, and causal reasoning to detect and analyze failed executions automatically.REFLECT queries an LLM using hierarchical summaries of past robot experiences generated from multisensory observations.
  • 5.4. Navigation: Navigation requires both perception and policy generation, with navigation modalities and vision-based models used to learn navigation implications.BADGR uses end-to-end self-supervised learning, while GNM targets generalization across environments and robot embodiments.
  • 5.4. Navigation: Vision-and-language navigation connects visual observations with actions over time while addressing multimodal processing, dataset, and environment-transfer challenges.VLN is designed for navigation in complex 3D environments using visual observations and language.

6. Applications of LLMs in Robotics

The survey describes LLM–robotics applications spanning navigation, language interaction, perception, collaboration, industrial automation, healthcare, environmental monitoring, agriculture, and education. Across these domains, large models provide semantic understanding, reasoning, multimodal analysis, and natural-language interaction.

  • Autonomous navigation and path planning: Large models assist autonomous navigation and path planning by interpreting sensor data, obstacles, target locations, and navigation objectives.They support path-planning solutions in unknown environments through semantic understanding and reasoning.
  • Speech interaction and NLP: LLMs support speech interaction by recognizing speech, understanding semantics, generating language, answering queries, executing tasks, and personalizing services.The surveyed applications include both user-facing dialogue and task execution.
  • Visual perception and object recognition: Large models aid visual perception through image and video analysis for object recognition, target detection, and scene understanding.Applications include autonomous driving, vision-based navigation, and industrial automation.
  • Human-robot collaboration and social robots: Natural-language processing and emotion analysis enable human–robot collaboration and social robots to interpret intentions, converse, and provide support.The survey identifies healthcare, education, and entertainment as application areas.
  • Industrial automation and robot control: Large models support industrial monitoring, anomaly detection, predictive maintenance, and adaptive control through sensor-data analysis.These capabilities are framed as applications in intelligent industrial automation.
  • Healthcare and rehabilitation robots: Healthcare and rehabilitation robots can analyze medical information for diagnosis, surgical planning, personalized therapy, physical assistance, and rehabilitation exercises.The cited applications include patient care and support for mobility-impaired patients.
  • Other applications: Large models and robots are also described for environmental monitoring, agriculture, and education through sensor analysis, decision support, tutoring, and personalized guidance.Examples include pollution and disaster monitoring, crop management, plant-health assessment, and learning assistance.
  • Application landscape: The survey summarizes applications across autonomous navigation, speech interaction, visual perception, collaboration, industrial automation, healthcare, environmental monitoring, agriculture, and education.It presents these combinations as potential sources of convenience and innovation in human life and work.

7. Challenges

The survey identifies challenges in multimodal dataset collection, training-environment transfer, robot morphology, and LLM deployment. These constraints concern data scarcity, real-world interaction, task-specific design, computation, latency, and unreliable connectivity.

  • 7.1. Datasets: Robotics datasets are diverse and scarce because training requires unified text, image, touch, policy-control, and 3D data.Collecting and processing these multimodal datasets requires substantial time and resources.
  • 7.2. Training Schemes: Game and simulation training can transfer poorly to real scenes, while real-world training introduces substantial cost and data-collection challenges.The passage gives an example of 90% accuracy in a game or simulation falling to 10% in a real scene.
  • 7.3. Shape: Robot morphology lacks a unified design approach, and task-specific pipelines can be more efficient than humanoid forms for some applications.The survey advocates modular designs with interchangeable components and unified module specifications.
  • 7.4. LLM Deployment: Current technical limitations prevent local deployment of LLMs on robots, motivating cloud-based and local two-brain architectures.Cloud interaction introduces latency, and signal loss may leave robots with only local control, causing unpredictable behavior.

7.5. Security

Security and broader societal challenges include biased or mistaken model guidance, privacy risks from cloud interaction, dialogue persistence limits, labor disruption, and questions about robots’ status.

  • 7.5. Security: Pre-training biases or misconceptions can produce problematic guidance, while physical robot actions may turn such errors into safety risks.The survey also identifies household data leakage risks when robots interact with cloud services.
  • 7.5. Security: Offline operation may reduce privacy-leakage risk but can compromise robot performance.The trade-off arises because household robots may require cloud interaction for certain tasks.
  • 7.6. Dialogue Consistency: Continuous dialogue is needed for robots to update goals and strategies during ambiguous tasks, while limited context can cause catastrophic forgetting.Dialogue persistence is identified as a challenge for long-term tasks such as scientific research.
  • 7.7. Social Impact: Greater robot capability may reduce demand for human labor and potentially disrupt social structures and stability.The survey frames these effects as possible consequences of robots performing repetitive tasks.
  • 7.8. Robot Status: Human-like appearance or intelligence may encourage emotional attachments and raise questions about whether embodied intelligence should remain viewed merely as a tool.Companion robots are presented as a context for this concern.

8. Promising Directions for Future Work

Future work should address safe real-world deployment, multimodal processing, modular robot design, and autonomous perception. These directions target practical constraints in training, sensing, embodiment, and task adaptability.

  • Safety: Agent safety during task execution requires attention because robotic actions could cause serious harm in ordinary tasks.The paper illustrates this concern with a robot asked to make lunch that accidentally sets the kitchen on fire.
  • Real-World Deployment: Training in simulated or gaming environments can leave agents insufficiently acclimatized to diverse real-world scenarios, compromising deployment performance.The paper attributes this problem to disparities in feedback between training environments and real-world settings.
  • Multimodal Processing: VLA models address multimodal input challenges by converting visual and natural-language inputs into multimodal sentences and producing actions in the same format.This approach is proposed for agents whose multimodal perception must be processed by LLM-based systems.
  • Modular Components: A modular design methodology would let robot components be swapped according to task-specific needs, such as using different limbs for cooking or yard work.The proposal depends on establishing unified specifications for robot modules.
  • Autonomous Perception: Autonomous perception could combine hearing, vision, and common sense to detect situations such as broken glass and handle them without direct instruction.The paper identifies LLMs used as the robot’s brain as a possible source of common-sense capabilities.

9. Conclusion

The survey synthesizes LLM-based robotics across representative models, core robotic modules, applications, challenges, and societal implications. It presents embodied intelligence as a promising future direction while emphasizing the need to address broader impacts proactively.

  • Conclusion: The survey reviews methods and technologies for LLM-based robots, including LLM development, expected robotic improvements, and representative models.It frames LLMs as a possible robotic brain and surveys models proposed in recent years.
  • Conclusion: The survey organizes robotics into perception, decision-making, control, and interaction modules and explains each module’s role.It also examines coordination among these modules in practical robotic systems.
  • Conclusion: The survey considers applications and challenges involving training, safety, robot shape, deployment, long-term task performance, and social and ethical impacts.The discussion extends from current application scenarios to possible effects of post-intelligent robots on human society.
  • Conclusion: LLM-based robotics offers a potential path toward embodied intelligence, while its societal impact requires proactive attention.The paper connects future robotic capability with hardware advances and broader implications for intelligent-system development.
Loading 2311.07226v2…