Embodied AI: From LLMs to World Models

I. INTRODUCTION

• Embodied AI has shifted from unimodal systems focused on individual components toward multimodal agents intended for complex tasks in dynamic environments.Unimodal methods are limited by narrow modality-specific information and gaps between components.Source: §I · p. 1
• MLLMs support semantic reasoning and task decomposition, but can fail to ground predictions in physical dynamics or adapt to environmental feedback in real time.They connect multimodal inputs with motor-action sequences in end-to-end embodied applications.Source: §I · p. 2
• World models build internal representations and predict future outcomes to support physical-law-compliant interactions, but struggle with open-ended semantic reasoning and task decomposition without explicit priors.Source: §I · p. 2
• Joint MLLM-WM architectures are proposed to combine contextual task reasoning with physics-aware simulation for complex physical-world tasks.The paper presents the combination as a way to bridge semantic intelligence and grounded physical interaction.Source: §I · p. 2
• The paper introduces embodied AI’s history, technologies, components, and hardware, and surveys its LLM/MLLM- and world-model-driven developments.It also outlines applications and future directions, including service robotics, rescue UAVs, industrial robots, autonomous embodied AI, hardware, and swarms.Source: §I · p. 2

A. The Historical View

• Embodied AI developed from theories linking intelligence to physical experience through behavior-based robotics, developmental humanoid learning, and deep-learning-driven navigation and manipulation.The historical account spans Turing’s 1950 proposal, cognitive science in the 1980s, robotics advances in the late 1980s and 1990s, and adaptive interaction.Source: A. The Historical View · p. 3

1) Key Technologies of Embodied AI:

• Computer vision, NLP, reinforcement learning, LLMs/MLLMs, and world models support embodied agents’ perception, cognition, and interaction.LLMs/MLLMs support instruction understanding and multimodal reasoning, while world models represent and predict environmental dynamics for imagination-based planning.Source: 1) Key Technologies of Embodied AI · p. 3

2) Key Components of Embodied AI:

• Active perception spans visual SLAM, 3D scene understanding, and active exploration, with semantic methods helping visual SLAM handle dynamic environments.The section also describes geometric visual SLAM as effective in static settings but challenged by dynamic scenes.Source: 2) Key Components of Embodied AI · pp. 3–4
• Embodied cognition covers self-planning, memory-driven self-reflection, and multimodal foundation models for planning, reasoning, and long-horizon tasks.Structured learning can be efficient within training distributions but lacks robustness to out-of-distribution scenarios; reflective agents use experience to support correction and future decisions.Source: 2) Key Components of Embodied AI · p. 5
• Dynamic interaction includes action control, behavioral interaction, and collaborative decision-making across single-agent, multi-agent, and human-in-the-loop settings.Reliable long-horizon behavioral interaction under sparse feedback remains challenging.Source: 2) Key Components of Embodied AI · pp. 6–7

• Hardware-friendly optimization must balance model performance with efficiency to support responsive, energy-aware embodied agents, especially at the edge.The section identifies model compression, compiler optimization, domain-specific accelerators, and hardware-software co-design as four hardware-optimization components.Source: C. Hardware · p. 7

1) Hardware-aware Model Compression:

• Quantization reduces weight and activation bit-widths, while pruning removes redundant parameters to lower model size and computational cost.Power, performance, and area metrics can guide task-specific trade-offs between accuracy and deployment on physical platforms.Source: 1) Hardware-aware Model Compression · p. 7

2) Compiler-level Optimization:

• Compiler toolchains optimize embodied models through operator fusion, redundant-computation elimination, loop reordering, and tiling for low-latency inference.The section names TVM, built on LLVM and CUDA, as a cross-platform code-generation example.Source: 2) Compiler-level Optimization · p. 7

3) Domain-specific Accelerators:

• Domain-specific accelerators target frequent operations with fast, energy-efficient hardware tailored to embodied AI workloads.FPGAs offer reconfigurability, C ग्रRAs improve dataflow-heavy computations, and ASICs offer high throughput and energy efficiency.Source: 3) Domain-specific Accelerators · p. 7

4) Hardware-software Co-design:

• Hardware-software co-design jointly optimizes algorithms with systems or hardware to improve deployment efficiency.Examples include using GPU resources effectively and tuning model operators with hardware configurations or matched quantization schemes.Source: 4) Hardware-software Co-design · p. 7

D. Benchmarks and Evaluation Metrics

• Embodied AI evaluation uses varied testbeds and metrics to assess task success, responsiveness, energy use, and other task-specific properties.Habitat emphasizes visual realism, ManiSkill object diversity, MuJoCo physical accuracy, and EmbodiedBench multimodal integration; sim-to-real transfer validation remains challenging.Source: D. Benchmarks and Evaluation Metrics · p. 7

E. From Unimodal to Multimodal

• Unimodal systems can excel in individual modules but face limited information within a modality and gaps in information transfer across modules.Multimodal methods integrate sensing modalities to support a more holistic environmental understanding and deeper integration among perception, cognition, and interaction.Source: E. From Unimodal to Multimodal · p. 8, Figure 4 · p. 8

1) Semantic Reasoning:

• LLMs use linguistic patterns, context, and pretrained knowledge to interpret instructions through hierarchical semantic abstraction and inference.Attention weighs relevant cues, while prompts align conceptual representations with intended outcomes.Source: 1) Semantic Reasoning · p. 8

2) Task Decomposition:

• LLMs decompose complex objectives into contextually ordered subtasks, while SayCan [22] constrains actions with a pretrained library and checks feasibility with value functions.This approach is limited by a fixed action library and specific environment, making expansion to new robots and environments difficult.Source: 2) Task Decomposition · p. 8

B. MLLMs Boost Embodied AI

• MLLMs bridge high-level multimodal inputs and low-level motor actions in end-to-end applications, with VLMs and VLAs as major model families.The development roadmap in Fig. 5 marks conceptual and practical milestones.Source: B. MLLMs Boost Embodied AI · pp. 8–9, Figure 5 · p. 9

1) VLMs for Embodied AI:

• VLMs combine visual and language instruction understanding for goal-driven agents, with representative methods supporting multi-task completion in fixed action spaces.PaLM-E links continuous sensor encodings to language; ShapeNet reduces navigation path-planning errors through 3D spatial reasoning.Source: 1) VLMs for Embodied AI · p. 9

2) VLAs for Embodied AI:

• VLAs connect multimodal inputs to low-level control through differentiable pipelines, and the section describes them as supporting adaptation across robots and environments.Examples include RT-2 [24], Octo [129], and PerAct, which respectively tokenize robot inputs for action generation, pretrain on 100K demonstrations, and use 3D voxels for millimeter-level grasp accuracy.Source: 2) VLAs for Embodied AI · p. 9

1) MLLMs for Active Perception:

• MLLMs support active perception through semantic scene grounding, scene understanding, and feedback-driven exploration.Examples include MLLM-enriched 3D SLAM, language-aligned scene features, and motion-level feedback used to revise plans.Source: 1) MLLMs for Active Perception · p. 9, C. Classification of MLLMs for… · p. 9

2) MLLMs for Embodied Cognition:

• For embodied cognition, MLLMs support self-planning, memory-driven reflection, and adaptation of multimodal foundation models to embodied settings.CoT-VLA [33] predicts subgoal images, while Reflexion stores linguistic feedback in episodic memory for future planning.Source: 2) MLLMs for Embodied Cognition · pp. 9–10

3) MLLMs for Dynamic Interaction:

• For dynamic interaction, MLLMs generate control signals or behavioral action sequences and support collaborative decisions among agents and humans.Examples include autoregressive actions, auxiliary policy heads, code generation, π-0 trajectories, Combo [146] decentralized cooperation, and VLAS [232] conversational interaction.Source: 3) MLLMs for Dynamic Interaction · p. 10

A. World Models Boost Embodied AI

• World models empower embodied AI by forming internal representations and predicting future states of the external world to support physics-compliant interaction.Fig. 6 presents a roadmap of world-model development milestones.Source: A. World Models Boost Embodied AI · p. 10, Figure 6 · p. 10

1) Internal Representations of the External World:

• Structured latent representations compress sensory inputs while encoding object dynamics, physical laws, spatial structure, and relations between entities.Disentangled properties support counterfactual reasoning and transfer, while uncertainty estimation and consistency constraints inform planning.Source: 1) Internal Representations of the… · p. 10

2) Future Predictions of the External World:

• Future predictions simulate action-sequence rewards across time horizons, filtering physically implausible or strategically incoherent trajectories.Uncertainty-aware simulation can replace costly trial and error with mental rehearsal, and prediction errors drive model refinement.Source: 2) Future Predictions of the External… · pp. 10–11

B. Classification of World Models for Embodied AI

• The classification highlights RSSM-, JEPA-, and Transformer-based world models, with hierarchical and diffusion approaches also represented.Fig. 6 shows the related structures.Source: B. Classification of World Models for… · p. 11

1) RSSM-based WMs for Embodied AI:

• RSSM underpins the Dreamer family –, learning temporal dynamics from visual inputs for action selection through latent trajectory optimization.Its probabilistic and deterministic hidden-state components represent environmental uncertainty and systematic patterns.Source: 1) RSSM-based WMs for Embodied AI · p. 11

2) JEPA-based WMs for Embodied AI:

• JEPA learns mappings from inputs to anticipated outcomes in abstract latent spaces rather than generating pixel-wise reconstructions.Self-supervised inference of occluded data supports transfer across visual and non-visual downstream domains.Source: 2) JEPA-based WMs for Embodied AI · p. 11

3) Transformer-based WMs for Embodied AI:

• Transformer-based world models use attention for parallel context weighting, and reported variants outperform RSSM in memory-intensive interactive scenarios.Google’s Genie uses an ST-Transformer and large-scale self-supervised video pretraining to create synthetic interactive environments.Source: 3) Transformer-based WMs for Embodied AI · p. 11

A. MLLMs and WMs

• MLLMs provide contextual task reasoning but overlook physical constraints, whereas world models simulate physics but lack high-level semantics; their combination bridges the two.Fig. 7 depicts the proposed joint MLLM-WM architecture.Source: A. MLLMs and WMs · p. 11, Figure 7 · p. 12

1) The Limitations of MLLMs for Embodied AI (without WMs):

• MLLMs can produce impractical plans when they fail to ground predictions in physical dynamics and may adapt poorly to real-time environmental feedback.The text attributes these limitations to static pretrained knowledge rather than continuous physical interaction.Source: 1) The Limitations of MLLMs for… · p. 11

2) The Limitations of WMs for Embodied AI (without LLMs/MLLMs):

• World models struggle with open-ended semantic tasks and generalizable task decomposition without explicit priors, while prediction depends heavily on domain-specific interaction records.The text notes that rigid-object manipulation models may fail on deformable materials without extensive retraining.Source: 2) The Limitations of WMs for Embodied… · p. 11

3) MLLMs Boosting WMs Reasoning:

• MLLMs could enrich world models through multimodal semantic representations, high-level subtask planning, and human-feedback-based updates to model priors.Examples include CLIP-based scene-language alignment, GPT-4V [252] plans using world-model context, and RLHF.Source: 3) MLLMs Boosting WMs Reasoning · pp. 11–12
• The described MLLM-to-world-model reasoning strategies have not been achieved in existing works.The text likewise says the described world-model support for MLLM decisions has not been achieved in existing works.Source: 3) MLLMs Boosting WMs Reasoning · p. 12, 4) WMs boosting MLLMs Interaction · p. 12

4) WMs boosting MLLMs Interaction:

• World models could refine MLLM decisions by supplying physical constraints, spatiotemporal context, and closed-loop interaction experience.Examples describe filtering motions against torque limits [254], preventing navigation collisions, and using task histories to correct kinematic errors.Source: 4) WMs boosting MLLMs Interaction · p. 12

B. Joint MLLM-WM-driven Embodied AI Architecture

• The architecture exchanges information among MLLMs, WMs, robots, and hardware to support complex tasks in physical worlds.Self-state metrics inform both models, with WMs representing physical state and MLLMs contextualizing it for task alignment.Source: B. Joint MLLM-WM-driven Embodied AI… · p. 12, 1) Robots →Self-State Inputing… · p. 12

1) Robots →Self-State Inputing →MLLMs/WMs →Hardware Embodiment →Robots:

• Self-state metrics feed both models: WMs build representations of physical state, while MLLMs contextualize those states for task alignment.Hardware embodiment implements the models in physical devices to address sim-to-real problems, with the bidirectional flow intended to respect mechanical limits and high-level goals.Source: 1) Robots →Self-State Inputing… · p. 12

2) MLLMs →Task Planning →WMs →Memory Updating →MLLMs:

• MLLMs decompose instructions into subtasks, WMs predict outcomes, and memory updates feed task experiences back to MLLMs for lifelong learning.The memory process represents forgetting past task memories, renewing current memories, and predicting future memories; past failures inform future planning.Source: 2) MLLMs →Task Planning →WMs →Memory… · p. 12

3) Environments →Active Perception →MLLMs/WMs →Dynamic Interaction →Environments:

• WMs predict key environmental changes for active perception, while MLLMs and WMs jointly support world representation, task decomposition, prediction, and iterative interaction.The passage describes continuous iteration as the basis for adaptive perception and interaction in dynamic environments.Source: 3) Environments →Active Perception… · p. 12

C. Discussions

• MLLMs support contextual reasoning and adaptive planning, while WMs provide physics-based simulation; their combination is described as balancing abstract reasoning with physical interaction.The passage also describes feasibility validation and iterative plan refinement, and says joint architectures support cross-modal generalization in partially observable or novel scenarios.Source: C. Discussions · p. 13
• Joint systems face synchronization delays, semantic-physical misalignment, scalable-memory demands, and unmet needs for broad multimodal data and robustness to noise and partial observability.The passage identifies lightweight MLLM inference, tighter feedback loops, and dynamic context filtering as needed to minimize latency.Source: C. Discussions · p. 13

VI. EMBODIED AI APPLICATIONS

• The applications overview highlights service and rescue robots and other domains, with joint MLLMs and WMs associated with active perception, embodied cognition, and dynamic interaction.The section presents these as application areas rather than reporting a comparative evaluation.Source: §VI · p. 13

A. Service Robotics

• Service robots combine language instructions with robot control for household tasks, while healthcare and public settings include assistance, navigation, and delivery applications.The section also describes few-shot skill acquisition from limited demonstrations and notes that current approaches remain limited on long-horizon tasks.Source: A. Service Robotics · p. 13

B. Rescue UAVs

• For rescue UAVs, joint WMs and MLLMs are presented as a strategy for long-horizon reasoning and autonomous decisions under uncertainty.WMs maintain evolving spatiotemporal representations for planning and risk prediction, including in GPS-denied conditions, while MLLMs ground commands in the UAV belief state.Source: B. Rescue UAVs · p. 14

C. Industrial Robots

• Industrial examples include robots adjusting grip force or finding paths around obstacles, plus factory part realignment and sensor-based package sorting and inventory alerts.The passage contrasts these flexible tasks with traditional robots designed for repeated movements.Source: C. Industrial Robots · p. 14

D. Other Applications

• Other applications include collaborative manufacturing, student-responsive educational robots, multistep virtual agents, and autonomous adaptation in space exploration.The passage also describes visual and touch feedback for handling fragile items and memory development in virtual agents.Source: D. Other Applications · p. 14

VII. FUTURE DIRECTIONS

• Future research prioritizes unified and reliable systems across autonomous embodied AI, hardware, swarms, and evaluation benchmarks.The section frames these directions against the move from simulation to real-world deployment.Source: §VII · p. 14

A. Autonomous Embodied AI

• Autonomous systems need adaptive sensory selection, environmental awareness supported by spatiotemporal and causal memory, and real-time physical interaction paired with MLLMs.The stated goals include adapting to environmental changes, predicting action consequences, and transferring behavior to new environments.Source: A. Autonomous Embodied AI · p. 14

B. Embodied AI Hardware

• Hardware directions include hardware-aware compression, graph-level compilation, domain-specific accelerators, and hardware-software co-design for real-time, energy-efficient execution.The passage names FPGA and CGRA as flexible architectures and ASIC designs as offering high efficiency and performance.Source: B. Embodied AI Hardware · pp. 14–15

C. Swarm Embodied AI

• Swarm research needs collaborative WMs for shared environmental representations, multi-agent representation learning, social-behavior modeling, and human-swarm interaction interfaces.These are presented as requirements for cooperation, communication, role allocation, group decisions, and real-world integration.Source: C. Swarm Embodied AI · p. 15

D. Explainability and Trustworthiness Embodied AI

• Trustworthy deployment requires human-understandable action justifications, ethical adherence, verifiable safety and certification, and robustness to attacks, noise, and distribution shifts.The passage identifies these as open challenges for agents operating in unstructured physical settings and morally ambiguous scenarios.Source: D. Explainability and Trustworthiness… · p. 15

E. Other Directions

• Other directions include lifelong learning without forgetting, human-in-the-loop feedback for learning goals and preferences, and moral decision-making aligned with human values.The passage links these directions to adaptation in dynamic environments and safe, reliable systems.Source: E. Other Directions · p. 15