PaperlayerView exact revision on arXiv

Source-linked AI summary

Data Pyramid for Embodied Manipulation: A Survey

Yifan Ye, Yankai Fu, Yaoxu Lv, Bohan Hou, Jun Cen, Lingdong Kong, Duo Zheng, Tianxing Chen, Jiaming Liu, Ziang Cao, Yunfan Lou, Wei Chow, Xian Sun, Yingshuo Wang, Kuangzhi Ge, Xiaowei Chi, Xidong Zhang, Zhibo Pang, Yiwu Zhong, Sirui Han, Zhihe Lu, Weihao Yuan, Qifeng Chen, Michael Yu Wang, Yao Mu, Ziwei Liu, Jianfei Yang, Ping Luo, Shanghang Zhang

arXiv:2607.24744v2cs.ROcs.CV

TL;DR

Embodied foundation models need data linking observations with physical states and actions, but heterogeneous sources must be selected and combined effectively. This survey organizes those sources into a data pyramid and finds that model pretraining increasingly uses heterogeneous mixtures beyond real-robot trajectories.

  • Problem

    Embodied foundation models require supervision linking observations, physical states, dynamics, and actions, raising how heterogeneous data sources should be selected and combined.

  • Method

    The paper builds a data-pyramid taxonomy of five embodied data categories and reviews how their construction and use shape embodied foundation-model pretraining.

  • Results

    Reviewed models show a shift from real-robot-dominated recipes toward increasingly heterogeneous mixtures of real-robot, simulation, general, egocentric, and occasional UMI-style data.

  • Takeaways & Limitations

    The pyramid frames embodied data sources by scalability and robot alignment, with distinct strengths and limitations across quality, diversity, reusability, and physical fidelity.

  • Takeaways & Limitations

    Real-robot data remain costly and difficult to scale, while embodiment, sensing, and action-representation differences complicate aggregation and cross-platform transfer.

Abstract

from arXiv · show

Multimodal foundation models learned to see and to speak by consuming the whole internet. Embodied agents admit no such shortcut, since they require data that couple observations with physical states and actions. These signals can be provided, to varying degrees, by multiple data sources. In this work, we organize the embodied data ecosystem as a "pyramid" spanning five complementary sources: real-robot data, UMI-style data, egocentric and exocentric data, simulation data, and general vision-language data. We organize the pyramid around the tension between scalability and robot alignment, and further characterize each source in terms of data quality, diversity, reusability, and physical fidelity. We then analyze recent embodied foundation models through the lens of their data recipes, examining how different sources are selected, aligned, and mixed during pretraining. For embodied brain models, vision-language-action models, and world-action models alike, we relate data composition to capabilities in perception, reasoning, planning, action generation, and world prediction. We close by discussing six open challenges: building large-scale tactile datasets, collecting failure and recovery data, developing scalable data-collection pipelines, aligning actions across embodiments, leveraging egocentric data for dexterous manipulation, and designing principled data recipes for robot learning. We hope this work paves the foundation for the design of next-generation embodied systems.

1 Introduction

The introduction frames embodied foundation-model training as a data problem requiring physical-state and action supervision beyond visual and linguistic pretraining. It proposes a data pyramid organized by scalability and robot alignment, analyzes heterogeneous data recipes, and identifies open challenges for collecting and using embodied data.

  • Motivation: Embodied agents must understand physical states and dynamics, reason about action effects, and execute behaviors, requiring more than visual and linguistic inputs.
  • Data-pyramid taxonomy: Scalability and robot alignment organize the taxonomy, while quality, diversity, reusability, physical fidelity, and collection scalability characterize each category.Scalability concerns expansion constraints such as hardware, labor, resets, and safety supervision; robot alignment concerns support for physical-robot learning and execution.
  • Data-pyramid taxonomy: The data pyramid orders five sources from real-robot data to general data, spanning UMI-style, egocentric and exocentric, and simulation data.The ordering moves from stronger physical grounding and actionable robot supervision toward greater accessibility and scalability.
  • Data-centric model analysis: Embodied foundation models increasingly use heterogeneous mixtures, shifting from recipes dominated by real-robot trajectories toward combinations including simulation, general, egocentric, and occasionally UMI-style data.The analysis relates dataset properties such as semantic coverage, temporal structure, action supervision, embodiment alignment, diversity, and physical fidelity to perception, reasoning, planning, action generation, and world prediction.
  • Open challenges: Open challenges include collecting failure, recovery, and tactile interaction data; reducing manual teleoperation; improving coverage; and integrating heterogeneous data across embodiments.The introduction also highlights quality and diversity assessment, multimodal standardization, human-to-robot and sim-to-real transfer, and evaluation of heterogeneous sources.

2 Real-Robot Data

Real-robot data are the most physically grounded source in the embodied-data pyramid, directly linking observations, robot states, actions, and physical consequences. Their value for reliable manipulation is constrained by costly collection, limited coverage, and cross-embodiment aggregation challenges, motivating broader teleoperation, synthetic augmentation, policy rollouts, and failure-recovery supervision.

  • Dataset evolution: Dataset development progressed from scripted, task-specific collections toward broader human-operated demonstrations and large-scale ecosystems spanning bimanual, mobile, humanoid, and dexterous systems.The shift reflects movement from training individual policies toward generalist robot models.
  • Collection paradigms: Scripted collection enables repeated and parallel acquisition at relatively low marginal cost by shifting human effort from episode-level control to task, control-pipeline, and execution-condition design.By contrast, leader-follower and integrated leader-follower teleoperation provide intuitive or precise control, with integrated systems supporting direct visual and force feedback and avoiding separate-arm kinematic-mapping errors.
  • Scale and coverage: Approximately 800K episodes across 12 tasks characterize MT-Opt, while RT-1 and RoboMIND provide 130K and 107K trajectories across hundreds of manipulation skills, respectively.More recent datasets have reached million-scale collection or thousands of hours of interaction.
  • Scale and coverage: Data diversity often matters more than raw trajectory count because task, embodiment, modality, and trajectory variation broaden represented states, behaviors, interaction conditions, and state-action distributions.Trajectory diversity includes differences in initial and terminal states, motion paths, contact sequences, and execution speeds.
  • Data characteristics: Real-robot datasets directly capture sensing noise, control latency, contact dynamics, hardware constraints, and executable platform-specific actions.This physical grounding supports learning reliable manipulation policies and recovery behaviors.
  • Limitations and future directions: High collection costs, hardware and safety requirements, environment resets, maintenance, and embodiment-specific representations restrict scale and complicate aggregation across robots, sensors, coordinate systems, and action formats.Future directions include simpler teleoperation, AIGC-based embodiment-specific augmentation, policy rollouts, and human interventions that convert failures into recovery trajectories.

3 UMI Data

UMI data uses portable, robot-independent interfaces to collect synchronized visual observations and end-effector actions from natural human demonstrations. Its scalability, diversity, and cross-embodiment potential come with tracking, calibration, embodiment-gap, and contact-feedback limitations.

  • Data collection and representation: Portable, robot-free UMI interfaces capture synchronized observations, six-degree-of-freedom gripper poses, and gripper states during natural human demonstrations.Typical systems combine cameras, a handheld gripper, pose tracking, an IMU, and visual–inertial SLAM.
  • Evolution of UMI systems: UMI has expanded from single-arm visual trajectories to large-scale, multimodal, and multi-embodiment data spanning mobile manipulators, humanoids, bimanual systems, dexterous hands, 3D views, and tactile or force sensing.FastUMI-100K exemplifies increased collection scale, while later systems add viewpoints and contact-rich measurements.
  • Cross-embodiment deployment: End-effector-centric relative trajectories separate task representation from robot-specific execution, enabling retargeting across different arms, mobile manipulators, and dexterous hands.Cross-embodiment transfer commonly predicts relative six-degree-of-freedom trajectories and gripper commands rather than joint-space actions.
  • Advantages: UMI’s portable collection enables scalable, low-cost, in-the-wild demonstrations across diverse environments while reducing dependence on a specific robot embodiment.Human operators can collect smooth real-world manipulation demonstrations, and end-effector-centric actions support reuse across robot platforms.
  • Limitations: UMI remains limited by embodiment discrepancies, fragile tracking and calibration, missing robot-specific proprioception and actuator dynamics, and insufficient force feedback for contact-rich tasks.Occlusion, rapid motion, visually challenging environments, and the difficulty of validating trajectories without robot execution can reduce reliability.

4 Egocentric and Exocentric Data

Egocentric and exocentric data provide authentic real-world observations and physical interactions, occupying a middle layer between simulation scalability and robot-aligned data. Their value can be expanded through multimodal sensing, structured reconstruction, and retargeting, but human recordings remain only partially observable and lack directly executable robot experience.

  • Data role: Egocentric and exocentric data offer more authentic real-world observations and physical interactions than simulation, but less direct alignment with robot execution than UMI data.Egocentric capture commonly follows the wearer, while exocentric systems add environmental viewpoints and scene context.
  • Supervision construction: Recorded streams gain semantic, geometric, multimodal, and robot-oriented supervision through annotation, pose reconstruction, sensor fusion, and human-to-robot retargeting.Examples include narrated action segments, calibrated hand and object poses, SLAM trajectories, inverse-kinematic mapping, and aligned robot demonstrations.
  • Capture systems: Wearable and environmental systems jointly measure appearance, geometry, motion, attention, muscle activation, and physical interaction.Visual sensors capture images and depth; tracking systems measure kinematics; auxiliary sensors expose gaze, muscle activity, and contact.
  • Limitations: Egocentric data remain limited by partial observability, hand-object occlusion, rapid camera motion, motion blur, device-dependent viewpoints, and uneven availability of dense supervision.Additional hardware, calibration, synchronization, and reconstruction are required for dense geometric and multimodal labels.
  • Robot alignment: Human recordings lack robot proprioception, actuator dynamics, and directly executable actions, so they are not equivalent to robot experience.This fundamental mismatch persists even when human motion is mapped into robot-compatible action spaces.

5 Simulation Data · 5.1 Data Overview · 5.2 Simulation Infrastructure and Asset Ecosystems

Simulation data complements costly real-robot collection by providing repeatable interactions, privileged labels, and low-cost trajectories, while recent datasets broaden task horizons and embodiment coverage. Its ecosystem couples robot-sensor systems, object and scene assets, and physics-rendering backends, with scalable asset generation requiring manipulation-oriented annotations.

  • 5.1 Data Overview: Simulation alleviates real-robot data bottlenecks through repeatable conditions, privileged states and labels, and low-cost trajectory generation.Real-robot collection remains constrained by cost, safety, hardware availability, and annotation difficulty.
  • 5.1 Data Overview: Simulated datasets have progressed from constrained rigid-body manipulation to longer-horizon, compositional, household, bimanual, and humanoid manipulation settings.Data-collection pipelines increasingly use motion planning and RL experts, demonstration augmentation, digital-twin reconstruction, and LLM-assisted synthesis.
  • 5.2 Simulation Infrastructure and Asset Ecosystems: Simulation infrastructure comprises three coupled components: embodiment-sensor systems, object and scene assets, and physics and rendering backends.Embodiment-sensor systems specify robot morphology, kinematics, actuation, controller interfaces, and physically associated sensing channels.
  • 5.2.1 Robotic Embodiments and Sensors: Robot assets are instantiated as controllable articulated models whose descriptions specify kinematics, limits, inertial properties, geometries, actuators, and sensor attachments.Stationary single-arm manipulators are most common, alongside dexterous hands, mobile manipulators, and dual-arm or bimanual systems.
  • 5.2.2 Object and Scene Assets: Object and scene assets determine category coverage, geometric variation, texture realism, and affordance structure, drawing on CAD, web-scale, scanned, and product-grounded resources.Usable assets additionally require collision geometry, consistent scale, stable poses, physical parameters, and relevant grasp or articulation annotations.
  • 5.2.2 Object and Scene Assets: ManiTwin provides over 100K AIGC-generated rigid-body objects with manipulation-oriented annotations, showing that asset scaling must be coupled with operable annotations for automated synthesis.Other resources extend assets with mobility, multisensory, physics-aware, or simulation-ready properties.
  • 5.2.3 Physics and Rendering Backends: Physics and rendering backends govern contact dynamics, constraints, numerical stability, parallelization, and visual realism, thereby shaping feasible simulated data distributions.MuJoCo supports efficient rigid-body and contact modeling, GPU-oriented systems emphasize parallel simulation, and specialized engines target deformable, multimodal, accelerator-friendly, multi-physics, or differentiable regimes.

5.3 Benchmarks and Simulation Datasets · 5.4 Synthetic Demonstration Generation

Section 5.3 surveys simulation benchmarks and datasets spanning diverse embodiments and interaction settings, while noting persistent gaps in tactile and deformable-object support. Section 5.4 organizes synthetic demonstration generation into human-executed, rule-based, trajectory-reuse, and autonomous or generative approaches, each trading scalability against behavioral coverage and validity.

  • 5.3 Benchmarks and Simulation Datasets: Simulation benchmarks span single-arm, dual-arm, dexterous, mobile, and humanoid settings, but rigid-object manipulation remains dominant while tactile and deformable interactions are limited.‘S,’ ‘D,’ and ‘H’ denote single-arm, dual-arm, and humanoid embodiments; dexterous-hand and mobile capabilities are reported separately.
  • 5.3 Benchmarks and Simulation Datasets: Large-scale simulation resources vary in volume, task coverage, embodiment diversity, and manipulation settings, increasingly including multiple embodiments and hundreds of thousands or millions of trajectories.Grasp-centric resources also provide substantially large collections of grasp and dexterous-hand samples.
  • 5.4 Synthetic Demonstration Generation: Human-executed demonstrations use teleoperation, VR control, motion-capture retargeting, or play interaction to record synchronized observations, states, actions, language, object states, and success labels.Here, simulation functions primarily as an instrumented recording environment.
  • 5.4 Synthetic Demonstration Generation: Rule-based execution decomposes tasks into geometric stages such as target selection, grasp computation, approach, interaction, and completion checking using privileged experts or planners.It improves reproducibility and reduces marginal episode-collection cost, but its behavior distribution is bounded by the specified expert’s expressiveness.
  • 5.4 Synthetic Demonstration Generation: Trajectory playback and demonstration expansion transform existing trajectories through replay, spatial changes, segmentation, stitching, or retargeting, reducing human effort while retaining seed motion structure and strategies.Coverage remains conditioned on the diversity and quality of the source trajectories and their compatibility with new configurations.
  • 5.4 Synthetic Demonstration Generation: Autonomous and generative rollouts use learned policies, reinforcement learning, task programs, scene-graph solvers, skill libraries, world models, and scalable task, asset, skill, and validation pipelines.GenSim, GenSim2, and RoboGen exemplify systems that generate tasks, scenes, code, or supervision before producing demonstrations with planning or reinforcement-learning solvers.
  • 5.4 Synthetic Demonstration Generation: Synthetic generation increasingly extends beyond RGB and proprioception to grasping, dexterous manipulation, and visuotactile signals such as deformation and marker displacement fields.Representative resources include GraspVLA/SynGrasp-1B, Dex1B, and visuotactile simulation frameworks such as Taxim.
  • 5.4 Synthetic Demonstration Generation: World-model approaches such as DreamGen generate robot videos or neural trajectories from visual states and language, then recover pseudo-actions, reducing manual collection while risking invalidity and visual-action misalignment.The passage also identifies semantic inconsistency as an additional risk.

5.5 World Models as Simulators · 5.6 Sim-to-Real Gap · 5.7 Advantages and Limitations

World models are becoming learned simulators for policy training, evaluation, and synthetic interaction-data generation, offering scalable alternatives or complements to physics-based simulation. Simulation remains cost-efficient and richly supervised, but sim-to-real transfer is limited by observation, interaction, and physical-fidelity gaps, while learned simulators require calibration against real-world behavior.

  • 5.5 World Models as Simulators: World models learn predictive dynamics from observations and interactions, generating future observations, latent states, rewards, or task outcomes conditioned on histories, tasks, and actions.They replace manually specified assets, dynamics, and rendering pipelines with learned predictive environment models.
  • 5.5.1 World Models for Policy Training and Post-Training: Imagined transitions and estimated rewards let policies be optimized, adapted, or post-trained in virtual environments without repeatedly interacting with physical robots.These methods can reduce real-world interaction costs and generate experience in parallel, provided learned simulators preserve causal action effects.
  • 5.5.2 World Models for Policy Evaluation: Learned rollouts can evaluate policy progress, completion, failure, and unsafe behavior for checkpoint selection, but visual realism alone is insufficient for reliable policy evaluation.WorldGym, WorldEval, and Veo-based evaluation examine nominal, out-of-distribution, and safety-related behaviors.
  • 5.5.3 World Models as Synthetic Data Engines: World models synthesize interaction data across scenes, instructions, actions, viewpoints, and embodiments while modeling temporal evolution and action consequences.DreamGen generates robot videos and pseudoactions, while GigaWorld-0 treats generative world models as embodied-learning data engines.
  • 5.5.3 World Models as Synthetic Data Engines: Learned simulation scales policy optimization, evaluation, and data generation, but prediction errors, action ambiguity, exploitation, and distribution shift can yield plausible yet physically invalid trajectories.Filtering, uncertainty estimation, and calibration against real-world behavior are therefore required.
  • 5.6 Sim-to-Real Gap: The sim-to-real gap consists primarily of observation mismatch and interaction mismatch, with the latter including distinct kinematic and dynamic gaps.Observation differences include texture, illumination, depth noise, calibration, occlusion, tactile response, and force signals; kinematic differences can often be reduced through alignment and hardware-consistent control.
  • 5.7 Advantages and Limitations: Simulation provides scalable, low-cost, parallelized, and safer supervision with privileged labels such as poses, segmentation, contact states, physical parameters, success signals, and dense rewards.It also avoids hardware wear and supports controllable virtual-environment data collection.
  • 5.7 Advantages and Limitations: Limited physical fidelity and constrained diversity weaken simulation data, especially for contact-rich manipulation, deformable objects, reflective materials, liquids, and long-horizon tasks.Policies must still internalize physical regularities that imperfect simulator dynamics cannot fully recover, particularly for contact, materials, and object interactions.

6 General Data

General data provide scalable, diverse, embodiment-agnostic semantic, perceptual, and reasoning priors that complement robot trajectories. However, they offer weak action and contact grounding, so robot-specific data must connect broad knowledge to executable behavior.

  • Data contributions: General data span vision-language, segmentation and localization, video, spatial and 3D perception, and planning, contributing complementary cognitive capabilities.These sources support semantic knowledge, spatial grounding, temporal and procedural understanding, geometric awareness, and executable-step organization.
  • Vision-language data: Vision-language corpora establish multimodal alignment, scene understanding, commonsense reasoning, task comprehension, and question answering, including under noisy instructions.They expose models to diverse scenes, activities, tools, materials, and linguistic expressions beyond the limited coverage of robot trajectories.
  • Spatial grounding: General perception annotations require robot-specific supplementation because robotic scenes contain occlusion, repeated categories, unusual viewpoints, motion blur, and partially visible functional parts.Robust systems combine large-scale general segmentation data with smaller task-relevant robot datasets and relation-aware annotation.
  • Data contributions: Web, video, and benchmark resources provide scale, diversity, accessibility, low marginal cost, and reuse across embodiments, platforms, and tasks.They require neither hardware, teleoperation, nor environment resets, making the corpora embodiment-agnostic.
  • Limitations: General data lack proprioception, actuator dynamics, contact forces, and physical consequences of actions, limiting their ability to specify robot behavior independently.Automatically generated annotations may also introduce hallucinated objects, incorrect relations, generic language, or physically invalid plans.

7 Data Applications in Embodied Foundation Models · 7.1 Overview

This section shifts from embodied data construction to data application, examining how different data types train embodied foundation models. It first summarizes data recipes and action representations, then discusses embodied brain, vision-language-action, and world action models.

  • 7.1 Overview: The section transitions from data construction to the application of embodied data in foundation-model training.It examines how different types of embodied data are used to train these models.
  • 7.1 Overview: It first summarizes the data recipes adopted by representative embodied foundation-model architectures.
  • 7.1 Overview: It also summarizes the action representations adopted by representative model architectures.
  • 7.1 Overview: The discussion covers three major model families: embodied brain models, vision-language-action models, and world action models.The passage abbreviates vision-language-action models as VLAs and world action models as WAMs.
  • 7.1 Overview: Embodied brain models are treated as one of the section’s three major model families.
  • 7.1 Overview: Vision-language-action models and world action models are treated alongside embodied brain models.The passage identifies these families as VLAs and WAMs, respectively.

7.2 Data Composition and Action Representation

Embodied foundation-model data recipes are shifting from single-source robot demonstrations toward heterogeneous, larger-scale mixtures, with growing use of egocentric data while the optimal composition remains unresolved. Action heterogeneity is addressed through structural and geometric representations that align control interfaces and coordinate conventions across embodiments.

  • Data composition: Successive model generations increasingly combine real-robot, general-purpose, egocentric, UMI, and simulation data rather than relying on a single source.The π series progresses from real-robot data in π0 to adding general-purpose data in π0.5 and egocentric data in π0.7; LingbotVA combines real-robot, UMI, and simulation data.
  • Data composition: The optimal data recipe remains unresolved, since robot-focused models can perform strongly and current evidence does not show that adding more sources is inherently preferable.Different sources may contribute complementary capabilities, including broad visual-semantic reasoning, cross-embodiment coverage, and physical interaction knowledge.
  • Data scale: Approximately 38,100 hours: Qwen-RobotManip constructs a multi-source corpus containing roughly 11.4K hours of open-source robot data and 24,808 hours of synthesized robot-compatible trajectories.This illustrates the rapid growth of pretraining scale relative to earlier robot-demonstration-centric pipelines.
  • Egocentric data: 20,854 hours: EgoScale scales action-labeled egocentric pretraining and reports consistent improvements across subsets ranging from 1,000 to 20,000 hours.Other studies use egocentric data as their primary or exclusive pretraining source; HumanScale uses up to 5,000 hours of curated egocentric video.
  • Action representation: Cross-embodiment action heterogeneity is handled through embodiment-specific projection, fixed-dimensional zero-padded interfaces, or semantic action slots.These strategies respectively share representations while preserving native interfaces, provide uniform tensors with validity masks, or assign consistent physical meanings to shared dimensions; Qwen-RobotManip uses an 80-dimensional canonical state-action vector.
  • Action representation: Geometric action representations may be robot-centric, camera-centric, or wrist-centric, but controlled comparisons have not established a coordinate convention that consistently improves transfer or control.Implementations should specify frame origin and axes, handedness, tool-center point, action mode, rotation parameterization, and physical units.

7.3 Applications for Embodied Brain

Embodied brain models use vision-language pretraining for physical-world understanding, emphasizing perception, physical reasoning, grounding, memory, and planning rather than direct action generation. Their data recipes combine action-free multimodal and video data with transferable supervision recovered from interaction signals, bridging physical understanding and executable behavior.

  • Model focus: Embodied brain models target spatiotemporal perception, physical reasoning, grounding, temporal memory, and task planning rather than direct action generation or dynamics prediction.Representative models include RynnBrain, HY-Embodied, and Pelican-VL.
  • Training data: General multimodal data provide semantic understanding, visual commonsense, OCR, memory, and spatial reasoning, while interaction data add affordance, grasp, trajectory, and planning supervision.Supervision need not contain low-level robot actions if it contributes to physically meaningful representations.
  • Action-free data: Action-free image-text, video-text, VQA, OCR, spatiotemporal QA, instruction-following, and re-annotated egocentric data provide supervision for recognition, language understanding, visual commonsense, and temporal reasoning.Robot-centric video objectives can additionally learn object permanence, motion patterns, hand-object interactions, state transitions, and coarse physical dynamics.
  • Action-free data: Video-only pretraining yields primarily observational knowledge, so executable control still requires action-conditioned trajectories, robot states, inverse-dynamics supervision, or policy adaptation.Action-free data do not necessarily identify causal actions or embodiment-specific feasibility.
  • Action-labeled data: Action-labeled data are converted into transferable affordance, grasp, pose, trajectory, and planning supervision, linking physical understanding with executable behavior.Real-robot data offer embodiment-aligned traces, handheld-gripper data improve scalability, and simulation supplies object states, contacts, success conditions, and failure labels.

7.4 Applications for Vision-Language-Action Models

Vision-language-action models extend vision-language understanding into executable robot behavior by grounding scenes, planning actions, predicting action chunks, and adapting to embodiments. Their training increasingly combines action-labeled data with continuous control methods, egocentric human videos, and action-free videos that provide proxy or intermediate supervision.

  • VLA capabilities: VLA models translate multimodal representations into executable behaviors, including scene understanding, action-sequence planning, low-level action-chunk prediction, and embodiment-specific adaptation.They must recognize task-relevant objects, states, and affordances while grounding decisions in spatial and temporal context.
  • Action supervision: Action-labeled data directly supervise mappings from visual-language observations to robot behaviors, while action tokens enable autoregressive generation alongside language tokens.This formulation reuses pretrained vision-language backbones and their acquired semantic knowledge.
  • Action supervision: Continuous action heads address discrete-token limitations in high-frequency, fine-grained, and dexterous manipulation, with diffusion policies denoising continuous action sequences.Small continuous-control errors can substantially affect execution, motivating expressive modeling of high-dimensional and potentially multimodal actions.
  • Additional supervision: Egocentric human videos with hand poses provide scalable dexterous-learning supervision through motion reconstruction, robot-hand retargeting, embodiment-gap reduction, and shared human-robot action spaces.Approaches use hand inpainting, robot-hand rendering, wrist poses, fingertip poses, and MANO parameters.
  • Additional supervision: Action-free videos support VLA learning through latent actions, geometric motion representations, and hierarchical supervision for task decomposition, subgoal prediction, and reasoning.These proxies describe action-relevant state transitions or visual dynamics but are not assumed to be executable robot actions.

7.5 Applications for World Action Models

World Action Models learn how physical worlds evolve under actions, enabling learned simulation, planning, policy evaluation, and synthetic interaction generation. Their data and objectives combine action-labeled trajectories for executable control with action-free observations for broad world-dynamics priors.

  • WAM Role: WAMs model physical-world dynamics and action effects, unlike embodied VLMs focused on perception and reasoning or VLAs mapping observations directly to behaviors.They can serve as learned simulators, planners, policy evaluators, or generators of synthetic interaction data.
  • Data Sources: WAM training data split into action-labeled trajectories that specify transition-causing actions and action-free multimodal or egocentric videos that lack explicit action variables.Action-labeled sources include real-robot, UMI-style, and simulation data; action-free sources include general multimodal, egocentric, and exocentric data.
  • Action Modeling: Action supervision follows two paradigms: continuous action denoising with diffusion or flow matching, and discrete action-token prediction with autoregressive objectives.Both paradigms align predicted future world evolution with executable robot control; WorldVLA uses autoregressive token prediction, while Genie Envisioner uses a flow-matching action expert.
  • Action-Free Learning: Action-free data teach how worlds typically evolve through observation-only prediction of future frames, visual latents, scene states, or language descriptions.Egocentric datasets additionally expose first-person interaction, hand-object occlusion, viewpoint changes, object-state transitions, and long-horizon task progress.
  • Data Pyramid Feedback: Action-free data provide scalable, diverse observational priors, while higher pyramid levels add action grounding; trained WAMs generate synthetic interaction data and replace physical environments for policy evaluation.Embodied training recipes are becoming more heterogeneous and larger-scale, but the optimal data mixture remains unresolved because source contributions are not systematically isolated.

8 Challenges and Future Directions

Future directions for robot-learning data concern what to collect, how to acquire it efficiently, and how to integrate heterogeneous sources. Key priorities include tactile sensing, failure and recovery data, scalable collection, cross-embodiment action alignment, egocentric-to-robot transfer, and principled data recipes.

  • How to collect: Scalable, adaptive pipelines should reduce costly manual teleoperation, broaden task and scene coverage, and target underrepresented, uncertain, or behaviorally informative data regions.This reframes collection around informational coverage and acquisition efficiency rather than dataset size alone.
  • What to collect: Tactile sensing remains a missing contact layer because visual observations provide only indirect evidence of contact force, slip, and material properties.Existing datasets center on RGB-D, proprioception, language, and action trajectories, while tactile sensing is not systematically incorporated.
  • What to collect: Failure and recovery trajectories are needed because success-centric datasets provide limited supervision for recognizing, diagnosing, and correcting execution errors.Future datasets should represent heterogeneous failures and their temporal and causal structures rather than using only binary success labels.
  • How to use: Cross-embodiment learning requires action alignment beyond a common format, using transferable interfaces such as Cartesian end-effector poses and canonical reference frames when transformations are reliable.End-effector actions may be represented relative to the robot base, camera, end-effector, or world frame.
  • How to use: Egocentric human data offers manipulation priors, but transfer remains difficult because human and robotic hands differ in kinematics, morphology, sensing, compliance, actuation, friction, and force limits.The paper frames egocentric data as useful for dexterous-manipulation priors while leaving optimal robot-policy use open.
  • How to use: No principled data recipe has yet established optimal proportions or compute-matched effects for combining real-robot, simulation, egocentric, UMI-style, and general multimodal data.Recent models increasingly mix pyramid layers, but source combinations, sampling ratios, and stage-wise allocations differ substantially.

9 Conclusion

The paper synthesizes embodied intelligence through a data pyramid organized by scalability and robot alignment, while using quality, diversity, reusability, and physical fidelity as complementary dimensions. It analyzes foundation-model data recipes and proposes a roadmap for selecting, collecting, combining, and evaluating data across capabilities, embodiments, and deployment settings.

  • Data pyramid: The embodied data pyramid spans five categories with distinct strengths and limitations, organized primarily by scalability and robot alignment.Quality, diversity, reusability, and physical fidelity provide complementary dimensions for characterizing the data ecosystem.
  • Data pyramid: Simulation data supports controllable, parallel generation of robot-oriented experience with privileged supervision, but remains constrained by environment coverage and the sim-to-real gap.General multimodal data scales semantic, perceptual, and reasoning priors, yet offers limited grounding in robot actions, contacts, and physical outcomes.
  • Data recipes: The paper analyzes embodied foundation models through data recipes, including action representations and their implications for data compatibility and cross-embodiment transfer.It examines heterogeneous data use across embodied brain, vision-language-action, and world-action models.
  • Data recipes: These model families use heterogeneous data to support reasoning and planning, executable control, and predictive world modeling, respectively.The mapping follows embodied brain models, vision-language-action models, and world-action models in that order.
  • Open challenges and roadmap: The paper identifies open challenges in data quality assessment, multimodal and action-space standardization, and cross-embodiment reuse, and offers a forward-looking roadmap for embodied data practice.The roadmap addresses selecting, collecting, combining, and evaluating data for different capabilities, embodiments, and deployment settings.
Loading 2607.24744v2…