Source-linked AI summary
Robots Need More than VLA and World Models
Elis Karcini, Faisal Mehrban, Quang Nguyen, Mac Schwager, Arash Ajoudani, Cesar Cadena, Jan Peters, Marco Hutter, Haitham Bou-Ammar
TL;DR
The paper argues that generalist robotics is constrained not only by policy scaling, but by the lack of mechanisms that turn broad, messy physical experience into grounded robot supervision. It proposes a larger physical-intelligence stack—combining data, embodiment, world-model, and deployment interfaces—and concludes that robotics must move toward world-scale physical supervision.
Problem
Generalist robotics lacks scalable mechanisms for converting heterogeneous physical experience into supervision that robot policies can use.
Method
The paper develops a position and research agenda centered on physical data engines, embodied autolabelling, task-preserving retargeting, and physics-grounded world models.
Results
The paper concludes that VLAs are only one layer of a broader physical-intelligence stack whose effectiveness depends on upstream and downstream grounding mechanisms.
Takeaways & Limitations
Robotics should shift from robot-native datasets and isolated policies toward systems that learn from world-scale behavioural data and continual physical deployment.
Takeaways & Limitations
The proposed physical data engine must integrate sources with partial supervision, imperfect realism, or missing actions, task labels, and embodiment information.
Abstract
from arXiv · showhide
Generalist robot intelligence is often framed as a policy-scaling problem: collect more robot demonstrations, train larger Vision-Language-Action (VLA) models, and expect broader generalisation. In this position paper, we argue that this framing is incomplete. The central bottleneck is not only policy learning, but the absence of mechanisms that convert the world's abundant unstructured behavioural data into grounded robot supervision. Human motion, internet video, simulation rollouts, and interactive demonstrations contain rich information about tasks, goals, contacts, failures, and physical constraints, yet most of this information is not directly usable by robot policies because it lacks embodiment-specific action labels, task semantics, and reward structure. We identify four missing components for the next generation of robotics: data interfaces for autolabelling unstructured behaviour, embodiment interfaces for retargeting human motion to robot actions, world-model interfaces for physics-grounded 3D reasoning, and reward interfaces for inferring task progress and success from video and language. We survey recent progress in robot foundation models, cross-embodiment datasets, learning from video, world models, and reward modelling, and propose a research agenda for building robotics systems that can learn not only from robot demonstrations, but from the broader physical world.
1 Introduction
Robotics has advanced through larger models and broader datasets, but its path to generalist intelligence remains unclear because physical experience is not naturally packaged as robot supervision. The paper argues for a grounding-centric pipeline that converts diverse physical experience into robot-usable supervision.
- The grounding bottleneck: Physical interaction produces abundant behavioural data, but unlike text and images, most of it is not directly usable by robots.The paper contrasts naturally digitised, densely supervised modalities with physical experience that lacks suitable grounding.
- The grounding bottleneck: Current progress scales robot datasets, pools experience across embodiments, and trains generalist VLA policies, yet still relies on costly robot demonstrations, curated datasets, or embodiment-specific actions.These dependencies make current progress expensive and difficult to scale.
- A grounding-centric pipeline: The paper surveys methods that widen supervision through heterogeneous robot trajectories, human and internet video, action-conditioned world models, and simulation-generated counterfactual interaction data.Its organising principle is the robotics learning pipeline rather than only data source or algorithmic family.
- A grounding-centric pipeline: Robot learning should shift from a robot-data-centric loop toward grounding mechanisms that produce robot-usable actions, contacts, object states, task phases, goals, and rewards.The proposed pipeline begins with broad physical experience, including human motion, video, simulation, tactile sensing, robot interaction, and language.
- A grounding-centric pipeline: The central claim is that robotics needs components transforming physical experience into robot-usable supervision, not another policy architecture alone.This reframes the field's bottleneck around supervision grounding rather than policy scaling in isolation.
- A grounding-centric pipeline: The paper identifies four missing pillars: embodied autolabelling, task-preserving cross-embodiment retargeting, physics-grounded consequence prediction, and task-conditioned reward grounding.These components are presented as mechanisms for making the world's physical experience learnable by robots.
2 Robot-Native Supervision: Progress and Scaling Limits
Robot-native supervision has enabled increasingly generalist policies through larger, more diverse datasets and improved VLA architectures, but scaling remains constrained by the need for grounded robot-labelled experience. Simulation and world models can expand experience, yet their value still depends on preserving physical variables relevant to control.
- Scaling Limits: The central scaling limit is converting broader physical experience into robot-usable actions, contacts, object states, task phases, goals, and rewards.Current results remain dependent on supervision already expressed in the coordinate system of robot learning.
- Robot-Native Supervision: Robot-native supervision pairs observations with embodiment-specific actions, task labels, language instructions, rewards, or success signals.It may include camera images, proprioception, end-effector poses, tactile readings, and force-torque measurements.
- Scaling Progress: Cross-embodiment datasets and generalist policies demonstrate that scaling across tasks, environments, and robot bodies can improve robot behaviour.Open X-Embodiment and RT-X pooled more than one million trajectories from 22 embodiments, while Octo pretrained on 800,000 trajectories.
- Scaling Progress: VLA systems map observations and textual descriptions to robotic actions, extending robot learning with multimodal reasoning and web-scale vision-language pretraining.RT-2 transferred semantic knowledge from internet-scale pretraining into robotic control by representing robot actions as tokens.
- Scaling Limits: Simulation transfers to physical control only when it captures or randomises the physical factors that matter.This requirement limits the reliability of cheaply generated experience despite its scalability and safety advantages.
- Scaling Limits: Generated experience is useful only when it preserves geometry, object state, contact, dynamics, embodiment constraints, task progress, and success or failure.Visually plausible rollouts that ignore contact, force, friction, or stability are not yet reliable robot supervision.
3 The Missing Components for Physical Intelligence
The paper proposes four components that turn broad physical experience into grounded robot intelligence: structured data extraction, task-preserving retargeting, physics-grounded prediction, and task-conditioned reward feedback. Together, these components connect observation, action, consequence prediction, and deployment learning beyond policy scaling alone.
- 3.1 Physical Data Engines and Embodied Autolabelling: A physical data engine converts heterogeneous experience into object states, contacts, task phases, latent actions, goals, rewards, and outcome labels.It treats human motion, video, sensing, failures, and deployment traces as partially labelled interaction data rather than raw unstructured inputs.
- 3.2 Task-preserving Retargeting across Embodiments: Task-preserving retargeting maps latent physical actions or human demonstrations to executable robot actions while preserving their intended effects on the world.The relevant invariant may be pose, contact, object-state transition, or task intent; matching human joint trajectories alone is insufficient.
- 3.3 Beyond Physics-Grounded World Models for Consequence Predictions: Physics-grounded world models generate task-relevant counterfactuals by predicting consequences under alternative actions, bodies, contacts, and goals.Prediction should prioritize task-relevant effects, such as drawer displacement or handle contact, rather than equally modelling every visual detail.
- 3.4 Self-Improving Deployment Loops: Task-conditioned reward grounding interprets physical states relative to goals so deployment outcomes can provide structured progress, success, failure, and correction signals.The same state may be successful for one goal and unsuccessful or irrelevant for another, motivating reward functions conditioned on task context.
- 3 The Missing Components for Physical Intelligence: The four components form a feedback loop in which outcomes update reward, world, retargeting, and policy models before redeployment.The loop turns failed rollouts, successful executions, and human corrections into grounded supervision rather than isolated episode records.
4 Conclusions
Generalist robotics should move beyond policy scaling by building grounding mechanisms that convert broad, weakly labelled physical experience into robot-usable supervision. The paper identifies a system-level path spanning data, embodiment, world-model, and deployment interfaces.
- The central bottleneck is converting broad, messy, weakly labelled physical experience into supervision that robots can use.Available experience includes human behaviour, internet video, simulations, tactile traces, failures, and corrections, but it often lacks robot actions, task labels, contacts, phases, and rewards.
- VLAs are one layer in a larger physical-intelligence stack whose effectiveness depends on upstream grounding and downstream interpretation of deployment outcomes.The required capabilities include autolabelling heterogeneous behaviour, retargeting physical effects across embodiments, predicting action consequences, and relating outcomes to attempted tasks.
- Generalist robotics should evaluate whether systems can convert weaker sources of physical experience into useful supervision, not only whether larger policies solve more tasks.The paper proposes questions about inferring contacts, object-state changes, and task phases; retargeting effects; predicting relevant consequences; and interpreting success and failure.
- A compounding system could turn heterogeneous experience into structured supervision, map task-relevant effects to robot actions, generate actionable counterfactuals, and learn from deployment outcomes.The proposed components are a physical data engine, embodiment interface, physics-grounded world model, and task-conditioned deployment loop.
- Progress depends on moving from robot-native datasets to world-scale physical supervision and from isolated policies to systems that learn from the physical world itself.This conclusion reframes advancement as scaling grounding mechanisms alongside policies.