Source-linked AI summary
Unifying Embodied Foundation Models via a Model-to-Policy Formulation
Xinyi Hong, Mingyang Ren
TL;DR
Robot learning requires task-conditioned policies whose actions shape their own subsequent observations, while embodied models differ in architectures, outputs, and objectives. The paper formulates these models with conditional output distributions and action-realization kernels, linking pre-training and post-training to closed-loop policy deployment. It presents the formulation as a common probabilistic language for embodied models with heterogeneous architectures, output representations, and learning objectives.
Problem
Robot policies must act on trajectories induced by their own actions, while embodied models use differing architectures, output representations, and learning objectives.
Source: §1 · p. 2Method
Results
The formulation provides a common probabilistic language for embodied models with different architectures, output representations, and learning objectives.
Source: §6 · p. 8Takeaways & Limitations
The framework clarifies how pre-training and post-training shape policies deployed in closed-loop interaction.
Source: §6 · p. 8
Abstract
from arXiv · showhide
Vision-language-action (VLA) models and world-action models (WAMs) have emerged as promising approaches to robot learning and control. These models integrate multimodal perception and action generation, but differ in their architectures, output representations, and learning objectives. In this paper, we present a unified, architecture-agnostic probabilistic formulation for embodied foundation models based on stochastic kernels. Our formulation has two key components: (i) we characterize embodied foundation models through conditional output distributions and action-realization kernels, which together induce task-conditioned robot policies; and (ii) we connect pretraining and post-training to policy deployment, accommodating model-specific pretraining objectives and offline or online post-training strategies. We instantiate the formulation for OpenVLA and DreamZero, covering autoregressive action-token prediction and joint video-action generation.
1 Introduction
Embodied foundation models must induce task-conditioned policies, not merely fit fixed prediction distributions, because their actions shape the trajectories they later encounter. The paper introduces a stochastic-kernel formulation connecting model outputs and action realization with training and closed-loop deployment.
1 Introduction
- Robot policies must perform well on trajectories induced by their own actions, unlike models that map inputs to predictions or generated content.The introduction frames robot learning as inducing task-conditioned distributions over actions from interaction history and task conditions.Source: §1 · p. 2
- Collecting real robot trajectories across diverse tasks, environments, and embodiments constrains scaling robot pre-training, while video pre-training offers a complementary source of motion and scene dynamics.The paper distinguishes these video-learned physical-interaction priors from visual realism alone.Source: §1 · p. 2
- The paper develops an architecture-agnostic probabilistic formulation based on stochastic kernels and instantiates it for OpenVLA and DreamZero.It connects model output distributions and action realization with pre-training, post-training, and closed-loop robot interaction.Source: §1 · p. 2
2 Task-Conditioned Robot Interaction Setting
The section models robot interaction as repeated task-conditioned action selection from partial trajectories. Each selected action changes the environment and supplies the next observation until the finite-horizon trajectory is complete.
2 Task-Conditioned Robot Interaction Setting
- At each step, the robot samples an action from a policy conditioned on its partial interaction history and task condition.The history contains observations and prior actions, beginning with the initial observation.Source: §2 · p. 2
- After executing an action, the robot receives a new observation and repeats action selection through T steps.The resulting trajectory contains the initial observation, all actions, and the final observation.Source: §2 · p. 2
- The policy maps partial trajectories and task conditions to probability distributions over actions.The paper denotes the observation space, action space, task-condition space, and partial-trajectory space by O, A, C, and T.Source: §2 · p. 2
3 Unified Formulation for Embodied Foundation Models
The formulation represents an embodied model as a conditional distribution over outputs and pairs it with an action-realization kernel to induce a deployed policy. It accommodates model-specific training and different output forms, including decoded action tokens and joint video-action chunks.
3 Unified Formulation for Embodied Foundation Models
- The conditional output distribution and action-realization kernel together specify how a model induces a task-conditioned policy.The model samples an output given interaction history and task condition, then the kernel maps that output to an executable action.Source: §3 · p. 3
- The output space can represent decoded action tokens, action chunks, latent or video plans, or joint video-action chunks, each paired with an appropriate action-realization procedure.For joint video-action chunks, the kernel selects executable actions from the available chunk and can incorporate scheduling and smoothing.Source: §3 · p. 3
- Pre-training estimates model parameters by minimizing an empirical loss whose per-example objective is specified by the chosen pre-training method.The formulation leaves the loss and training examples specific to the selected method.Source: §3 · p. 3
- During post-training, αk ∈ [0, 1] controls the relative weighting of offline and online losses: αk = 0 uses only offline data, αk = 1 uses only online data, and αk ∈ (0, 1) combines both.Source: §3 · p. 4
- At deployment, sampling model outputs and realizing actions through the kernel implements the post-trained policy, with deterministic or stochastic realization.The distinction separates deterministic decoding from stochastic sampling.Source: §3 · p. 4
4 Vision-Language-Action (VLA) Model
OpenVLA is framed as a vision-language model that predicts action-token sequences and converts them into robot commands. Its pre-training and downstream supervised fine-tuning use teacher-forced action-token cross-entropy, while deployment uses greedy decoding and deterministic detokenization.
4 Vision-Language-Action (VLA) Model
- OpenVLA conditions on the current image and language instruction, producing action tokens that are detokenized into a seven-dimensional robot action.Its model combines DINOv2, SigLIP, an MLP projector, and Llama 2 7B; the action tokens are generated autoregressively.Source: §4 · pp. 4–5
- OpenVLA pre-training further optimizes a pretrained vision-language model on Open X-Embodiment robot demonstrations using normalized, discretized action-token targets.Teacher forcing supervises the seven action tokens with cross-entropy.Source: §4 · pp. 5–6
- Downstream supervised fine-tuning uses the same teacher-forced action-token cross-entropy objective on offline robot demonstrations.The paper sets αk = 0 because this stage uses offline data only.Source: §4 · p. 6
- At deployment, greedy token decoding followed by action detokenization makes OpenVLA's output and action realization deterministic.The generated token sequence is converted into the executable action command.Source: §4 · p. 6
5 World Action Models (WAMs)
World-action models jointly predict future observations and robot actions, unlike VLAs that primarily predict actions. DreamZero instantiates this approach with joint video-action generation for closed-loop control.
World-action models
- Unlike VLAs that primarily predict actions from visual and language inputs, WAMs additionally predict future environment observations and actions jointly.Source: §5 · pp. 6–7
Model Architecture
- DreamZero uses a joint video-action Diffusion Transformer that conditions on visual history, current proprioceptive state, and language instruction.Its output space contains predicted video frames and robot actions.Source: §5 · p. 6
Pre-training
- DreamZero is pretrained on approximately 500 hours of AgiBot G1 teleoperation data and DROID data for Franka, with separate models for the two embodiments.Source: §5 · p. 7
- Training uses the same flow time for video and action within each chunk, with teacher forcing on preceding clean chunks.The joint flow-matching loss supervises video and action velocity prediction, allowing the shared DiT to learn environment dynamics and robot control.Source: §5 · p. 7
Post-training
- DreamZero is adapted to downstream manipulation tasks through offline supervised fine-tuning that retains the joint flow-matching objective.The supplied passages name shirt folding, fruit packing, and table bussing as task-specific demonstrations.Source: §5 · pp. 7–8
Deployment
- DreamZero conditions joint video-action generation on visual history, proprioceptive state, and language, then uses updated observations for closed-loop interaction.It executes action chunks asynchronously while generating subsequent chunks; action selection may include smoothing.Source: §5 · p. 8
6 Discussion
The discussion presents a stochastic-kernel formulation connecting embodied-model outputs and action realization to policies deployed in closed-loop interaction. It identifies distribution shifts and generalization across tasks, environments, and embodiments as future areas of investigation.
Discussion
- The formulation connects pre-training, post-training, and closed-loop interaction by relating heterogeneous model outputs to task-conditioned robot policies.Source: §6 · p. 8
- The framework provides a common probabilistic language for embodied models with different architectures, output representations, and learning objectives.Source: §6 · p. 8
- Future work may investigate how pre-training and post-training strategies affect policy-induced distribution shifts and generalization across tasks, environments, and embodiments.Source: §6 · p. 8