PaperlayerView exact revision on arXiv

Source-linked AI summary

EO-1: An Open Unified Embodied Foundation Model for General Robot Control

Delin Qu, Haoming Song, Qizhi Chen, Zhaoqing Chen, Xianqiang Gao, Dong Wang, Xinyi Ye, Qi Lv, Modi Shi, Guanghui Ren, Cheng Ruan, Maoqing Yao, Haoran Yang, Jiacheng Bao, Bin Zhao, Xuelong Li

arXiv:2508.21112v5cs.ROcs.AI

TL;DR

Generalist robot policies still face open-world generalization challenges and lack flexible integration of reasoning with physical action. EO-1 combines interleaved embodied data with a unified model trained for text reasoning and action generation, and the paper reports gains across embodied reasoning and robot-control evaluations. The authors also identify remaining challenges in generalizing beyond the action domain and handling complex scenarios.

  • Problem

    Open-world generalization and flexible integration of multimodal reasoning with robot action remain central challenges for generalist robot policies.

    Source: §1 · pp. 1–2

  • Method

    EO-1 uses a unified model trained on interleaved embodied data, combining autoregressive text decoding with flow-matching action generation.

    Source: §2.1 · p. 4, §6 · p. 25

  • Results

    EO-1 surpasses existing open-source models across multiple embodied reasoning and robot-control benchmarks, with reported real-robot evaluations showing stronger reasoning and dexterous control in open-world generalization.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The evaluations support interleaved vision-text-action learning as an approach for open-world understanding and generalization across embodied reasoning and robot-control tasks.

    Source: §4.9 · p. 23, §6 · p. 25

    The authors identify limited robot-control data as a challenge for generalization outside the action domain and list navigation, obstacle avoidance, failure analysis, and human interaction as areas for future work.

    Source: §4.7 · p. 19, §6 · p. 25

Abstract

from arXiv · show

The human ability to seamlessly perform multimodal reasoning and physical interaction in the open world is a core goal for general purpose embodied intelligent systems. Recent vision-language-action (VLA) models, which are co-trained on large-scale robot and visual-text data, have demonstrated notable progress in general robot control. However, they still fail to achieve human-level flexibility in interleaved reasoning and interaction. In this work, we introduce EO-Robotics, consists of EO-1 model and EO-Data1.5M dataset. EO-1 is a unified embodied foundation model that achieves superior performance in multimodal embodied reasoning and robot control through interleaved vision-text-action pre-training. The development of EO-1 is based on two key pillars: (i) a unified architecture that processes multimodal inputs indiscriminately (image, text, video, and action), and (ii) a massive, high-quality multimodal embodied reasoning dataset, EO-Data1.5M, which contains over 1.5 million samples with emphasis on interleaved vision-text-action comprehension. EO-1 is trained through synergies between auto-regressive decoding and flow matching denoising on EO-Data1.5M, enabling seamless robot action generation and multimodal embodied reasoning. Extensive experiments demonstrate the effectiveness of interleaved vision-text-action learning for open-world understanding and generalization, validated through a variety of long-horizon, dexterous manipulation tasks across multiple embodiments. This paper details the architecture of EO-1, the data construction strategy of EO-Data1.5M, and the training methodology, offering valuable insights for developing advanced embodied foundation models. Project Page: https://eo-robotics.ai/eo-1.

1. Introduction

The introduction identifies open-world generalization and the integration of reasoning with action as central challenges for generalist robots. EO-1 addresses them with interleaved embodied data and a unified model, reporting results across embodied reasoning and robot-control benchmarks.

1. Introduction

  • Open-world generalization remains a central challenge because robots must combine world knowledge, human-level reasoning, and dexterous action across diverse and unforeseen situations.The introduction notes progress in constrained environments but identifies open-world capability as unresolved.Source: §1 · p. 1
  • Prior robot-only VLA training is described as restricting policies to narrow task domains and weakening inherited semantic knowledge and instruction following.Earlier approaches extend vision-language models using domain-specific robotic data.Source: §1 · p. 1
  • Even co-training with web and robot data leaves a gap: action-at-the-end outputs overlook temporal and causal dependencies, motivating flexible integration of reasoning and acting.The paper frames the goal as reasoning guiding action and action informing subsequent reasoning.Source: §1 · p. 2
  • EO-1 proposes interleaved embodied pretraining, requiring a unified architecture and multimodal data that jointly integrate text, images, videos, and robot actions.The proposal draws inspiration from interleaved multimodal pretraining in understanding and generation systems.Source: §1 · p. 2
  • The data pipeline combines web vision-language data with annotated robot episodes, then orders reasoning QA and control actions temporally in flexible interleaved formats.Annotations cover physical common sense, planning, localization, affordances, and multi-view correspondence.Source: §1 · p. 2
  • EO-1 is a 3B-parameter model trained on EO-Data1.5M, and EO-Robotics releases model weights, training code, and dataset components.The reported task range includes pick-and-place, articulated manipulation, long-horizon planning, and dexterous skills.Source: §1 · pp. 2–3
  • EO-1 surpasses existing open-source models across embodied reasoning and robot-control benchmarks, with real-robot evaluations reporting stronger reasoning and dexterous control in open-world generalization.The named benchmarks include ERQA, LIBERO, SimplerEnv, and EO-Bench.Source: §1 · p. 3

2. The EO-1 Model and Training Paradigm

EO-1 formulates embodied reasoning and robot control as interleaved outputs of a shared transformer. Its training combines language-token prediction with action flow matching and rectifies interleaved action segments to preserve causal context.

2. The EO-1 Model and Training Paradigm

  • The model’s unified transformer is designed to capture temporal dynamics and causal relationships across vision, text, and action for cross-modal transfer and alignment.The shared architecture is the central formulation of the model and training paradigm.Source: §2 · p. 3

2.1. Problem Formulation

  • EO-1 models action chunks and reasoning text conditioned on observations, task context, and prior actions, while not generating outputs for vision or robot-state tokens.Observations include multi-view images and robot state; language context can be embodied QA or an overall task prompt.Source: §2.1 · p. 3

2.2. Model and Training

  • A shared decoder-only transformer processes interleaved multimodal inputs and produces text through a language head and continuous actions through a flow-matching head.The backbone is initialized from Qwen2.5-VL; continuous robot state and noisy action are embedded alongside text and image inputs.Source: Figure 1 · p. 4, §2.2 · p. 4
  • EO-1 avoids extra action-specific parameters, integrating discrete embodied reasoning and continuous robot control through shared model parameters.The formulation describes action generation as integrating the predicted vector field from random noise over τ = 0 to τ = 1.Source: §2.1 · p. 4, §2.2 · p. 5
  • Interleaved rectifying sampling splits sequences around action-generation segments and replaces noisy middle-segment actions with clean actions in subsequences that continue past them.This supports both flow-matching denoising and indirect gradients through clean actions used by subsequent interleaved generation.Source: §2.2 · p. 5, Figure 2 · p. 5
  • The unified model is trained end-to-end with next-token prediction for text and flow-matching denoising for robot actions.The total objective is the sum of the autoregressive and flow-matching losses.Source: §2.2 · pp. 5–6

3. Dataset and Benchmark

EO-1’s training corpus combines web multimodal data, robot-control episodes, and interleaved embodied data. The section describes how robot videos become reasoning QA and interleaved training examples, and presents EO-Bench as a structured evaluation of embodied reasoning.

3. Dataset and Benchmark

  • EO-1’s pretraining corpus comprises web multimodal data, robot-control data, and interleaved embodied data, with scale and composition summarized in Table 1.The corpus spans text, image, video, and robot-control data.Source: §3 · p. 6, Table 1 · p. 6

3.1. Data Source

  • The data sources combine 5.7 million web multimodal samples with 1.2 million robot-control episodes drawn from five public sources.Web data covers visual instruction following, video understanding and reasoning, and referring-expression and spatial grounding; robot data includes multiple embodiments and task types.Source: §3.1 · pp. 6–7

3.2.1. Embodied Reasoning Data Pipeline

  • Robot-video curation filters for visual diversity, splits clips into subtasks, generates temporal and spatial reasoning questions, and cleans or rewrites QA pairs.VLMs generate candidate answers, which human annotators select or rewrite; the resulting datasets include 70 thousand physical-commonsense QA pairs, 0.7 million task-reasoning pairs, and 1.5 million spatial-reasoning pairs.Source: §3.2.1 · pp. 7, 9

3.2.2. Interleaved Vision-Text-Action Data Pipeline

  • Three formats interleave reasoning QA with robot control along video time: temporal planning, spatial trajectory prediction, and free-chatting QA.Planning answers are merged into action instructions; trajectory answers connect predicted gripper coordinates to action generation, while free-chatting QA uses the original task instruction.Source: §3.2.2 · pp. 9–10

3.3. Embodied Onevision Reasoning Benchmark

  • 648 manually labeled EO-Bench QA pairs span spatial understanding, task reasoning, physical dynamic reasoning, and physical commonsense, with 370, 140, 84, and 48 pairs, respectively.The benchmark aims to target distinct reasoning angles for interpretable diagnosis rather than conflate multiple capabilities in one question.Source: §3.3 · pp. 10–11
  • EO-Bench organizes embodied evaluation into spatial understanding, physical commonsense, task reasoning, and state estimation, with interpretable subcategories for complementary capabilities.Subcategories include trajectory reasoning, multiview pointing, counterfactual reasoning, task planning, process verification, and object-state recognition.Source: §3.3 · p. 11

4. Experimental Results

The section evaluates EO-1 across embodied reasoning, simulated and real-world robot control, open-world generalization, and integrated reasoning-control tasks, reporting benchmark and real-world completion results across robots and task types. It also examines control and generalization performance in relation to hybrid decoding, task-aligned interleaved data, data scale, and task context.

4.1. Implementation Details; 4.2. Open-sourced Benchmark Evaluation Setups

  • EO-1 trains on 135B tokens and, at inference, predicts 16-step action chunks with 10 denoising iterations using 6 GB of GPU memory on one RTX 4090.The evaluation setup covers reasoning benchmarks RoboVQA, ERQA, and EO-Bench, plus the LIBERO and SimplerEnv manipulation benchmarks; EO-Bench comprises 700 tasks across four reasoning categories.Source: §4.1 · p. 11, §4.2 · p. 12

4.3. Benchmark Evaluation Results

  • RoboVQA BLEU-4 is 58.5 versus GPT-4o’s 47.2, ERQA accuracy is 45.5 versus InternVL2.5 8B’s 45.2, and EO-Bench Spatial and Temporal scores are 36.4 and 38.9.Mainstream VLMs average 32 on EO-Bench; the Spatial leaderboard covers multiview reasoning, trajectory prediction, and visual grounding, while Temporal evaluates multimodal temporal reasoning in manipulation.Source: §4.3 · p. 12, Table 2 · p. 13
  • LIBERO success averages 98.2%, exceeding OpenVLA-OFT by 1.1%, π0 by 4.0%, and GR00t-N1 by 4.3%; SimplerEnv reaches 72.7%, 76.5%, and 63.0%.On SimplerEnv, EO-1 surpasses π0 by 3.5%, 5.1%, and 8.3% on WidowX, Google-VM, and Google-VA, respectively. The authors report these results with lightweight fine-tuning on a modest mixed-modality dataset.Source: Table 3 · p. 13, §4.3 · p. 13, Table 4 · p. 14

4.5. Mastering Diverse Manipulations on Multiple Embodiments

  • Figure 6 reports that across all robot platforms and task categories, EO-1 achieves 86.0%, versus 43.0% for Fast, 71.0% for GR00T-N1.5, and 68.0% for π0.EO-1 scores 81.0% on Agibot G-1 long-horizon tasks, 94.0% on Franka Panda pick-and-place tasks, and 83.0% on embodied reasoning control tasks.Source: Figure 6 · p. 16, §4.5 · p. 16

4.6. Specializing to Long-horizon Dexterity

  • EO-1 averages 81.0% completion on four specialized long-horizon tasks, exceeding GR00T-N1.5 at 68.0% and π0 at 67.0%.It scores 56.0% on Roast Beef Steak, 85.0% on Make Breakfast Sandwich, and 95.0% on Sorting Grocery Items. Fine-tuning used 150 demonstration trajectories per task, 25 epochs, and a 9:1 sub-task-to-overall-task data mixture.Source: Figure 7 · p. 17, §4.6 · p. 17

4.7. Emerging Open-world Embodied Generalization

  • Against GR00T-N1.5 and π0, EO-1 achieves an average completion score of 87.0%, outperforming their 75.0% and 66.0%, with gains consistent across embodiments and task types.On 12 tasks probing visual, action, and language variation, EO-1 scores 73.0% overall (67.0% Action, 79% Language, 72.0% Visual). The authors report challenges generalizing out of the Action Domain with limited robot-control data, and marked improvement where densely annotated interleaved Bridge data is available.Source: §4.7 · pp. 18–20

4.8. Enhanced Generalization with Unified Reasoning

  • On the reasoning-control benchmark, EO-1 scores 84.0% on Make Breakfast Sandwich, 55.0% on Roast Beef Steak, 79.0% on Visual Rearrangement, and 76.0% on Tic-Tac-Toe.It has the highest average completion across the four tasks; the largest reported margin over the best baseline is 40.0 points on Tic-Tac-Toe. EO-1 integrates planning and control in one decoder, while the compared GR00T-N1.5 and π0 baselines use an external LLM planner and separate controller.Source: §4.8 · pp. 20–21, Figure 11 · p. 22

4.9. Discussion: Architecture and Data Contributions to Generalization

  • The discussion examines how shared-parameter autoregression and flow matching, alongside interleaved vision-text-action supervision, contribute to visual, action, and language generalization.This passage states the evaluation objective and benchmark axes, not findings from the discussion.Source: §4.9 · p. 22
  • 98.2% overall success rate for EO-1 (base) versus 88.0% for EO-1 (fast) on LIBERO, while EO-1 (base) also outperforms EO-1 (fast) across WidowX generalization regimes and data scales.The hybrid model combines discrete autoregression for vision–language tokens with continuous flow-matching denoising for actions; discrete-only decoding struggles with low-frequency action modes and distribution shifts.Source: §4.9 · pp. 22–24, Table 5 · p. 23, Figure 12 · p. 23
  • 0.82 versus 0.59 Language Generalization at 50k demonstrations for EO-1 (interleaved) and EO-1 (base), respectively, after starting at 0.27 and 0.01 with 50 demonstrations.EO-1 (interleaved) consistently outperforms EO-1 (base) as interleaved demonstrations scale from 50 to 50k, with faster and more robust generalization.Source: §4.9 · p. 23, Figure 12 · p. 23
  • 0.77 to 0.66 on Agibot Sandwich and 0.71 to 0.34 on WidowX Generalization accompany adding generic instruction-following data in EO-1 (llava).The reported drops suggest that multimodal data misaligned with embodied control can weaken physical grounding.Source: §4.9 · p. 23, Table 5 · p. 23
  • 0.75 performance on Agibot Sandwich is unchanged between EO-1 (base) and EO-1 (interleaved), whereas WidowX Generalization improves from 0.67 to 0.73.The discussion attributes the contrast to the limited dataset for Agibot versus large-scale, diverse Bridge Data for WidowX, underscoring the value of task-aligned multimodal data.Source: §4.9 · pp. 23–24, Table 5 · p. 23

5. Related Work

Related work contrasts discrete action-token prediction with continuous action denoising, then discusses co-training and unified multimodal architectures. It identifies limits in instruction following, temporal interaction modeling, knowledge preservation, and training efficiency.

Vision-Language-Action Models

  • Autoregressive VLA approaches discretize actions but can incur inference latency and limited resolution, while diffusion or flow-matching action experts offer fast inference and dexterous control.The section also notes challenges combining autoregression and continuous denoising, including knowledge preservation and training efficiency.Source: §5 · p. 24
  • Co-training web and robot data has shown promise for generalization, but current approaches still overlook embodied interactions’ temporal dynamics and causal relationships, leaving instruction following suboptimal.The section calls for improved co-training, cross-modal transfer, and larger interleaved datasets.Source: §5 · p. 24

Unified Multimodal Models

  • Unified multimodal models combine understanding and generation in shared architectures, with later designs adding modality-specific parameters or dual visual encoding.The section also describes approaches that assemble frozen language models with adapters or learnable queries, noting their struggles in robotic learning.Source: §5 · pp. 24–25
  • Prior work’s unified understanding-and-generation designs inspire EO-1’s architecture and exhibit open-world capabilities.The section names representative works including Black et al. (2024, 2025), Bjorck et al. (2025), and others.Source: §5 · p. 25

6. Conclusion and Future Work

EO-1 combines a unified model, interleaved embodied data, and an open training recipe for perception, reasoning, planning, and robot control. The authors report strong performance against VLA baselines and identify complex scenarios, efficient inference, and broader data and embodiment coverage as future work.

6. Conclusion and Future Work

  • EO-1 combines autoregressive decoding and flow-matching denoising in one model, co-trained on web data and EO-Data1.5M, and is released with an open recipe and EO-Bench.The dataset is curated for interleaved vision-text-action data intended to improve open-world generalization.Source: §6 · p. 25
  • Evaluations across embodied reasoning and robot control report that EO-1 consistently outperforms strong VLA baselines in multimodal reasoning and dexterous robot control.EO-Bench is presented as a benchmark for evaluating embodied reasoning capabilities.Source: §6 · p. 25
  • Future work targets navigation, obstacle avoidance, failure analysis, human interaction, more efficient architecture or asynchronous inference, and additional data and robot embodiments.The authors specifically mention human data and humanoid robots as additions to the training recipe.Source: §6 · p. 25

B. Training Dataset Statistics

This appendix material labels a pre-training dataset statistics table and provides example figures spanning embodied reasoning and interaction tasks.

B. Training Dataset Statistics

D. Data Annotation Pipeline Details

The data preprocessing pipeline is organized into four components for curating and annotating videos and cleaning the resulting data.

D. Data Annotation Pipeline Details

D.1. Video Filtering and Curation

EO-Data1.5M video curation uses feature-based clustering and human inspection to improve visual variety and dataset coverage. A unified interface also supports selecting videos with viewpoint diversity for multiview reasoning.

D.1. Video Filtering and Curation

  • Pretrained visual features and K-means clustering organize robot videos, after which human-assisted inspection removes same-scene samples to improve visual variety.Videos are selected in balanced sets from each cluster.Source: §D.1 · p. 37
  • The curation process addresses redundancy in robot datasets collected in constrained environments by manually pruning clusters to improve dataset coverage.Source: §D.1 · p. 37
  • A unified interface lets curators select videos with sufficient viewpoint diversity for multiview and cross-scene reasoning tasks.Source: §D.1 · p. 37

D.2. Video Splitting and Captioning

Curated videos are divided into short clips, each focused on a single subtask. Captions describe robot actions and support both captioning QA and later reasoning annotations.

D.2. Video Splitting and Captioning

  • Annotators or pretrained VLMs segment curated videos into meaningful short clips, each containing a single subtask.Source: §D.2 · p. 37
  • Descriptive captions are generated for each clip to capture the specific robot actions.Source: §D.2 · p. 37
  • Clip captions serve both video-captioning QA and contextual prompting for subsequent reasoning annotations.Source: §D.2 · p. 37

D.3. QA Generation for Temporal and Spatial Reasoning

The section describes prompted QA construction for temporal and spatial reasoning, with VLM-generated questions and human-refined answers. Temporal reasoning targets task sequencing and physical common sense.

D.3. QA Generation for Temporal and Spatial Reasoning

  • VLMs use carefully designed templates to generate diverse questions, and human annotators refine the final answers.Source: §D.3 · p. 37

D.3.1. Temporal Reasoning

  • Task-planning questions test understanding of sequential subtask structure.Source: §D.3.1 · p. 37
  • Physical common-sense questions assess reasoning about object dynamics and physical feasibility.Source: §D.3.1 · p. 37

D.3.2. Spatial Reasoning

Spatial-reasoning QA covers four subtasks designed to capture fine-grained spatial relations. Rule-based filtering and LLM rewriting then improve answer validity and textual diversity.

D.3.2. Spatial Reasoning

  • Spatial QA includes trajectory prediction, object pointing, object referring, and multiview reasoning to capture fine-grained spatial relations.Source: §D.3.2 · pp. 37–38
  • Multiview data use the object-pointing prompting strategy, while curators select observations with sufficient viewpoint diversity.Source: §D.3.2 · p. 37, Figure 25 · p. 39
  • Rule-based filtering removes invalid or ambiguous spatial answers, and LLM rewriting increases textual diversity and introduces natural fuzziness.The authors state that this post-processing makes the dataset more robust.Source: §D.3.2 · p. 38

Related papers

Loading 2508.21112v5…