PaperlayerView exact revision on arXiv

Source-linked AI summary

Embodied-R1.5: Evolving Physical Intelligence via Embodied Foundation Models

Yifu Yuan, Yaoting Huang, Xianze Yao, Yutong Li, Shuoheng Zhang, Linqi Han, Pengyi Li, Jiangeng Sun, Wenting Jia, Zhao Zhang, Yuhao Liu, Ruihao Liao, Yucheng Hu, Qiyu Wu, Yuxiao Li, Zibin Dong, Fei Ni, Yan Zheng, Shuyang Gu, Yi Ma, Hongyao Tang, Han Hu, Jianye Hao

arXiv:2606.11324v2cs.ROcs.AIcs.LG

TL;DR

Existing embodied models cover only parts of the capability spectrum, while some multi-capability approaches use separate models rather than one unified architecture. Embodied-R1.5 unifies embodied capabilities in one model and achieves state of the art on 16 out of 24 embodied VLM benchmarks.

  • Problem

    Existing embodied models cover only parts of the capability spectrum, and some multi-capability approaches rely on separate models rather than one unified model.

    Source: §1 · p. 3

  • Method

    Embodied-R1.5 unifies embodied capabilities in one model, supported by three automated data pipelines, a 15B-token corpus, multi-task balanced RL, and a Planner-Grounder-Corrector framework.

    Source: §1 · p. 3

  • Results

    16 out of 24 Embodied VLM benchmarks achieve state-of-the-art results with Embodied-R1.5, which averages 70.4% across 21 main accuracy-based benchmarks.

    Source: §1 · p. 3

  • Takeaways & Limitations

    In the evaluated settings, internalized embodied reasoning can partially substitute for large-scale action pretraining.

    Source: §9 · p. 24

    The PGC closed-loop framework has been validated on tabletop manipulation; extending it to mobile manipulation and navigation remains a next step.

    Source: §9 · p. 24

Abstract

from arXiv · show

We introduce Embodied-R1.5, a unified Embodied Foundation Model (EFM) that integrates comprehensive embodied reasoning capabilities, spanning embodied cognition, task planning, correction, and pointing, within a single architecture toward general physical intelligence. Leveraging three automated data construction pipelines to significantly expand the data coverage of critical capabilities, we build a large-scale data system of over 15B tokens, and design a multi-task balanced RL recipe to alleviate heterogeneous task conflicts. We further introduce a Planner-Grounder-Corrector (PGC) closed-loop framework that enables a single model to autonomously execute and self-correct over long-horizon tasks. With only 8B parameters, Embodied-R1.5 achieves SOTA on 16 out of 24 embodied VLM benchmarks, surpassing leading models like Gemini-Robotics-ER-1.5 and GPT-5.4. Benefiting from the internalized embodied capabilities, Embodied-R1.5 can be fine-tuned into a VLA with only a small amount of data, outperforming leading VLA models like $π_{0.5}$ across 4 popular manipulation benchmark suites. We further conduct extensive zero-shot real-robot experiments, validating performance in instruction following, affordance grounding, articulated object manipulation, and long-horizon complex tasks, demonstrating strong generalization to the physical world. We open-source model weights, datasets, training code, and EmbodiedEvalKit, an evaluation framework tailored for embodied tasks, to facilitate future research in EFMs.

1 Introduction

Embodied-R1.5 unifies embodied cognition, planning and correction, and pointing in a single foundation model, with a closed-loop framework for long-horizon autonomy. The introduction presents strong benchmark results, lightweight downstream VLA adaptation, and zero-shot real-robot evaluation as evidence of its capabilities.

1 Introduction

  • The EFM unifies spatial cognition and reasoning, task planning and correction, and pointing and location, supported by three automated data pipelines totaling over 15B tokens and multi-task balanced RL.The training recipe also includes two-stage training to address interference among heterogeneous tasks.Source: §1 · p. 3, Figure 2 · p. 4
  • Prior work is limited by fragmented capabilities, multi-task interference across differing output formats, and inadequate validation of closed-loop autonomy on long-horizon tasks.The introduction notes that some models cover only parts of the capability spectrum, while others rely on separate models for different tasks.Source: §1 · p. 3
  • The PGC framework enables a single model to plan, ground, execute, and self-correct long-horizon tasks without human intervention; the authors provide EmbodiedEvalKit for 25+ benchmarks and open-source weights, data, and code.Source: §1 · pp. 3–4
  • With 8B parameters, Embodied-R1.5 achieves SOTA on 16 of 24 benchmarks and averages 70.4% across 21 main accuracy-based benchmarks, surpassing Gemini-Robotics-ER-1.5 (Team et al., 2025b) and GPT-5.4 by 17.0% and 21.7%, respectively.The model’s results are reported as evidence of its embodied reasoning performance.Source: §1 · pp. 3–4
  • Embodied-R1.5-VLA scores 92.4% on SimplEREnv Google Robot Visual Matching, over 20% above π0.5, and outperforms ManipLLM (Li et al., 2024b) by 11% on PartNet-Mobility.The authors attribute the light action-data adaptation to embodied reasoning already internalized by the foundation model.Source: §1 · pp. 3–4
  • Zero-shot real-robot experiments cover instruction following, affordance grounding, articulated object manipulation, and long-horizon complex tasks, demonstrating generalization to diverse real-world scenarios.Source: §1 · p. 4

2 Unified Embodied Capabilities & Architecture

Embodied-R1.5 unifies embodied cognition, planning and correction, and pointing and location in one model, forming a progressive reasoning chain from perception to decision to execution.

2 Unified Embodied Capabilities & Architecture

  • The three capabilities form complementary levels of embodied reasoning, unified in one model so information flows across dimensions without external communication.Cognition and spatial reasoning establish physical-world understanding; planning and correction schedule tasks and monitor execution; pointing and location translate reasoning into actionable points and trajectories.Source: §2 · p. 4, §2.1 · p. 5

2.1 Unified Embodied Capabilities

  • Embodied cognition covers spatial relations, metric reasoning, 3D scene perception, and robot-centric object and scene understanding.These capabilities support downstream planning and localization, precise manipulation, collision avoidance, and identification of targets, obstacles, and task context.Source: §2.1 · p. 5
  • Embodied planning and correction span long-horizon task decomposition, next-step planning, process detection, error localization, and corrective replanning or rollback.The model monitors task progress during execution and feeds corrective suggestions into the next planning round.Source: §2.1 · p. 5
  • Embodied-R1.5 strengthens open-vocabulary pointing and affordance understanding, treating coordinates as referenceable symbols throughout multi-step reasoning.Its grounding and trace capabilities cover objects, regions, functional parts, and ordered trajectories, including object and end-effector flows in 2D and spatially grounded traces in 3D; pointing is the signature capability of the Embodied-R series (Yuan et al., 2025b).Source: §2.1 · p. 5

2.2 Architecture

3 Training Data Construction

Embodied-R1.5’s training data system combines restructured open-source resources with three automated pipelines targeting gaps in spatial reasoning, planning and correction, and pointing. Across 34 datasets, it exceeds 15B tokens and balances capability categories with general visual cognition, reasoning, and instruction-following data.

Training Data Construction

  • The embodied data system combines 34 datasets and exceeds 15B tokens, integrating open-source resources with three pipelines targeting critical capability gaps.The overall mixture is balanced across capability categories and includes general visual cognition, logical reasoning, and instruction-following data as regularization against catastrophic forgetting.Source: §3 · pp. 6–7

Pipeline 1: 3D Scene Annotation for Spatial Reasoning

  • ER1.5-Spatial’s automated pipeline builds about 20K QA samples from 3D semantic scene graphs to address fine-grained tabletop spatial reasoning.It chains semantic understanding, monocular geometry estimation, open-vocabulary segmentation, plane alignment, and quality control on real robot-scene images.Source: §3 · p. 6

Pipeline 2: Failure-Aware Annotation for Planning and Correction

  • ER1.5-Correction organizes about 800K samples across planning and execution failures and detection, localization, and correction, forming six QA types.Planning errors are generated with five structured perturbations; execution errors use video truncation, description replacement, and simulated physics perturbations, followed by validity checks, balancing, and human review.Source: §3 · pp. 6–7

Pipeline 3: Affordance and Trajectory Data for Pointing

  • ER1.5-Pointing combines affordance annotations with end-effector and object trajectories, including traces for articulated-object manipulation.Trajectories are extracted from 3D poses or image-based tracking, and the method reports semantic generalization from low-fidelity simulated motion data; the filtered, normalized dataset contains about 400K samples.Source: §3 · p. 7

4 Training Strategy

Embodied-R1.5 uses supervised fine-tuning to establish a multi-task foundation, then reinforces it with RL and verifiable rewards, particularly for pointing. Its training recipe balances heterogeneous tasks and matches rewards to different output types.

4 Training Strategy

  • The two-stage strategy reserves low-resource, high-value tasks for RL, where verifiable rewards provide efficient learning signals and particularly boost pointing.The full data system is used for one SFT epoch before reinforced fine-tuning.Source: §4 · p. 8

4.1 Stage 1: Supervised Fine-Tuning

  • SFT jointly trains the vision encoder and Qwen3-VL-8B-Instruct on heterogeneous outputs as token sequences for one epoch.Full-parameter fine-tuning uses a causal language modeling objective; joint vision-encoder training adapts visual features to embodied scenarios and initializes RL rollouts.Source: §4.1 · p. 8

4.2.1 Multi-Task Balanced RL Recipe

  • Difficulty-aware filtering, degenerate-group masking, and mixed-batch reward normalization balance RL updates across heterogeneous tasks.Rollout pass rates select about 200K medium-difficulty samples; advantages use the within-group mean and mixed-batch standard deviation.Source: §4.2.1 · p. 8

4.2.2 Multi-Task Reward Design

  • Five reward families cover exact matching, IoU, point distance, trajectory RMSE, and semantic similarity, with piecewise-linear decay providing partial credit for continuous outputs.Point rewards use nearest-neighbor distance or point-in-region checks; trajectory rewards combine 2D RMSE and depth MAE, with penalties for length mismatch.Source: §4.2.1 · p. 8, §4.2.2 · pp. 9–10

4.2.1 Multi-Task Balanced RL Recipe

  • Without forcing explicit reasoning chains, RL reward design encourages near-zero reasoning for pointing and structured thought for complex planning.Only answer tags are constrained; final-answer quality determines reward, so additional reasoning is learned when it improves planning quality.Source: §4.2.1 · pp. 8–9

5 Closed-Loop PGC Autonomy Framework

The PGC framework uses one Embodied-R1.5 model to plan, ground, and correct long-horizon tasks through asynchronous capability calls, with a lightweight harness coordinating execution and memory.

5 Closed-Loop PGC Autonomy Framework

  • A single Embodied-R1.5 instance serves as Planner, Grounder, and Corrector, avoiding multi-model cascading or multi-agent orchestration.The framework is a lightweight stateless harness for control-flow scheduling and memory management; reasoning and error attribution come from the model.Source: §5 · pp. 10–11
  • The Planner decomposes a long-horizon instruction into sub-tasks and replans after each completion using the latest observation, while the Grounder selects pointing capabilities to produce spatial commands.For pressing a dispenser switch, the Grounder first uses OFG to localize the functional part, then VTG to plan a pressing trajectory.Source: §5 · pp. 10–11
  • An asynchronous Corrector classifies execution as SUCCESS, PROCESS, or FAIL using observations and history in a FIFO memory buffer, feeding failures back to the Planner for retry or replanning.The described switch-pressing sub-task succeeds on its second attempt after an initial failure.Source: §5 · p. 11
  • Shared model context lets the Corrector use the Planner’s reasoning and Grounder’s spatial understanding, while the framework’s ceiling depends on model capability rather than engineering complexity.The framework is described as enabling long-chain real-world tasks to run autonomously without human intervention.Source: §5 · p. 11

6 EmbodiedEvalKit

EmbodiedEvalKit addresses inconsistent embodied-VLM evaluation with an open, unified framework that standardizes benchmark data, model outputs, and metric computation.

6 EmbodiedEvalKit

  • EmbodiedEvalKit replaces ad-hoc, non-comparable evaluation scripts with a unified framework for embodied VLMs.Existing tools such as VLMEvalKit (Duan et al., 2024) and lmms-eval (Zhang et al., 2024) target standard VQA and captioning and do not handle embodied outputs and metrics.Source: §6 · pp. 11–12
  • Its four-layer design standardizes benchmark data, supports multiple inference backends, normalizes grounding outputs, and computes official metrics by capability category.The data layer uses HuggingFace Parquet; the inference layer handles model-specific coordinate systems and prompts, including across open-source and proprietary models.Source: §6 · p. 12
  • The open, extensible framework supports 25+ embodied benchmarks and 20+ models; this paper reports 24 benchmarks and 13 baselines through the unified pipeline.Pre-processed evaluation data is released for reproducibility, and adding a benchmark requires a data adapter and evaluator following standardized templates.Source: §6 · p. 12

7 Experiments

Section 7 evaluates Embodied-R1.5 across embodied VLM capabilities, downstream manipulation, zero-shot transfer, long-horizon closed-loop execution, and design choices, using benchmark results, real-world demonstrations, and ablations. It also examines autonomous contact-rich manipulation by pairing the model’s semantic understanding and localization during approach with a force-aware policy that regulates contact using force/torque feedback.

7.1 Embodied VLM Benchmarks

  • Embodied-R1.5 achieves state of the art on 16 of 24 embodied VLM benchmarks across planning, pointing, and cognition.The evaluation also reports minimal degradation on seven general vision benchmarks relative to Qwen3-VL-8B, with MMMU and ScienceQA improving.Source: Table 1 · p. 13, §7.1 · pp. 13–14, Table 2 · p. 14
  • Pointing & Location averages 72.8 with state-of-the-art results on all nine pointing benchmarks, while Cognition & Spatial Reasoning averages 70.2.Planning & Correction averages 65.3, with a +24.0pp margin over Gemini-Robotics-ER-1.5; Embodied-R1.5 also performs competitively across all eight cognition and spatial benchmarks.Source: §7.1 · pp. 13–14, Table 4 · p. 15

7.2 Robotic Manipulation in Simulation

7.4 Long-Horizon Closed-Loop Demonstrations

  • The PGC framework autonomously executes four zero-shot long-horizon tasks and recovers from repeated human perturbations without manual reset.The Corrector detects deviations and triggers re-localization and re-execution; all tasks are reported as fully autonomous, with no human intervention at any stage.Source: §7.4 · pp. 18–19

7.3 Zero-Shot Manipulation Transfer

  • In zero-shot real-robot tests, Embodied-R1.5 performs best across all task categories, reaching 66.6% on Cup Disassembly while all baselines score 0%.The five task categories are tested over six trials each without fine-tuning, using the PGC closed-loop framework.Source: §7.3 · pp. 17–18, Table 12 · p. 19

Q1: Does RFT improve over pure SFT?

  • RFT improves over SFT by +3.8pp on Pointing, +2.7pp on Planning, and +1.3pp on Cognition; the full RL recipe balances all three dimensions.The full recipe combines difficulty-aware data filtering, dynamic filtering, and global batch reward normalization, achieving Cognition 70.2 / Planning 65.3 / Pointing 72.8.Source: Table 14 · p. 20, Q1: Does RFT improve over pure SFT? · pp. 20–22

7.3 Zero-Shot Manipulation Transfer

  • On zero-shot RoboTwin2.0, Embodied-R1.5 achieves 65.0% average success despite baselines using 400 demonstrations per task.On ManiSkill-Affordance, its average success is 0.76 on seen categories and 0.66 on unseen categories, compared with ManipLLM’s 0.59 and 0.54, respectively.Source: §7.3 · pp. 16–17, Table 11 · p. 19, Table 13 · p. 21

Q1: Does RFT improve over pure SFT?

  • With GR00T on LIBERO, Embodied-R1.5 reaches 92.0% at 10K steps versus 82.0% with Qwen3-VL-8B, and 97.0% versus 96.0% at 80K steps.The advantage is reported across both GR00T and OpenVLA-OFT action experts, supporting transfer beyond a warm-start effect.Source: Q1: Does RFT improve over pure SFT? · p. 21

Q4: Can Embodied-R1.5 enable contact-rich manipulation via force-aware policy?

  • Embodied-R1.5 handles semantic understanding and spatial localization during approach, while a force-aware flow-matching policy takes over at contact to regulate interaction dynamics.This vision-to-force handover aims to support visual generalization across objects and scenes and force generalization across contact conditions.Source: Q1: Does RFT improve over pure SFT? · p. 22
  • Both Clean the Vase and Plug the Charger were executed fully autonomously, testing pressure control on curved geometry and sub-millimeter alignment under tight contact constraints, respectively.The evaluation uses two representative contact-rich manipulation tasks (figure 11).Source: Q1: Does RFT improve over pure SFT? · p. 22
  • The experiments demonstrate that Embodied-R1.5’s embodied reasoning composes with force-aware policies for contact-rich manipulation beyond either modality alone.The complete technical report is available in ForceFlow (Zhang et al., 2026b).Source: Q1: Does RFT improve over pure SFT? · p. 22

8 Related Work

Related work progresses from EFMs targeting individual embodied capabilities to broader unified reasoning models, while also using EFMs to strengthen VLA training. Embodied-R1.5 unifies embodied capabilities in one model, and its VLA variant demonstrates reduced action-learning data needs.

8 Related Work

9 Conclusion

Embodied-R1.5 unifies embodied reasoning in an 8B-parameter model, achieves strong benchmark and real-robot results, and is released with open-source infrastructure. The conclusion also identifies 2D perception, lightweight action generation, and tabletop-only PGC validation as areas for future work.

9 Conclusion

  • Embodied-R1.5 integrates three embodied reasoning dimensions in a shared 8B-parameter Transformer, supported by a 15B-token corpus and a multi-task balanced RL recipe.Three automated data-construction pipelines expand the corpus; the shared architecture replaces the fragmented multi-model paradigm of prior work.Source: §9 · pp. 23–24
  • 70.4% average on 21 main accuracy-based benchmarks, Embodied-R1.5 reaches SOTA on 16 of 24 and surpasses Gemini-Robotics-ER-1.5 and GPT-5.4 by 17.0% and 21.7%, respectively.With a small amount of action data, Embodied-R1.5-VLA outperforms π0.5 across 4 manipulation benchmark suites, including 92.4% on SimplerEnv Google Robot Visual Matching. Zero-shot real-robot experiments cover instruction following, affordance grounding, articulated object manipulation, and long-horizon closed-loop execution.Source: §9 · p. 24
  • The Planner-Grounder-Corrector framework lets a single model orchestrate long-horizon execution with self-correction, and the release includes model weights, data, training code, and EmbodiedEvalKit.The conclusion presents the closed-loop framework as validating the translation of internalized reasoning into long-horizon robotic autonomy.Source: §9 · p. 24
  • Future work includes adding native 3D perception, tighter coupling between reasoning tokens and action generation, and extending PGC beyond tabletop manipulation to mobile manipulation and navigation.The current model operates on 2D images, the VLA extension uses a lightweight action head, and PGC has been validated on tabletop manipulation.Source: §9 · p. 24

Contributions · A Data Composition Details

The contributions section assigns project leadership, dataset and pipeline development, evaluation, robotics experiments, and research supervision to named contributors. The appendix points readers to Section 3 for the full data composition details.

Contributions

  • Yifu Yuan proposed the methodology and research direction, developed the complete training dataset, and led implementation, training, inference, and experimental analysis.Yifu Yuan also led EmbodiedEvalKit design and manuscript drafting.Source: Contributions · p. 24
  • Yifu Yuan, Hongyao Tang, and Yi Ma served as co-project leads, while Shuyang Gu, Yi Ma, Hongyao Tang, and Jianye Hao were corresponding authors.The co-project leads oversaw overall project execution.Source: Contributions · p. 24
  • Yaoting Huang, Linqi Han, and Jiangeng Sun contributed to model evaluation.Source: Contributions · p. 24
  • Yaoting Huang, Linqi Han, Pengyi Li, Jiangeng Sun, Wenting Jia, Yucheng Hu, Zhao Zhang, and Yuxiao Li performed data construction and cleaning.Source: Contributions · p. 24
  • Shuoheng Zhang, Xianze Yao, Pengyi Li, Yuhao Liu, Yutong Li, Ruihao Liao, Qiyu Wu, and Yuxiao Li conducted the real-world robotic platform setup and experiments.Source: Contributions · p. 24
  • Shuyang Gu, Zibin Dong, Fei Ni, Yan Zheng, Han Hu, and Jianye Hao provided research guidance and supervision.Source: Contributions · p. 24

A Data Composition Details

A.1 Embodied Cognition & Spatial Reasoning Data · A.2 Embodied Planning Data · A.3 Embodied Correction Data

The three data pipelines expand embodied cognition, long-horizon planning, and failure correction training with spatial, egocentric, and real-robot or simulated examples. They combine existing datasets with automated construction and perturbation methods to cover diverse reasoning and recovery tasks.

A.1 Embodied Cognition & Spatial Reasoning Data

  • Multi-view and video spatial datasets train reasoning over object counts, distances, directions, topological relations, depth ordering, occlusion, appearance order, and size.The data requires integrating spatial information across viewpoints as a foundation for 3D perception in embodied scenes.Source: §A.1 · p. 34
  • DepthLM-derived QA pairs use sensor-ground-truth indoor depth at specified image coordinates, covering absolute metric and relative depth.Construction excludes boundary, infinite, and physically inconsistent pixels and normalizes focal lengths; the pipeline selects high-quality indoor datasets.Source: §A.1 · p. 34
  • Approximately 106K samples from Robo2VLM and RoboBrain 2.0 provide scene-understanding QA from the robot’s first-person manipulation viewpoint.Source: §A.1 · p. 34
  • ER1.5-Spatial comprises approximately 20K QA pairs generated from structured 3D semantic scene graphs produced by an automated pipeline from single RGB images.The pairs cover spatial relations, distance metrics, scene cognition, and appearance order.Source: §A.1 · p. 34

A.2 Embodied Planning Data

  • Approximately 950K planning QA pairs cover manipulation, navigation, and egocentric scenarios, while first-person video samples require predicting actions from observed progress.The planning data combines open-source datasets with EgoPlan-IT and EgoRe samples.Source: §A.2 · pp. 34–35

A.3 Embodied Correction Data

  • 800K ER1.5-Correction samples cover failure detection, localization, and recovery across planning and execution failures, yielding six QA task types.The dataset complements 78K RoboFAC video QA pairs and uses real-robot and simulation data sources.Source: §A.3 · p. 35
  • Planning failures are generated by perturbing correct sub-task plans, while execution failures use truncated videos, altered descriptions, simulated physical perturbations, and reformatted RoboFail records.Correct plans are retained as positive samples, and all data types undergo human sampling verification.Source: §A.3 · p. 35

A.4 Embodied Pointing Data · A.5 General Knowledge Data · B Automatic Data Construction Pipeline

Embodied-R1.5 expands embodied pointing data across four capabilities, including automated functional-affordance and trajectory generation, while retaining general vision-language training data. The appendix also maps the three automated data-construction pipelines to their detailed sections.

A.4 Embodied Pointing Data

  • Pointing is Embodied-R1.5’s signature capability, with data organized around four capabilities and expanded through automated functional-affordance and trajectory-generation pipelines.The data also integrates existing open-source sources.Source: §A.4 · p. 35
  • Referring expression grounding trains precise localization of target objects from natural-language descriptions.Training data aggregates sources including RefCoco, SAM2 (Ravi et al., 2025), Pixmo-Points, and RoboRefit (Lu et al., 2023).Source: §A.4 · p. 35
  • Region grounding supports placement and spatial arrangement by identifying suitable free regions and assessing placement feasibility.Data combines RoboPoint (Yuan et al., 2024), RefSpatial (Zhou et al., 2025), and ER1-Point, plus human-interaction videos and synthetic Regular Rearrangement scenes where a model points to complete block patterns.Source: §A.4 · pp. 35–36
  • Functional grounding localizes object parts whose physical properties inform grasping strategies.The data draws on open-source affordance sets, simulated articulated-object annotations, and human grasping patterns on functional parts; PartNet-Maniskill covers 17 articulated-object categories from ManiSkill (Tao et al., 2024).Source: §A.4 · p. 36
  • Visual Trace Generation supplies feasibility-curve annotations for trajectories used in embodied-task planning.Automatically extracted from simulation and human-interaction data, traces describe either end-effector motion or manipulated-object motion.Source: §A.4 · p. 36

A.5 General Knowledge Data

B Automatic Data Construction Pipeline

  • The appendix details three automated data-construction pipelines across spatial reasoning, correction, functional affordance, and trajectory data.Sections B.1 and B.2 cover ER1.5-Spatial and ER1.5-Correction; B.3 covers functional-affordance and trajectory construction.Source: B Automatic Data Construction Pipeline · p. 36

B.1 Pipeline 1: 3D Scene Annotation for Spatial Reasoning Data

The pipeline turns single RGB images of tabletop scenes into structured 3D semantic scene graphs, enabling programmatic generation of spatial-reasoning training data. It combines semantic labeling, metric geometry estimation, instance segmentation, and gravity-aligned coordinates to build ER1.5-Spatial.

Spatial reasoning QA generation

  • The pipeline generates ER1.5-Spatial, a dataset of approximately 20K tabletop spatial-reasoning QA samples, from single RGB images and structured 3D scene annotations.Its annotations include object categories, spatial positions, and inter-object relations.Source: §B.1 · pp. 36, 37, 39

Data source

  • Images sampled from Open X-Embodiment datasets, including BridgeData V2 and DROID, provide varied tabletop compositions, layouts, and lighting conditions.Representative frames are processed independently.Source: §B.1 · p. 37

Semantic understanding

  • Qwen3-VL and RAM++ supply complementary object labels, which a VLM normalization module merges into consistent categories for downstream detection and segmentation.Qwen3-VL is used for rich descriptions and long-tail objects, while RAM++ provides broader category coverage and faster inference.Source: §B.1 · p. 37

Geometry estimation and 2D instance segmentation

  • MoGe-2 estimates metric-scale depth, surface normals, and camera intrinsics, while Grounded-SAM-2 uses label-conditioned detection and segmentation to produce per-instance masks.Instances are filtered by detection confidence and mask area; metric-scale depth enables distance QA without a separate scale-recovery step.Source: §B.1 · p. 37

Horizontal plane alignment

  • RANSAC-based plane alignment sets the tabletop to z=0 and the vertical axis to gravity, with scenes lacking a clear dominant plane discarded.The pipeline lifts masked pixels into 3D and checks that most object-box bottoms lie within 7 cm of z=0.Source: §B.1 · p. 38

Spatial reasoning QA generation

  • The resulting scene metadata supports QA generation for spatial relations, distance metrics, scene cognition, and appearance order.These tabletop-level questions form the ER1.5-Spatial dataset for the training data system.Source: §B.1 · p. 39

B.2 Pipeline 2: Failure-Aware Data Construction for Correction Data … C.4 Chain-of-Thought Reasoning Examples

The paper builds failure-aware correction data and automated affordance and trajectory supervision, then illustrates how Embodied-R1.5 handles grounding, spatial reasoning, and multi-step embodied queries. These sections emphasize scalable data construction and qualitative examples rather than new benchmark comparisons.

B.2 Pipeline 2: Failure-Aware Data Construction for Correction Data

  • Correction QA organizes planning and execution failures across detection, localization, and correction, using perturbed plans and video–instruction inconsistencies to construct failures.Planning perturbations delete, duplicate, swap, or replace steps, objects, actions, or locations while retaining correct plans as positives; execution failures use truncation, description replacement, and injected physical perturbations, alongside native failure records.Source: §B.2 · pp. 39–40
  • ∼800K+ samples form ER1.5-Correction, with approximately balanced cognitive tiers and contributions including ∼802K BridgeData samples and ∼3.5K ManiSkill samples.Human sampling review checks failure validity, subtask-boundary truncations, plausible-but-incorrect replacements, and balanced detection labels; RoboFail, RoboFAC, and GEMBench also contribute.Source: §B.2 · p. 40

B.3 Pipeline 3: Functional Affordance & Trajectory Data Construction

  • Two automated pipelines generate functional-affordance data for OFG and trajectory data for VTG.They address costly real-world part annotations and the lack of explicit 2D/3D trajectory labels in many robot demonstrations.Source: §B.3 · p. 40, §B.3.1 · p. 40, §B.3.2 · p. 41

B.3.1 Object Functional Affordance Data

  • OFG annotations come from rendered articulated objects with semantic part labels and from hand–object contact regions that indicate grasp affordances.Simulation renders vary viewpoints and lighting; interaction data supplies grounding signals for where and how objects should be grasped for different intents.Source: §B.3.1 · pp. 40–41

B.3.2 Trajectory Data

  • VTG provides robot-centric end-effector traces and object-centric motion traces in 2D and 3D, paired with instructions and filtered for noisy or implausible paths.End-effector poses are projected into image coordinates or normalized in world coordinates; object trajectories come from tracking and segmentation or simulation pose sequences, supporting tasks such as pushing, pouring, and sweeping.Source: §B.3.2 · p. 41

C Qualitative Visualizations

  • Qualitative visualizations span zero-shot RoboTwin manipulation, pointing, spatial reasoning, and chain-of-thought examples across grounding and reasoning queries.The figures show pointing locations for grasping, placement, and affordances, alongside referring-expression grounding, free-space identification, functional-part localization, and spatial relations.Source: C Qualitative Visualizations · p. 41, Figure 15 · p. 43, Figure 16 · p. 44

C.4 Chain-of-Thought Reasoning Examples

  • Chain-of-thought examples illustrate pointing, feasibility planning, state verification and correction, and action understanding over images or observation sequences.Examples include scene descriptions of a robotic arm holding a showerhead near a wall-mounted holder; figures show final decisions emitted within an <answer> tag.Source: §C.4 · p. 45, Figure 18 · p. 46, Figure 19 · p. 47

Related papers

Loading 2606.11324v2…