PaperlayerView exact revision on arXiv

Source-linked AI summary

GEM: Generative Supervision Helps Embodied Intelligence

Ruowen Zhao, Bangguo Li, Zuyan Liu, Yinan Liang, Junliang Ye, Fangfu Liu, Diankun Wu, Zhengyi Wang, Xumin Yu, Yongming Rao, Han Hu, Jun Zhu

arXiv:2605.28548v1cs.CV

TL;DR

Embodied VLMs often lack the low-level spatial and physical knowledge needed for execution. GEM adds depth-generation supervision during pre-training, and achieves state-of-the-art performance across embodied benchmarks while GEM-VLA improves task execution in simulation and real-world evaluations.

  • Problem

    Standard embodied VLM training emphasizes high-level semantic understanding, leaving limited spatial and physical knowledge for embodied execution.

  • Method

    GEM jointly trains VLMs with depth-map generation, progressive optimization, and the GEM-4M dataset covering grounding, reasoning, and planning.

  • Results

    GEM achieves state-of-the-art performance across embodied benchmarks, including VSI-Bench gains from 50.4 to 62.8 for 2B and from 57.9 to 70.6 for 8B.

  • Takeaways & Limitations

    Depth supervision supports stronger spatial perception, while GEM-VLA sets new simulation records and generalizes robustly to real-world robotic evaluations.

  • Takeaways & Limitations

    GEM-VLA has not been pretrained on large-scale robot datasets, and further scaling of model size and training data remains needed.

Abstract

from arXiv · show

Embodied Vision-Language Models (VLMs) have demonstrated impressive performance and generalization in robotics, particularly within Vision-Language-Action frameworks. However, a significant gap remains between the high-level semantic focus of standard text-guided pre-training paradigms and the low-level spatial and physical knowledge critical for execution in embodied environments. In this paper, we introduce GEM, a Generative-supervised Embodied vision-language Model designed to bridge this divide. We propose integrating a depth map generation task directly into the VLM pre-training phase. By training this generative objective jointly with the main model, we observe substantial improvements in embodied intelligence, significantly enhancing both semantic understanding and physical operation capabilities. To support this paradigm, we curate and release GEM-4M, a comprehensive large-scale dataset featuring a mixture of grounding, reasoning, and planning data paired with high-quality depth supervision. Extensive experiments demonstrate that GEM achieves state-of-the-art results across diverse embodied benchmarks. Furthermore, our deployed action model, GEM-VLA, exhibits vastly superior task execution abilities in both simulation environments and real-world evaluations. Code, models, and datasets are available at https://zhaorw02.github.io/GEM/

1 Introduction

GEM addresses the gap between high-level semantic training and the fine-grained spatial and physical knowledge required for embodied execution. It combines depth-map generation, large-scale embodied pre-training data, and extension to GEM-VLA, achieving strong performance across embodied benchmarks.

  • Motivation: Scaling massive visual question-answering datasets improves high-level semantic benchmarks and passive comprehension, but leaves limitations for embodied spatial and physical capabilities.
  • Method: GEM introduces depth-map prediction as an intrinsic generative target to capture fine-grained structural details and complete spatial and geometric relations.Its architecture uses a hybrid autoregressive-diffusion design that blends generative and representational modeling.
  • Data and Extension: GEM-4M provides high-quality, large-scale embodied pre-training data covering physical grounding, spatial-temporal planning, and physical reasoning.The dataset includes extensive embodied question-answering pairs.
  • Results: GEM and GEM-VLA demonstrate remarkable performance across benchmarks spanning recognition through real-world operations, with GEM establishing new state-of-the-art results.GEM consistently outperforms leading open-source general-purpose models and spatial and embodied specialists on key reasoning benchmarks.

2 Related Work

Related work advances embodied intelligence through data-driven reasoning methods and generalist Vision-Language-Action models. These efforts target capabilities such as affordance understanding, spatial reasoning, planning, and instruction execution.

  • Embodied Reasoning: Data-driven methods enhance embodied reasoning through object affordances, object counting, spatial relationship understanding, and action planning.
  • Vision-Language-Action Models: Robotic manipulation has progressed from single-task specialists toward generalist models trained on broad and diverse datasets.This evolution is driven by advances in VLMs and large-scale robot action datasets.
  • Vision-Language-Action Models: VLA models integrate pretrained VLM backbones with robot action output heads to interpret and execute human instructions.They inherit perceptual and linguistic representations from pretrained VLMs and demonstrate improved adaptability and zero-shot capabilities.

3 Method

GEM augments a VLM with a DiT-based depth generator and jointly trains semantic and generative objectives to encode both textual meaning and physical structure. Its method further includes progressive alignment, the GEM-4M grounding/reasoning/planning dataset, and a VLA extension for continuous robot-action prediction.

  • 3.1 Architecture: GEM projects final-layer visual tokens through a connector into a DiT-based depth generator, whose reconstruction objective supplies physical structural supervision alongside semantic VLM training.The connector produces c = Cϕ(h_o), which conditions depth-map generation; the generative loss is combined with cross-entropy so representations retain semantic and structural information.
  • 3.2 Progressive training: Progressive training prevents modality interference by first aligning the connector, then adapting the connector and DiT head, and finally jointly training the VLM, connector, and depth head.The final phase uses L_total = L_CE + λL_flow after the two stabilization phases.
  • 3.3 GEM-4M dataset: GEM-4M is a high-quality, large-scale supervised fine-tuning dataset designed to improve perception and reasoning grounded in physical knowledge.Its data engine covers grounding, spatial reasoning, and action-planning supervision from embodied datasets and annotations.
  • 3.3 GEM-4M dataset: The dataset includes 1M question-answer pairs for grounding tasks such as open-vocabulary detection, instruction-based localization, and object-affordance recognition.These data target object recognition and localization in embodied scenarios.
  • 3.4 VLA extension: GEM extends to a VLA by adding a DiT-based action expert that conditions continuous-action diffusion policies on key–value tokens from multimodal observation histories.The action expert is denoted A_ω and uses the conditioning representation c_act extracted from backbone attention blocks.

4 Experiments

GEM improves spatial reasoning across embodied benchmarks, with depth generative supervision strengthening distance perception and spatial capabilities. Its GEM-VLA extension outperforms baselines in simulation and challenging real-world manipulation, while removing depth supervision reduces performance.

  • Spatiotemporal embodied reasoning: GEM improves performance across CV-Bench, VSI-Bench, MMSI-Bench, and EmbSpatial relative to Qwen3-VL at both model scales.On VSI-Bench and MMSI-Bench, the improvements are roughly 10%.
  • Spatial placement and grounding: The GEM-8B variant achieves the best overall performance on RefSpatial, Where2Place, and RoboSpatial against general-purpose and embodied specialist models.These benchmarks evaluate object placement and referring in embodied environments.
  • Simulation manipulation: GEM-VLA outperforms all baselines across LIBERO’s Spatial, Object, Goal, and Long task suites, indicating improved generalization across diverse manipulation tasks.LIBERO contains 10 tasks per suite with 50 trials per task.
  • Ablation study: Removing depth generative supervision lowers performance, because full GEM integrates global structural and semantic features in a shared representation space.The full models particularly outperform ablated counterparts on distance-related VSI-Bench questions, indicating stronger relative-distance perception.
  • Real-world manipulation: On challenging real-world tasks, GEM-VLA achieves higher success rates across deformable manipulation and a significantly higher average progress score on long-horizon table bussing than π0-FAST and π0.5.The evaluated tasks include cloth folding, zipper unzipping, and table bussing.
  • VLA ablation: Removing depth generative supervision during VLA fine-tuning reduces performance on almost all tasks, showing that auxiliary depth prediction enables more accurate manipulation.The ablation freezes the depth head and trains only the VLM backbone and action expert with the action objective.

5 Ablation Studies

The ablations show that depth supervision is more effective than RGB reconstruction and that progressive training better integrates semantic and structural features than direct end-to-end training. GEM’s depth-generative supervision also produces richer low-level structural representations than standard SFT.

  • 5 Ablation Studies: Replacing depth supervision with RGB reconstruction reduces performance, especially on distance-related VSI-Bench questions.Depth provides more explicit cues about spatial relationships, including relative distance.
  • 5 Ablation Studies: Direct end-to-end training fails to effectively integrate semantic and structural features compared with the proposed progressive three-stage strategy.The comparison uses depth-head and understanding-backbone training on VSI-590K and evaluates the same spatial reasoning benchmarks.
  • 5 Ablation Studies: Qwen3-VL-SFT visual features generate depth maps with limited structural details, indicating representations dominated by high-level semantic signals.These representations lack explicit spatial and geometric information, helping explain standard SFT models’ suboptimal embodied reasoning performance.
  • 5 Ablation Studies: GEM visual features produce high-fidelity depth generation, demonstrating that depth-generative supervision captures low-level structural information from 2D inputs.The resulting structural cues benefit spatial perception in embodied environments.

6 Conclusion · Appendix

GEM bridges high-level semantic reasoning and low-level physical grounding by making depth generation an intrinsic target for scene geometry. Its progressive training recipe jointly improves depth synthesis and language objectives to fuse structural and semantic representations.

  • 6 Conclusion: GEM is introduced as a generative-supervised embodied vision-language framework.The framework targets the gap between semantic reasoning and physical grounding.
  • 6 Conclusion: GEM addresses the gap between high-level semantic reasoning and low-level physical grounding.Its design explicitly connects semantic and physical aspects of embodied intelligence.
  • 6 Conclusion: Depth generation is learned as an intrinsic target for scene geometry.This target supplies structural information within the embodied vision-language framework.
  • 6 Conclusion: A progressive training recipe optimizes depth synthesis and language objectives.The recipe is designed to coordinate the two learning objectives during training.
  • 6 Conclusion: The training recipe better fuses structural and semantic representations.This fusion follows from jointly optimizing depth synthesis and language objectives.
  • 6 Conclusion: The work also constructs a large-scale dataset to support the framework.The supplied passage introduces this dataset construction as part of the paper’s broader contribution, but its details are truncated.

A SimplerEnv Evaluation on WidowX Robot Tasks

The study evaluates GEM in SimplerEnv under the WidowX setup, using four manipulation task suites designed to test visual robustness and precise control. GEM-VLA achieves the highest average performance across all task types on the Simpler WidowX Robot benchmark.

  • Benchmark setup: The Simpler WidowX benchmark evaluates Put Carrot on Plate, Put Eggplant in Basket, Put Spoon on Towel, and Stack Blocks.These four suites assess robustness to visual variations and precise manipulation.
  • Evaluation setup: GEM and its action expert are fine-tuned from scratch on BridgeD following the StarVLA implementation.The evaluation is conducted in the SimplerEnv simulation environment under the WidowX robot setup.
  • Results: GEM-VLA achieves the highest average performance across all task types on the Simpler WidowX Robot benchmark.Table 5 reports success rates for the benchmark and identifies GEM-VLA as the top average performer.

B More Implementation Details · C Details on GEM-4M construction

The section details GEM’s architecture, staged pretraining, real-world action-model finetuning, and baseline reproduction settings. It also identifies the training-stage configuration table and real-world finetuning loss-curve figure.

  • B More Implementation Details: GEM uses Qwen3-VL as its VLM backbone, with a lightweight 2-layer MLP connector and a DiT-based depth generator.These components combine generative supervision with representation learning.
  • B More Implementation Details: GEM follows three training phases: connector initialization, depth-generator initialization, and end-to-end joint training.The strategy progressively initializes components before joint optimization.
  • B More Implementation Details: GEM-VLA uses a flow-based action expert based on RDT2 for real-world robotic-task evaluation.The action model is extended specifically to enable real-world task evaluation.
  • B More Implementation Details: For each task, GEM-VLA uses 200 trajectories, 50k finetuning steps, global batch size 256, action chunk size 32, and three camera views.The views are one top camera and two wrist cameras, left and right.
  • B More Implementation Details: Table 6 reports the detailed configuration for each GEM VLM-pretraining stage.The passage identifies the table as covering training-stage configurations.
  • B More Implementation Details: Figure 6 shows the GEM-VLA loss curves during real-world task finetuning.The figure concerns loss behavior during the described real-world finetuning process.
  • B More Implementation Details: The authors re-implement π0.5 and π0-FAST with the official OpenPI codebase, training both baselines to convergence on 8 GPUs with per-GPU batch size 32.The implementations use the same optimizer and scheduler settings for π0-FAST as for π0.5.
  • B More Implementation Details: The π0.5 reproduction uses discrete state inputs, action horizon 24, 32-dimensional action space, bfloat16 precision, and AdamW optimization.The π0-FAST implementation retains π0.5’s optimizer and scheduler settings.

C.1 Embodied Grounding Data

GEM augments physical-manipulation grounding with 100k high-quality samples generated from open-source robot action datasets, beginning by labeling foreground objects in each video’s first frame.

  • C.1 Embodied Grounding Data: 100k high-quality samples are generated from open-source robot action datasets to enhance grounding in physical manipulation scenarios.The cited sources are Wu et al. (2024), O’Neill et al. (2024), and Khazatsky et al. (2024).
  • C.1 Embodied Grounding Data: The first generation stage extracts each robot-operation video’s first frame and uses Qwen3-VL to identify all foreground object labels.The process follows prompt templates provided in Figure 11.

C.2 Physical, Spatial Reasoning Data · C.3 Spatiotemporal Planning Data

GEM-4M includes dedicated physical-spatial reasoning and spatiotemporal planning data. These resources combine 3D scene-based spatial QA, embodied grounding examples, robot-video subtask QA, and tracked object trajectories for action learning.

  • C.2 Physical, Spatial Reasoning Data: 100k 3D spatial perception samples are constructed from ScanNet, ScanNet++, and ARKitScenes using Open3D point clouds and spatial-semantic metadata.The construction follows methodologies established in VSI-Bench.
  • C.2 Physical, Spatial Reasoning Data: Embodied grounding examples localize instructed target objects, such as bananas, cucumbers, and pots, with highlighted scene bounding boxes.The examples include single-object and multiple-object localization requests.
  • C.2 Physical, Spatial Reasoning Data: The spatial annotations include room dimensions, object centers, category counts, and 3D bounding boxes with rotations, extents, and centers.
  • C.2 Physical, Spatial Reasoning Data: Generated QA pairs cover object counts, absolute and relative distances, object and room sizes, relative directions, and other spatial attributes.The questions follow VSI-Bench templates.
  • C.3 Spatiotemporal Planning Data: Robot videos from public datasets are paired with subtask annotations, and corresponding frames are converted into QA pairs using the RoboVQA template.Representative examples are shown in Figure 9.
  • C.3 Spatiotemporal Planning Data: Trajectory data is generated to teach object motion and action execution by identifying manipulated objects from each subtask description.Qwen3 is used with prompt templates in Figure 12 for object identification.
  • C.3 Spatiotemporal Planning Data: SAM3 segments the manipulated object in the initial frame, whose centroid initializes trajectory tracking before cubic-spline smoothing.The tracking procedure uses the object centroid as the initial state.

D Limitation and Future Work · E Simulation Rollouts Visualization

GEM achieves strong embodied performance but remains limited by model scale and the absence of large-scale robot-data pretraining. The paper also visualizes GEM-VLA’s qualitative simulation rollouts across representative LIBERO and SimplerEnv manipulation tasks.

  • D Limitation and Future Work: GEM still has room to scale in model size and training data despite strong embodied recognition and manipulation performance.The authors identify scaling as an ongoing limitation.
  • D Limitation and Future Work: GEM-VLA has not been pretrained on large-scale robot datasets, limiting its acquisition of richer physical knowledge.Future work will incorporate large-scale robot-data pretraining and further evaluate and refine the methods.
  • E Simulation Rollouts Visualization: The paper presents qualitative visualizations of GEM-VLA policy rollouts on simulation benchmarks.The visualized trajectories consist of key trajectory points representing model-predicted paths for task completion.
  • E Simulation Rollouts Visualization: On LIBERO, GEM-VLA demonstrates successful rollouts across LIBERO-Long, LIBERO-Goal, LIBERO-Object, and LIBERO-Spatial manipulation tasks.These representative rollouts are shown in Figure 13.
  • E Simulation Rollouts Visualization: On SimplerEnv with the WidowX robot setup, rollouts cover Put Carrot on Plate, Put Eggplant in Basket, Put Spoon on Towel, and Stack Blocks.These four task suites are presented in Figure 14.
  • E Simulation Rollouts Visualization: The rollout examples include stacking a green block onto a yellow block, placing a carrot on a plate, and putting a spoon on a towel.These examples correspond to the qualitative manipulation visualizations.
Loading 2605.28548v1…