Source-linked AI summary
Igniting VLMs toward the Embodied Space
Andy Zhai, Brae Liu, Bruno Fang, Chalse Cai, Ellie Ma, Ethan Yin, Hao Wang, Hugo Zhou, James Wang, Lights Shi, Lucy Liang, Make Wang, Qian Wang, Roy Gan, Ryan Yu, Shalfun Li, Starrick Liu, Sylas Chen, Vincent Chen, Zach Xu
TL;DR
Existing VLMs have limited spatial and embodiment understanding, while embodied action comprehension and generation remain bottlenecks. WALL-OSS addresses these gaps with a tightly coupled VLA architecture, multimodal training curriculum, and Unified Cross-Level CoT. It reports strong long-horizon manipulation, instruction-following, reasoning, and baseline-comparison results.
Problem
Existing VLMs have limited spatial and embodiment understanding, and transferring them to embodied domains exposes mismatches in modalities, pretraining distributions, and training objectives.
Method
WALL-OSS combines a tightly coupled architecture and multi-strategy curriculum with Unified Cross-Level CoT for reasoning, subgoal decomposition, and fine-grained action synthesis.
Results
WALL-OSS achieves high success on complex long-horizon manipulations, strong instruction-following and reasoning performance, and outperforms strong baselines.
Takeaways & Limitations
The results support tightly coupled multimodal reasoning and action modeling as a scalable approach to embodied intelligence.
Abstract
from arXiv · showhide
While foundation models show remarkable progress in language and vision, existing vision-language models (VLMs) still have limited spatial and embodiment understanding. Transferring VLMs to embodied domains reveals fundamental mismatches between modalities, pretraining distributions, and training objectives, leaving action comprehension and generation as a central bottleneck on the path to AGI. We introduce WALL-OSS, an end-to-end embodied foundation model that leverages large-scale multimodal pretraining to achieve (1) embodiment-aware vision-language understanding, (2) strong language-action association, and (3) robust manipulation capability. Our approach employs a tightly coupled architecture and multi-strategies training curriculum that enables Unified Cross-Level CoT-seamlessly unifying instruction reasoning, subgoal decomposition, and fine-grained action synthesis within a single differentiable framework. Our results show that WALL-OSS attains high success on complex long-horizon manipulations, demonstrates strong instruction-following capabilities, complex understanding and reasoning, and outperforms strong baselines, thereby providing a reliable and scalable path from VLMs to embodied foundation models.
X SQUARE ROBOT
WALL-OSS is an end-to-end embodied foundation model designed to improve spatial and embodiment understanding, language–action association, and manipulation. Its tightly coupled architecture and multi-strategy curriculum unify reasoning, subgoal decomposition, and fine-grained action synthesis.
- WALL-OSS uses large-scale multimodal pretraining to target embodiment-aware vision–language understanding, language–action association, and robust manipulation.
- Unified Cross-Level CoT combines instruction reasoning, subgoal decomposition, and fine-grained action synthesis in one differentiable framework.
- WALL-OSS achieves high success on complex long-horizon manipulations and strong instruction-following, understanding, and reasoning performance.
- The model outperforms strong baselines, supporting a reliable and scalable path from VLMs to embodied foundation models.
1 Introduction
Existing VLMs remain limited in embodied understanding and executable action generation because embodied domains expose gaps in modalities, data, pretraining distributions, and training objectives. WALL-OSS addresses these gaps with tightly coupled architecture, curriculum design, embodied supervision, and unified reasoning-to-action mapping.
- Embodied action understanding and generation remain a core bottleneck because current vision-language systems are largely disembodied and do not generate executable actions.
- Aligned vision–language–action data are scarce and heterogeneous, making from-scratch action training struggle to generalize.
- Naive action-head fine-tuning can cause weight drift, while loosely coupled knowledge-insulation designs restrict semantic–control binding and weaken action execution outside the vision-language prior.
- Embodied settings create gaps between compressed visual-text representations and continuous 3D space-time actions, motivating distinct action modeling objectives.
- WALL-OSS uses a tightly coupled Mixture-of-Experts architecture, staged action modeling, embodied VQA, discretized action priors, and unified CoT mapping to bridge these gaps.
- The framework reports state-of-the-art performance across diverse manipulation benchmarks, especially for instruction following, compositional reasoning, and long-horizon control.
2 Related Work
Related work transfers VLM priors to robot control through multimodal co-training, action modeling, and hierarchical reasoning. WALL-OSS combines discrete action priors, continuous control, tightly coupled routing, and a single-model reasoning-to-action formulation.
- Foundation models in language, vision and manipulation: WALL-OSS follows a discrete-priors-to-continuous-control recipe, first implanting discrete action priors and then optimizing a flow-matching head for high-frequency actions.
- Foundation models in language, vision and manipulation: A tightly coupled Mixture-of-Experts with static routing enforces stronger language–action binding.
- Multimodal co-training with robot data: Multimodal and cross-source co-training combines web image–text, dialogs, long videos, and multi-embodiment robot data to preserve open-world competence while injecting embodied semantics.
- Language reasoning and subtask decomposition: Unlike pipeline designs with separate planners and controllers, WALL-OSS uses one model for instruction, chain-of-thought, subtasks, and continuous actions.
- Language reasoning and subtask decomposition: The unified design can include or skip intermediate reasoning and interleave reasoning with execution, while the curriculum and routing reduce interface-induced error accumulation.
- Figure 2 distinguishes transfer paradigms using initialized pretrained-VLM weights and separates Discrete Action Modeling from Continuous Action Modeling.
3 WALL-OSS
WALL-OSS is an end-to-end embodied foundation model combining a tightly coupled architecture with staged training for spatial understanding and action generation. Unified Cross-Level CoT links instruction reasoning, subtask planning, and continuous actions within one differentiable model.
- Architecture: WALL-OSS uses a transformer-based architecture that conditions on vision and language while producing phase-specific outputs.The model uses egocentric and arm-mounted camera views with text instructions as multimodal inputs.
- Training: Training proceeds from Inspiration to Integration, first strengthening embodied vision-language and discrete action priors, then replacing discrete prediction with continuous flow-matching control.Integration freezes the VLM while training the flow head before joint optimization.
- Architecture: A Mixture-of-Experts design assigns task-specific feed-forward networks to form a tightly coupled cross-modal structure.The architecture routes action-centric features to an Action FFN and vision-language features to a Vision-Language FFN.
- Unified Cross-Level CoT: Unified Cross-Level CoT spans instruction, reasoning, subtask plans, and continuous actions, while allowing full-chain or direct instruction-to-action mappings.The path-drop objective supports conditioning on or bypassing intermediate reasoning within one end-to-end model.
- Unified Cross-Level CoT: Uni-CoT adaptively invokes subtask decomposition and can interleave reasoning with execution for asynchronous control.The model may emit actions for completed subtasks while continuing to reason during inference.
4 Data Composition
WALL-OSS uses a multisource embodied dataset exceeding 10,000 hours, combining self-collected actions, open-source trajectories, and multimodal VQA. The corpus spans diverse platforms, scenes, task horizons, and spatial-temporal reasoning annotations, with standardized data processing for joint training.
- Sources: The corpus exceeds 10,000 hours and combines self-collected robot actions, open-source actions, and multimodal VQA.These sources target task complexity, cross-morphology and cross-environment generalization, and language-vision preservation.
- Platforms and scenes: The dataset covers desktop arms, mobile stands, wheeled bi-arm systems, and wheeled humanoids across household manipulation scenes.Scenes include kitchen cleaning, dressing and organizing, mobile pick-and-place, and assembly.
- Task spectrum: Two task families span short-horizon precision manipulation and long-horizon routines requiring decomposition, progress tracking, and real-time decisions.The long-horizon tasks have clear goals but implicit procedures.
- Annotation and quality control: A multi-model annotation pipeline with human spot checks supplies fine-grained trajectory steps and CoT-style supervision.Quality control includes timestamp synchronization, outlier filtering, manual audits, and augmentation.
- Standardization: The corpus standardizes coordinate frames, units, morphology, perception inputs, and timestamps across heterogeneous robot platforms.Missing joints are handled through masking or placeholders in a shared expressive DoF template.
- Multimodal VQA: General VQA maintains language-vision competence, while embodied VQA targets spatial-temporal understanding, task reasoning, perception, and affordances.The unified labels explicitly link instruction, CoT, subtasks, and continuous actions for Uni-CoT and the coupled MoE architecture.
5.1 Setup
The evaluation suite measures embodied understanding, reasoning, planning, execution, and robustness across VQA and six manipulation tasks. It includes novel-task adaptation, zero-shot instruction following, baseline comparisons, and blind third-party testing.
- Evaluation suite: The suite combines an Embodied VQA benchmark with six robotic manipulation tasks covering instruction understanding, long-horizon planning, action accuracy, and robustness.The VQA benchmark evaluates scene captioning, object grounding, and action planning.
- Task design: Set-Table, Tidy-Bedroom, and Place-by-Color are novel tasks unseen during pre-training, testing adaptation to new task formulations.
- Task design: The task suite spans single-instruction, reasoning, and long-horizon settings, including zero-shot Instruction-Pick-Place and multi-stage Set-Table and Tidy-Bedroom.
- Reasoning tasks: Place-by-Color tests direct visual matching and text-based color reasoning, while Block-Spell requires object identification or arithmetic inference before spelling with blocks.
- Baselines and protocol: WALL-OSS is compared with π0 and Diffusion-Policy under flat and GPT4-Subtask instruction paradigms, with matched supervision and backbone considerations.
- Baselines and protocol: Blind third-party evaluators used standardized environments, initial states, and scoring rubrics without knowing which model version they tested.
5.2 Results and Analysis
WALL-OSS improves embodied scene understanding, zero-shot instruction following, action accuracy, generalization, and reasoning-intensive manipulation. Its gains are especially evident under distribution shift, reduced data, and tasks requiring explicit decomposition or multimodal grounding.
- Embodied scene understanding: WALL-OSS improves embodied scene understanding over Qwen2.5-VL-3B on the Embodied VQA benchmark, particularly for object grounding and scene captioning.
- Zero-shot instruction following: WALL-OSS demonstrates zero-shot interpretation and execution of manipulation instructions without task-specific fine-tuning.
- Zero-shot instruction following: 85% average task progress on seen-object instructions falls to 61% on novel-object instructions, with failures mainly involving grasp or placement pose inaccuracies.
- Action accuracy and generalization: 100% ID success on Collect-Waste with 1000 demonstrations is achieved by WALL-OSS and π0, versus 80% for Diffusion-Policy.
- Action accuracy and generalization: With 500 demonstrations on Pick-Place-Cup, pre-trained models remain above 90% success while Diffusion-Policy drops below 20%; in OOD Collect-Waste, WALL-OSS and π0 stay above 80%.
- Long-horizon planning: WALL-OSS jointly predicts actions and subtasks, using contextual subtask cues to disambiguate options and maintain coherent progress in long-horizon tasks.
- Reasoning and decomposition: CoT substantially improves the text-based Place-by-Color condition, while Block-Spell baselines show near-zero progress without task decomposition.
- Multimodal co-training: Multimodal co-training yields the highest fine-grained instruction-following accuracy, while action-only WALL-OSS remains above π0, whose accuracy is close to random chance.
6 Discussion
WALL-OSS addresses the language–vision–action gap through tightly coupled architecture, multimodal curriculum, and staged training. The discussion reports broad gains in grounding, reasoning, and OOD control while noting that precise manipulation remains a relative weakness and route efficiency is dynamic.
- Unified architecture: WALL-OSS uses tightly coupled MoE routing, multimodal curriculum, and multi-stage training to map instructions through CoT and subtasks to actions.
- Unified architecture: The framework aims to improve embodied VL understanding and action generation together without heavily disrupting pretrained vision-language priors.
- Empirical findings: WALL-OSS reports enhanced embodied grounding and reasoning, competitive action generation, and stronger OOD generalization than Diffusion-Policy, while π0 remains superior for precise manipulation.
- Embodied modeling challenge: Action modeling must connect high-level state analysis, intent formation, and task decomposition with low-level spatial perception and control.
- Future directions: The paper treats strictly end-to-end VL→A learning and intermediate-route approaches as convergent paths whose relative efficiency toward AGI remains dynamic.