Source-linked AI summary
Wall-OSS-0.5 Technical Report
Ryan Yu, Pushi Zhang, Starrick Liu, Brae Liu, Miracle Kang, Shalfun Li, Lights Shi, Ellie Ma, Ping Yang, Chris Pan, Jerry Chen, Dongxiu Liu, Rain Sun, Miles Guo, Byron Zhang, Hugo Zhou, Zach Xu, Vincent Chen, Harrison Huang, James Wang, Dance Kuzi, Andy Zhai, Hang Su, Roy Gan, Lucy Liang, Hao Wang, Qian Wang
TL;DR
VLA results are usually reported only after fine-tuning, leaving uncertain whether pretraining itself produces executable robot behavior. Wall-OSS-0.5 uses gradient-bridged co-training and direct physical-robot evaluation, showing zero-shot capability before adaptation and stronger performance afterward while retaining grounded vision-language competence.
Problem
VLA evidence is typically reported after downstream fine-tuning, leaving whether pretraining alone produces an executable real-robot policy unresolved.
Method
Wall-OSS-0.5 combines multimodal cross-entropy, action-token cross-entropy, and flow matching so VLM-native gradients shape control while continuous actions remain the deployment interface.
Results
The pretrained model completes several zero-shot tasks, including held-out Rope Tightening at 82%, while fine-tuning reaches 60.5% average task progress on 15 tasks and surpasses π0.5 by 17.5%.
Takeaways & Limitations
The results support deployment-oriented VLA pretraining as a directly measurable source of robot capability and a stronger downstream adaptation prior.
Takeaways & Limitations
Validation of gradient-bridge dynamics is limited to a 3B VLM backbone, while single-frame inputs constrain long-horizon memory and the tokenizer is tied to a fixed 26-dimensional action representation.
Abstract
from arXiv · showhide
Large-scale Vision-Language-Action (VLA) pretraining is increasingly adopted as the foundation for robot policies, yet the evidence for pretrained VLAs is almost invariably reported after task-specific fine-tuning. This leaves a foundational question unanswered: does VLA pretraining itself yield executable robot behavior, or does it merely furnish a better initialization for downstream policy learning? We present Wall-OSS-0.5, an open-source 4B VLA built upon a 3B VLM backbone augmented with action-generation components, designed so that pretrained robotic capability is directly measurable on physical hardware. The model is pretrained across more than 20 embodiments, processing over one million robot trajectories per epoch alongside a grounded multimodal corpus. We adopt a gradient-bridged co-training recipe in which three objectives play distinct and complementary roles: discrete action prediction routes strong VLM-native gradients into the backbone, multimodal prediction preserves grounded vision-language understanding, and continuous flow matching serves as the deployment-time action interface. Before task-specific fine-tuning, the pretrained checkpoint achieves non-trivial zero-shot real-robot behavior, completing several tasks, including a held-out deformable manipulation task, at high task progress on a 17-task suite. After fine-tuning, the same checkpoint serves as a stronger adaptation prior, reaching 60.5% average task progress on 15 real-robot tasks and outperforming π_0.5 by 17.5%. Multimodal evaluations further confirm that action training does not erode grounded vision-language competence: the model preserves broad vision-language ability while strengthening embodied grounding. Together, these results reposition VLA pretraining from an initialization strategy to a directly testable, already useful source of robot capability.
1 Introduction
Wall-OSS-0.5 addresses whether VLA pretraining alone can produce executable real-robot behavior rather than only a better downstream initialization. Its deployment-oriented design combines gradient-bridged co-training with direct evaluation of pretrained, adapted, and multimodal capabilities.
- Motivation: VLA evidence is typically reported after fine-tuning, leaving whether pretraining alone yields an executable real-robot policy unresolved.The paper frames this as a concrete operational question for physical-robot evaluation.
- Approach: Wall-OSS-0.5 is an open-source 4B VLA built on a 3B VLM backbone with action-generation components and evaluated directly before task-specific adaptation.The deployment-oriented criterion requires useful out-of-the-box manipulation, retained instruction-grounded VLM competence, and a useful downstream prior.
- Results: Several zero-shot tasks reach high progress before fine-tuning, including Block Sorting at 100%, Fruit Sorting at 96%, Ring Stacking at 86%, and Rope Tightening at 82%.Rope Tightening is a held-out deformable task in the 17-task real-robot suite.
- Results: After fine-tuning, Wall-OSS-0.5 reaches 60.5% average task progress on 15 real-robot tasks, surpassing π0.5 by 17.5%.The margin widens to 26% on the 10-task manipulation subset.
- Approach: Three objectives jointly shape the policy: multimodal cross-entropy preserves grounded understanding, action-token cross-entropy bridges VLM gradients to control, and flow matching trains deployment actions.The discrete pathway primarily supports training, while the continuous pathway provides the deployment-time action interface.
- Results: Multimodal evaluation shows stable overall performance alongside a 21.8% gain on embodied grounding, while deployment optimization sustains 15 Hz control at high input resolution.The inference stack combines CUDA Graph capture and custom fused kernels for a reported 4× speedup over PyTorch eager mode.
2 Methods
Wall-OSS-0.5 combines discrete action-token training, multimodal prediction, and continuous flow matching so the VLM backbone becomes action-aware while continuous actions remain the deployment interface. Its tokenizer, action-space supervision, and optimized inference stack target semantic action representations, stable training, and real-time robot control.
- Backbone Routing: The Mixture-of-Transformers routes vision, language, proprioception, and discrete actions through the VL Expert while continuous action signals use the Action Expert.Shared attention context preserves end-to-end gradient flow, so the expert split is a routing decomposition rather than gradient isolation.
- Vision-Aligned RVQ Action Tokenizer: The Vision-Aligned RVQ tokenizer compresses observation-conditioned delta-action chunks into structured discrete tokens for backbone training rather than merely low-distortion reconstruction.Its codebooks capture coarse motion at early levels and fine residual corrections at later levels, while auxiliary objectives align action latents with visual features and future observations.
- Gradient-Bridged Co-Training: Three jointly trained objectives give discrete action prediction the gradient-bridging role, multimodal prediction the vision-language preservation role, and flow matching the deployment-time action role.The discrete pathway primarily shapes the backbone during training, while the continuous pathway generates action chunks at inference.
- Action-Space Supervision: Action-space supervision emphasizes high-noise flow-matching steps, where the global action trajectory is formed, and improves convergence speed, peak performance, and training stability.The induced weighting is motivated by the smooth, low-frequency structure of robot trajectories.
- Action Interface: At inference, discrete tokens are not decoded into executable actions; multi-step flow-matching denoising generates continuous action chunks from the action interface.The action-token and continuous-action query tokens feed separate pathways, with the discrete pathway confined to training-time gradient transfer.
- Inference Optimization: CUDA Graph capture removes CPU dispatch bubbles during denoising, while fused kernels eliminate intermediate materialization and deliver 2–10× speedups over native implementations.On an RTX 5090, the optimized stack reaches approximately 21 Hz at 224×224 and 15 Hz at 448×448, yielding a 4× end-to-end speedup over PyTorch eager mode.
3 Data Recipe and Management
Wall-OSS-0.5 combines diverse self-collected and open-source robot data with multimodal and embodied-bridge supervision. Its preprocessing unifies heterogeneous action data, improves quality and balance, and aligns understanding with executable behavior.
- Data composition: The pretraining mixture combines self-collected manipulation data, open-source multi-embodiment datasets, and targeted multimodal corpora balanced across task, embodiment, and source diversity.The self-collected corpus spans tabletop bimanual systems, mobile manipulators, and the embodiment-free XRZero-G0 device.
- Data composition: XRZero-G0 enables low-cost collection across diverse scenes without tying data to a specific robot morphology, expanding environmental and task diversity.The corpus also covers household, industrial, and office environments, plus controlled collection rooms.
- Preprocessing: Action-space unification maps heterogeneous datasets into shared schemas for end-effector poses, joint positions, grippers, mobile bases, lifts, waists, and heads.For joint-only datasets, end-effector poses are recovered with forward kinematics, while physical action semantics are normalized across embodiments.
- Preprocessing: Timestamp alignment, error repair, and stationary-frame filtering reduce mismatched supervision, recording defects, and redundant pauses in training data.Stationary-frame filtering removes near-identical observations paired with inconsistent near-zero and non-zero actions and produces more compact execution cadence.
- Sampling: Power sampling with source-task groups and per-group caps mitigates inter-source and intra-source imbalance while allocating over one million trajectories per epoch.The sampled epoch contains approximately 60% self-collected and 40% open-source trajectories.
- Multimodal data: Embodied bridge data is generated directly from action trajectories and organizes supervision around object, spatial, scene, and task understanding.This construction explicitly connects multimodal understanding to action prediction using data aligned with the same trajectories and task contexts.
4 Experiments and Results
Pretraining produces measurable zero-shot robot capability, including transfer to held-out tasks, while fine-tuning further strengthens performance across real-robot benchmarks. Co-training also shifts multimodal competence toward embodied grounding, with a documented trade-off in open-domain VQA.
- Zero-shot capability: At the 400k checkpoint, six tasks reach at least 60% task progress without task-specific fine-tuning, including four above 80% and two held-out tasks.Rope Tightening, an unseen deformable task, reaches 82%.
- Zero-shot capability: 72.6% average task progress makes semantic understanding the strongest zero-shot capability dimension, while precision-demanding categories remain harder.High-performing tasks include Block Sorting at 100%, Fruit Sorting at 96%, and Ring Stacking at 86%.
- Zero-shot capability: Unseen-task progress rises from 24.2% to 53.6%, alongside seen-task progress rising from 26.1% to 50.0%, despite checkpoint fluctuations.The authors interpret the parallel improvement primarily as cross-scene and cross-prop transfer, not entirely novel skill learning from scratch.
- Zero-shot capability: Zero-shot performance remains below 20% on Towel Folding, Table Setting, and Charger Plugging, whose precision, state-perception, and isolated action patterns limit transfer.These tasks are characterized as beyond current zero-shot reach and requiring fine-tuning.
- Fine-tuned performance: 60.5% average task progress after fine-tuning surpasses π0.5 by 17.5%, with Wall-OSS-0.5 leading on 10 of 15 tasks.The model leads π0.5 on both manipulation and reasoning subset averages.
- Fine-tuned performance: Fine-tuning amplifies pretrained capabilities: Color Block Sorting and Ring Stacking remain strongest, while Drawer Organization improves from a low zero-shot baseline to 52%.The evidence links strong post-fine-tuning results both to existing pretrained capability and to adaptation gains on weaker tasks.
- Fine-tuned performance: Scaling the fine-tuning task set improves shared-task performance, raising the five-task average from 73.96% to 83.75% across 5-to-19-task scaling.The ten-task shared subset also rises from 59.98% to 64.78% when scaling from 10 to 19 tasks.
- Embodied multimodal understanding: Co-training shifts capability toward embodied perception: Embodied Grounding improves by 21.8 points, Where2Place by 11.0, and EO-Bench by 3.9.The authors attribute the embodied gains to robot-view grounding and spatial-decision supervision from bridge samples.
5 Ablation Studies
The ablations support gradient-bridged co-training as the strongest training strategy, while action-space supervision and the Vision-Aligned RVQ tokenizer improve robot performance under controlled comparisons.
- Co-Training Strategies: 57.0% average task progress made co-training outperform flow-only (36.6%), stop-gradient (31.9%), and stop-gradient to co-training (49.6%) on 5 ablation tasks.The comparison used identical configurations for 70k steps; the tasks were selected for tractable from-scratch training.
- Co-Training Strategies: Removing the gradient bridge or substituting flow-only or two-stage training degraded real-robot performance by 7.4–25.1 percentage points.Co-training jointly uses action-token cross-entropy, flow matching, and multimodal cross-entropy with gradients reaching the VLM backbone.
- Co-Training Strategies: VQA scores stayed tightly clustered across strategies, with stop-gradient marginally ahead, while co-training produced stronger action-task performance.This indicates the action-performance differences were not accompanied by a large separation in VQA scores.
- Action-Space Loss: 96.5% peak average success rate at 25k steps was achieved by action-space loss, exceeding velocity-space loss by 6.2%.Action-space loss also reached 95.8% average success rate at 20k steps, while velocity-space loss remained below 90.3% after 35k steps.
- Vision-Aligned RVQ Tokenizer: 77.5% VQA accuracy with the RVQ tokenizer improved over 75.7%, indicating preserved and strengthened multimodal understanding.The reported gains are attributed to auxiliary visual-action alignment and next-frame prediction objectives used to train the tokenizer.
- Vision-Aligned RVQ Tokenizer: 48.1% average task progress with the RVQ tokenizer exceeded FAST’s 29.3% under identical co-training settings.The tokenizer comparison evaluated four real-robot tasks; continuous actions were generated through flow matching rather than directly decoded discrete tokens.
6 Related Work
Related work spans continuous and discrete action representations, architectures for integrating action generation with pretrained VLMs, and increasingly broad multi-embodiment robot datasets. Wall-OSS-0.5 combines these directions through per-layer VL and Action Experts with shared attention and a deployment-oriented data mixture.
- Action Representations: Prior VLA systems model continuous actions through diffusion or flow matching, while RT-2 and OpenVLA exemplify discrete text-like action tokens.Wall-OSS-0.5 focuses on discrete and latent action representations alongside continuous action generation.
- Action-Ready VLM Architectures: Architectural approaches range from training generalist controllers from scratch to reusing pretrained VLMs with modular or embodiment-specific action components.Examples include RT-1, Gato, Octo, RDT-1B, CogACT, and HPT.
- Action-Ready VLM Architectures: Wall-OSS-0.5 extends this design space with a Mixture-of-Transformers architecture containing VL and Action Experts at every Transformer layer.The experts retain separate parameters but interact through shared attention.
- Robot Data and Embodied Grounding: Robot datasets such as Open X-Embodiment, DROID, RoboMIND, RoboCOIN, and AgiBot World expand available scale and embodiment diversity.Wall-OSS-0.5 uses over one million trajectories per epoch across more than 20 embodiments and organizes the mixture around deployment rather than scale alone.
7 Discussion and Limitations
The discussion argues that large-scale VLA pretraining can yield measurable real-world behavior and a stronger downstream adaptation prior. It links this outcome to gradient bridging, MoT routing, and embodied bridge data, while identifying scale, temporal, action-space, evaluation, and deployment boundaries.
- Discussion: Large-scale VLA pretraining can produce measurable real-world robotic behavior while also providing a stronger prior for downstream adaptation.The claim is presented as an operational conclusion of the report.
- Design Implications: Discrete action-token cross-entropy shapes the backbone toward controllable representations even when deployment uses continuous actions.Continuous-only training exposes the backbone primarily to a weaker residual flow-matching signal.
- Design Implications: MoT routing and embodied bridge data jointly transform a pretrained multimodal model into an executable policy without collapsing it into a narrow task-specific controller.MoT preserves gradient flow while allocating action-specific capacity, and bridge data makes multimodal training more action-aware.
- Limitations: The gradient-bridge dynamics have only been validated with a 3B VLM backbone, so scaling to larger backbones may alter interactions among the training signals.This is the first limitation identified by the authors.
- Limitations: Single-frame inputs constrain long-horizon tasks, while the fixed 26-dimensional action representation limits direct applicability to dexterous and other high-DoF embodiments.The benchmark also lacks multi-robot collaboration, long-duration deployment, and broader open-world interaction settings.
8 Contributors
Wall-OSS-0.5 is presented as a collaborative effort by the X Square Robot team, with contributors and project roles listed in the report.
- Contributors: The report identifies Wall-OSS-0.5 as a collaborative effort of the X Square Robot team.The full contributor list follows the statement.
- Contributors: The contributor list marks core contributors with ∗, the project lead with †, and the corresponding author with ‡.The listed contributors include Ryan Yu, Pushi Zhang, Lucy Liang, Hao Wang, and others.
A.1 Pretrained Model Zero-Shot Evaluation Per-Task Results
The zero-shot evaluation reports per-task progress for seen and unseen tasks across milestone checkpoints, using standardized trajectory-based scoring. Results are organized by task categories including semantic understanding, rigid-object, deformable-object, fine-grained, and long-horizon manipulation.
- Task coverage: Seen and unseen tasks are reported separately in Tables 5 and 6 across milestone checkpoints.The tables list per-task task progress for all seen and unseen tasks.
- Evaluation protocol: Each task is evaluated over 10 trajectories using predefined scoring rubrics with a maximum score of 100.
- Task categories: The reported task categories cover semantic understanding, rigid-object, deformable-object, fine-grained, and long-horizon multi-step manipulation.
- Seen tasks: Table 5 identifies the highest task progress across checkpoints for each seen task.
- Unseen tasks: Table 6 identifies the highest task progress across checkpoints for each unseen task.
A.2 Multimodal Understanding Evaluation Detailed Results
The evaluation materials provide detailed benchmark and task descriptions alongside per-task results and a unified real-robot scoring protocol. Task progress is measured step by step across manipulation stages, while success rate requires completing every step.
- Benchmark details: Table 7 reports detailed scores and descriptions for each benchmark, including an internally constructed Embodied Grounding benchmark sampled and annotated from robot action trajectories.
- Fine-tuning results: Table 8 compares fine-tuning per-task task progress under identical data and evaluation protocols, using approximately 500 trajectories per task.
- Multi-task results: Table 9 reports detailed multi-task fine-tuning task progress, with dashes marking tasks excluded from a configuration.
- Real-robot protocol: Real-robot evaluation covers 31 tasks under a unified step-wise scoring protocol.Each task has a maximum score of 10 accumulated across key manipulation stages.
- Real-robot protocol: Task progress = actual score/maximum score × 100, providing a fine-grained evaluation metric.
- Real-robot protocol: Success rate is the proportion of trajectories that fully complete all task steps, with each task evaluated over 10 trajectories.