Wall-OSS-0.5 Technical Report

1 Introduction

• The work frames deployment-oriented VLA pretraining around directly executable manipulation skills, retained instruction-grounded VLM competence, and priors that make downstream adaptation more sample-efficient.This addresses the gap between VLA results typically reported after fine-tuning and the question of whether pretraining alone yields an executable real-robot policy.Source: §1 · p. 1
• Before task-specific fine-tuning, the model completes several tasks at high progress in a 17-task real-robot suite, including Block Sorting (100%), Fruit Sorting (96%), Ring Stacking (86%), and held-out Rope Tightening (82%).Held-out tasks track the learning curves of seen tasks during pretraining, indicating reusable manipulation capability rather than task-template memorization.Source: §1 · pp. 1–2, Figure 1 · p. 2
• 60.5% average task progress on 15 real-robot tasks, surpassing π0.5 by 17.5% and widening the margin to 26% on the 10-task manipulation subset; embodied grounding gains 21.8% with stable overall multimodal performance.These results show the pretrained checkpoint also serves as a stronger adaptation prior after fine-tuning.Source: §1 · p. 2
• Gradient-bridged co-training uses action-token cross-entropy to steer the VLM backbone toward control, multimodal cross-entropy to anchor grounding, and flow matching to train continuous deployment actions.Action-token gradients are strong and aligned with flow matching, while multimodal updates complement action optimization; ablations show all three components are jointly necessary.Source: §1 · pp. 2–3, Figure 2 · p. 4
• The recipe combines Mixture-of-Transformers routing, a Vision-Aligned RVQ Action Tokenizer, and Action-Space Supervision while retaining end-to-end gradient flow and the deployment-time action interface.The VL and Action Experts route different token types; Action-Space Supervision applies loss in raw action space to accelerate convergence and stabilize continuous generation.Source: §1 · p. 3, Figure 2 · p. 4
• Single-stage training mixes self-collected manipulation data, open-source multi-embodiment trajectories, and a 90M-sample multimodal corpus, including embodied bridge samples synthesized from action trajectories.Robot trajectories teach execution, multimodal samples preserve grounded understanding, and embodied bridge samples connect the two.Source: §1 · p. 3
• CUDA Graph capture and custom fused kernels deliver 4× end-to-end speedup over PyTorch eager mode and sustain 15 Hz control at high input resolution.The open-source 4B VLA is built on a 3B VLM backbone and is evaluated directly as a pretrained real-robot policy.Source: §1 · pp. 1, 3

2.1.1 Backbone Routing

• Three experts’ input streams route through the VL Expert, while noisy continuous actions use an Action Expert that shares sequence-level attention and receives flow-matching gradients.Attention masking makes discrete and continuous action tokens mutually invisible in the forward pass, allowing their pathways to be evaluated independently.Source: §2.1.1 · pp. 4–5

2.1.2 Vision-Aligned RVQ Action Tokenizer

• The learned Vision-Aligned RVQ tokenizer encodes observation-conditioned delta-action chunks into discrete tokens that are reconstructable, visually aligned, and physically smooth.Its objectives include visual-action alignment, next-frame prediction, and DCT-domain reconstruction; the tokenizer is designed to expose structured action semantics for VLM training, not only low-distortion reconstruction.Source: §2.1.2 · p. 5, Figure 3 · p. 5

2.2.1 Gradient-Bridged Co-Training

• Joint training uses action-token cross-entropy to bridge gradients into the backbone, multimodal cross-entropy to anchor vision-language understanding, and flow matching for inference-time actions.The flow-matching contribution to backbone updates stabilizes at roughly 5%; action and multimodal data are mixed 9:1, and λ_act = λ_mm = 0.01 scale the cross-entropy losses against the smaller flow loss.Source: §2.2.1 · pp. 5–6

2.2.2 Action-Space Supervision

• Action-Space Supervision emphasizes high-noise timesteps, where global trajectory shape is formed, and improves convergence speed, peak performance, and training stability in a controlled ablation.It retains velocity prediction but defines loss on the recovered action, equivalent to a (1 − τ)^2-weighted velocity-space loss.Source: §2.2.2 · p. 6

2.2.3 Action Interface

• At inference, the Action Expert generates continuous action chunks through multi-step flow-matching denoising; discrete action tokens provide a training signal, not executable outputs.The model accepts arbitrary camera views, goal- or step-level instructions, and perturbed or dropped proprioception during training. Its 26-dimensional action space uses relative positions, 6D rotations, gripper state, base velocity, lift height, and head actuation over a one-second horizon.Source: §2.2.3 · p. 7

2.2.4 Optimization and Configuration

• Muon updates expert matrix parameters while AdamW handles visual embeddings and the language-model head; distributed DMuon reduces Muon’s added optimizer overhead by approximately 100×.DMuon assigns matrices to owner ranks, performs Newton–Schulz updates there, and overlaps asynchronous state broadcasts with forward computation.Source: §2.2.4 · p. 7

2.3 Inference Optimization

• Approximately 21 Hz at 224×224 and 15 Hz at 448×448, with T=10, deliver a 4× speedup over PyTorch eager mode on one RTX 5090.The inference pipeline captures each denoising step as a CUDA Graph and fuses remaining operators into custom CUDA kernels.Source: §2.3 · p. 8

3.1 Robot Manipulation Data

• The manipulation corpus combines diverse self-collected and curated open-source data, mapped to a shared action schema with timestamp-synchronized camera streams.Platform-specific camera counts and viewpoints are retained. The mixture is balanced across task diversity, embodiment coverage, and data sources.Source: §3 · p. 8, §3.1 · p. 8

3.1.1 Self-Collected Robot Manipulation Data

• Self-collected data spans tabletop bimanual and mobile manipulators, plus XRZero-G0, an embodiment-free device for collecting data across diverse scenes.It covers household, industrial, office, and controlled-room scenes, with tasks labeled by complexity, duration, and special attributes. Episode-level brief and detailed instructions, expanded with paraphrases, complement atomic sub-trajectory instructions for fine-grained supervision.Source: §3.1.1 · pp. 8–9

3.1.2 Open-Source Multi-Embodiment Data

• Open-source manipulation data extends embodiment and scene coverage through unified curation, metadata verification, field mapping, and filtering for continuity, plausibility, and frame anomalies.Figure 4 describes 10 retained subsets and their resulting embodiment coverage.Source: §3.1.2 · p. 9, Figure 4 · p. 9

3.1.3 Data Preprocessing and Sampling Strategy

• Action preprocessing standardizes physical conventions and representations, aligns video with the nearest-timestamp state when exact matches are unavailable, and repairs or discards detected recording errors.Near-stationary frames paired with inconsistent actions are filtered to reduce supervision noise and redundant pauses during execution.Source: §3.1.3 · p. 10

3.2 Multimodal Data

• Multimodal data totals approximately 90 million samples, including 78 million open-source samples and 12 million embodied bridge samples constructed from action trajectories.The bridge data uses the same trajectories, observations, and task contexts as action learning, spanning object, spatial, scene, and task understanding objectives. Open-source data covers general vision-language, embodied perception, and embodied cognition, with model-based and human quality control.Source: §3.2 · pp. 10–12, Figure 5 · p. 11

4.1.1 Results

• Before fine-tuning, six tasks reach at least 60% progress at 400k, including four above 80% and two unseen tasks; Rope Tightening reaches 82%.The zero-shot suite covers 17 tasks: 12 seen and 5 held-out tasks, scored for task progress over 10 trajectories.Source: §4.1.1 · p. 12, §4.1 · p. 12, Table 3 · p. 14

4.1.2 Analysis

• Unseen-task progress rises from 24.2% to 53.6% and seen-task progress from 26.1% to 50.0%; semantic understanding averages 72.6% at 400k.The authors interpret parallel improvement as evidence of cross-scene and cross-prop transfer, while noting that held-out tasks may have semantically related experience in the pretraining corpus and are not entirely novel skills.Source: §4.1.2 · pp. 12–13, Figure 6 · p. 13
• Zero-shot performance remains uneven: Towel Folding scores 10%, Table Setting 9%, and Charger Plugging 9%, while several tasks show sharp gains at later checkpoints.The authors attribute the lowest scores to demanding precision and state perception, plus action patterns that are relatively isolated in the dataset; they report Block Sorting rising from about 50% to 100% and Fruit Sorting from 61% to 96%.Source: §4.1.2 · p. 14

4.2.1 Baseline Comparison

• 60.5% average task progress beats π0.5’s 43.0% by 17.5% and DreamZero’s 33.4% by 27.1%, with the highest score on 10 of 15 tasks.Wall-OSS-0.5 also leads on manipulation (61.1% vs. 35.0%) and reasoning (59.3% vs. 58.9%).Source: §4.2.1 · p. 14, Table 4 · p. 15
• Wall-OSS-0.5 leads π0.5 on three of five reasoning tasks and on examples including Color Block Sorting (96% vs. 42%) and Drawer Organization (52% vs. 7%).Color Block Sorting and Ring Stacking were already strong zero-shot tasks, whereas Drawer Organization illustrates fine-tuning’s amplification of a low zero-shot baseline; Pencil Case Packing remains difficult at 18.5%.Source: §4.2.1 · p. 15

4.2.2 Multi-Task Fine-Tuning Scaling

• 83.75% average progress on the shared 5-task subset under 19-task fine-tuning exceeds 73.96% under 5 tasks; the shared 10-task subset rises from 59.98% to 64.78%.The nine added tasks also reach 65.59% average progress despite differing backgrounds and embodiment distributions; the authors suggest broader coverage of reusable action, language, and state patterns may explain the gains.Source: §4.2.2 · pp. 15–16, Figure 7 · p. 16

4.3 Embodied Multimodal Understanding

• Co-training improves embodied benchmarks relative to Qwen2.5-VL-3B: Embodied Grounding +21.8, Where2Place +11.0, and EO-Bench +3.9, while RealWorld VQA falls by 15.0 and ERQA by 5.5.Embodied Grounding has a 9.0 baseline score; the authors say 12 million bridge samples provide robot-view grounding and spatial-decision supervision to counterbalance action-token training pressure.Source: Figure 8 · p. 17, §4.3 · p. 17
• In embodied-scene examples, Wall-OSS-0.5 more accurately localizes manipulation targets and selects actionable placement positions than the Qwen2.5-VL-3B backbone.The backbone is more likely to drift toward visually salient but less task-relevant areas.Source: §4.3 · p. 17

5.1 Evaluation of Co-Training Strategies

• 57.0% average task progress with co-training exceeded flow-only (36.6%), stop-gradient (31.9%), and stop-gradient to co-training (49.6%) on 5 ablation tasks.The from-scratch strategies were trained for 70k steps; these tasks were selected for tractable training, so their absolute results are not directly comparable to the 17-task pretrained evaluation.Source: §5.1 · pp. 17–18, Figure 10 · p. 19
• Stop-gradient marginally led on VQA but had the lowest action-task score, while co-training converged faster on flow loss and was adopted for pretraining and fine-tuning.Stop-gradient’s slower flow-loss convergence and higher final loss indicated Action Expert underfitting; discrete action tokens also provided more efficient adaptation signals during fine-tuning.Source: §5.1 · p. 19, Figure 10 · p. 19

5.2 Effect on Action-Space Loss

• 96.5% peak average success rate at 25k steps with action-space loss exceeded the velocity-space peak by 6.2%, and it reached 95.8% at 20k while velocity-space stayed below 90.3% after 35k.The controlled LIBERO comparison found a small early-training exception: velocity-space loss slightly outperformed action-space loss at 10k steps.Source: §5.2 · pp. 19–20
• After 20k steps, action-space loss maintained 92.5%–96.5% performance while velocity-space loss fluctuated; its low-noise gradients are sparse early, then performance rises after high-noise supervision learns trajectory structure.The authors relate this behavior to action-space loss being equivalent to (1 −𝜏)2-reweighting of velocity-space loss, with useful robot-action information concentrated in low-frequency components.Source: §5.2 · p. 20

5.3 Effect of the Vision-Aligned RVQ Action Tokenizer

• RVQ improved average task progress from 29.3% to 48.1% and VQA accuracy from 75.7% to 77.5% versus FAST under identical co-training settings.Real-robot evaluation used continuous flow-matching actions, so the tokenizer gains were not confined to discrete decoding; visual-action alignment and next-frame prediction objectives also contributed to visual-semantic understanding.Source: Figure 12 · p. 20, §5.3 · pp. 20–21

Action representations

• Unlike π0.5 (1), which co-trains FAST-based (18) autoregression with flow matching, Wall-OSS-0.5 uses a learned Vision-Aligned RVQ tokenizer, treats discrete actions as a gradient bridge, and adds Action-Space Supervision.The related-work discussion contrasts these choices with approaches that tokenize actions primarily for trajectory reconstruction or encode behavioral semantics and context.Source: §6 · p. 21

Architectures for action-ready VLMs

• Building on cognition-action separation in CogACT (97) and embodiment-specific stems in HPT (98), Wall-OSS-0.5 uses VL and Action experts at each Transformer layer.Its Mixture-of-Transformers experts interact through shared attention while retaining separate parameters, routing by modality and function rather than generic knowledge sharding. Prior designs also include from-scratch controllers, shared-transformer action tokens, continuous heads, planner-controller hierarchies, video world models, and memory modules.Source: §6 · pp. 21–22

Robot data and embodied grounding

• Wall-OSS-0.5 organizes a mixture of over one million trajectories per epoch across more than 20 embodiments around deployment, not scale alone.Task-level power sampling mitigates long-tail imbalance, while synthesized embodied bridge samples from action corpora link visual understanding to executable action semantics rather than relying on generic VQA data alone (3, 1).Source: §6 · p. 22

Design implications

• Action-token cross-entropy supplies VLM-native supervision that shapes controllable backbone representations, complementing the weaker residual signal of continuous-only training.Stop-gradient formulations also prevent representation learning and continuous-control optimization from reinforcing one another.Source: §7 · p. 22
• MoT routing and embodied bridge data jointly support executable policies by allocating capacity for continuous actions while preserving gradients and action-aware grounded understanding.Together, these choices aim to avoid collapsing a pretrained multimodal model into a narrow task-specific controller.Source: §7 · p. 22

Limitations

• Current limitations include validation only with a 3B VLM, single-frame inputs, a fixed 26-dimensional action representation, manual scoring rubrics, and incomplete real-robot benchmark coverage.The benchmark does not yet cover multi-robot collaboration, long-duration deployment, or broader open-world interaction; scaling may change training-signal interactions, while single-frame input may constrain long-horizon memory and state tracking.Source: §7 · p. 22

Future Work

• Future work will scale to larger VLMs, add temporal observations and hierarchical planning, and explore action representations for diverse robot morphologies.The open-source project will continue releasing model weights, training code, and evaluation tools to support research and reproducibility.Source: §7 · p. 22

A.1 Pretrained Model Zero-Shot Evaluation Per-Task Results

• Each task is evaluated over 10 trajectories using a predefined scoring rubric with a maximum score of 100.The reported categories are semantic understanding, rigid-object, deformable-object, fine-grained, and long-horizon multi-step manipulation.Source: §A.1 · p. 29
• Table 5 lists zero-shot per-task progress for seen tasks and marks the highest-progress checkpoint for each task.Source: Table 5 · p. 29
• Table 6 lists zero-shot per-task progress for unseen tasks and marks the highest-progress checkpoint for each task.Source: Table 6 · p. 29

• Table 7 lists detailed scores and task descriptions for each benchmark in Section 4.Source: §A.2 · p. 29, Table 7 · p. 30
• Embodied Grounding is an internally constructed benchmark with evaluation samples drawn and annotated directly from robot action dataset trajectories.Source: Table 7 · p. 30

A.3 Fine-Tuning Evaluation Per-Task Detailed Results

• Table 8 reports per-task fine-tuning task progress for three models using official pretrained weights under identical data and evaluation protocols.Each task uses approximately 500 trajectories; task progress is capped at 100.Source: Table 8 · p. 30

A.4 Multi-Task Fine-Tuning Per-Task Detailed Results

• Table 9 gives multi-task fine-tuning per-task task progress, with dashes marking tasks omitted from a configuration.Task progress has a maximum of 100.Source: Table 9 · p. 31

A.5 Evaluation Task Descriptions and Scoring Rubrics

• Real-robot evaluation covers 31 tasks, each scored stepwise to a maximum of 10 points and reported as task progress: actual score divided by maximum score, multiplied by 100.Each task is evaluated over 10 trajectories.Source: §A.5 · p. 31
• Success rate is the proportion of trajectories that fully complete all task steps.The protocol evaluates 10 trajectories per task.Source: §A.5 · p. 31
• The rubric assigns points to manipulation stages; for object matching, each pair is scored for grasp and placement, followed by retraction.The listed rule awards 2 points for grasp, 2.5 for placement, and 1 for retraction across two pairs.Source: Table 10 · p. 32, §A.5 · p. 33
• The fruit-basket-placement rubric awards 3 points per correctly placed specified fruit and 1 point for retraction.The listed task has three fruits.Source: Table 10 · p. 32, §A.5 · p. 33