PaperlayerView exact revision on arXiv

Source-linked AI summary

HoloBrain-0 Technical Report

Xuewu Lin, Tianwei Lin, Yun Du, Hongyu Xie, Yiwei Jin, Jiawei Li, Shijie Wu, Qingze Wang, Mengdi Li, Mengao Zhao, Ziang Li, Chaodong Huang, Hongzhe Bi, Lichao Huang, Zhizhong Su

arXiv:2602.12062v1cs.RO

TL;DR

Robotic agents must generalize across out-of-distribution states and heterogeneous embodiments while remaining reliable in real-world deployment. HoloBrain-0 addresses this with an embodiment-aware VLA architecture, a pre-train-then-post-train strategy, and open infrastructure, achieving state-of-the-art simulation performance and strong real-world manipulation results, with a 0.2B variant comparable to larger baselines.

  • Problem

    Real-world VLA deployment requires generalization across out-of-distribution visual and physical states and compatibility across diverse robot embodiments.

  • Method

    HoloBrain-0 combines embodiment-aware VLA modeling with explicit camera and kinematic priors, scalable pretraining and post-training, and the RoboOrchard infrastructure.

  • Results

    HoloBrain-0 achieves state-of-the-art performance across simulation benchmarks and strong results on challenging real-world manipulation tasks; its 0.2B variant is comparable to larger baselines.

  • Takeaways & Limitations

    The fully open-sourced ecosystem provides pretrained foundations, post-trained checkpoints, and infrastructure for reproducible robotic manipulation research and deployment.

  • Takeaways & Limitations

    Precise instruction following remains underevaluated, especially for easily confusable instructions.

Abstract

from arXiv · show

In this work, we introduce HoloBrain-0, a comprehensive Vision-Language-Action (VLA) framework that bridges the gap between foundation model research and reliable real-world robot deployment. The core of our system is a novel VLA architecture that explicitly incorporates robot embodiment priors, including multi-view camera parameters and kinematic descriptions (URDF), to enhance 3D spatial reasoning and support diverse embodiments. We validate this design through a scalable ``pre-train then post-train" paradigm, achieving state-of-the-art results on simulation benchmarks such as RoboTwin 2.0, LIBERO, and GenieSim, as well as strong results on challenging long-horizon real-world manipulation tasks. Notably, our efficient 0.2B-parameter variant rivals significantly larger baselines, enabling low-latency on-device deployment. To further accelerate research and practical adoption, we fully open-source the entire HoloBrain ecosystem, which includes: (1) powerful pre-trained VLA foundations; (2) post-trained checkpoints for multiple simulation suites and real-world tasks; and (3) RoboOrchard, a full-stack VLA infrastructure for data curation, model training and deployment. Together with standardized data collection protocols, this release provides the community with a complete, reproducible path toward high-performance robotic manipulation.

1 Introduction

HoloBrain-0 addresses the difficulty of building general robotic agents by combining embodiment-aware VLA modeling, scalable data curation, and open infrastructure. It achieves strong simulation and real-world results, including comparable performance from a lightweight 0.2B variant.

  • Motivation: General-purpose robotic agents remain difficult because real-world deployment requires robust generalization across out-of-distribution visual, physical, and robot-embodiment variations.Large-scale pretraining helps address these challenges but increases the need for cross-embodiment compatibility.
  • Framework: HoloBrain-0 integrates a VLA architecture, scalable data strategy, and full-stack infrastructure into one framework.The framework is designed to address generalization and deployment challenges through a unified system.
  • Architecture: Its embodiment-aware architecture projects multi-view images into unified 3D coordinates, encodes kinematic chains, and predicts relative-joint and SE(3) motions.These components support compatibility across single- and dual-arm manipulators, mobile robots, and human-captured data.
  • Data strategy: The data strategy combines heterogeneous multi-embodiment demonstrations for pretraining with test-driven post-training data collection for dexterous, long-horizon bimanual tasks.The post-training process dynamically adjusts collection based on model performance.
  • Infrastructure: RoboOrchard covers data acquisition, validation, organization, scalable training, and synchronous or asynchronous deployment.It provides web-based acquisition, automated quality checks, MCAP organization, Arrow-based datasets, and flexible inference modes.
  • Results: HoloBrain-0 reaches state-of-the-art performance across several simulation benchmarks and strong results on challenging real-world manipulation tasks.Its 0.2B variant performs comparably to larger baselines, supporting efficient low-latency on-device deployment.

2 Problem Formulation

HoloBrain-0 formulates robotic manipulation as conditional action generation from multimodal observations, language instructions, camera parameters, and robot kinematic priors. The policy outputs an action chunk for subsequent time steps.

  • Inputs: The policy receives multi-view RGB images, depth maps, and the robot’s current proprioceptive joint state.The formulation operates directly on the current observation frame.
  • Inputs: For multi-task execution, the model is additionally conditioned on natural-language instructions.Language specifies the task alongside the visual and proprioceptive observations.
  • Embodiment priors: Camera parameters and robot kinematic priors, including URDF descriptions, are incorporated to ground the model in the physical world.Camera parameters include intrinsics and extrinsics for the available views.
  • Output: Given these inputs, the model generates an action chunk for the subsequent t_out steps, with N_j corresponding to the robot’s degrees of freedom.N denotes the number of camera views, while the output is represented over the robot’s joint dimensions.

3 Method: Model Architecture

HoloBrain-0 combines a vision-language backbone with spatial and action modules that explicitly encode 3D geometry and embodiment structure. Its deployment strategy addresses action-chunk latency through trajectory consistency and smooth fusion during inference.

  • Core architecture: The architecture combines a VLM, Spatial Enhancer, and embodiment-aware Action Expert for end-to-end robotic control.The VLM encodes language and visual observations, while the Spatial Enhancer and Action Expert provide spatial and action modeling.
  • Spatial representation: The Spatial Enhancer projects multi-view image features through camera geometry into a unified 3D coordinate system.It uses camera intrinsics and extrinsics, samples 3D points, predicts depth distributions, and produces depth-aware positional embeddings.
  • Action representation: The Action Expert masks most joint angles and represents joint state primarily through 6D joint poses to improve embodiment generalization.Cartesian link poses provide a more consistent geometric reference than embodiment-specific joint-angle conventions.
  • Action representation: The model predicts hybrid actions combining joint-angle residuals with Cartesian link-pose displacements.This output space includes angle-space and pose-space transformations for each joint.
  • Real-time control: SimpleRTC enforces consistency with the preceding action chunk before smoothly blending toward the new trajectory during denoising.The method uses an inference-delay interval followed by a transition window, with Linear, Quadratic, or Exponential decay choices.
  • Training strategy: Teacher forcing replaces initial noisy trajectory steps with ground-truth actions during training to reduce the training–inference distribution gap.The prefix length is sampled from a Poisson distribution, and a small teacher-forcing ratio preserves denoising capability on fully noisy inputs.

4 Method: Data Strategy

HoloBrain-0 builds a heterogeneous pre-training corpus around cross-embodiment 3D grounding, then applies rigorous consistency checks and iterative data strategies to improve robustness to OOD states.

  • Cross-Embodiment and Spatial Grounding: Selected datasets include multi-view camera intrinsics and extrinsics plus complete URDFs to support cross-embodiment spatial grounding.The corpus spans diverse robotic platforms and sensor configurations.
  • Geometric and Semantic Diversity: Simulation data supplies accurate depth maps and camera parameters, providing high-fidelity geometric supervision for learning 3D world dynamics.This complements semantic and object diversity from human-centric and robotic manipulation data.
  • Data Cleaning: 3D consistency verification projects joint 6D poses into images and filters samples with reprojection errors, validating camera parameters, action labels, and URDF accuracy.Additional filters assess task type, trajectory plausibility, and instruction-video consistency.
  • Pre-training Data Corpus: 156 million frames from 3,500+ hours and seven embodiments combine proprietary, real-world, simulation, and human-video data.Sampling ratios are manually adjusted according to data quality and task richness.
  • Iterative Test-Driven Data Strategy: Post-training uses proactive state expansion and reactive failure-cluster targeting to align training data with anticipated and observed OOD conditions.The approach shifts from scale-oriented dataset expansion toward quality-driven iteration.

5 Full-Stack Infrastructure: RoboOrchard

RoboOrchard addresses fragmented robotic-learning toolchains with a modular, artifact-driven infrastructure spanning data acquisition, storage, training, deployment, and user interaction.

  • Architecture: RoboOrchard links decoupled modules through standardized artifacts rather than tightly coupled code across system layers.Unified Artifact Specifications define self-describing datasets and self-contained model artifacts.
  • Architecture: Its three layers separate hardware abstraction, the data-to-policy middleware pipeline, and user interaction with visualization.The Hardware Abstraction Layer bridges simulation and real hardware through unified interfaces.
  • Data Acquisition: The acquisition stack combines ROS 2 recording, MCAP storage, integrity monitoring, and a visual Companion App for task orchestration.The Integrity Monitor flags sensor frame drops and timestamp anomalies.
  • Data Storage: RODataset stores high-bandwidth sensor data in Apache Arrow for zero-copy loading while structuring global metadata separately.The format integrates with the HuggingFace ecosystem.
  • Training: RoboOrchardLab provides Pydantic-based type-safe configuration, hook-based training, unified model interfaces, and portable model artifacts.Artifacts package weights with preprocessing pipelines and environment dependencies.
  • Deployment: Deployment decouples inference from control, supports synchronous and asynchronous modes, and enables transitions between real robots and simulation.The architecture uses standardized interfaces and a client-server inference design to reduce dependency conflicts and support model switching.

6 Experiments

HoloBrain-0 is evaluated across diverse real-world and simulation settings, including long-horizon manipulation, robustness to distribution shifts, humanoid embodiments, and auxiliary-task co-training. Across these evaluations, it reports strong performance, including gains over π0.5, state-of-the-art benchmark results, and robust grasping with limited data.

  • Evaluation Setup: The evaluation covers 10 real-world manipulation tasks and four simulation benchmarks, with analyses of multi-task co-training and asynchronous inference.The real-world suite includes seven basic tasks, two long-horizon dexterous tasks, and one general object pick-and-place task.
  • Real-World Results: 5.65% and 8.02% average success-rate gains over π0.5 are achieved by HoloBrain-0-GD and HoloBrain-0-QW, respectively, across 10 real-world tasks.The comparison uses π0.5 as the baseline and reports average success rates across the evaluated tasks.
  • Real-World Results: 25% and 30% success-rate improvements over π0.5 occur on fold clothes and fold paper box, respectively.These are the two long-horizon dexterous manipulation tasks highlighted in the real-world analysis.
  • Real-World Results: HoloBrain-0-QW reaches 93.5% on seen and 97.5% on unseen objects in Grasp Anything, with grasping success reported as largely independent of training-set membership.The test set includes difficult objects with irregular geometries, contributing to the lower seen-object result.
  • Simulation Results: 90.8% success for the 0.2B-parameter HoloBrain-0-GD and 92.3% for HoloBrain-0-QW are reported under the simulation benchmark’s randomization setting.The results are presented as SOTA-level performance and are associated with precision and stability in multi-task execution.
  • Simulation Results: HoloBrain-0-GD achieves 74.0% on zero-shot LIBERO-Plus, exceeding OpenVLA-OFT at 69.6% and X-VLA at 69.7%.LIBERO-Plus evaluates generalization across seven distribution-shift axes, while standard LIBERO reports 97.4% for HoloBrain-0-QW.
  • Simulation Results: HoloBrain-0-QW scores 4.685 on GenieSim 2.2, surpassing X-VLA’s 4.541 on a humanoid upper-body embodiment.The benchmark contains 10 manipulation tasks evaluated over 250 episodes.
  • Ablation and Analysis: Co-training Grasp Anything with seven basic tasks increases average success rates across the basic tasks, supporting auxiliary skill-focused data as a data-efficiency strategy.The ablation targets robust grasping because grasping is described as foundational to the basic tasks.

7 Related Work

Related work spans end-to-end action modeling, data strategies for distribution shift, and infrastructure for embodied-AI development. HoloBrain-0 positions RoboOrchard as an efficiency- and reproducibility-oriented infrastructure response to existing deployment bottlenecks.

  • Action Models: Action-modeling research explores action distributions through imitation, Gaussian mixtures, and diffusion, alongside architectures designed to improve manipulation performance.The cited approaches represent multiple lines of progress in end-to-end robot manipulation.
  • Data Strategy: Data-collection methods for distribution shift include adversarial collection and state-space augmentation, but existing approaches can be labor-intensive or causally misaligned.The passage identifies these as major paradigms for improving robustness.
  • Robotic Infrastructure: LeRobot lowers access barriers through standardized multimodal formats, PyTorch implementations, hardware interfaces, and cloud-hosted datasets.The infrastructure supports rapid prototyping and cross-institutional data sharing.
  • Robotic Infrastructure: RoboOrchard addresses high-throughput data-handling and deployment-reproducibility bottlenecks by prioritizing efficiency and adopting MCAP.It is presented as a response to limitations identified in existing robotic platforms.

8 Conclusion

HoloBrain-0 bridges foundation-model research and reliable real-world deployment through embodiment-aware architecture, scalable training, and open infrastructure. It achieves strong simulation and real-world performance while providing a reproducible ecosystem for generalist robotic agents.

  • HoloBrain-0 explicitly injects kinematic chains and multi-view camera parameters to support 3D spatial reasoning and cross-embodiment generalization.
  • The framework combines a “pre-train then post-train” paradigm with open-source RoboOrchard infrastructure for training and deployment.
  • HoloBrain-0 achieves state-of-the-art performance across RoboTwin 2.0, LIBERO, LIBERO-Plus, and GenieSim, alongside challenging real-world manipulation tasks.
  • The open-sourced ecosystem includes pre-trained foundations, post-trained checkpoints, and software infrastructure for reproducible development of generalist robotic agents.

9 Future Work

Future work targets higher task success, instruction adherence, and generalization while reducing data acquisition costs. Planned directions include off-policy reinforcement learning, rigorous instruction-following evaluation, and advanced co-training strategies.

  • The next HoloBrain release will integrate off-policy reinforcement learning with expert demonstrations, policy rollouts, and human interventions.A Value Model is planned within the VLA framework.
  • Precise instruction following remains underevaluated, especially for easily confused instructions, motivating clearer benchmarks and improved reliability.
  • Mixing data from different tasks during post-training produced preliminary synergistic gains, motivating further study of co-training for generalizability and few-shot embodiment transfer.

A HoloBrain-0 Implementation Details

The implementation uses two vision-language base models and documents architecture-specific parameter handling, with supporting details provided for the Action Expert and hyperparameter configuration.

  • GroundingDINO Tiny and Qwen2.5-VL-3B serve as base models for HoloBrain-0-GD and HoloBrain-0-QW, respectively.
  • For GroundingDINO Tiny, architectural components and parameters are retained while the BERT module is frozen during training.
  • For Qwen2.5-VL, the language model is pruned to its first layer and the vision encoder is frozen.
  • Figure A1 details the Action Expert architecture, while Table A1 lists HoloBrain-0 hyperparameter settings.

B Real-World Experimental Details

The real-world evaluation combines a dual-arm Piper setup, diverse object and task protocols, and data strategies designed to improve state coverage and recover deployment failures. Iterative targeted collection raises success and generalization while quality controls improve dataset reliability.

  • Real-world setup: Experiments use a dual-arm AgileX Piper robot with three Intel RealSense D435 RGB-D cameras for fixed-base tabletop manipulation.
  • Real-world setup: Two global camera configurations accommodate the extensive workspace required by cloth folding and other task field-of-view needs.
  • Task diversity: The “Grasp Anything” dataset contains over 200 distinct objects spanning rigid, irregular, deformable, and semi-transparent categories.
  • Task protocols: Evaluation protocols standardize object placements, task instructions, SOPs, and progress-score definitions across 10 real-world tasks.The success thresholds differ for place shoe, fold towel, and place to slot; other tasks require their maximum achievable score.
  • Data collection: Cloth folding uses 8 SOP stages and paper box folding uses 6 stages, with SOPs constructed by backward induction from target states and key waypoints.
  • Data collection: Strict SOP adherence can reduce episode information entropy and cause overfitting, motivating proactive vision and robot state expansion.
  • Data collection: Step-wise state expansion selects keyframes and augments states around trajectory waypoints to concentrate data on error-correction behaviors.
  • Failure recovery: Test-Driven Failure Recovery uses deployment failure frequencies to target state expansion and data collection for high-frequency failures.

C.4 Failure Mode Analysis

Despite iterative refinement, HoloBrain-0 retains long-tail failure modes in complex cloth folding. These failures involve recovery from disordered states, task-stage recognition, garment generalization, and flattening efficiency.

  • Long-tail failures persist in complex cloth folding despite iterative refinements through version 4.0.
  • Progress Reversion: Tangled configurations can reset progress to the initial flattening phase because robust mid-level recovery strategies are lacking.
  • State Confusion: Partially folded garments can resemble crumpled states, causing stage misidentification and flattening actions to recur mid-trajectory.
  • Insufficient Clothes Generalization: The policy generalizes poorly across garment materials, sizes, and intricate designs; low-friction fabrics can cause empty grasps or slippage.Performance remains consistent on standard solid-colored T-shirts.
  • Low Flattening Efficiency: Repetitive flattening primitives are less efficient than human tensioning and grasp-selection heuristics in highly disordered states.
Loading 2602.12062v1…