PaperlayerView exact revision on arXiv

Source-linked AI summary

Hy-Embodied-0.5-VLA: From Vision-Language-Action Models to a Real-World Robot Learning Stack

He Zhang, Lingzhu Xiang, Haitao Lin, Zeyu Huang, Minghui Wang, Dingyan Zhong, Yubo Dong, Yihao Wu, Yongming Rao, Dongsheng Zhang, Wanjia He, Ling Chen, Kai Huang, Jiahao Chen, Sichang Su, Xumin Yu, Ziyi Wang, Chengwei Zhu, Xiao Teng, Yuchun Guo, Yufeng Zhang, Yuandong Liu, Rui Wang, Zisheng Lu, Han Hu, Zhengyou Zhang

arXiv:2606.14409v2cs.ROcs.AI

TL;DR

Deployable generalist robots require coordinated solutions across data, model design, training, adaptation, and execution rather than stronger policies alone. HyVLA-0.5 builds this stack with high-fidelity UMI data, an embodied-native continuous-action VLA, FlowPRO offline preference refinement, and hardware-aware deployment. It reports leading simulated success rates and improved precision-critical real-world task performance, while leaving zero-shot generalization and deployment speed as open limits.

  • Problem

    Deployable generalist robots require data, training, adaptation, and execution layers co-designed around real-hardware constraints, while existing approaches face data, representation, and last-mile dexterity bottlenecks.

  • Method

    HyVLA-0.5 co-designs high-fidelity egocentric UMI data, an embodied VLM with a flow-matching action expert and end-effector-frame delta chunks, FlowPRO offline preference refinement, and asynchronous deployment.

  • Results

    HyVLA-0.5 achieves the best simulated success rate in both settings, reaching 90.9% on Clean and 90.1% on Randomized, while UMI pre-training improves precision-critical real-world tasks.

  • Takeaways & Limitations

    The paper presents a complete pipeline in which reusable high-fidelity supervision, embodiment-agnostic action interfaces, failure-driven offline refinement, and asynchronous execution support cross-embodiment deployment.

  • Takeaways & Limitations

    HyVLA-0.5 does not study zero-shot generalization because the authors consider current data scale insufficient for such claims, and deployment-time execution speed remains an open direction.

Abstract

from arXiv · show

In this report, we present Hy-Embodied-0.5-VLA, abbreviated as HyVLA-0.5, an end-to-end system that spans the full robot learning stack: data collection, model design, continued pre-training and supervised fine-tuning, RL post-training, and real-world deployment. Each component serves a distinct role in this stack.

1 Introduction

HyVLA-0.5 treats deployable generalist robotics as a full-stack co-design problem spanning data, modeling, adaptation, reinforcement learning, and execution. It combines custom high-fidelity demonstrations, embodied-native continuous-action modeling, FlowPRO preference post-training, and hardware-aware deployment.

  • Deployable generalist robots require data, training, adaptation, and execution layers co-designed around real-hardware constraints.
  • HyVLA-0.5 addresses this need with an end-to-end pipeline spanning custom data collection, VLA modeling, RL post-training, and real-world deployment.
  • The custom fingertip UMI device and motion-capture cage collect over 10K hours of egocentric, sub-millimeter-precision demonstrations with reusable downstream supervision.
  • Its embodied-native 4B MoT backbone uses a flow-matching action expert, compact temporal memory, and delta-chunk actions for continuous high-frequency control and cross-embodiment adaptation.
  • Two supervised-fine-tuning tracks study same-platform adaptation and UMI-only transfer to morphologically different robots, while asynchronous inference and Bézier smoothing support real-hardware execution.
  • FlowPRO converts paired success/failure rollouts into reward-free, critic-free offline preference refinement without training reward or value networks.

2 Model Architecture

HyVLA-0.5 combines an embodied MoT VLM with a continuous-action expert and a compact memory encoder for goal-conditioned, chunk-level robotic control. Its design preserves modality-specific processing while enabling cross-modal grounding and efficient causal spatiotemporal perception.

  • Architecture: HyVLA-0.5 uses an embodied MoT VLM, a conditional-flow-matching action expert, and a compact memory encoder as its three main components.The VLM provides multimodal context, the action expert predicts continuous action chunks, and the memory encoder aggregates multi-frame observations.
  • Problem formulation: The policy consumes multi-view visual frames, language instructions, and proprioceptive state, then predicts a future action chunk of horizon H.The visual history contains K frames from each of n camera viewpoints, while language specifies the task goal and state anchors actions to the robot configuration.
  • Action expert: Continuous flow matching models p(A_t | o_t) directly instead of discretizing actions into language-like tokens.The action expert predicts a velocity field that transports noisy action chunks toward target actions.
  • Action expert: Shared self-attention couples separate understanding and action-generation towers, while block-wise causal attention isolates perception, state, and noisy-action blocks.The perception block cannot attend to robotics-specific blocks, the state block supports caching, and the noisy-action block attends to the full prefix.
  • Compact memory encoder: Temporal-spatial attention processes historical frames causally and spatial patches bidirectionally, reducing per-layer cost from O(n^2K^2) to O(Kn^2 + nK^2).The encoder reuses the ViT projections and uses fixed sinusoidal temporal encoding; with K=1, it reduces exactly to the pre-trained ViT block.

3 Pre-training and Supervised Fine-tuning

HyVLA-0.5 is pre-trained on a large, high-fidelity UMI corpus and then supervised-fine-tuned for simulated and real robot embodiments. The training setup combines embodiment-independent demonstrations with target-platform adaptation and cross-embodiment transfer.

  • 3.2 Pre-training: HyVLA-0.5 is pre-trained on more than 10K hours of in-house Hy-UMI-10K demonstrations, the sole pre-training data source.The corpus contains more than 1M episodes spanning 70 tasks and diverse scene-based task families.
  • 3.1 Hy-UMI-10K: High-Fidelity Manipulation Dataset: External optical motion capture labels each gripper’s 6-DoF trajectory at sub-millimetre precision in a globally consistent frame.This replaces visual-SLAM pose recovery and prioritizes high-quality labels for fine-motor tasks, while making deployment less convenient in the wild.
  • 3.2 Pre-training: The pre-training model initializes its VLM from Hy-Embodied-0.5-MoT and trains a separately initialized 370M-parameter action expert under the flow-matching objective.All model parameters remain trainable during pre-training.
  • 3.3 Supervised Fine-tuning: Supervised fine-tuning uses task-specific demonstrations across one simulated embodiment and four real-world platforms.The real-world data are organized into Track-A for same-platform adaptation and Track-B for UMI-only transfer to morphologically different robots without target-robot teleoperation.
  • 3.3 Supervised Fine-tuning: The real-world SFT recipe samples actions at 50 Hz with horizon H=50 and trains for 60K steps using a global batch size of 32.RoboTwin 2.0 uses a distinct larger-scale configuration with downsampled actions, horizon H=20, and global batch size 128.

4 Reinforcement Learning Post-Training

FlowPRO adds failure-driven, preference-based offline RL on top of an SFT-pretrained HyVLA-0.5 policy. It turns teleoperated corrections into dense per-state preferences and optimizes a flow-matching-aware objective that uses proximal anchoring and supervised data.

  • Design principles: FlowPRO exploits failures directly through per-state, per-chunk contrastive signals rather than discarding or merely relabeling negative trajectories.Its design also avoids reward and critic models and anchors implicit reward magnitudes to prevent reward hacking.
  • FlowPRO pipeline: FlowPRO iteratively collects on-robot preference pairs, densifies them into per-state tuples, and optimizes mixed batches of new, historical, and SFT data.The previous round’s policy supplies the frozen reference policy for the next round.
  • RPRO objective: RPRO aligns preferred and dispreferred actions using a flow-matching loss, a symmetric proximal regularizer, and a supervised regression term.The proximal term anchors implicit rewards near zero, while the supervised term preserves direct regression toward preferred actions.
  • Optimization behavior: Contrastive gradient cancellation makes samples with identical preferred and dispreferred actions contribute no contrastive gradient, allowing SFT demonstrations to share the RPRO loss.The remaining gradients come from the proximal and supervised terms.
  • Data collection: Intervention-and-rollback records erroneous segments as negative trajectories and pairs them with operator corrections from an earlier state.Smooth Interpolation synthesizes missing counterpart actions after the preferred and dispreferred trajectories diverge.
  • Evaluation: FlowPRO is evaluated on four real-robot bimanual tasks: Bottle, Cap, USB, and Zip.The supplied passage identifies the evaluation tasks but reports no outcome values.

5 Deployment

HyVLA-0.5 deployment preserves a robot-agnostic delta-chunk interface while adapting commands to heterogeneous platforms. An asynchronous runtime and cubic Bézier stitching maintain continuous execution and smooth transitions between independently predicted chunks.

  • Embodiment-Agnostic Platform Mapping: Platform mapping keeps the learned action interface unchanged across embodiments and defers robot-specific kinematics to deployment-time composition and inverse kinematics.The policy predicts a 20-dimensional dual-arm chunk, while embodiment-specific transformations produce target-robot joint commands.
  • Embodiment-Agnostic Platform Mapping: Track A retains the same world frame between data collection and deployment, whereas Track B maps data from fixed-base arms or a floating-base humanoid to the target embodiment.For the humanoid, a deterministic heuristic sets additional head and torso dimensions rather than the policy predicting them.
  • Asynchronous Inference and Execution: Asynchronous execution overlaps policy inference, action-buffer overwriting, Bézier smoothing, and servo-rate command execution.A producer–consumer runtime hides much of backbone latency while recording executed poses for tangent estimation.
  • Latency-Aware Bézier Chunk Stitching: Cubic Bézier stitching reconnects delayed action chunks to the current robot state with C1-continuous position and tangent transitions.Historical and future motion directions determine the inner control points, while position, orientation, and gripper commands are smoothed separately.
  • Latency-Aware Bézier Chunk Stitching: The truncation ratio α and connection parameter γ trade off retained policy actions against room for correcting delayed chunk boundaries.Smaller γ preserves more predicted actions; larger γ provides a smoother landing target but skips more of the retained chunk.
  • Latency-Aware Bézier Chunk Stitching: Fig. 8 compares raw orange chunks with blue asynchronously Bézier-smoothed trajectories across both arms’ x, y, and z dimensions.The smoothed trajectories reduce visible discontinuities at chunk boundaries.

6 Evaluation

HyVLA-0.5 is evaluated across simulation, real-robot manipulation, force-aware control, and FlowPRO post-training. It achieves strong benchmark performance, transfers across embodiments, and improves real-world execution through preference-based refinement.

  • 6.2 Real-World Tasks: Track A evaluates four bimanual tasks after same-platform teleoperated fine-tuning, while Track B evaluates two tasks after UMI-only transfer to different robots.The real-robot evaluation spans three platforms and six benchmark tasks.
  • 6.1 Simulated Tasks: Removing the compact memory encoder lowers Clean/Randomized success from 90.9% / 90.1% to 88.8% / 88.6%.Removing UMI pre-training further lowers the scores to 88.1% / 87.9%.
  • 6.2 Real-World Tasks: UMI pre-training substantially improves cross-embodiment success on Put Away the Accessory and Clean Up the Table over π0 and π0.5.The comparison uses the same UMI fine-tuning data without target-robot teleoperation.
  • 6.2 Real-World Tasks: Force-aware HyVLA-0.5 reliably selects the lighter box by comparing grasp-phase force profiles when object position is randomized.The policy uses tip force/torque signals encoded from a 50-step F/T window for each hand.
  • 6.3 Real-World Reinforcement: Across four real-robot tasks, RPRO achieves the highest success rate with the shortest completion time among the compared post-training methods.RPRO directly injects preference information into the continuous action-generation loss and uses negative trajectories.

7 Related Work

Related work progresses from discrete-token VLA control toward continuous flow-matching policies, embodied-native backbones, robot-agnostic demonstrations, and preference-based post-training. Deployment research also addresses action refinement and inference latency.

  • Generalist VLA Models: Early VLA systems used discrete autoregressive action tokens, limiting control frequency and spatial precision.Flow-matching policies later restored continuous, high-frequency execution capabilities.
  • Embodied VLM Backbones: Contemporary VLAs increasingly use domain-specific embodied backbones to improve the fine-grained visual acuity required for manipulation.Hy-Embodied-0.5 is presented among these embodied VLM backbones.
  • Pre-training and Post-training Recipes for VLAs: HyVLA-0.5 differs from aggregated teleoperation pre-training paradigms by sourcing its foundational signal predominantly from human-centric UMI data.Its action expert is optimized under a singular flow-matching loss.
  • Hand-Held Demonstrations and UMI: UMI-style hand-held rigs provide robot-agnostic demonstrations, while later systems expand their morphological applicability toward humanoid and mobile platforms.The related work positions UMI as a foundation for cross-platform demonstration capture.
  • Preference Post-Training in Continuous Control: Existing continuous-control post-training methods differ in how directly they exploit failure information from autonomous rollouts.Vanilla SFT discards failure signals, whereas DAgger uses them to trigger expert correction.
  • Asynchronous Inference and Action-Chunk Smoothing: Action-chunk deployment methods face jitter, boundary discontinuities, and idle gaps, motivating high-frequency refinement and asynchronous execution.Inference-Time and Training-Time RTC are cited as related approaches to this deployment problem.

8 Discussion

The discussion frames HyVLA-0.5 as a co-designed robot-learning stack while identifying open boundaries in data collection, execution efficiency, and zero-shot generalization.

  • HyVLA-0.5 Pipeline: HyVLA-0.5 combines high-fidelity UMI data, temporal action representations, FlowPRO refinement, and asynchronous chunk stitching for cross-embodiment deployment.These components target data quality, action representation, failure correction, and deployment timing.
  • Future Work: Future data collection should move beyond motion capture while preserving high-precision supervision and address differences between egocentric UMI and robot-mounted cameras.The authors propose exoskeleton-based collection, controlled label noise, and systematic visual augmentation studies.
  • Future Work: Deployment research must improve execution speed while maintaining safety and precision, potentially by combining deployment-time adaptation with reinforcement learning.The discussion treats practical task cadence as part of real-world success.
  • Future Work: HyVLA-0.5 does not study zero-shot generalization because the authors consider the current data scale insufficient for such claims.Rigorous evaluation of emerging zero-shot capabilities remains open.

A RoboTwin 2.0 Evaluation Details

The RoboTwin 2.0 appendix supplements aggregate evaluation with per-task results and describes data cleaning and action decoding procedures.

  • Per-task results: Table 3 reports HyVLA-0.5 per-task success rates under Clean and Randomized settings on the 50-task RoboTwin 2.0 suite.The breakdown complements the aggregate comparison in Table 1.
  • Data filtering: RoboTwin demonstrations are offline-cleaned by clustering episode-length distributions with HDBSCAN to identify abnormal episodes.The cleaning step targets implausible inverse-kinematics solutions associated with abnormal episode lengths.
  • Action decoding: Action decoding combines relative-EEF and EEF predictions, doubling chunk size before fusing motion and interpolating quaternion orientations with SLERP.Relative-EEF supplies smooth local motion, while EEF anchors targets globally and reduces drift accumulation.

B.1 UMI-to-Robot Deployment Derivation

UMI demonstrations use a world frame without torso poses, so deployment on Astribot S1 requires mapping targets into the chassis frame and inferring whole-body poses. The reported heuristic aligns torso orientation and height with the demonstrated gripper configuration.

  • UMI demonstrations lack torso poses and use their own world frame, whereas Astribot S1 targets are defined in its chassis frame.The missing frame mapping and torso pose affect reasonable upper-body poses and efficient inverse-kinematics solving.
  • Heuristic torso/head pose inference: The heuristic estimator consumes bimanual gripper poses to infer the world-to-chassis transform, torso pose, and head pose.
  • Heuristic torso/head pose inference: The inferred torso forward axis points toward the centroid of the two grippers, while torso height places both grippers inside a comfortable upper-body reach shell.
  • Heuristic torso/head pose inference: The method assumes the UMI world and robot chassis frames differ by pure translation with identical orientation, an assumption stated to hold on Astribot S1.

B.2 Track-B Reachability and Data Hygiene

Because UMI demonstrations are collected without a robot in the loop, they do not guarantee reachability for arbitrary target morphologies. Offline reachability and IK checks filter deployment-incompatible tasks and trajectories without changing the policy or action representation.

  • Robot-free UMI demonstrations intrinsically lack reachability guarantees for arbitrary target morphologies.
  • Unitree G1 & Astribot S1: Unitree G1 and Astribot S1 deployments verify reachability offline and exclude tasks exceeding each humanoid platform’s reachable shell.
  • JAKA K1: JAKA K1 post-training corpora undergo a single-pass IK feasibility check, removing trajectories that violate the arm’s kinematics.
  • Neither protocol changes the policy or action representation; both enforce distributional alignment between post-training data and the physical deployment frontier.

C FlowPRO Hyperparameters

FlowPRO uses a staged optimization setup with three rounds of 25,000 steps, fixed batch composition, and an empirically weighted trajectory-distance metric. Stage 2 initializes training and remains the frozen reference policy.

  • Trajectory distance combines end-effector position distance, weighted rotational geodesic distance, and normalized gripper-width difference.The empirical weights are 1, 0.5, and 0.2, respectively.
  • The Stage-2 checkpoint serves as both the initialization θ and the frozen reference policy.
Loading 2606.14409v2…