Source-linked AI summary
NebulaVLA: A Dual-Frequency Vision-Language-Action Model With Guide Action for Robotic Manipulation
Cong Zhao, Shuai Tian, Xu Zhang, Baocheng Ni, Xinguo Song, Xueying Sun, Shu Jiang, Shouchang Yang, Bo Tang, Jin Deng, Ge Zhu, YongCheng Wang, Jin Xu, Ri Yang
TL;DR
VLA deployment is limited by efficiency-performance trade-offs, cross-embodiment representation, and execution smoothness. NebulaVLA addresses these challenges with asynchronous dual-frequency control, GESTURE-7, and Guide Action, achieving 85.5% success on LIBERO-Plus while reducing latency and outperforming synchronous baselines.
Problem
VLA models face coupled limitations in computational efficiency, cross-embodiment action representation, and smooth continuous execution.
Method
NebulaVLA decouples semantic reasoning from reactive control with dual frequencies, while GESTURE-7 unifies action semantics and Guide Action enforces smoothness.
Results
85.5% overall success on LIBERO-Plus surpasses InternVLA-M1 (81.3%) by 4.2%, while heterogeneous frequencies reduce average step latency from 115 ms to 42 ms.
Takeaways & Limitations
NebulaVLA provides efficient, responsive robotic manipulation with improved cross-embodiment transferability and reduced execution jitter over synchronous baselines.
Takeaways & Limitations
Frequency allocation is empirically determined and static, while adaptive scheduling remains an open challenge across task domains.
Abstract
from arXiv · showhide
Real-world deployment of Vision-Language-Action (VLA) models is often bottlenecked by efficiency-performance trade-offs, cross-embodiment generalization, and execution smoothness. We present NebulaVLA, an asynchronous dual-frequency architecture that decouples high-level semantic reasoning from low-level action control, optimizing computational resources and modularity. To bridge semantic gaps across heterogeneous robots, we introduce GESTURE-7, a unified language-grounded action representation. Furthermore, our Guide Action algorithm enforces kinematic continuity via mask-based smoothness constraints. Comprehensive evaluations demonstrate that NebulaVLA significantly outperforms synchronous baselines, achieving an 85.5\% average success rate on LIBERO-Plus and accelerating action generation by \textasciitilde 2.7$\times$. This asynchronous design enables highly efficient and responsive control for practical robotics.
1 Introduction
NebulaVLA addresses VLA bottlenecks in synchronous efficiency-performance trade-offs, embodiment representation, and execution smoothness with a dual-frequency architecture, GESTURE-7, and Guide Action. Experiments report reduced inference latency, higher task success than synchronous baselines, improved cross-embodiment transfer, and suppressed execution jitter.
- Motivation: VLAs unify perception, language, and control for natural-language instruction following and complex robotic task execution.This transition moves robotic manipulation from modular pipelines toward end-to-end learning.
- Motivation: Synchronous VLA architectures cannot simultaneously support low-frequency semantic grounding and high-frequency motor control, while inadequate embodiment representations hinder cross-morphology generalization.Mapping continuous kinematics to discrete tokens also requires expansive vocabularies and large-scale demonstration data.
- Contributions: NebulaVLA introduces a dual-frequency VLA model that systematically addresses these efficiency, embodiment, and control limitations.The design is grounded in a fast-slow cognitive paradigm.
- Contributions: 10 Hz VLM deliberative reasoning is decoupled from 20 Hz DiT reactive control, optimizing compute-latency and enabling distributed deployment.The heterogeneous architecture assigns semantic reasoning to System 2 and motor control to System 1.
- Contributions: GESTURE-7 provides a 7D continuous end-effector representation, while Guide Action uses trajectory anchors and mask constraints to improve continuity and suppress execution jitter.Three-stage training combines System 2 SFT and GRPO-based RL with System 1 SFT using weighted flowmatching loss; experiments also report reduced inference latency, higher task success, and improved cross-embodiment transferability.
2 Related Work
Prior VLA research has expanded architectural scale and unified perception-action modeling, but challenges remain in computational efficiency, cross-embodiment action alignment, and smooth asynchronous execution. NebulaVLA addresses trajectory continuity by conditioning generation on executed anchors and enforcing mask constraints during denoising.
- VLA Architectures and Frequency Design: VLA architectures have progressed from foundational Transformers to scalable systems integrating spatial grounding, unified perception-action modeling, and knowledge transfer.The related frameworks include open-source scalable designs and unified understanding-generation-action systems such as π0, InternVLA, and GR series.
- Embodiment Representation and Action Tokenization: Unified action representation remains central to cross-embodiment generalization, while discrete tokenization requires extensive demonstrations and struggles with cross-morphology semantic alignment.The passage contrasts discrete methods such as FAST and Mind-to-Hand with continuous approaches including PUMA/DOMINO and KineVLA.
- Trajectory Smoothness and Chunk Boundary Continuity: Asynchronous chunk-based control can suffer jitter and boundary discontinuities when inference latency is misaligned with control frequency.Existing approaches predominantly rely on post-hoc smoothing, which operates outside the generative process and cannot guarantee semantic consistency across chunks.
- Trajectory Smoothness and Chunk Boundary Continuity: NebulaVLA conditions next-chunk generation on executed trajectory anchors and enforces mask constraints during diffusion denoising.The Guide Action mechanism preserves known actions at each step, maintains semantic continuity at overlapping boundaries, and removes post-processing smoothing.
3 Method
NebulaVLA separates semantic planning from action control through asynchronous System 2 and System 1 modules operating at different frequencies. Its method further addresses cross-embodiment action representation and trajectory smoothness with GESTURE-7 and Guide Action.
- Dual-frequency architecture: System 2 performs high-level semantic planning at ~10 Hz, while System 1 controls actions at ~20 Hz using guidance, high-frequency vision, and action history.System 2 uses Qwen3-VL to produce abstract action guidance; System 1 uses a Q-Former and Diffusion Transformer to generate execution actions.
- Guide Action: Guide Action conditions each generated action chunk on unexecuted prior steps to replace hard trajectory switches with seamless boundary alignment.It embeds chunk concatenation into the generative pipeline and uses a binary mask to preserve the anchor region M[1..k] = 1 while generating M[k + 1..T] = 0.
- Training pipeline: The training pipeline uses System 2 supervised fine-tuning and reinforcement learning before System 1 supervised fine-tuning with continuous actions.The three stages comprise SFT and GRPO-based RL for System 2, followed by joint System 2/System 1 optimization with frozen visual encoders and masked L2 supervision.
- Unified action representation: GESTURE-7 encodes end-effector states as a 7-dimensional vector and uses shared action keywords across heterogeneous robot embodiments.The vector contains [x, y, z, r, p, w, gesture], and actions such as grasp receive the same representation across grippers and dexterous hands.
4 Experiment
NebulaVLA is evaluated in simulation and on a real AgiBot A2 robot, showing strong LIBERO-Plus robustness, faster heterogeneous-frequency inference, and smoother trajectories with Guide Action. Ablations support the staged training strategy while identifying Camera perturbations as a limitation linked to GESTURE-7.
- Simulation Results: 85.5% overall success rate on LIBERO-Plus surpasses InternVLA-M1 (81.3%) by 4.2% and exceeds OpenVLA-OFT+ (79.6%), GR00T N1.5 (59.0%), and π0.5 (58.0%).LIBERO-Plus evaluates VLA policies across 7 perturbation dimensions and 10,030 tasks using Success Rate.
- Real-World Results: NebulaVLA-Heterogeneous achieves the highest success rates and lowest inference latency on both AgiBot A2 tasks, Pick-and-Place and Packaging Line Material Feeding.Experiments use stereo RGB inputs and an NVIDIA RTX 4070 GPU, with InternVLA-M1 as the comparison baseline.
- Real-World Results: 42 ms average step latency, down from 115 ms, accompanies improved success rates for the heterogeneous-frequency design versus the homogeneous-frequency variant.Decoupling System 1’s fast action response from System 2’s deep semantic reasoning enables the latency reduction and supports separate-device deployment.
- Training Strategy Ablation Study: 85.5% overall success rate for NebulaVLA(ALL) improves over NebulaVLA(w/o SFT, w/o RL) (83.8%) by 1.7% and NebulaVLA(w/o RL) (83.0%) by 2.5%.Stage 2 SFT improves Layout by 6.2%, while the staged S2 SFT, S2 RL, and S1 SFT strategy validates progressive stage-wise optimization.
- Guide Action Ablation Study: 25.6% average jerk reduction over the native asynchronous baseline is achieved by Guide Action across joints 7–13, with reductions of 41.2% in joint 12 and 45.1% in joint 13.Guide points align each new action chunk with the preceding trajectory, suppressing boundary discontinuities over 10 real-world trials.
5 Conclusions and Limitations
NebulaVLA addresses computational efficiency, cross-embodiment action representation, and trajectory continuity through heterogeneous frequency coordination. Although it outperforms synchronous baselines across simulation and real-world tasks, its empirically determined static frequency allocation remains a limitation.
- Conclusions: NebulaVLA targets computational efficiency versus control fidelity, cross-embodiment action representation, and trajectory continuity in robotic manipulation.These are identified as the paper’s three core challenges.
- Conclusions: The model decouples semantic planning at ˜10 Hz from reactive motor control at ˜20 Hz through a heterogeneous dual-frequency architecture.The architecture is presented as enabling modular deployment.
- Limitations: NebulaVLA consistently outperforms synchronous baselines across diverse manipulation tasks in simulation and real-world deployment.The results are described as validating heterogeneous frequency coordination.
- Limitations: The current frequency allocation is empirically determined and static, while optimal configurations may vary across task domains.Dynamic adaptive scheduling remains an open challenge.
A GESTURE-7: Natural Language-Based End-Effector State Representation
GESTURE-7 represents robot end-effector states with a natural-language-grounded 7-dimensional vector combining position, orientation, and a discrete gesture token. Its embodiment-shared keywords, representative keypoints, and quantization support unified and practical state prediction.
- Representation: Natural-language gesture keywords denote target hand postures, allowing different robot embodiments performing the same action to share identical keywords.This shared vocabulary is intended to facilitate cross-embodiment representation.
- Implementation details: Representative keypoints differ by morphology: the middle-finger root represents dexterous hands, while the gripper center represents parallel grippers.Position and orientation predictions are quantized to integer centimeters and nearest-degree angles to address mechanical tolerances and reduce learning burden and hallucination.
- Extensibility: The gesture vocabulary G can be freely extended according to task requirements and includes keywords for dexterous hands and grippers.Figure 5 illustrates seven dexterous-hand gesture categories and two gripper gesture categories.
B System 2 Supervised Fine-Tuning
NebulaVLA’s System 2 is adapted by directly fine-tuning task-specific data from original Qwen3-VL weights, avoiding additional pre-training. Five supervised tasks develop language-based state perception and GESTURE-7 trajectory representation and prediction.
- Fine-tuning strategy: Direct fine-tuning on task-specific data starts from original Qwen3-VL weights and bypasses additional pre-training stages.The authors report limited transfer from large-scale cross-embodiment manipulation data and multimodal perception capabilities for their target tasks, alongside a risk of catastrophic forgetting.
- Supervised tasks: Five supervised fine-tuning tasks target System 2’s natural language-based state perception and GESTURE-7-based trajectory representation and prediction.The tasks span pose regression, keypoint regression, 2D waypoint prediction, 3D trajectory prediction, and Chain-of-Thought generation.
- Spatial perception: Pose regression predicts the current end-effector state in GESTURE-7 format, while keypoint regression estimates 2D hand keypoint coordinates from visual observations.Both tasks provide spatial perception inputs for downstream manipulation reasoning and planning.
- Trajectory prediction: 2D waypoint prediction estimates future image-plane waypoints at 2Hz, and 3D trajectory prediction forecasts future end-effector GESTURE-7 trajectories at 2Hz.The 3D prediction operates over a future time horizon from the current visual frame.
- Structured planning: Chain-of-Thought generation produces subtask reasoning, a natural language subtask description, and a future GESTURE-7 trajectory from observations, instructions, and end-effector state.Together, the tasks equip System 2 with precise spatial perception and structured trajectory planning capabilities.
C Reward Functions of System 2 Reinforcement Learning
After System 2 Supervised Fine-Tuning, GRPO reinforces five Stage 1 tasks with specialized reward functions. These rewards assess GESTURE-7 state distance, keypoint localization, waypoint trajectories, trajectory predictions, and output format consistency.
- Reward design: GRPO reinforces five Stage 1 tasks with specialized reward functions after System 2 Supervised Fine-Tuning to improve trajectory planning.The reinforcement stage applies to the five tasks introduced in Stage 1.
- Reward design: GESTURE-7 distance combines weighted position distance, orientation distance, and gesture-consistency penalty, with all three weights set to 1.0.Position and orientation use Euclidean distances; gesture mismatch is a binary 0/1 indicator.
- Task-specific rewards: Keypoint localization reward measures predicted 2D keypoint accuracy against ground-truth annotations using an exponential penalty controlled by bandwidth σ.Larger σ produces more forgiving rewards.
- Task-specific rewards: Waypoints reward combines point-level Euclidean distance with trajectory-level Dynamic Time Warping distance to evaluate trajectory quality.This follows the approach of Mind to Hand.
- Task-specific rewards: Trajectory reward uses point-wise GESTURE-7 distance over actions sampled at 2Hz, typically covering K∈{2, 3, 4} points across a 1–2-second horizon.The short horizon limits hallucination risk from increasingly diverse and uncertain distant action predictions.
- Task-specific rewards: Format reward assigns 1 or 0 for regex-based conformity to a predefined output template and contributes to total reward with weight 0.5.The constraint ensures structural consistency between inference mode and subtask stages.