Source-linked AI summary
HiVLA: A Visual-Grounded-Centric Hierarchical Embodied Manipulation System
Tianshuo Yang, Guanyu Chen, Yutian Chen, Zhixuan Liang, Yitian Liu, Zanxin Chen, Chunpu Xu, Haotian Liang, Jiangmiao Pang, Yao Mu, Ping Luo
TL;DR
End-to-end VLA fine-tuning can compromise pretrained VLM reasoning on narrow manipulation data. HiVLA decouples VLM planning from DiT-based action generation through visual-grounded plans, and reports substantial gains over strong baselines, especially for long-horizon and fine-grained manipulation.
Problem
Fine-tuning VLAs on scarce, domain-specific manipulation data can degrade the reasoning capabilities inherited from pretrained VLMs.
Method
HiVLA uses a VLM planner to produce semantic subtasks and target bounding boxes, then a DiT action expert integrates global, local, and language features through cascaded cross-attention.
Results
12.7% improvement over H-RDT and 37.7% over π0 is reported in simulation, with particular strength in long-horizon skill composition and fine-grained manipulation of small objects in cluttered environments.
Takeaways & Limitations
The modular architecture enables independent scaling of the VLM planner and action expert and provides interpretability through explicit intermediate plans.
Abstract
from arXiv · showhide
While end-to-end Vision-Language-Action (VLA) models offer a promising paradigm for robotic manipulation, fine-tuning them on narrow control data often compromises the profound reasoning capabilities inherited from their base Vision-Language Models (VLMs). To resolve this fundamental trade-off, we propose HiVLA, a visual-grounded-centric hierarchical framework that explicitly decouples high-level semantic planning from low-level motor control. In high-level part, a VLM planner first performs task decomposition and visual grounding to generate structured plans, comprising a subtask instruction and a precise target bounding box. Then, to translate this plan into physical actions, we introduce a flow-matching Diffusion Transformer (DiT) action expert in low-level part equipped with a novel cascaded cross-attention mechanism. This design sequentially fuses global context, high-resolution object-centric crops and skill semantics, enabling the DiT to focus purely on robust execution. Our decoupled architecture preserves the VLM's zero-shot reasoning while allowing independent improvement of both components. Extensive experiments in simulation and the real world demonstrate that HiVLA significantly outperforms state-of-the-art end-to-end baselines, particularly excelling in long-horizon skill composition and the fine-grained manipulation of small objects in cluttered scenes.
1 Introduction
HiVLA addresses catastrophic forgetting in end-to-end VLAs by decoupling VLM-based planning from low-level control through visual-grounded structured plans. Its DiT action expert combines global context, high-resolution local features, and subtask guidance, achieving strong results in simulation and real-world evaluations.
- End-to-end VLA fine-tuning on scarce, domain-specific manipulation data can degrade the reasoning capabilities inherited from pretrained VLMs.
- HiVLA uses a VLM to decompose complex instructions and ground target objects, producing a semantic subtask label and precise bounding box.
- Its DiT action expert sequentially conditions actions on global visual context, high-resolution position-aware local features, and language-based skill guidance.
- 12.7% absolute success rate improvement over H-RDT and 37.7% over π0 is reported on RoboTwin 2.0, alongside strength in long-horizon composition and fine-grained cluttered-scene manipulation.
- The architecture explicitly decouples high-level VLM planning from low-level control, preserving reasoning capabilities while allowing separate component improvements.
2 Related Work
Related work spans monolithic and hierarchical VLAs, with hierarchical systems using intermediate representations to separate reasoning from control. Visual grounding methods vary from implicit alignment to explicit spatial representations, each addressing perception-action coupling differently.
- Monolithic VLAs include single-system models that decode actions directly and dual-system models that use a VLM backbone to guide an action expert.
- Hierarchical models decouple task planning from policy execution through interpretable bridges such as textual subtasks or spatial keypoints.
- Visual-centric VLAs use vision-language alignment and integrated grounding techniques to localize targets and direct diffusion-transformer attention.
- Explicit grounding systems use segmentation masks to isolate target objects, but dense-mask generation may require external expert models and compromise general visual capabilities.
- HiVLA’s pipeline combines VLM-generated subtasks and bounding boxes with a DiT action expert using cascaded cross-attention.
3 Problem Formulation
The paper formulates language-guided manipulation as conditional sequence generation from multimodal observation histories and language instructions to future robot actions. The central challenge is converting complex instructions and cluttered visual inputs into grounded, robust motor commands.
- At each timestep, the policy receives multi-view visual observations and the robot’s proprioceptive state as multimodal input.
- Given observation history and a language instruction, the policy generates a sequence of future actions over prediction horizon H.
- Each action specifies target control commands, such as joint positions, for the robot arm and gripper.
- The formulation’s core difficulty is bridging abstract, long-horizon language instructions with precise motor-command sequences.
- Robust execution requires task decomposition, visual grounding of the correct object amid clutter, and translation of grounded intent into actions.
4 Method
HiVLA uses a VLM planner to produce structured, visually grounded subtasks, then a DiT action expert converts those plans into motor commands through conditional flow matching and hierarchical visual-language conditioning.
- High-Level VLM Planner: The VLM planner interprets language, visual observations, gripper state, previous subtasks, and visual history to determine the next action.
- High-Level VLM Planner: The bounding box invokes a crop tool that extracts a high-resolution object-centric patch, which joins the subtask description and original observation as DiT guidance.
- DiT Action Expert: The DiT action expert models p(A_t|S_0:t, L_sub,t, B_t) to translate grounded plans into low-level motor commands.
- Conditional Flow Matching: Conditional Flow Matching learns a vector field connecting Gaussian noise to action sequences under the planner’s context.
- Conditional Flow Matching: During inference, an ODE solver integrates the learned vector field from noise to generate the conditioned action sequence.
- Hierarchical Transformer Architecture: Each transformer block sequentially fuses global scene features, position-aware high-resolution local features, and task-specific guidance through cascaded cross-attention.
- Hierarchical Transformer Architecture: Absolute positional embeddings preserve the target’s location in the original camera frame while retaining fine-grained crop details for precise manipulation.
5 Experiments
HiVLA is evaluated across simulation and real-world manipulation settings, with comparisons to state-of-the-art baselines, robustness tests, and architectural ablations. It achieves strong performance on long-horizon tasks, small-object manipulation, and cluttered scenes.
- Experimental Setup: Evaluations cover RoboTwin2.0 simulation, real-world manipulation, nine tasks, and four state-of-the-art baselines.The task suite includes four Easy Tasks and five Hard Tasks, while baselines include π0, π0.5, StarVLA, and H-RDT.
- Simulation Results: 83.3% total average success rate establishes HiVLA as the strongest simulator performer across the evaluated tasks.HiVLA reaches 96.0% on Easy Tasks and 73.2% on Hard Tasks, where coupled VLAs average below 40%.
- Ablation Study: 8.8% performance loss on Hard Tasks occurs when specific subtask skills are replaced by the global instruction.The ablation attributes the difference to decomposed one-to-one language conditions for the diffusion policy.
- Robustness: 57.0% success under 100% target-bounding-box shift demonstrates resilience to spatial guidance noise.The robustness study contrasts tolerance to bounding-box perturbations with strict adherence to language instructions.
- Ablation Study: 83.3% average success results from coarse-to-fine conditioning that combines global context, local crops, and language skill.Using only local or global visual features produces approximately 70% average performance; low-resolution crops and missing absolute positional encoding degrade fine-grained manipulation.
6 Conclusion
HiVLA decouples VLM-based planning from DiT-based action generation through visual grounding and cascaded cross-attention. Simulation and real-world experiments show stronger performance than state-of-the-art baselines, especially for long-horizon and fine-grained manipulation.
- Conclusion: HiVLA decouples high-level task decomposition and visual grounding from low-level action generation.The VLM produces grounded plans, while the DiT action expert uses global context, position-aware local features, and subtask guidance.
- Conclusion: 12.7% over H-RDT and 37.7% over π0 are the reported simulation improvements over state-of-the-art baselines.The gains are accompanied by particular strength in long-horizon skill composition and fine-grained manipulation of small objects in cluttered environments.
- Conclusion: Independent component scaling and explicit intermediate plans support a modular, interpretable foundation for complex robotic manipulation.The conclusion identifies these properties as benefits beyond performance gains.
1 DiT Model Details
The DiT action expert uses a high-capacity Transformer with multimodal conditioning for visual, language, state, and action inputs. Its implementation combines grouped-query attention, SwiGLU feed-forward layers, and frozen pretrained vision features.
- Implementation Details: Training uses PyTorch and HuggingFace Accelerate on two NVIDIA H200 GPUs with AdamW optimization.The configuration includes weight decay of 1 × 10^-2, gradient clipping at 1.0, and a learning rate peaking at 1 × 10^-4 after warmup.
- Architecture Specifications: The action expert is a 16-layer Transformer with hidden dimension 2,176, 16 attention heads, and 8 key-value heads.It uses Grouped Query Attention, SwiGLU feed-forward networks, and LayerNorm.
- Input Conditioning: Frozen DINOv2 and SigLIP backbones provide pretrained visual representations during training.Separate MLP projectors map visual, language, state, and action embeddings into the Transformer latent space.
2 VLM Planner Agent Analysis
The VLM Planner Agent is evaluated for grounding and subtask prediction under different model scales, fine-tuning settings, and visual-history conditions. Domain-specific fine-tuning substantially improves the lightweight Qwen3-VL planner, while historical observations support task progression.
- Experimental Setup and Metrics: A 210K-instance dialogue dataset from HiVLA-HD supports independent evaluation of Qwen3-VL planning.The planner is assessed using bounding-box mIoU and strict exact-match subtask prediction.
- Fine-Tuning vs. Zero-Shot Capabilities: 42.85% zero-shot subtask accuracy versus 3.45% mIoU illustrates stronger zero-shot semantic prediction than native spatial grounding for GPT-4o.Larger models improve zero-shot subtask accuracy and grounding, but precise long-horizon manipulation remains insufficient without fine-tuning.
- Fine-Tuning vs. Zero-Shot Capabilities: 90.37% mIoU and 98.57% subtask accuracy are achieved by the fine-tuned Qwen3-VL 8B planner.The result demonstrates a large domain-specific improvement while retaining the pretrained VLM’s generalizable priors.
- The Necessity of Visual History: 95.24% subtask accuracy without visual history falls from 98.57%, showing that historical frames provide temporal context for task progression.The ablation removes historical frames during fine-tuning.
- Extensibility and Future Scaling: The decoupled planner can be replaced by advanced foundation models as plug-and-play components for future scaling.The passage states that domain-specific fine-tuning is currently the most optimal deployment strategy while larger models improve zero-shot capability.
3 Task Visualization
The experiments document task instructions and execution visualizations across RoboTwin simulation and real-world scenarios, alongside the VLM planner’s structured-plan prompt.
- Task visualizations cover both the RoboTwin simulation environment and real-world scenarios.Simulation execution sequences are shown in Fig. 4, while corresponding real-world processes appear in Fig. 5.
- The VLM planner receives historical and current observations with state information and generates a structured subtask plan.
- The task list provides the specific natural language instructions used in the RoboTwin simulation and real-world experiments.