Source-linked AI summary
DySL-VLA: Efficient Vision-Language-Action Model Inference via Dynamic-Static Layer-Skipping for Robot Manipulation
Zebin Yang, Yijiahao Qi, Tong Xie, Bo Yu, Shaoshan Liu, Meng Li
TL;DR
VLA models achieve strong robotic manipulation performance, but their computational cost hinders real-time deployment. DySL-VLA dynamically skips layers according to action importance while preserving informative layers and using skip-aware training. It reports improved successful length over DeeR-VLA, reduced trainable parameters and training steps, and lower latency than RoboFlamingo.
Problem
VLA models’ high computational demands create latency and power-consumption challenges for real-world robots with limited resources and battery capacity.
Method
DySL-VLA combines dynamic-static layer skipping, prior-post skipping guidance, and skip-aware two-stage knowledge distillation to allocate computation by action importance.
Results
2.1% average successful length improvement over DeeR-VLA is reported on Calvin D →D, with 85.7× fewer trainable parameters, 13.7× fewer training steps, and 3.75× latency reduction over RoboFlamingo.
Takeaways & Limitations
DySL-VLA provides an inference-acceleration approach that preserves informative layers while reducing training and inference costs.
Takeaways & Limitations
Skipping-controller mechanisms can add non-negligible serial inference latency, and fine-grained skipping may provide little latency reduction when only half the layers are activated.
Abstract
from arXiv · showhide
Vision-Language-Action (VLA) models have shown remarkable success in robotic tasks like manipulation by fusing a language model's reasoning with a vision model's 3D understanding. However, their high computational cost remains a major obstacle for real-world applications that require real-time performance. We observe that the actions within a task have varying levels of importance: critical steps demand high precision, while less important ones can tolerate more variance. Leveraging this insight, we propose DySL-VLA, a novel framework that addresses computational cost by dynamically skipping VLA layers based on each action's importance. DySL-VLA categorizes its layers into two types: informative layers, which are consistently executed, and incremental layers, which can be selectively skipped. To intelligently skip layers without sacrificing accuracy, we invent a prior-post skipping guidance mechanism to determine when to initiate layer-skipping. We also propose a skip-aware two-stage knowledge distillation algorithm to efficiently train a standard VLA into a DySL-VLA. Our experiments indicate that DySL-VLA achieves 2.1% improvement in success length over Deer-VLA on the Calvin dataset, while simultaneously reducing trainable parameters by a factor of 85.7 and providing a 3.75x speedup relative to the RoboFlamingo baseline at iso-accuracy. Our code is available on https://github.com/PKU-SEC-Lab/DYSL_VLA.
1 Introduction
VLA models enable end-to-end robotic control but their computational demands limit real-time deployment. DySL-VLA exploits unequal action and layer importance through dynamic-static skipping and skip-aware training.
- VLA models use images and language instructions to generate robot control commands, supporting end-to-end manipulation.
- High computational demands cause latency and power consumption that conflict with robotic platforms’ limited resources and battery capacity.
- Existing acceleration methods often apply uniform computation despite unequal action importance, limiting speedup on less important actions.
- DySL-VLA statically keeps informative layers while dynamically skipping unnecessary layers according to action importance.Prior-post skipping guidance and skip-aware two-stage knowledge distillation support important-action correctness and training convergence.
- 3.75× latency reduction over RoboFlamingo accompanies 2.1% average successful length improvement over DeeR-VLA and 85.7× fewer trainable parameters.The same contribution passage also reports 13.7× fewer training steps.
2 Background
VLA models predict robot actions from visual observations and language but face high computation costs and low control frequency. DySL-VLA addresses this by allocating more computation to important actions while reducing information loss and training overhead.
- VLA models are fine-tuned from pretrained vision-language models to perform end-to-end robot control.
- Existing pruning, quantization, and mixture-of-layers methods allocate equal computation across actions, wasting acceleration opportunities on unimportant actions.
- Early-exit methods dynamically adjust computation but can lose information by skipping final layers, requiring additional action heads or backbone training to recover performance.
- DySL-VLA adaptively applies more computation to important actions while using dynamic-static layer skipping to reduce information loss.It trains lightweight skipping controllers and adapters instead of the LLM backbone.
3 Efficient VLA Inference via Dynamic-Static Layer-Skipping
DySL-VLA reduces VLA inference cost by statically retaining informative layers and dynamically skipping other layers according to action importance. Prior-post guidance protects important actions while adapters and skip-aware training reduce information loss and training cost.
- Observation and Overview: Informative VLA layers can be critical to performance, while directly skipping all less informative layers also causes extremely low accuracy.DySL-VLA therefore distinguishes statically retained layers from dynamically skipped layers.
- Observation and Overview: Skipping controllers alone keep relatively few layers across actions, cannot guarantee important-action correctness, and may provide little latency reduction because of controller overhead.With half the layers activated, controller-based skipping gains little latency over the baseline.
- Dynamic-static Layer Skipping: DySL-VLA statically keeps informative layers, dynamically skips other layers to the next static layer, and uses adapters to fit activations for the next retained layer.This skips multiple layers at once rather than only one layer per decision.
- Skip-aware Two-stage Knowledge Distillation: The method avoids training the LLM backbone and multiple action heads, reducing training cost while achieving higher speedup with low information loss.DySL-VLA trains lightweight adapters and skipping controllers instead.
- Dynamic-static Layer Skipping: Trajectory continuity guides skipping because smooth actions usually correspond to non-critical motion, whereas continuity breaks during fine operations such as grasping or releasing.The method estimates current action importance from the recent action trajectory.
- Prior-post Skipping Guidance: Prior-post guidance shifts the skipping-allow point forward when continuity decreases and backward when continuity improves, keeping enough layers for important actions while reducing controller latency.Pre-skip prediction controls when skip controllers are active; the controllers still make the actual skipping decision.
4 Experiments
DySL-VLA is evaluated on CALVIN and LIBERO, where it improves task performance while reducing latency and training costs. Ablations examine the contributions of its skipping guidance, layer selection, and hyperparameters.
- Accuracy comparison: On CALVIN D →D, DySL-VLA improves average successful length by 2.1% over DeeR-VLA while reducing trainable parameters 85.7× and training steps 13.7×.The benchmark measures average successful length across five subtasks.
- Accuracy comparison: On LIBERO, DySL-VLA improves average SR by 1.2% over DeeR-VLA and reduces trainable parameters 31.4×.It also improves average SR by 41.3% over FlexiDepth.
- Latency Comparison: 3.75× latency reduction over full RoboFlamingo is achieved on CALVIN through dynamic skipping of unnecessary VLA layers.The method also reports 2.03× reduction versus FlexiDepth and 1.42× versus DeeR-VLA.
- Latency Comparison: On LIBERO, latency reductions versus FlexiDepth and DeeR-VLA reach 1.54× and 1.47× on A6000, and 1.46× and 1.43× on Jetson Orin.Compared with full RoboFlamingo, reductions are 1.93× on A6000 and 1.96× on Jetson Orin.
- Latency Comparison: DySL-VLA reaches a 23.2Hz control frequency on Jetson Orin because OpenVLA-oft predicts eight actions per inference using action chunks.The reported latency is higher on Jetson Orin because of its limited computation resources.
- Ablation studies: Removing pre-skip prediction, post-skip verification, dynamic-static skipping, or skip-aware distillation causes accuracy degradation.The experiments attribute these drops to weaker protection of important actions, information loss, or interference during training.
5 Conclusion
The paper presents DySL-VLA as an action-aware acceleration framework for VLA models. Its experiments report improved CALVIN performance alongside substantially lower training cost.
- Conclusion: DySL-VLA dynamically skips unnecessary layers according to action importance while statically retaining informative layers to reduce information loss.The framework also uses prior-post skipping guidance to retain enough layers for important actions.
- Conclusion: 2.1% average successful length improvement over DeeR-VLA is reported on CALVIN D →D.The same result is accompanied by reductions in trainable parameters and training steps.
- Conclusion: 85.7× fewer trainable parameters and 13.7× fewer training steps are reported relative to the comparison setting.These reductions are reported together with the CALVIN performance improvement.