Source-linked AI summary
Algorithm-Architecture Co-Design for Efficient VLA Inference via Speculative Inference and Verification
Chunyu Qi, Zhuoran Song, Jian Weng, Haozhe Jiang, Xueyuan Liu, Naifeng Jing, Guanghui He, Xiaoyao Liang, Haibing Guan
TL;DR
VLA inference is too costly and slow for real-time robotic control. SpecVLA adaptively predicts long action sequences in inactive states and verifies them selectively in active states, significantly reducing end-to-end latency while maintaining task success rates and enabling real-time manipulation.
Problem
VLA models face high computational cost and inference latency exceeding 600 ms, while limited action lengths hinder real-time robotic control.
Method
SpecVLA adaptively adjusts action lengths by predicting environmental states and coordinating full VLA inference with lightweight sVLA verification.
Results
SpecVLA significantly reduces VLA end-to-end latency while maintaining task success rates across evaluated robotic manipulation benchmarks.
Takeaways & Limitations
SpecVLA enables real-time robotic manipulation with improved efficiency and maintained reliability.
Takeaways & Limitations
The approach can sacrifice model expressive depth, producing suboptimal actions that prolong physical execution.
Abstract
from arXiv · showhide
Vision-Language-Action (VLA) models have demonstrated remarkable capabilities in the field of embodied AI, but their high computational cost and limited predicted action length hinder real-time deployment. Although Dadu-Corki, a dedicated accelerator for efficient embodied AI, has been introduced, it does not exploit the inherent interaction patterns between the robot and its environment, which results in a relatively short predicted action length. We observe that robotic environments naturally alternate between active states-where precise actions are crucial-and inactive states-where actions have limited impact on task success. This insight enables a new scheduling opportunity: long-action-length speculative prediction in inactive states, paired with selective verification in active states. We propose SpecVLA, an algorithm-system co-design framework that adaptively balances action length, inference latency, and task reliability. On the algorithm side, SpecVLA introduces a state-aware VLA inference execution paradigm and a hardware-friendly construction of a smaller verification model (sVLA) using differential residuals and block-wise mixed-precision quantization. On the system side, we develop a heterogeneous architecture consisting of a GPU and a robotic-specific hardware module, along with a speculative dataflow that decouples VLA and sVLA through parallel execution. Comprehensive evaluations on OpenVLA and RDT across LIBERO and ManiSkill benchmarks show that SpecVLA reduces end-to-end latency significantly while preserving task success rate. By enabling long-action-length speculative prediction with timely verification, SpecVLA achieves real-time robotic manipulation with both high efficiency and reliability.
1 Introduction
SpecVLA addresses VLA inference latencies exceeding 600 ms by exploiting continuity in robotic environments and distinguishing active from inactive states. It co-designs adaptive action prediction, lightweight verification, hardware-friendly quantization, and heterogeneous execution for efficient real-time control.
- Motivation: VLA inference latencies exceeding 600 ms hinder real-time control and can reduce productivity while increasing costs in construction automation.Delayed action predictions are identified as the cause of these effects.
- Limitations of Existing Systems: Dadu-Corki hides communication and inference latency by streaming environmental frames to a GPU server during robot actions but overlooks frame continuity.Its heterogeneous system executes VLA inference on a GPU server and robot actions on an FPGA platform.
- Key Observation: Robotic execution alternates between active states requiring precise fine-grained control and inactive states where intermediate actions have limited impact on task success.Active states include aligning, grasping, or placing, whereas inactive states include moving the gripper toward a target pose.
- System Design: SpecVLA combines speculative long-action prediction with lightweight safety checks and a heterogeneous GPU–robotic-specific accelerator system.The GPU performs full-precision VLA inference, while the dedicated robotic hardware module supports system-level efficiency.
- Algorithm Design: SpecVLA adaptively determines predicted action-sequence length and verification by identifying active and inactive states at runtime.This algorithm-system co-designed framework exploits continuity in environmental frames for adaptive robotic action prediction.
- Hardware Design: SpecVLA introduces hardware-friendly VLA quantization based on differential computation and block-wise mixed-precision quantization.The method is listed as a core contribution of the framework.
2 Background and Motivation
This section introduces VLA inference for robotic-arm control and identifies three bottlenecks: high computational cost, sVLA accuracy degradation at longer action lengths, and poor hardware utilization from sequential verification. It motivates state-aware speculative execution and heterogeneous parallelization to address these limitations.
- Challenges: More than 3 TFLOPs make full-precision VLAs too slow for real-time control, while naive quantization often destroys accuracy.This establishes the central efficiency–accuracy challenge motivating specialized inference methods.
- Observations: Temporal redundancy between consecutive frames enables a lightweight sVLA based on mixed-precision differential residual quantization.Frame-to-frame feature residuals have a significantly narrower distribution, supporting block-wise quantization and substantially lower inference latency.
- Challenges: sVLA accuracy degrades at longer action prediction lengths, despite comparable success rates to the original VLA below two actions.The success rate measures completed tasks divided by total tasks, while action length counts low-level actions predicted per inference call.
- Observations: Environmental state correlates with prediction requirements: active states demand precise actions, whereas inactive states permit longer speculative predictions with limited impact on task success.Active states involve interaction with or approach to the target object, potentially changing its position or orientation.
- Challenges and Design Opportunity: Serialized VLA–sVLA verification increases latency because sVLA follows VLA and is invoked more frequently, preventing practical acceleration.A heterogeneous design assigns VLA inference to the GPU and sVLA inference to the NPU, using speculative dataflow to decouple their dependency.
3 SpecVLA Algorithm
SpecVLA adaptively generates long action sequences by predicting environmental states and coordinating VLA speculation with selective sVLA verification. It constructs sVLA with differential, block-wise mixed-precision quantization for hardware-efficient inference.
- State-aware speculative inference: SpecVLA predicts the robot’s environmental state and adaptively adjusts action length to maximize sequence length while maintaining an acceptable task success rate.The framework coordinates VLA and sVLA execution to balance speculative action length and reliability.
- State-aware speculative inference: In inactive states, SpecVLA uses VLA alone to produce long Predicted Actions without verification, whereas active states trigger verification before execution.Inactive states indicate limited action influence; active states indicate that upcoming actions may significantly affect task success.
- Environmental state prediction: Environmental state classification uses robot–object distance and robot-arm status: an open arm with no nearby objects is inactive, while proximity to an object is active.Distances from onboard perception sensors are compared with threshold T_d to determine whether objects are too close.
- Action verification: SpecVLA validates Predicted Actions against sVLA Expected Actions using weighted L1 distance after min–max normalization, accepting them below threshold T_s and rejecting them otherwise.Weights are set as w[k] = 1/σ[k], reducing the influence of high-variance dimensions and emphasizing stable dimensions.
- sVLA construction: sVLA computes input-feature residuals and applies block-wise mixed-precision quantization using thresholds T_z and T_h, removing zero blocks before layer-wise accumulation.Blocks are zeroed when insignificant, quantized at high precision when important, and otherwise assigned low precision.
4 SpecVLA Heterogeneous Architecture
SpecVLA combines speculative parallel execution with a GPU–robotic-specific hardware architecture to reduce inference dependencies and efficiently support verification, state prediction, and mixed-precision sVLA computation. A reconfigurable shared distance-computation datapath further accelerates state prediction and action verification while reducing hardware overhead.
- Sequential and Speculative Dataflow: The sequential design serializes VLA prediction, sVLA verification, and action execution because each stage depends on newly produced data.Prediction depends on newly observed images, verification on predicted actions, and execution on verification results.
- Sequential and Speculative Dataflow: Speculative dataflow decouples VLA prediction from sVLA verification by running VLA inference in advance, enabling concurrent execution on separate GPUs.The two-GPU design assigns sVLA verification to GPU2 and speculative VLA prediction to GPU1.
- Speculative Execution Control: SpecVLA safely recovers failed speculative actions through compensatory reverse motions, verifying after at most one motion primitive to avoid irreversible state transitions.Physically irreversible effects such as full gripper closure require multi-step execution, while verification occurs before that threshold.
- Heterogeneous Architecture: The heterogeneous architecture pairs a GPU with a robotic-specific module that accelerates sVLA through block splitting, unified precision, and a SIMD-based PE array.The module also integrates a state predictor and preprocessing module to support the SpecVLA dataflow.
- Reconfigurable State Predictor: A reconfigurable state predictor uses one distance-computation datapath for environmental state prediction and action verification, reducing area overhead while accelerating both operators.Lightweight control signals switch the shared subtract–compute–reduce pipeline between the two modes.
5 Evaluation
SpecVLA is evaluated across OpenVLA and RDT robotic manipulation workloads, showing reliable task performance, improved speculative efficiency, and substantial latency reductions. Its hardware, quantization, and verification designs jointly improve efficiency while preserving success rates.
- Evaluation Setup: SpecVLA is evaluated with OpenVLA and RDT across LIBERO and ManiSkill robotic manipulation environments.OpenVLA is deployed on a Franka Emika Panda arm and evaluated using four LIBERO task suites.
- Task Reliability: SpecVLA incurs negligible success-rate loss over baselines in noise-free environments and maintains reliability under modeled sensor noise.Success rate measures the proportion of successfully completed tasks.
- Speculation Efficiency: Inactive states comprise an average 74.2% of execution, enabling long-action speculative prediction while active states receive verification.The analysis measures active/inactive-state ratio, rollback frequency, effective action length, and path length.
- Model Compression: The proposed quantization improves average success rate by 2.50% over QVLA, 6.25% over QuantVLA, 4.25% over Eventful Transformer, 3.50% over MaskVD, and 3.25% over SP-VLA.It also compresses theoretical FLOPs to 10.22% of OpenVLA.
- Latency: SpecVLA achieves average speedups of 12.1×, 2.9×, and 1.9× over CPU, GPU, and Dadu-Corki-ADAP, respectively.The comparison uses full-precision VLA execution on Intel CPU and NVIDIA A100 GPU, plus Dadu-Corki-ADAP.
- Ablation and Sensitivity: Verification improves task success rate by approximately 7%, speculative dataflow reduces latency by 79.8%, and the robotic-specific accelerator reduces end-to-end latency by 20.6%.The chosen action length L=8 and threshold T_d=0.2 provide the reported success-rate and speedup trade-off.
6 Related Works
Related work spans optimization-based robot decision-making, LLM-controlled robotics, and hardware accelerators, alongside temporal redundancy pruning, dynamic layer-skipping, and speculative execution. SpecVLA is distinguished by parallel prediction with serial, sequential action verification against real-time environmental feedback.
- Robot control and accelerators: Traditional robots use optimization-based algorithms, whereas emerging applications use LLMs for manipulation, planning, and navigation.The passage states that LLM-based applications demonstrate capabilities far superior to traditional algorithms.
- Robot control and accelerators: Many accelerators have been designed as robots increasingly become the next generation of computing platforms.
- Speculative execution and verification: SpecVLA relates to temporal redundancy pruning, dynamic layer-skipping, and speculative execution, but introduces parallel prediction with serial verification.It sequentially validates each action against real-time environmental feedback for closed-loop responsiveness during abrupt physical mutations.
7 Conclusion
SpecVLA is an algorithm–architecture co-design framework that adaptively verifies predicted actions based on environmental state to reduce VLA end-to-end latency while maintaining success rate, enabling real-time robotic manipulation.
- Conclusion: SpecVLA adaptively verifies predicted actions according to the environmental state.This state-aware verification is the framework’s key idea.
- Conclusion: SpecVLA significantly reduces VLA model end-to-end latency while maintaining success rate.The reported benefit addresses the excessive latency of current VLAs, which is insufficient for practical applications.
- Conclusion: SpecVLA enables real-time robotic manipulation in practical applications.This follows from the combination of lower end-to-end latency and maintained success rate.