PaperlayerView exact revision on arXiv

Source-linked AI summary

SP-VLA: A Joint Model Scheduling and Token Pruning Approach for VLA Model Acceleration

Ye Li, Yuan Meng, Zewen Sun, Kangye Ji, Chen Tang, Jiajun Fan, Xinzhu Ma, Shutao Xia, Zhi Wang, Wenwu Zhu

arXiv:2506.12723v3cs.CVcs.AI

TL;DR

VLA models are costly to execute in real time, while existing acceleration methods leave temporal redundancy in sequential actions and spatial redundancy in visual inputs insufficiently addressed. SP-VLA jointly schedules a full VLA model and lightweight generator by action type and prunes tokens using spatio-semantic importance. It achieves lossless acceleration on LIBERO and SimplerEnv, with reported gains in performance, inference frequency, and latency.

  • Problem

    VLA models have high computational cost and low execution frequency, while existing acceleration methods overlook temporal redundancy in sequential action generation and spatial redundancy in visual input.

  • Method

    SP-VLA jointly schedules a full VLA model and lightweight action generator for deliberative and intuitive actions, then applies spatio-semantic token pruning during VLA invocation.

  • Results

    1.5× lossless speedup in LIBERO and 2.4× in SimplerEnv are accompanied by up to 6% performance gain, with inference frequency and latency improving by 1.4× and 2.2× respectively.

  • Takeaways & Limitations

    The joint model-data strategy supports frequency-aware, task-adaptive VLA acceleration while maintaining high accuracy across the reported environments.

  • Takeaways & Limitations

    Intuitive action generation remains limited to model lightweighting, so deliberative and intuitive action generation are not yet completely separated.

Abstract

from arXiv · show

Vision-Language-Action (VLA) models have attracted increasing attention for their strong control capabilities. However, their high computational cost and low execution frequency hinder their suitability for real-time tasks such as robotic manipulation and autonomous navigation. Existing VLA acceleration methods primarily focus on structural optimization, overlooking the fact that these models operate in sequential decision-making environments. As a result, temporal redundancy in sequential action generation and spatial redundancy in visual input remain unaddressed. To this end, we propose SP-VLA, a unified framework that accelerates VLA models by jointly scheduling models and pruning tokens. Specifically, we design an action-aware model scheduling mechanism that reduces temporal redundancy by dynamically switching between VLA model and a lightweight generator. Inspired by the human motion pattern of focusing on key decision points while relying on intuition for other actions, we categorize VLA actions into deliberative and intuitive, assigning the former to the VLA model and the latter to the lightweight generator, enabling frequency-adaptive execution through collaborative model scheduling. To address spatial redundancy, we further develop a spatio-semantic dual-aware token pruning method. Tokens are classified into spatial and semantic types and pruned based on their dual-aware importance to accelerate VLA inference. These two mechanisms work jointly to guide the VLA in focusing on critical actions and salient visual information, achieving effective acceleration while maintaining high accuracy. Extensive experiments show that our method achieves 1.5$\times$ lossless acceleration in LIBERO and 2.4$\times$ in SimplerEnv, with up to 6% average performance gain. Inference frequency and latency improve by 2.2$\times$ in SimplerEnv and 1.4$\times$ in LIBERO.

1 INTRODUCTION

SP-VLA targets computational and temporal-spatial redundancies that limit VLA models in real-time settings. It jointly schedules a lightweight generator with the VLA model and prunes tokens using spatial and semantic information, achieving substantial acceleration while preserving accuracy.

  • Motivation: VLA models remain computationally expensive because large parameter counts produce slow inference unsuitable for real-time scenarios.OpenVLA exceeds 7 billion parameters, while the RT-X series contains over 55 billion parameters.
  • Motivation: Existing acceleration methods mainly reduce single-step computation redundancy through compression, overlooking sequential decision-making characteristics of VLA models.Prior approaches include pruning, quantization, caching, early exit, and imitation-learning-based quantization.
  • Joint acceleration framework: SP-VLA reduces temporal redundancy by assigning intuitive actions to a lightweight model and deliberative actions to the VLA model.The schedule dynamically switches models according to action type, enabling frequency-adaptive execution.
  • Joint acceleration framework: SP-VLA reduces spatial redundancy with spatio-semantic token pruning that combines edge features, token positions, and semantic importance.The method preserves spatial information needed for VLA perception while reducing less relevant visual tokens.
  • Results: 1.5× lossless speedup in LIBERO and 2.4× in SimplerEnv accompany up to 6% performance gain and higher inference frequency and lower latency.Inference frequency and latency improve by 1.4× in LIBERO and 2.2× in SimplerEnv.

2 RELATED WORK

Prior VLA work includes large-scale embodied models and efficiency methods, while SP-VLA combines action scheduling with token pruning to address temporal and spatial redundancy. Its design reflects both mixed action dynamics and the spatial sensitivity of VLA perception.

  • Vision-Language-Action Models: VLA research includes large-scale models such as RT-1, RT-2, RT-X, RT-H, and OpenVLA for embodied control.OpenVLA builds on LLaMA 2 reasoning capabilities, while diffusion models have also been used to improve temporal action coherence.
  • Acceleration for Vision-Language-Action Models: VLA efficiency methods use quantization, frequency-domain action compression, token reuse, and other mechanisms to reduce inference cost.QAIL quantizes policies, Fast analyzes action spectra, and VLA-Cache reuses less informative tokens.
  • SP-VLA: VLA action sequences contain deliberative and intuitive components, motivating model scheduling between a VLA model and a lightweight action generator.The scheduling strategy assigns different action types to different models for frequency-adaptive acceleration.
  • SP-VLA: SP-VLA combines action-aware model scheduling with token pruning according to task complexity to accelerate temporal and spatial computation.This joint strategy is designed around the mixed dynamics of sequential actions and the visual information required for control.

3 A JOINT MODEL SCHEDULING AND TOKEN PRUNING APPROACH FOR VLA MODEL ACCELERATION

SP-VLA jointly reduces temporal and spatial redundancy by scheduling between a VLA model and lightweight action generator while pruning tokens using semantic and spatial information. Its scheduling and pruning adapt to action type, motion speed, and token importance to accelerate inference while preserving task performance.

  • Action Type-Aware Model Scheduling: SP-VLA classifies actions as intuitive or deliberative, assigning intuitive actions to a lightweight generator and deliberative actions to the VLA model.The scheduler analyzes historical trajectories in an action buffer before selecting the model for each time step.
  • Action Type-Aware Model Scheduling: The lightweight generator predicts translational and rotational components, while the binary end-effector state is reused from the previous time step.End-effector state transitions remain delegated to the VLA model, and predicted actions pass a validity check before execution.
  • Action Type-Aware Model Scheduling: Small-step, high-frequency model switching accelerates inference while maintaining action-direction accuracy.Velocity thresholds and the proportion of VLA-generated actions in the buffer determine when the lightweight model can be used.
  • Action Type-Aware Model Scheduling: The lightweight generator fits short intuitive action segments with Ridge Regression using recent actions stored in an action buffer.The action buffer models the relationship between time and velocity, while regularization stabilizes parameter fitting.
  • Spatio-Semantic Dual-Aware Token Pruning: Token pruning combines semantic attention importance with Canny-derived contour information and preserves the original token ordering.The selected semantic and spatial token sets are merged before entering the LLM backbone.
  • Experimental Results: 1.35× speedup is achieved in LIBERO without performance loss, while 1.5× acceleration incurs less than a 3% accuracy drop.These results are reported for comparisons with state-of-the-art methods after preserving relative token positions and incorporating Canny edge information.
  • Spatio-Semantic Dual-Aware Token Pruning: Token pruning is disabled at low speed and becomes more aggressive as velocity increases, aligning visual computation with motion conditions.The retained-token ratio is defined using minimum and maximum velocity thresholds.

4 EXPERIMENTAL RESULTS

Experiments across LIBERO and SimplerEnv show that SP-VLA accelerates VLA inference while preserving or improving task performance. Ablations and analyses indicate that model scheduling addresses temporal redundancy, whereas token pruning must preserve spatial information such as object contours.

  • Main results: SP-VLA achieves a 1.35× speedup without accuracy loss across LIBERO tasks and up to 1.5× speedup with a 3% accuracy drop.On SimplerEnv, it achieves 2× speedup with improved performance, including about 27% higher performance and 1.8× speedup on Visual Aggregation Drawer.
  • Main results: SP-VLA achieves SOTA performance across diverse SimplerEnv tasks while improving both speed and accuracy.
  • Ablation results: Token pruning is sensitive to spatial information: disrupting token order or removing object-edge information substantially degrades performance and can cause task failure.These findings motivate preserving positional and contour information during visual-token reduction.
  • Ablation results: Model scheduling provides the strongest individual acceleration, achieving 1.27× speedup with only a 1% accuracy drop.The joint application of model scheduling and token pruning yields the best overall acceleration without compromising accuracy.
  • Frequency and latency: SP-VLA increases SimplerEnv inference frequency by 2.2× and reduces per-inference latency by 2.2×, while improving LIBERO frequency by 1.4×.Measurements were conducted on a single NVIDIA RTX 4090 in SimplerEnv.
  • Visualizations: Visualizations show that SP-VLA prunes redundant image regions while preserving object-edge information needed for spatial perception.

5 CONCLUSION

SP-VLA jointly schedules a full VLA model and lightweight generator according to action type, while pruning tokens using spatio-semantic importance. Experiments report acceleration and performance gains across LIBERO and SimplerEnv, alongside improved inference frequency and latency.

  • SP-VLA jointly schedules models and prunes tokens to accelerate VLA models.It switches between a full VLA model and lightweight generator for deliberative or intuitive actions, and prunes tokens using spatio-semantic importance during VLA invocation.
  • SP-VLA achieves 1.5× lossless speedup in LIBERO and 2.4× in SimplerEnv, with up to 6% performance gain.
  • Inference frequency and latency improve by 1.4× in LIBERO and 2.2× in SimplerEnv.

6 ETHICS STATEMENT

The work concerns accelerating embodied models through robotic-arm simulations and real-world experiments, without human subjects or sensitive personal data.

  • The study uses robotic-arm simulations and real-world experiments and does not involve human subjects or sensitive personal data.
  • The authors report no foreseeable misuse or harm and no issues involving bias, discrimination, privacy, or security.
  • The experiments were conducted in compliance with standard research-integrity practices.

7 REPRODUCIBILITY STATEMENT

The authors report reproducibility measures including repeated runs, public backbones and environments, and planned release of implementation details.

  • Multiple random seeds and repeated runs were used to confirm result robustness.
  • The backbone models, checkpoints, and experimental environments are publicly available and open-source or accessible.
  • Hyperparameters and implementation details will be documented and released in a public GitHub repository.

A.1 THE USE OF LARGE LANGUAGE MODELS (LLMS)

The work used LLMs only to polish language for clarity and readability, not to generate ideas or conduct the research.

  • LLMs were used solely to polish the language for clarity and readability.
  • No LLMs were used for idea generation or experimental design.
  • No LLMs were used for data analysis or other parts of the research process.

A.2 LANTENCY AND FREQUENCY

SP-VLA improves VLA inference frequency and latency in both LIBERO and SimplerEnv while preserving model accuracy.

  • 1.4× inference speedup is achieved by SP-VLA in the LIBERO environment.The measurement uses an RTX 4090.
  • 2.2× inference speedup is achieved by SP-VLA in the SimplerEnv environment.The measurement uses an RTX 4090.
  • Approximately 1.4× and 2.2× inference-rate improvements are achieved for OpenVLA and CogACT, respectively, while preserving model accuracy.OpenVLA and CogACT otherwise achieve only 3–4 Hz inference frequency.

A.3 SENSITIVITY ANALYSIS OF KEY PARAMETERS.

The experiments examine parameter sensitivity, baseline limitations, environment robustness, and the separate contributions of model scheduling and token pruning. Results indicate that SP-VLA remains robust across settings, with scheduling providing the larger contribution and the two mechanisms complementing each other, while intuitive-action generation remains preliminary.

  • Baseline analysis: VLA-specific spatial and temporal redundancies make token-lightweighting strategies designed for VLMs unsuitable for VLA applications.VLA models are sensitive to relative token positions and object contours, and sequential decision-making creates temporal redundancy.
  • Simulation environments: 1.5× lossless acceleration is achieved on LIBERO, while SimplerEnv reaches up to 2.4× acceleration with a 6% performance gain.LIBERO evaluates stability across diverse tasks, whereas SimplerEnv focuses on generalization.
  • Acceleration contributions: Model scheduling produces larger speedups than token pruning, with intuitive-action proportions increasing from 18% in LIBERO-Spatial to 28% in LIBERO-Long.The corresponding speedups are 1.18× and 1.39×, respectively.
  • Robustness across acceleration ratios: At 1.35× acceleration, SP-VLA reaches 74.90% overall accuracy, while at 1.5× acceleration accuracy drops by only 3%.These results indicate stable performance across a range of acceleration ratios.
  • Acceleration contributions: Different tasks tolerate the two acceleration methods differently: LIBERO-Goal supports 26.70% token pruning, whereas LIBERO-Long supports only 5.67%.Most acceleration in LIBERO-Long comes from intuitive action generation, and the two methods work synergistically.
Loading 2506.12723v3…