VLA-Cache: Efficient Vision-Language-Action Manipulation via Adaptive Token Caching

Siyu Xu; Yunke Wang; Chenghao Xia; Dihao Zhu; Tao Huang; Chang Xu

Source-linked AI summary

VLA-Cache: Efficient Vision-Language-Action Manipulation via Adaptive Token Caching

Siyu Xu, Yunke Wang, Chenghao Xia, Dihao Zhu, Tao Huang, Chang Xu

arXiv:2502.02175v2Feb 4, 2025cs.ROcs.CVcs.LG

TL;DR

VLA models are computationally expensive for real-time robotic control, and existing acceleration methods do not specifically exploit temporal redundancy in visual inputs. VLA-Cache is a training-free method that reuses static visual tokens across frames, recomputes task-relevant tokens, and adapts reuse by decoder layer. Across simulated and real-world evaluations, it achieves about 1.7× speedup while maintaining task performance, although its gains diminish in dynamic environments and its applicability beyond evaluated LLaMA2-based architectures remains open.

Problem
VLA inference is costly for real-time control, while generic acceleration methods lack task-specific designs for the temporal redundancy of robotic visual streams.
Method
VLA-Cache selectively reuses static visual-token KV representations, filters task-relevant tokens using decoder attention, and adapts reuse ratios across layers using attention concentration.
Results
1.7× speedup is achieved across three VLA models and two simulation environments while maintaining performance, with practical real-time speedup on a Kinova Jaco2 robot arm.
Takeaways & Limitations
The method provides a training-free, plug-and-play acceleration strategy that preserves action performance while exploiting temporal redundancy across frames.
Takeaways & Limitations
Acceleration gains decrease in substantially dynamic environments, and applicability to VLA systems with different backbones or more complex designs remains open.

Abstract

from arXiv · show

Vision-Language-Action (VLA) models have demonstrated strong multi-modal reasoning capabilities, enabling direct action generation from visual perception and language instructions in an end-to-end manner. However, their substantial computational cost poses a challenge for real-time robotic control, where rapid decision-making is essential. This paper introduces VLA-Cache, a training-free inference acceleration method that reduces computational overhead by adaptively caching and reusing static visual tokens across frames. Exploiting the temporal continuity in robotic manipulation, VLA-Cache identifies minimally changed tokens between adjacent frames and reuses their cached key-value representations, thereby circumventing redundant computations. Additionally, to maintain action precision, VLA-Cache selectively re-computes task-relevant tokens that are environmentally sensitive, ensuring the fidelity of critical visual information. To further optimize efficiency, we introduce a layer adaptive token reusing strategy that dynamically adjusts the reuse ratio based on attention concentration across decoder layers, prioritizing critical tokens for recomputation. Extensive experiments on two simulation platforms (LIBERO and SIMPLER) and a real-world robotic system demonstrate that VLA-Cache achieves up to 1.7x speedup in CUDA latency and a 15% increase in control frequency, with negligible loss on task success rate. The code and videos can be found at our project page: https://vla-cache.github.io.

1 Introduction

VLA models offer end-to-end multimodal robotic control but incur high inference costs, while generic acceleration methods require modifications or retraining and lack task-specific design. VLA-Cache exploits temporal redundancy in visual inputs to accelerate inference with minor task-success degradation.

VLA models integrate vision and language to generate robotic actions end to end, supporting adaptability and generalization in robotic control.
Generic lightweighting, quantization, and early-exit methods require architectural modification or retraining and may not balance inference speed with action accuracy.
Static visual regions recur across adjacent observations, creating redundant computation because background tokens are repeatedly processed despite limited utility for action decisions.
Over 1.7× acceleration is achieved across LIBERO, SIMPLER, and three VLA models, with only minor task-success drops and real-world speedup on a Kinova Jaco2 arm.

2 Related Work

Related work spans VLA architectures for end-to-end visuomotor control and acceleration methods developed for vision-language models. Existing token-level methods mainly reduce redundancy within individual frames, whereas robotic control also contains temporal structure.

VLA models extend vision-language models with action outputs for end-to-end visuomotor control and are commonly fine-tuned on robot data.
Vision-language acceleration methods include quantization, pruning, and token-level techniques, while VLA efficiency work also uses architectural modification and quantization-aware training.

3 Methodology

VLA-Cache reuses visual information across adjacent frames while protecting task-relevant regions from stale reuse. It selects visually stable tokens, filters them with decoder attention, and adapts reuse across layers according to attention concentration.

3.1 KV Cache for VLA Token Reusing: Temporal KV caching reuses visual token representations across adjacent frames instead of recomputing overlapping visual content from scratch.
3.4 Layer Adaptive Token Reusing: VLA-Cache combines dynamic token selection with adaptive per-layer caching to reduce redundant computation without model modification or retraining.
3.2 Temporal Redundancy in Robotic Perception: Static token selection compares corresponding image patches across frames, retaining patches whose similarity exceeds τ and refining them with a Top-k filter.
3.3 Retaining Task-Relevant Information: The method preserves task fidelity by recomputing tokens that are visually static but semantically important, using decoder cross-attention to identify task-relevant regions.
3.3 Retaining Task-Relevant Information: 82.6% success rate is recovered after filtering task-relevant tokens from the reuse set, compared with 74.2% for naive static-token reuse, while retaining FLOPs and latency gains.
3.4 Layer Adaptive Token Reusing: Layer-adaptive reuse estimates attention concentration through entropy reduction and permits higher reuse fractions in layers with larger cumulative entropy reductions.

4 Implementations

VLA-Cache accelerates VLA inference by reusing cached visual-token representations across temporally adjacent frames while recomputing dynamic or task-relevant tokens. Its theoretical analysis accounts for token processing costs and the overhead of token selection, attention filtering, and layer-adaptive reuse.

Temporal KV caching: VLA-Cache reuses previously computed visual-token key-value representations for tokens that remain unchanged across frames.Dynamic or task-relevant tokens are freshly computed, preserving critical visual information while avoiding redundant computation.
Temporal KV caching: Unlike single-frame pruning or token merging, VLA-Cache exploits temporal redundancy across frames in closed-loop robotic control.This design is intended to align token reuse with high-frequency robotic action prediction.
Theoretical complexity: Static token selection costs approximately O(H^2), task-relevance filtering costs O(L_tL_vD), and entropy-based layer adaptation adds O(L^2D).The added costs remain significantly lower than the baseline per-layer computation cost.
Theoretical complexity: The theoretical analysis separates standard per-layer Transformer computation from the reduced-cost computation after token reuse.The baseline and reduced expressions are presented using terms involving token count, hidden dimension, and intermediate dimension.
Theoretical complexity: VLA-Cache reduces the effective per-layer token count to L_r = α × P_final, yielding theoretical FLOP savings.The supplied analysis introduces the reduced token count and associated savings expressions without providing their full derivations.

5 Experiment

Experiments across LIBERO, SIMPLER, and real-robot tasks show that VLA-Cache reduces computation and latency while largely preserving task performance. Its benefits extend across VLA models, viewpoints, reuse settings, and dynamic backgrounds.

VLA-Cache reduces FLOPs by 27.31% and improves latency by 1.63× over standard OpenVLA on LIBERO, with only a 0.3% success-rate drop.
Nearly 14 Hz higher control frequency with OpenVLA-OFT demonstrates additive gains for a faster action-chunking architecture.
Aggressive token reduction harms success, whereas VLA-Cache remains stable at moderate reuse rates such as 100 tokens.
Figure 4 shows reuse across LIBERO, dynamic-background real-world scenes, and OpenVLA-OFT’s main and wrist-camera views.
VLA-Cache achieves roughly 20% fewer FLOPs and a 1.37× reduction in inference latency than CogACT in SIMPLER, with comparable success rates.
Across four real-robot tasks, VLA-Cache improves average success rate by 2.4% and exceeds the baseline on three tasks.
42% fewer FLOPs and 35% lower latency are achieved under dynamic background motion while maintaining the same success rate as the baseline.

6 Conclusion

VLA-Cache is a training-free acceleration method that reuses static visual tokens while filtering task-relevant tokens. Experiments across three VLA models, two simulation environments, and a Kinova Jaco2 show 1.7× speedup while maintaining performance.

VLA-Cache selectively reuses static visual tokens and filters task-relevant ones to reduce redundant computation without sacrificing accuracy.
A layer-adaptive reuse strategy adjusts token reuse according to attention concentration to improve success rates.
1.7× speedup is demonstrated across OpenVLA, CogAct, and OpenVLA-OFT in LIBERO and SIMPLER while maintaining performance.
Real-world deployment on a Kinova Jaco2 robot arm demonstrates practical speedup under real-time control scenarios.

A Limitations

VLA-Cache has reduced acceleration gains in highly dynamic environments and has been evaluated only on a limited set of VLA architectures.

Dynamic backgrounds or object motion increase non-reusable tokens, reducing acceleration gains.Dynamic regions require full recomputation.
Experiments focus on OpenVLA, CogAct, and OpenVLA-OFT, all based on LLaMA2 decoders.Applicability to architectures with different backbones or more complex VLA systems remains open.
Real-world deployment requires careful operation and continuous monitoring for safe, interpretable, and reliable behavior.

C Complexity Analysis Details

VLA-Cache reduces inference computation by selecting static visual tokens, removing task-relevant tokens from reuse, and adapting reuse across decoder layers.

Token Selection: Reusable tokens are selected through patch-wise similarity, top-k filtering, and cross-modal attention-based relevance filtering.The final reusable set is the static set excluding task-relevant tokens.
Layer-Adaptive Reuse: Attention distributions vary across decoder layers, motivating layer-adaptive reuse based on attention concentration.Early layers show dispersed attention, intermediate layers fluctuate, and final layers partially rebound.
Complexity Analysis: The added overheads remain modest relative to the full forward pass, enabling efficient token reuse.
Inference Procedure: VLA-Cache compares adjacent-frame visual tokens and reuses cached key-value entries for static, task-irrelevant tokens.Dynamic or task-relevant tokens are recomputed.
Token Reuse Mechanism: Dynamic cache updates recompute changed tokens while reused tokens inherit prior-frame cache values.The implementation retains skipped-token states and updates newly computed key-value entries.
Inference Procedure: The largest computational gain occurs when generating the first action token at each timestep.Subsequent tokens are decoded autoregressively without additional cost.
Experimental Settings: Experiments use OpenVLA with 256 visual tokens and default settings k=100 and τtask=0.5, with τ=0.996 in simulation and τ=0.85 for real-world Jaco2.Simulated evaluations use an NVIDIA RTX 4090 GPU in BF16 precision; CogACT additionally uses 10-step DDIM sampling with CFG 1.5.
Task Definitions: The evaluated LIBERO and SIMPLER suites include manipulation tasks such as object picking, drawer operation, placement, and spatial rearrangement.

E.2 Additional Simulation Results

Additional simulation results show that VLA-Cache can match or exceed baseline success on some LIBERO subtasks, with visualizations documenting task executions.

LIBERO-Spatial Results: VLA-Cache occasionally surpasses the baseline success rate on individual LIBERO-Spatial subtasks.The same pattern is also observed for SparseVLM and FastV.
Visualization Results: Figure 5 presents evaluation examples of OpenVLA with VLA-Cache across LIBERO tasks.
LIBERO-Spatial Results: Table 6 compares success rates across different tasks in the LIBERO-Spatial benchmark.
Real-World Results: Table 7 reports the real-world PickPot task under dynamic background conditions involving human or object motion.

E.3 Additional Ablations and Comparisons

Ablations support attention-based task-relevance filtering and show robust efficiency–accuracy behavior across static-token budgets and relevance thresholds, while architectural scope remains bounded.

Task-Relevance Proxy: Attention-based filtering achieves higher success rate and lower latency than an object-mask alternative on LIBERO-Spatial with OpenVLA-OFT.
Task-Relevance Proxy: Attention scores provide a lightweight, task-adaptive relevance proxy that can capture fine-grained or contextual manipulation signals missed by object masks.
Sensitivity Analysis: VLA-Cache remains robust across a wide range of static-token budgets and relevance thresholds, with k=100 and τ=0.5 providing a strong balance.
Sensitivity Analysis: Table 9 varies the static-token budget k with τ=0.5, while Table 10 varies relevance threshold τ with k=100.
Sensitivity Analysis: Efficiency improves monotonically with larger k and τ while success rate remains consistently high.
Architectural Applicability: VLA-Cache is not directly applicable to standalone diffusion policies without a vision-language backbone.It remains compatible with hybrid VLM and diffusion-policy architectures such as CogACT.

E.4 Real Robot Experiment

The real-robot evaluation uses a Kinova Jaco arm with camera-based observations and four single-instruction manipulation tasks. VLA-Cache is compared with OpenVLA across 100 randomized, single-attempt trials, with outcomes recorded using binary success criteria.

Robot Setup: The setup uses a 6-DoF Kinova Jaco arm rigidly fixed to a frame and a Sony AX53 camera facing the operating table.The camera transmits video in real time for robot evaluation.
Task Definitions and Success Criteria: The real-world benchmark defines four tasks: picking up an orange pot, placing a blue cube in a box, putting a sausage in a blue pan, and wiping a table.Each task specifies a terminal success condition, and evaluation assigns only success or failure without partial credit.
Trial Protocol and Randomization: The evaluation comprises 100 trials per method, with 20 trials for PickPot and PutSausage and 30 trials for PlaceCube and WipeTable.Initial robot configurations and object placements are randomized; trials use a fixed horizon with no intervention or reset.
Results: Table 11 reports per-task successes, failures, and success rates, while Figure 7 shows evaluation examples comparing OpenVLA with VLA-Cache on the Kinova Jaco arm.Average success is computed across all 100 trials.