PaperlayerView exact revision on arXiv

Source-linked AI summary

FastVLA: Task-Conditioned and Temporally Adaptive Visual Token Compression for Efficient Vision-Language-Action Models

Jiuyi Xu, Chengyuan Qian, Zhengzhong Tu, Yang Sui, Yangming Shi

arXiv:2610.22146v1cs.RO

TL;DR

Closed-loop VLA inference is costly because high-resolution, often multi-view observations produce long visual-token sequences, while task-critical information varies across control steps. FastVLA uses task-to-vision attention and an inverse-Simpson measure with sliding-window calibration to adapt compression per step. It reports lower compute and latency with comparable task success in simulation and on a physical robot.

  • Problem

    VLA models process hundreds of visual tokens at each control step, while decision-critical tokens depend on the task and the needed token budget varies across steps.

    Source: §1 · p. 1

  • Method

    FastVLA scores token importance with task-to-vision attention and sets a per-step compression rate from inverse-Simpson uniformity under online sliding-window calibration.

    Source: §1 · p. 2, §3.3 · pp. 4–5

  • Results

    Across simulation benchmarks, FastVLA reduces LLM-backbone FLOPs by 35–59% and action latency by 9–32% while maintaining task success; on a physical Franka robot, latency falls 20% at comparable success.

    Source: §5 · pp. 8–9

  • Takeaways & Limitations

    FastVLA achieves a comparable or better efficiency-performance trade-off than existing training-free compression, pruning, and reuse baselines while remaining training-free.

    Source: §1 · p. 2, §5 · p. 9

Abstract

from arXiv · show

Vision-Language-Action (VLA) models present a promising paradigm for general-purpose robot manipulation, but closed-loop deployment is often limited by inference efficiency: at each timestep, high-resolution, often multi-view images are encoded into hundreds of visual tokens that dominate transformer computation and latency. Token compression via attention is a natural solution but is challenging for VLA control because (1) decision-critical tokens are determined by the task, not generic visual saliency, and (2) the token budget varies substantially across steps, so static metrics and budgets waste computation or discard control-critical tokens. We introduce FastVLA, a training-free, task-aware, and temporally adaptive visual token compression framework. At each step, FastVLA scores token importance via task-to-vision attention and adaptively sets the compression rate from the importance-distribution uniformity, measured by an inverse-Simpson metric with online sliding-window calibration. We evaluate FastVLA on OpenVLA, OpenVLA-OFT, and CogACT across LIBERO and SIMPLER, and on a physical Franka Research 3 robot. In simulation, it reduces LLM-backbone FLOPs by 35-59\% and action latency by 9-32\% (a 1.09$\times$-1.46$\times$ speedup) while maintaining task success, achieving a comparable or better efficiency-performance trade-off than existing plug-and-play token compression/pruning/reuse baselines. On the real robot, it cuts end-to-end latency by 20\% (1.25$\times$ speedup) at comparable success. Code is available at https://github.com/jiuyixu25/FastVLA.

1. Introduction

FastVLA targets VLA inference costs from long visual-token sequences and the mismatch between fixed compression rates and task-dependent, step-varying information needs. It uses task-conditioned attention and an adaptive per-step budget; evaluations report lower compute and latency with comparable task success.

1. Introduction

  • High-resolution, often multi-view observations become hundreds of visual tokens that dominate VLM-backbone computation and latency during closed-loop action prediction.These visual tokens outweigh the few text tokens in a short task.Source: §1 · p. 1
  • Generic VLM compression is often suboptimal for VLA control because it targets global semantic content in single-shot understanding rather than sequential decisions shaped by actions.FastVLA uses task-conditioned importance rather than generic visual saliency.Source: §1 · pp. 1–2
  • Task-critical visual information varies across steps, so a fixed compression rate can waste computation on concentrated evidence or discard needed broader context.For example, localization may use broad scene context, while grasping or contact can depend on a small region.Source: §1 · p. 2
  • FastVLA sets a per-step compression rate from task-to-vision attention and the importance distribution’s inverse-Simpson uniformity, calibrated with an online sliding window.It prunes more aggressively when importance is concentrated relative to recent steps and more conservatively when broader context is needed.Source: §1 · p. 2
  • 35–59% fewer LLM-backbone FLOPs and 9–32% lower action latency in simulation, with task success maintained; real-robot latency falls 20% at comparable success.Evaluations span OpenVLA, OpenVLA-OFT, and CogACT across LIBERO and SIMPLER, plus a physical Franka Research 3 robot.Source: §1 · p. 2

2. Related Work

VLA models use large pretrained representations for robot manipulation, but long visual-token sequences make closed-loop inference costly. Prior compression approaches prune, merge, or reuse tokens, while FastVLA combines task-conditioned importance with a compression rate that adapts at each step.

2.1. Vision-Language-Action Models for Robot Manipulation

  • VLA models extend language-conditioned robot manipulation with large pretrained representations, but high-resolution observations create long visual-token sequences that incur substantial computation and latency at every control step.The cited examples include SayCan [1], RT-1 [5], RT-2, PaLM-E [9], Octo [23], OpenVLA [14], OpenVLA-OFT [15], CogACT [16], and π0 [3] / π0.5 [10].Source: §2.1 · p. 2

2.2. Visual Token Compression for Efficient VLA Inference

  • VLM compressors prune or merge tokens using similarity, attention, or diversity, but their static policies target single-image inference and can degrade VLA control performance.Examples include ToMe [4], FastV, SparseVLM [29], and DivPrune [2].Source: §2.2 · p. 2
  • Existing approaches also reuse or cache tokens, add task-, phase-, or motion-aware importance cues, or use hybrid and learned schemes.Reuse-based methods include VLA-Cache [26] and FlashVLA [21]; importance methods include SpecPrune-VLA [25], ADP [20], and VLA-Pruner [19]; hybrid or learned approaches include EfficientVLA [27] and LightVLA [11].Source: §2.2 · pp. 2–3
  • FastVLA scores task-relevant patches, measures their distributional uniformity, and calibrates a dynamic compression rate with an online sliding window.The framework retains task-relevant, information-diverse tokens while reducing LLM-backbone computation.Source: Figure 1 · p. 3
  • FastVLA differs by pruning independently at each step without caches, conditioning importance on the language task, and adapting the compression rate from recent task-relevant evidence.The paper notes that caching can add state and management overhead, while LightVLA requires training and is tied to a backbone.Source: §2.2 · p. 3

3. Method

FastVLA is a training-free visual-token pruning method that scores tokens using task-conditioned attention and adapts the retained-token budget to the importance distribution at each action step.

3. Method

  • FastVLA first scores visual-token importance with task-conditioned attention, then adaptively sets how many tokens to retain at each step.The framework is training-free and reduces the visual-token load on the VLA model’s LLM backbone.Source: §3 · p. 3

3.1. Preliminary: Visual Token Compression in VLA Models

  • FastVLA uses hard, task-conditioned top-K pruning, while merging and caching are distinct compression approaches that can be combined with pruning.Pruning discards tokens, merging pools similar tokens, and reuse/caching recomputes only changed regions or states.Source: §3.1 · p. 3

3.2. Task-conditioned visual token importance

  • Task-to-vision scoring retained near-saturated LIBERO success at 10%–50% pruning and nontrivial performance at ≥ 70%, while vision-to-vision and all-to-vision degraded faster and approached failure at 90%.The figure compares average LIBERO success across pruning rates for the three attention-based scoring strategies.Source: §3.2 · p. 4, Figure 2 · p. 4

3.3. Adaptive Step-aware Compression Rate

  • Within a single LIBERO-Long trial, importance shifts between nearly uniform and highly concentrated, so fixed pruning can waste computation or discard decision-critical tokens.Approaching or grasping can depend on a localized region, while target localization or scene verification can require broader context.Source: §3.3 · p. 4, Figure 3 · p. 4
  • Inverse Simpson uniformity emphasizes squared weights and down-weights the long tail, providing a larger dynamic range than normalized Shannon entropy for tracking dominant-token concentration.The measures are compared across action steps in Figure 4.Source: §3.3 · p. 4, Figure 4 · p. 4
  • FastVLA calibrates each step’s uniformity against a sliding window because its range drifts across suites and backbones, then maps the relative position to a bounded pruning rate around target ρ.A more concentrated distribution than the recent past warrants more aggressive pruning; a more uniform one warrants caution, with the rate bounded by [ρ − τ, ρ + τ].Source: §3.3 · p. 5

4. Experiments

Experiments compare FastVLA with acceleration baselines across LIBERO, SIMPLER, and real-robot manipulation, finding lower computation and latency with task success largely preserved. Ablations examine whether adaptive allocation, calibration, and the diversity metric account for the results.

4.1. Experiment Setup

  • Across one RTX 4090, FastVLA was compared with VLM pruning, VLA compression, caching and reuse, and learned pruning using success, FLOPs, latency, control frequency, and CUDA latency.For methods that could not be reproduced, the paper reports results from their original papers for reference.Source: §4.1 · p. 5, §4 · p. 5

4.3. Experiment Results

  • On LIBERO, OpenVLA-OFT FastVLA reduced LLM FLOPs from 3.98 to 1.65 T (−58.5%) and latency from 118.52 to 81.17 ms (1.46×), with success changing from 97.0% to 96.5%.Against LightVLA, FastVLA does not define the better efficiency frontier at equal success, but it is training-free and works across three backbones and a real robot at inference time.Source: Table 1 · p. 6, §4.3 · p. 6

4.2. Evaluation Environments

  • LIBERO covers four task suites, SIMPLER includes four tasks across Visual Matching and Variant Aggregation, and the robot evaluation uses pick and pick-and-place.The robot policy is OpenVLA-OFT fine-tuned with LoRA on teleoperated demonstrations; each task and condition has 50 rollouts alternating baseline and FastVLA.Source: §4.2 · p. 6

4.3. Experiment Results

  • On SIMPLER, FastVLA reduced latency in both settings while average success stayed essentially unchanged in Visual Matching and rose 1.0 points in Variant Aggregation.Variant Aggregation latency fell from 174.94 to 157.06 ms (−10.2%, 1.11×); control frequency rose from 5.72 to 6.37 Hz (+11.4%).Source: Table 2 · p. 7, §4.3 · p. 7
  • On Franka, latency fell 20.0% on pick and 18.5% on pick-and-place, while success remained comparable at 84.0% vs. 80.0% and 68.0% vs. 66.0%.Control frequency increased by 25.0% on pick and 22.7% on pick-and-place.Source: Table 3 · p. 7, §4.3 · p. 7
  • Qualitative rollouts compare full observations with FastVLA-pruned views across a simulated LIBERO-Long task and a real-world pick-and-place task; retained tokens concentrate around the robot arm and target objects.Each strip shows eight uniformly sampled frames from an external third-person camera.Source: Figure 5 · p. 7, §4.3 · p. 8

4.4. Ablation Studies

  • Ablations found adaptive allocation beat a compute-matched fixed policy; min–max and rank calibration were indistinguishable, while inverse-Simpson led Shannon and Gini by 2.3–3.0 and 1.7–2.0 points.Inverse-Simpson’s advantage was not explained by token count; it retained only 0.6–1.8% more tokens, while rank calibration was proposed as a drop-in substitute for noisy observations.Source: §4.4 · p. 8, Table 4 · p. 8, Table 5 · p. 8

5. Conclusion

FastVLA uses inverse-Simpson concentration with sliding-window calibration to adapt visual-token budgets online. It reduces simulation compute and latency while maintaining task success, and improves real-robot speed at comparable success.

5. Conclusion

  • 35-59% lower LLM-backbone FLOPs and 9-32% lower action latency across three backbones and two simulation benchmarks, while maintaining task success.The conclusion reports these results for FastVLA across the simulation benchmarks.Source: §5 · p. 8
  • 20% lower latency and up to 25% higher control frequency on the physical robot, at comparable success and within 0.6 points of the uncompressed baseline on both backbones.The conclusion describes FastVLA as entirely training-free.Source: §5 · p. 9
  • Robot-specific structure such as 3D geometry or proprioception, and combinations with reuse or early-exit strategies, are proposed as directions for further acceleration.Source: §5 · p. 9

Related papers

Loading 2610.22146v1…