PaperlayerView exact revision on arXiv

Source-linked AI summary

VLA-ACL: Action-Consistent Visual Token Pruning for Efficient Vision-Language-Action Models

Owen Du, Yang Yue, Jie Zhang, Jiaqi Pi, Chi Bene Chen, Gao Huang

arXiv:2610.08133v1cs.ROcs.CV

TL;DR

VLA models process long visual-token sequences at each control step, making computation costly, while existing pruning relies on indirect heuristics or costly VLA fine-tuning. VLA-ACL learns action-consistent token selection with the VLA frozen, retaining competitive performance while pruning visual tokens across LIBERO and real-world tasks.

  • Problem

    VLA inference processes long, visual-token-heavy sequences at every control step, while existing selectors use indirect heuristics or require costly base-model fine-tuning.

    Source: §I · p. 1

  • Method

    VLA-ACL trains a lightweight visual-token scorer to match pruned-context actions to full-context teacher actions, using ground-truth actions as auxiliary supervision while keeping the VLA frozen.

    Source: §I · p. 1

  • Results

    Across LIBERO and real-world tasks, VLA-ACL maintains competitive success rates while pruning up to 87.5% of visual tokens.

    Source: §I · p. 2, §V · p. 7

  • Takeaways & Limitations

    The results support action consistency as a learning signal for visual-token selection while retaining the information needed for control.

    Source: §V · p. 7

Abstract

from arXiv · show

Vision-Language-Action (VLA) models achieve strong robotic manipulation performance but incur high computational costs from processing long token sequences at every control step, limiting real-time deployment. Visual token pruning offers a direct solution, as visual patches dominate the input sequence and contain considerable redundancy. Existing approaches, however, either rely on indirect training-free heuristics, such as attention scores and motion thresholds, or require costly fine-tuning of the base VLA model. We introduce VLA-ACL (Action Consistency Learning), which learns a lightweight visual token pruning policy through action-level supervision while keeping the base VLA model entirely frozen. The training objective encourages actions produced from pruned visual contexts to remain consistent with the full-context teacher, with ground-truth actions as auxiliary supervision. This directly ties token selection to its effect on the downstream control output. Experiments on LIBERO and real-world manipulation tasks show that VLA-ACL prunes up to 87.5% of visual tokens while retaining competitive performance, reduces computation by up to 75%, and achieves a 1.5x inference speedup. These results establish a stronger performance-efficiency trade-off than existing frozen-VLA pruning methods and demonstrate the value of action-level supervision for visual token selection. Code is available at https://github.com/du-owen/VLA-ACL.

I. INTRODUCTION

VLA models process long, visual-token-heavy sequences at each control step, creating computational costs that limit real-time deployment. VLA-ACL learns token selection from action consistency while keeping the base VLA frozen.

I. INTRODUCTION

  • VLA policies process long multimodal sequences dominated by visual tokens at every control step, and this computation and latency constrain real-time response.Visual redundancy motivates pruning or caching, but effective control still depends on retaining useful visual information.Source: §I · p. 1
  • Training-free selectors use indirect signals, while training-based approaches fine-tune the VLA; VLA-ACL instead learns from action impact with the base model frozen and without accessing or modifying its internal states.The objective matches pruned-context actions to full-context teacher actions and adds ground-truth action supervision; training uses a lightweight scorer and takes a few hours on one GPU.Source: §I · pp. 1–2
  • Up to 87.5% of visual tokens and 75% of FLOPs are removed while maintaining competitive success rates on LIBERO and real-world tasks, with a stronger trade-off than frozen-VLA pruning baselines.Fig. 1 compares LIBERO-Long success rates against computational budget and reports a better trade-off than frozen-VLA pruning and caching methods.Source: Figure 1 · p. 1, §I · p. 2

II. RELATED WORK

VLA models predict robot actions from visual observations and language, but long visual sequences remain computationally expensive. Prior acceleration methods prune tokens with heuristics or jointly train the pruner and VLA; VLA-ACL learns action-linked selection while freezing the VLA.

A. Vision-Language-Action Models

  • VLA models use pretrained vision-language representations to predict robot actions, but processing long visual token sequences remains computationally expensive for real-world tasks.OpenVLA uses a large language backbone, while OpenVLA-OFT uses parallel decoding of action chunks.Source: A. Vision-Language-Action Models · p. 2

B. Efficient VLA Inference

  • Training-free pruning scores tokens with attention, diversity, saliency, or temporal signals, but layer- and time-varying heuristics can require tuned thresholds and conflict with fused-attention or graph-compiled inference.Deeper-layer attention may be unavailable until a previous forward pass, degrading performance at low retention rates.Source: B. Efficient VLA Inference · p. 2
  • Training-based methods learn pruning jointly with the VLA, but LightVLA and Grid-S require backbone retraining that can take multiple days even on eight frontier GPUs.These approaches use a Gumbel-softmax selector or differentiable grid sampling, respectively.Source: B. Efficient VLA Inference · p. 2
  • VLA-ACL combines learned pruning with a frozen VLA: one per-view token budget controls selection before the language backbone, and action consistency replaces intermediate token-importance proxies.Its fixed pruning mechanism avoids temporal controllers and remains compatible with existing inference-acceleration techniques.Source: B. Efficient VLA Inference · p. 2

III. METHOD

VLA-ACL trains a lightweight patch scorer to preserve a frozen VLA’s full-context action predictions after visual-token pruning, with ground-truth actions as auxiliary supervision. At inference, it keeps the highest-scoring tokens before they enter the language backbone.

III. METHOD

  • A lightweight patch scorer is trained with action-level supervision so pruned-context predictions match full-context teacher predictions, while the VLA remains frozen.Soft Top-K gating enables gradient propagation during training; the overview depicts separate full-context teacher and soft-pruned action passes.Source: §III · p. 2, Figure 2 · p. 3

A. Token-Pruning: Inference

  • At inference, the scorer retains the K highest-scoring tokens per view before the frozen language backbone predicts actions; discarded tokens are excluded, and retained tokens keep their original RoPE positions.The selected embeddings are combined with proprioceptive inputs and language instructions for action prediction.Source: A. Token-Pruning: Inference · p. 3, §III · p. 3

B. Action Consistency Learning

VLA-ACL trains a patch scorer to preserve the frozen VLA’s action predictions after visual token pruning, using full-context teacher actions and ground-truth actions as supervision. Differentiable soft selection and gating let action-level gradients update the scorer while a fixed token budget controls pruning.

B. Action Consistency Learning

  • The action-consistency loss encourages pruned-context actions to match full-context teacher actions and uses demonstrated actions as auxiliary supervision.Gradients pass through the frozen VLA and differentiable gate to update only the patch scorer; the ground-truth term can anchor predictions when the teacher is inaccurate.Source: B. Action Consistency Learning · pp. 3–4
  • Soft Top-K maps patch logits to continuous scores with total selection mass K, providing a differentiable relaxation of hard Top-K selection.As α approaches zero, the scores approach binary Top-K; cosine annealing progressively sharpens selection while keeping K fixed.Source: B. Action Consistency Learning · p. 3
  • The differentiable gate combines original visual embeddings with black-image embeddings according to each patch’s selection score.A score near 1 preserves the original patch, while a score approaching 0 replaces it with the black-patch embedding; this is an in-distribution replacement for the frozen backbone.Source: B. Action Consistency Learning · pp. 3–4, Figure 3 · p. 4
  • The training objective uses a fixed token budget K, directly setting the inference pruning ratio without an additional sparsity-regularization hyperparameter.The soft-pruned context acts as a differentiable surrogate for hard-pruned embeddings, allowing action-level gradients to reach the scorer.Source: B. Action Consistency Learning · p. 4

IV. EXPERIMENTS

The experiments evaluate VLA-ACL on LIBERO and real-world manipulation, compare it with frozen-VLA pruning and caching methods built on OpenVLA-OFT, and test transfer to π0.5. All experiments and measurements use a single NVIDIA A100 GPU.

IV. EXPERIMENTS

  • Experiments cover LIBERO, real-world manipulation, frozen-VLA pruning and caching baselines built on OpenVLA-OFT, and an additional application to π0.5.All experiments and measurements are conducted using a single NVIDIA A100 GPU.Source: §IV · p. 4

A. Training

Training uses a five-layer bidirectional Transformer patch scorer over vision embeddings and evaluates two fixed token budgets spanning aggressive and moderate pruning. Uniform hyperparameters are used across LIBERO and real-world tasks and both VLA backbones.

A. Training

  • A five-layer bidirectional Transformer scores vision embeddings separately for each camera view.LIBERO uses two views with 256 tokens each; two real-world tasks use two wrist-camera views and one static-camera view.Source: A. Training · p. 4
  • K = 32 and K = 64 correspond to visual-token pruning rates of 87.5% and 75%, respectively.These budgets span aggressive and moderate pruning regimes, trading off performance against efficiency.Source: A. Training · p. 4
  • Across LIBERO, real-world tasks, and both backbones, λ = 0.5 and µ = 0.5 equally weight teacher consistency and ground-truth supervision.The training setup anneals α from 2.0 to 0.1 across 5000 steps to close the gap between soft and hard pruning regimes.Source: A. Training · p. 4

B. Simulation Experiments

On LIBERO, VLA-ACL reduces computation while retaining success rates comparable to full-context inference, and it sustains aggressive pruning at every control step. The experiments also report transfer to π0.5 and compare pruning and caching policies.

B. Simulation Experiments

  • At K = 32, VLA-ACL reduces FLOPs by 75% and achieves a 1.5× speed-up relative to OpenVLA-OFT, with a −1.4% change in average success rate.At K = 64, it reduces FLOPs by 65%, achieves a 1.4× speedup, and exceeds the baseline’s average by +0.2%.Source: B. Simulation Experiments · p. 4
  • VLA-ACL maintains strong performance under aggressive pruning and achieves the lowest latency; VLA-ADP and VLA-Cache rely on full-context forwards during inference.The figure caption describes VLA-ACL as sustaining a more aggressive pruning schedule at every control step while retaining similar performance across LIBERO suites.Source: Figure 4 · p. 5, B. Simulation Experiments · p. 5
  • LIBERO-Long is where the K = 32 and K = 64 configurations differ most, with long-horizon trajectories more sensitive to reduced visual context at K = 32.VLA-Cache runs slower than base OpenVLA-OFT across the tested GPU configurations despite reducing FLOPs.Source: B. Simulation Experiments · p. 5
  • On π0.5, K = 64 reduces TFLOPs by 67% and pre-fill latency by 2.2× at 96.1% average success (−0.8 versus the unpruned model).It outperforms SAFE-Pruner by +0.6 in success rate with less than half its TFLOPs while retaining torch.compile.Source: B. Simulation Experiments · p. 5
  • VLA-ACL prunes at a constant rate every control step, including the first frame, without temporal controllers or a fallback to full context.Its policy indicates which patches contribute most to the frozen VLA’s action, unlike training-free methods’ indirect heuristics such as attention values.Source: B. Simulation Experiments · p. 5

C. Real-world experiments

The real-world evaluation compares VLA-ACL at K = 64 with OpenVLA-OFT across four manipulation tasks. It reports comparable aggregate performance alongside substantially lower computation and faster inference.

Experiment setup

  • The four tasks cover instruction following and precise perception, spatial understanding, goal-directed behavior, and multi-subtask long-horizon execution.Task 4 combines instruction following with long-horizon subtasks; tasks 2 and 3 include multiple subtasks.Source: C. Real-world experiments · p. 5

Experiment results

  • More than 50% fewer FLOPs and around 1.5× speed-up accompany performance comparable to full-context baselines across four real-world tasks.VLA-ACL trails OpenVLA-OFT by one failed trial on tasks 1 and 3 and matches or exceeds it on the other two.Source: Table II · p. 6, C. Real-world experiments · p. 6

Experiment setup

D. Ablation study

The ablations assess the action-consistency objective, token-selection strategy, and gating mechanism. They report benefits from auxiliary ground-truth supervision, action-consistent selection, and black-patch gating.

Action consistency objective

  • The combined teacher-consistency and ground-truth objective outperforms teacher-only and ground-truth-only variants across K configurations.Teacher and ground-truth actions usually coincide closely, but episodic gaps can spike; Fig. 6 tracks their ℓ1 distances during training.Source: Table III · p. 6, D. Ablation study · p. 6, Figure 6 · p. 6

Selection method

  • At equal token budgets, VLA-ACL outperforms the selection baselines in both action loss and success rate.Original-index RoPE improves on plain attention, which falls to approximately 3% success rate on LIBERO-Long at low token retention.Source: Table IV · p. 7, D. Ablation study · p. 7

Gating mode

  • At α = 0.1, black-patch gating yields a smaller soft-to-hard action gap than noise gating for K ≥ 32, while multiplication has persistently high action loss.Fig. 7 compares the soft-pruned actions with hard Top-K actions across K values.Source: Figure 7 · p. 7, D. Ablation study · p. 7

V. CONCLUSION

VLA-ACL learns visual-token pruning for frozen VLAs by training a lightweight scorer to preserve full-context actions, with ground-truth demonstrations as auxiliary supervision. It reports competitive success rates while reducing computation across LIBERO and real-world tasks.

Conclusion

  • A lightweight patch scorer is trained to make pruned-context actions match full-context actions from the same frozen VLA, with ground-truth demonstrations as auxiliary supervision.The framework is evaluated on LIBERO across two VLA backbones and on real-world tasks.Source: §V · p. 7
  • Up to 87.5% of visual tokens are pruned and FLOPs are reduced by up to 75% while maintaining competitive success rates.Training costs only a few GPU hours.Source: §V · p. 7
  • VLA-ACL extends the FLOPs-and-latency Pareto frontier among frozen-VLA visual-token pruning methods.Source: §V · p. 7

APPENDIX

The appendix describes implementation across OpenVLA-OFT and π0.5, reports π0.5 results, and outlines the real-world platform and data collection. Its π0.5 comparison shows a strong success-rate and compute trade-off, with baseline latency unavailable for direct measurement.

A. Implementation Details

  • VLA-ACL uses a jointly trained patch scorer across all four π0.5 suites without per-suite specialization or backbone-specific hyperparameter tuning.For π0.5 flow matching, the scorer is trained against pruned-context and full-context teacher velocities at the same sampled noise level.Source: A. Implementation Details · p. 10
  • Baseline latency for π0.5 pruning methods could not be measured because their implementations were unavailable; the expected latency advantage is not directly verified.Training and evaluation were conducted on a single NVIDIA A100 GPU.Source: A. Implementation Details · p. 10, B. More results · p. 10

B. More results

  • On π0.5, VLA-ACL uses 67% fewer TFLOPs and achieves 2.2× speedup in pre-fill latency relative to the base model.Against SAFE-Pruner, it uses 0.701 vs. 1.482 TFLOPs and achieves 96.1% vs. 95.5% average success; it trails unpruned π0.5 by 0.8 points.Source: B. More results · p. 10

C. Real-world experiment setup

  • The real-world platform is a bimanual AgileX PiPER-X with a static camera and two wrist cameras, and each of four tasks uses 50–100 expert demonstrations.The trajectories are used to fine-tune OpenVLA-OFT with imitation learning.Source: Figure 8 · p. 10, C. Real-world experiment setup · p. 10

Related papers

Loading 2610.08133v1…