PaperlayerView exact revision on arXiv

Source-linked AI summary

TRIAGE: Efficient Vision-Language-Action Inference Needs Both Reuse and Pruning

Sooyoung Kwon, Carlee Joe-Wong, Kyunghan Lee

arXiv:2610.34519v1cs.ROcs.AIcs.CVcs.LG

TL;DR

Embedded VLA inference must process hundreds of visual tokens per control step, while existing accelerators apply pruning or reuse uniformly to skipped tokens. TRIAGE assigns tokens to recomputation, reuse, or pruning using importance and cache validity. It stays within 3 pp of full recomputation while reducing Jetson AGX Orin prefill latency by up to 48%.

  • Problem

    Existing VLA acceleration methods either prune or reuse all visual tokens they do not recompute, although onboard prefill latency grows with the many visual tokens processed at each control step.

    Source: §I · pp. 1–2

  • Method

    TRIAGE recomputes important tokens with invalid caches first, then reuses valid caches and prunes the remaining tokens.

    Source: §I · p. 2, D. TRIAGE · p. 4

  • Results

    TRIAGE stays within 3 pp of full recomputation while reducing Jetson AGX Orin prefill latency by up to 48%.

    Source: §I · p. 2

  • Takeaways & Limitations

    The recomputation budget fixes the number of recomputed tokens and directly sets prefill latency, allowing its selection to target a control-loop latency limit.

    Source: D. TRIAGE · p. 4

    TRIAGE hyperparameters are tuned empirically rather than derived from first principles.

    Source: §VI · p. 7

Abstract

from arXiv · show

Vision-language-action (VLA) models are increasingly used for robot manipulation because they map camera images and a language instruction directly to actions. Robots that cannot be tethered to a workstation, such as mobile manipulators and humanoids, must run them on onboard embedded boards, where the prefill latency of the language backbone grows with the hundreds of visual tokens processed at every control step. To meet real-time limits, training-free accelerators recompute only a subset of the visual tokens and skip the rest, either by pruning them or by reusing their cached key-value (KV) states. The two options save nearly the same computation, yet existing methods apply only one of them to every skipped token. Through a token-level diagnostic, we show that which option keeps the action closer to full recomputation depends on the token's drift, the change in its shallow-layer value since its cache was computed. On both backbones tested, reusing tokens below a drift threshold was no worse than pruning them, whereas beyond it, reuse yields up to 4.4x the action error of pruning; the same diagnostic calibrates this threshold per backbone. Building on this finding, we propose TRIAGE, which deems a cache valid when its drift is below this threshold and it is not too old. TRIAGE recomputes a fixed budget of tokens, prioritizing important tokens with invalid caches, reuses every other token with a valid cache, and prunes the rest; prune-only and reuse-only methods are the two extremes of this partition. Because the budget fixes the number of recomputed tokens, it directly sets the prefill latency. On LIBERO and SimplerEnv, TRIAGE stays within 3 pp of full recomputation in average success rate while cutting prefill latency on a Jetson AGX Orin by up to 48%, and with only 7% of visual tokens recomputed, it beats the strongest baseline by 24.4 pp on LIBERO-Object.

I. INTRODUCTION

VLA inference on embedded boards faces prefill costs from hundreds of visual tokens, while existing accelerators either prune or reuse every skipped token. TRIAGE uses drift and importance to choose per token among recomputation, reuse, and pruning, achieving near-full-recomputation success rates with lower latency.

I. INTRODUCTION

  • On embedded boards, prefill latency grows with input-token count, and visual tokens account for roughly 90% of the sequence.This makes visual-token computation a main contributor to inference cost.Source: §I · p. 1
  • Existing methods either prune or reuse all visual tokens they do not recompute, despite pruning risking lost relevant tokens and reuse retaining stale caches.Pruning is irreversible, while repeatedly reusing stale key-value caches can accumulate errors.Source: §I · pp. 1–2
  • The token-level diagnostic finds reuse no worse than pruning below a drift threshold, but up to 4.4× worse in action error beyond it.The threshold depends on the backbone and can be calibrated by rerunning the diagnostic.Source: §I · p. 2
  • TRIAGE recomputes important tokens with invalid caches first, then reuses valid caches and prunes the remaining tokens.Importance allocates the recomputation budget; cache validity determines the fate of tokens outside it.Source: §I · p. 2
  • TRIAGE stays within 3 pp of full recomputation while reducing Jetson AGX Orin prefill latency by up to 48%.Across two backbones and two benchmarks, it also beats the strongest baseline by 24.4 pp on LIBERO-Object when recomputing 7% of visual tokens.Source: §I · p. 2

II. RELATED WORK

Prior VLA acceleration methods use pruning or reuse, with each visual token outside the recomputed set receiving one fixed treatment. TRIAGE instead partitions tokens among recomputation, reuse, and pruning.

II. RELATED WORK

  • Visual tokens dominate VLA backbone input; reducing their cost applies across decoding styles, and can combine with orthogonal accelerations such as quantization or layer skipping.OpenVLA uses 256 visual tokens per step and OpenVLA-OFT uses 512.Source: §II · p. 2
  • VLA pruning methods use temporal and action cues, but discard every visual token they do not keep, including tokens with accurate caches.Examples include smoothed action-decode attention [10], deep-layer attention, motion-adaptive ratios, and forecast saliency [13].Source: §II · p. 2
  • VLA reuse methods vary their policies, but reuse every visual token they do not recompute, even when its cache is stale.Examples include static-patch caching, learned policies [15], action-confidence gating [16], and reuse-head selection [17].Source: §II · p. 2
  • TRIAGE chooses per token among recomputation, reuse, and pruning, making prune-only and reuse-only methods special cases of its partition.The method reuses accurate caches that pruning would discard and prunes stale caches that reuse would keep.Source: §II · p. 2

III. WHEN IS A CACHED TOKEN WORTH KEEPING?

The section asks whether reuse or pruning better preserves the action for a token outside the recomputed set. It identifies shallow-layer drift as the signal for that decision.

A. Problem Formulation

  • On LIBERO-Spatial, reuse is never worse below δ = 0.3 but clearly worse above 0.5; drift grows with cache age.The figure plots reuse-minus-prune action error against token drift and shows pooled and cache-age-specific results.Source: Figure 2 · p. 3
  • Prune-only and reuse-only methods are the two extremes of the recompute, reuse, and prune partition; the section focuses on choosing reuse or pruning given the recomputed set.Drift measures how far a cached state has moved from what the model would compute for the current frame.Source: A. Problem Formulation · pp. 2–3
  • Drift is the relative change in a token’s shallow-layer attention value since its last recomputation.It is measured at a shallow layer, and once δ is calibrated, other shallow layers can be used.Source: A. Problem Formulation · p. 3

B. Token-level Diagnostic

The diagnostic compares reuse and pruning against full recomputation while varying token drift and cache age. Reuse is no worse below a drift threshold, but can be worse above it, with the threshold dependent on the backbone.

B. Token-level Diagnostic

  • Probes vary cache age and token groups spanning three drift levels and three importance levels, comparing reuse or pruning for 16 tokens while recomputing all others.The study uses LIBERO-Spatial full-recomputation actions as the reference and collects 14,442 OpenVLA probes and 9,925 OFT probes.Source: B. Token-level Diagnostic · p. 3
  • Below drift 0.3, reuse is no worse than pruning on either backbone, with pooled confidence intervals entirely below zero.No individual cache age rises above zero in this drift range.Source: B. Token-level Diagnostic · p. 3
  • Cache age affects how much reuse helps, but drift accounts for the threshold: reuse remains better through ages 20 or 40 on OpenVLA, while its advantage vanishes from age 10 on OFT.For simplicity, the paper uses δ = 0.3 for both backbones; success rates are statistically unchanged at δ = 0.2 or 0.4.Source: B. Token-level Diagnostic · pp. 3–4
  • Above drift 0.5, reuse yields 1.6× (OpenVLA) and 4.4× (OFT) the action error of pruning.At high drift, older caches are clearly worse than pruning, while caches from one query earlier stay close to zero.Source: B. Token-level Diagnostic · p. 3
  • The reuse-versus-pruning threshold depends on the backbone: between drifts 0.3 and 0.5, reuse remains harmless on OpenVLA but is slightly worse than pruning on OFT.The diagnostic can calibrate the threshold per backbone using full-recomputation rollouts and no training.Source: B. Token-level Diagnostic · p. 3

IV. TRIAGE: RECOMPUTE, REUSE, OR PRUNE

TRIAGE uses shallow-layer attention and drift to assign visual tokens to recomputation, reuse, or pruning. Importance ranks tokens for a fixed recomputation budget, while cache validity governs reuse versus pruning.

A. Overview

  • Shallow layers provide per-token attention and drift signals; TRIAGE uses them to assign tokens to recomputation, reuse, or pruning, with deep layers computing only recomputed tokens and reading cached KVs for reused tokens.Drift and cache age determine validity; attention measures importance.Source: A. Overview · p. 4

B. Importance

  • The importance score combines shallow attention with a forecast of deep-layer saliency, since shallow attention can miss tokens that become important deeper in the model.The saliency forecast uses SAFE-Pruner’s deep-layer signal [13].Source: B. Importance · p. 4
  • TRIAGE transfers the first-query saliency memory across frames through patch correspondence and does not refresh it, while current shallow attention remains part of each importance score.The fixed memory supports the design goal of avoiding full-token queries at any step and bounding every step’s latency.Source: B. Importance · p. 4
  • The important set contains the top B + κ tokens by importance, with κ = 70, so valid-cache tokens can be reused rather than recomputed.The two importance signals are fused with equal weights, using λ = 1 as in SAFE-Pruner.Source: B. Importance · p. 4

C. Validity

Cache validity combines drift, cache age, and cache existence; TRIAGE then uses a fixed recomputation budget to prioritize important tokens with invalid caches and decide whether the remaining tokens are reused or pruned.

C. Validity

  • A cache is valid only when the token is cached, its drift is below δ, and its age is at most T.The drift threshold is δ = 0.3 and the age limit is T = 10; the threshold may differ across backbones or tasks.Source: C. Validity · p. 4

D. TRIAGE

  • The recomputation budget B sets prefill latency by fixing the number of recomputed tokens, and the budget is allocated first to important tokens with invalid caches.Important tokens that cannot be reused are prioritized by importance; remaining slots may refresh important tokens with valid caches.Source: D. TRIAGE · p. 4
  • Maintenance refreshes tokens outside the importance set, while the other budget priorities recompute important tokens whose caches cannot be trusted and optionally refresh important valid-cache tokens.Without maintenance, background reuse vanishes after about T queries; tokens left uncached initially are computed only after entering the importance set.Source: D. TRIAGE · pp. 4–5
  • Tokens outside the recompute set are reused when their caches are valid and pruned otherwise, regardless of importance.This rule assigns every token not recomputed to reuse or pruning based on validity alone.Source: D. TRIAGE · p. 4

A. Setup

The evaluation uses LIBERO and SimplerEnv with two VLA backbones, training-free baselines, and latency measurements on embedded and reference hardware. TRIAGE remains close to full recomputation while occupying the success-rate/prefill-latency Pareto frontier across the reported settings.

A. Setup

  • The evaluation covers four LIBERO suites and selected SimplerEnv Google Robot tasks, using OpenVLA-7B and OpenVLA-OFT backbones.The baseline comparison includes full recomputation, VLA-Cache, VLA-Pruner, SAFE-Pruner, and SpecPrune-VLA.Source: A. Setup · p. 5
  • Latency evaluation reports prefill and end-to-end times, including p99, on a Jetson AGX Orin and an RTX 3090.The reported metrics also include success rate and analytic prefill FLOPs.Source: A. Setup · p. 5

B. Both Pruning and Reuse Are Needed

  • Across budgets, drop-all degrades steadily as the budget shrinks, reaching 22.4% at B = 18; keep-all remains competitive at large budgets but collapses below B = 46, reaching 0.0%.Source: B. Both Pruning and Reuse Are Needed · p. 5
  • At B = 18 on LIBERO-Object with OpenVLA, TRIAGE reaches 75.8%, versus at most 51.4% for every other method.It leads VLA-Pruner by 24.4 pp at B = 18.Source: B. Both Pruning and Reuse Are Needed · p. 5
  • TRIAGE lies on the success-rate/prefill-latency Pareto frontier across every backbone, benchmark, and device setting.The figure plots Jetson AGX Orin and RTX 3090 results averaged over LIBERO suites or SimplerEnv settings.Source: Figure 5 · p. 6
  • TRIAGE stays within 3 pp of full recomputation in average success rate and has the lowest Jetson p99 latency on LIBERO.Table I’s caption identifies these results; measurements use bf16 and replay the same observation stream.Source: Table I · p. 6, B. Both Pruning and Reuse Are Needed · p. 6

C. Main Results

TRIAGE offers a strong success-rate–latency trade-off across OpenVLA, OpenVLA-OFT, and SimplerEnv, staying close to full recomputation while reducing prefill latency. Its results vary by setting, but it remains especially competitive on the Jetson.

C. Main Results

  • TRIAGE stays within 3 pp of full recomputation in average success rate and lies on the success-rate–prefill-latency Pareto frontier across backbones, benchmarks, and devices.Methods matching its success rate are slower, while methods as fast or faster are less accurate; it also has the lowest Jetson p99 latency in every setting.Source: C. Main Results · p. 6
  • On OpenVLA, TRIAGE reaches 73.7%, 3.6 pp above SAFE-Pruner at the same FLOPs and 3.5 pp above SpecPrune-VLA at similar latency.VLA-Cache reaches 75.2% but is slower; TRIAGE reduces prefill latency to 0.73× full recomputation on Jetson and 0.43× on RTX 3090.Source: C. Main Results · pp. 6–7
  • On OpenVLA-OFT, TRIAGE averages 95.8%, 1.3 pp below full recomputation and 0.3 pp above SAFE-Pruner at the same FLOPs.It has the lowest Jetson prefill latency (0.52× full recomputation) and end-to-end latency (0.68×); SpecPrune-VLA is 1.0 pp more accurate but slower.Source: C. Main Results · p. 7
  • On SimplerEnv without retuning, TRIAGE averages 54.5% versus 54.8% for full recomputation, while SpecPrune-VLA and VLA-Pruner trail by 4.2 and 6.1 pp.On Jetson, TRIAGE has the lowest prefill (0.74× full recomputation), end-to-end, and p99 latency.Source: C. Main Results · p. 7, Table II · p. 7

VI. DISCUSSION AND LIMITATIONS

TRIAGE combines reuse and pruning according to cache drift, and its evaluated performance offers a strong success-rate–latency trade-off. The discussion identifies empirical hyperparameters, workload-dependent thresholds, and post-hoc motion detection as boundaries or opportunities.

VI. DISCUSSION AND LIMITATIONS

  • TRIAGE's hyperparameters were tuned on development episodes and frozen before evaluation, but their values are empirical rather than derived from first principles.The discussion names δ, T, κ, λ, and µ.Source: §VI · p. 7
  • The drift threshold differs across the tested backbones: reuse became harmful from 0.5 on OpenVLA and from 0.3 on OpenVLA-OFT.The threshold may also shift with backbone, task, and scene speed, motivating an adaptive deployment-time threshold.Source: §VI · p. 7
  • TRIAGE detects motion only after it occurs; optical flow, codec motion vectors, or learned patch selection could identify moving regions before backbone execution.The paper suggests reusing static regions more aggressively and recomputing moving regions first as a possible route to better latency and accuracy.Source: §VI · p. 7

VII. CONCLUSION

  • TRIAGE chooses per token among recomputation, reuse, and pruning, and stays within 3 pp of full recomputation with no compared baseline both more accurate and faster.The conclusion says its advantage grows as the recomputation budget shrinks and the drift threshold may need recalibration for other backbones.Source: §VII · p. 7

APPENDIX

The appendix tests hyperparameter sensitivity and examines how reused-cache errors change with cache age and drift. Its findings indicate that the drift test filters most stale caches, while an age limit safeguards against some deep-layer error.

A. Sensitivity Analysis

  • Every tested hyperparameter change has a 95% interval containing zero, and none shifts LIBERO-Goal success by more than 4 pp from the 70.6% default.Removing the age limit lowers success from 70.6% to 68.6% (paired difference −2.0 pp, 95% CI [−6.2, +2.2]); T acts as a safeguard.Source: A. Sensitivity Analysis · p. 7, Figure 6 · p. 8

B. Deep-Layer Error of Reused Caches

Loading 2610.34519v1…