PaperlayerView exact revision on arXiv

Source-linked AI summary

Just Noticeable Difference Modeling for Token Compression in Vision-Language-Action Models

Zhuoyuan Li, Rui Zhao, Jin Wang, Hanwei Zhu, Cong Zhang, Giuseppe Valenzise, Weisi Lin, Kin-Man Lam

arXiv:2608.21247v1cs.CVcs.RO

TL;DR

Token compression for VLA models must reduce inference cost without changing latency-sensitive robot actions, but existing redundancy and importance cues only indirectly assess this tolerance. The paper introduces Action-JND, a token-wise estimator of action-preserving perturbation, and uses it for KV reuse and pruning. On LIBERO with OpenVLA and OpenVLA-OFT, Action-JND consistently improves compression reliability, especially at aggressive ratios.

  • Problem

    Existing VLA compression criteria use indirect redundancy or importance cues rather than measuring how much a token can change before altering downstream actions.

  • Method

    Action-JND defines noticeability through language-conditioned robot-action deviation and estimates maximum tolerable token perturbations with a lightweight deep-feature estimator.

  • Results

    Action-JND consistently improves compression reliability on LIBERO with OpenVLA and OpenVLA-OFT, particularly under aggressive compression ratios.

  • Takeaways & Limitations

    Action-JND provides a criterion for more reliable closed-loop VLA compression while improving inference efficiency for robot control.

Abstract

from arXiv · show

Token compression has become a key technique for reducing the inference cost of large foundation models, with approaches such as token pruning and KV-cache reuse widely adopted in vision-language models and recently explored for embodied agents. In embodied agents, tokens not only support perception and semantic understanding but also directly affect latency-sensitive closed-loop robot action prediction. Existing schemes typically guide compression using redundancy or importance cues, such as visual similarity, attention scores, and saliency. However, these cues only indirectly measure the key factor for safe compression: how much a token can change before causing an unacceptable deviation in downstream actions. This receiver-dependent tolerance is closely related to the principle of just noticeable difference (JND). Classical JND characterizes signal tolerance in the human visual system, while machine-oriented JND extends this concept to downstream machine responses. Building on this progression, we introduce Action-JND, which extends JND modeling to embodied perception by defining noticeability through the language-conditioned action response of a vision-language-action (VLA) policy in closed-loop control. A token change is considered admissible only when the induced action deviation remains within a tolerated margin. To realize this concept, we develop a lightweight token-wise JND estimator in deep visual-feature space to predict the maximum tolerable perturbation while preserving policy responses. The resulting action-tolerance score serves as a plug-and-play criterion for VLA compression paradigms, including stale-KV reuse and token pruning, prioritizing action-tolerant tokens for compression. Experiments on the LIBERO benchmark with OpenVLA and OpenVLA-OFT demonstrate that Action-JND consistently improves compression reliability, especially under aggressive compression ratios.

I. INTRODUCTION

The paper argues that existing VLA token-compression criteria use indirect redundancy or importance signals rather than action-level tolerance. It introduces Action-JND, which estimates tolerable token changes from language-conditioned robot-action deviations and applies them to KV reuse and pruning.

  • Motivation: Existing compression methods estimate token compressibility from redundancy or importance cues, which only indirectly indicate whether compression preserves downstream actions.Signals include visual similarity, attention, and saliency.
  • JND Perspective: Machine-oriented JND defines noticeability through changes in downstream machine responses rather than signal distortion alone.The embodied setting extends this receiver-dependent boundary to VLA action responses.
  • Action-JND: Action-JND defines the maximum compression-induced token change that preserves a language-conditioned robot action within a tolerated deviation during closed-loop control.The receiver is a robotic policy whose response includes translation, rotation, and gripper control.
  • Method: A lightweight token-wise estimator in deep visual-feature space learns tolerable perturbations and converts their magnitudes into action-tolerance scores.Larger scores indicate tokens that tolerate larger feature perturbations while preserving the policy response.
  • Applications: The learned criterion integrates with stale-KV reuse and token pruning, prioritizing tokens whose representation changes are less likely to alter predicted actions.The framework is designed as a plug-and-play criterion for existing VLA compression paradigms.
  • Results: Experiments across compression ratios show that Action-JND improves compression reliability for VLA models, particularly under aggressive compression.The evaluation uses OpenVLA and OpenVLA-OFT on LIBERO.

A. Vision-Language-Action Models

VLA models generate executable robot actions from visual observations and language instructions, making efficient inference important for responsive closed-loop control. Existing compression methods remain comparatively underexplored for embodied control and rely on proxy signals rather than explicit token action sensitivity.

  • VLA Models: VLA models map visual observations and language instructions to executable robot actions for active robotic control.Representative systems formulate control as vision-language generation using robotic trajectory data.
  • VLA Models: OpenVLA-OFT improves deployment practicality through choices including parallel decoding, action chunking, continuous actions, and an ℓ1 regression objective.These choices redesign the action-generation process rather than only compressing input tokens.
  • Token Compression: Token compression methods reduce memory, bandwidth, or dense visual-token computation through KV-cache compression, merging, pruning, and selection.Their extension to embodied robotic control remains comparatively underexplored.
  • Embodied Efficiency: Closed-loop VLA control repeatedly processes temporally adjacent observations, so inference latency limits how frequently robot actions can be updated.Efficient inference is therefore important for responsive control.
  • Compression Gap: Existing embodied compression decisions rely on proxy signals such as visual stability, attention importance, scheduling confidence, or token redundancy rather than explicit action sensitivity.These cues identify potentially compressible tokens but do not estimate how much alteration a token can tolerate before affecting the predicted action.
  • JND Background: Classical JND models human perceptual thresholds, while machine-oriented JND evaluates whether perturbations preserve recognition, multimodal responses, or static task performance.The paper positions embodied JND as a further extension in which the receiver produces robot actions.

B. Receiver-Dependent Formulation

Receiver-dependent JND defines the largest representation perturbation whose downstream response remains within a tolerated deviation. Action-JND applies this principle to language-conditioned VLA policies by preserving robot-action responses under visual-token perturbations.

  • B. Receiver-Dependent Formulation: Receiver-dependent JND maximizes perturbation magnitude subject to a receiver-response discrepancy staying below tolerance.The receiver may be a human visual system, machine task, or robotic policy.
  • B. Receiver-Dependent Formulation: Machine-oriented JND measures noticeability through changes in a downstream task head rather than direct signal distortion.Task discrepancy can use classification loss, regression error, or distribution divergence.
  • C. Action-JND Formulation: The estimator learns token-wise perturbations that are large while preserving the frozen policy’s predicted action.Visual-token features are perturbed before the policy produces the response used to evaluate action noticeability.
  • C. Action-JND Formulation: Action-JND treats a frozen VLA policy as the receiver and preserves its language-conditioned action response after visual-feature perturbation.The formulation supports continuous actions and discrete action-token distributions, including single-step and multi-step responses.
  • C. Action-JND Formulation: For discrete-action policies such as OpenVLA, action discrepancy is measured by KL divergence between clean and perturbed action-token distributions.The discrepancy is selected according to the policy’s action parameterization.
  • C. Action-JND Formulation: For continuous-action policies, the action discrepancy combines weighted translation, rotation, and gripper-control differences.The weights λ_p, λ_r, and λ_g determine the contribution of each action component.
  • C. Action-JND Formulation: The relaxed learning objective balances perturbation magnitude against action preservation through λ_mag and λ_act.The resulting perturbation magnitude characterizes tolerance to action-preserving changes.

IV. ACTION-JND-AWARE TOKEN COMPRESSION

Action-JND-aware compression learns token-wise action tolerance from projected visual features and uses it to rank tokens for reuse or pruning. The estimator preserves view-specific spatial structure, including separate processing for dual-view inputs.

  • IV. ACTION-JND-AWARE TOKEN COMPRESSION: The framework trains an Action-JND estimator on projected visual-token features from a frozen VLA policy.The estimator predicts token-wise perturbation maps that are converted into action-tolerance scores.
  • 1) Action-JND Estimator Architecture:: The estimator predicts perturbations as large as possible while preserving the VLA policy’s action response.Its output is a learned approximation of the tolerable token perturbation.
  • IV. ACTION-JND-AWARE TOKEN COMPRESSION: Action-tolerance scores rank candidate tokens for temporal KV-cache reuse and direct visual-token pruning.Existing framework-specific cues can first identify candidates, after which Action-JND ranks them by tolerance.
  • 1) Action-JND Estimator Architecture:: For single-view inputs, visual tokens are reshaped into a 16×16 grid and processed through spatial and channel-mixing residual blocks.A bottleneck convolution and output projection produce the token-wise perturbation map.
  • 1) Action-JND Estimator Architecture:: For dual-view inputs, independent estimators process primary-view and wrist-view tokens before concatenating their outputs.The view-specific design preserves each camera’s spatial topology and avoids mixing coordinate systems.

2) Learning Objective:

The Action-JND estimator is trained while the VLA policy remains frozen, comparing clean and perturbed action responses. Its objective encourages large perturbations subject to action consistency, with discrepancy adapted to discrete or continuous actions.

  • 2) Learning Objective:: The VLA policy is frozen during training, while only the Action-JND estimator is updated.Clean and perturbed visual features pass through the same policy to produce comparable action responses.
  • 2) Learning Objective:: The loss encourages action consistency between clean and perturbed branches while maximizing predicted perturbation magnitude.This creates perturbations that remain action-tolerable but are as large as possible.
  • 2) Learning Objective:: The perturbation-magnitude term averages the ℓ2 norm of predicted token perturbations across tokens.M(ˆ∆_t) measures the magnitude used in the learning objective.
  • 2) Learning Objective:: For discrete action-token prediction, the action discrepancy is the KL divergence between clean and perturbed action distributions.This instantiation applies to OpenVLA-style discrete action outputs.
  • 2) Learning Objective:: For continuous action regression, the action discrepancy uses a regression distance such as weighted ℓ1 distance.The same training framework therefore supports both discrete and continuous VLA action parameterizations.

C. From JND Maps to Token Scores

Action-JND converts predicted token perturbations into token-wise action-tolerance scores, then uses those scores to rank compression candidates without changing the underlying VLA compression frameworks.

  • Score construction: Action-JND produces token-wise scores indicating how much feature perturbation each token can tolerate while preserving the policy’s action response.The predicted perturbation is used for scoring rather than injected into visual features during inference.
  • Score construction: The relative score is used for KV-cache reuse, while the absolute score is used for direct visual token pruning.This separates scale-normalized scoring from feature-magnitude-sensitive scoring across the two compression operations.
  • Candidate ranking: Action-JND ranks tokens within candidate sets by descending action tolerance and can also order framework-specific fallback candidates when needed.The method refines candidate ordering rather than replacing the compression framework’s candidate construction.
  • Pipeline integration: The unified interface supports both stale-KV reuse and token pruning by prioritizing tokens with larger JND scores for compression.Both operations use the same principle: less action-critical representations are selected for compression.
  • KV-cache reuse: For KV-cache reuse, Action-JND ranks candidates after temporal-stability and task-relevance filtering, while selected tokens reuse previous-timestep key/value states.Nonselected tokens continue to use current-step hidden states, preserving the original reuse schedule.
  • Token pruning: For token pruning, temporally stable, low-attention candidates are ranked by Action-JND, with additional candidates filled according to action tolerance if the pruning pool is insufficient.The pruning pipeline uses aligned patch similarity, attention-based protection, and JND-guided selection.

A. Experimental Setup

Experiments evaluate Action-JND on LIBERO using OpenVLA and OpenVLA-OFT, comparing it with several KV-cache reuse and token-pruning baselines across multiple compression settings and efficiency metrics.

  • Benchmarks and backbones: Experiments use four LIBERO suites—Spatial, Object, Goal, and Long—to evaluate language-conditioned robotic manipulation and VLA policy learning.The suites cover spatial reasoning, object-centric manipulation, goal-conditioned execution, and long-horizon tasks.
  • Baselines: Action-JND is compared with VLA-Cache and KV-cache-compatible adaptations of FastV, SparseVLM, and DivPrune, plus token-pruning baselines.The pruning comparisons include FastV, SparseVLM, DivPrune, and a VLA-Cache-based pruning baseline.
  • Compression settings: KV-cache reuse is tested at 30%, 40%, 60%, and 80% target ratios, while token pruning is tested at 25%, 50%, 75%, and 87.5%.KV-cache experiments include adaptive soft and strict_guarded policies; pruning uses a budgeted policy.
  • Metrics: Evaluation reports suite success rates, average accuracy, accuracy differences, FLOPs, relative FLOPs, CUDA latency, latency differences, and control frequency.Control frequency measures how often the policy can update robot actions during closed-loop control.

B. Experimental Analysis

The analysis examines Action-JND under KV-cache reuse and token pruning, emphasizing task preservation, inference efficiency, and qualitative compression behavior.

  • Scope: The analysis covers both KV-cache reuse and token pruning.
  • Evaluation focus: Task preservation is a central focus of the evaluation.
  • Evaluation focus: Inference efficiency and qualitative compression behavior are also examined.

1) Result Analysis on KV-Cache Reuse:

Action-JND improves the reliability of KV-cache reuse across OpenVLA and OpenVLA-OFT, with increasingly larger advantages over VLA-Cache at higher reuse ratios.

  • Cross-backbone results: Action-JND achieves competitive or superior task performance across reuse ratios on both OpenVLA and OpenVLA-OFT.It obtains the highest average accuracy under the soft policy on both backbones and generally offers a more favorable accuracy–efficiency trade-off under strict_guarded reuse.
  • Robustness under aggressive reuse: At higher reuse ratios, manually designed compression criteria suffer rapid performance degradation.
  • Quantitative gains: 23.70 and 41.65 percentage points improve OpenVLA average accuracy over VLA-Cache at 60% and 80% reuse, respectively.On OpenVLA-OFT, the corresponding improvements are 6.20 and 10.10 percentage points.
  • Efficiency: Action-JND reduces CUDA latency and increases control frequency relative to vanilla inference while preserving task performance under aggressive reuse.

2) Result Analysis on Token Pruning:

Action-JND improves OpenVLA token-pruning reliability, with larger gains at more aggressive pruning ratios while substantially reducing inference cost and preserving task performance.

  • Accuracy: From 50% pruning onward, Action-JND achieves the highest average accuracy among compared OpenVLA token-pruning schemes.Its advantage becomes more pronounced as the pruning ratio increases.
  • Accuracy: 1.45, 3.05, and 11.65 percentage points: Action-JND’s average-accuracy improvements at 50%, 75%, and 87.5% pruning, respectively.
  • Accuracy: 6.55 percentage points: Action-JND’s advantage over the strongest competing method at 87.5% pruning.
  • Pruning rationale: Physical token removal makes errors from deleting action-sensitive tokens difficult to recover in subsequent layers, motivating action-tolerance-guided pruning.
  • Accuracy: At 87.5% pruning, Action-JND reaches 42.00% average accuracy versus 30.35% for VLA-Cache-based pruning and 35.45% for the strongest competing baseline.
  • Efficiency: At 87.5% pruning, Action-JND uses 33.06% of vanilla FLOPs, lowers CUDA latency from 52.88 ms to 30.16 ms, and raises control frequency from 18.91 to 33.15 Hz.

3) Qualitative Visualization:

Qualitative LIBERO comparisons show Action-JND-aware compression protecting action-sensitive information during KV-cache reuse and token pruning, supporting successful execution under aggressive compression.

  • Qualitative visualization: Action-JND qualitative examples cover both KV-cache reuse and token pruning under aggressive compression on LIBERO.
  • Qualitative visualization: The visualizations show Action-JND better protecting action-sensitive information, with successful executions in cases where competing methods fail or require more control steps.
  • Implication: The paper applies learned action-tolerance scores to KV-cache reuse and token pruning, demonstrating more reliable compression and improved inference efficiency for closed-loop robot control.
Loading 2608.21247v1…