Knowing When to Stop: Adaptive Action Chunking via Internal Cross-Attention Dynamics in VLAs

Runze Xu; Xiaolong Shan; Shuang Dai; Yu Wang; Jincheng Yu

Source-linked AI summary

Knowing When to Stop: Adaptive Action Chunking via Internal Cross-Attention Dynamics in VLAs

Runze Xu, Xiaolong Shan, Shuang Dai, Yu Wang, Jincheng Yu

arXiv:2609.00908v1Sep 1, 2026cs.RO

TL;DR

Fixed execution horizons force a trade-off between inference efficiency and responsiveness because short chunks can oscillate while long chunks become misaligned with new observations. The paper uses cross-attention entropy to detect sustained high-entropy plateaus and adaptively truncate action execution, improving success across simulated and real-world evaluations over fixed-horizon baselines. Its current scope is limited by reliance on identifiable cross-attention pathways and the absence of evaluation on World Action Models.

Problem
Fixed execution horizons trade efficiency against accuracy and cannot consistently accommodate changing task stages.
Method
A training-free truncation mechanism monitors sustained high and temporally stable cross-attention entropy to dynamically select the execution horizon during inference.
Results
The method improves average success over fixed-horizon baselines across simulated and real-world evaluations while adding low inference latency.
Takeaways & Limitations
Cross-attention dynamics provide a practical internal signal for adaptive action execution in VLAs.
Takeaways & Limitations
The method depends on identifiable cross-attention pathways and has not yet been evaluated on World Action Models.

Abstract

from arXiv · show

Action chunking is a standard execution strategy in modern Vision-Language-Action (VLA) frameworks, but fixed execution horizons impose a trade-off between efficiency and accuracy. Short chunks require frequent inference and may cause oscillatory behavior, whereas long chunks can become misaligned with newly observed states. We address this limitation with an adaptive action chunking approach based on internal cross-attention dynamics in the action expert. We observe that, as the prediction horizon extends, action-to-observation cross-attention becomes increasingly dispersed and its entropy rises toward a plateau. This pattern is associated with higher action prediction error and provides an online signal that the current observation offers limited grounding for further open-loop execution. Based on this observation, we introduce a training-free truncation mechanism that detects sustained high-entropy plateaus and dynamically selects the execution horizon during inference. The method uses attention weights already computed by the policy and introduces negligible additional overhead. Evaluations on $π_{0.5}$ and X-VLA across RoboTwin 2.0, LIBERO, and three real-world manipulation tasks show improved average task success over fixed-horizon and adaptive chunking baselines, while preserving efficient closed-loop control. These results show that cross-attention dynamics can provide a practical internal signal for adaptive action execution in VLAs.

I. INTRODUCTION

VLA action chunking improves temporal execution, but fixed horizons cannot balance reactivity and efficiency across task stages. The paper uses action-to-VLM cross-attention entropy to detect weak grounding and dynamically truncate execution without training or architectural changes.

Motivation: Fixed horizons trade computational efficiency and precision: short horizons require frequent inference, while long horizons can misalign with new observations.Optimal horizons also vary across tasks and within task stages.
Core observation: Action-to-VLM attention entropy rises toward a high-value plateau as the prediction horizon extends, indicating reduced concentration on current perceptual context.Sustained high entropy is associated with elevated offline action error.
Method: The proposed training-free truncation mechanism monitors cross-attention entropy and dynamically determines the execution horizon when entropy reaches a plateau.It creates a dynamic perception-action loop with near-zero computational overhead.
Contributions: Evaluations across multiple VLA models and simulated and real-world environments show improved average success over evaluated fixed-horizon baselines and competitive adaptive-baseline performance.The mechanism requires no architectural modification and adds low inference latency.

II. RELATED WORK

Prior VLA work largely focuses on policy training and action generation, while execution commonly relies on manually chosen fixed horizons. This paper instead adapts chunk length using attention dispersion as an inference-time grounding signal.

VLA models: VLA systems combine visual observations, language instructions, and robot actions, commonly using a pretrained VLM to condition a dedicated action-generation module.Earlier systems use autoregressive discretized action tokens, while newer approaches employ continuous methods.
Execution strategies: Existing work primarily uses manually chosen fixed horizons during execution rather than adapting chunk length online.The paper studies when future actions cease to be sufficiently grounded in the current observation.
Action chunking: Action chunking mitigates cumulative errors and improves temporal modeling, but fixed horizons require task-specific tuning and restrict adaptive generalization.BID identifies chunk lengths but maintains static horizons during execution.

III. METHODS

The method unifies different VLA architectures through action-to-perception cross-attentional retrieval. It analyzes action-query attention over VLM and action tokens to obtain an internal signal for future-action grounding.

Architectural formulation: π0.5 uses static VLM keys and values, whereas X-VLA jointly updates concatenated VLM and action tokens through self-attention.Despite this architectural difference, both can be analyzed through their action-perception interaction.
Cross-attentional retrieval: The action expert uses action queries over the predicted horizon to retrieve information from VLM-provided perceptual features.This interaction is represented by an attention matrix whose rows index predicted action steps and columns index perceptual features.
Grounding signal: The unified attention formulation enables an internal signal associated with reduced perceptual grounding of future actions.The analysis instantiates the method with π0.5 and discusses X-VLA-specific adaptations separately.

B. Attention Dispersion and Entropy Dynamics

Later action queries become less selectively grounded in visual context, with attention entropy increasing by action index and eventually reaching a stable plateau. The method quantifies this dispersion by aggregating and normalizing action-to-VLM attention across layers.

Qualitative attention dynamics: In layers 9–15, early actions attend sharply to language tokens, while later actions attend diffusely across visual tokens.The later-action pattern motivates attention dispersion as an empirical risk signal for reduced perceptual grounding.
Entropy computation: Cross-attention entropy is computed from normalized action-to-VLM attention distributions after aggregating coefficients across layers.Higher entropy indicates a more divergent allocation of attention across VLM tokens.
Entropy interpretation: The entropy of each action token is bounded by 0 ≤ E_j ≤ log N, with the upper bound reached when attention is uniformly distributed.This architecture-dependent bound distinguishes genuinely dispersed plateaus from locally flat but concentrated attention.
Entropy dynamics: Entropy increases with action index before reaching a stable plateau, whose onset varies across task stages.Coarse approach maneuvers plateau later, whereas precise pre-interaction phases plateau earlier.

3) Entropy as Accuracy Proxy:

Cross-attention entropy is associated with action-prediction error beyond action position alone. Error rises sharply near the high-entropy regime, supporting entropy as an online truncation signal.

Average action-prediction MSE rises sharply as entropy approaches E = 6.5, approximately 95% of the theoretical upper bound ln(968).The analysis sorts valid entropy–MSE pairs into 12 equal-frequency bins before comparing their means.
The top MSE quartile has 10.4× higher mean error and 12.1× higher error standard deviation than the remaining samples.
These results support high attention entropy as an online risk signal for truncating predictions.

C. Adaptive Action Chunking

The method detects sustained high-entropy plateaus in the predicted action sequence and uses their earliest occurrence to adapt execution length. It preserves the full horizon when no qualifying plateau is found and reuses attention weights without retraining or additional sampling.

The method computes a cross-attention entropy sequence and identifies the earliest sustained high-dispersion regime during inference.
A moving-average filter smooths the entropy sequence before its first-order changes are evaluated.
A plateau requires both high smoothed entropy and temporally stable entropy changes within a k-step window.The high-entropy condition uses threshold η relative to log N, while stability requires the average change to remain below τ.
If no window satisfies both conditions, the policy executes the full predicted horizon Hp rather than truncating at a low-entropy plateau.
The rule retains longer chunks while attention remains concentrated and truncates after sustained dispersion associated with increased prediction error.It uses attention weights already produced by the action expert and requires neither retraining nor additional sampling.
The evaluation covers generalizability across Hang mug, Stack cups, and Pen Tidying real-world tasks.

IV. EXPERIMENTS

The method is evaluated with π0.5 and X-VLA on RoboTwin 2.0, LIBERO, and real-world tasks against fixed-length and adaptive baselines. The experiments use shared hyperparameters across models and benchmarks, with one shorter smoothing-window exception for π0.5 on LIBERO.

Evaluations use π0.5 and X-VLA across RoboTwin 2.0, LIBERO, and three real-world tasks.
Comparisons against fixed-length, multi-sampling, and self-attention-based baselines assess success rates and generalizability.
π0.5 on LIBERO uses k = 3 because the policy predicts only Hp = 10 actions per query, while τ and η remain unchanged.All four LIBERO suites share this setting.

A. Simulation

Simulation evaluations on RoboTwin 2.0 and X-VLA show that adaptive action chunking performs strongly across multi-task settings, while LIBERO results further test transfer across task suites.

RoboTwin 2.0: 18.11 steps is the method’s average execution horizon, exceeding the strongest fixed-chunk baseline while maintaining higher average success.The result suggests longer execution during high-confidence phases can reduce inference frequency.
X-VLA: X-VLA evaluations use predicted horizons of 30 actions and compare fixed execution horizons of 15, 20, 25, and 30.Shorter fixed horizons are omitted because of low success rates.

2) LIBERO Experiments:

LIBERO experiments show that adaptive execution transfers to a separately trained policy and improves performance across all four task suites, especially on long-horizon tasks.

LIBERO results: 97.25% overall average success is achieved with adaptive execution, versus 94.88% for the strongest fixed-chunk baseline.The comparison uses fixed chunk sizes of 2, 4, 6, 8, and 10 without modifying policy parameters.
LIBERO results: 94.5% success on LIBERO-Long exceeds the best fixed setting by 4.5 percentage points.This is the largest reported improvement among the four LIBERO suites.
LIBERO results: The method achieves the highest success rate on every LIBERO task suite.The evaluation uses the open-source trained π0.5 model and compares adaptive execution against fixed execution chunks.

B. Realworld Experiments

Real-world evaluations across three manipulation tasks show consistent gains from adaptive action chunking over fixed and self-attention-based alternatives.

Evaluation protocol: Each task is evaluated over 20 independent trials after separate π0.5 policies are fine-tuned from 50 expert demonstrations.The protocol uses 40,000 fine-tuning steps with batch size 32.
Real-world results: 14.2% average success-rate improvement is obtained over the best fixed-chunk configuration across the three real-world tasks.The best fixed configuration is reported as chunk = 20,30.
Real-world results: 6.7% average success-rate improvement is obtained relative to the strongest self-attention-based adaptive baseline.The evaluation covers both single-arm and dual-arm tasks.
Baseline comparison: Multi-sampling struggles because its small chunk size fragments execution and causes intermittent pauses during movement.These pauses are reported to reduce overall success rate.

C. Efficiency Analysis

The method uses internal attention signals to adapt execution with low latency, while its applicability remains constrained by the need for identifiable cross-attention pathways.

Latency: 29.5 ms additional latency is incurred by multi-sampling for each inference cycle.The overhead remains substantial for high-frequency control despite parallel sampling passes.
Latency: Less than 7 ms of additional latency is introduced by the proposed attention-based strategy relative to fixed chunking.By contrast, multi-sampling increases each inference cycle by 29.5 ms.
Mechanism: Attention entropy enables training-free dynamic truncation by using cross-attention dynamics already computed within the action expert.The approach is presented as a practical internal signal for adaptive closed-loop control.
Limitations: Architectures with highly interleaved or hierarchical structures may be difficult to support because they lack easily aggregated cross-attention pathways.The paper also reports no evaluation on emerging World Action Models with distinct generative dynamics.