Beyond Token Importance: Preserving Spatial Scaffolds for Efficient Vision-Language-Action Inference

1 Introduction

• VLA pruning methods that select tokens by task relevance, temporal redundancy, or action-stage dynamics do not explicitly preserve scene spatial structure.This matters because manipulation requires both locating task-relevant objects and grounding physical actions, including gripper poses and contact points, in scene geometry.Source: §1 · p. 1
• At an 85% pruning ratio, Stride outperforms semantic and random pruning, but its performance collapses at the nearby 87.5% ratio.This Stride–Random Reversal suggests retained tokens’ spatial arrangement matters in addition to their individual semantic relevance.Source: §1 · p. 2
• The spatial coverage radius measures the largest blind spot after pruning and is strongly correlated with pruning-induced error across policies and rates.The paper proposes using semantic relevance to decide which regions receive more budget and geometric structure to select tokens within each region.Source: §1 · p. 2, Figure 1 · p. 2
• GeoScaffold partitions each image into spatial regions, allocates token budgets using semantic weights, and selects within-region scaffold tokens using center-seeded farthest point sampling.A spatial floor keeps at least one token in each region, while farthest point sampling reduces the local coverage radius.Source: §1 · pp. 2–3
• At 90% pruning on π0.5 and LIBERO, GeoScaffold reaches 75.1% average success across four task suites, 48.5 percentage points above FastV, with a 1.96× prefill speedup.On OpenVLA-OFT at 87.5% pruning, it reaches 79.9% average success versus 72.6% for FastV and 51.8% for VLA-Cache; on π0.5 it also retains 93.2% success with a 1.78× prefill speedup at 80% pruning.Source: §1 · pp. 2–3

2.1 Related Work

• Spatially grounded VLAs use scene geometry—including object locations, contact regions, free space, and trajectories—to support manipulation beyond semantic recognition.The cited related work also describes SpatialVLA and DreamVLA as explicitly modeling spatial information.Source: §2.1 · p. 3
• Existing efficient VLA methods reduce inference cost through training-time changes, architecture design, caching, or pruning, while GeoScaffold revisits token selection in the training-free setting.Examples include dynamic early exiting, Mamba-based architectures, temporal key-value reuse, and action-aware dynamic pruning.Source: §2.1 · p. 3

2.2 Preliminaries

• At each control step, a VLA receives RGB views, a language instruction, and proprioceptive state, then predicts a continuous action chunk.The pipeline encodes observations into a multimodal prefix and conditions action generation on that prefix.Source: §2.2 · pp. 3–4
• Visual-token pruning selects a budgeted subset of the visual prefix, ideally preserving the full model’s predicted action chunk while physically shortening the sequence.The paper defines deviation from the full-model action chunk as the quantity to minimize; non-visual tokens remain unchanged.Source: §2.2 · p. 4

3.1 Stride–Random Reversals Reveal Spatial Aliasing

• At 85% pruning, Stride reaches 81.3% average success across four LIBERO suites, versus 71.8% for Random and 56.4% for semantic pruning.Stride also outperforms both baselines on LIBERO-Spatial, Object, Goal, and Long at this ratio.Source: §3.1 · p. 4
• Raising Stride’s pruning ratio from 85% to 87.5% cuts its average success from 81.3% to 31.0%, a 50.3-point drop as its per-view budget falls from 38 to 32 tokens.Random and semantic pruning degrade more smoothly over the same interval, and Stride also exhibits non-monotonic success across other budget changes.Source: §3.1 · pp. 4–5

3.2 Quantifying Spatial Scaffold Collapse via Coverage Radius

• Spatial coverage radius is the maximum distance from any image-grid location to its nearest retained token, measuring the largest blind spot after pruning.A larger radius means less uniform visual support across the image.Source: §3.2 · p. 5
• Across 27 method–ratio configurations, coverage radius correlates with failure rate at Pearson 0.85 and Spearman 0.84; at fixed budgets, the average correlation over nine ratios is 0.93.The fixed-budget comparison holds token count constant, focusing the comparison on how tokens are spatially arranged.Source: §3.2 · p. 5
• Coverage radius correctly ranks 22 of 27 pairwise comparisons, including the Stride reversal, while most mismatches occur near saturated success rates.The 85%, 87.5%, and 90% comparisons show how aliasing with the two-dimensional grid can enlarge uncovered regions and worsen Stride performance.Source: §3.2 · pp. 5–6
• Coverage radius is not the sole determinant of success, so the proposed design preserves both semantic relevance and geometric coverage.Task relevance, camera-view differences, and manipulation stage can also affect success.Source: §3.2 · p. 6

3.3 GeoScaffold

• GeoScaffold uses shallow-layer attention to score regions, gives each region a one-token floor, distributes remaining budget by region weight, and selects within-region scaffold tokens using center-seeded farthest point sampling.The method is training-free; it physically removes unselected tokens after the specified layer to accelerate prefill inference.Source: §3.3 · pp. 6–7

Main results on π0.5

• At R = 0.9, FastV success rate falls from 75.0% at R = 0.8 to 26.6%, while LIBERO-Spatial and LIBERO-Long fall to 15.8% and 13.6%, respectively.SparseVLM follows the same collapse pattern.Source: §4 · p. 8
• At R = 0.9, GeoScaffold improves average success rate over FastV by 48.5 percentage points; with only 10% of visual tokens retained (25 tokens per image), it achieves 75.1% average success rate.Source: §4 · p. 8

Inference efficiency

• At R = 0.8, GeoScaffold reduces prefill latency from 27.68 ms to 15.58 ms, a 1.78× prefill speedup, and achieves a 1.28× end-to-end speedup with a 4% success-rate drop.The farthest point sampling step can be vectorized with negligible computation and latency overhead.Source: §4 · p. 8

Cross-architecture validation

• At R = 0.875 on OpenVLA-OFT, GeoScaffold achieves 79.9% average success across four LIBERO suites, versus 72.6% for FastV and 51.8% for VLA-Cache.On LIBERO-Long, GeoScaffold reaches 66.4%, versus 40.4% for FastV and 26.2% for VLA-Cache.Source: §4 · p. 9

Single-level geometric pruning

• At R = 0.85, GeoScaffold exceeds Random, Global-FPS, and Stratified-Random by at least 15 percentage points.Global-FPS is 7.7 percentage points worse than Random, while Stratified-Random is 3.5 points better than Random.Source: §5 · p. 9

Region- versus token-level attention

• At R = 0.85, success rises from 72.0% for Region-Semantic to 81.8% for Region-Stride and 90.5% with center-seeded two-dimensional farthest point sampling.The results indicate attention is more effective for inter-region budget allocation than token-level selection within regions.Source: §5 · p. 9

Spatial scaffold design

• At R = 0.85, center seeding alone adds 4.6 points and two-dimensional farthest point sampling alone adds 5.4 points; together they improve 8.7 points over Region-Stride.Center seeding mainly helps when one token is retained per region; two-dimensional sampling matters more when regions retain multiple tokens.Source: §5 · p. 9

6 Conclusion

• The conclusion identifies spatial coverage radius as the strongest single-variable predictor of pruning-induced error across strategies and pruning rates.GeoScaffold assigns budgets across regions with attention, selects within regions geometrically, applies a per-region spatial floor, and uses center-seeded farthest point sampling.Source: §6 · pp. 9–10
• At 90% pruning on π0.5+LIBERO, GeoScaffold improves four-suite average success by 48.5 percentage points over FastV while delivering a 1.96× prefill speedup.The method transfers to OpenVLA-OFT without modification.Source: §6 · p. 10

Appendix

• The appendices cover additional per-suite LIBERO results, OpenVLA-OFT cross-architecture results, complexity and latency, limitations and assumptions, broader impact, and assets and licenses.Source: Appendix · p. 11

• Across the four task suites, Figure 3 shows per-suite success rates for Stride, Random, and semantic pruning across pruning ratios, with the Stride–Random reversal recurring in every suite.Source: Figure 3 · p. 12

B Cross-architecture results on OpenVLA-OFT

• At R = 0.875, GeoScaffold reaches 79.9% average success, versus 72.6% for FastV and 51.8% for VLA-Cache.On LIBERO-Long, the methods score 66.4%, 40.4%, and 26.2%, respectively.Source: Table 3 · p. 12, B Cross-architecture results on… · p. 13
• At moderate pruning rate R = 0.75 on OpenVLA-OFT, GeoScaffold achieves a 90.8% four-suite average, slightly ahead of FastV at 90.5% and above VLA-Cache at 83.8%.Source: B Cross-architecture results on… · p. 12
• The larger gap at R = 0.875 than at R = 0.75 suggests the spatial-scaffold benefit transfers to OpenVLA-OFT under aggressive token budgets.The reported pattern is described as similar to that observed on π0.5.Source: B Cross-architecture results on… · pp. 12–13

C Latency and efficiency analysis

• Pruning reduces every subsequent backbone layer’s FLOP cost, while the action expert benefits through cross-attention to the shortened visual prefix.The pruning overhead reuses the layer-K attention map and is dominated by a scatter_add and vectorized FPS lookup, with total cost O(nT).Source: C Latency and efficiency analysis · p. 13
• 1.78×, 1.95×, and 1.96× prefill speedups are measured at pruning rates R=0.80, R=0.875, and R=0.90, respectively.At R=0.80, prefill latency falls from 27.68 ms unpruned to 15.58 ms; end-to-end speedup plateaus at ∼1.33×.Source: C Latency and efficiency analysis · p. 13, Table 4 · p. 13
• Prefill gains saturate above R=0.875 as residual cost is dominated by the unprunable image encoder and text/action prefix, while denoise latency stays within 5%.The latency profile was measured for single-step π0.5 inference on one RTX 4090, averaging 100 LIBERO-Spatial episodes after torch.compile autotune steps were discarded.Source: C Latency and efficiency analysis · p. 13

D Limitations

• The main evaluation uses LIBERO simulation benchmarks, with no validation yet on real-robot deployments.GeoScaffold also assumes visual tokens retain an underlying two-dimensional patch-grid structure, which may not directly apply to irregular tokenization or learned token merging.Source: D Limitations · p. 13
• Coverage radius is not the only factor affecting VLA performance; semantic relevance, camera viewpoint, manipulation stage, and task difficulty also matter.Cross-architecture validation covers π0.5 and OpenVLA-OFT, with broader evaluation on additional VLA families left for future work.Source: D Limitations · pp. 13–14

E Broader Impact

• Inference acceleration may lower computational cost and improve research accessibility, but policies still require task-specific safety, robustness, and monitoring evaluation before real-world deployment.The paper notes that more efficient robot policies may also accelerate deployment in settings where failures could cause physical harm.Source: E Broader Impact · p. 14

F Existing Assets and Licenses

• The assets are used only for research evaluation, and pretrained model weights and benchmark data are not redistributed.Checkpoint terms that depend on upstream base models follow the corresponding model-card and repository terms.Source: F Existing Assets and Licenses · p. 14, Table 5 · p. 14