Sparse-WAM: Accelerating World Action Models via Action-Guided Sparse Imagination

1 INTRODUCTION

• Dense future-frame tokens are repeatedly processed during denoising, increasing inference latency and limiting control frequency.WAM imagination tokens form the majority of the visual-action sequence and evolve throughout denoising.Source: §1 · p. 1, Figure 1 · p. 2
• Prior methods omit future imagination at inference or compress future-frame tokens, but do not target regions by action relevance.In WAMs, noisy future tokens are jointly denoised with action tokens, making relevant regions difficult to determine in advance.Source: §1 · p. 1
• Sparse-WAM is training-free and uses action-to-future attention to retain frame-specific regions alongside shared spatial context.Pilot combines lightweight scoring with cross-step reuse of token selections to reduce selection and sparse-execution overhead.Source: §1 · p. 2
• On LIBERO, Sparse-WAM achieves a 1.98× speedup over dense eager inference with a 0.30 percentage-point decrease in average task success.The evaluation covers three WAMs across LIBERO, RoboLab-120, LIBERO-Plus, and real-world robotic tasks.Source: §1 · p. 2

Efficient World Action Models

• WAM efficiency methods omit future prediction at inference or reduce its cost through compact subgoals, lower-resolution representations, asymmetric denoising, or feature reuse.Sparse-WAM retains joint visual–action denoising and selectively allocates future-frame computation without additional training.Source: §2 · pp. 2–3

Token-Level Caching and Pruning

• VLA pruning uses observed-input or task/action cues, while video methods use token importance or trajectory predictability; these signals do not directly identify evolving future regions that support action prediction.Sparse-WAM uses action-to-future attention and reuses selected positions across denoising steps.Source: §2 · p. 3

Efficient LLM Serving

• MoE routing and scheduling optimize expert execution, while 4-bit quantization reduces memory and computation; Sparse-WAM instead selects future-frame tokens during joint denoising.The related-work discussion cites Wang et al. (2025), Wang et al. (2026b), and Hu et al. (2026).Source: §2 · p. 3

3.1 WORLD ACTION MODELS

• In a shared-Transformer WAM, future-frame tokens and an H-step action chunk are jointly denoised; head-averaged action-to-future attention serves as a proxy for token relevance.The attention weights retain normalization over all keys visible to each action query.Source: §3.1 · p. 3

3.2 OBSERVATIONS

• Each future frame predominantly receives attention from roughly H/F contiguous action queries, with boundary queries also attending to neighboring frames; hotspots shift across frames.These patterns motivate scoring frames with their temporally associated action queries and selecting regions separately by frame.Source: §3.2 · p. 4, Figure 2 · p. 4
• Excluding hotspots increases cross-frame attention overlap, while contextual regions such as the gripper remain attended, motivating shared anchor positions alongside frame-specific selection.The figure presents this as its second observation.Source: §3.2 · p. 4, Figure 2 · p. 4
• Attention distributions substantially overlap between consecutive denoising steps across all three camera views, despite continued changes in future representations.This consistency motivates reusing token selections to reduce repeated selection overhead.Source: §3.2 · p. 4, Figure 2 · p. 4

4.1 ONLINE ACTION-GUIDED TOKEN SELECTION

• Each future frame is scored from its associated action queries, with central queries weighted more heavily, and layers ranked for localized attention.Queries near group boundaries also attend to neighboring frames.Source: §4.1 · p. 5

Appendix A.1 provides the query-group definition and layer-quality calculations.

• Per-frame core tokens follow shifting hotspots, while shared anchors favor strong, consistent cross-frame attention; disjoint sets fix each frame at Kc + Ks tokens.The shared anchors retain contextual regions that receive consistent attention across frames.Source: Appendix A.1 · pp. 5–6

4.2 PILOT: ONLINE SELECTION AND SPARSE EXECUTION

• Pilot profiles attention from dense conditional passes, reuses selections and cached visual predictions within each action chunk, and updates omitted future latents from cached predictions while recomputing actions.Sparse steps process selected future tokens compactly alongside all observation and action tokens.Source: §4.2 · pp. 6–7

5.1 EXPERIMENTAL SETTINGS

• Evaluation covers FastWAM-Joint on LIBERO, LIBERO-Plus, and real-world tasks, plus Cosmos 3 Nano Policy and Edge on RoboLab-120; inference runs on an NVIDIA RTX 4090.Reported metrics are task success rate, speedup, and FLOPs as a percentage of the dense model; main-table speedups compare with dense eager inference and include execution optimizations.Source: §5.1 · p. 7

5.2 SIMULATION RESULTS

• Sparse-WAM achieved the largest evaluated speedup on LIBERO at 1.98×, reducing FLOPs by 50.35% while attaining 98.45% success, 0.30 percentage points below dense inference, and the highest accelerated RoboLab-120 average success on Edge (23.00%) and Nano (35.50%).Source: §5.2 · pp. 7–8

5.3 REAL-WORLD EXPERIMENTS

• 2.08× speedup with 75.00% average success on real-world manipulation, compared with 77.78% for dense FastWAM-Joint.The tasks are object packing, cup stacking, and battery insertion on the AgileX Cobot Magic platform.Source: §5.3 · p. 8, Table 3 · p. 9

5.4 ABLATION STUDIES

• With 184 of 360 tokens retained per future frame, combining 80 core tokens and 104 stable anchors achieved 35.5% success, outperforming core-only, stable-only, and random selection.The combined selection maintained approximately 1.81× speedup, comparable to core-only selection.Source: §5.4 · p. 9
• Increasing the future-token pruning ratio from 28.89% to 68.89% raised speedup from 1.45× to 2.36× while lowering success from 37.5% to 28.3%.The sweep demonstrates an efficiency–control trade-off as pruning becomes more aggressive.Source: §5.4 · p. 9

6 LIMITATIONS AND FUTURE WORK

• Efficiency was benchmarked on an NVIDIA RTX 4090 and three WAMs; evaluation on other hardware and broader architectures remains future work.The paper also identifies performance degradation under environmental perturbations and proposes adaptive token selection as a future direction.Source: §6 · p. 9

7 CONCLUSION

• Sparse-WAM achieves approximately 1.8× speedup on RoboLab-120 and 2.0× on LIBERO while largely preserving task success across evaluated settings.The conclusion also covers Cosmos 3 Edge, Cosmos 3 Nano Policy, FastWAM-Joint, and real-world manipulation tasks.Source: §7 · p. 10
• The authors interpret the results as evidence that dense computation over evolving imagination tokens is partially redundant for action prediction.They conclude that sparsifying this computation enables more efficient inference while largely preserving task performance.Source: §7 · p. 10

A ADDITIONAL METHOD DETAILS

• The appendix evaluates the following scores at a full denoising step τd, suppressing that index in subsequent definitions.The passage frames these definitions as support for the selection rule and scorer.Source: A ADDITIONAL METHOD DETAILS · p. 13

A.1 TOKEN SELECTION DETAILS

• Frame-aligned query groups define per-frame scoring; central queries receive greater weight, and layer quality uses attention mass and normalized spatial entropy.The highest-quality layers form the core-layer set.Source: §A.1 · p. 13
• Shared anchors are chosen outside per-frame core sets using scores favoring high mean and low variability, yielding Kc + Ks retained tokens per frame.Anchor positions are shared, but their representations remain frame-specific.Source: §A.1 · p. 13

A.2 ATTENTION PROFILING IN PILOT

• Pilot scores action-to-future attention using the backbone’s normalization and visibility constraints, without computing a full attention matrix or value-weighted output.It reuses the original normalizer to preserve attention mass and stores only spatial scores.Source: §A.2 · p. 13

B.1 BENCHMARKS AND EVALUATION PROTOCOLS

• Evaluation covers LIBERO’s 40 tasks and 2,000 rollouts, RoboLab-120’s 120 tasks and 1,200 rollouts, and LIBERO-Plus’s 560 perturbation rollouts.Compared methods use identical selected variants, initial-state indices, environment seeds, and model seeds.Source: §B.1 · p. 14

B.2 IMPLEMENTATION DETAILS

• Default configurations use a 32-action horizon, six informative layers, fixed token budgets, and online layer and position selection for each action chunk.Cosmos uses four aligned queries per frame, while FastWAM-Joint uses sixteen; Table 4 presents backbone and default Sparse-WAM settings.Source: §B.2 · p. 14, Table 4 · p. 14

B.3 COMPARATIVE METHODS

• Comparisons use ToCa and WorldCache cache reuse, SpecPrune-VLA observation-based pruning, and C3ache joint visual–action residual reuse.The approaches differ in which tokens or predictions they cache, prune, or reuse.Source: §B.3 · p. 14

B.4 TIMING DETAILS

• Cosmos 3 Edge timing used batch size 1, BF16, four denoising steps, and 30 synchronized CUDA-event measurements after five warm-ups.Latencies are medians except C3ache, which is amortized over complete refresh/reuse cycles; Table 5 reports latency per action chunk.Source: §B.4 · p. 15, Table 5 · p. 15
• Sparse-WAM achieves 1.55× under eager execution, 1.85× with optimizations relative to dense eager, and 1.56× relative to equally optimized dense inference.These Cosmos 3 Edge latency comparisons include scoring and selection overhead; Table 5 separates sparse-inference gains from backend gains.Source: Table 5 · p. 15, §B.4 · p. 15

B.5 REAL-WORLD FINETUNING AND EVALUATION

• FastWAM-Joint is fully fine-tuned on eight H800 GPUs with global batch size 64, BF16, and task-specific budgets of 15,000–20,000 steps.The learning rate is 10^-4, with cosine scheduling, weight decay 0.01, and gradient clipping 1.0.Source: §B.5 · p. 15
• Each prediction uses a single observation without history and produces 32 actions; dense and sparse variants are evaluated on the same tasks.The tasks are packing three objects into a container, mouse-battery assembly, and stacking three cups.Source: §B.5 · p. 15
• The reported comparison measures success rates and latency for dense and sparse variants on the three manipulation tasks.The passage refers to Table 3 but supplies no numerical outcomes.Source: §B.5 · p. 15

• The extended analyses measure robustness, selection reuse, and attention consistency motivating the method.Source: C EXTENDED EXPERIMENTS · p. 16

C.1 ROBUSTNESS ANALYSIS

• 63.04% success and 207.2 ms latency: Sparse-WAM is 7.85 percentage points below dense FastWAM-Joint, but more robust than action-only FastWAM on LIBERO-Plus.The comparison is reported for the seven perturbation categories; Table 6 presents LIBERO-Plus success rates and generation latency per action chunk.Source: §C.1 · p. 16, Table 6 · p. 16

C.2 SELECTION REUSE AND REFRESH FREQUENCY

• 35.5% success with a 1.81× speedup: one full-refresh step in the four-step schedule, versus refreshing every step at 36.8% success.More frequent refreshes improve success but reduce acceleration, supporting one scoring pass per action chunk with later-step reuse.Source: §C.2 · p. 16

C.3 CROSS-STEP AND CROSS-FRAME ATTENTION OVERLAP

• 81.11% consecutive-step overlap on Cosmos 3 Edge and 97.97% on FastWAM-Joint exceed cross-frame overlap of 68.89% and 81.85%, respectively.This pattern supports reusing selections across denoising steps while choosing core positions separately for each future frame.Source: §C.3 · p. 16