Source-linked AI summary
Light Interaction: Training-Free Inference Acceleration for Interactive Video World Models
Jiacheng Lu, Haoyi Zhu, Sipei Yi, Enze Xie, Yu Li, Cheng Zhuo
TL;DR
Long interactive video trajectories are expensive because memory, attention, and repeated denoising costs grow. Light Interaction provides training-free trajectory-dependent adaptive computation through context management, denoising caching, and hardware-aware sparse attention, achieving up to 2.59× speedup without retraining. Its scope includes assumptions about relevance signals, short-step denoising, and model-dependent sparse-attention speedups.
Problem
Scaling interactive video world models to long trajectories is prohibitively expensive because context memory grows, attention is quadratic, and denoising repeatedly executes the Transformer.
Method
Light Interaction adapts spatial and temporal context, reuses denoising outputs during reliable revisiting, and accelerates remaining attention with autoregressive-aware 3D block sparsity and fused kernels.
Results
2.59× speedup is achieved on HY-WorldPlay and Matrix-Game-3.0 without model retraining.
Takeaways & Limitations
Interaction dynamics can guide inference computation by discarding unreliable retrieved memory, adapting temporal history, and reusing outputs when regions are revisited.
Takeaways & Limitations
The framework assumes a camera-pose-aware relevance signal, validates denoising caching only for K ≤4 models, and has model-dependent sparse-attention speedups.
Abstract
from arXiv · showhide
Interactive video world models generate video chunk by chunk in response to user-controlled camera movements, enabling applications such as real-time game simulation, virtual scene navigation, and embodied AI training. However, scaling to long interactive trajectories is prohibitively expensive due to growing context memory, quadratic attention complexity, and repeated denoising steps. We present Light Interaction, a training-free inference acceleration framework for interactive video world models. Our key insight is that interaction naturally enables trajectory-dependent adaptive computation: retrieved spatial memory can be discarded during novel exploration, temporal context can be adjusted according to local latent dynamics, and early-step model outputs can be reused when the camera revisits familiar regions. Based on this insight, Light Interaction combines adaptive context management, denoising cache acceleration, and hardware-software co-designed 3D block sparse attention with fused Triton kernels. Evaluated on HY-WorldPlay and Matrix-Game-3.0, Light Interaction achieves up to 2.59x speedup without model retraining while maintaining competitive visual quality.
1 Introduction
Interactive video world models face severe long-trajectory costs, motivating Light Interaction’s training-free, trajectory-dependent adaptive computation and hardware-aware acceleration. The framework combines adaptive context management, denoising reuse, and sparse attention, achieving up to 2.59× speedup without retraining.
- Generating 10 seconds of HY-WorldPlay video on one A100 GPU can take over 200 seconds because context, attention, and denoising costs grow.
- Existing methods separately compress KV history, reuse denoising outputs, or sparsify attention, but do not reliably adapt computation to camera trajectories or avoid autoregressive overhead.
- Light Interaction uses retrieval similarity to gate spatial memory and denoising reuse, while local latent dynamics determine the temporal context window.
- Denoising cache acceleration reuses early-step outputs during reliable revisiting while preserving the final step for quality correction.
- Hardware-software co-designed 3D block sparse attention preserves text and current-chunk tokens, sparsifies historical visual KV blocks, and uses fused Triton kernels.
- 2.59× speedup is achieved on HY-WorldPlay and Matrix-Game-3.0 without model retraining.
2 Related Work
Related work accelerates interactive or video generation through retrieval, context compression, denoising reuse, and sparse attention. Light Interaction differs by adapting these mechanisms to camera trajectories and autoregressive execution constraints.
- Autoregressive generation predicts frames sequentially, supporting streaming and interactive applications, while retrieval primarily preserves long-term spatial consistency.
- KV compression and fixed temporal windows use uniform policies, whereas Light Interaction adapts retrieved spatial memory and temporal context to camera trajectories.
- Existing denoising caches reuse activations or estimate output similarity, but content-agnostic policies can be unreliable during novel exploration.
- Sparse attention reaches 2.28–2.30× speedups in standard bidirectional generation, but causal constraints and data reordering weaken gains in autoregressive generation without hardware-aware kernels.
3 Light Interaction
Light Interaction combines trajectory-dependent context selection, denoising reuse, and hardware-aware sparse attention to reduce inference cost in interactive video generation. It discards unreliable retrieved memory during exploration, adapts temporal context to latent dynamics, reuses denoising outputs during revisiting, and makes remaining attention efficient.
- Light Interaction combines adaptive context management, denoising cache acceleration, and AR-aware 3D sparse attention with fused execution.The framework reconstructs context selectively, reuses denoising computations under reliable revisiting, and optimizes historical attention for autoregressive generation.
- Adaptive Context Management: Temporal context is selected at chunk level and adjusted from smoothed latent dynamics, shrinking under large dynamics and expanding under stable dynamics.The method estimates dynamics from recent stable historical latent units, smooths them with an exponential moving average, and adapts the window within budget Lm.
- Adaptive Context Management: Adaptive context management discards unreliable retrieved spatial memory during exploration and retains geometrically relevant memory during revisiting.Pose-aware similarity identifies whether historical views are valid; discarded memory reduces effective context length in unseen regions.
- Lightweight Denoising Cache Acceleration: Denoising cache reuse is activated for reliable revisiting, approximates intermediate steps with the first-step output, and computes the first and final steps normally.When K ≤2, no intermediate reuse is applied; the final step remains a normal correction step.
- Hardware-Software Co-designed 3D Sparse Attention: 3D block sparse attention sparsifies only historical visual KV blocks while retaining text and current-frame tokens, then uses fused operators to reduce execution overhead.The design preserves local spatiotemporal structure and addresses block gathering, layout conversion, padding, and gather/scatter costs in autoregressive execution.
4 Experiments
Experiments evaluate Light Interaction on two interactive video models using quality, runtime, memory, and component-level analyses. The method delivers the strongest overall trade-offs while its modules provide complementary efficiency and quality effects.
- Light Interaction achieves the best quality–efficiency trade-off on HY-WorldPlay and the fastest runtime on Matrix-Game-3.0 with competitive visual quality.
- On HY-WorldPlay, Light Interaction provides the best fidelity and self-comparison consistency, with 2.59× speedup, 140.36 s lower latency, and 21.91 GB less peak memory.
- On Matrix-Game-3.0, Light Interaction achieves a 1.61× speedup and the fastest runtime while maintaining competitive quality.
- Effectiveness of Individual Components: The three components are complementary: temporal context management mainly reduces latency and memory, spatial management improves fidelity, and their combination yields the strongest standalone quality gain.
- Effectiveness of Individual Components: Denoising cache reduces effective denoising cost from 4 steps to about 3 on average and achieves the highest PSNR against the original model through dynamic reuse.
- Effectiveness of Individual Components: 3D sparse attention is a major acceleration source, while context management compensates for its standalone quality degradation by improving retained-context quality.
- Latency and Kernel Studies: Progressive enabling reduces KV reconstruction, denoising, and remaining generation costs in sequence, while kernel fusion brings a 1.40× speedup to sparse attention.
- Hyperparameter Studies: Increasing the retained ratio improves reconstruction quality but weakens sparsity's runtime advantage, making the adopted setting a quality–efficiency balance.
5 Conclusion
Light Interaction is a training-free framework that reduces interactive world-model computation through trajectory-dependent adaptive context, denoising reuse, and hardware-aware sparse attention. It achieves up to 2.59× speedup without retraining, but depends on geometric relevance signals, short-step denoising, and model-specific execution structure.
- Light Interaction combines adaptive context management, denoising cache acceleration, and hardware-software co-designed 3D block sparse attention with Triton fused kernels.
- Evaluated on HY-WorldPlay and Matrix-Game-3.0, Light Interaction achieves up to 2.59× speedup without model retraining.
- The framework assumes camera-pose-aware relevance signals, validates denoising caching only for K ≤4 models, and has speedups dependent on autoregressive memory organization.
Broader Impacts
Light Interaction improves accessibility for interactive world-model research and more responsive applications, while faster generation may lower barriers to synthetic-media creation at scale.
- Light Interaction improves accessibility for research and supports more responsive embodied-AI, game-simulation, and virtual-scene-navigation applications.
- Faster generation lowers the barrier to creating synthetic media at scale, although the inference-only method does not expand model capabilities.
A Leave-One-Out Ablation of Light Interaction on HY-WorldPlay
The leave-one-out ablation evaluates whether each Light Interaction component is necessary within the integrated pipeline, using a fixed HY-WorldPlay subset because the analysis is computationally intensive.
- The ablation results are averaged over a small fixed evaluation subset rather than the full benchmark.
- Removing any single component weakens the overall speed–quality trade-off, although each removal produces a different failure mode.The experiment starts from the full system and removes one component at a time.
- Removing 3D Sparse Attention yields the strongest quality recovery but substantially increases latency.Denser attention preserves more information while reducing practical acceleration.
- Removing KV Cache Management degrades both speed and memory efficiency, identifying historical-context control as a core requirement.
- Removing Denoising Cache increases runtime, but its degradation is smaller than the other component removals.The ablation characterizes denoising cache as a lightweight complementary accelerator.
B Additional Implementation Details
The implementation uses model-specific temporal settings, fixed-volume sparse blocks, a dense warm-up period, and timing procedures focused on the dominant affected stages.
- HY-WorldPlay retains only the most recent temporal unit, while Matrix-Game-3.0 does not enable parameterized temporal-window adaptation.The HY-WorldPlay setting uses L_t = 1 because local dynamics are typically strong.
- Sparse-attention block sizes are (4, 8, 4) for HY-WorldPlay and (4, 4, 8) for Matrix-Game-3.0, with 128 tokens per block.
- Sparsification applies only to the historical visual KV cache; text tokens and current-frame denoising KV remain fully preserved.
- The first three chunks use full dense computation before adaptive pruning and denoising-cache reuse are enabled.This warm-up avoids unstable decisions when historical information is insufficient.
- Timing analysis focuses on KV reconstruction and denoising computation, while latent-dynamics and pose/FoV similarity overhead is negligible relative to the backbone.Sparse index generation is counted as part of sparse attention.