PaperlayerView exact revision on arXiv

Source-linked AI summary

Video Compression Meets Video Generation: Latent Inter-Frame Pruning with Attention Recovery

Dennis Menn, Yuedong Yang, Bokun Wang, Xiwen Wei, Mustafa Munir, Feng Liang, Radu Marculescu, Chenfeng Xu, Diana Marculescu

arXiv:2603.05811v2cs.CV

TL;DR

Video generation models recompute redundant latent tokens, limiting computational efficiency for real-time use. LIPAR prunes temporally redundant latent patches and recovers pruned-token attention without training. It reports 1.45× average inference speedup while preserving high visual fidelity, with scope currently focused on conditioned video generation.

  • Problem

    Video generation remains computationally costly because models recompute tokens despite temporal redundancy, constraining real-time deployment.

  • Method

    LIPAR performs training-free end-to-end latent inter-frame pruning and uses Attention Recovery to approximate unpruned attention outputs.

  • Results

    1.45× average inference speedup was achieved while preserving high visual fidelity and outperforming existing training-free pruning methods.

  • Takeaways & Limitations

    The method bridges pixel-space compression principles and latent generative pipelines for conditioned video generation.

  • Takeaways & Limitations

    LIPAR currently focuses on conditioned video generation because its pruning mask relies on a source video; extension to text-to-video remains future work.

Abstract

from arXiv · show

Current video generation models suffer from high computational latency, making real-time applications prohibitively costly. In this paper, we address this limitation by exploiting the temporal redundancy inherent in video latent patches. To this end, we propose the Latent Inter-frame Pruning with Attention Recovery (LIPAR) framework, which detects and skips recomputing duplicated latent patches. Additionally, we introduce a novel Attention Recovery mechanism that approximates the attention values of pruned tokens, thereby removing visual artifacts arising from naively applying the pruning method. Empirically, our method increases video editing throughput by $1.53\times$, achieving an average of 19.3 FPS on an NVIDIA RTX 4090 with the 1.3B Self-Forcing model (4-step denoising, FP16). The proposed method does not compromise generation quality and can be seamlessly integrated with the model without additional training. Our approach effectively bridges the gap between traditional compression algorithms and modern generative pipelines.

1 Introduction

Video generation remains computationally expensive because latent diffusion models recompute every token despite temporal redundancy. LIPAR adapts compression to latent video patches, pruning redundant tokens while recovering attention to preserve quality.

  • Motivation: Real-time video generation is constrained by high computational costs, including difficulty reaching 30 fps on a single GPU.These constraints persist despite causal attention and few-step distillation advances.
  • Motivation: Latent diffusion models allocate fixed computation to every token, unlike compression algorithms that skip repeated pixel patches.Redundancy is difficult to identify in latent space before generation.
  • Related Work: Existing token-merging methods incur frequent similarity-search overhead, operate only in selected layers, and can introduce visual artifacts.These limitations reduce their computational and quality benefits.
  • Contribution: LIPAR identifies redundant latent patches, prunes them end-to-end, and uses Attention Recovery to reduce the training-inference discrepancy.The framework is training-free and allows all layers to benefit from pruning.
  • Results: 1.45× throughput speedup was achieved on 51 DAVIS video-text prompts, reaching 12.2 FPS on one A6000 GPU with 20% lower GPU usage.The method also obtained an 86.4% win-tie rate against unpruned results in evaluation by 14 human participants.
  • Empirical Evidence: 0.69 and 0.77 Pearson correlations were observed between pixel-space and latent-space changes for WAN 2.1 and WAN 2.2 VAEs, respectively.These results motivate transferring temporal redundancy-based compression from pixels to latents.
  • Empirical Evidence: 46% latent compression preserved high visual fidelity, with averaged LPIPS ≤0.05 compared with the original decoded videos.The experiment used ten DAVIS videos and replaced nearly unchanged patches with patches from previous frames.

4 Problem Formulation

The formulation requires pruning and denoising to remain approximately compatible, with Multi-head Self-Attention as the critical sequence-dependent operation. The analysis identifies noise-induced attention and value-aggregation errors that Attention Recovery must address.

  • 4.1 Target Objective: The target is for pruning, denoising, and recovery to reproduce the output of the full unpruned token sequence.The denoising operation should approximately commute with pruning.
  • 4.1 Target Objective: Approximating Multi-head Self-Attention outputs is sufficient because subsequent cross-attention and linear layers operate independently per token.Thus, sequence-level approximation is concentrated in self-attention.
  • 4.2 MSA Approximation: The MSA approximation groups pruned tokens with kept tokens whose keys are similar, while accounting for position-dependent RoPE rotations.The one-dimensional example prunes x2, x3, and x5 and generalizes to the broader case.
  • 4.2 MSA Approximation: O(δ) bounds the approximation error, where δ is the maximum difference between original keys and RoPE-rotated kept-token keys.The bound follows from Lipschitz continuity of self-attention with respect to keys.
  • 4.3 The Impact of I.I.D. Noise: Independent Gaussian noise makes naive token duplication invalid because it creates artificial correlations and amplifies noise during attention.The analysis decomposes tokens into clean signal and noise components and projects noise through W_Q, W_K, and W_V.
  • 4.3 The Impact of I.I.D. Noise: Duplicated noise changes the attention-score distribution from N(0,D) to χ2_D, adding positive bias and higher variance that inflate duplicated-token weights.The duplicated case has expected bias D and variance 2D.
  • 4.3 The Impact of I.I.D. Noise: Duplicating values changes aggregate noise variance from O(nID) to O(n2ID), producing quadratic variance growth.Empirically, forcing duplicated tokens to be equal creates noisy patterns and significantly degrades generated-video quality.

5 Methods

LIPAR accelerates conditioned video generation by pruning temporally redundant latent patches, recovering attention for pruned tokens, and restoring fixed-size patches for decoding. Its pruning and recovery design addresses computational savings while preserving visual quality.

  • LIPAR Overview: LIPAR operates in three stages: latent inter-frame pruning, Attention Recovery, and latent patch restoration.Pruning reduces sequence length before transformer processing, while restoration reconstructs patches required for decoding.
  • Attention Recovery: Attention Recovery aligns pruned-sequence attention with the original computation through M-Degree Approximation and Noise-Aware Duplication.M-Degree Approximation preserves the m closest keys and values, while Noise-Aware Duplication copies only clean components to avoid artificial noise correlations.
  • Latent Inter-Frame Pruning: Latent inter-frame pruning removes patches whose consecutive-frame differences fall below a predefined threshold, bypassing redundant computation.The method compares patches at corresponding spatial locations across consecutive frames.
  • Latent Inter-Frame Pruning: Motion-aware pruning reduces mispruning from subtle latent movements by using neighboring-token information and both short-term and long-term temporal differences.The design reflects object-level dynamics and supports the i.i.d. noise assumption required by Attention Recovery.
  • Latent Patch Restoration: Restoration reconstructs pruned and denoised patches by duplicating corresponding patches from the previous frame, producing fixed-dimensional inputs for decoding.The restoration step maps the processed latents back toward the format required for pixel-space decoding.
  • Attention Recovery: M-Degree Approximation estimates the attention exponential sum using the m largest terms, which causal attention approximates by selecting the most recent indices.The approximation error is bounded by O(δ), where δ is the maximum token approximation error.
  • Attention Recovery: Even with full duplication, LIPAR retains a linear speedup by reducing self-attention queries and accelerating feed-forward and cross-attention layers.The method is compatible with parallelism tools such as FlashAttention and reduces attention memory usage and computational complexity.

6 Experiments

Experiments show that LIPAR improves throughput while preserving visual quality in video editing, outperforming existing training-free pruning methods. Attention Recovery is central to avoiding artifacts from pruning, and latency scales linearly with retained tokens.

  • Comparison with Other Models: 18.4% of participants preferred LIPAR over Self-Forcing, 13.3% preferred the baseline, and 68.3% tied.The study involved 14 participants performing pairwise comparisons.
  • Comparison with Other Models: 1.45× higher throughput than Self-Forcing was achieved on an NVIDIA A6000 GPU while maintaining comparable or better visual quality than competing V2V models.The evaluation used 51 DAVIS video-prompt pairs and measured the full generation pipeline.
  • Comparison with Training-Free Pruning Methods: LIPAR preserves more visual quality than Importance-based Token Merging, IDM, and ToMe, which introduce patch artifacts or severe blurring.Quantitative results also show LIPAR consistently outperforming the compared pruning methods across nearly all metrics, especially at higher pruning rates.
  • Generation Quality VS. Proposed Techniques: 33.8% token pruning retains clear, high-fidelity video only when Attention Recovery combines M-Degree Approximation with Noise-Aware Duplication.Direct pruning causes artifacts, while partial recovery introduces noisy patterns by violating the diffusion model’s i.i.d. noise assumption.
  • Latency Profiling: Pearson r = 0.999 between remaining-token percentage and latency confirms an O(n) relationship, enabling latency prediction before video editing.The measurements used 480×832 videos with 72 frames on an NVIDIA A6000 GPU.

8 Motion-Controlled Video Generation

LIPAR generalizes to motion-controlled video generation with the bidirectional-attention Wan 2.2 5B model. On TTM, it maintains baseline quality while increasing diffusion-denoising throughput.

  • Motion-Controlled Video Generation: LIPAR was extended to the training-free TTM task using the Wan 2.2 5B model with bidirectional attention.TTM transforms warped videos into natural videos that follow user-specified motion trajectories.
  • Motion-Controlled Video Generation: 1.5× higher inference throughput was achieved on TTM while maintaining performance comparable to the baseline on VBench and Warp Error.Throughput measures only the diffusion denoising process.

9 Conclusion

The paper connects temporal redundancy in pixel and latent spaces to training-free latent compression for video generation. LIPAR uses Attention Recovery to achieve speedup while preserving visual fidelity.

  • Conclusion: A strong correlation between temporal changes in pixel and latent spaces motivates copying unchanged latent patches instead of recalculating them.The paper presents this as a bridge between pixel-level video compression and latent video generation.
  • Conclusion: LIPAR achieves an average 1.45× inference speedup while preserving high visual fidelity and outperforming existing training-free pruning methods.The method is presented as a foundational step toward integrating pixel-level compression with latent video generation.

10 Related Work - Real-time Interactive Video Generation

Real-time interactive video generation targets low-latency editing and motion control, but achieving responsive performance remains difficult on consumer-grade GPUs.

  • Real-time Interactive Video Generation: Real-time video editing aims to provide instantaneous prompt-based edits for live applications.This setting reduces reliance on sophisticated pre-made filters.
  • Real-time Interactive Video Generation: Motion control uses explicit motion signals, such as warped static images, to guide synthesis and enable intuitive interactions.Examples include dragging a dog’s head to turn.
  • Real-time Interactive Video Generation: Achieving a real-time 30 FPS response on a consumer-grade GPU remains challenging for motion-conditioned video generation.

11 Latents Compression Experiment

The experiment tests whether temporally similar latent patches can be compressed without degrading decoded video quality. Increasing the similarity threshold raises compression while maintaining high visual similarity at substantial rates.

  • The study measures whether replacing patches with temporal predecessors affects decoded results.The compressed latents are evaluated against the original decoded video using a similarity criterion.
  • The compression rule replaces p_{t+1}^{x,y} with p_t^{x,y} when their L1 distance is below θ.Patches that do not meet the threshold remain unchanged.
  • 44.5% of latent patches can be compressed while decoded results remain similar to the uncompressed video.Compression replaces sufficiently similar patches with their temporal predecessors.
  • 46% compression preserves high visual similarity, with LPIPS ≤0.05 compared with the original decoded video.The rate is obtained by increasing θ across ten input videos.
  • The results establish substantial temporal redundancy in the latent space that can support latent-space video compression.The conclusion follows from high similarity despite compressing a large fraction of patches.

12 Deriving Target Objective

The target objective is to make pruning approximately commute with the denoising Transformer. Because point-wise layers are unaffected, the key requirement is preserving self-attention behavior under pruning.

  • The objective is to ensure D(P(x_t)) approximately equals P(D(x_t)) for the denoising network.Here x_t is the token sequence, P is pruning, and D is the denoising network.
  • Block-level equivalence is sufficient for global approximation because the denoising network is composed of stacked attention blocks.The condition is imposed across the Transformer blocks.
  • Feed-Forward Network and Cross-Attention layers are unaffected by pruning because they operate point-wise on video tokens.Each output depends only on the corresponding token.
  • Multi-Head Self-Attention is the critical layer because each token's calculation depends on the entire token sequence.This inter-token dependency makes attention behavior sensitive to pruning.
  • The required self-attention condition is MSA(P(x_t)) approximately equal to P(MSA(x_t)).Attention Recovery is designed around this approximation target.

13 General Case for the Impact of I.I.D Noise

The analysis explains why duplicating tokens changes self-attention statistics. Independent and duplicated noise produce different distributions, and the duplicated case inflates attention weights through a positive bias.

  • Duplicated noise follows a weighted sum of chi-squared variables, whereas independent noise is approximated by a Gaussian distribution.The distinction depends on whether the noise terms are duplicated or independent.
  • For large token dimension D, the Gaussian distribution is justified as a Central Limit Theorem approximation.The approximation applies to the independent-noise case.
  • The duplicated case introduces a positive bias in the attention interaction.This bias is associated with the shared noise structure.
  • Learned Transformer projection matrices typically have a heavily positive trace, making the duplicated-token bias large.The positive trace is linked to identical tokens attending to themselves.
  • The bias effectively inflates attention weights on duplicated tokens.This explains why direct token reuse can create an attention discrepancy under pruning.

14 Latent Inter-Frame Pruning and Restoration Full Algorithms

The full pipeline detects redundant latent patches, constructs a temporally informed keep mask, restores pruned patches from predecessors, and evaluates the resulting video-editing system.

  • Latent Inter-Frame Pruning: The keep-mask algorithm combines short- and long-term temporal differences, then applies median blur, morphology, and dilation.Adaptive 3D Gaussian thresholding is used to account for neighboring differences and motion.
  • Latent Patch Restoration: The restoration algorithm copies temporal predecessors into positions marked as pruned while retaining the first frame unchanged.Kept positions receive the current latent patch, and pruned positions reuse the previous restored patch.
  • Latent Inter-Frame Pruning: LIPAR identifies similar patches at the same spatial location across consecutive frames to bypass recomputation.The pruning criterion compares temporally adjacent latent patches using an L1 threshold.
  • Latent Patch Restoration: Subtle latent movements can cause mispruning, allowing restoration artifacts to propagate through decoding and degrade video quality.This motivates the method's attention-recovery and motion-aware design.
  • Implementation and Evaluation: The implementation uses Self-Forcing with four-step denoising, trims the KV cache to six recent frames, and prunes 32% of tokens on average.Experiments use an NVIDIA A6000 GPU with fixed pruning thresholds and a fixed random seed.
  • Implementation and Evaluation: Evaluation uses 51 DAVIS video-prompt pairs and reports Warp Error, VBench scores, and throughput against real-time video-to-video and training-free pruning methods.The benchmark includes qualitative and quantitative comparisons.

16 Webpage for Human Evaluation Test

The evaluation compares LIPAR with baseline methods using human judgments and qualitative motion-control results, while documenting its efficiency advantages and current scope limitations.

  • Human Evaluation: 51 video-prompt pairs from DAVIS were evaluated with a Two-Alternative Forced Choice study comparing LIPAR against four baselines.Participants viewed two videos side-by-side and judged overall quality, including temporal consistency and frame quality.
  • Throughput Comparison: LIPAR computes the pruning mask once with approximately 10 ms overhead, unlike token-merging baselines that repeatedly perform expensive merge operations.This efficiency advantage is reported despite identical pruning rates.
  • Throughput Comparison: LIPAR prunes end-to-end across all layer components, whereas the cited baseline token-merging approach is restricted to the Self-Attention module.The broader pruning scope is presented as maximizing acceleration.
  • Qualitative Motion Control: In an owl motion-control example, 47% of tokens are pruned while outputs remain realistic, resemble the baseline, and follow the warped-video motion trajectories.The unchanged background provides substantial temporal redundancy for pruning.
  • Limitations and Future Work: LIPAR currently focuses on conditioned video generation because it derives the pruning mask from the source video.Extending the method to text-to-video generation is identified as future work.
  • Limitations and Future Work: Attention Recovery in bidirectional architectures requires auxiliary conditioning, such as a clean image condition, to preserve the i.i.d. Gaussian noise assumption.The paper identifies noise filtering as a possible way to remove this constraint.
  • Limitations and Future Work: LIPAR compares the previous frame at the same spatial location when computing temporal redundancy, without explicitly compensating for camera motion.The paper proposes incorporating optical-flow estimation in future work to improve efficiency.
Loading 2603.05811v2…