PaperlayerView exact revision on arXiv

Source-linked AI summary

Attn-Forcing: Learning Full-Context Attention Proxies for Streaming Video Generation

Junyao Gao, Sibo Liu, Jiaxing Li, Yanan Sun, Hanxin Zhu, Weidong Zhang, Jun Zhang, Cairong Zhao

arXiv:2610.30036v1cs.CV

TL;DR

Longer attention contexts improve streaming video quality but impose prohibitive memory and computation costs, while existing selection and compression approaches lack explicit attention-level supervision. Attn-Forcing uses learnable attention proxies and Attention Matching Loss to preserve historical attention contributions under bounded resources. The paper reports consistent quality improvements and state-of-the-art performance over existing baselines.

  • Problem

    Longer attention contexts improve streaming video quality, but their memory and computational costs make them impractical, and existing bounded-context approaches lack explicit supervision for preserving historical contributions to current queries.

    Source: §1 · p. 1

  • Method

    Attn-Forcing uses learnable visual attention proxies updated with historical latents and an Attention Matching Loss to align their attention responses with full-context attention.

    Source: §1 · p. 2, §3.2 · p. 4, §3.3 · p. 6

  • Results

    Attn-Forcing consistently improves streaming video generation quality and achieves state-of-the-art performance over existing baselines.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    Attention proxies provide a way to preserve historical attention contributions during streaming generation under bounded memory and computation.

    Source: §1 · p. 2, §5 · p. 9

    A fixed number of proxies may be insufficient to preserve relevant history as generation horizons grow substantially or visual content becomes more diverse.

    Source: §A.5 · p. 16

Abstract

from arXiv · show

Increasing the attention context length significantly improves streaming video generation quality, while incurring prohibitive memory and computational costs as the context grows. Existing methods attempt to capture this contextual information through attention selection or compression, but remain limited by bounded representations and lack explicit attention-level supervision to retain historical contributions to current queries. To address this, we introduce Attn-Forcing, a novel framework that preserves full-context attention contributions for streaming video generation under bounded memory and computation. We achieve this by introducing attention proxies that continuously approximate the attention responses induced by the historical context. Specifically, we implement the proxies as learnable visual latents and employ a Proxy Update Module to incorporate visual context from historical latents. Moreover, we propose an Attention Matching Loss to explicitly constrain the attention behavior of the learned proxies using full-context attention supervision. Extensive qualitative and quantitative experiments demonstrate that Attn-Forcing consistently improves streaming video generation quality and achieves state-of-the-art performance over existing baseline methods.

1 INTRODUCTION

Longer attention contexts improve streaming video quality but are impractical under bounded memory and computation. Attn-Forcing addresses this by preserving historical attention contributions with proxies and attention-level supervision, and reports state-of-the-art results over baselines.

1 INTRODUCTION

  • Longer KV contexts improve VBench-Long scores, but maintaining them becomes impractical under bounded memory and computational budgets.Self-Forcing’s inference scores rise as its KV cache grows, despite training with a fixed local attention window.Source: §1 · p. 1, Figure 1 · p. 2
  • Existing approaches select or compress historical context, but bounded representations and absent attention-level supervision limit preservation of historical contributions to current queries.The approaches include adaptive retention or retrieval and compact representations.Source: §1 · p. 1
  • Extensive qualitative and quantitative experiments report state-of-the-art performance over baseline methods, while ablations support the proposed designs.The paper identifies improving streaming video generation quality as an experimental outcome.Source: §1 · p. 2
  • Attn-Forcing uses learnable attention proxies and Attention Matching Loss to preserve full-context attention contributions under bounded memory and computation.The paper presents this as a DMD training framework for streaming generation.Source: §1 · p. 2

2 RELATED WORK

Video diffusion has shifted from temporal modules on image backbones toward scalable transformer architectures. Long-video work extends generation horizons, while streaming approaches address long rollouts but remain constrained by fixed-length caches and bounded representations.

2.1 VIDEO DIFFUSION MODELS

  • Diffusion Transformers have become a basis for video generation, following earlier approaches that added temporal modules to image diffusion backbones.The paper builds on Wan2.1-T2V-1.3B and adapts it for streaming long-video generation.Source: §2.1 · pp. 2–3

2.2 LONG VIDEO GENERATION

  • Long-video methods extend pretrained short-video models through inference or temporal-modeling changes, including noise rescheduling, progressive denoising, and positional extrapolation.The cited approaches include FreeNoise, FIFO-Diffusion, RIFLEx, and Diff-VF.Source: §2.2 · p. 3
  • Streaming autoregressive video diffusion uses causal distillation, self-generated histories, and long-rollout strategies, but fixed-length KV caches discard historical information.The passage names CausVid, Self-Forcing, LongLive, and Self-Forcing++.Source: §2.2 · p. 3
  • Context-preservation methods retrieve, prune, or compress historical states, but still use bounded representations without explicit attention-level supervision for full-context contributions.The paper contrasts these approaches with attention proxies trained using Attention Matching Loss.Source: §2.2 · p. 3

3 METHOD

Attn-Forcing combines bounded KV caching with learnable proxies for evicted history and an attention-level loss that aligns proxy responses with an approximation of full-history attention.

3.1 PRELIMINARY

  • Autoregressive generation caches sink and local key-value states for causal attention, while DMD distills a multi-step bidirectional diffusion model into a few-step causal generator.The DMD objective perturbs each generated chunk and estimates its score using a pretrained diffusion teacher and a learnable fake-score model.Source: §3.1 · pp. 3–4

3.2 ATTENTION PROXIES

  • Learnable proxy latents query newly evicted visual latents through cross-attention, then a gated update fuses the captured context into the proxies.The updated proxies pass through the DiT to produce key-value states appended between sink and local states; these let evicted context contribute without retaining the full historical cache.Source: §3.2 · pp. 4–5

3.3 ATTENTION MATCHING LOSS

  • Positive random features and two online accumulated statistics approximate historical attention responses as new key-value states leave the local window.The statistics' sizes are independent of history length, so memory and attention computation do not grow with accumulated history.Source: §3.3 · p. 5
  • Attention Matching Loss aligns proxy responses to the approximate full-history attention response, providing a tractable surrogate for directly optimizing that approximation.The objective is applied to the last L DiT layers and averaged across them; the approximation is used as a supervision target rather than directly for generation.Source: §3.3 · p. 6
  • The overall training objective combines DMD with weighted attention matching, and the matching loss updates only proxy-related parameters.λAM controls the strength of attention matching.Source: §3.3 · p. 6

4 EXPERIMENTS

Attn-Forcing is evaluated on short- and long-horizon video generation, with ablations and analyses examining its design choices and inference costs. It outperforms baselines on key quality measures, while the analyses show trade-offs from longer local attention and reduced proxy capacity.

4.2 COMPARISON WITH SOTA METHODS

  • On 60s MovieGen-prompt videos, Attn-Forcing produces stronger generation quality and motion dynamics than the qualitative comparison methods.The comparison also reports more dynamic long videos while maintaining competitive temporal consistency and visual quality.Source: Figure 3 · p. 7, §4.2 · p. 8

4.1 IMPLEMENTATION DETAILS

  • The evaluation covers 5s and 30s videos using 946 prompts and five seeds each, plus 60s videos using 128 MovieGen prompts.The implementation uses Wan2.1-T2V-1.3B, three proxy latents, 128 random features, and applies Attention Matching Loss to the last six DiT layers.Source: §4.1 · pp. 6–7

4.2 COMPARISON WITH SOTA METHODS

  • At 5s, Attn-Forcing achieves the best quality and total VBench scores, 84.94 and 84.10, and the second-best semantic score, 80.74.The comparison uses similar-scale open-source video generation models.Source: §4.2 · p. 7
  • On 30s videos, Attn-Forcing achieves best total and quality scores of 83.85 and 85.26; on 60s videos, it achieves the best total score of 82.37.For 60s videos, its dynamic degree is 72.14, and it has the best aesthetic and imaging quality scores.Source: §4.2 · p. 7, Table 2 · p. 7

4.3 ABLATION STUDIES

  • Reducing random features from 128 to 64 or 32 lowers total scores from 82.37 to 79.50 and 79.31, with dynamic degree falling from 72.14 to around 47.Background consistency and motion smoothness remain relatively stable; image quality falls from 69.42 to 67.52 and 68.05.Source: §4.3 · p. 8, Table 3 · p. 8
  • Supervising six DiT layers with LAM achieves the best overall ablation performance, with total score 82.37 and dynamic degree 72.14.The final model uses the last six DiT layers.Source: Table 3 · p. 8, §4.3 · p. 9

4.4 ANALYSIS

  • Extending LongLive’s local attention size from 12 to 120 raises background consistency and motion smoothness but lowers total score, dynamic degree, and image quality; direct bAhist use blurs over time.The 120-window comparison reports total scores of 79.26 and 75.01, and Figure 5 shows increasing blur over time when bAhist is used directly.Source: §4.4 · p. 9, Figure 5 · p. 9, Table 4 · p. 9
  • Proxy updates and one extra DiT forward pass add approximately 13% end-to-end inference time on an H20 GPU compared with LongLive.The proxies produce native KV states through the standard DiT forward pass while preserving full-context attention contributions.Source: §4.4 · p. 9

5 CONCLUSION

Attn-Forcing preserves full-context attention contributions during streaming video generation under bounded memory and computation. Learnable proxies incorporate evicted visual context, while Attention Matching Loss aligns their attention responses with full-context supervision; experiments report improved quality and state-of-the-art performance over existing baselines.

5 CONCLUSION

  • Attn-Forcing preserves full-context attention contributions for streaming video generation under bounded memory and computation.It uses learnable attention proxies that incorporate evicted visual context and produce native KV states for subsequent generation.Source: §5 · p. 9
  • Attention Matching Loss uses efficient full-context supervision constructed from online accumulated statistics to improve proxy approximations of historical attention responses.Source: §5 · p. 9
  • Experiments on short- and long-horizon video generation show improved quality and state-of-the-art performance over existing baselines.Source: §5 · p. 9

A.1 DERIVATION OF THE POSITIVE RANDOM FEATURE APPROXIMATION

The appendix derives the positive random feature representation of the softmax kernel used in Eq. 9. Its implementation uses Gaussian orthogonal random projections to reduce finite-sample kernel-estimator variance.

A.1 DERIVATION OF THE POSITIVE RANDOM FEATURE APPROXIMATION

  • The attention scaling is absorbed into the query and key when setting up the softmax-kernel derivation.Source: §A.1 · p. 13
  • The softmax kernel is rewritten and substituted to obtain the positive random feature representation used in Eq. 9.Source: §A.1 · p. 13
  • Gaussian orthogonal random projections maintain Gaussian marginals while reducing variance in the finite-sample kernel estimator.The approximation uses a finite-dimensional positive random feature mapping.Source: §A.1 · p. 13

A.2 TRAINING AND INFERENCE PROCEDURE

Training updates proxies and historical attention statistics after visual latents are evicted, then uses proxy KV states and attention matching in subsequent chunks. Inference uses the updated proxies but does not require the historical statistics or LAM.

A.2 TRAINING AND INFERENCE PROCEDURE

  • After eviction begins, training updates proxy latents from evicted states and historical attention statistics from their cached layer-wise KV states.Source: §A.2 · p. 14
  • From subsequent chunks, proxy KV states are inserted between sink and local KV states, while accumulated statistics provide historical attention supervision for the last six DiT layers.LAM is combined with the DMD objective, and proxy-related parameters are optimized with LAM only during generator updates.Source: §A.2 · p. 14
  • At inference, evicted visual latents update the proxies for later chunks, while historical attention statistics and LAM are not required.Figure 6 provides an additional qualitative comparison on 60-second videos generated using MovieGen prompts.Source: §A.2 · p. 15, Figure 6 · p. 15

A.3 IMPLEMENTATION DETAILS

The Proxy Update Module incorporates newly evicted visual context into current proxy latents through cross-attention and subsequent refinement. The proxies use sink-frame RoPE positions, and specified update inputs are stop-gradient.

A.3 IMPLEMENTATION DETAILS

  • The Proxy Update Module summarizes newly evicted visual hidden states through cross-attention with the current proxy latents.The projections use the proxies for queries and the evicted states for keys and values.Source: §A.3 · p. 15
  • A second cross-attention block and an FFN refine and fuse the summarized context with the current proxies before producing a candidate update.Source: §A.3 · p. 15
  • The candidate update is gated to obtain updated proxy latents; proxies take sink-frame RoPE positions, and current LAM queries and evicted hidden states are stop-gradient inputs.Source: §A.3 · pp. 15–16

A.4 ADDITIONAL QUALITATIVE RESULTS

This subsection presents an additional visual comparison on 60-second videos.

A.4 ADDITIONAL QUALITATIVE RESULTS

  • The subsection presents additional visual comparison results for 60-second videos.Source: §A.4 · p. 16

A.5 LIMITATIONS

Attn-Forcing’s bounded design has two limitations: fixed proxy capacity may not capture increasingly long or diverse histories, and approximate attention supervision introduces error.

A.5 LIMITATIONS

  • A fixed number of proxies bounds overhead but may become insufficient for substantially longer horizons or increasingly diverse visual content.The limitation concerns capacity to preserve all relevant historical information as history grows or varies.Source: §A.5 · p. 16
  • Attention Matching Loss uses a random-feature approximation rather than exact full-context attention, introducing approximation error into its supervision signal.The approximation enables efficient long-context supervision with constant-size online statistics.Source: §A.5 · p. 16

Related papers

Loading 2610.30036v1…