PaperlayerView exact revision on arXiv

Source-linked AI summary

Backend-Agnostic Sparse Attention for Fast High-Resolution Visual Generation

Liao Ma, Jiayi Song, Yunfeng Wu, Songhua Liu, Peilin Zhao

arXiv:2610.08772v1cs.CV

TL;DR

Dense self-attention makes high-resolution DiT generation costly, while window methods trade cross-window continuity against practical speed. BASA uses structured window shifts and sparse context to retain regular computation while enabling information exchange. It narrows the theoretical-to-measured acceleration gap across FLUX and Wan while maintaining competitive generation quality.

  • Problem

    Dense self-attention's quadratic complexity makes high-resolution generation costly, while partitioned windows block cross-boundary interaction and fine-grained sliding windows incur practical speed and portability costs.

    Source: §1 · p. 2

  • Method

    BASA varies regular local-window partitions across DiT blocks, adds pooled global K/V memory, and routes selected spatial windows for video context.

    Source: §1 · p. 2

  • Results

    Across FLUX and Wan, BASA narrows the theoretical-to-measured acceleration gap while maintaining competitive generation quality, achieving over 90% of theoretical speedup on FLUX.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    BASA demonstrates the practicality of backend-compatible structured sparsity for efficient high-resolution generation.

    Source: §5 · p. 10

Abstract

from arXiv · show

Diffusion Transformers (DiTs) have achieved strong performance in image and video generation, but the quadratic complexity of full attention makes high-resolution generation computationally expensive. Window attention offers an efficient alternative, yet existing methods face a practical trade-off: partitioned window attention typically achieves computational efficiency consistent with its theoretical complexity. However, isolated windows block cross-window interaction, often introducing visible grid-like artifacts in the generated results. Fine-grained sliding-window attention effectively restores interactions across neighboring windows and improves visual quality. However, its irregular computation patterns create a substantial gap between theoretical and practical speedups and require specialized kernels tailored to each hardware backend. To tackle these challenges, we propose BASA, a backend-agnostic sparse attention, which brings the best of both worlds: visual quality and practical acceleration. Specifically, BASA replaces visual self-attention with shifted local-window attention. By introducing a structured window-shifting scheme across DiT blocks, we allow tokens divided by window boundaries in one layer to communicate in the following layers, thereby achieving global information exchange and eliminating window-induced visual artifacts. Notably, our design introduces no additional irregular operators or customized kernels, making it readily deployable on existing attention backends and closing the gap between theoretical sparsity and practical acceleration. Experiments demonstrate that BASA achieves measured speedups exceeding 90\% of the theoretical estimates on FLUX and delivers a 4.52$\times$ attention speedup on Wan while maintaining competitive generation quality.

1 INTRODUCTION

Dense self-attention makes high-resolution DiT generation costly, while existing window methods trade practical speed against cross-window continuity. BASA combines structured sparse attention with cross-window and global information exchange, and reports acceleration on FLUX and Wan while maintaining competitive generation quality.

1 INTRODUCTION

  • Dense self-attention's quadratic complexity makes high-resolution image and video generation increasingly costly.Partitioned windows can track theoretical savings in measured acceleration but block cross-boundary interaction, producing grid-like artifacts.Source: §1 · p. 2
  • Fine-grained sliding-window methods improve cross-boundary interaction but incur indexing and memory-access overhead and often need backend-specific kernels.These practical costs create a gap between theoretical and measured speedups.Source: §1 · p. 2
  • BASA varies regular local-window offsets across DiT blocks and denoising steps so tokens separated by one partition can interact under later partitions.Pooled global K/V memory adds low-cost scene context; video routing retrieves relevant uncompressed spatial windows.Source: §1 · p. 2
  • BASA targets cross-window communication with regular computation and backend portability, without customized sparse kernels.This design addresses the trade-off between hardware-efficient regular sparsity and effective cross-window interaction.Source: §1 · p. 2
  • BASA achieves over 90% of theoretical speedup on FLUX and 4.52× attention speedup on Wan while maintaining competitive generation quality.The evaluation also reports reduced FLUX attention latency and competitive temporal consistency.Source: §1 · pp. 2–3

2 RELATED WORK

Related work addresses high-resolution generation through training-free extrapolation and adaptation, and reduces attention costs with sparse patterns. BASA is positioned as backend-compatible structured sparsity for bridging theoretical savings and practical acceleration.

2.1 HIGH-RESOLUTION VISUAL GENERATION

  • Training-free approaches address high-resolution image extrapolation through trajectory fusion, receptive-field adaptation, refinement, feature fusion, patch conditioning, and resolution changes.Other work explores flow/DiT extrapolation and efficient adaptation for ultra-high-resolution generation.Source: §2.1 · p. 3
  • High-resolution video work explores hierarchical local–global attention, tiled attention, sliding windows, guidance, and native-4K generation.BASA instead focuses on backend-compatible structured sparsity to bridge theoretical savings and practical wall-clock acceleration.Source: §2.1 · p. 3

2.2 EFFICIENT SPARSE ATTENTION FOR DIFFUSION TRANSFORMERS

  • Sparse attention restricts enlarged visual-token interactions to local windows, neighborhoods, or blocks, while regular windows can be reorganized into compact dense attention calls.Fixed boundaries may disrupt spatial continuity, motivating finer-grained patterns that better preserve dense-attention behavior.Source: §2.2 · p. 3

A Overall DiT pipeline with BASA and module replacement

BASA's overview presents sparse attention as a combination of local and global visual interactions at reduced attention cost. It is used as a drop-in replacement for visual self-attention while the rest of the generation pipeline remains unchanged.

A Overall DiT pipeline with BASA and module replacement

  • BASA is applied as a drop-in replacement for visual self-attention in DiT blocks, leaving the remaining generation pipeline unchanged.The overview describes the framework as recovering local and global visual interactions under reduced attention cost.Source: Figure 3 · p. 4
  • Sliding, head-dependent, or content-dependent sparse patterns may require specialized execution, so FLOP reductions may not yield proportional wall-clock speedups.Source: A Overall DiT pipeline with BASA and… · p. 4
  • BASA combines regular local-window computation with structured cross-window and global information exchange.This design addresses the trade-off between fidelity and efficient execution on existing attention backends.Source: A Overall DiT pipeline with BASA and… · p. 4

3 METHOD

BASA combines shifted regular local windows with pooled global K/V memory and content-adaptive routing, then adapts sparse attention under a frozen dense teacher. Its shifts vary across layers and denoising steps so tokens separated by one partition can interact in later partitions.

3.1 SHIFTED WINDOW SPARSE ATTENTION

  • Interleaved offsets vary window boundaries across layers and denoising steps, allowing tokens separated in one partition to interact later and diversifying cross-window communication.The schedule interleaves candidate offsets before advancing them across denoising steps.Source: §3.1 · pp. 4–5

3.2 K/V-POOLED GLOBAL MEMORY

  • Spatially pooled K/V gives each local window inexpensive global context, but compression may lose instance detail and leave similar objects indistinguishable.For video, pooling is spatial only, retaining the temporal dimension.Source: §3.2 · p. 5

3.3 CONTENT-ADAPTIVE TRUE-WINDOW ROUTING

  • Content-adaptive routing adds selected original K/V windows to local windows and pooled global memory, restoring selected long-range details without full dense attention.For video, routing operates over spatial windows that retain the full temporal axis.Source: §3.3 · p. 6

3.4 DENSE-TEACHER SPARSE ADAPTATION

  • A frozen dense teacher supervises a sparse student, with only sparse-attention parameters optimized via flow-matching, prediction, and attention-output alignment losses.Aligning attention outputs rather than maps avoids materializing the dense attention matrix.Source: §3.4 · p. 6
  • Video adaptation adds scheduler timestep weighting, and inference discards the teacher to use only the sparse student.Source: §3.4 · p. 6

4 EXPERIMENTS

BASA is evaluated for image and video resolution extrapolation against sparse-attention baselines, with quality and efficiency measured under controlled generation settings. Results show strong attention speedups alongside competitive generation quality, while image ablations distinguish the contributions of shifted windows and pooled global memory.

4.1 EXPERIMENTAL SETTINGS

  • Evaluation uses FLUX.1-dev and Wan2.1-T2V-1.3B on one A100 GPU, with image and video extrapolation evaluated against dense references and sparse-attention baselines.Image tests use 1,000 COCO2014 captions and 20 denoising steps; video tests use 100 videos from 20 VBench prompts, 50 denoising steps, and a twofold increase in both spatial dimensions.Source: §4.1 · pp. 7–8

4.2 EMPIRICAL STUDIES

  • For image extrapolation, w = 16 is the balanced operating point, while w = 32 favors reference preservation; BASA-8 yields greater fidelity loss with little latency reduction.BASA-32 improves all reference-similarity metrics over Swin and distilled STA-32, while CLEAR has competitive fidelity but uses more attention computation.Source: §4.2 · p. 8
  • 4.52× attention speedup on Wan and 2.443× faster latency for BASA with w = 32 on FLUX show acceleration across video and image extrapolation.The video efficiency table reports attention-only latency after warm-up and speedup relative to dense Wan; the image result retains 97.60 CLIP-I and 95.95 DINO similarity.Source: §4.2 · p. 8, Table 3 · p. 8
  • On video extrapolation, p = (4, 4) yields 90.00% sparsity and 4.52× speedup at w = (10, 10), while p = (2, 2) reduces speedup to 1.99×.The smaller pooling factor retains more global K/V tokens, illustrating a context-efficiency trade-off.Source: §4.2 · p. 8, Table 3 · p. 8

4.2.1 VIDEO GENERATION QUALITY

  • The faster video setting achieves 4.52× speedup with only minor reductions in subject consistency, background consistency, and aesthetic quality; motion smoothness is almost unchanged.Imaging quality rises from 65.22% to 67.02%; CLEAR has strong VBench scores but only 1.05× measured speedup, while Swin and STA have other quality or speed trade-offs.Source: Table 4 · p. 9, §4.2.1 · p. 9

4.3 ABLATION STUDIES

  • In image ablations, full BASA has the strongest reference fidelity across six similarity metrics; removing pooled global memory weakens several metrics, while shifted windows still outperform isolated local windows.The reported changes include PSNR from 24.09 to 23.33 and FID from 20.97 to 24.05 after removing pooled memory; the two components have complementary roles.Source: Table 5 · p. 9, §4.3 · p. 9

4.2 EMPIRICAL STUDIES

  • FLOP reduction alone does not ensure practical acceleration: CLEAR, Swin, and STA have high theoretical sparsity but low MFU and limited wall-clock gains.BASA's measured performance keeps pace with its theoretical advantage.Source: §4.2 · p. 9

5 CONCLUSION

BASA combines regular local attention with cross-window and long-range information exchange to make high-resolution generation more efficient. Experiments on FLUX and Wan show narrower gaps between theoretical and measured acceleration while maintaining stable quality with reasonably sized windows.

5 CONCLUSION

  • BASA combines regular local attention with cross-window and long-range information exchange using standard Q/K/V projections and SDPA, without customized sparse kernels.This is presented as a backend-agnostic sparse attention framework.Source: §5 · p. 9
  • Experiments on FLUX and Wan show BASA narrows the gap between theoretical and measured acceleration for image and video generation while maintaining stable quality with reasonably sized windows.The authors describe these results as demonstrating the practicality of backend-compatible structured sparsity for efficient high-resolution generation.Source: §5 · p. 10

AI USE STATEMENT

The authors report using generative AI for language editing, limited code assistance, and visual references during figure design, while retaining responsibility for the final work.

AI USE STATEMENT

  • Generative AI was used for language editing and polishing, limited code drafting and editing, and visual references for workflow-figure layout.AI-generated visual references were used only as guidance and were not incorporated into the final figures.Source: AI USE STATEMENT · p. 10
  • The authors state that AI tools were not used to develop hypotheses, design the methodology, make scientific or experimental decisions, interpret results, or generate scientific claims.Source: AI USE STATEMENT · p. 10
  • The authors reviewed AI-assisted work, manually inspected and refined code, and accept responsibility for the final content and artifacts.Source: AI USE STATEMENT · p. 10

A.1 COMPLEXITY AND BACKEND EXECUTION

The appendix analyzes BASA's attention cost and explains how regular window execution supports practical acceleration on standard attention backends. Its implementation separates attention branches when needed while preserving the exact joint-softmax result.

A.1 COMPLEXITY AND BACKEND EXECUTION

  • BASA's pooled global memory remains substantially cheaper than dense attention for small routing counts and moderate pooling factors.For pooling factors of 4 in both dimensions, the global branch retains about 1/16 as many visual K/V tokens as dense attention; the local branch contributes M/N of the dense visual K/V range.Source: §A.1 · p. 13
  • The reported theoretical speedup includes shared Q/K/V and output-projection costs, which limit achievable full-MSA speedup, as well as BASA's reduced attention cost.The text also notes that pooling adds only a small regular reduction overhead relative to attention computation.Source: §A.1 · p. 13
  • BASA executes local windows as regular dense attention problems through standard SDPA, avoiding query-dependent sparse indexing and customized sparse-attention kernels.For divisible resolutions, cyclic shift, reshape, and permutation form non-overlapping windows that are folded into the batch dimension.Source: §A.1 · pp. 13–14
  • The local-window and text/global-memory branches can be evaluated separately while remaining mathematically equivalent to one softmax over concatenated K/V tokens.This avoids replicating shared global K/V context for every local window.Source: §A.1 · p. 14
  • Warm-up selects and caches the faster two-way or three-way attention decomposition for each static tensor shape, while shifts do not trigger repeated compilation.Source: §A.1 · p. 14
  • BASA's regular formulation narrows the gap between theoretical and measured acceleration, although reshape, pooling, branch-merging, kernel-launch overheads, and small-window utilization can prevent an exact match.Source: §A.1 · p. 14

A.2 IMPLEMENTATION DETAILS AND EXPERIMENTAL CONFIGURATION

The appendix specifies the adaptation, generation, and timing protocols used to evaluate BASA on FLUX and Wan. It also reports video ablations and provides qualitative examples of image and video resolution extrapolation.

A.2 IMPLEMENTATION DETAILS AND EXPERIMENTAL CONFIGURATION

  • Sparse models are adapted from dense pretrained models using teacher distillation; FLUX uses LoRA rank 128 and scaling 128 while the transformer, VAE, and text encoders remain frozen.The shifted partition index cycles across optimization steps so training exposes the student to inference-time shifts.Source: §A.2 · p. 14
  • For FLUX image extrapolation, 1,000 COCO2014 prompts use 1024 × 1024 dense source images that are resized to 2048 × 2048 for the high-resolution SDEdit stage.The source generation uses 20 denoising steps and guidance scale 3.5; the high-resolution stage uses 20 steps, SDEdit strength 0.7, and guidance scale 7.5.Source: §A.2 · p. 14
  • For Wan video extrapolation, 100 videos from 20 VBench prompts are enlarged from 832 × 480 to 1664 × 960 using the same source video, prompt, and random seed across methods.The protocol uses 81 frames, 15 fps, 50 denoising steps, guidance scale 5.0, and V2V denoising strength 0.7.Source: §A.2 · p. 15
  • Video BASA preserves the complete temporal extent of each selected spatial window; its full configuration enables pooled global memory and one routed true K/V window per query window.Unless otherwise specified, candidate-window radius is one and the routing configuration is shared across denoising steps.Source: §A.2 · p. 15
  • BASA is implemented at the attention-processor level using cyclic shift, reshape, permutation, standard SDPA, and regular spatial reduction, with LoRA weights merged for inference.The implementation leaves the surrounding FLUX or Wan pipeline interface unchanged.Source: §A.2 · p. 15
  • Latency measurements profile the warmed-up attention-processor call, including projections and attention but excluding the VAE, text encoders, scheduler, image I/O, and other pipeline components.The same hardware, precision, input shapes, and timing convention are used for comparisons; experiments were conducted on a single NVIDIA A100-80GB GPU.Source: §A.2 · p. 15
  • Removing shifted windows caused the largest reported video-ablation degradation: Aesthetic Quality fell from 56.87% to 50.76% and Dynamic Degree from 0.34 to 0.29.Removing pooled global memory reduced subject consistency, background consistency, and motion smoothness, while removing routing reduced Dynamic Degree from 0.34 to 0.29 and Imaging Quality from 67.02% to 65.94%.Source: §A.2 · p. 15, Table 6 · p. 16
  • The qualitative figures show BASA image extrapolation from 1K to 2K with FLUX and video extrapolation from 480 × 832 to 960 × 1664 with Wan2.1-1.3B.Source: Figure 5 · p. 16, Figure 6 · p. 17

A.3 PROMPTS FOR QUALITY SAMPLE GENERATION

The appendix collects varied scene prompts for high-resolution samples and qualitative comparisons, alongside examples of video and image resolution extrapolation.

A.3 PROMPTS FOR QUALITY SAMPLE GENERATION

  • The comparison prompt collection includes a motocross rider, stormy lighthouse, terraced rice fields, and other detailed scenes.The appendix separately labels prompts for quality ultra-resolution comparisons and for 1K-to-2K images.Source: §A.3 · pp. 18–19
  • Sample prompts span detailed scenes including a candlelit library, desert caravan, hummingbird, mountain, phoenix, and action or fantasy subjects.Several request high-resolution or 8K detail and specify lighting, texture, or composition.Source: §A.3 · p. 19
  • Figure 7 presents BASA video extrapolation from 480×832 to 960×1664 using Wan2.1-1.3B.The caption specifies a 10 × 10 latent window and pooling size 4.Source: Figure 7 · p. 20
  • Additional prompts cover nature, portraits, still life, fantasy, urban scenes, and dramatic landscapes.Examples include cherry blossoms, glassware, a robot tending a garden, a stormy cliff, and a steampunk airship.Source: §A.3 · pp. 17, 18, 20, 21
  • Figure 8 compares low-resolution inputs with BASA extrapolations and reports sharper, more coherent local structures while preserving overall image content.Orange boxes mark regions with insufficient fine detail or noticeable over-smoothing in the inputs.Source: Figure 8 · p. 21
Loading 2610.08772v1…