Source-linked AI summary
BLASST: Dynamic BLocked Attention Sparsity via Softmax Thresholding
Jiayi Yuan, Cameron Shinn, Kai Xu, Jingze Cui, George Klimiashvili, Guangxuan Xiao, Perkz Zheng, Bo Li, Yuxin Zhou, Zhouhai Ye, Weijie You, Tian Zheng, Dominic Brown, Pengbo Wang, Markus Hoehnerbach, Richard Cai, Julien Demouth, John D. Owens, Xia Hu, Song Han, Timmy Liu, Huizi Mao
TL;DR
Long-context inference faces computational and memory bottlenecks because attention retains quadratic cost. BLASST reuses online softmax statistics to skip negligible attention blocks, and its optimized kernels report speedups in both prefill and decode. Its prefill design currently retains value-block loads because memory bandwidth is not the bottleneck in the targeted regime.
Problem
Attention’s O(n^2) operations and memory accesses make long-context deployment challenging, while optimized FlashAttention still computes the full attention matrix.
Source: §1 · pp. 1–2Method
BLASST dynamically prunes negligible attention blocks by reusing online softmax statistics, without training or pre-computation.
Source: §1 · p. 2Results
BLASST achieves up to 1.52x speedup for prefill at 71.9% sparsity and 1.48x speedup for decode at 73.2% sparsity over FlashAttention baselines.
Source: §1 · p. 2Takeaways & Limitations
BLASST substantially accelerates attention with minimal accuracy degradation and supports both prefill and decode in inference frameworks.
Source: §6 · p. 11The current prefill design retains value-block loads because prefill is compute-bound in its targeted regime; skipping them may suit future memory-bandwidth-bound workloads.
Source: §4.1 · p. 6
Abstract
from arXiv · showhide
The growing demand for long-context inference capabilities in Large Language Models (LLMs) has intensified the computational and memory bottlenecks inherent to the self-attention mechanism. To address this challenge, we introduce BLASST, a drop-in, dynamic sparse attention mechanism that accelerates inference by using only a fixed scalar threshold to skip attention blocks. Our method targets practical inference deployment by removing the barriers to adoption present in existing works. As such, BLASST eliminates training requirements, avoids expensive pre-computation passes, accelerates both prefill and decode across all major attention variants (MHA, GQA, MQA, and MLA), provides optimized support for modern hardware, and easily integrates into existing frameworks. This is achieved by reusing online softmax statistics to identify negligible attention scores, skipping softmax, value block loads, and the subsequent matrix multiplication. We demonstrate the BLASST algorithm by delivering optimized kernels with negligible latency overhead. Our automated threshold calibration procedure reveals a simple inverse relationship between optimal threshold and context length, meaning we require only a single threshold each for prefill and decode per model. Preserving benchmark accuracy, we demonstrate a 1.52x speedup for prefill at 71.9% sparsity and a 1.48x speedup for decode at 73.2% sparsity on modern GPUs.
1 INTRODUCTION
Long-context inference is constrained by attention’s quadratic cost, while existing sparse methods can add deployment barriers. BLASST dynamically skips negligible attention blocks using online softmax statistics, with optimized kernels and threshold calibration aimed at practical inference.
1 INTRODUCTION
- Attention requires O(n^2) operations and memory accesses, and FlashAttention still computes the full attention matrix despite optimizing memory bandwidth.Longer context windows make this bottleneck more severe.Source: §1 · pp. 1–2
- BLASST skips blocks whose local maximum is sufficiently below the running row maximum, avoiding softmax exponentiation, value-block loads, and attention–value multiplication.The pruning decision reuses online softmax statistics and requires one comparison per block with a scalar threshold.Source: Figure 1 · p. 1, §1 · p. 2
- BLASST targets adoption barriers in sparse attention with a training-free method that needs no pre-computation and integrates through a single scalar threshold input.The introduction identifies training, pre-computation, phase coverage, modern-GPU support, and framework integration as adoption hurdles.Source: §1 · p. 2
- BLASST achieves up to 1.52x speedup for prefill at 71.9% sparsity and 1.48x speedup for decode at 73.2% sparsity over FlashAttention baselines.The kernels support MHA, MQA, GQA, and MLA while maintaining numerical stability.Source: §1 · p. 2
- Optimized CUDA kernels target compute-bound prefill and memory-bound decode, reusing computed statistics to keep block-skipping overhead minimal.The kernels are designed for Blackwell and Hopper GPUs.Source: §1 · p. 2
- Automated calibration and sparsity-aware training extend deployment flexibility, with calibration finding an inverse threshold–context-length relationship and training improving the accuracy–sparsity frontier.The contribution list also identifies optimized kernels for both phases and availability in TensorRT-LLM and FlashInfer.Source: §1 · p. 2
2 RELATED WORKS
Related work spans compute-focused sparse attention, memory-focused token and KV-cache methods, and alternative attention architectures. BLASST is positioned as a training-free approach that reduces compute and memory accesses across prefill and decode without costly pre-computation.
2.1 Compute-Optimized Sparsity
- Compute-optimized methods include static local or block patterns, retrieval-head methods, dynamic selection, and training-aided gates, each with distinct overhead or training trade-offs.The cited examples include Sparse Transformer [Child et al., 2019], MInference, and SeerAttention [Gao et al., 2025b].Source: §2.1 · p. 3
- BLASST differs from SpargeAttention by supporting both prefill and decode with specialized kernels, direct skip decisions, decode-time value-load skipping, automated calibration, and sparsity-aware training.SpargeAttention targets prefill and uses a separate prediction step.Source: §2.1 · p. 3
2.2 Memory-Optimized Sparsity
- Token/KV methods reduce decode-time memory accesses through query-conditioned token selection, KV-cache compression, or persistent patterns; BLASST instead reduces compute and memory accesses in both phases while remaining training-free.Examples include H2O, Quest [Tang et al., 2024], and TidalDecode.Source: §2.2 · p. 3
2.3 New Attention Variants
- Alternative attention mechanisms include sliding-window, linear or gated attention, and state-space models; BLASST integrates with FlashAttention implementations without proxy scores or complex pre-computation.The passage contrasts BLASST with approaches that often require architectural changes and training.Source: §2.3 · p. 3
3 METHODOLOGY
BLASST dynamically skips attention blocks whose scores are negligible, reusing online-softmax statistics to avoid computing their contribution. The section describes threshold calibration, compatibility with attention variants, and an optional sparsity-aware training extension.
3.1.1 Key Insight
- BLASST skips blocks whose local maximum falls sufficiently below the running maximum, using the condition (block max − running max) < ln(λ) as a block-level proxy.This approximates score importance relative to the unknown global maximum and avoids computing blocks expected to contribute negligibly after softmax.Source: §3.1 · p. 3, §3.1.1 · p. 4
3.1.2 Algorithm Design
- The modified FlashAttention pass tiles query and KV blocks and skips blocks failing the threshold test, avoiding softmax-weight work, value-block loads, and matrix multiplication.These omissions save CUDA-core instructions, tensor-core computation in prefill, and memory bandwidth in decode.Source: §3.1.2 · p. 4
3.2 Calibration for Optimal Sparsity
- Threshold calibration targets fixed sparsity because accuracy degradation is consistent by sparsity level, while a fixed threshold yields different sparsity across context lengths.Accuracy remains stable to about 60–70% sparsity, then drops sharply; 75% sparsity requires λ ≈1e−4 at 8K and 1e−5 at 64K.Source: §3.2 · pp. 4–5, Figure 2 · p. 5
- Calibration measures sparsity for candidate thresholds across samples, fits λ · L = α · exp(β · s), and uses λ = α · exp(β · S)/L for a target sparsity.The fitted scale factor yields an inverse relationship between threshold and context length, and the target sparsity can be changed at runtime without recalibration.Source: §3.2 · pp. 4–5
3.3 Extensibility to Attention Variants
- Because BLASST relies on tiled online softmax, it is compatible with attention variants including MLA, where it can reduce computation and memory accesses despite a compute-bound decode bottleneck.The passage describes compatibility through online-softmax use within MLA’s latent space.Source: §3.3 · p. 5
3.4 Sparsity-Aware Training
- The sparsity-aware training extension applies BLASST during fine-tuning, so skipped blocks receive no gradients and the model adapts to sparse attention without architectural changes or auxiliary losses.It is presented as an extension to improve the accuracy-sparsity trade-off.Source: §3.4 · p. 5
4 KERNEL DESIGN
BLASST reuses online-softmax statistics already maintained by FlashAttention to make block-skip decisions with minimal kernel changes and negligible decision overhead. Its kernels specialize for compute-bound prefill and memory-bound decode.
4 KERNEL DESIGN
- BLASST reuses each thread’s local and running maxima from online softmax for block decisions, with a few added instructions whose latency is hidden behind existing operations.The decision coordinates a per-thread threshold predicate, warp vote, and shared-memory atomic across the softmax warpgroup.Source: §4 · pp. 5–6
4.1 Prefill Kernel: Compute-Bound Optimization
- Prefill skips softmax and attention-value multiplication for negligible blocks while still computing QK products and loading Value blocks from HBM.The design targets compute-bound prefill; Value loading is retained because bandwidth is not the bottleneck and conditional loads would cost more than they save.Source: §4.1 · p. 6
4.2 Decode Kernel: Memory-Bound Optimization
- Decode skips Value loads for pruned blocks, then batches consecutive query-key products so conditional loads do not serialize the pipeline.The described schedule reduces Value-load completion time from 38 to 31 time units; it maintains B shared-memory buffers for score tiles.Source: §4.2 · pp. 6–7
- For compute-bound MLA decode, BLASST also skips softmax operations for pruned blocks, adding savings beyond reduced memory traffic.The decode optimization is otherwise focused on skipping memory-intensive Value-matrix loads.Source: §4.2 · pp. 6–7
5 EXPERIMENTS
Across long-context and reasoning benchmarks, BLASST generally retains accuracy at substantial sparsity, while optimized kernels accelerate prefill and decode. Calibration and sparsity-aware training further address deployment across context lengths and accuracy-sparsity trade-offs.
5.1 Experimental Setup
- The evaluation covers two 8B models, long-context benchmarks and reasoning tasks, with greedy decoding for long-context tasks and best-of-N sampling for reasoning tasks.The reasoning evaluation uses 10 samples per problem for MATH500, GPQA, and LiveCodeBench and 20 for AIME 2024.Source: §5.1 · pp. 7–8
5.2 Main Results
- At ∼75% sparsity, BLASST shows minimal accuracy degradation; at 50% sparsity, Qwen3-8B scores 96.23 vs. 95.87 on MATH500 and 76.50 vs. 75.00 on AIME 2024 against dense.Table 2 covers Llama-3.1-8B and Qwen3-8B across prefill-only, decode-only, and combined optimization scenarios.Source: §5.2 · p. 8, Table 2 · p. 8
5.3 GPU Kernel Performance
- At approximately 70% sparsity on Blackwell, BLASST delivers 1.52× prefill and 1.48× decode speedup; on Hopper, prefill reaches 1.52× at 71.0% sparsity.At 0% sparsity, performance is 0.96–1.00× baseline, with no significant performance degradation.Source: §5.3 · pp. 8–9
5.4 Calibration Results
- For a 50% target, calibrated λ = a/L keeps observed sparsity within an average 1.2% of target across context lengths, while fixed thresholds range from 23% at 4K to 75% at 64K.The calibrated parameter a also transfers across task types without heavy retuning.Source: §5.4 · p. 9, Table 6 · p. 10
5.5 Sparsity-Aware Training Results
- At 50–75% target sparsity, sparsity-aware training reduces accuracy degradation by up to 1.7× compared with training-free sparsity application on RULER.At low sparsity, sparse-trained models slightly outperform the dense baseline; the authors report that training concentrates information in high-scoring blocks.Source: §5.5 · pp. 9–10, Figure 6 · p. 10
5.6 Ablation Studies
- Despite substantial variation in sparsity across layers and heads, one shared threshold automatically prunes more concentrated attention more aggressively and preserves more diffuse attention.This adaptation does not require explicit top-k selection or head pruning.Source: §5.6 · p. 10, Figure 7 · p. 11
- BLASST composes with prefill-optimized XAttention and KV-cache compression such as RocketKV, with minimal accuracy degradation reported for the XAttention-plus-BLASST combination.The authors describe the methods as largely orthogonal and BLASST as a flexible component in sparse-attention pipelines.Source: §5.6 · p. 10, Table 7 · p. 11
- At 200K tokens, BLASST reaches ∼58% prefill sparsity on RepoQA with minimal accuracy drop, while its accuracy degradation is more stable than XAttention’s at 70–90% sparsity on RULER.Applying sparsity in both prefill and decode provides additional computational savings with negligible incremental cost; tile-row reordering has dataset-dependent but generally negligible effects.Source: §5.6 · pp. 10–11
6 CONCLUSION
BLASST dynamically prunes attention computations using online-softmax statistics and is presented as a practical sparse-attention method for inference. The conclusion emphasizes deployment flexibility and reports more stable accuracy degradation than XAttention at high sparsity.
6 CONCLUSION
- BLASST dynamically prunes attention computations by reusing online-softmax statistics and requires no pre-computation or training.It supports both prefill and decode and is integrated into multiple inference frameworks.Source: §6 · p. 11
- On RULER-16K for Qwen3-8B at high achieved sparsity levels, BLASST shows more stable accuracy degradation than XAttention and maintains better accuracy at aggressive sparsity settings.The passage attributes this effectiveness to using actual softmax statistics rather than proxy-based importance scores.Source: Figure 8 · p. 11
- The conclusion presents BLASST as a practical foundation for efficient long-context transformers, with calibration and sparsity-aware training described as further sources of robustness and flexibility.The paper identifies hardware-aware sparse patterns, learned sparsity, and adaptive hybrid methods as future directions.Source: §6 · p. 11
A.1 Large Model Evaluations
BLASST retains accuracy at high sparsity in large-model long-context evaluations and shows near-baseline accuracy with MLA at 60% sparsity.
A.1 Large Model Evaluations
- The large-model evaluations cover long-context summarization and retrieval tasks.The results are reported for Qwen3-30B-A3B-Instruct and Llama-3.1-70B-Instruct.Source: §A.1 · p. 15
- BLASST maintains accuracy comparable to the dense baseline on LongBench as sparsity reaches 70%, and retains > 97% accuracy on needle-in-a-haystack tasks up to 80%.These evaluations use Qwen3-30B-A3B-Instruct and Llama-3.1-70B-Instruct, respectively.Source: Table 9 · p. 15, Table 10 · p. 15, §A.1 · p. 15
A.2 MLA Compatibility
- DeepSeek-R1 NVFP4 maintains near-baseline accuracy on GPQA Diamond, MMLU Pro, and LiveCodeBench at 60% sparsity with MLA.The paper identifies this result as evidence of compatibility with Multi-Head Latent Attention.Source: §A.2 · p. 15, Table 11 · p. 15
A.3 Calibration Stability Across Datasets
Calibration transfers across diverse tasks: achieved sparsity remains near the 50% target, although retrieval-focused decode tasks require lower calibrated values of a.
A.3 Calibration Stability Across Datasets
- At a 50% target, achieved sparsity stays close to 50% across datasets, supporting one calibration on a diverse mixed sample for deployment across workloads.The table reports separate prefill and decode results; the per-dataset breakdown is illustrative rather than the recommended calibration procedure.Source: §A.3 · p. 15, Table 12 · p. 15
- For decode, niah single and qa yield lower a values because retrieval-focused decoding attends narrowly to relevant spans.Prefill datasets maintain similar achieved sparsity levels.Source: §A.3 · p. 15
A.4 Tile Row Reordering
BLASST supports tile-row reordering with negligible overhead; its observed effect varies by dataset, while accuracy at a given sparsity changes negligibly.
A.4 Tile Row Reordering
- Processing local-window tiles first can populate the running maximum with high-scoring local and sink tokens earlier, informing later skip decisions.The paper attributes the local-window and sink-token pattern to StreamingLLM (Xiao et al., 2024b).Source: §A.4 · p. 15
- Tile-row reordering is supported at the kernel-scheduling level with negligible overhead.Source: §A.4 · p. 15
- Reordering gives similar performance on VT and noticeable improvements on FWE, while its impact on accuracy at a given sparsity is negligible.The comparison is between sequential Standard Cummax and reverse-order Reordered Cummax processing.Source: §A.4 · p. 16, Figure 9 · p. 16
B ERROR BOUND ANALYSIS
The appendix bounds the approximation error from skipped attention blocks and describes an artifact for reproducing kernel benchmarks across specified hardware and software environments.
B ERROR BOUND ANALYSIS
- The skip criterion bounds each skipped block’s unnormalized attention mass, and the output error is the total contribution of skipped tokens, with a δ^2Vmax renormalization correction negligible when δ ≪ 1.The analysis defines Vmax as the maximum value-vector norm and S as the set of skipped blocks.Source: B ERROR BOUND ANALYSIS · p. 16
C.1 Abstract
- The artifact benchmarks custom kernels against a state-of-the-art baseline across prefill and decode, sweeping thresholds and measuring sparsity, execution time, bandwidth, and speedups.The evaluation targets NVIDIA H200 and B200 architectures.Source: §C.1 · p. 16
C.2 Artifact check-list (meta-information)
- The checklist identifies BLASST Skip-Softmax, public code under Apache 2.0, TensorRT-LLM and FlashInfer workflows, closed binaries for sparsity measurement, and single-GPU kernel benchmarks.Source: §C.2 · pp. 16–17
C.3.2 Hardware dependencies
- Hardware evaluation requires a many-core x86 host, an H200 or B200 GPU depending on the kernels, and about 100 GB of SSD storage.Source: §C.3.2 · p. 17
C.3.3 Software dependencies
- The artifact requires Docker with NVIDIA Container Toolkit or Singularity, plus a compatible Linux host and drivers for Hopper or Blackwell.Startup scripts use the official TensorRT-LLM release container.Source: §C.3.3 · p. 17
C.3.4 Data sets
- Core benchmarks use randomly initialized tensors, and closed sm100 binaries collect exact sparsity percentages dynamically during execution.Source: §C.3.4 · p. 17
C.4 Installation
- Installation begins by cloning the repository with submodules and launching its container startup script, with Singularity as a fallback when Docker is unavailable.Source: §C.4 · p. 17
C.5 Experiment workflow
The evaluation workflow is organized by hardware architecture and, for Hopper, by attention phase. README instructions compile the kernels and launch automated benchmarks that sweep threshold scales and compare BLASST with dense SOTA baselines.
C.5 Experiment workflow
- Hopper evaluations use separate prefill and decode directories, while Blackwell combines both phases in one directory.Source: §C.5 · p. 17
- Users select the architecture and phase subdirectory, then follow its README to compile kernels and start automated benchmarks.Source: §C.5 · p. 17
- The scripts sweep threshold scale factors and benchmark both BLASST kernels and dense SOTA baselines.Source: §C.5 · p. 17
- Measurements of sparsity, execution time, and memory bandwidth are logged to standard output.Source: §C.5 · p. 17
C.6 Evaluation and expected result
- Expected outputs are documented in each repository subdirectory's README and are intended to align with Table 5.Source: §C.6 · p. 17