Source-linked AI summary
Fast Vision Transformers with HiLo Attention
Zizheng Pan, Jianfei Cai, Bohan Zhuang
TL;DR
Efficient ViT design is limited by the gap between FLOPs and actual throughput on target hardware. This paper introduces LITv2 and HiLo attention, which separates local high-frequency from global low-frequency processing; HiLo and LITv2 achieve competitive performance with faster, more efficient execution, while the head-splitting ratio remains task-specific.
Problem
FLOPs are an indirect efficiency metric that can diverge from actual ViT speed on target platforms, complicating speed–accuracy design under limited computational budgets.
Method
LITv2 uses HiLo attention, splitting heads between local-window high-frequency attention and global low-frequency attention over average-pooled keys and values.
Results
LITv2 achieves competitive performance on ImageNet and downstream tasks, while HiLo is faster than existing attention mechanisms across FLOPs, throughput and memory consumption.
Takeaways & Limitations
HiLo makes ViT backbones more practical for low-latency vision applications by improving efficiency especially on high-resolution images.
Takeaways & Limitations
The high/low-frequency head-splitting ratio is selected by ImageNet grid search, but its optimal value is task-specific and must be set manually.
Abstract
from arXiv · showhide
Vision Transformers (ViTs) have triggered the most recent and significant breakthroughs in computer vision. Their efficient designs are mostly guided by the indirect metric of computational complexity, i.e., FLOPs, which however has a clear gap with the direct metric such as throughput. Thus, we propose to use the direct speed evaluation on the target platform as the design principle for efficient ViTs. Particularly, we introduce LITv2, a simple and effective ViT which performs favourably against the existing state-of-the-art methods across a spectrum of different model sizes with faster speed. At the core of LITv2 is a novel self-attention mechanism, which we dub HiLo. HiLo is inspired by the insight that high frequencies in an image capture local fine details and low frequencies focus on global structures, whereas a multi-head self-attention layer neglects the characteristic of different frequencies. Therefore, we propose to disentangle the high/low frequency patterns in an attention layer by separating the heads into two groups, where one group encodes high frequencies via self-attention within each local window, and another group encodes low frequencies by performing global attention between the average-pooled low-frequency keys and values from each window and each query position in the input feature map. Benefiting from the efficient design for both groups, we show that HiLo is superior to the existing attention mechanisms by comprehensively benchmarking FLOPs, speed and memory consumption on GPUs and CPUs. For example, HiLo is 1.4x faster than spatial reduction attention and 1.6x faster than local window attention on CPUs. Powered by HiLo, LITv2 serves as a strong backbone for mainstream vision tasks including image classification, dense detection and segmentation. Code is available at https://github.com/ziplab/LITv2.
1 Introduction
Efficient ViT design should prioritize measured throughput alongside FLOPs because theoretical complexity can diverge from real hardware speed. LITv2 addresses this gap with HiLo attention, which separates local high-frequency and global low-frequency processing for efficient vision tasks.
- Motivation: FLOPs can misrepresent deployment speed because hardware-unfriendly operations and memory access costs slow some ViTs despite low theoretical complexity.Focal-Tiny is much slower than Swin-Ti on GPUs although their FLOPs are comparable.
- LITv2: LITv2 evaluates ViTs using throughput and combines competitive accuracy with faster practical speed on GPUs.The architecture is evaluated on ImageNet, COCO and ADE20K across classification, detection and segmentation.
- HiLo Attention: HiLo separates attention heads into high-frequency and low-frequency paths to capture local fine details and global structures efficiently.Hi-Fi uses local-window self-attention, while Lo-Fi uses global attention over down-sampled feature maps.
- HiLo Attention: The Hi-Fi path assigns a few heads to local-window self-attention, while the Lo-Fi path average-pools each window before modeling global relationships.This design matches high-frequency details with local attention and low-frequency structures with down-sampled global attention.
- Architecture Modifications: Replacing LITv1’s fixed relative positional encoding with depthwise convolution improves dense-prediction efficiency while adding receptive field through zero-padded filters.The convolutions incorporate implicitly learned positional information and enlarge the receptive field of early MLP blocks.
2 Related Work
Related work improves ViTs through architectural changes, efficient attention, and frequency-domain processing. HiLo differs by combining local and global self-attention to model high- and low-frequency relationships together.
- Vision Transformers: Vision Transformers have been improved with convolutional layers, pyramid feature maps, locality enhancements, and neural architecture search.These directions target stronger representations and better-performing architectures for vision tasks.
- Efficient Attention Mechanisms: Efficient attention methods reduce standard MSA’s quadratic complexity through low-rank, kernelized, memory-based, sparsity, spatial-reduction, or local-window mechanisms.Existing computer-vision methods often focus on either local or global attention at the same layer.
- Frequency Domain Analysis: Frequency-domain methods associate low frequencies with global structures and color information, and high frequencies with fine object details such as sharp edges.HiLo applies this distinction to attention by capturing both local and global relationships through self-attention.
3 Background
Standard ViTs use multi-head self-attention within Transformer blocks, but early-stage attention and high-resolution processing create efficiency bottlenecks. These limitations motivate efficient alternatives for pyramid features and dense prediction.
- Multi-head Self-Attention: A standard MSA layer transforms input tokens into query, key, and value matrices for each attention head.The input sequence has N tokens and D hidden dimensions, with learnable projections for each head.
- Multi-head Self-Attention: The outputs of all self-attention heads are concatenated and passed through a linear projection.The projection combines the head-specific representations into the layer output.
- Multi-head Self-Attention: Standard MSA has computational cost 4ND^2 + 2N^2D, with the quadratic term arising from attention over all tokens.The 3ND^2 and ND^2 terms arise from the input projections and output projection, respectively.
- Transformer Blocks: A standard vision Transformer block combines an MSA layer with a position-wise FFN inside residual LayerNorm-based updates.The FFN uses two fully connected layers with GELU nonlinearity.
- Bottlenecks of LITv1: LITv1 removes early MSA layers because they focus on local patterns, but later standard MSAs remain a speed bottleneck on high-resolution feature maps.The limitation is especially relevant to dense prediction tasks.
4 Method
HiLo separates attention into high- and low-frequency branches, using local attention for fine details and pooled-key/value global attention for broader dependencies. Its head splitting and window-size choices reduce computational cost, while LITv2 further modifies positional encoding and feed-forward layers for efficiency.
- HiLo Attention: Hi-Fi captures fine-grained local details with non-overlapping local-window self-attention instead of global attention.The design is intended to reduce computational complexity and improve hardware friendliness.
- HiLo Attention: Lo-Fi applies average pooling within each window, then uses global attention from original queries to reduced keys and values.Spatially reducing keys and values lowers the complexity of both attention computations.
- HiLo Attention: HiLo splits the original MSA heads between Hi-Fi and Lo-Fi using split ratio α, avoiding the cost of duplicating all heads.The split also decomposes the output projection into smaller matrices, reducing model parameters.
- HiLo Attention: A larger window size improves HiLo efficiency on high-resolution images by reducing FLOPs.The paper presents this as a practical guideline for adopting HiLo in existing frameworks.
- HiLo Attention: HiLo concatenates the outputs of the Hi-Fi and Lo-Fi branches to form the attention output.This is the operational combination of the two frequency-specific attentions.
- LITv2 Architecture: LITv2 removes relative positional encodings, adds a 3 × 3 zero-padded depthwise convolution to each FFN, and replaces attention layers with HiLo.The convolution incorporates position information and enlarges the receptive field of early MLP blocks.
5 Experiment
Experiments evaluate LITv2 across ImageNet-1K, COCO, and ADE20K, measuring accuracy, efficiency, throughput, and memory. Results show strong performance and speed, while ablations clarify HiLo’s frequency separation, α, window size, and architectural choices.
- Experimental Setup: Experiments cover image classification on ImageNet-1K, object detection and instance segmentation on COCO, and semantic segmentation on ADE20K.Performance is measured with Top-1 accuracy, AP, and mIoU, respectively.
- COCO Detection and Instance Segmentation: LITv2 achieves the best AP among compared methods on COCO with compelling fast inference speed.Using a larger HiLo window improves efficiency with a slight performance drop.
- ADE20K Semantic Segmentation: LITv2-S, LITv2-M, and LITv2-B surpass Swin-Ti, Swin-S, and Swin-B by 2.8%, 0.5%, and 1.2% mIoU, respectively, with higher FPS.The comparison uses ADE20K semantic segmentation results.
- Effect of α: With α = 0.9, HiLo achieves the best performance among tested α values while allocating more heads to Lo-Fi and reducing FLOPs.The pure Hi-Fi setting performs badly, while high-frequency signals remain important for fine object details in dense prediction.
- HiLo Analysis: Hi-Fi captures more high-frequency components, while Lo-Fi mainly captures low-frequency components, supporting HiLo’s intended frequency disentanglement.The frequency magnitudes are visualized using FFT outputs from the two attention branches.
- Throughput and Attention Ablations: HiLo is consistently faster than many attention mechanisms on CPUs and GPUs, including 1.4× faster than SRA and 1.6× faster than local window attention on CPUs.The broader benchmark compares throughput across multiple attention mechanisms.
6 Conclusion and Future Work
The paper concludes that LITv2 is a fast vision Transformer backbone with competitive performance, while HiLo improves efficiency across multiple resource measures. It identifies task extension and architectural refinements as future directions.
- LITv2 is a fast vision Transformer backbone that outperforms most state-of-the-art models on ImageNet and downstream tasks.
- HiLo provides efficiency advantages over existing attention mechanisms in FLOPs, throughput, and memory consumption, especially on high-resolution images.
- Future work may explore convolutional stems, overlapping patch embeddings, and applications of HiLo to speech recognition and video processing.
- HiLo’s head-splitting ratio is manually set using an ImageNet grid search, although its optimal value may differ across tasks.
Appendix
The appendix organizes supplementary analyses and documents LITv2’s architecture specifications. It defines the principal architectural notation and distinguishes modified ConvFFN blocks from earlier MLP blocks.
- Supplementary organization: The supplementary material covers architecture specifications, HiLo computational-cost derivations, window-size studies, pooled-query experiments, and throughput benchmarks.
- Architecture specifications: Table I specifies LITv2 using patch size, channel dimension, attention heads, HiLo split ratio, window size, FFN expansion ratio, and token-merging notation.
- Architecture specifications: ConvFFN Block denotes the modified early-stage feed-forward blocks, distinguishing them from the earlier MLP Blocks in LITv1.
A Architecture Specifications of LITv2
LITv2 retains LITv1’s stage depth and width while using HiLo selectively across stages. Its computational cost is decomposed into projection, attention/value aggregation, and output-projection components.
- Architecture: LITv2 generally keeps the same network depth and width as LITv1 while excluding multi-head self-attention from its first two stages.
- Architecture: At the last stage, α = 1.0 and s = 1 make HiLo behave as standard multi-head self-attention.
- Computational cost: The computational-cost analysis treats an HiLo layer as having N tokens, D hidden dimensions, and window size s.
- Computational cost: Each attention branch’s cost is divided into Q/K/V projections, attention and value aggregation, and the final linear projection.
C Effect of Window Size
The window-size study finds that window size does not change HiLo’s parameter count, while performance and efficiency depend on resolution and padding overhead. Window size 2 is the default for image classification.
- Parameterization: HiLo’s parameter count does not depend on window size.
- Small-resolution efficiency: At 224×224 image-classification resolution, window sizes have comparable FLOPs, speed, and memory, with differences mainly from window-partition padding.
- Image-classification setting: Window size 2 performs best on CIFAR-100 and is therefore used as LITv2’s default setting for image classification.
- Large-resolution setting: A larger window, such as 4, can improve efficiency at larger resolutions with a slight performance drop.
- Pooled-query alternative: Computing Lo-Fi queries from pooled feature maps changes the output spatial size, requiring interpolation to restore consistency with the original input.
E Details of Throughput Benchmark for Different Attention Mechanisms
The throughput benchmark evaluates attention mechanisms under standardized ViT-B conditions and compares their practical efficiency on CPUs and GPUs. HiLo uses fewer FLOPs and achieves higher speed than competing methods at similar parameter counts.
- Benchmark setup: The benchmark uses 224×224 inputs, producing 14×14 feature maps at 1/16 scale, with 12 heads of 64 dimensions each.These configurations are applied to all compared methods for a fair comparison.
- Benchmark setup: HiLo uses a window size of 2 and alpha of 0.9 for 1/16-scale feature maps, while other methods use their default settings.The comparison accounts for method-specific hyperparameters.
- Results: HiLo uses fewer FLOPs and runs faster on both CPUs and GPUs than compared methods with a similar number of parameters.The result is reported for a single attention layer under the benchmark settings.
F More Visualisations on Spectrum Analysis
The spectrum visualisations compare Hi-Fi and Lo-Fi outputs using frequency magnitudes averaged across samples. Hi-Fi emphasizes high frequencies, whereas Lo-Fi mainly emphasizes low frequencies.
- Frequency analysis: Hi-Fi captures more high-frequency content in LITv2, while Lo-Fi mainly focuses on low-frequency content.The paper presents this as evidence that the two attention paths disentangle frequency patterns.
- Visualisation procedure: The visualisation procedure computes a shifted logarithmic Fourier magnitude for each output feature map and averages it over samples.The output feature maps are treated as tensors with batch, channel, height, and width dimensions.
- Hi-Fi visualisation: Figure II shows 14×14 frequency magnitudes from eight Hi-Fi output channels averaged over 100 samples.The visualisation corresponds to Hi-Fi outputs in LITv2-S.
- Lo-Fi visualisation: Figure III shows 14×14 frequency magnitudes from eight Lo-Fi output channels averaged over 100 samples.The visualisation corresponds to Lo-Fi outputs in LITv2-S.