Source-linked AI summary
Boosting Crowd Counting via Multifaceted Attention
Hui Lin, Zhiheng Ma, Rongrong Ji, Yaowei Wang, Xiaopeng Hong
TL;DR
Crowd counting is challenged by large-scale variation that fixed-size CNN kernels and transformer attention do not handle well. MAN combines global transformer attention with learnable local and instance attention, achieving state-of-the-art performance on four crowd-counting datasets. Its design improves local spatial encoding while its training mechanisms address attention distribution and label-noise effects.
Problem
Large-scale variations challenge fixed-size CNN kernels and fixed-size transformer attention in crowd images, limiting local spatial relation encoding.
Method
MAN combines vanilla transformer global attention, Learnable Region Attention, Local Attention Regularization, and Instance Attention Loss for crowd counting.
Results
MAN achieves state-of-the-art performance on ShanghaiTech, UCF-QNRF, JHU++, and NWPU.
Takeaways & Limitations
Learnable local attention improves transformer spatial encoding for scale-varying crowds, while training mechanisms reduce the influence of label noise.
Takeaways & Limitations
A region-attention filter mechanism can make each exclusive region depend on only two discrete points, limiting learnability and flexibility.
Abstract
from arXiv · showhide
This paper focuses on the challenging crowd counting task. As large-scale variations often exist within crowd images, neither fixed-size convolution kernel of CNN nor fixed-size attention of recent vision transformers can well handle this kind of variation. To address this problem, we propose a Multifaceted Attention Network (MAN) to improve transformer models in local spatial relation encoding. MAN incorporates global attention from a vanilla transformer, learnable local attention, and instance attention into a counting model. Firstly, the local Learnable Region Attention (LRA) is proposed to assign attention exclusively for each feature location dynamically. Secondly, we design the Local Attention Regularization to supervise the training of LRA by minimizing the deviation among the attention for different feature locations. Finally, we provide an Instance Attention mechanism to focus on the most important instances dynamically during training. Extensive experiments on four challenging crowd counting datasets namely ShanghaiTech, UCF-QNRF, JHU++, and NWPU have validated the proposed method. Codes: https://github.com/LoraLinH/Boosting-Crowd-Counting-via-Multifaceted-Attention.
1. Introduction
Crowd counting must handle large-scale variation, but fixed-size CNN kernels and transformer attention struggle with local spatial structure. MAN combines learnable local attention, regularization, and instance attention to improve counting across four datasets.
- Large-scale variations from camera viewing angle and 2D projection limit fixed-kernel CNN crowd counting.
- Transformer crowd-counting models commonly use fixed-size attention, which inadequately encodes local structure under large-scale variation.
- MAN combines global transformer attention, learnable local attention, and instance attention in a crowd-counting model.
- Learnable Region Attention dynamically selects an exclusive local region for each feature location, extracting relevant local information across scale changes.
- Local Attention Regularization supervises LRA by penalizing deviations among attention distributions, while Instance Attention dynamically focuses training on important instances.
- Experiments on ShanghaiTech, UCF-QNRF, JHU++, and NWPU report a solid advance in counting performance.
2. Related Works
Related work spans detection, regression, point-supervised counting, transformers, and variable attention. MAN extends variable attention with learnable regions designed for scale variation in crowd images.
- Detection methods count predicted person boxes but are limited by occlusion in congested areas and the need for additional annotations.
- Regression methods estimate counts from pseudo-density maps, while later approaches use point supervision to avoid inaccurate pseudo-map generation.
- Transformer vision methods combine global interactions with local mechanisms such as windows, blocked attention, focal attention, and sliding windows.
- MAN extends variable attention into 2D Learnable Region Attention, removing fixed local-window constraints and targeting scale robustness.
3. The Proposed Method
MAN combines global transformer attention with learnable local region attention, local attention regularization, and instance attention to encode spatial relations across crowd scales and noisy annotations.
- MAN consists of Learnable Region Attention, Local Attention Regularization, and Instance Attention Loss.
- Global self-attention captures pairwise feature relations but is position-agnostic, motivating local attention for spatial information.
- Learnable Region Attention: LRA dynamically selects a suitable local region for each feature location instead of using fixed-size windows, supporting scale variations.
- Learnable Region Attention: Differentiable parameter arrays make the learned region maps trainable and more flexible than regions determined by two discrete vertices.
- Local Attention Regularization: LRA combines with global attention in the complete attention module, while Local Attention Regularization penalizes deviations among local attention allocations.
- Instance Attention: Instance Attention dynamically selects or weights instances during training to reduce the influence of noisy point annotations.
4. Experiments
Experiments evaluate MAN on four crowd-counting datasets using MAE and MSE, with ablations, attention visualizations, and running-cost comparisons. MAN improves counting performance across benchmarks, while the full combination of modules performs best and remains lightweight compared with vanilla transformers.
- Datasets and Evaluation Metrics: MAN is evaluated on ShanghaiTech, UCF-QNRF, JHU-Crowd++, and NWPU-CROWD using MAE and MSE, where lower values indicate better performance.MAE emphasizes accuracy, while MSE emphasizes robustness.
- Counting Accuracy: MAN improves the second-best method’s MAE and MSE from 80.6 to 77.3 and from 139.8 to 131.5, respectively.On JHU++, the corresponding values improve from 59.4 to 53.4 and from 244.0 to 209.9.
- Counting Accuracy: Compared with BL, MAN improves MAE and MSE by 9.6% and 11.3% on ShanghaiTech A, 12.9% and 15.1% on UCF-QNRF, 28.8% and 30.0% on JHU-Crowd++, and 27.4% and 28.9% on NWPU-CROWD.The improvements are reported for both evaluation metrics across all four datasets.
- Ablation Studies: The full model achieves its best UCF-QNRF performance with LAR, improving BL by 12.9% for MAE and 15.1% for MSE.Using LRA without global attention decreases performance, indicating the importance of both global and local information.
- Effect of δ: Instance Attention Loss performs best at δ = 0.9, while smaller δ values reduce accuracy and δ values between 0.8 and 1 outperform supervision using all annotations.At δ = 0.9, the model cuts off 10% of annotations with the largest prediction deviations.
- Visualizations of LRA: LRA visualizations show narrower attention regions for smaller-scale crowds, while the number of people attended to in each region remains approximately similar.The visualization attributes balanced attention allocation to Local Attention Regularization.
- Running Cost Evaluation: MAN has model size and inference time close to VGG19+Trans and much smaller than ViT-B, with only a marginal FLOPs difference from VGG19-Trans.The comparison uses FLOPs on 384 × 384 inputs and inference time on 1024 × 1024 images.
5. Conclusion
The paper enhances transformer-based crowd counting with multifaceted attention and reports state-of-the-art performance on four crowd-counting datasets. Its components include learnable region attention, regularized local attention, and instance attention, with only a small computational-cost increase.
- Contributions: Multifaceted Attention Network improves transformer spatial local-context encoding for crowd counting through Learnable Region Attention.LRA is designed as a structural contribution to the transformer model.
- Efficiency: The computational cost of MAN only increases a little compared with the referenced vanilla-encoder configurations.Table 3 compares model size, FLOPs, and inference time.
- Contributions: Local Attention Regularization balances attention across proposed regions, while Instance Attention Loss reduces the influence of label noise during training.Together, these mechanisms modify the training pipeline alongside the architectural change.
- Conclusion: The proposed method achieves state-of-the-art performance on four crowd-counting datasets and is intended for broader vision-task applications.The paper identifies wider vision tasks as a future direction.