Source-linked AI summary
Crowd Counting using Deep Recurrent Spatial-Aware Network
Lingbo Liu, Hongjun Wang, Guanbin Li, Wanli Ouyang, Liang Lin
TL;DR
Unconstrained crowd counting is challenged by perspective-driven scale and rotation variation, while fixed multi-scale methods cannot fully address these conditions. The paper proposes a recurrent spatial-aware framework that adaptively transforms and refines attended density-map regions. It reports improvements of 12% on WorldExpo’10 and 22.8% on UCF CC 50 against existing best-performing methods.
Problem
Perspective causes large scale and rotation variation in unconstrained crowd scenes, while fixed multi-scale methods cannot cover all scales and ignore rotation variation.
Method
The Deep Recurrent Spatial-Aware Network recurrently selects regions with a Spatial Transformer Network and refines their density maps through residual learning.
Results
12% improvement on WorldExpo’10 and 22.8% improvement on UCF CC 50 were achieved compared with existing best-performing methods.
Takeaways & Limitations
The framework simultaneously models crowd-density variation and pose changes as a unified learnable approach for crowd-map refinement.
Takeaways & Limitations
Training requires all input images or patches to be transformed to the same resolutions because the model contains fully connected layers.
Abstract
from arXiv · showhide
Crowd counting from unconstrained scene images is a crucial task in many real-world applications like urban surveillance and management, but it is greatly challenged by the camera's perspective that causes huge appearance variations in people's scales and rotations. Conventional methods address such challenges by resorting to fixed multi-scale architectures that are often unable to cover the largely varied scales while ignoring the rotation variations. In this paper, we propose a unified neural network framework, named Deep Recurrent Spatial-Aware Network, which adaptively addresses the two issues in a learnable spatial transform module with a region-wise refinement process. Specifically, our framework incorporates a Recurrent Spatial-Aware Refinement (RSAR) module iteratively conducting two components: i) a Spatial Transformer Network that dynamically locates an attentional region from the crowd density map and transforms it to the suitable scale and rotation for optimal crowd estimation; ii) a Local Refinement Network that refines the density map of the attended region with residual learning. Extensive experiments on four challenging benchmarks show the effectiveness of our approach. Specifically, comparing with the existing best-performing methods, we achieve an improvement of 12% on the largest dataset WorldExpo'10 and 22.8% on the most challenging dataset UCF_CC_50.
1 Introduction
Crowd counting in unconstrained scenes is difficult because perspective, occlusion, lighting, scale, and rotation variations challenge fixed architectures. The proposed Deep Recurrent Spatial-Aware Network adaptively selects and refines regions to address these variations, improving benchmark performance.
- Crowd counting estimates the number of people in unconstrained scenes for applications including video surveillance and traffic monitoring.
- Perspective, occlusion, lighting, camera setting, scale, and rotation variations make unconstrained crowd counting challenging.
- The Deep Recurrent Spatial-Aware Network uses recurrent adaptive region selection and residual learning to refine an initial crowd density map.
- The spatial transform module adaptively handles scale and rotation variation in crowd counting.
- 12% improvement on WorldExpo’10 and 22.8% improvement on UCF CC 50 were reported against existing best-performing methods.
2 Related Work
Deep crowd-counting methods commonly use multi-scale architectures, but fixed receptive fields cannot cover all scale variation in diverse scenes. This work uses iterative spatial transformation within an LSTM framework for adaptive region selection.
- Deep learning Methods for Crowd Counting: Existing crowd-counting methods use multi-scale architectures to model variation in people’s scales and regional crowd density.
- Deep learning Methods for Crowd Counting: Fixed static receptive fields handle only limited scale variation and do not scale well to diverse large-scale changes.
- Deep learning Methods for Crowd Counting: A Spatial Transformer Network selects and transforms an input subregion using parameters for location, resize scale, and rotation angle.
- Deep learning Methods for Crowd Counting: The proposed approach is described as the first to use multiple Spatial Transformer iterations within an LSTM framework for crowd counting.
3 Proposed Method
The proposed network first estimates a global crowd density map, then recurrently selects and refines attended regions using spatial transformation, global context, and residual learning.
- The architecture combines a Global Feature Embedding module with a Recurrent Spatial-Aware Refinement module for crowd-density estimation.GFE extracts global features and produces the initial map; RSAR subsequently refines it.
- Global Feature Embedding: GFE uses three CNN columns with different kernel sizes and channel numbers, concatenating their outputs into a global feature for initial-map generation.A 1 × 1 convolution converts the global feature into the initial density map M0.
- Attentional Region Localization: RSAR uses an LSTM-driven spatial transformer to select, crop, translate, scale, and rotate a region from the preceding density map.The selected region is resized by bilinear interpolation before refinement.
- Region Density Map Refinement: The Local Refinement Network combines the attended region with global context and learns a residual density map for that region.The inverse spatial transformer maps the residual back before adding it to the preceding density map.
- Networks Optimization: Training requires images or patches with uniform resolution because the network contains fully connected layers, and optimizes the end-to-end loss with Adam.The training density-map annotations preserve crowd counts over the image up to negligible deviation.
4 Experiments
Experiments evaluate the method with MAE and MSE on four public benchmarks, compare it with prior methods, and use ablations to examine spatial transformation, global context, and recurrent refinement.
- Evaluations and Comparisons: The experiments compare the method with published state-of-the-art approaches on ShanghaiTech, UCF CC 50, MALL, and WorldExpo’10.The study also includes extensive ablation experiments on ShanghaiTech.
- Evaluation Metric: MAE and MSE evaluate crowd-counting accuracy, with estimated counts obtained by summing the estimated density map.MAE and MSE are computed over testing images using ground-truth and estimated counts.
- Evaluations and Comparisons: 49.7% lower MAE and 39.5% lower MSE are reported against CP-CNN on ShanghaiTech Part B.The method outperforms the competing methods on both ShanghaiTech parts.
- Evaluations and Comparisons: 22.8% improvement over ConvLSTM-nt is reported on UCF CC 50 using the MAE metric.The comparison uses several deep-learning state-of-the-art methods.
- Evaluations and Comparisons: The method achieves state-of-the-art average MAE across five WorldExpo’10 scenes and the lowest MAE in Scenes 2 and 5.Scenes 2 and 5 are identified as the two most challenging testing scenes.
- Ablation Study: Removing global context reduces performance across all metrics, while performance improves with more refinement iterations and is best at n = 30.Performance slightly drops by 40 steps, and the refined maps become more accurate after multi-step refinement.
5 Conclusion
The paper presents Deep Recurrent Spatial-Aware Network as a unified framework for crowd-map refinement, modeling crowd-density and pose variations. Experiments on four challenging benchmarks report superior performance over existing state-of-the-art methods.
- The proposed network simultaneously models crowd-density variations and pose changes within a unified learnable module.
- The framework can be regarded as a general approach for refining crowd maps.
- Experiments on four challenging benchmarks show superior performance compared with existing state-of-the-art methods.
- Future work will investigate incorporating the model into other crowd-flow prediction frameworks.