Source-linked AI summary
ADCrowdNet: An Attention-injective Deformable Convolutional Network for Crowd Understanding
Ning Liu, Yongchao Long, Changqing Zou, Qun Niu, Li Pan, Hefeng Wu
TL;DR
Accurate crowd understanding remains difficult in highly congested, noisy scenes because existing methods can degrade under noise, occlusion, and diverse distributions. ADCrowdNet addresses this with attention-based crowd priors and multi-scale deformable convolution, reporting lower errors across crowd and vehicle-counting datasets. Its remaining limitation is incomplete recognition in severely occluded regions.
Problem
Existing DNN-based crowd methods can lose accuracy in highly congested noisy scenes affected by background noise, occlusions, and non-uniform crowd distributions.
Method
ADCrowdNet concatenates an Attention Map Generator that identifies crowd regions and congestion with a Density Map Estimator using multi-scale deformable convolutions.
Results
ADCrowdNet reports lower MAE than CSRNet across five crowd-counting datasets and the TRANCOS vehicle-counting dataset, with improvements ranging from 3.0% to 32.8%.
Takeaways & Limitations
The framework is reported to improve crowd counting and density-map estimation across four crowd datasets and one vehicle-counting dataset.
Takeaways & Limitations
In a severely occluded example, AMG fails to highlight the whole crowd region, so ADCrowdNet recognizes only part of it.
Abstract
from arXiv · showhide
We propose an attention-injective deformable convolutional network called ADCrowdNet for crowd understanding that can address the accuracy degradation problem of highly congested noisy scenes. ADCrowdNet contains two concatenated networks. An attention-aware network called Attention Map Generator (AMG) first detects crowd regions in images and computes the congestion degree of these regions. Based on detected crowd regions and congestion priors, a multi-scale deformable network called Density Map Estimator (DME) then generates high-quality density maps. With the attention-aware training scheme and multi-scale deformable convolutional scheme, the proposed ADCrowdNet achieves the capability of being more effective to capture the crowd features and more resistant to various noises. We have evaluated our method on four popular crowd counting datasets (ShanghaiTech, UCF_CC_50, WorldEXPO'10, and UCSD) and an extra vehicle counting dataset TRANCOS, and our approach beats existing state-of-the-art approaches on all of these datasets.
1. Introduction
Crowd understanding requires accurate density maps and counting in congested, noisy scenes, where existing deep models can degrade. ADCrowdNet combines attention and multi-scale deformable convolution, and reports improvements across crowd and vehicle-counting benchmarks.
- Highly congested noisy scenes challenge accurate crowd density estimation and counting because of background noise, occlusions, and diverse crowd distributions.
- Existing DNN-based methods can suffer accuracy degradation in highly congested noisy scenes despite earlier progress.CSRNet is described as being affected by background noises, occlusions, and non-uniform crowd distributions.
- ADCrowdNet combines a visual attention mechanism for noise suppression with a multi-scale deformable convolution scheme for congested environments.The attention mechanism focuses computation on pertinent crowd regions, while deformable convolution models complex geometric transformations and diverse distributions.
- AMG first generates crowd-region and congestion priors, then DME uses them to estimate density maps through a two-stage architecture.The attention map generator supplies candidate crowd regions and fine-grained congestion context for the density map estimator.
- 3.0%, 18.8%, 3.0%, 13.9%, and 5.1% lower MAE than CSRNet are reported on ShanghaiTech Part A, Part B, UCF CC 50, WorldExpo10, and UCSD, respectively.On TRANCOS, ADCrowdNet achieves 32.8% lower MAE than CSRNet.
2. Related Work
Related work progresses from detection and regression toward CNN-based density-map estimation, motivated by the difficulty of handling congested scenes and preserving spatial information.
- Detection-based crowd methods use sliding windows and handcrafted features, but their assumptions limit applicability in more challenging congested scenes.
- Regression-based methods predict object counts or densities, with count regression discarding object-location information during learning.Density regression instead estimates pixel-level density whose regional integral gives the object count.
- CNN-based approaches learn crowd counts or density maps from images using architectures such as layered, end-to-end, dual-column, and multi-column networks.
- CP-CNN and CSRNet use multi-level context and dilated convolution, respectively, and are described as state-of-the-art approaches.
3. Attention-Injective Deformable Convolutional Network
ADCrowdNet concatenates an attention map generator with a density map estimator. AMG highlights crowd regions and congestion, while DME uses multi-scale deformable convolutions to handle occlusion, distribution variation, and perspective distortion.
- Architecture: ADCrowdNet trains AMG and DME as concatenated modules: AMG generates attention maps, and DME estimates density maps from attention-weighted images.AMG is trained with crowd and background images before DME is trained on the pixel-wise product of the input and attention map.
- Attention map: An attention map is an image-sized weight map with higher values in crowd regions, generated from AMG’s two-category classification features.
- Attention map: AMG fuses class-specific feature maps using confidence scores, upsamples the result, and normalizes it to form the attention map.The class feature maps are Fc and Fb, and their confidence scores are Pc and Pb.
- Attention map: The attention map indicates regional congestion: higher values correspond to more congested crowds, and its pixel-wise product with the image feeds DME.
- AMG: AMG uses VGG-16 convolutional layers and multi-rate dilated convolutions to localize people clusters with enlarged receptive fields.
- DME: DME combines a VGG-16 convolutional front end with a multi-scale deformable-convolution back end to address occlusion, crowd diversity, and perspective distortion.
- Deformable convolution: Deformable convolution learns offsets that adjust sampling locations during training instead of using uniform sampling.This dynamic sampling scheme is presented as suitable for congested noisy crowd scenes.
4.1. Datasets and Settings
ADCrowdNet is evaluated on four crowd-counting datasets spanning congested, noisy, extremely congested, and sparse scenes, using counting and density-map quality metrics.
- Datasets: The evaluation covers ShanghaiTech, UCF CC 50, WorldExpo’10, and UCSD crowd-counting datasets.The datasets include different crowd densities and background conditions.
- Datasets: ShanghaiTech contains 1,198 images divided into congested Part A and sparse Part B subsets.Part A has 300 training and 182 testing images, while Part B has 400 training and 316 testing images.
- Datasets: UCF CC 50 contains 50 images with 94–4543 persons per image and uses five-fold cross-validation.Its limited image count and wide variation in person counts make it challenging.
- Datasets: WorldExpo’10 contains 3980 images from five scenes, with 3380 for training and 600 for testing, and provides ROI regions.
- Datasets: UCSD contains 2000 sparse-scene images, uses ROI regions, and enlarges images to 952×632 before training.800 images are used for training and the remainder for testing.
- Evaluation metrics: MAE and MSE evaluate counting accuracy, while PSNR and SSIM evaluate generated density-map quality after resizing maps and ground truth to the original input size.
4.2. Training
Training uses positive and negative samples for AMG and augmented image patches for DME, with Adam optimization at a learning rate of 1e-5.
- AMG training: AMG training uses positive samples from four crowd-counting datasets and 650 Internet-sourced negative background images.The negative images include outdoor scenes where people may appear, distinguishing samples primarily by whether the image contains people.
- Optimization: Adam is used for training with a learning rate of 1e-5.The text specifies this setting for both AMG and DME training.
- DME training: Each DME training image produces nine quarter-sized patches, including four non-overlapping quarters and five random crops.
- DME training: Mirroring the cropped patches doubles the DME training dataset.
- DME training: DME ground truth follows the procedure in [17], and Euclidean distance measures the difference between generated density maps and ground truth.
4.3. Results and Analyses
The experiments compare ADCrowdNet variants with prior methods across crowd and vehicle-counting datasets, emphasizing noisy and congested scenes. Results generally favor attention-aware, deformable designs, while also revealing reduced effectiveness when attention erases already-defined regions or misses severe occlusions.
- Alternative study: AMG-DME outperformed DME on ShanghaiTech Part A, Part B, and WorldExpo’10, especially for noisy examples.In one ShanghaiTech Part A example, AMG-DME estimated 198 people versus the ground truth 171, while DME mistook distant trees for people.
- Alternative study: AMG-DME had approximate performance to DME on UCF CC 50, with slightly higher AME but lower MSE.The passage attributes this pattern to extremely congested scenes with relatively little background noise.
- Quantitative results: On ShanghaiTech, ADCrowdNet variants AMG-DME and AMG-bAttn-DME performed better than existing approaches, while DME was generally close to CSRNet.The comparison used estimation errors on ShanghaiTech and UCF CC 50.
- Quantitative results: ADCrowdNet achieved the best average accuracy on WorldExpo’10, while DME achieved the best UCSD accuracy for both MAE and MSE.The WorldExpo’10 result covered scenes 1, 4, and 5; the comparison included Switching-CNN, MCNN, and CSRNet.
- Quantitative results: ADCrowdNet outperformed CSRNet on all five datasets for density-map PSNR and SSIM, including 7.03% higher PSNR and 55.76% higher SSIM on UCF CC 50.On UCSD, the reported improvements were 31.81% for PSNR and 8.13% for SSIM.
- Quantitative results: ADCrowdNet achieved the best performance at all GAME levels on the TRANCOS vehicle-counting dataset.TRANCOS evaluation used Grid Average Mean Absolute Error to measure counting accuracy.
5. Conclusion
ADCrowdNet combines multi-scale deformable convolutional layers with an attention-aware training scheme for congested, noisy crowd understanding. Across four crowd-counting datasets and TRANCOS, it achieved significant improvements over recent state-of-the-art approaches.
- Conclusion: ADCrowdNet generally achieved more accurate crowd counting and density-map estimation than existing methods in highly congested noisy environments.The conclusion attributes this scope to suppressing problems from noises, occlusions, and diversified crowd distributions.
- Conclusion: ADCrowdNet achieved significant improvements over recent state-of-the-art approaches on ShanghaiTech, UCF CC 50, WorldEXPO’10, UCSD, and TRANCOS.The first four datasets evaluate crowd counting, while TRANCOS evaluates vehicle counting.
- Conclusion: The proposed architecture is designed for crowd understanding of congested noisy scenes.Its stated components are multi-scale deformable convolutional layers and an attention-aware training scheme.