Source-linked AI summary
Revisiting Perspective Information for Efficient Crowd Counting
Miaojing Shi, Zhaohui Yang, Chao Xu, Qijun Chen
TL;DR
Perspective distortion causes drastic person-scale changes and makes density regression on small person areas difficult, while perspective information is often unavailable or costly to acquire. PACNN generates perspective maps and integrates them into multi-scale density regression, achieving efficient and effective crowd counting over state-of-the-art methods on standard benchmarks.
Problem
Perspective distortion causes drastic person-scale changes, making density regression on small person areas difficult, while perspective information is often unavailable or hard to acquire.
Method
PACNN generates ground-truth perspective maps for training and uses predicted multi-scale perspective maps as pixel-level weighting layers to combine multi-scale density outputs.
Results
PACNN demonstrates effectiveness and efficiency over state-of-the-art methods on ShanghaiTech, WorldExpo’10, UCF CC 50, and UCSD benchmarks.
Takeaways & Limitations
Perspective-aware weighting makes the combined density map robust to perspective distortion while retaining image-based inference efficiency.
Takeaways & Limitations
The generated perspective maps are not perfect, and retaining raw K-NN distances provides barely any significant benefit in experiments.
Abstract
from arXiv · showhide
Crowd counting is the task of estimating people numbers in crowd images. Modern crowd counting methods employ deep neural networks to estimate crowd counts via crowd density regressions. A major challenge of this task lies in the perspective distortion, which results in drastic person scale change in an image. Density regression on the small person area is in general very hard. In this work, we propose a perspective-aware convolutional neural network (PACNN) for efficient crowd counting, which integrates the perspective information into density regression to provide additional knowledge of the person scale change in an image. Ground truth perspective maps are firstly generated for training; PACNN is then specifically designed to predict multi-scale perspective maps, and encode them as perspective-aware weighting layers in the network to adaptively combine the outputs of multi-scale density maps. The weights are learned at every pixel of the maps such that the final density combination is robust to the perspective distortion. We conduct extensive experiments on the ShanghaiTech, WorldExpo'10, UCF_CC_50, and UCSD datasets, and demonstrate the effectiveness and efficiency of PACNN over the state-of-the-art.
1. Introduction
Crowd counting is challenged by perspective distortion, which changes person scale and makes density regression difficult, especially for small person areas. PACNN addresses this by predicting perspective maps and using them to adaptively combine multi-scale density outputs for efficient, robust counting.
- Perspective distortion causes drastic person-scale changes, making density regression on small person areas difficult.
- Perspective information is useful for modeling scale changes but is often unavailable because acquiring it requires camera parameters or scene geometry.
- Patch-based and multi-scale approaches address varying scales, but patch-based inference increases computational cost.
- PACNN generates ground-truth perspective maps from sampled person locations and trains the network to predict perspective maps for new images.
- PACNN uses predicted perspective maps in two weighting layers to adaptively combine three multi-scale density outputs pixel by pixel.
2. Related work
Crowd-counting research includes detection-based, regression-based, and CNN-based methods, with perspective information used in different ways. PACNN differs by predicting perspective maps directly and using them to guide multi-scale density combination.
- Traditional methods: Traditional detection-based methods identify individual pedestrians or parts, but dense crowds and complex backgrounds remain difficult.
- Traditional methods: Traditional regression-based methods extract image features and apply regression functions to estimate crowd counts.
- Perspective information: Perspective information was traditionally used to normalize regression features or detection results according to person-scale changes.
- Modern methods: Modern CNN methods commonly use multi-column or multi-scale architectures and patch-based schemes to handle varying head sizes, crowd densities, and contexts.
- Modern methods: In modern methods, perspective information is often implicit, whereas PACNN predicts perspective maps directly and uses them to adaptively combine multi-scale density outputs.
3. Perspective-aware CNN
The paper generates perspective maps from sampled person-scale cues and integrates predicted maps into multi-scale density regression. PACNN uses perspective-aware pixel-level weighting to adapt density estimates to perspective distortion.
- Ground-truth perspective generation: Perspective values represent the number of image pixels corresponding to one meter at each real-scene location.
- Ground-truth perspective generation: For dense crowds, sampled perspective values are inferred from observed head size using average distances to K-nearest neighbors.
- Ground-truth perspective generation: The method averages sampled perspective values by image row and fits them with a tanh function to reduce outlier influence.
- Ground-truth perspective generation: The fitted map decreases from bottom to top and is constant within each row, representing vertical person-scale change.
- Network architecture: Perspective-aware weights assign location-dependent contributions to the multi-scale density maps, unlike uniform averaging.
- Loss function and training: The network jointly optimizes density and perspective regressions using MSE and DSSIM losses for pixel accuracy and local pattern consistency.
4. Experiments
Experiments evaluate PACNN through ablations and comparisons across ShanghaiTech, UCF CC 50, WorldExpo’10, and UCSD. Perspective-aware weighting improves accuracy, while image-based inference provides a speed advantage over patch-based approaches.
- Ablation study: On ShanghaiTech, De1 performs better for the denser SHA split, whereas De2 performs better for the sparser SHB split, matching their intended scale specialization.The reported MAEs for De1, De2, and De3 are 81.8, 86.3, and 93.1 on SHA, and 16.0, 14.5, and 18.2 on SHB.
- Ablation study: PACNN’s perspective-aware weighting substantially outperforms direct averaging on ShanghaiTech, reaching MAE 66.3 and MSE 106.4 on SHA, and MAE 8.9 and MSE 13.5 on SHB.Without perspective maps, direct averaging reaches MAE 76.5 on SHA and 12.9 on SHB, compared with 81.8 and 14.5 for the respective single-scale baselines.
- ShanghaiTech comparison: PACNN achieves strong ShanghaiTech results with image-based inference, requiring 230ms for a 1024*768 input while patch-based methods can be approximately 5x slower.Combining PACNN with the trained backbone from [17] further reaches MAE 62.4 and MSE 102.0 on SHA, and MAE 7.6 and MSE 11.8 on SHB.
- UCF CC 50: On UCF CC 50, PACNN obtains MAE 267.9 and MSE 357.8, while PACNN + [17] achieves the lowest reported MAE 241.7 and MSE 320.7.Standalone PACNN does not attain the best standalone MAE or MSE, whose reported values are 258.4 and 320.9.
- WorldExpo’10: PACNN records the lowest mean MAE 7.8 over the five WorldExpo’10 scenes, although it does not outperform the state of the art in every individual scene.The evaluation trains and tests within each scene’s provided region of interest and averages scene-level MAE.
- UCSD: On UCSD, PACNN achieves the lowest reported MAE 0.89 and MSE 1.18, supporting performance across sparser crowd densities.The dataset exhibits uneven crowd distribution and strong person-scale changes caused by perspective distortion.
5. Conclusion
The paper proposes PACNN to estimate crowd counts using perspective-aware density regression. It generates ground-truth perspective maps for training and uses predicted maps to combine multi-scale density outputs, with experiments reporting efficiency and effectiveness over state-of-the-art methods.
- PACNN automatically estimates crowd counts while predicting perspective and density maps at test time.
- Ground-truth perspective maps are generated for training, and predicted maps guide two weighting layers that adaptively combine multi-scale density outputs.
- The combined density map is reported to be robust to perspective distortion in crowd images.
- Experiments on standard crowd-counting benchmarks report the proposed method’s efficiency and effectiveness over state-of-the-art methods.