PaperlayerView exact revision on arXiv

Source-linked AI summary

Perspective-Guided Convolution Networks for Crowd Counting

Zhaoyi Yan, Yuchen Yuan, Wangmeng Zuo, Xiao Tan, Yezhen Wang, Shilei Wen, Errui Ding

arXiv:1909.06966v1cs.CV

TL;DR

Crowd counting must handle dramatic perspective-driven scale variation, while existing multi-scale and multi-column approaches model only discrete scales. PGCNet uses perspective-guided spatially variant smoothing and a trainable perspective-estimation branch, achieving state-of-the-art performance across four benchmark datasets and introducing a large-scale dataset.

  • Problem

    Existing crowd-counting methods struggle with continuous intra-scene scale variation because multi-scale and multi-column architectures consider discrete representative scales.

  • Method

    PGCNet uses perspective-guided convolution for spatially variant feature smoothing and includes a perspective-estimation branch trainable with or without perspective annotations.

  • Results

    PGCNet achieves state-of-the-art performance on four benchmark datasets, including 57.0 MAE on ShanghaiTech Part A and 8.8 MAE on Part B.

  • Takeaways & Limitations

    Perspective information can guide scale-adaptive density-map estimation in a single-column crowd-counting network, while Crowd Surveillance expands available high-resolution data.

  • Takeaways & Limitations

    Accurate perspective estimation requires fair pre-training of PENet; without it, the guidance remains messy and significantly degrades performance.

Abstract

from arXiv · show

In this paper, we propose a novel perspective-guided convolution (PGC) for convolutional neural network (CNN) based crowd counting (i.e. PGCNet), which aims to overcome the dramatic intra-scene scale variations of people due to the perspective effect. While most state-of-the-arts adopt multi-scale or multi-column architectures to address such issue, they generally fail in modeling continuous scale variations since only discrete representative scales are considered. PGCNet, on the other hand, utilizes perspective information to guide the spatially variant smoothing of feature maps before feeding them to the successive convolutions. An effective perspective estimation branch is also introduced to PGCNet, which can be trained in either supervised setting or weakly-supervised setting when the branch has been pre-trained. Our PGCNet is single-column with moderate increase in computation, and extensive experimental results on four benchmark datasets show the improvements of our method against the state-of-the-arts. Additionally, we also introduce Crowd Surveillance, a large scale dataset for crowd counting that contains 13,000+ high-resolution images with challenging scenarios.

1. Introduction

Crowd counting is challenged by perspective-driven continuous scale variation, which fixed-receptive-field and discrete multi-scale methods handle imperfectly. PGCNet addresses this with perspective-guided feature smoothing, an adaptable perspective-estimation branch, and strong benchmark performance.

  • Motivation: Perspective-driven intra-scene scale variation is a major challenge for crowd counting in complex urban scenes.Crowd counting supports congested-scene analysis for crowd management, control, and security.
  • Limitations of existing methods: Multi-column and multi-scale methods consider discrete scales, can be difficult to train, and increase computation with additional columns or scales.The paper specifically notes high feature correlation as a training problem for MCNN.
  • Limitations of existing methods: CSRNet uses fixed receptive fields and performs relatively poorly at smaller or larger pedestrian scales.The paper motivates PGCNet as a single-column CNN for continuous scale variation using perspective information.
  • Proposed method: PGC uses perspective information to guide spatially variant feature-map smoothing before successive convolutions, adapting Gaussian-kernel size to pedestrian scale.Larger people receive larger smoothing kernels, while smaller people receive smaller ones.
  • Proposed method: The perspective-estimation branch supports supervised training and weakly supervised training when pre-trained.This addresses the limited availability of perspective annotations in existing datasets.
  • Results and contributions: 57.0 MAE on ShanghaiTech Part A and 8.8 MAE on ShanghaiTech Part B are reported as state-of-the-art results, alongside a new large-scale dataset.The contribution list also reports results across four benchmark datasets.

2. Related Work

CNN crowd-counting research has explored multi-scale, context, and multi-task strategies, while perspective information has usually been used during preprocessing. The paper instead targets continuous per-person scale variation by encoding perspective into the network architecture.

  • CNN-based methods: CNN-based crowd-counting methods include multi-scale, context, multi-task, and related architectures.Examples include MCNN, Switching-CNN, CP-CNN, SANet, and CSRNet.
  • CNN-based methods: Existing scale-handling methods commonly fuse features from discrete receptive-field sizes or assign discrete regressors.CSRNet enlarges receptive fields by stacking dilated convolutions rather than using multiple columns.
  • CNN-based methods: The proposed method handles continuous scale variation for each pedestrian instead of simply fusing features from different scales.Perspective information is treated as an estimator of pedestrian scale.
  • Perspective normalization: Perspective information has generally supported density-map preprocessing and is seldom directly encoded into the CNN architecture.PACNN remains based on a multi-column architecture with discrete scales.
  • Perspective normalization: The paper advocates explicit perspective normalization during network training to address continuous scale variation.This contrasts with approaches that combine perspective maps after processing.

3. Proposed Method

PGCNet uses perspective information to adapt feature smoothing and receptive fields continuously to people’s scale, then predicts density maps with successive convolutions. Its perspective estimation branch can be trained with or without perspective annotations.

  • Perspective-Guided Convolution: PGC uses spatially variant Gaussian filtering followed by spatially invariant convolution to adapt receptive fields to perspective-dependent scales.The two-stage design smooths feature maps spatially according to perspective before applying conventional convolution.
  • Perspective-Guided Convolution: Perspective values are normalized and transformed into a blur map with trainable parameters that determine the Gaussian smoothing scale.The normalization uses a sigmoid-like function with learned α and β, while the blur map uses additional trainable parameters.
  • Efficient Approximation: Candidate Gaussian filters are reduced with PCA into basis filters and coefficient maps for an efficient approximation of spatially variant smoothing.The approximation uses sampled filters, PCA eigenvectors, and coefficient maps before convolutional combination.
  • Perspective-Guided Convolution: PGC adaptively uses larger or smaller receptive fields for people at larger or smaller scales while appending conventional convolution to enforce scale consistency.The design targets continuous intra-scene scale variation rather than only discrete representative scales.
  • Perspective Estimation: PENet estimates perspective maps through an encoder-decoder branch that can be trained without corresponding perspective annotations after pretraining.The branch is incorporated into PGCNet for end-to-end or weakly supervised learning when perspective labels are unavailable.
  • Complete Architecture: PGCNet combines a backbone, PENet, and DMPNet, with DMPNet using backbone features and estimated perspective maps to produce the final density map.The DMPNet applies dilated convolution after spatially variant Gaussian smoothing within its PGC modules.

4. The Crowd Surveillance Dataset

Crowd Surveillance is introduced as a large-scale, high-resolution crowd-counting dataset designed to broaden data volume, count variation, image quality, and scenario difficulty. It contains 13,945 images and 386,513 marked people.

  • Dataset Scale: Crowd Surveillance contains 13,945 high-resolution images with 386,513 marked people.The dataset is described as nearly three times larger than the combination of four other datasets in Table 1.
  • Data Collection: The dataset was built from online image crawling and real-life surveillance video acquired from cooperative partners with necessary permissions.Its sources combine web imagery with surveillance-video acquisition.
  • Dataset Statistics: Crowd Surveillance exhibits high data volume and substantial crowd-count variance across images.Figure 3 presents the statistical histogram used to compare count distributions across datasets.
  • Dataset Comparison: Compared with ShanghaiTech Part A and WorldExpo’10, Crowd Surveillance offers the highest average resolution and more challenging scenarios with complicated backgrounds.The dataset is less crowded on average but emphasizes image quality and scenario complexity.

5. Experimental Results

Experiments evaluate PGCNet on four datasets, compare it with baselines, and examine implementation choices, backbone extensibility, and PENet pre-training. PGCNet achieves strong results, while pre-training remains important for reliable perspective guidance.

  • Evaluation setup: Experiments use ShanghaiTech, WorldExpo’10, UCF CC 50, and Crowd Surveillance, with MAE and MSE evaluating crowd counting and perspective estimation.For datasets without perspective annotations, Ours A uses estimated maps from a pre-trained PENet, whereas Ours B trains end-to-end without perspective map annotations.
  • Benchmark comparisons: 57.0 MAE on ShanghaiTech Part A and 8.8 MAE on Part B represent the reported state-of-the-art performance of PGCNet on these benchmarks.On ShanghaiTech, PGCNet achieves the best Part A result for both MAE and MSE and is slightly surpassed by SANet on Part B.
  • Benchmark comparisons: 8.1 average MAE is achieved by PGCNet on WorldExpo’10, the best result against the compared methods.The method uses the official ground-truth perspective map to guide PGC processing on this dataset.
  • Benchmark comparisons: 244.6 MAE is achieved by Ours B on UCF CC 50, a 14.8 improvement over Ours A, supporting the feasibility of end-to-end training without perspective annotations.Ours A also gains significantly over CSRNet* on both MAE and MSE.
  • Benchmark comparisons: On Crowd Surveillance, Ours A gains 2.1 MAE over the baseline and Ours B gains a further 0.5 MAE through end-to-end training.Crowd Surveillance contains 10,880 training images and 3,065 testing images in the reported experiment.
  • Ablation and architecture: K = 7 provides the best trade-off for a PGC block, requiring approximately 10ms while smaller and larger filters perform worse or cost more computation.Stacking five PGC blocks reaches peak values of 57.0 and 8.8 MAE on ShanghaiTech Parts A and B; additional blocks degrade performance through over-smoothing.
  • Ablation and architecture: Three PGC blocks improve truncated ResNet-101 by 19.9/7.6 MAE, reaching 89.7/18.6 MAE on ShanghaiTech Parts A/B.This result supports extending PGC beyond the VGG-16-based CSRNet* backbone.
  • Ablation and architecture: PENet pre-training is necessary for accurate perspective estimation because untrained outputs provide confusing guidance for spatially variant Gaussian smoothing.Results with PENet pre-training significantly outperform those without it on UCF CC 50 and Crowd Surveillance.

6. Conclusion

The paper presents PGCNet, which uses perspective-guided convolution to address continuous intra-scene scale variation in crowd counting. It also introduces a perspective estimation branch and the Crowd Surveillance dataset, with experiments showing superiority over state-of-the-art methods.

  • Conclusion: PGCNet uses perspective-guided convolution as an insertable module for handling continuous intra-scene scale variation.The network incorporates perspective information into its processing rather than relying only on discrete scale features.
  • Conclusion: A perspective estimation branch and learning strategy make PGCNet end-to-end trainable even without perspective map annotations.The conclusion identifies this branch as part of the complete PGCNet approach.
  • Conclusion: Crowd Surveillance is introduced as a large-scale dataset intended to promote crowd-counting research.The dataset is presented alongside the model contribution.
  • Conclusion: Experiments on four benchmark datasets show superiority of PGCNet against state-of-the-art methods.The conclusion summarizes the reported cross-dataset evaluation outcome.

Supplementary Material

The supplementary materials provide additional architectural, training, visualization, reliability, and density-map details for the perspective estimation network and PGCNet.

  • Supplementary contents: The supplementary materials describe PENet’s architecture and its three training phases.
  • Supplementary contents: They include visualizations of estimated perspective maps from each PENet training phase.
  • Supplementary contents: They report additional analysis of PENet prediction reliability.
  • Supplementary contents: They provide more density maps predicted by PGCNet.

A. The Architecture of PENet and the Training Details

PENet uses an encoder–decoder architecture and is trained in three phases: reconstruction, perspective-map prediction from RGB images, and integration into end-to-end PGCNet training.

  • Architecture: PENet uses Convolution-LeakyReLU encoder blocks and UpConv-ReLU decoder blocks, with encoder features downsampled by a factor of two per block.The architecture is documented in Table 11.
  • Training phases: In phase one, PENet reconstructs perspective maps at one-eighth of the original image resolution.This resolution matches the perspective input required by the PGC block.
  • Training phases: In phase two, the decoder is fixed while the encoder learns to construct perspective maps from corresponding RGB images.This stage produces 0.101 MAE and 0.142 MSE.
  • Training phases: In phase three, PENet is either used to provide estimated-map guidance directly or embedded as an end-to-end perspective-estimation branch.

B. The Visualization of Estimated Perspective Maps in Each Phase of Training PENet

Visualizations indicate that PENet reconstructs perspective maps effectively, predicts roughly accurate maps across scenes, and functions in both direct-estimation and end-to-end settings.

  • Phase-wise visualization: The first-phase visualizations compare the input or ground truth with PENet’s reconstructed perspective map.The reconstruction is described as performing well.
  • Phase-wise visualization: The second-phase examples show roughly accurate perspective maps across different scenes, indicating robustness in the reported visualizations.
  • Phase-wise visualization: Together with quantitative results, the first two phases show that PENet predicts meaningful perspective maps both quantitatively and qualitatively.
  • Phase-wise visualization: In phase three, images with larger visual angles correspond to directly estimated maps containing more large values.This visual relationship is reported for the compared examples.
  • Phase-wise visualization: PENet works both for direct perspective-map prediction and as the perspective estimator in the end-to-end architecture.

C. Reliability of the Prediction of PENet

The reliability experiment compares ground-truth and estimated perspective guidance on three crowd-counting benchmarks, reporting only small MAE decreases when estimated maps are used.

  • Reliability evaluation: The experiment evaluates ground-truth versus estimated perspective maps as guidance for spatially variant smoothing on ShanghaiTech Part A/B and WorldExpo’10.The comparison is summarized in Table 10.
  • Reliability evaluation: Using estimated guidance yields MAEs of 58.1, 9.0, and 8.3, with decreases of 1.1, 0.2, and 0.2, respectively.The values correspond to ShanghaiTech Part A, ShanghaiTech Part B, and WorldExpo’10 in the reported order.
  • Reliability evaluation: The experiment is motivated by the absence of perspective annotations in the intended deployment setting.

D. More Density Maps Predicted by the Proposed PGCNet

The paper presents visual comparisons of perspective-map and density-map outputs, including PENet phase results and CSRNet-versus-PGCNet density estimations.

  • Density-map visualizations: The reported visual comparison indicates that PGCNet estimates pedestrian counts more accurately than CSRNet in sparse and congested scenes.
  • PENet visualizations: Figures 8 and 9 visualize PENet’s first- and second-phase results, comparing inputs, reconstructions or outputs, and corresponding RGB images or ground truth.
  • PENet visualizations: Figure 10 compares perspective maps from direct estimation and end-to-end training across example RGB images.
  • PENet visualizations: Table 11 documents the PENet architecture used for the perspective-estimation branch.
  • Density-map visualizations: Figures 11 and 12 compare density-map estimations produced by CSRNet and PGCNet.
Loading 1909.06966v1…