PaperlayerView exact revision on arXiv

Source-linked AI summary

HA-CCN: Hierarchical Attention-based Crowd Counting Network

Vishwanath A. Sindagi, Vishal M. Patel

arXiv:1907.10255v1cs.CV

TL;DR

Crowd counting remains difficult in highly congested scenes, and CNN-based models generalize poorly to new datasets without costly annotations. HA-CCN uses hierarchical spatial and global attention, then adapts the pretrained network with image-level-label weak supervision. The resulting framework reports improved counting and cross-dataset performance while reducing reliance on point-wise annotations.

  • Problem

    Crowd counting is challenging in congested scenes, while CNN-based models have poor generalization to new datasets and typically require expensive point-wise annotations for adaptation.

  • Method

    HA-CCN enhances VGG16 features with spatial and global attention, and adapts the pretrained counting network using pseudo-ground truth generated from image-level labels.

  • Results

    The proposed framework achieves significant improvements across challenging datasets and provides faster inference than patch-based testing methods.

  • Takeaways & Limitations

    Weakly supervised image-level-label adaptation reduces reliance on labor-intensive point-wise annotations for newer datasets and scenes.

Abstract

from arXiv · show

Single image-based crowd counting has recently witnessed increased focus, but many leading methods are far from optimal, especially in highly congested scenes. In this paper, we present Hierarchical Attention-based Crowd Counting Network (HA-CCN) that employs attention mechanisms at various levels to selectively enhance the features of the network. The proposed method, which is based on the VGG16 network, consists of a spatial attention module (SAM) and a set of global attention modules (GAM). SAM enhances low-level features in the network by infusing spatial segmentation information, whereas the GAM focuses on enhancing channel-wise information in the higher level layers. The proposed method is a single-step training framework, simple to implement and achieves state-of-the-art results on different datasets. Furthermore, we extend the proposed counting network by introducing a novel set-up to adapt the network to different scenes and datasets via weak supervision using image-level labels. This new set up reduces the burden of acquiring labour intensive point-wise annotations for new datasets while improving the cross-dataset performance.

I. INTRODUCTION

HA-CCN addresses the difficulty of crowd counting in congested, variable scenes and the poor cross-dataset generalization of CNN-based models. It combines hierarchical attention with weakly supervised target-dataset adaptation using image-level labels.

  • Crowd counting is difficult because scenes exhibit perspective distortion, scale variation, occlusion, illumination changes, clutter, and nonuniform person distributions.
  • HA-CCN selectively enhances VGG16 features through spatial attention in lower layers and global attention in higher layers.The spatial module infuses segmentation information, while global attention augments channel-wise information.
  • Weakly supervised adaptation fine-tunes a source-pretrained counting network on target datasets using image-level labels instead of point-wise annotations.The source dataset uses point-wise annotations, while the target dataset uses image-level labels for fine-tuning.
  • The proposed design uses attention at multiple VGG16 levels to improve multi-scale feature concatenation.
  • The paper presents the attention-based network and weakly supervised adaptation as its main contributions.

II. RELATED WORK

Prior work spans CNN-based crowd counting, attention mechanisms, and weak supervision, but the paper distinguishes HA-CCN through selectively placed, mask-guided attention and image-level-label adaptation.

  • Crowd Counting: CNN-based crowd-counting methods improve robustness through multi-column networks, regressor selection, context modeling, and other specialized designs.
  • Attention mechanisms: Attention has been applied across vision tasks including captioning, visual question answering, pose estimation, classification, detection, and fine-grained recognition.
  • Attention mechanisms: HA-CCN places a mask-guided spatial module after conv3 and global attention modules after conv4 and conv5.The spatial module is explicitly supervised with foreground/background masks, and the selective placement avoids unnecessary over-parameterization.
  • Weak Supervision: Weak supervision has been less explored for crowd counting, with earlier approaches requiring model-specific adaptation or labor-intensive region-level count annotations.
  • Weak Supervision: The proposed weakly supervised setup generates pseudo-ground truth from image-level labels and uses it to fine-tune the counting network.Unlike related semi-supervised approaches, it generates pseudo-ground truth from weak image-level labels.

III. HIERARCHICAL ATTENTION FOR CROWD COUNTING

HA-CCN addresses the difficulty of fusing multi-scale VGG16 features by enhancing different layers with hierarchical attention before producing a density map. The processed conv3, conv4, and conv5 features are concatenated and trained against ground-truth density maps.

  • III. HIERARCHICAL ATTENTION FOR CROWD COUNTING: HA-CCN enhances VGG16 features at different levels before multi-scale concatenation to improve feature effectiveness.The design uses conv3, conv4, and conv5 features with spatial or channel-wise enhancement.
  • III. HIERARCHICAL ATTENTION FOR CROWD COUNTING: SAM enhances conv3 features, while GAMs perform channel-wise enhancement on conv4 and conv5 features.The enhanced higher-level features are passed through convolutional blocks and upsampling before fusion.
  • III. HIERARCHICAL ATTENTION FOR CROWD COUNTING: Processed features from multiple layers are concatenated and passed through a fusion module to produce the final density map.The fusion module uses convolutional layers to map the concatenated features to a one-channel output.
  • III. HIERARCHICAL ATTENTION FOR CROWD COUNTING: The network minimizes Euclidean distance between each predicted density map and its corresponding ground-truth density map.The training objective is defined over N training samples, with Fd(Xi, Θ) denoting the estimated density and Di the ground-truth density.
  • III. HIERARCHICAL ATTENTION FOR CROWD COUNTING: Ground-truth density maps sum Gaussian kernels centered at annotated people, and their total density estimates the number of people.The generated density map has one-fourth the input image resolution.

A. Spatial attention module

The spatial attention module learns foreground-background segmentation from conv3 features and uses it to emphasize relevant spatial regions. This suppresses background regions while enriching low-level features before fusion with other layers.

  • A. Spatial attention module: SAM uses conv3 feature maps to predict a pixel-wise segmentation map for spatial attention.The module consists of four 3×3 convolutional layers and outputs values in [0, 1] across spatial locations.
  • A. Spatial attention module: The predicted segmentation map multiplies the conv3 features element-wise, producing actuated low-level features.This operation incorporates segmentation awareness into the feature representation.
  • A. Spatial attention module: Segmentation infusion suppresses irrelevant background regions and boosts foreground regions in the conv3 feature maps.Figure 4 compares the input, pre-infusion features, and post-infusion features; density maps use red for high density and blue for low density.
  • A. Spatial attention module: The actuated conv3 features are fused with features from other layers to generate the final density map.The fusion step connects spatially enhanced low-level features with the network’s multi-layer representation.
  • A. Spatial attention module: SAM is trained with cross-entropy loss using segmentation maps obtained by thresholding existing ground-truth density annotations.Pixels containing head regions are labeled foreground, while other pixels are labeled background.

B. Global attention modules

Global attention modules enhance higher-level feature maps by learning channel-wise attention, while the broader weak-supervision setup uses image-level labels to adapt counting networks to new datasets.

  • Global attention modules: GAMs consume higher-level feature maps and learn channel-wise attention to emphasize important channels and suppress unnecessary information.They perform spatial pooling, pass pooled features through fully connected and sigmoid layers, then multiply the resulting attention vector with the feature map.
  • Weak supervision via image-level labels: The weakly supervised setup adapts a pretrained HA-CCN to target datasets using image-level density labels instead of point-wise annotations.A CAM module produces class-wise pixel scores that are aggregated into image-level classification scores for training.
  • Weak supervision via image-level labels: Crowd-density classification reformulates regression into six density classes, making image-level labeling easier than specifying exact counts.The classes range from zero density to very high density.
  • Weak supervision via image-level labels: CAM class-wise score maps are converted into pseudo ground-truth density maps using class scores and source-dataset average counts.These maps provide coarse regional density supervision for target images.
  • Weak supervision via image-level labels: Pseudo ground-truth maps are less sharp than actual ground-truth maps but provide more regional information than image-level labels alone.During target adaptation, these maps supervise the counting network while VGG-16 weights remain fixed and later convolutional layers are updated.

A. Hierarchical attention-based counting

The study evaluates HA-CCN through ablation experiments, standard metrics, and inference-speed considerations. The fully convolutional design permits whole-image inference rather than patch-based testing.

  • Training and implementation details: Experiments evaluate the proposed method against configurations and recent approaches on three public datasets.The datasets are ShanghaiTech, UCF-QNRF, and UCF CROWD 50.
  • Training and implementation details: The network is trained end-to-end with Adam at a learning rate of 0.00005 and momentum 0.9 on a Titan Xp GPU.Training uses a 10% validation split, 224×224 crops from nine random locations, horizontal flips, and random noise.
  • Training and implementation details: MAE and MSE measure counting performance using ground-truth and estimated counts across test samples.N denotes test samples; y_i is the ground-truth count and y′_i is the estimated count.
  • Training and implementation details: Because the network is fully convolutional, entire test images are forwarded during inference, making testing faster than patch-based methods.The comparison names Switching-CNN, IG-CNN, CP-CNN, and SA-Net as patch-based examples.

2) Architecture ablation:

The ablation study tests progressively richer VGG16-based configurations on ShanghaiTech Parts A and B. Results indicate that spatial and global attention improve multi-scale counting, with explicit SAM supervision outperforming self-supervision.

  • Architecture ablation: The ablation study uses ShanghaiTech Parts A and B, containing 482 and 716 images respectively.Together, the parts contain 330,165 head annotations and include training and test subsets.
  • Architecture ablation: Five configurations range from VGG16 and multi-scale concatenation to VGG16+MS+SAM+GAM, the proposed HA-CCN.The study includes self-supervised and explicitly supervised SAM variants.
  • Architecture ablation: Global attention further improves performance, indicating the significance of channel-wise importance in the network.The proposed HA-CCN combines multi-scale features, SAM, and GAM.
  • Architecture ablation: Explicitly supervised SAM performs better than the self-supervised spatial-attention variant.The comparison is part of the ShanghaiTech ablation experiments.
  • Architecture ablation: HA-CCN achieves the best MAE among the evaluated configurations across all density levels.Figure 7 plots MAE for different configurations at different density levels.

3) Comparison with recent methods:

HA-CCN is compared with recent crowd-counting methods on ShanghaiTech, UCF CROWD 50, and UCF-QNRF. The reported comparisons place the proposed method ahead of existing methods on the first two datasets and at state-of-the-art level on UCF-QNRF.

  • Comparison with recent methods: The comparison covers ShanghaiTech, UCF CROWD 50, and UCF-QNRF datasets.UCF CROWD 50 uses 50 annotated images and five-fold cross-validation; UCF-QNRF contains 1,535 images and 1.25 million annotations.
  • Comparison with recent methods: HA-CCN outperforms all listed existing methods on the ShanghaiTech and UCF CROWD 50 datasets.The comparison includes Switching-CNN, CP-CNN, IG-CNN, D-ConvNet, CSRNet, ic-CNN, SA-Net, ADCrowdNet, and Residual Regression.
  • Comparison with recent methods: On UCF-QNRF, HA-CCN achieves state-of-the-art results against five compared approaches.The compared methods include Idrees et al., MCNN, CMTL, Switching-CNN, and Idrees et al..
  • Comparison with recent methods: Table IV reports cross-dataset performance using MAE/MSE, with S, NS, and C denoting target-trained, source-trained, and performance-drop settings.The table caption defines the evaluation abbreviations.
  • Comparison with recent methods: Qualitative examples show input images, ground-truth maps, and estimated density maps for ShanghaiTech, UCF CROWD 50, and UCF-QNRF.These examples are presented in Figures 8, 9, and 10.

B. Cross dataset performance

The paper evaluates HA-CCN under dataset shift and introduces weakly supervised adaptation using image-level labels. Weak supervision improves generalization, although it remains below fully supervised target-dataset performance.

  • Cross dataset performance: When trained on ShanghaiTech A and tested on ShanghaiTech B, UCF CROWD 50, and WorldExpo ’10, HA-CCN is relatively more robust to dataset-distribution changes.The comparison uses MCNN, Switching-CNN, and D-ConvNet as baselines and measures overall count error with MAE/MSE.
  • Cross dataset performance: Cross-dataset performance remains considerably worse than performance when the network is fully supervised on the target set.The paper addresses this gap with the weakly supervised technique described in the following section.
  • Cross dataset performance: The weakly supervised setup adapts the network using image-level labels instead of point-wise or count annotations for target data.The setup uses source and target training sets with cropped multi-scale patches and introduces label noise for 15% of target training samples.
  • Cross dataset performance: The counting network is first fully supervised on a diverse source dataset, after which a CAM module is trained and fine-tuned on target samples using image-level labels.The counting-network weights are fixed while the CAM module is trained with binary cross entropy on aggregated class scores.
  • Cross dataset performance: Weak supervision significantly improves generalization, with Log-Sum-Exponential aggregation outperforming Global Average Pooling and Global Max Pooling.The evaluated configurations are HA-CCN+W-A, HA-CCN+W-M, and HA-CCN+W-L.
  • Cross dataset performance: Weakly supervised results are comparable to several recent fully supervised techniques.The cited comparisons include Hydra-CNN, MCNN, Walach et al., and Switching-CNN.

VI. CONCLUSIONS

The proposed crowd counting network combines spatial and global attention mechanisms, and introduces weakly supervised adaptation using image-level labels. Experiments showed significant improvements on challenging datasets, while future work targets broader feature integration and backbone architectures.

  • The network combines spatial attention for pixel-level foreground-background enhancement with global attention for channel-wise importance.
  • A weakly supervised setup adapts counting models to different datasets using image-level labels.
  • Extensive experiments on challenging datasets demonstrated significant improvements over recent state-of-the-art approaches.
  • Future work will explore improved cross-layer feature integration, other backbone networks, and additional weakly supervised or semi-supervised approaches.
Loading 1907.10255v1…