Source-linked AI summary
Multi-Level Bottom-Top and Top-Bottom Feature Fusion for Crowd Counting
Vishwanath A Sindagi, Vishal M. Patel
TL;DR
Crowd counting in congested scenes must handle severe scale variation, while straightforward multi-scale fusion has limited effectiveness. The paper introduces bidirectional multi-level fusion with complementary feature extraction and scale-aware supervision, and reports significant improvements over existing methods on three datasets.
Problem
Congested-scene crowd counting is difficult because of scale variation, and straightforward fusion has limited ability to combine multi-scale features effectively.
Method
The method combines multi-level bottom-top and top-bottom fusion, SCFB cross-scale residual extraction, and scale-aware density-map supervision based on annotations and superpixel segmentation.
Results
The proposed approach outperforms existing methods by a significant margin on ShanghaiTech, UCF CROWD 50, and UCF-QNRF.
Takeaways & Limitations
Bidirectional multi-level fusion with scale-complementary features and scale-aware supervision improves crowd counting across three highly congested-scene datasets.
Abstract
from arXiv · showhide
Crowd counting presents enormous challenges in the form of large variation in scales within images and across the dataset. These issues are further exacerbated in highly congested scenes. Approaches based on straightforward fusion of multi-scale features from a deep network seem to be obvious solutions to this problem. However, these fusion approaches do not yield significant improvements in the case of crowd counting in congested scenes. This is usually due to their limited abilities in effectively combining the multi-scale features for problems like crowd counting. To overcome this, we focus on how to efficiently leverage information present in different layers of the network. Specifically, we present a network that involves: (i) a multi-level bottom-top and top-bottom fusion (MBTTBF) method to combine information from shallower to deeper layers and vice versa at multiple levels, (ii) scale complementary feature extraction blocks (SCFB) involving cross-scale residual functions to explicitly enable flow of complementary features from adjacent conv layers along the fusion paths. Furthermore, in order to increase the effectiveness of the multi-scale fusion, we employ a principled way of generating scale-aware ground-truth density maps for training. Experiments conducted on three datasets that contain highly congested scenes (ShanghaiTech, UCF_CC_50, and UCF-QNRF) demonstrate that the proposed method is able to outperform several recent methods in all the datasets.
1. Introduction
Crowd counting in congested scenes is challenged by scale variation, missing scale labels, and the limited effectiveness of straightforward feature fusion. The paper proposes multi-level bidirectional fusion, scale-complementary extraction, and principled scale-aware supervision.
- Scale variation within and across images, alongside occlusions, perspective changes, and background clutter, makes congested-scene crowd counting difficult.
- Crowd-counting datasets typically provide only head locations, leaving scale labels unavailable for training robust models.
- Standard addition or concatenation fuses multiple layers simultaneously but requires the network to learn feature importance automatically, producing sub-optimal fusion.
- Bottom-top fusion transfers spatial details upward, while top-bottom fusion propagates semantic context downward to suppress noise in lower layers.
- MBTTBF exchanges spatial information and high-level context bidirectionally across multiple levels, reducing noise propagation while preserving lower-layer details.
- SCFB extracts complementary features from adjacent scales, while scale-aware supervision combines annotations and superpixel segmentation in an MRF framework.
- The method outperforms existing methods by a significant margin on ShanghaiTech, UCF CROWD 50, and UCF-QNRF.
2. Related work
Related work has progressed from CNN architectures addressing scale variation toward methods incorporating additional cues, learning strategies, datasets, and multi-scale features.
- CNN-based crowd-counting methods have produced dramatic error-rate improvements compared with traditional approaches.
- Switching-CNN addresses scale variation with multiple independent regressors selected by a switch classifier based on image patches.
- Recent methods expand beyond scale robustness through dataset-specific capacity, adversarial loss, unlabeled-data ranking, cascaded fusion, scale-based aggregation, and weak supervision.
- Idrees et al. introduced a large-scale high-density dataset containing approximately 1.25 million head annotations and a localization task for crowded images.
- Other work incorporates segmentation, semantic priors, attention, perspective, context, multiple views, and multi-scale features, while synthetic-data adaptation uses SSIM-based CycleGAN.
3. Proposed method
The proposed network fuses multi-scale features through hierarchical bottom-top and top-bottom paths, scale-complementary extraction, attention-based fusion, and scale-aware supervision.
- Network overview: The network uses three branches: a VGG16 main branch, multi-level bottom-top fusion, and multi-level top-bottom fusion.Multi-scale features from conv3–conv6 are reduced to 32 channels before fusion.
- Multi-level bottom-top fusion: Bottom-top fusion propagates spatial information from lower layers upward through two hierarchical fusion levels.The first level progressively combines conv3–conv6 features, while the second combines features produced by the first level.
- Multi-level top-bottom fusion: Top-bottom fusion propagates high-level context downward through two hierarchical levels to reduce noise in lower-layer features.The top-bottom path progressively combines features from deeper layers with lower-layer features using SCFB blocks.
- Self-attention-based fusion: A self-attention fusion module combines four feature maps from the two fusion paths and forwards the result through a 1×1 convolution to produce the density map.The attention maps weight the fusion-path features element-wise before aggregation.
- Scale complementary feature extraction: SCFB uses cross-scale residual connections and convolutional layers to extract complementary features from adjacent network layers.Unlike direct addition or concatenation, the block is designed to learn relevant features from each fused scale.
- Scale-aware supervision: Scale-aware supervision divides head annotations into four size categories, producing separate density maps for progressively larger heads.The scale-aware maps supervise the individual SCFB branches.
4. Details of implmentation and training
The network is trained end-to-end and evaluated using MAE and MSE. Supervision is applied both to the final overall density map and to intermediate SCFB paths using scale-aware ground truths.
- Training: The network weights are optimized end-to-end using Adam with a learning rate of 0.00005 and momentum of 0.9.Random noise and random image flipping are used for data augmentation.
- Evaluation: Performance is evaluated with mean absolute error (MAE) and mean squared error (MSE).MAE and MSE quantify errors between predicted and ground-truth counts.
- Supervision: Final supervision uses the overall density map, while SCFB paths use corresponding scale-aware ground truths.Both levels use Euclidean loss.
5. Experiments and results
Experiments evaluate the proposed fusion network through ablations and comparisons on three congested crowd datasets. Multi-directional, multi-level fusion with SCFB and scale-aware supervision improves qualitative and quantitative results, while the method achieves leading benchmark performance.
- Experimental setup: Experiments analyze the proposed network through an ablation study and compare it with recent state-of-the-art methods.The evaluation uses ShanghaiTech, UCF CROWD 50, and UCF-QNRF.
- Ablation study: The ablation study compares baseline, addition, concatenation, bottom-top, top-bottom, bidirectional, multi-level, SCFB, and scale-aware-supervision configurations.The configurations are evaluated progressively from simple fusion to the proposed method.
- Ablation study: Simple addition or concatenation at the network end does not significantly improve over the baseline.The authors attribute this to supervision directly affecting initial convolutional layers in the main branch.
- Ablation study: Bottom-top or top-bottom fusion improves considerably over the baseline, while bidirectional and multi-level fusion reduce count error further.The additional fusion level in experiment vii is reported to reduce count error further.
- Ablation study: Replacing fusion blocks with SCFB improves performance, and adding scale-aware supervision produces further improvements.The comparison is between experiments viii and ix, with and without scale-aware supervision.
- Ablation study: Qualitatively, simple concatenation produces background noise and lost details, whereas the proposed configuration yields more detailed density maps with less background clutter.The comparison covers experiments iii, vi, and ix in Figure 6.
- Comparison with recent methods: The proposed method achieves the best results on ShanghaiTech Part A, UCF CROWD 50, and UCF-QNRF, while ranking close second on ShanghaiTech Part B.On ShanghaiTech Part B, it is reported to trail only CAN.
6. Conclusion
The proposed method improves multi-layer feature fusion for congested crowd counting by combining scale-complementary features with bidirectional hierarchical propagation and principled density-map supervision. It achieves significant improvements over existing fusion schemes and state-of-the-art counting methods on three popular datasets.
- Method: The method extracts scale-complementary features from adjacent layers before propagating them hierarchically in bottom-top and top-bottom directions.This produces a more effective fusion of features from multiple backbone layers.
- Training supervision: Its ground-truth density maps are created in a principled way by combining image and location annotations from the dataset.
- Qualitative evaluation: The qualitative evaluation on ShanghaiTech compares input images, ground-truth density maps, and predicted density maps.
- Results: Significant improvements over existing fusion schemes and state-of-the-art counting methods are reported on three popular crowd counting datasets.