Source-linked AI summary
Crowd Counting with Deep Structured Scale Integration Network
Lingbo Liu, Zhilin Qiu, Guanbin Li, Shufan Liu, Wanli Ouyang, Liang Lin
TL;DR
Crowd counting must handle substantial variation in people’s scales, which remains insufficiently addressed by conventional multiscale feature fusion and pixel-wise losses. DSSINet combines CRF-based mutual feature refinement with a dilated multiscale structural-similarity loss, and experiments on four benchmarks show superior performance over state-of-the-art methods.
Problem
Crowd counting remains challenging because people’s scales vary greatly, while existing fusion strategies and pixel-wise losses do not fully capture complementary features or regional scale correlations.
Method
DSSINet uses CRF-based message passing to mutually refine multiscale features and DMS-SSIM loss to model local correlations across regions of various sizes.
Results
Extensive experiments on four challenging benchmarks show that DSSINet achieves superior performance compared with state-of-the-art methods.
Takeaways & Limitations
Structured feature representation learning and hierarchically structured loss optimization provide the paper’s approach to handling crowd scale variation and producing locally consistent density estimates.
Abstract
from arXiv · showhide
Automatic estimation of the number of people in unconstrained crowded scenes is a challenging task and one major difficulty stems from the huge scale variation of people. In this paper, we propose a novel Deep Structured Scale Integration Network (DSSINet) for crowd counting, which addresses the scale variation of people by using structured feature representation learning and hierarchically structured loss function optimization. Unlike conventional methods which directly fuse multiple features with weighted average or concatenation, we first introduce a Structured Feature Enhancement Module based on conditional random fields (CRFs) to refine multiscale features mutually with a message passing mechanism. In this module, each scale-specific feature is considered as a continuous random variable and passes complementary information to refine the features at other scales. Second, we utilize a Dilated Multiscale Structural Similarity loss to enforce our DSSINet to learn the local correlation of people's scales within regions of various size, thus yielding high-quality density maps. Extensive experiments on four challenging benchmarks well demonstrate the effectiveness of our method. Specifically, our DSSINet achieves improvements of 9.5% error reduction on Shanghaitech dataset and 24.9% on UCF-QNRF dataset against the state-of-the-art methods.
1. Introduction
Crowd counting remains difficult because people appear at widely varying scales, while conventional feature fusion and pixel-wise losses do not fully model complementary scale information or regional scale correlations. DSSINet addresses these limitations through structured feature refinement and multiscale structural-similarity optimization.
- Huge variation in people’s scales remains a major unresolved challenge for crowd counting in diverse unconstrained scenes.
- Conventional methods extract multiscale features but typically fuse them through weighted averaging or concatenation, limiting their use of complementary information.Deeper features encode semantic information, whereas shallower features retain low-level appearance details.
- People’s head radiuses are often locally uniform, but scale correlations vary by region, making fixed-size pixel-wise losses inadequate.Large-head regions require larger neighborhoods, whereas distant small-head regions can be captured with smaller ones.
- DSSINet uses CRF-based Structured Feature Enhancement Modules to mutually refine multiscale features through message passing across three scaled-image subnetworks.The subnetworks share parameters, and complementary information from other scales enhances each scale-specific feature.
- DSSINet applies Dilated Multiscale Structural Similarity loss to learn local scale correlations across regions of different sizes and produce locally consistent density maps.The framework progressively fuses side-output density maps top-down to obtain a high-resolution density map.
2. Related Work
Prior crowd-counting methods commonly use multiscale architectures to handle variation in people’s sizes, while CRFs and MS-SSIM provide related feature-refinement and similarity-learning foundations. DSSINet combines these directions through CRF-based multiscale refinement and progressively fused density-map outputs.
- Crowd Counting: Multiscale architectures are mainstream in crowd counting because they extract features from multiple network columns or branches to handle people’s scale variation.Earlier approaches include combining deep and shallow networks and using multi-column CNNs.
- Conditional Random Fields: CRFs have refined CNN features and outputs through message passing in tasks including semantic segmentation, contour prediction, and action recognition.Prior work modeled pixel relationships, fused features with Attention-Gated CRFs, or enhanced view-specific features.
- Conditional Random Fields: DSSINet is presented as the first crowd-counting method to use CRFs to mutually refine multiple features at different scales.Its framework includes four CRF-based Structured Feature Enhancement Modules for multiscale feature refinement.
- Multiscale Structural Similarity: MS-SSIM compares luminance, contrast, and structure across multiscale image regions and has been used as a loss in image restoration.Its effectiveness had not yet been verified for high-level tasks such as crowd counting.
3. Method
DSSINet addresses crowd-scale variation through mutually refined multiscale features and a dilated structural-similarity loss. Its architecture combines parallel scaled-image subnetworks, CRF-based feature enhancement, progressive density-map refinement, and multiscale loss optimization.
- DSSINet Overview: DSSINet uses three parameter-shared parallel subnetworks, each processing a different scaled version of the same image for multiscale feature extraction.The inputs form a three-level image pyramid containing scales 2, 1, and 0.5 relative to the original image.
- DSSINet Overview: Features at matching resolutions are grouped across subnetworks because different receptive fields and image sizes provide complementary appearance and semantic information.Shallow features contain appearance details, whereas deeper features encode higher-level semantic information.
- Density-map Generation: DSSINet generates a high-resolution density map top-down by progressively refining shallower side outputs with upsampled deeper maps.The final map M0 has resolution H × W and fine spatial details of people’s distribution.
- Structured Feature Enhancement Module: The CRF-based Structured Feature Enhancement Module mutually refines scale-specific features through message passing and dynamically learned cross-feature information.Mean-field inference yields each refined feature as its original feature plus information received from features at other scales; 1 × 1 convolutions implement this exchange.
4. Experiments
Experiments evaluate DSSINet across four crowd-counting benchmarks and ablate its feature-refinement, multiscale-input, and structural-loss components. The method achieves strong benchmark performance, while complexity analysis reports fewer parameters than existing state-of-the-art methods.
- State-of-the-art comparison: DSSINet achieves 9.5% relative MAE improvement and 8.1% relative MSE improvement over SANet on Shanghaitech Part A.On Part B, it decreases MSE from 13.6 to 10.34.
- State-of-the-art comparison: 24.9% MAE improvement over CL accompanies the lowest reported MAE of 99.1 and MSE of 159.2 on UCF-QNRF.The comparison includes eight methods on this challenging benchmark.
- State-of-the-art comparison: DSSINet obtains MAE 216.9 on UCF CC 50, outperforming all other methods and improving 19.1% over SANet.Evaluation uses five-fold cross-validation on the 50-image dataset.
- State-of-the-art comparison: DSSINet achieves the best average MAE across five WorldExpo’10 scenes and the lowest MAE in three individual scenes.Unlike methods relying on temporal information or perspective maps, it generates density maps and estimates counts without those inputs.
- Ablation studies: CRF-2 FeatRefine performs best in the SFEM ablation, whereas excessive CRF iterations degrade performance by mixing multiscale features too strongly.The final model therefore uses two CRF iterations.
- Ablation studies: DMS-SSIM-5 achieves MAE 60.63 and MSE 96.04, outperforming Euclidean and SSIM losses; DMS-SSIM-6 declines when applied to 121×121 regions.The larger region may contain nonuniform people’s scales, weakening the local-correlation assumption.
- Complexity analysis: DSSINet contains 8.858 million parameters, about half as many as existing state-of-the-art methods, and processes a 720×576 frame in 450 ms on an NVIDIA 1080 GPU.Shared subnetworks and 1×1 SFEM convolutions reduce the parameter count.
5. Conclusion
DSSINet addresses people’s scale variation through structured feature refinement and hierarchically structured loss optimization, achieving superior performance on four benchmarks.
- DSSINet uses a CRF-based Structured Feature Enhancement Module to mutually refine multiple features and improve their robustness.
- Its Dilated Multiscale Structural Similarity Loss learns local scale correlations across regions of varying sizes, producing locally consistent estimates.
- Extensive experiments on four benchmarks show superior performance compared with state-of-the-art methods.