Source-linked AI summary
Point in, Box out: Beyond Counting Persons in Crowds
Yuting Liu, Miaojing Shi, Qijun Zhao, Xiaofang Wang
TL;DR
Crowd counting methods based on density regression use inexpensive point supervision but do not detect individuals, while detection methods generally require costly bounding boxes. PSDDN uses point annotations to initialize and refine pseudo boxes, constrain local box sizes, and train progressively; experiments report competitive detection and counting across several benchmarks. Its pseudo-box scale depends on dataset-specific assumptions, and fully supervised comparisons have methodological differences.
Problem
Regression-based crowd counting can use point annotations but cannot detect individuals, whereas detection-based methods commonly require expensive bounding-box annotations.
Method
PSDDN initializes pseudo boxes from nearest-neighbor head distances, updates them online, applies locally constrained regression, and uses curriculum learning with point supervision.
Results
PSDDN performs close to density-based methods for counting, improves results when combined with a prior method, and produces competitive detection results across ShanghaiTech, UCF CC 50, WiderFace, and TRANCOS.
Takeaways & Limitations
Point-level annotations can support a detection network that provides head size and location information alongside crowd counts.
Takeaways & Limitations
The pseudo-box scale factor lacks an exact selection because real bounding boxes are unavailable, so its default varies across datasets; comparisons with fully supervised methods also have methodological differences.
Abstract
from arXiv · showhide
Modern crowd counting methods usually employ deep neural networks (DNN) to estimate crowd counts via density regression. Despite their significant improvements, the regression-based methods are incapable of providing the detection of individuals in crowds. The detection-based methods, on the other hand, have not been largely explored in recent trends of crowd counting due to the needs for expensive bounding box annotations. In this work, we instead propose a new deep detection network with only point supervision required. It can simultaneously detect the size and location of human heads and count them in crowds. We first mine useful person size information from point-level annotations and initialize the pseudo ground truth bounding boxes. An online updating scheme is introduced to refine the pseudo ground truth during training; while a locally-constrained regression loss is designed to provide additional constraints on the size of the predicted boxes in a local neighborhood. In the end, we propose a curriculum learning strategy to train the network from images of relatively accurate and easy pseudo ground truth first. Extensive experiments are conducted in both detection and counting tasks on several standard benchmarks, e.g. ShanghaiTech, UCF_CC_50, WiderFace, and TRANCOS datasets, and the results show the superiority of our method over the state-of-the-art.
1. Introduction
Crowd counting is important but difficult in dense scenes, and existing approaches trade off inexpensive supervision against individual detection. PSDDN addresses this gap by using point annotations to produce head detections and counts, with experiments spanning multiple benchmarks.
- Crowd counting supports video surveillance, safety monitoring, and behavior modeling but is difficult because of occlusion, perspective distortion, scale variation, and density variation.
- Regression-based methods estimate counts from density maps using point annotations, but they do not provide individual person detections at test time.
- Detection-based crowd counting directly detects individuals but has been less explored because it typically requires expensive bounding-box annotations.
- PSDDN uses only point-level head annotations to predict both the locations and sizes of human heads while bypassing bounding-box annotation.
- The method combines pseudo-ground-truth initialization and updating, locally constrained regression, and curriculum learning based on pseudo-ground-truth difficulty.
- Experiments cover crowd detection and counting on ShanghaiTech, UCF CC 50, WiderFace, and vehicle counting on TRANCOS.
2. Related works
Related work spans detection-based counting, density-regression counting, and point supervision. Prior methods either rely on extensive box annotations or provide localization without modeling proper person size, motivating PSDDN’s point-supervised detection approach.
- Detection-based crowd counting: Detection-based methods use motion, appearance, or deep detectors, but heavy occlusion and exhaustive bounding-box annotations limit their application to crowded scenes.
- Detection-based crowd counting: Small-object and face-detection methods address localization with box supervision, but their target crowds are less dense than those in crowd-counting benchmarks.
- Regression-based crowd counting: Modern regression-based methods estimate density maps whose integrals produce crowd counts, achieving substantial progress with deep neural networks.
- Regression-based crowd counting: DecideNet combines detection- and regression-based density maps, while composition-loss methods jointly regress density and localization maps.
- Point supervision: Point supervision reduces annotation effort in pose estimation, object detection, and segmentation by marking key points or selected object locations.
3. Method
PSDDN extends anchor-based detection to point-supervised crowd counting by initializing and updating pseudo boxes, constraining local box sizes, and using curriculum learning.
- Network overview: PSDDN uses ResNet-101 features with two detection scales whose predictions are up-sampled and summed into a final prediction.The detection layers predict anchor classification scores and four box offsets, followed by multi-task classification and regression losses.
- Pseudo ground truth initialization: Pseudo ground truth boxes are initialized as square boxes whose size equals each head point’s nearest-neighbor distance, then matched to similarly sized anchors.The initialization is typically closer in dense crowds but often larger than real boxes in sparse crowds.
- Pseudo ground truth updating: During training, the highest-scoring suitable positive anchor iteratively replaces each initialized pseudo box, while boxes without positive samples are ignored.The replacement is selected among predictions smaller than the initial nearest-neighbor-based size, and dense anchors support repeated updates.
- Locally-constrained regression loss: Because pseudo-box centers are accurate but widths and heights are not, PSDDN replaces direct box regression with a locally-constrained regression loss.The center loss compares predicted and pseudo-box centers, while size constraints use neighboring boxes within a narrow horizontal band.
- Locally-constrained regression loss: The size constraint penalizes predicted widths or heights outside local mean ± 3 standard deviations, while tolerating individual head-size variation.The local statistics are computed from boxes within rows i−1:i+1 across the feature map.
- Curriculum learning: Curriculum learning orders images by pseudo-box difficulty, starting with mostly medium-sized boxes before progressively adding harder folds.Very sparse scenes tend to produce oversized initial boxes, whereas very dense scenes tend to produce boxes that are too small and difficult to detect.
4. Experiments
Experiments evaluate PSDDN for crowd counting and person detection across ShanghaiTech, UCF CC 50, WiderFace, and TRANCOS, using point-supervised training and dataset-specific detection criteria. The proposed components improve counting and detection performance, while results remain competitive across dense crowds, face detection, and vehicle counting.
- Experimental setup: Experiments cover crowd counting and detection on ShanghaiTech, UCF CC 50, WiderFace, and TRANCOS, with MAE and MSE for counting.Detection uses IoU on WiderFace and center-distance and predicted-size criteria on ShanghaiTech and UCF CC 50.
- Ablation study: The full PSDDN progressively lowers SHA MAE from 168.6 for Pv0 to 85.4 for Pv3, while MSE falls from 268.3 to 159.2.The same qualitative improvement is reported on SHB, and each component provides a clear benefit.
- Counting comparison: PSDDN combined with [20] achieves MAE 65.9 on SHA and 9.1 on SHB, outperforming the state-of-the-art comparison reported by the authors.The standalone detection-based method performs close to recent density-based methods, while the combined result is reported as superior.
- UCF CC 50: On UCF CC 50, PSDDN achieves counting MAE 359.4 and MSE 514.8, with detection AP 0.536 despite the dataset’s very dense crowds.The paper also reports reasonable bounding boxes for tiny heads in this dataset.
- Detection ablation: Detection ablations raise AP on SHA from 0.308 for Pv0 to 0.554 for Pv3 and on SHB from 0.015 to 0.663.The full model includes pseudo-ground-truth updating, locally-constrained regression, and curriculum learning.
- Generalization: PSDDN generalizes to WiderFace and TRANCOS, attaining WiderFace AP 0.605, 0.605, and 0.396 on easy, medium, and hard sets, and TRANCOS GAME0-GAME3 values of 4.79, 5.43, 6.68, and 8.40.On TRANCOS, PSDDN outperforms the best regression-based method on GAME1, GAME2, and GAME3 and is competitive on GAME0.
5. Conclusion
The paper presents PSDDN, a point-supervised deep detection network for person detection and counting in crowds. It combines iterative pseudo-box refinement, local-band regression, and curriculum learning, with experiments demonstrating effectiveness across standard benchmarks.
- PSDDN performs person detection and counting in crowds using point-level head annotations.
- Pseudo ground-truth bounding boxes are initialized from head points and updated iteratively during training.
- Bounding-box regression compares each predicted box with ground-truth boxes within a local band area.
- Curriculum learning addresses density variation by training from relatively accurate and easy pseudo ground truth first.
- Experiments on several standard benchmarks show PSDDN's efficiency and effectiveness for person detection and crowd counting.