Source-linked AI summary
Distribution Matching for Crowd Counting
Boyu Wang, Huidong Liu, Dimitris Samaras, Minh Hoai
TL;DR
Sparse dot annotations and Gaussian-smoothed supervision can impair crowd-counting generalization. DM-Count instead matches normalized predicted and ground-truth distributions with Optimal Transport and Total Variation stabilization, achieving stronger theory and benchmark results. It outperforms prior methods across four datasets, including an approximately 16% reduction in published NWPU MAE.
Problem
Gaussian smoothing of sparse dot annotations can hurt generalization, while Bayesian supervision requires kernels and remains underdetermined.
Method
DM-Count matches normalized predicted and ground-truth density maps with Optimal Transport, combining counting, OT, and Total Variation losses.
Results
DM-Count significantly outperforms previous state-of-the-art methods on four crowd-counting benchmarks and reduces published NWPU MAE by approximately 16%.
Takeaways & Limitations
The paper supports treating crowd counting as distribution matching without Gaussian-smoothed ground-truth annotations.
Takeaways & Limitations
The Bayesian loss is underdetermined because fewer annotated dots than pixels permit infinitely many zero-loss predictions differing from the ground truth.
Abstract
from arXiv · showhide
In crowd counting, each training image contains multiple people, where each person is annotated by a dot. Existing crowd counting methods need to use a Gaussian to smooth each annotated dot or to estimate the likelihood of every pixel given the annotated point. In this paper, we show that imposing Gaussians to annotations hurts generalization performance. Instead, we propose to use Distribution Matching for crowd COUNTing (DM-Count). In DM-Count, we use Optimal Transport (OT) to measure the similarity between the normalized predicted density map and the normalized ground truth density map. To stabilize OT computation, we include a Total Variation loss in our model. We show that the generalization error bound of DM-Count is tighter than that of the Gaussian smoothed methods. In terms of Mean Absolute Error, DM-Count outperforms the previous state-of-the-art methods by a large margin on two large-scale counting datasets, UCF-QNRF and NWPU, and achieves the state-of-the-art results on the ShanghaiTech and UCF-CC50 datasets. DM-Count reduced the error of the state-of-the-art published result by approximately 16%. Code is available at https://github.com/cvlab-stonybrook/DM-Count.
1 Introduction
Crowd counting commonly estimates density maps from sparse dot annotations, but Gaussian-based supervision can be sensitive to annotation smoothing choices and may weaken generalization. DM-Count addresses these shortcomings through distribution matching and reports stronger theoretical and empirical results.
- Background: Density-map estimation sums predicted pixel values and is more robust to occlusion than detection-then-counting for large crowds.It avoids early binarized decisions.
- Problem: Training data provide sparse binary dot annotations rather than each person’s spatial extent, making pixel-wise supervision heavily unbalanced.The predicted map is dense and real-valued, whereas annotations are sparse binary matrices.
- Problem: Gaussian-smoothed supervision depends on selecting suitable blob widths despite large variation in people’s sizes and shapes across perspective images.The resulting network’s performance is highly dependent on the quality of this pseudo ground truth.
- Problem: Bayesian loss also requires Gaussian kernels and is underdetermined, allowing zero-loss predicted maps that differ substantially from the ground truth.Its infinitely many solutions can lead to predicted density maps unlike the ground truth.
- Contributions: DM-Count uses Optimal Transport between normalized predicted and ground-truth density maps, adds Total Variation for stability, and reports tighter bounds and improved benchmark performance.The authors report approximately a 16% reduction in published state-of-the-art MAE on NWPU.
2 Previous Work
Prior crowd-counting methods include detection, direct regression, and density-map estimation, with density maps favored for dense scenes. Optimal Transport provides a distribution-level way to measure dissimilarity while accounting for the locations of probability mass.
- Crowd Counting Methods: Crowd counting methods fall into detection-then-counting, direct count regression, and density-map estimation categories.Density-map estimation is described as more robust than the other two approaches.
- Density Map Estimation: Density-map methods commonly train against Gaussian-smoothed annotations, using fixed, adaptive, or multiple kernel widths.Adaptive widths can be selected from distances to annotated dots’ nearest neighbors.
- Optimal Transport: Optimal Transport measures the minimum cost of transforming one probability distribution into another.Its cost accounts for both transported mass and the distance between locations.
- Optimal Transport: The Monge-Kantorovich formulation minimizes transport cost over couplings whose marginals equal the source and target probability measures.The dual formulation expresses the same OT cost through constrained potentials.
3 DM-Count: Distribution Matching for Crowd Counting
DM-Count treats crowd counting as distribution matching without Gaussian-smoothed annotations, combining counting, Optimal Transport, and Total Variation losses. OT aligns normalized density distributions, while TV addresses approximation and training-stability issues in Sinkhorn optimization.
- DM-Count avoids Gaussian preprocessing by matching the normalized predicted density map with the normalized ground truth density map using Optimal Transport.
- The counting loss minimizes the absolute difference between the ground-truth and predicted total counts.
- OT compares distributions after converting unnormalized density maps into probability density functions by dividing by their total masses.
- Sinkhorn optimization approximates the OT solution but can fit dense regions more accurately than low-density regions when iterations are limited.The stated time complexity is O(n^2 log n/ϵ^2).
- The Total Variation loss supplements OT to address poorer low-density approximation and increase training stability.The overall objective combines counting, OT, and TV losses, with tunable weights for the OT and TV terms.
4 Generalization Bounds and Theoretical Analysis
The analysis characterizes how Gaussian-smoothed training affects generalization and derives bounds for DM-Count’s counting, OT, TV, and combined losses. It also identifies dependence on sample size, image size, transport cost, and loss-weight balancing.
- Bound Behavior: As the number of samples K grows, the sample-dependent complexity and concentration terms decrease toward zero.The theorem analysis states that the resulting upper bounds are tighter than the Gaussian-smoothed-method bound.
- Gaussian Smoothed Methods: Gaussian-smoothed training can generalize poorly: with sufficient training data, the real-ground-truth risk is bounded above by smoothed-ground-truth risk plus the annotation discrepancy.The lower-bound analysis also states that real-ground-truth risk can be as large as the expected annotation discrepancy when smoothed-ground-truth risk is zero.
- Bayesian Loss: The Bayesian loss is underdetermined because fewer annotated dots than pixels yield infinitely many zero-loss predicted density maps different from the ground truth.Its Gaussian likelihoods are centered at annotated dots, but the resulting equations do not uniquely determine the predicted density map.
- DM-Count Losses: Theorem 2 provides generalization bounds for the counting, OT, TV, and overall losses under bounded-loss and nonzero-mass assumptions.The overall bound includes the hypothesis complexity term, OT cost coefficient, TV coefficient, and a sample-dependent concentration term.
- Empirical Illustration: On toy data, DM-Count produces more accurate crowd count and localization than pixel-wise and Bayesian losses.The pixel-wise loss yields a blurry map with higher counting error, while the Bayesian loss assigns high values at many unannotated locations.
- Bound Behavior: The counting-loss complexity coefficient is O(n), whereas the OT and TV coefficients are O(n^2), so larger images require more training images.When combining losses, λ1 and λ2 should be small enough to balance their contributions.
5 Experiments
Experiments on four benchmark datasets show that DM-Count is effective and stable across settings, outperforming competing losses and state-of-the-art methods. Additional studies examine visualization quality, hyper-parameters, Sinkhorn iterations, component contributions, and annotation noise.
- Quantitative Results: DM-Count outperforms all other methods except CAN under MSE in NWPU, where the two methods are comparable.The same hyper-parameters are used across experiments, and the authors report stable performance across datasets.
- Quantitative Results: DM-Count outperforms pixel-wise and Bayesian losses with the same network architecture and training procedure across all experiments.It also achieves state-of-the-art performance on all four datasets without the multi-scale or deeper architectures used by some competing methods.
- Quantitative Results: On UCF-QNRF, DM-Count reduces Bayesian-loss MAE from 88.7 to 85.6 and MSE from 154.8 to 148.3.On NWPU, it reduces MAE from 105.4 to 88.4 and NAE from 0.203 to 0.169.
- Qualitative Results: DM-Count produces sharper density maps and more accurate localization in both dense and sparse regions than pixel-wise and Bayesian losses.On UCF-QNRF, its average PSNR and SSIM are 40.65 and 0.55, versus 34.79 and 0.43 for pixel-wise loss and 34.55 and 0.42 for Bayesian loss.
- Hyper-parameter and Sinkhorn Studies: Using 100 Sinkhorn iterations improves DM-Count beyond lower-iteration settings, after which performance plateaus.The authors use 100 iterations in all experiments because fewer iterations produce inaccurate OT solutions.
- Ablation Studies: The OT, counting, and TV losses are all essential, with the OT loss identified as the most important component.The component analysis is conducted on UCF-QNRF.
- Robustness to Noisy Annotations: DM-Count is more robust to annotation errors than pixel-wise and Bayesian losses when training uses randomly perturbed annotations.The added noise ranges from 0 to 5% of image height, about 80 pixels on average.
6 Conclusion
The conclusion argues that Gaussian smoothing can weaken generalization for crowd counting and presents DM-Count as a distribution-matching alternative. DM-Count uses Optimal Transport without Gaussian-smoothed annotations and significantly outperforms prior state-of-the-art methods on four benchmarks.
- Gaussian smoothing of ground-truth dot annotations can hurt a model’s generalization bound when testing on real ground-truth data.
- DM-Count treats crowd counting as distribution matching and uses Optimal Transport without Gaussian smoothing of annotated dots.
- DM-Count has a tighter generalization error bound than Gaussian-smoothed methods and significantly outperforms previous state-of-the-art methods on four benchmarks.
Broader Impact
The paper frames accurate and interpretable crowd counting as potentially useful for crowd control, public safety, and disputed crowd-size records. It emphasizes transparency alongside counting accuracy for critical applications.
- More accurate crowd-size estimates could guide crowd control and improve public safety.
- Better crowd localization can make estimated counts more interpretable and increase transparency in critical applications.
- Transparent, accurate, and objective counting methods could support historical records and public acceptance of estimates for politically disputed crowd sizes.