Source-linked AI summary
Divide and Grow: Capturing Huge Diversity in Crowd Images with Incrementally Growing CNN
Deepak Babu Sam, Neeraj N Sajjan, R. Venkatesh Babu
TL;DR
Crowd counting is challenged by large appearance diversity, especially because visible features depend on crowd density. IG-CNN grows a CNN tree of automatically specialized regressors and routes patches with a classifier; it reports better performance on major benchmarks, while specialization quality can limit generalization.
Problem
Crowd counting must handle large appearance diversity, from visible facial features in less dense scenes to blob-like people in highly dense crowds, without reliable handcrafted specialty criteria.
Method
IG-CNN recursively copies a base density regressor, differentially trains child regressors on jointly clustered data, and routes test patches to leaf experts with a classifier.
Results
IG-CNN reports better count accuracy on standard benchmarks, including the lowest count error on UCF CC 50 and significant improvements over the base CNN at higher growth levels.
Takeaways & Limitations
The method automatically mines progressively finer crowd specialties and adapts model capacity to training-data complexity.
Takeaways & Limitations
If learned specialties lack generalizable features, expert-classifier performance may decay on the test set.
Abstract
from arXiv · showhide
Automated counting of people in crowd images is a challenging task. The major difficulty stems from the large diversity in the way people appear in crowds. In fact, features available for crowd discrimination largely depend on the crowd density to the extent that people are only seen as blobs in a highly dense scene. We tackle this problem with a growing CNN which can progressively increase its capacity to account for the wide variability seen in crowd scenes. Our model starts from a base CNN density regressor, which is trained in equivalence on all types of crowd images. In order to adapt with the huge diversity, we create two child regressors which are exact copies of the base CNN. A differential training procedure divides the dataset into two clusters and fine-tunes the child networks on their respective specialties. Consequently, without any hand-crafted criteria for forming specialties, the child regressors become experts on certain types of crowds. The child networks are again split recursively, creating two experts at every division. This hierarchical training leads to a CNN tree, where the child regressors are more fine experts than any of their parents. The leaf nodes are taken as the final experts and a classifier network is then trained to predict the correct specialty for a given test image patch. The proposed model achieves higher count accuracy on major crowd datasets. Further, we analyse the characteristics of specialties mined automatically by our method.
1. Introduction
Crowd counting is difficult because crowd appearance and visible features vary sharply with density, while manually defined specialties are dataset- and model-dependent. IG-CNN addresses this by growing a hierarchy of automatically specialized regressors and routing patches to the appropriate expert.
- Motivation: Crowd counting is difficult because dense scenes reduce people to blobs, while less dense scenes expose more facial or body features.Occlusion, pose changes, and viewpoint variation further compound the diversity.
- Motivation: Typical CNN regressors trained across all densities vary widely in performance, overestimating sparse crowds and underpredicting dense crowds.Uneven density distributions in datasets can contribute to this variation.
- Motivation: Specialized regressors perform better on their own crowd types, but defining specialties and division thresholds manually is difficult and can produce suboptimal solutions.Prior mixture-of-experts approaches do not work well in this scenario.
- Proposed approach: IG-CNN grows from a base density regressor by recursively copying parent networks, differentially training child regressors on jointly formed clusters, and using leaf nodes as experts.A classifier routes each test image patch to the appropriate expert.
- Proposed approach: The method provides a hierarchical clustering procedure that jointly creates image clusters and neural-network experts without handcrafted dataset-dependent specialization criteria.The resulting system is designed to adapt and grow with dataset complexity.
2. Previous Work
Previous crowd-counting work includes detection, density regression, multi-scale and multi-column CNNs, and progressively growing or specialty-based networks. These approaches motivate a CNN tree that learns progressively finer specialties without requiring manually supplied specialty information.
- Crowd counting: Detection-based crowd counters rely on head or body features but fail in highly dense crowds where discriminative features are absent.This limitation has motivated regression-based methods.
- Crowd counting: CNN crowd-counting methods predict density maps, counts, or iterative corrections, with density-map training helping preserve feature detectors.Other approaches combine density and count losses or use feedback mechanisms.
- Crowd counting: Multi-scale and multi-column CNNs address scale variability using different input scales, receptive fields, or network depths.Examples include architectures pairing shallow networks for dense crowds with deeper networks for sparse crowds.
- Growing networks: Growing-network research shows that neural models can incrementally enlarge capacity through data-dependent neurons, added layers, or added width.These ideas span supervised, unsupervised, transfer-learning, and developmental settings.
- Specialization-based methods: Specialization methods use class hierarchies, generalist-specialist branches, or CNN trees, but some require coarse labels or other specialty information.The paper targets automatic specialty discovery for crowd regression.
3. Our Approach
IG-CNN addresses crowd diversity by progressively growing a tree of specialized CNN density regressors through differential training. A classifier selects the appropriate leaf expert for each test patch, while training and splitting are guided by count-based validation performance.
- Creating Experts with Hierarchical Differential Training: IG-CNN begins with a base CNN density regressor trained on the full dataset, then recursively replicates and specializes regressors on subsets of training data.Each child initially copies its parent’s weights, and differential training jointly forms clusters and expert regressors.
- Creating Experts with Hierarchical Differential Training: For each training patch, only the child regressor with the lower count error is fine-tuned, minimizing the oracle loss of the regressor set.Ties are assigned to the first regressor, allowing differentiation to develop progressively.
- Growing CNN Architecture: The resulting CNN tree has leaf regressors specialized for distinct crowd subsets, while all regressors retain the same architecture.Specialty subsets can be skewed and depend on both the dataset and the regressors.
- Growing CNN Architecture: At test time, an expert classifier routes each image patch to a specialized regressor, whose predictions are combined over overlapping regions.The classifier is trained from labels assigned to the regressor with minimum count error for each patch.
- Pretraining of Base CNN: The base regressor predicts density maps using l2 loss, whose density values can be summed to obtain crowd counts.Ground-truth maps are generated by placing normalized Gaussians at annotated head locations, and training uses stochastic gradient descent with momentum.
- Training Algorithm for IG-CNN: Tree growth stops when Actual MAE on a fixed validation set no longer improves, whereas Oracle MAE measures performance assuming the correct expert is always selected.The classifier-dependent Actual MAE therefore evaluates the complete routing-and-regression system rather than ideal expert assignment alone.
4. Experiments
IG-CNN is evaluated by routing test-image patches to crowd-type-specialized regressors across three datasets. It achieves strong counting performance, including lower error on UCF CC 50 and comparable performance on the challenging WorldExpo’10 dataset.
- Evaluation scheme: Test-image patches are routed by an expert classifier to regressors specialized for specific crowd types.The evaluation uses the classifier to select regressors for each patch.
- Shanghaitech dataset: IG-CNN grows to three levels and eight expert regressors for both Part A and Part B of Shanghaitech.Table 1 compares its performance with other models on the two dataset partitions.
- Shanghaitech dataset: IG-CNN outperforms all other methods on Shanghaitech Part B in both MAE and MSE, and achieves better count accuracy on Part A.Its Part A advantage over CP-CNN is narrow, while CP-CNN’s reported baseline was improved using adversarial training.
- UCF CC 50 dataset: 4.4 points lower MAE is reported on UCF CC 50, with comparable MSE performance.IG-CNN uses two hierarchical growth levels, producing four expert regressors on this challenging dataset.
- WorldExpo’10 dataset: WorldExpo’10 is challenging because sparse crowds provide little density variability, limiting the model’s ability to generate experts for different crowd types.Despite this limitation, IG-CNN shows comparable performance to other models.
5. Analysis and Ablations
Hierarchical growth improves oracle accuracy and mines crowd specialties, but deeper trees make expert classification harder and can worsen final MAE. IG-CNN outperforms alternative specialization strategies while revealing density-related expert groupings.
- 5.1. Effect of Growing: Deeper IG-CNN levels substantially reduce oracle MAE, but gains saturate after level 3 as expert-classifier errors increase.At higher levels, fewer training samples per expert contribute to switching errors.
- 5.2. Expert Specialty Characteristics: Crowd-count distributions indicate that automatically mined specialties separate patches by density-related characteristics.Patches with few people tend to one regressor, while denser patches distribute across other experts.
- 5.3. Hierarchical Training Vs Baseline Methods: IG-CNN achieves lower MAE than a standard mixture-of-experts model, whose regressors show insufficient specialization.The comparison uses a VGG-16 gating CNN with eight regressors initialized from the base CNN.
- 5.3. Hierarchical Training Vs Baseline Methods: Hierarchical IG-CNN training yields better performance than N-way differential training with comparable expert counts.Both the oracle loss and final expert-classifier performance are reported as inferior for N-way differential training.
6. Conclusion
IG-CNN addresses crowd-density diversity by expanding a base density regressor into a recursively specialized CNN tree. Its leaf experts are selected by a classifier, and the resulting specialties correlate with observable crowd characteristics such as density.
- 6. Conclusion: IG-CNN recursively splits copied child regressors into finer experts using differential training, forming a CNN tree without manually specified specialty criteria.An expert classifier predicts the appropriate leaf expert for each test patch.
- 6. Conclusion: The model achieves better performance on standard crowd-counting benchmarks and mines specialties correlated with observable crowd density.