Source-linked AI summary
Locality-constrained Spatial Transformer Network for Video Crowd Counting
Yanyan Fang, Biyun Zhan, Wandi Cai, Shenghua Gao, Bo Hu
TL;DR
Video crowd counting can use consistency across neighboring frames, but changes in density maps and crowd membership complicate estimation. LSTN relates frame density maps through a locality-constrained transformer and is reported effective across the evaluated datasets; the paper also introduces FDST.
Problem
Video crowd counting must account for density-map changes from translation, rotation, and scaling, as well as count changes from people entering, leaving, or being occluded.
Source: §1 · p. 1Method
LSTN estimates a density map for each frame, then uses a locality-constrained spatial transformer to relate maps between neighboring frames.
Source: §1 · p. 2, §3.2 · p. 3Results
LSTN is reported to achieve the best performance on FDST, outperform ConvLSTM on UCSD, and reach state-of-the-art MAE and MSE on Mall.
Source: §4.2 · p. 4, §4.3 · p. 5, §4.4 · p. 5Takeaways & Limitations
The authors conclude that LSTN is effective for video crowd counting and present FDST as a large-scale dataset for performance evaluation.
Source: §5 · p. 6Expo’10 is unsuitable for evaluating this method because its annotations are at 2 fps and neighboring-frame populations change substantially.
Source: §3.4 · p. 4
Abstract
from arXiv · showhide
Compared with single image based crowd counting, video provides the spatial-temporal information of the crowd that would help improve the robustness of crowd counting. But translation, rotation and scaling of people lead to the change of density map of heads between neighbouring frames. Meanwhile, people walking in/out or being occluded in dynamic scenes leads to the change of head counts. To alleviate these issues in video crowd counting, a Locality-constrained Spatial Transformer Network (LSTN) is proposed. Specifically, we first leverage a Convolutional Neural Networks to estimate the density map for each frame. Then to relate the density maps between neighbouring frames, a Locality-constrained Spatial Transformer (LST) module is introduced to estimate the density map of next frame with that of current frame. To facilitate the performance evaluation, a large-scale video crowd counting dataset is collected, which contains 15K frames with about 394K annotated heads captured from 13 different scenes. As far as we know, it is the largest video crowd counting dataset. Extensive experiments on our dataset and other crowd counting datasets validate the effectiveness of our LSTN for crowd counting.
1. INTRODUCTION
The introduction frames video crowd counting as a way to use consistency across neighboring frames, while noting that changes in appearance, movement, and occlusion complicate density estimation. It presents LSTN and the FDST dataset as contributions to this problem.
1. INTRODUCTION
- Crowd movement is described as predictable and consistent, motivating use of spatial-temporal consistency across neighboring frames for more robust video counting.The cited prior work is invoked in support of predictable crowd trajectories.Source: §1 · p. 1
- Prior video methods using LSTM or ConvLSTM accumulate history features implicitly, which may hurt current-frame density estimation when crowd identities change through entry, exit, or occlusion.The introduction links this problem to the difference between people represented in history frames and those in the current frame.Source: §1 · p. 1
- LSTN explicitly models spatial-temporal correlations between neighboring frames, using block similarity to weight differences between warped and ground-truth density maps.The blockwise design accommodates people walking in or out of view or being occluded, and avoids relying on irrelevant history frames.Source: §1 · pp. 1–2
- 15,000 frames, 394,081 annotated heads, and 13 scenes make FDST a larger, frame-wise annotated video crowd-counting dataset than WorldExpo’10.WorldExpo’10 is described as containing 3,980 frames and 199,923 annotated heads, with annotations only every 30 seconds.Source: §1 · p. 2
- The paper’s stated contributions are LSTN, the frame-wise annotated FDST dataset, and experiments validating the model for video crowd counting.The network first estimates a density map for each frame, then relates neighboring-frame maps.Source: §1 · pp. 1–2
2. RELATED WORK
Related work covers CNN-based density-map estimation for single images, recurrent methods for video, and spatial transformers. The paper distinguishes LST by explicitly relating density maps across neighboring frames rather than implicitly accumulating history features.
Crowd counting for single image
- CNN methods estimate density maps for single images, but can produce inconsistent head counts across neighboring video frames.Architectures discussed include MCNN, Hydra CNN, Switch-CNN, and CSRNet.Source: §2 · p. 2
Video crowd counting
- LSTM and ConvLSTM methods integrate current-frame features with history, but may be affected by irrelevant history and do not explicitly model neighboring-frame spatial-temporal dependencies.The cited video approaches include ConvLSTM and LSTM.Source: §2 · p. 2
Spatial transformer network (STN)
- Unlike prior spatial-transformer use for single-image crowd counting [18], this work uses spatial transformation to relate density maps between neighboring video frames.Spatial transformers were introduced as differentiable modules for learning input-to-output spatial transformations.Source: §2 · p. 2
3. OUR APPROACH
LSTN estimates a density map for each frame, then uses a locality-constrained spatial transformer to predict the next frame’s map. Its blockwise similarity weighting accommodates changes in population between neighboring frames while exploiting their spatial-temporal consistency.
3.1. Density map regression module
- Each annotated head is represented as a delta function smoothed by a 2D Gaussian kernel, and VGG-16 estimates each frame’s density map.Pixels near annotated points have higher probability of belonging to a head.Source: §3.1 · p. 3
3.2. LST module
- LST divides frames into H × W blocks and weights next-frame density-map consistency by corresponding-block similarity, allowing more difference for changed or occluded populations.The method describes scaling and translation when people move toward or away from the camera, and rotation with camera motion.Source: §3.2 · pp. 3–4
3.3. Loss function
- The training objective combines density-map regression and LST losses, and testing integrates each estimated frame’s density map to obtain head counts.The parameter λ balances the two losses.Source: §3.3 · p. 4
3.4. Implementation details
- Training uses Adam at 1e-8 with batch size 5; frames are resized to 640 × 360, W = 2, and H = 1 for Mall and FDST or 2 for UCSD.The density-map module is pretrained before fine-tuning the whole network while fixing VGG-16’s first 10 layers.Source: §3.3 · p. 4, §3.4 · p. 4
- Expo’10 is unsuitable for evaluating this method because its 2-fps annotations and marked ROIs coincide with large population changes between neighboring frames.Source: §3.4 · p. 4
4. EXPERIMENTS
The experiments evaluate crowd-counting accuracy with MAE and MSE across FDST, UCSD, and Mall, and test the contribution of LST’s similarity term. LSTN is reported to perform best on FDST, outperform ConvLSTM on UCSD, achieve state-of-the-art performance on Mall, and benefit from the similarity term across the evaluated datasets.
4.1. Evaluation metric
- Performance is evaluated using mean absolute error (MAE) and mean squared error (MSE) over testing frames.For each frame, z_i is the actual count and ẑ_i is the estimated count; T is the total number of testing frames.Source: §4.1 · p. 4
4.2. Fudan-ShanghaiTech Video Crowd counting dataset
- LSTN achieves the best performance on FDST, and its improvement over the version without LST supports the effectiveness of LST.The comparison uses a dataset split of 9,000 training frames and 6,000 testing frames; Table 2 reports the results.Source: §4.2 · p. 4, Table 2 · p. 5
4.3. The UCSD dataset
- LSTN outperforms the ConvLSTM-based method on UCSD.Frames are resized to 952 × 632 following.Source: §4.3 · p. 5
4.4. The Mall dataset
- On Mall, LSTN achieves state-of-the-art performance in both MAE and MSE.The evaluation follows the dataset’s train-test setting in.Source: §4.4 · p. 5
4.5. The importance of similarity term in LST
- Adding the similarity term improves video crowd-counting performance on UCSD, Mall, and FDST.The term weights the difference between warped and ground-truth density maps according to the similarity of temporal-neighboring blocks.Source: §4.5 · p. 5, Table 5 · p. 5
5. CONCLUSION
LSTN relates neighboring frames’ density maps through similarity-weighted block comparisons, while FDST provides a large-scale dataset for evaluation. Experiments validate LSTN’s effectiveness for video crowd counting.
5. CONCLUSION
- Experiments validate LSTN’s effectiveness for video crowd counting.The conclusion reports effectiveness without specifying benchmark values.Source: §5 · p. 6
- LSTN estimates each frame’s density map, then relates neighboring maps using similarity-weighted block differences.The block-wise weighting accounts for people walking in or out and occlusions.Source: §5 · p. 6
- For performance evaluation, the authors describe FDST as, as far as they know, the largest video crowd-counting dataset in terms of both the number of scenes and frames.Source: §5 · p. 6