Source-linked AI summary
Spatiotemporal Modeling for Crowd Counting in Videos
Feng Xiong, Xingjian Shi, Dit-Yan Yeung
TL;DR
CNN-based crowd-counting methods process video frames independently, leaving neighboring-frame temporal correlations unused. The paper applies ConvLSTM and bidirectional ConvLSTM to capture spatial and temporal dependencies, with experiments across four datasets showing improved or comparable performance and transfer-learning robustness.
Problem
CNN-based crowd-counting methods treat video frames independently and ignore strong temporal correlations between neighboring frames.
Method
The paper uses ConvLSTM for crowd counting and extends it to bidirectional ConvLSTM to capture spatial, temporal, and long-range information in both directions.
Results
The models outperform existing methods on UCF CC 50, Mall, and WorldExpo, achieve comparable results on UCSD, and bidirectional ConvLSTM achieves state-of-the-art performance in both transfer-learning tasks.
Takeaways & Limitations
Explicitly exploiting temporal information has a clear advantage on three datasets, while the model remains robust when generalizing from previous learning experience under transfer learning.
Takeaways & Limitations
WorldExpo scene 3 contains labels that omit many people whose heads are hidden, creating a potential problem for evaluating spatiotemporal models.
Abstract
from arXiv · showhide
Region of Interest (ROI) crowd counting can be formulated as a regression problem of learning a mapping from an image or a video frame to a crowd density map. Recently, convolutional neural network (CNN) models have achieved promising results for crowd counting. However, even when dealing with video data, CNN-based methods still consider each video frame independently, ignoring the strong temporal correlation between neighboring frames. To exploit the otherwise very useful temporal information in video sequences, we propose a variant of a recent deep learning model called convolutional LSTM (ConvLSTM) for crowd counting. Unlike the previous CNN-based methods, our method fully captures both spatial and temporal dependencies. Furthermore, we extend the ConvLSTM model to a bidirectional ConvLSTM model which can access long-range information in both directions. Extensive experiments using four publicly available datasets demonstrate the reliability of our approach and the effectiveness of incorporating temporal information to boost the accuracy of crowd counting. In addition, we also conduct some transfer learning experiments to show that once our model is trained on one dataset, its learning experience can be transferred easily to a new dataset which consists of only very few video frames for model adaptation.
1. Introduction
Crowd counting estimates people in images or video, with ROI counting focusing on people within a region at a given time. Although CNN methods perform well, they process video frames independently and miss temporal correlations; the paper addresses this with spatiotemporal modeling.
- Crowd counting estimates the number of people in still images or video for applications including surveillance, traffic control, and emergency management.
- ROI counting estimates people within a region at a given time, whereas LOI counting counts people crossing a detecting line over a period.
- Regression-based methods learn mappings from visual features to crowd counts or density maps and have achieved state-of-the-art performance in crowded scenes.
- CNN-based methods learn features end to end and achieve strong dataset results, but treat video frames as independent still images.
- The proposed ConvLSTM variant captures spatial and temporal dependencies to use information from video sequences for crowd counting.
2. Related Work
Prior work established CNN-based density-map regression and ConvLSTM-based spatiotemporal modeling in other sequence tasks. These developments motivate applying convolutional recurrent structures to crowd counting.
- CNN crowd-counting methods replaced handcrafted features with learned end-to-end representations, including multi-column and stage-wise architectures.
- Density-map regression preserves crowd location information and allows counts for arbitrary regions through integration over the density function.
- ConvLSTM extends FC-LSTM with convolutional input-to-state and state-to-state connections, preserving locality for spatiotemporal data.
- ConvLSTM-based models have also been used for motion prediction, pixel-level prediction in natural videos, and speech recognition.
3. Our Crowd Counting Method
The method estimates crowd density maps from annotated person points and models video sequences with ConvLSTM to capture spatial and temporal dependencies. A bidirectional extension uses information in both temporal directions, while ConvLSTM-nt removes temporal connections for comparison.
- 3.1. Crowd density map: Crowd density maps are formed by summing normalized Gaussian kernels centered at annotated person locations.The density map preserves crowd-location information, and its integral estimates the crowd count.
- 3.1. Crowd density map: Boundary-near annotations can place probability mass outside the image, causing small discrepancies between integrated density and crowd count.The paper notes that partially visible pedestrians may not need to be counted as whole people.
- 3.2. ConvLSTM model: ConvLSTM replaces fully connected state transitions with convolutions, preserving locality while modeling temporal correlations from neighboring spatial regions.Its inputs, states, and gates are spatial tensors, and outputs depend on local neighbors’ inputs and past states.
- 3.2. ConvLSTM model: The model maps consecutive video frames to corresponding estimated density maps, using four ConvLSTM layers and 1 × 1 filters for density-map prediction.Training uses Euclidean distance between estimated and ground-truth density maps; all models use four layers with 128, 64, 64, and 64 hidden states.
- 3.3. From ConvLSTM to bidirectional ConvLSTM: Bidirectional ConvLSTM processes sequences forward and backward, then combines the hidden sequences to access long-range information in both directions.ConvLSTM-nt removes connections between cells and serves as a gated CNN-like variant for assessing temporal information.
4. Experiments
Experiments across four datasets evaluate ConvLSTM variants for crowd-density estimation, showing benefits from temporal modeling and transfer learning under limited adaptation data.
- Datasets and evaluation: The study compares methods on UCF CC 50, UCSD, Mall, and WorldExpo’10, and also evaluates transfer between UCSD and Mall.
- UCF CC 50 dataset: UCF CC 50 contains only 50 highly diverse, extremely dense images, so only ConvLSTM-nt is evaluated without temporal information.Pedestrian counts range from 94 to 4,543, averaging 1,280.
- UCF CC 50 dataset: ConvLSTM-nt achieves the lowest MAE and MSE among methods without additional training data on UCF CC 50.Shang et al. uses additional data, making direct comparison unfair.
- UCSD dataset: Bidirectional ConvLSTM achieves comparable MAE and MSE with state-of-the-art methods on UCSD, while comparisons among variants show temporal information boosts performance.Bidirectional ConvLSTM also produces the density maps closest to ground truth in illustrative examples.
- WorldExpo dataset: On WorldExpo, bidirectional ConvLSTM has the lowest average MAE and the best result for scene 5, while temporal variants outperform one another in most scenes.Scene 3 contains labels that omit people whose heads are hidden, although motion reveals them in the video.
- Transfer learning experiments: In transfer learning, bidirectional ConvLSTM achieves state-of-the-art performance in both tasks and reaches an MAE of 2.63 with 50 adaptation frames from Mall.The source model is trained on 800 frames, while the target dataset contributes only 50 adaptation frames.
5. Conclusion
The models advance video crowd counting through joint spatiotemporal modeling, outperforming existing methods on three datasets and matching them on UCSD. The experiments also support transfer learning and motivate future active-learning extensions.
- The models outperform existing crowd-counting methods on UCF CC 50, Mall, and WorldExpo, while achieving comparable results on UCSD.
- On UCF CC 50, the model performs well on extremely dense crowd images even when temporal information is unavailable.
- On the Mall, WorldExpo, and UCSD datasets, explicitly exploiting temporal information shows a clear advantage.
- The transfer-learning experiments show that learning experience from one dataset can generalize to a new dataset with few video frames for adaptation.
- Future work will extend the model with active learning to query labels for less-confident regions and reduce labeling effort.