Source-linked AI summary
FCN-rLSTM: Deep Spatio-Temporal Neural Networks for Vehicle Counting in City Cameras
Shanghang Zhang, Guanhang Wu, João P. Costeira, José M. F. Moura
TL;DR
City-camera vehicle counting is difficult because these videos have low spatial and temporal resolution and high occlusion. The paper proposes FCN-rLSTM, which combines dense prediction with temporal modeling through residual learning, and reports improved counting accuracy, including TRANCOS MAE reduction from 5.31 to 4.21.
Problem
City-camera vehicle counting is challenging because videos have low spatial and temporal resolution and high occlusion.
Method
FCN-rLSTM connects FCN and LSTM through residual learning to jointly estimate vehicle density and vehicle count.
Results
FCN-rLSTM reduces TRANCOS MAE from 5.31 to 4.21 compared with baseline methods.
Takeaways & Limitations
The residual connection accelerates training by 5 times on average, while FCN-rLSTM can potentially extend to visual tasks requiring dense prediction and temporal correlation.
Takeaways & Limitations
The paper notes that existing methods fail to model temporal correlation in sequential traffic frames.
Abstract
from arXiv · showhide
In this paper, we develop deep spatio-temporal neural networks to sequentially count vehicles from low quality videos captured by city cameras (citycams). Citycam videos have low resolution, low frame rate, high occlusion and large perspective, making most existing methods lose their efficacy. To overcome limitations of existing methods and incorporate the temporal information of traffic video, we design a novel FCN-rLSTM network to jointly estimate vehicle density and vehicle count by connecting fully convolutional neural networks (FCN) with long short term memory networks (LSTM) in a residual learning fashion. Such design leverages the strengths of FCN for pixel-level prediction and the strengths of LSTM for learning complex temporal dynamics. The residual learning connection reformulates the vehicle count regression as learning residual functions with reference to the sum of densities in each frame, which significantly accelerates the training of networks. To preserve feature map resolution, we propose a Hyper-Atrous combination to integrate atrous convolution in FCN and combine feature maps of different convolution layers. FCN-rLSTM enables refined feature representation and a novel end-to-end trainable mapping from pixels to vehicle count. We extensively evaluated the proposed method on different counting tasks with three datasets, with experimental results demonstrating their effectiveness and robustness. In particular, FCN-rLSTM reduces the mean absolute error (MAE) from 5.31 to 4.21 on TRANCOS, and reduces the MAE from 2.74 to 1.53 on WebCamT. Training process is accelerated by 5 times on average.
1. Introduction
The paper targets vehicle counting in difficult citycam videos by combining dense spatial prediction with temporal modeling in an end-to-end FCN-rLSTM architecture. Its residual connection, Hyper-Atrous feature processing, and joint density-count learning improve training and counting accuracy.
- Problem: Citycam vehicle counting matters for traffic applications but is constrained by low frame rate, low resolution, high occlusion, and large perspective.These conditions produce varied vehicle scales and make counting especially difficult.
- Problem: Existing frame differencing, detection, motion, density estimation, and deep learning methods lose efficacy under citycam conditions.The paper identifies sensitivity to environmental changes, occlusion, resolution, frame rate, and perspective across these approaches.
- Approach: FCN-rLSTM combines FCN-based density prediction with LSTM temporal modeling in a residual-learning framework for sequential vehicle counting.The FCN maps pixels to density, while LSTM layers learn temporal dynamics and residual functions relative to per-frame density sums.
- Results: 5 times average training acceleration accompanies the residual FCN-LSTM connection, while TRANCOS MAE decreases from 5.31 to 4.21.The reported improvement is against the stated baseline methods on TRANCOS.
- Approach: Hyper-Atrous convolution combines feature maps while preserving resolution and improving foreground-background discrimination for density estimation.A subsequent 1×1 convolution re-weights the combined features, avoiding the need for foreground segmentation.
- Approach: Jointly learning vehicle density and vehicle count supplies additional supervision and improves both tasks in an end-to-end trainable network.The framework uses the related objectives to support better local optimization with fewer parameters.
2. Related Work
Related work spans traditional motion, detection, differencing, density, and deep learning approaches to vehicle counting. The paper identifies a shared gap: existing methods do not model temporal correlation in sequential traffic frames, while prior CNN-RNN work does not address object counting with FCN-LSTM residual learning.
- Traditional methods: Frame differencing methods are easy to implement but suffer from noise, abrupt illumination changes, and background changes.Their reliance on sequential-frame differences creates sensitivity to changing visual conditions.
- Traditional methods: Detection-based methods perform poorly in low-resolution and highly occluded videos, while motion-based methods fail with low frame rates and limited motion information.Both approaches depend on visual or temporal signals that citycam footage often lacks.
- Density estimation: Density estimation avoids detecting or tracking individual objects, but accuracy decreases with large perspective and varied vehicle scales.The reviewed formulation maps pixel features to object densities whose image-region integral gives the count.
- Deep learning: Deep learning methods improve counting performance, but reviewed approaches lack either FCN pixel-wise prediction or full-resolution density-map reconstruction.One cited method uses CNN features with ridge regression, while another FCN method produces density maps smaller than the input image.
- Open gap: Existing counting methods fail to model temporal correlation across sequential traffic frames.This limitation motivates adding recurrent temporal modeling to dense spatial prediction.
3. FCN-rLSTM for vehicle counting
FCN-rLSTM jointly estimates pixel-level vehicle density and global vehicle count by combining FCN and LSTM in an end-to-end residual-learning framework. Its Hyper-Atrous design preserves feature resolution while the multi-task objective supplies both density and count supervision.
- FCN-rLSTM architecture: FCN-rLSTM maps image pixels to vehicle density with an FCN and estimates global vehicle count from LSTM-based residual regression.The base count is the density-map integral, while the LSTM learns a residual count that is added to it.
- FCN-rLSTM architecture: Hyper-Atrous combination integrates atrous convolution with feature maps from different layers to preserve resolution and improve foreground-background discrimination.A subsequent 1×1 convolution re-weights the combined feature volume.
- FCN-rLSTM architecture: LSTM layers model temporal correlations across sequential vehicle-count frames using gated recurrent states designed to learn long-term dependencies.The described gates are forget, input, and output gates.
- Residual connection: The residual connection uses the summed density of each frame as a base count and learns the remaining difference through a fully connected layer.Compared with direct FCN–LSTM connection, this design is reported to ease training and increase counting accuracy.
- Multi-task learning: The network jointly supervises pixel-level density and global vehicle count through end-to-end training with density and count losses.The global count-loss weight λ is tuned for best accuracy, and simultaneous learning can train each task with fewer parameters.
- Training: FCN-rLSTM training optimizes the network parameters with batch-based Adam and backpropagation over image sequences and density-map labels.The algorithm outputs parameters for the FCN, LSTM, and fully connected layers.
4. Experiments
Experiments evaluate FCN-rLSTM across vehicle and pedestrian counting datasets, varied cameras, weather, perspectives, and temporal settings. Results show strong accuracy, robustness, and faster convergence, while TRANCOS highlights the importance of temporal consistency for selecting model configurations.
- WebCamT: FCN-rLSTM outperforms baseline approaches and alternative configurations on WebCamT across varied congestion states, perspectives, weather conditions, and times of day.The evaluation uses city-camera videos with low resolution, low frame rate, and high occlusion, covering downtown and parkway cameras.
- WebCamT: FCN-rLSTM estimates vehicle counts more accurately than FCN-HA, particularly reducing large errors caused by oversized vehicles.Figures 5 and 6 compare the two configurations on parkway and downtown cameras.
- WebCamT: FCN-rLSTM accurately counts vehicles across six downtown and parkway cameras over long time sequences.The camera positions are shown in Figure 7, while Figure 9 presents the corresponding counting results.
- WebCamT: FCN-rLSTM converges faster than FCN-HA and direct FCN-LSTM connections, with shading showing MAE over epochs and dark lines showing smoothed trends.The convergence comparison covers parkway and downtown cameras.
- TRANCOS: 4.21 MAE is achieved on TRANCOS, decreasing from 5.31 for Baseline 3 and from 10.99 for Baseline 2-Hydra.Because TRANCOS samples lack consistent temporal information, FCN-rLSTM cannot learn temporal patterns there; FCN-HA performs better, while FCN-rLSTM still outperforms all baselines.
- UCSD: FCN-rLSTM outperforms all baseline methods and FCN-HA on the UCSD pedestrian-counting dataset, supporting robustness across counting tasks.UCSD contains surveillance-camera frames with an ROI and is evaluated using separate training and test frames.
5. Discussion & Conclusion
FCN-rLSTM jointly estimates vehicle density and count by connecting FCN and LSTM through residual learning, achieving effective and robust performance across counting tasks and datasets. Its main scope limitation is that available memory restricts the unrolled frame window, preventing learning very long-term temporal information.
- FCN-rLSTM jointly estimates vehicle density and vehicle count by connecting FCN with LSTM in a residual learning framework.The residual connection references count regression to the sum of densities in each frame.
- The residual design avoids learning unreferenced functions and significantly accelerates network training.
- Extensive evaluations on different counting tasks and three datasets demonstrate the proposed methods’ effectiveness and robustness.
- The unrolled sequential-frame window is restricted by available memory capacity, preventing FCN-rLSTM from learning very long-term temporal information.