Source-linked AI summary
Plug-and-Play CNN for Crowd Motion Analysis: An Application in Abnormal Event Detection
Mahdyar Ravanbakhsh, Moin Nabi, Hossein Mousavi, Enver Sangineto, Nicu Sebe
TL;DR
Crowd abnormal-event detection is constrained by hand-crafted features and scarce abnormal examples for supervised CNN training. The paper uses pretrained CNN features, binary quantization, a TCP measure, and optical-flow fusion, reporting state-of-the-art-comparable results without fine-tuning.
Problem
Crowd abnormal-event detection relies on difficult hand-crafted features, while scarce abnormal examples make CNN fine-tuning prone to overfitting.
Method
The method tracks temporal variations in pretrained CNN features through binary maps and a TCP measure, then fuses them with optical flow for abnormality localization.
Results
The approach is comparable with state-of-the-art methods on challenging abnormality-detection datasets.
Takeaways & Limitations
Combining semantic CNN information with optical flow provides complementary appearance and motion patterns while avoiding additional fine-tuning costs.
Takeaways & Limitations
Errors mainly occur when abnormal objects are small, partially occluded, or moving at a normal speed.
Abstract
from arXiv · showhide
Most of the crowd abnormal event detection methods rely on complex hand-crafted features to represent the crowd motion and appearance. Convolutional Neural Networks (CNN) have shown to be a powerful tool with excellent representational capacities, which can leverage the need for hand-crafted features. In this paper, we show that keeping track of the changes in the CNN feature across time can facilitate capturing the local abnormality. We specifically propose a novel measure-based method which allows measuring the local abnormality in a video by combining semantic information (inherited from existing CNN models) with low-level Optical-Flow. One of the advantage of this method is that it can be used without the fine-tuning costs. The proposed method is validated on challenging abnormality detection datasets and the results show the superiority of our method compared to the state-of-the-art methods.
1. Introduction
The paper addresses limitations of hand-crafted crowd-motion features and scarce abnormal-event training data by using pretrained CNN semantics. It tracks temporal CNN-feature changes and combines them with optical flow to represent and localize crowd abnormalities without fine-tuning.
- Hand-crafted crowd features require task-specific prior knowledge that is difficult to define in complex surveillance scenes.
- Limited real-world abnormal-event examples make CNN fine-tuning prone to overfitting, motivating reuse of existing image-trained CNN models.
- Temporal changes in CNN features are used to capture motion properties even when consecutive image patches differ only slightly.
- The method combines a statistical Temporal CNN Pattern measure with low-level optical flow to provide complementary appearance and motion information.
- Method Overview: The pipeline extracts CNN binary maps, computes TCP over spatio-temporal blocks, and fuses the result with optical flow to localize abnormal segments.
- The authors report that the approach is comparable with state-of-the-art methods on common abnormality-detection datasets.
2. Binary Fully Convolutional Net (BFCN)
The Binary Fully Convolutional Net preserves spatial correspondence while converting high-dimensional CNN features into compact binary representations. These representations are produced without fine-tuning and support temporal pattern measurement and localization.
- 2.1. Frame-based Fully Convolutional Network: The architecture combines convolutional feature maps with binary-map representations of local features.
- 2.1. Frame-based Fully Convolutional Network: A Fully Convolutional Network preserves input-to-feature-map spatial relations, supporting localization while processing images of different sizes.
- 2.2. Binary Quantization Layer (BQL):: High-dimensional feature maps are hashed into compact binary codes to reduce computational cost and avoid specifying cluster centers in advance.
- 2.2. Binary Quantization Layer (BQL):: Iterative Quantization Hashing projects feature vectors into binary space, and its learned weights initialize the Binary Encoding Layer.
- 2.2. Binary Quantization Layer (BQL):: The Binary Quantization Layer is implemented as convolutional filters that transform feature vectors through a sigmoid and thresholding operation.
- 2.2. Binary Quantization Layer (BQL):: For each frame, the network returns a binary bitmap whose spatial structure supports localization, and sequential frame maps form the video representation.
- 2.2. Binary Quantization Layer (BQL):: Because abnormal-event data are scarce, experiments use the plugged-in binary layer without fine-tuning to avoid possible overfitting.
- 2.2. Binary Quantization Layer (BQL):: The BQL quantizes pool5 feature maps into 7-bit binary maps after the fully convolutional network.
3. Temporal CNN Pattern (TCP)
The TCP method represents crowd motion by tracking temporal changes in CNN-derived binary appearance patterns over overlapping video blocks. It measures local irregularity from prototype histograms and maps the resulting abnormality scores back to image regions.
- Overlapped Video Blocks: The method extracts overlapping video blocks from binary maps produced by an FCN, with each binary-map pixel representing a corresponding image patch.The middle patch of each block is used to indicate the block, and overlapping blocks provide finer temporal information.
- Histogram of Binary Codes: CNN feature patterns are quantized into binary prototypes whose temporal changes encode motion-related appearance variation.Consecutive frames are expected to have similar prototypes unless significant motion occurs.
- TCP Measure: Higher histogram uniformity indicates greater prototype diversity and increases the chance of abnormality, whereas concentrated distributions yield higher irregularity.The paper defines histogram irregularity through the non-uniformity of the prototype distribution.
- TCP Measure: The aggregated histogram represents the distribution of appearance codes across a video block, while its dominant prototype is the most frequent binary code.The TCP measure compares prototype samples with the dominant prototype using histogram-based differences and the L2 norm.
- TCP Map: TCP values are computed for all blocks, assigned to their temporally centered patches, and assembled into a frame-sized TCP map.The resulting map preserves the patchwise spatial arrangement of the binary map.
- Evaluation: The TCP representation is evaluated on UCSD and UMN datasets, with UCSD comparisons reporting ERR and AUC against prior methods.The evaluation includes frame-level anomaly detection and pixel-level anomaly localization on UCSD.
- Up-sampling TCP Maps: Upsampling assigns each patch’s TCP value to all corresponding image pixels to produce a map with the original frame resolution.This enables localization of the coarse abnormal region in the input frame.
4. Fusion with optical-flow Maps
The method fuses TCP maps with optical-flow maps to combine CNN-derived temporal appearance variation with frame-to-frame motion. The fused map assigns equal importance to both inputs and yields motion-segment maps across the video.
- Optical-flow Maps: Optical flow is extracted between consecutive frames and represented at the input-frame resolution.The resulting optical-flow maps are stacked over the video and organized into overlapping temporal blocks.
- Optical-flow Maps: Each aligned optical-flow block sums flow values over the corresponding spatial region across its temporal interval.This produces region-level flow values compatible with the TCP block representation.
- Feature Fusion: The method fuses optical-flow maps d_t and TCP maps c_t as mseg_t = αd_t + βc_t to form motion-segment maps.The importance factors control each map’s influence in the final representation.
- Feature Fusion: The implementation selects α = 0.5 and β = 0.5, giving equal influence to optical flow and TCP in the fused motion map.The fused maps are extracted for the entire video.
5. Experimental Results
The method is evaluated on UCSD and UMN crowd-abnormality datasets using frame-level detection and pixel-level localization protocols. Results are generally strong, while optical-flow fusion improves localization and errors occur with small, occluded, or normally moving abnormal objects.
- Evaluation setup: Evaluation uses UCSD and UMN datasets with both frame-level anomaly detection and pixel-level anomaly localization protocols.UCSD includes Ped1 and Ped2 subsets; UMN results compare the proposed TCP signal with Commotion Measure.
- Frame-level results: The UMN plots show TCP and Commotion Measure signals across 11 sequences against ground-truth abnormal-frame intervals.Each sequence begins with normal frames and ends with abnormality; the TCP frame indicator sums patch-level TCP values and is normalized to [0, 1].
- Overall results: Most cases were detected and localized correctly, including cars, bicycles, and skateboards, but some localization and detection results were slightly below state-of-the-art methods.The method achieved these results without additional learning costs.
- Error analysis: Missed detections commonly involved very small or partially occluded abnormal objects, or objects whose motion appeared normal in the scene.The paper gives a skateboard as an example of partial occlusion and a car moving at pedestrian speed as an example of normal-looking motion.
- Stream analysis: 93.6% AUC for TCP-only is slightly below 95.7% for the fused version in frame-level Ped1 evaluation.The comparison isolates the contribution of combining the TCP and optical-flow streams.
- Stream analysis: Pixel-level TCP-only performance dropped 9.3% relative to the fused version, showing the importance of optical flow for anomaly localization.The reported explanation is that optical flow refines abnormal segments using fine motion regions.
6. Discussion
The method captures crowd dynamics from temporal changes in CNN-derived binary patterns. Its TCP measure treats histogram uniformity as a local abnormality signal, enabling context-dependent detection.
- Architecture: The approach uses a pre-trained FCN followed by a binary quantization layer that produces 7-bit spatial binary maps without fine-tuning.The hashing-based quantization is trained once offline and reused across datasets, supporting the plug-and-play design.
- Temporal measure: TCP captures temporal changes by measuring irregularity in histograms of binary CNN patterns across consecutive frames.Consecutive frames are expected to have similar patterns unless a large semantic change occurs.
- Abnormality formulation: A flatter binary-pattern histogram indicates greater visual inconsistency and therefore a higher abnormality signal.This formulation is intended to handle context-dependent abnormal events.
7. Conclusions
The paper combines pre-trained CNN semantics, binary quantization, an unsupervised temporal-pattern measure, and optical flow for crowd anomaly detection and localization. Experiments report results comparable to state-of-the-art methods without additional learning costs.
- Method: A pre-trained FCN with a binary quantization layer provides spatially consistent, low-dimensional semantic embeddings for video frames.The binary layer is plugged into the network rather than learned through further fine-tuning.
- Method: An unsupervised TCP measure captures temporal CNN patterns, while optical-flow fusion supplies complementary appearance and motion information.The paper evaluates the combined approach on challenging crowd-abnormality datasets.
- Conclusion: Qualitative and quantitative evaluations show performance comparable to state-of-the-art methods.The paper identifies adding a trainable TCP layer and exploring alternative binary fully convolutional networks as future directions.