Source-linked AI summary
Improved Anomaly Detection in Crowded Scenes via Cell-based Analysis of Foreground Speed, Size and Texture
Vikas Reddy, Conrad Sanderson, Brian C. Lovell
TL;DR
Crowded-scene anomaly detection is difficult because tracking can fail under occlusion and motion-only methods may miss size- or appearance-based anomalies. The paper uses foreground-segmented, cell-based analysis of motion, size, and texture, with independent feature modelling and foreground-only optical flow. On the UCSD Anomaly dataset, it reports better results than MPPCA, social force, and MDT, while processing substantially faster than MDT.
Problem
Crowded scenes make tracking unreliable under occlusion and overlap, while motion-only methods may miss anomalies due to object size or appearance.
Method
The method segments foreground objects, divides frames into non-overlapping cells, and independently analyses foreground motion, size, and texture, including foreground-only optical flow.
Results
The proposed method obtains better results than MPPCA, social force, and MDT, with more accurate anomaly localisation than MDT and several-orders-of-magnitude faster processing.
Takeaways & Limitations
The approach combines accurate crowded-scene anomaly localisation with substantially lower computational cost than MDT.
Takeaways & Limitations
A pedestrian entering a region with minimal or no activity during training can be falsely classified as anomalous regardless of observed features.
Abstract
from arXiv · showhide
A robust and efficient anomaly detection technique is proposed, capable of dealing with crowded scenes where traditional tracking based approaches tend to fail. Initial foreground segmentation of the input frames confines the analysis to foreground objects and effectively ignores irrelevant background dynamics. Input frames are split into non-overlapping cells, followed by extracting features based on motion, size and texture from each cell. Each feature type is independently analysed for the presence of an anomaly. Unlike most methods, a refined estimate of object motion is achieved by computing the optical flow of only the foreground pixels. The motion and size features are modelled by an approximated version of kernel density estimation, which is computationally efficient even for large training datasets. Texture features are modelled by an adaptively grown codebook, with the number of entries in the codebook selected in an online fashion. Experiments on the recently published UCSD Anomaly Detection dataset show that the proposed method obtains considerably better results than three recent approaches: MPPCA, social force, and mixture of dynamic textures (MDT). The proposed method is also several orders of magnitude faster than MDT, the next best performing method.
1. Introduction
Video anomaly detection commonly models normal activity, but crowded scenes challenge tracking-based methods because occlusion and overlap make trajectories unreliable. The proposed approach avoids tracking, suppresses background dynamics through foreground analysis, and combines motion, size, and texture evidence at cell locations.
- Motivation: Anomaly detection is generally framed as binary classification using training examples from the normal class.Typical systems model expected activity and label outliers as anomalous.
- Limitations of Existing Methods: Tracking-based methods are suitable for sparse scenes but struggle in crowded environments because occlusion and object overlap disrupt reliable trajectories.
- Limitations of Existing Methods: Motion-only approaches may miss anomalies caused by object size or appearance, motivating models that also capture spatial and temporal scene variation.
- Proposed Direction: The proposed method targets crowded scenes by segmenting foreground objects, dividing frames into cells, and analysing foreground motion, size, and texture.Optical flow is computed only for foreground pixels to refine motion estimation while ignoring irrelevant background dynamics.
- Proposed Direction: Motion, size, and texture are analysed independently, then combined to classify each cell as normal or anomalous and infer anomaly types.
2. Proposed Algorithm
The algorithm extracts foreground-object features from non-overlapping cells, models normal dynamics separately by feature type, and classifies cells with sequential checks. Spatio-temporal post-processing then reduces isolated noise in the anomaly masks.
- Feature Extraction: Feature extraction splits input images into non-overlapping cells and computes foreground-object features for motion, size, and texture.
- Model Estimation: Normal dynamics are modelled separately for each feature type at every cell location.
- Cell Classification: Cells are classified sequentially using a speed check followed by a size-and-texture check when required.The second classifier is skipped when the speed check already identifies an anomaly.
- Cell Classification: Low-likelihood size observations are further evaluated with texture evidence to validate whether a cell is anomalous.
- Post-processing: Spatio-temporal post-processing reduces isolated random noise in the generated anomaly masks.
2.1. Feature Extraction
The method extracts motion, size, and texture features from foreground-containing cells after segmentation, adapting each feature to crowded-scene challenges. Foreground-only motion estimation, neighborhood-based size estimation, and texture filtering address tracking failures, overlap, occlusion, and false alarms.
- Feature extraction: Foreground segmentation filters distractions, and each frame is partitioned into non-overlapping cells for motion, size, and texture extraction.The analysis uses foreground masks to identify relevant cells and restrict feature computation to foreground objects.
- Motion: Foreground-only optical flow estimates each cell’s motion, then temporal averaging smooths the resulting feature.The iterative Lucas-Kanade algorithm computes pixel displacement between consecutive frames; flow components and foreground-pixel counts define the average motion.
- Size: Neighborhood-weighted foreground occupancy approximates object size when overlap and occlusion make connected components ineffective.A 3×3 Gaussian mask emphasizes the center cell while reducing contributions from neighboring cells that may contain other objects.
- Texture: Size analysis can raise sensitivity but may increase false alarms when nearby people merge into a large-looking foreground object.Texture is introduced to increase selectivity in such cases.
- Texture: Four-orientation Gabor-wavelet responses form a 4D texture descriptor, collected only from cells containing foreground pixels.The orientations are 0, 45, 90, and 135 degrees, and restricting collection reduces background modeling.
2.2. Scalable Semi-Parametric Model Estimation
The method models scalar motion and size features with a smoothed, discretized density estimate, while texture descriptors use an adaptively grown codebook and correlation-based similarity. These choices address scalability, high-dimensional storage, and contrast-sensitive distance issues.
- Parametric models are unsuitable because surveillance-scene feature distributions have changing, randomly varying numbers of modes.
- Motion and size are modelled with a smoothed histogram computed by Gaussian kernel density estimation at discrete points.Training samples are temporarily stored, then discarded after the probability mass function is computed.
- The discretized density estimate trades accuracy against computation by evaluating continuous variables only at fixed points.The step size, valid range, sample count, and Gaussian bandwidth determine the resulting normalized probability mass function.
- Direct density estimation is impractical for 4D texture descriptors because covering the feature space requires prohibitive storage across cell locations.Even 20 equally spaced points per dimension produce 20^4 points in 4D space.
- Pearson correlation and an online adaptively grown codebook model texture patterns while reducing storage and contrast-related distance problems.A new vector updates its best-matching entry when correlation exceeds 0.9; otherwise, it expands the codebook.
2.3. Cell Classification
Each cell is classified using sequential motion, size, and texture checks. Motion detects departures from normal speeds, while texture verifies size-based anomalies and is restricted to foreground-containing cells.
- Each cell is checked by up to two sequential classifiers, and classification stops when the first classifier identifies an anomaly.
- A cell is anomalous when its motion probability is below threshold T, detecting speeds slower or faster than normal.The motion probability model can represent several normal speeds.
- A cell is also anomalous when its size probability is below T and its closest texture-codebook correlation is below 0.9.The closest-match correlation is the maximum correlation between the texture descriptor and any codebook entry.
- Texture verifies size anomalies because size alone cannot distinguish one large object from multiple small objects.
- Texture is computed only for foreground-containing cells and combined with size because thin foreground edges can produce irrelevant texture measurements.
2.4. Spatio-Temporal Post-Processing
Spatio-temporal post-processing removes isolated or intermittent cell alarms by consulting neighboring cells across space and time. The procedure retains an anomaly only when neighboring evidence supports it.
- An initially anomalous cell is verified using its immediate spatial and temporal neighbors to suppress noise-driven alarms.
- A cell remains anomalous only if at least two cells in each temporal plane t-1, t, and t+1 are also anomalous.Otherwise, it is re-classified as normal.
- Figure 3 illustrates anomaly detection and localisation results highlighted in red for the Ped1 and Ped2 UCSD subsets.
3. Experiments
Experiments evaluated the method on the UCSD dataset using frame-level detection and within-frame localisation, comparing it with social force, MPPCA, and MDT. At the EER, the method outperformed the alternatives on both evaluations, while running substantially faster than MDT.
- Dataset: The UCSD evaluation covered crowded Ped1 and Ped2 surveillance scenes containing anomalies such as skateboarders, bikers, vehicles, carts, and lawn walking.Ped1 images are 238×158 pixels, while Ped2 images are 360×240 pixels.
- Evaluation protocol: The prescribed protocol measured frame-level anomaly detection and within-frame anomaly localisation, counting localisation as correct when at least 40% of detected anomaly pixels matched ground truth.The localisation evaluation used marked anomalous regions in 10 Ped1 and 9 Ped2 test sequences.
- Baselines: The method was compared with social force, MPPCA, and mixture of dynamic textures (MDT), with results reported using false negative rate rather than the true positive rate used in the source comparison.Social force and MPPCA use optical-flow features, whereas MDT uses appearance and scene-dynamics features.
- Quantitative results: At the EER, the proposed method outperformed the other methods in both frame-level detection and within-frame localisation, most notably on anomaly localisation.EER is the point where false negative and false positive rates are equal.
- Efficiency: 12 fps (720 frames per minute) was achieved on 240×160 images, compared with MDT’s 2.4 frames per minute.The implementation ran on a standard 3 GHz PC, making it several orders of magnitude faster than MDT.
- Observed errors: A false positive occurred because a pedestrian entered a cell with minimal or no activity during training, causing any foreground object there to be considered anomalous.This error was independent of the observed features during testing.
4. Main Findings and Future Directions
The proposed crowded-scene anomaly detector combines foreground-restricted, cell-level motion, size, and texture analysis, outperforming recent methods while remaining computationally efficient. Future work targets richer motion features, long-term adaptation, and scene-dependent cell sizing.
- Main Findings: Foreground-only optical flow provides a more precise object-motion estimate while suppressing dynamic background effects.The method segments foreground regions before computing optical flow, rather than using all pixels or fixed pixel locations.
- Main Findings: Cell-level motion, size, and texture features are modelled independently for precise anomaly detection.Motion and size use approximated kernel density estimation, while texture uses an online adaptively grown codebook.
- Main Findings: The method outperforms MPPCA, social force, and MDT on the UCSD Anomaly dataset, with considerably more accurate localisation and several-orders-of-magnitude faster execution than MDT.The reported comparison covers annotated surveillance videos and identifies MDT as the next best-performing method for localisation.
- Future Directions: Future work includes more descriptive motion features to detect events such as wrong-way traffic and adaptive model updates for changing context.The paper gives day-versus-night traffic density as an example of context that may change over time.
- Future Directions: Cell-size effects remain to be analysed under changes in image resolution, perspective, view angle, and object scale across a scene.The paper suggests larger cells where objects appear large and smaller cells where they appear small.
- Main Findings: Across frame-level detection and within-frame localisation, the proposed method outperforms other approaches at equal error rate, especially for localisation.Figure 4 reports this pattern on UCSD Ped1 and Ped2 evaluations.