PaperlayerView exact revision on arXiv

Source-linked AI summary

Crowded Scene Analysis: A Survey

Teng Li, Huan Chang, Meng Wang, Bingbing Ni, Richang Hong, Shuicheng Yan

arXiv:1502.01812v1cs.CV

TL;DR

Crowded-scene analysis is difficult because occlusions, ambiguities, and complex crowd behavior challenge existing visual methods. This paper surveys state-of-the-art approaches across motion patterns, behavior recognition, and anomaly detection, while organizing relevant features, models, evaluations, and datasets. It concludes that no generally accepted solution yet exists and identifies multi-sensor fusion and other future directions.

  • Problem

    Visual occlusions, ambiguities, and complex crowd dynamics make automated analysis challenging, while anomaly detection remains an open problem with varied assumptions and objectives.

  • Method

    The paper conducts a focused survey of crowded-scene analysis, covering features, physical and computational models, algorithms, evaluation protocols, datasets, and three task areas.

  • Results

    The survey synthesizes methods for motion pattern segmentation, crowd behavior recognition, and anomaly detection, including their models, evaluations, and reported system performance.

  • Takeaways & Limitations

    Crowded-scene analysis has promising real-world applications, but current research still lacks a generally accepted solution; multi-sensor fusion is identified as a way to reduce confusion and improve accuracy.

Abstract

from arXiv · show

Automated scene analysis has been a topic of great interest in computer vision and cognitive science. Recently, with the growth of crowd phenomena in the real world, crowded scene analysis has attracted much attention. However, the visual occlusions and ambiguities in crowded scenes, as well as the complex behaviors and scene semantics, make the analysis a challenging task. In the past few years, an increasing number of works on crowded scene analysis have been reported, covering different aspects including crowd motion pattern learning, crowd behavior and activity analysis, and anomaly detection in crowds. This paper surveys the state-of-the-art techniques on this topic. We first provide the background knowledge and the available features related to crowded scenes. Then, existing models, popular algorithms, evaluation protocols, as well as system performance are provided corresponding to different aspects of crowded scene analysis. We also outline the available datasets for performance evaluation. Finally, some research problems and promising future directions are presented with discussions.

I. INTRODUCTION

Crowded scene analysis addresses the limitations of conventional surveillance and human monitoring in increasingly frequent, visually complex crowds. This survey focuses on high-level motion and behavior understanding, organizing recent work around three related tasks while also covering features, models, applications, and research directions.

  • Problems and Motivations: Crowded scenes create major monitoring challenges because many individuals, occlusions, ambiguities, and complex dynamics undermine conventional analysis methods.These difficulties affect both detection and tracking and increase computational complexity.
  • Problems and Motivations: Crowded scene analysis supports security, safety, public management, congestion analysis, anomaly detection, crowd statistics, and mass-gathering management.The surveyed applications include railway stations, shopping malls, music festivals, and sports events.
  • Problems and Motivations: The survey discusses challenges and technical advances in feature extraction, segmentation, and model learning for crowded-scene visual behavior analysis.It aims to provide comprehensive review and general discussions of these problems.
  • Problems and Motivations: The survey provides a focused review of recent high-level crowded-scene understanding for motion and behavior analysis, covering work not previously surveyed together.Crowd counting and density estimation are explicitly outside its scope.
  • Problems and Motivations: It divides the topic into motion pattern segmentation, crowd behavior recognition, and anomaly detection, treating the three aspects as closely related.The reviewed methods generally analyze behaviors or activities using motion features.
  • Problems and Motivations: Feature representation is summarized separately because it is shared across methods and forms an indispensable basis for the three analysis tasks.The paper also introduces crowd-dynamics knowledge and physical models before discussing task-oriented methods.

C. Organization of The Paper

The paper first establishes crowd concepts and dynamics models, then organizes crowded-scene analysis around features and task-specific methods. It distinguishes structured from unstructured scenes and macroscopic from microscopic analysis, while presenting continuum- and agent-based modeling approaches.

  • Background and Crowd Concepts: Crowded scenes are categorized as structured or unstructured according to whether participants move coherently or chaotically across directions and times.Structured scenes typically contain one main crowd behavior at each spatial location, whereas unstructured scenes contain multiple behaviors.
  • Background and Crowd Concepts: Crowd analysis operates at macroscopic and microscopic levels, respectively modeling global mass motion or individual pedestrian movements using collective information.The levels correspond to different granularities of crowd behavior.
  • Crowd Dynamics Models: Crowd dynamics studies how crowds form and move above critical density and how individual interactions influence crowd status.Earlier applications supported urban-infrastructure planning such as entrances and corridors.
  • Crowd Dynamics Models: Continuum-based models treat crowds as physical fluids and are better suited to medium- and high-density macroscopic behavior.They use density fields, partial differential equations, dynamic potentials, and velocity fields to represent or guide motion.
  • Crowd Dynamics Models: Agent-based models represent individuals as autonomous agents that sense environments and make decisions according to predefined rules, making them suitable for low-density microscopic analysis.The social force model represents pedestrian interaction forces and has reproduced specific crowd phenomena.
  • Crowd Dynamics Models: The social force model uses individual mass, actual and desired velocities, relaxation, and interaction forces to formulate pedestrian motion.The interaction force is defined as the sum of attraction and repulsive forces, and generalized versions are used in crowd behavior analysis.

B. Crowd Model in Video Analysis

Crowd models represent either holistic scene dynamics or interactions among individual agents, with both levels sometimes combined for video analysis. These models support recognition, classification, tracking, and behavior analysis in crowded scenes.

  • Applications: Physics-inspired crowd models have been used for recognition and classification, while crowd simulation findings have informed automated scene analysis.These approaches connect crowd simulation research with video-analysis tasks.
  • Macroscopic models: Macroscopic models treat high-density crowds as complex dynamic systems using holistic properties such as motion fields and dynamical potentials.Motion fields describe flow velocity, while potentials can represent quantities including density or pressure.
  • Microscopic models: Microscopic agent-based models represent individuals as autonomous agents whose interactions generate crowd behavior.The social force model assumes pedestrian interaction forces are significant for analyzing crowd behaviors.
  • Combined modeling: Global motion and individual movements can be jointly used, improving individual tracking while supplying basic units for holistic scene models.The two representation levels are complementary rather than mutually exclusive.
  • Feature representation: Motion information is the dominant feature basis, although scene structure, geometry, and viewing direction can also contribute.Feature representation is treated as a prerequisite for subsequent crowded-scene analysis tasks.

1) Optical Flow:

Optical flow provides dense, robust instantaneous motion measurements, but it lacks long-range temporal information and cannot fully represent changing flow structure. Particle flow and streak flow extend the temporal representation, while local spatio-temporal features address complex non-uniform motion.

  • Optical Flow: Optical flow does not capture long-range temporal dependencies or fully represent spatial and temporal flow properties.These missing properties can matter for applications requiring extended motion context.
  • Particle Flow: Particle flow moves a grid of particles through numerical integration, producing trajectories from initial positions to later positions.It has been applied to crowd segmentation and abnormal crowd behavior detection, but ignores spatial changes and incurs significant time delay.
  • Streak Flow: Streak flow uses streaklines to represent crowd motion over a time period and was introduced to improve flow representation for crowd video analysis.The paper compares optical flow, particle flow, and streak flow in a changing locally uniform flow field.
  • Local spatio-temporal features: Dense local spatio-temporal motion patterns model relationships within patches or cubes when highly variable local motion makes optical flow insufficient.These representations characterize motion as a whole through spatio-temporal distributions.

2) Motion Histogram:

Motion histograms and related descriptors represent local motion distributions, while trajectory and tracklet features provide more semantic individual motion information. Feature choice depends on density, occlusion, camera view, resolution, and scene structure.

  • Motion histograms: Orientation distribution functions model the probability density of motion orientation without retaining flow magnitude.Removing magnitude yields a simpler one-dimensional representation for motion-pattern learning.
  • Motion histograms: Multi-scale histograms of optical flow preserve motion and spatial context by extracting features from image patches or spatio-temporal cubes.Features from multiple spatial structures are concatenated for event representation.
  • Motion histograms: Spatio-temporal features show strong descriptive power and have been widely used in tasks including crowd anomaly detection.Their local motion distributions support motion understanding.
  • Trajectory/Tracklet: Trajectory and tracklet features are more semantic, but complete trajectories become difficult to obtain as crowd density and scene clutter increase.Tracklets provide shorter, more conservative trajectory fragments that are less likely to drift.
  • Feature selection: Optical-flow-like features suit wide-view outdoor scenes, trajectory or tracklet features suit higher-resolution indoor scenes, and local spatio-temporal features suit dense severe-occlusion settings.The recommended representation varies with field of view, density, occlusion, and target resolution.

A. Flow Field Model Based Segmentation

Flow-field and local-motion methods segment crowded-scene motion by modeling coherent dynamics or clustering motion features. They handle dense and unstructured scenes differently, with limitations involving spatial detail, overlapping patterns, density, and tracking noise.

  • Flow-field representations: Flow-field models represent crowds as time-dependent regions with qualitatively different dynamics and support motion-pattern segmentation.Methods include edge-based, graph-based, watershed, and related segmentation approaches.
  • Flow-field representations: Lagrangian particle dynamics examines particle clouds and uses Lagrangian coherent structures to locate boundaries between crowd segments.The approach maps coherent structures to boundaries separating flow regions with different dynamics.
  • Limitations: Flow-field methods succeed in dense scenes but may miss long-range patterns, overlapping labels, or lower-density behavior, while some clustering methods require tuning or suffer tracking noise.Particle flow ignores spatial changes, and flow-field segmentation can over-segment scenes as crowd density decreases.
  • Similarity-based clustering: Similarity-based methods cluster trajectories, tracklets, or optical-flow features, then estimate semantic regions from the spatial extents of the clusters.Local feature clustering can separate mutually overlapping motion features and applies to structured and unstructured scenes.
  • Similarity-based clustering: Meta-tracking assigns pixels to orientation distributions, computes particle trajectories called meta-tracks, and hierarchically merges nearby meta-tracks into motion patterns.The method is designed to extract dominant motion patterns and main entry/exit areas.

C. Probability Model Based Clustering

Probability models provide compact, global representations of crowded-scene motion by modeling spatio-temporal relationships, while experiments compare their segmentation and source/sink capabilities with representative alternatives. The reported evaluations show method-dependent behavior across scene structure, computational cost, and prior knowledge.

  • Probability-model representations: Probability models such as GMM, RFT, and LDA represent clustered motion features and support longer-term, global scene analysis.They can model spatio-temporal interrelationships among events more compactly than direct clustering of high-dimensional motion vectors.
  • Probability-model representations: RFT treats tracklets as documents, quantizes observations into words using location and velocity direction, and uses MRF priors for spatial-temporal coherence.The model learns semantic regions from object motions while enforcing coherence between tracklets.
  • Probability-model representations: LDA models hierarchical motion structure by representing single-agent patterns over pixel features and interaction patterns over single-agent patterns.This supports discovery of both individual and interaction-level motion patterns.
  • Experiments: The pilot evaluation compares five segmentation methods across six videos spanning crowd density, scene structure, video length, and layout complexity.The videos range from 100 to 5000 frames and include structured, semi-structured, and unstructured scenes.
  • Experiments: In unstructured scenes, meta-tracking handles multiple motion patterns better than the compared methods, while the methods produce more similar segmentations in structured and semi-structured scenes.Optical flow with DBSCAN performs poorly, coherent filtering yields scattered pieces, and meta-tracking can appear cluttered while mixing patterns.
  • Experiments: 0.4 seconds is the reported average execution time for coherent-filtering on 100-frame clips, compared with 5.6 seconds for watershed, 6.3 for DBSCAN, 9.3 for meta-tracking, and 22.4 for FTLE.The timing comparison excludes motion-feature extraction and reflects whether methods process adjacent frames or the whole sequence.
  • Source/sink seeking: For a 30-minute unstructured station video, meta-tracking and RFT extract similar regular paths, while RFT appears better when annotated source/sink priors are incorporated.Meta-tracking finds sources and sinks automatically through clustering without human intervention.

E. Summary

Crowd motion-pattern segmentation uses largely unsupervised flow-field, similarity-based, and probability models, supporting path discovery and tracking. Crowd behavior analysis distinguishes holistic representations from object-based approaches, with each suited to different density and structure conditions.

  • Motion pattern segmentation: Flow-field and similarity-based clustering methods support largely unsupervised motion segmentation in crowded videos.Flow fields simulate crowd motion, while local tracklets can be more discriminative than local optical flow in high-density crowds.
  • Motion pattern segmentation: Learned motion patterns support path or source/sink seeking and can improve scene-specific tracking.
  • Crowd behavior analysis: Crowd behavior analysis has received less attention for reliable real-world activity classification than for crowd-motion simulation.
  • Crowd behavior analysis: Holistic approaches treat the crowd as one entity, whereas object-based approaches model individuals and their behaviors.Holistic methods suit structured medium- or high-density scenes but cannot handle local behaviors in unstructured scenes.
  • Crowd behavior analysis: Individual-object features may be too sparse in highly crowded scenes, making crowd-level behavior understanding advantageous when actions are unresolved.
  • Crowd behavior analysis: Flow-based studies model lanes, divergent or convergent regions, and events such as merges, splits, running, and evacuation.Other approaches recognize bottlenecks, fountainheads, arches, blockings, and large-scale behaviors using dynamical or viscous-fluid models.

B. Object-Based Approach

Object-based behavior analysis can localize individual activities and interactions, but dense crowds make individual detection and segmentation difficult. The survey also emphasizes that reported performances are not directly comparable across heterogeneous behavior definitions and experiments.

  • Object-based behavior analysis: Object-based methods can detect abnormal individual actions, such as one person running while the surrounding crowd walks.
  • Object-based behavior analysis: Dense crowds complicate individual isolation, motivating low-level features and probability models instead of single-object tracking.
  • Object-based behavior analysis: Hierarchical Bayesian models connect low-level features, atomic activities, semantic regions, and multi-agent interactions without labeled training.
  • Object-based behavior analysis: Mixture models of dynamic pedestrian-agents represent collective crowd behavior and can infer past behavior and predict future behavior from partial observations.
  • Object-Based Approach: Holistic methods summarize whole-crowd states but cannot separate simultaneous activities without object detection and tracking.
  • Object-Based Approach: Object-based methods support activity localization and semantic queries but fail when dense crowds prevent reliable individual detection.
  • Object-Based Approach: Reported behavior-recognition performances cannot be directly compared because studies use different behavior definitions, data, and experimental conditions.
  • Crowd anomaly detection: Crowd anomaly detection remains open, with studies differing in supervision, interpretation, assumptions, objectives, and global-versus-local scale.

B. Local Anomaly Detection

Local anomaly detection estimates where abnormal events occur using vision-based models that learn visual regularities and physics-inspired models that represent crowd dynamics. Surveyed methods include HMMs, dynamic textures, bag-of-words, sparse reconstruction, and manifold learning.

  • Local anomaly detection: Local anomaly detection models estimate where anomalies occur using crowd-dynamics models or computer-vision representations.
  • Vision-Based Approach: Vision-based methods learn event patterns from visual features, often by constructing clusters of possible events.
  • Hidden Markov Model: Distribution-based HMMs encode temporal relationships among local motion patterns at each spatial location.
  • Hidden Markov Model: Coupled HMMs model spatial relationships, while unusual events are detected as statistical deviations from same-scene video sequences.
  • Hidden Markov Model: Multiple HMMs can represent different local crowd behaviors using high-frequency and spatio-temporal information from wavelet transforms.
  • Dynamic Texture Model: Dynamic textures model video as observations from linear dynamical systems and are reported as more suitable than optical flow for local unusual-event detection.
  • Dynamic Texture Model: Mixtures of dynamic textures jointly detect temporal and spatial anomalies using representations of appearance and dynamics.
  • Dynamic Texture Model: MDT produces multi-scale temporal anomaly maps by assigning higher anomaly scores to patches with lower probability under their regional model.

2) Physics-Inspired Approach:

Physics-inspired anomaly detectors represent crowd dynamics through flow fields, interaction forces, energy, entropy, or chaotic invariants. These representations support global and local detection, but threshold-based energy methods require empirical scene-specific calibration.

  • Flow Field Model: Flow-field methods model evolving crowd motion and detect changes using finite-time Lyapunov exponent fields.
  • Flow Field Model: Particle trajectories can be clustered into representative crowd flows, whose chaotic invariants support probabilistic normal-versus-abnormal classification.
  • Flow Field Model: Global motion saliency represents flow vectors by phase angle and velocity magnitude before applying spectral residual detection.
  • Social Force Model: Social-force models estimate pedestrian interaction forces from particle flows and map them into the image plane for abnormal-behavior localization.
  • Social Force Model: Variants optimize or revise social-force interactions using particle swarm optimization or collision probability to locate spatio-temporal instability.
  • Crowd Energy Model: Crowd-energy methods use density, velocity, pressure, potential energy, kinetic energy, or dispersion to detect abnormal activities.
  • Crowd Energy Model: Behavior entropy can localize abnormal behaviors, with scene behavior entropy rising during running, dispersion, gathering, or regressive walking.
  • Crowd Energy Model: Energy-based detection typically requires empirically determined thresholds when applied to different crowd scenes.

C. Summary

The survey reviews anomaly detection methods and benchmarks for crowded scene analysis, while emphasizing that differing datasets and evaluation criteria limit direct comparison.

  • Anomaly detection: Anomaly detection methods are difficult to compare objectively because anomalies may be defined subjectively, sometimes according to what algorithms can detect.The survey compares recent techniques in Table V as a broad overview rather than a directly equivalent benchmark.
  • Crowd representation: Physical-system models are convenient for crowd representation but limited in recognizing certain patterns, motivating data-driven or hybrid models.The survey identifies generative topic models as promising among data-driven approaches.
  • Datasets: The survey presents dataset descriptions, database sizes, labeling levels, and accessibility alongside the benchmark listings.These fields are identified as the information provided for Table VI.
  • Datasets: The benchmark collection spans datasets for crowd behavior, anomaly detection, activity analysis, counting, tracking, density estimation, flow analysis, and event recognition.PETS2009 includes separate subsets for calibration, training, counting and density estimation, tracking, and flow or event analysis.
  • Datasets: Datasets differ in accessibility, labeling, sensing setup, crowd density, camera conditions, and task coverage.Examples include publicly available and non-public datasets, labeled and unlabeled sequences, stationary-camera footage, and multi-view multi-sensor recordings.

VIII. CONCLUSIONS AND FUTURE DEVELOPMENTS

The survey concludes that crowded scene analysis remains without a generally accepted representation or solution. It identifies flow-field and generative-topic models as promising directions and highlights multimodal, unified, real-time, and generalizable systems as open priorities.

  • Conclusions: The review covers motion pattern segmentation, crowd behavior recognition, and anomaly detection, with feature representation and crowd dynamics providing supporting foundations.The survey notes that anomaly detection has received substantial effort and that feature representation is correlated with all three subtopics.
  • Conclusions: Crowded scene analysis still lacks a generally accepted solution, while flow-field and generative-topic models are identified as promising representations.The flow-field model treats crowds as physical fluids or particles; the topic model represents scenes as mixtures of events or activities.
  • Future developments: Multi-sensor fusion is proposed to reduce confusion from clutter and occlusion, using modalities such as audio, radar, laser, and multiple cameras.The survey also points to multi-camera contexts as an avenue for crowded-scene surveillance.
  • Future developments: A unified online framework should jointly perform tracking, model learning, and behavior detection while exploiting contextual interactions among these modules.Tracking supplies motion features for learned crowd models, while crowd knowledge and detected events can support tracking and learning.
  • Future developments: Deep learning is presented as a prospective approach because it can learn representations from multiple features when sufficient training data are available.The survey states that no public accepted crowded scene representation currently exists.
  • Future developments: Future systems should address real-time computation and generalization beyond video data collected under specified conditions.The survey observes that current solutions often prioritize accurate understanding without considering computation.
Loading 1502.01812v1…