Source-linked AI summary
Background Subtraction in Real Applications: Challenges, Current Models and Future Directions
T. Bouwmans, B. Garcia-Garcia
TL;DR
Background subtraction methods must separate foreground from background accurately and quickly for downstream video analysis, while real applications involve diverse camera, environment, and object challenges. This survey reviews those applications and the background models they use, concluding that model suitability depends on the application and that robust recent methods remain needed in difficult environments.
Problem
Real applications require accurate, fast foreground masks, but background subtraction must address diverse camera locations, environments, and moving-object properties.
Method
The review surveys applications using background subtraction, examines their environmental and object-related challenges, and identifies background models used in practice.
Results
Recent RPCA and deep learning models should be considered for natural, maritime, and aquatic environments, where more robust methods are required.
Takeaways & Limitations
Controlled environments can use basic models, whereas traffic surveillance needs developments for illumination and sleeping or beginning foreground objects, and maritime settings require greater robustness.
Takeaways & Limitations
Honeybee surveillance remains constrained by tiny, fast, and chaotic targets, while higher-resolution or multiple-camera solutions increase cost and complexity.
Abstract
from arXiv · showhide
Computer vision applications based on videos often require the detection of moving objects in their first step. Background subtraction is then applied in order to separate the background and the foreground. In literature, background subtraction is surely among the most investigated field in computer vision providing a big amount of publications. Most of them concern the application of mathematical and machine learning models to be more robust to the challenges met in videos. However, the ultimate goal is that the background subtraction methods developed in research could be employed in real applications like traffic surveillance. But looking at the literature, we can remark that there is often a gap between the current methods used in real applications and the current methods in fundamental research. In addition, the videos evaluated in large-scale datasets are not exhaustive in the way that they only covered a part of the complete spectrum of the challenges met in real applications. In this context, we attempt to provide the most exhaustive survey as possible on real applications that used background subtraction in order to identify the real challenges met in practice, the current used background models and to provide future directions. Thus, challenges are investigated in terms of camera, foreground objects and environments. In addition, we identify the background models that are effectively used in these applications in order to find potential usable recent background models in terms of robustness, time and memory requirements.
1 Introduction
Background subtraction supports many fixed-camera video applications, but real-world environments and foreground objects create diverse challenges. This review surveys real applications and highlights models that may be usable under practical robustness, time, and memory constraints.
- Fixed-camera video applications use background initialization and subtraction across diverse environments and moving foreground objects.
- The review surveys real applications according to their aims, environments, and objects of interest.
- It includes publications that address background subtraction in real applications using experiments on corresponding videos.
- The paper highlights recent background subtraction models that can be directly used in real applications.
- The review targets computer-vision researchers and engineers, as well as biologists and ethologists studying visual surveillance.
2 Background Subtraction: A Short Overview
Background subtraction separates static and moving foreground objects through initialization, modeling, maintenance, and foreground detection. These stages use different algorithmic requirements, including batch processing for initialization and incremental updates for maintenance.
- Background initialization computes the first background image, while background modeling defines how the background is represented.
- Background maintenance updates the model over time, and foreground detection classifies pixels as background or moving objects.
- Initialization requires offline batch algorithms, whereas maintenance requires online incremental algorithms that process incoming data sequentially.
- Background initialization, modeling, and maintenance require reconstructive algorithms.
- The process can initialize a background from N training frames, compare it with the current image, and update it using the foreground mask.
3 A Preliminary Overview
Background subtraction and initialization serve applications ranging from surveillance and interaction to image processing and object modeling. These applications differ in camera configuration, environments, and foreground objects, while some require only a clean background.
- Background initialization can provide clean backgrounds for video inpainting, privacy protection, and computational photography.
- Some applications require only a clean background and therefore do not involve moving-object detection.
- Background subtraction supports surveillance of human activities, including traffic analysis, sports, and human-machine interaction.
- Animal and insect observation uses detection for behavior analysis, species monitoring, ecosystem assessment, and protection.
- Natural-environment applications detect foreign objects such as bottles, wood, or mines in rivers and oceans.
- Optical motion capture extracts filmed movements for character animation and interactions between real and virtual worlds.
- Applications may use one or multiple views, and background subtraction has also been extended to slowly moving or freely moving cameras.
4 Intelligent Visual Surveillance
Intelligent visual surveillance uses background subtraction to detect static and moving foreground objects across traffic, airport, maritime, animal, and other settings. The review highlights camera, environment, object, and implementation challenges, alongside background-model choices used in practice.
- Applications: Surveillance targets include abandoned stationary objects and moving vehicles, airplanes, boats, persons, and luggage for security and traffic statistics.Applications include road, airport, maritime, and consumer-behavior surveillance.
- Camera challenges: Traffic surveillance must handle fixed, aerial, and satellite cameras, with aerial and satellite videos adding distance, motion, shadow, weak-contrast, and small-object challenges.Camera quality also ranges from low-quality noisy video with compression artifacts and low frame rates to HD cameras.
- Environment and object challenges: Traffic scenes vary by weather, illumination, foliage motion, shadows, object-road color similarity, transient stops, merged detections, and nighttime headlights.These conditions can create noisy or false detections and degrade background estimation.
- Implementation challenges: Real-time applications require low computation time, making processing efficiency an implementation challenge.The review identifies real-time detection as a recurring application requirement.
- Traffic background models: Traffic studies use uni-modal models such as the median, histogram, and single Gaussian for road scenes, while dynamic environments favor multi-modal MOG models.RGB is common; intensity and YCrCb are also used, and shadow detection is a frequent additional strategy.
- Airport and maritime models: Airport tarmac surveillance uses median or single-Gaussian models in the reviewed works, whereas maritime scenes favor multi-modal models because water is dynamic.Airport studies use RGB, while maritime systems may use RGB by day and infrared at night, with morphology or saliency to address water-induced false positives.
5 Intelligent visual observation of animals and insects
Animal and insect surveillance spans diverse targets whose size, motion, camouflage, occlusion, and environmental conditions challenge background subtraction. Real applications therefore use adapted models and processing strategies, with MOG frequently reported as suitable in controlled settings.
- Applications: Animal surveillance supports ecological monitoring, behavior analysis, species protection, livestock management, and robotic locomotion studies.Applications include birds, fish, dolphins, honeybees, mice, pigs, hinds, and lizards.
- Bird surveillance: Bird videos require models that handle slow motion, rapid lighting correction, small targets, and cryptic coloration.Rapid adaptation may absorb motionless birds into the background; modified ViBe and PBAS add second-frame readiness, morphological filtering, and convex-hull processing.
- Fish surveillance: Fish applications address erratic motion, noisy sea surfaces, underwater conditions, recognition, counting, tracking, behavior analysis, and catch measurement.Reported approaches include sliding averages, Zivkovic-Heijden GMM, textons-based KDE, logical-AND fusion, and motion trajectories.
- Honeybee surveillance: Honeybee detection is constrained by approximately 6 × 14-pixel targets, fast and chaotic motion, identical appearance, changing illumination, and shadows.Higher-resolution or additional cameras improve detection but increase cost and physical and computational complexity.
- Model selection: MOG was reported as best suited for honeybee detection in hive-entrance video and as more suitable than SG for fish detection in tanks.For fish, SG and MOG had favorable time and memory requirements, whereas KDE was too slow and memory-intensive.
- Farm and forest surveillance: Pig and hind surveillance must handle dynamic backgrounds, illumination changes, camouflage, rapid motion, and foreground objects becoming motionless.Modified MOG, wavelet methods, homomorphic filtering, and multi-layer RPCA were applied in these settings.
6 Intelligent Visual Observation of Natural Environments
Natural-environment surveillance must separate targets from backgrounds altered by foliage, waves, weather, illumination, water conditions, and camouflage. Existing methods show mixed performance across aquatic and ecological scenes, motivating specialized background models.
- Challenges: Natural environments generate multimodal and structurally changing backgrounds, while foliage, waves, weather, and camouflage can create false detections or hide targets.These conditions occur in forests, rivers, oceans, pools, tanks, and open-sea underwater scenes.
- Ecological tools: MotionMeerkat supports ecological motion extraction with Running Gaussian Average or MOG, but rapid lighting changes and camouflage remain problems.DeepMeerkat applies convolutional-neural-network background modeling, while VIAME lacked background-subtraction algorithms in the reviewed description.
- Ocean and maritime environments: Maritime scenes are more difficult than calm water because waves, boat wakes, weather, and illumination produce highly dynamic backgrounds.On the Singapore-Marine dataset, 34 ChangeDetection.net algorithms produced false positives in water or false negatives when suppressing water background.
- Ocean and maritime environments: Open-sea conditions vary with sunlight, twilight, night, haze, rain, fog, solar angle, tides, visibility, speckle, glint, and color gamut.These factors alter contrast, statistical distributions, and the visibility of distant objects.
- Underwater environments: Underwater environments include pools, fish tanks, and open sea, with ripples, splashes, reflections, algae, illumination changes, water flow, and murky water.Fish, algae, plankton, and lens contamination complicate background modeling and can produce false positives.
- Reported performance: At blob level, ViBe excelled in nearly all evaluated videos, while codebook performed best in high-resolution videos under the reported conditions.At pixel level, detection rates ranged from 83.2% for APMM to 83.4% for ViBe, but false-alarm rates were relatively high in low-light and violent-weather scenes.
- Reported performance: GSM outperformed MOG-EM, ZHGMM, and EA-KDE on UnderwaterChangeDetection.eu, while SC-GMM outperformed 18 algorithms on two underwater datasets.The comparisons included classical models and advanced methods such as PBAS, SuBSENSE, and SOBS.
7 Miscellaneous Applications
Background subtraction supports diverse visual-analysis applications, including sports feedback and optical motion capture, where silhouettes are extracted for pose or movement analysis.
- In sports, background subtraction supports rapid decisions, athletic-performance analysis, and surveillance in dangerous activities.
- A stationary camera can capture an empty indoor background, subtract current images, and obtain a user silhouette for pose comparison.One augmented-reality desktop application reported 93.67% pose-matching accuracy.
- Optical motion-capture systems use silhouettes from multiple cameras to compute a 3D voxel approximation of an actor and reproduce movements on an avatar.The approach is used in computer games, virtual clothing, and virtual-reality applications.
7.3 Human-machine interaction
Human-machine interaction systems use real-time video and background subtraction to locate body pixels or moving objects that drive interactive and educational responses.
- Games and ludo-applications use fixed-camera video to support real-time interaction between humans and machines.
- Arts and Games: Interactive art and games locate a participant’s body pixels with background subtraction and use them for graphical responses or image compositing.
- Ludo-Multimedia Applications: The Aqu@theque interface lets visitors select live fish, automatically identify species, and display educational information and virtual representations.Its virtual tank also models fish behavior with multi-agent methods.
7.4 Vision-based Hand Gesture Recognition
Vision-based hand-gesture systems detect, track, and recognize hands for interfaces, control, learning, robotics, games, and virtual-environment interaction.
- Human-Computer Interface: Hand gestures provide an alternative to keyboards, mice, and joysticks in human-computer interfaces using real-time background removal.
- Human-Computer Interface: A vehicle-interface system combined background subtraction with superpixel segmentation to detect which occupant interacted with the center-console control.PBAS was selected because small foreground objects can decay into the background model.
- A hand-detection method used a first frame as background and thresholded YCrCb differences, with skin-color information improving segmentation.Experiments on indoor and outdoor scenes reported efficient hand detection.
- Robotics: Robotic control systems can detect five hand gestures and translate them into commands for robot movements using template matching.The cited system used a running average for hand detection and the 1$ algorithm for matching.
7.5 Content-based video coding
Content-based video coding separates static or dynamic backgrounds from moving objects so video objects can be encoded and transmitted more efficiently.
- Video coding segments frames into video objects and background, encoding them separately when a fixed-camera background remains unchanged.
- For H.264 coding, MOG-based dynamic background modeling generated a common reference frame while targeting rate-distortion optimization.
- Surveillance-coding methods addressed H.264/AVC limitations with nonparametric generation, SWRA, timely maintenance, adaptive prediction, BFDS, running averages, and panorama backgrounds.The panorama approach included motion compensation for moving cameras.
- RPCA-based Low-Rank and Sparse Decomposition represented backgrounds as low-rank components and moving objects as sparse components, achieving up to 3 dB PSNR gain over H.264/AVC.The gain was reported especially at relatively low bit rates.
- Table 10 overviews background models, maintenance, foreground detection, color spaces, and author strategies for optical motion capture and video coding.
7.6 Background substitution
Background subtraction can serve as the first step in extracting foreground video content before compositing it onto a new background.
- Background substitution: Background subtraction extracts the foreground from an input video before it is combined with a new background.This use is also called background cut or video matting.
7.7 Carried baggage detection
The reviewed applications illustrate background subtraction for carried-baggage detection, fire detection, and OLED defect inspection, using application-specific processing stages.
- Carried baggage detection: Carried-baggage detection combines background modeling with gradient and phase-congruency features, contour classification, and anisotropic-diffusion completion.The method targets cast-shadow removal, gradual-illumination-change invariance, and closed-contour extraction.
- Fire detection: Fire detection systems first identify moving pixels with background subtraction, then check fire colors and analyze temporal-spatial wavelet activity.The color and wavelet stages evaluate whether moving pixels match predefined fire characteristics.
- OLED inspection: OLED inspection addresses manufacturing defects that can affect display quality and lifetime through an automated computer-vision system.OLED displays are described as grayscale-feature environments in the supplied passage.
8 Discussion
The discussion organizes solved and unsolved application challenges and identifies background models that may improve robustness while meeting real-time and resource constraints. It highlights fuzzy, RPCA, and deep-learning approaches alongside improvements to widely used conventional models.
- Challenges by application: Table 11 organizes solved and unsolved background-subtraction challenges by application.The review groups challenges according to the environments and objects of interest in each application.
- Prospective models: Recent fuzzy, RPCA, and deep-learning background models are proposed as prospective responses to unsolved application challenges.The survey identifies these model families as possible directions for future real applications.
- Prospective models: Choquet-integral foreground detection was successfully tested for moving-vehicle detection.This is the cited fuzzy-model example among the prospective solutions.
- Prospective models: Online RPCA methods such as incPCP and ReProCS offer advantages in detection, real-time operation, and memory requirements.The review also reports successful incPCP testing for vehicle counting and online RPCA use for vehicle and person detection.
- Prospective models: Deep-learning methods are robust to relevant unsolved challenges but are often too time- and memory-consuming for current real applications.Their computational and memory costs remain a practical deployment constraint.
- Existing models: Improvements to MOG, codebook, ViBe, and PBAS are identified as potential methods for real applications.The discussion considers upgraded versions rather than only the original algorithms commonly used in reviewed studies.
- Datasets: Available datasets cover human-surveillance settings with varied modalities, but datasets for animals and insects remain scarce.Examples include RGB-D, infrared, multispectral, fish, and other animal-surveillance datasets.
- Implementation practice: Many studies use MOG algorithms available in OpenCV even though more recent algorithms are available in BGSLibrary and LRS Library.The review contrasts commonly selected OpenCV implementations with newer library alternatives.
9 Conclusion
The conclusion presents moving-object detection as an early, real-time-sensitive stage across diverse video applications. It argues that model choice should reflect application-specific challenges, with recent RPCA and deep-learning methods especially relevant to difficult natural environments.
- Application requirements: Moving-object detection is the first video-analysis step before tracking, recognition, or behavior analysis, so foreground masks must be precise and fast.The conclusion links this requirement to real-time constraints.
- Application challenges: Applications differ in camera location, environment, and moving-object type, creating distinct critical situations for background subtraction.Camera placement can produce objects that are small or large relative to image size.
- Model selection: Background models must handle environment-specific challenges and the differing appearance properties of foreground objects.The review frames the choice as either developing application-specific models or selecting suitable models for detected challenges.
- Model selection: A challenge-specific model-selection strategy is suggested as a route toward a universal background model.The proposal is to develop dedicated models for particular challenges and select them when those challenges are detected.
- Application-specific conclusions: Basic models are sufficiently robust in controlled environments such as indoor optical motion capture.Traffic surveillance needs developments for illumination changes and sleeping or beginning foreground objects.
- Application-specific conclusions: Natural maritime and aquatic environments require methods more robust than leading ChangeDetection.net competition methods, making recent RPCA and deep-learning models relevant.This conclusion specifically concerns maritime and submarine settings.