PaperlayerView exact revision on arXiv

Source-linked AI summary

Background Subtraction in Real Applications: Challenges, Current Models and Future Directions

T. Bouwmans, B. Garcia-Garcia

arXiv:1901.03577v1cs.CV

TL;DR

Real applications need precise foreground masks quickly, but cameras, environments, and moving objects pose distinct challenges. This survey reviews background-subtraction applications and concludes that suitable models vary by application, with natural maritime and aquatic settings requiring more robust methods.

  • Problem

    Real applications require precise foreground masks quickly before tracking, recognition, or behavior analysis, across camera, environment, and object-specific conditions.

    Source: §9 · p. 40

  • Method

    The paper surveys applications using background subtraction and reviews challenges related to different environments and moving objects.

    Source: §9 · p. 39

  • Results

    Background-model suitability varies by application: basic models suffice in controlled indoor motion capture, statistical models suit traffic with specific developments, and natural maritime and aquatic settings need more robust methods.

    Source: §9 · p. 40

  • Takeaways & Limitations

    The authors suggest dedicated models for particular challenges, with suitable models selected when those challenges are detected.

    Source: §9 · p. 40

    Honeybee surveillance involves tiny, fast, and chaotic targets, while higher-resolution or closer cameras increase cost and complexity.

    Source: §5.4 · p. 22

Abstract

from arXiv · show

Computer vision applications based on videos often require the detection of moving objects in their first step. Background subtraction is then applied in order to separate the background and the foreground. In literature, background subtraction is surely among the most investigated field in computer vision providing a big amount of publications. Most of them concern the application of mathematical and machine learning models to be more robust to the challenges met in videos. However, the ultimate goal is that the background subtraction methods developed in research could be employed in real applications like traffic surveillance. But looking at the literature, we can remark that there is often a gap between the current methods used in real applications and the current methods in fundamental research. In addition, the videos evaluated in large-scale datasets are not exhaustive in the way that they only covered a part of the complete spectrum of the challenges met in real applications. In this context, we attempt to provide the most exhaustive survey as possible on real applications that used background subtraction in order to identify the real challenges met in practice, the current used background models and to provide future directions. Thus, challenges are investigated in terms of camera, foreground objects and environments. In addition, we identify the background models that are effectively used in these applications in order to find potential usable recent background models in terms of robustness, time and memory requirements.

1 Introduction

Background subtraction serves diverse fixed-camera applications, but research datasets and methods do not fully reflect real-world challenges and practical constraints. This review surveys real-application studies and highlights recent models with potential for practical use.

1 Introduction

  • The review addresses a literature gap: large-scale research datasets often omit challenging situations encountered in real applications, and no full survey had grouped their models and challenges.The introduction also notes a gap between methods used in real applications and methods developed in fundamental research.Source: §1 · p. 2
  • Applications span human surveillance, including traffic, airport, and maritime settings, animal and insect observation, and tasks such as motion capture, gesture recognition, and background substitution.Animal observation may involve natural settings such as forests, rivers, oceans, and submarine environments.Source: §1 · p. 2
  • The review classifies real-application studies by aims, environments, and objects of interest, selecting publications that test background subtraction on corresponding application videos.It also highlights recent background subtraction models that may be directly usable in real applications.Source: §1 · p. 3

2 Background Subtraction: A Short Overview

Background subtraction segments static and moving foreground objects from video and provides a compromise between detection quality and computation time. It comprises initialization, modeling, maintenance, and foreground detection.

2 Background Subtraction: A Short Overview

  • The four stages are background initialization, background modeling, background maintenance, and pixel classification into background or moving objects.Initialization computes the first background image; modeling represents it; maintenance updates the model over time; and classification detects foreground pixels.Source: §2 · pp. 3–4
  • Initialization requires offline batch algorithms, maintenance requires online incremental algorithms, and initialization, modeling, and maintenance are reconstructive while foreground detection is discriminative.Source: §2 · p. 4
  • The process initializes from N training frames, repeatedly compares current and background images for foreground detection, then updates the background using the prior background, current image, and detection mask.Comparisons use corresponding pixels, regions, or clusters characterized by color, edge, texture, stereo, or motion features; feature and step choices depend on application challenges.Source: §2 · p. 4

3 A Preliminary Overview

In fixed-camera video, background initialization produces a clean background, while background subtraction separates foreground objects for a wide range of applications. The overview distinguishes clean-background tasks from applications that detect moving objects and focuses on the latter.

3.1 Background Initialization based Applications

  • Background initialization supplies a clean background for video inpainting, privacy protection, and computational photography; these tasks do not require moving-object detection and are not reviewed further.Video inpainting fills spatio-temporal holes, while computational photography extracts a clean background plate from images containing foreground clutter.Source: §3.1 · pp. 4–5

3.2 Background Subtraction based Applications

  • Background subtraction supports surveillance of people and vehicles, animal and insect behavior studies, detection of foreign objects in nature, and analysis of human activities.Applications include traffic, public-space, and store surveillance; ecological observation and species census; and sports and athletic-performance analysis.Source: §3.2 · p. 5
  • For visual hull computation, foreground silhouettes from multiple views define cones whose intersection bounds a 3D object, supporting image-based modeling and optical motion capture.The visual-hull technique is based on shape-from-silhouette reconstruction introduced by Laurentini [203].Source: §3.2 · p. 6
  • Other applications include human-machine interaction, hand-gesture recognition, content-based video coding, background substitution, baggage and fire detection, and OLED defect detection.Background subtraction can provide object shape information for video coding and extract foreground video for combination with a replacement background.Source: §3.2 · p. 7
  • The review covers fixed-camera applications, including single- and multi-view setups, while moving-camera applications are out of scope despite existing extensions.The cited overview notes that application challenges vary with camera location, environment, and moving-object type.Source: §3.2 · p. 7

4 Intelligent Visual Surveillance

Visual surveillance uses background modeling and foreground detection to identify static objects of security concern and moving objects for tasks such as traffic statistics, airport operations, and consumer-behavior analysis.

4.1 Traffic surveillance

  • Traffic scenes combine varied cameras and environments with shadows, illumination shifts, moving foliage, similar-colored road users, transient stops, and real-time constraints.Aerial and satellite footage also presents challenges such as camera motion, distant or small objects, weak contrast, and low image quality.Source: §4.1 · pp. 12–13
  • Traffic studies often use unimodal median, histogram, or single-Gaussian models for road views and multimodal models for dynamic scenes, commonly adding shadow detection.RGB is frequent, while intensity and YCrCb are also used to address illumination changes.Source: §4.1 · p. 13

4.2 Airport surveillance

  • Airport surveillance addresses weather and lighting changes, deformable ground vehicles, and prolonged blobs formed with aircraft or other vehicles; reviewed works use median or single-Gaussian models in RGB.The tarmac is described as a unimodal background.Source: §4.2 · p. 14

4.3 Maritime surveillance

  • Maritime surveillance favors multimodal background models for dynamic water and often adds morphology or saliency to reduce water-motion false positives.RGB is common in daylight and infrared at night; approaches include Bayesian segmentation, IMBS, MOG variants, median and single-Gaussian models, ViBe, and modified histograms.Source: §4.3 · pp. 14–15

4.4 Coastal Surveillance

  • Coastal surveillance uses behavior subtraction to detect boats, motor vehicles, and people near shorelines for wildlife, erosion, and human-impact monitoring.Examples include measuring whether people or vehicles approach seals or disturb sand dunes.Source: §4.4 · p. 15

4.5 Store surveillance

  • Store surveillance tracks customer paths to derive dwell time, product interactions, and shopper-group statistics, using uni- or multimodal RGB models for indoor scenes.The paper cites Leykin and Tuceryan [211][210] and Avinash et al. on shopper groups.Source: §4.5 · p. 15

4.6 Military surveillance

  • Military target detection uses infrared or SAR imagery for people and vehicles in difficult environments; SAR processing combines temporal-median background estimation, a log-ratio operator, and CFAR detection.Because military videos are confidential, published results use public civilian datasets.Source: §4.6 · p. 16

5 Intelligent visual observation of animals and insects

Animal and insect surveillance uses background subtraction before tracking or recognition across natural, study, and farm environments, where appearance, behavior, and scene conditions vary. The section describes application-specific detection challenges and the background models or methods used to address them.

5.1 Birds Surveillance

  • Birds at feeder stations can be absorbed into the background because they move less than it, while nesting videos require fast lighting correction and sensitivity to small, cryptically colored animals.Ko et al. used distribution-based subtraction; Goehner et al. modified ViBe and PBAS for second-frame readiness, added morphological opening and closing, and used convex hulls around foreground regions.Source: §5.1 · pp. 20–21

5.2 Fish Surveillance

  • Fish surveillance spans species recognition, counting, behavior study, and catch measurement, but fish are deformable and variable in appearance; methods include MOG for tank detection and textons-based KDE for erratic movement.Other approaches combine running-average foreground detection with three CFD methods, or use SuBSENSE masks and trajectories to separate fish from noise; deep-sea fish have also been detected with MOG and tracked using Camshift with Kalman filtering.Source: §5.2 · p. 21

5.3 Dolphins Surveillance; 5.5 Spiders Surveillance; 5.6 Lizards Surveillance

  • Dolphin tracking supports study of social dynamics and human impacts, with background subtraction used to visualize recurring paths; controlled-box spider localization used grayscale background differences, while lizard census work compared thresholds and methods.The spider method first captures a grayscale background without the spider and marks pixels as candidates when their difference from the captured image exceeds a threshold.Source: §5.3 · pp. 21–22, §5.5 · p. 23, §5.6 · p. 23

5.4 Honeybees Surveillance

  • MOG was judged best suited to detecting honeybees at hive entrances, where bees are tiny, fast, erratic, and often occlude one another under variable outdoor lighting.A typical bee occupies approximately 6 × 14 pixels; when individuals cannot be segmented, pollen assessment still requires determining whether a group carries a pollen load.Source: §5.4 · p. 22

5.7 Mice Surveillance

  • Mice behavior analysis is time-consuming when experts review footage frame by frame, so fast detection and recognition are needed; fRMC was used for in-cage mice detection.The work used the Caltech resident intruder mice dataset.Source: §5.7 · p. 23

5.8 Pigs Surveillance

  • Pig monitoring in farrowing pens must handle moved nesting material, sudden lighting changes that can make most of an image foreground, and sleeping animals that may be absorbed into the background.Approaches include PM-MOG, modified MOG with a Dyadic Wavelet Transform to extract sow shapes, and illumination-reflectance estimation with HWF and WQI followed by CFD detection.Source: §5.8 · pp. 23–24

5.9 Hinds Surveillance

  • Camera-trap detection of hinds must contend with camouflage, long periods of immobility, and rapid movement; multi-layer RPCA was reported robust to challenges including illumination changes.Hinds may blend into forest backgrounds or be incorporated into them while sleeping, but can also move quickly to escape predators.Source: §5.9 · p. 24

6 Intelligent Visual Observation of Natural Environments

Natural environments pose difficult background-subtraction challenges because foreground objects can blend into backgrounds that change dynamically. The survey reviews practical applications, their challenges, and background models used across forests, rivers, oceans, and underwater settings.

6.1 Forest Environments

  • Forest applications address lighting transitions, foliage motion, and occlusion; Janzen et al.’s system reduced review images to one third and identified ungulate crossings with 60%–92% success.Other approaches included robust PCA for animals and joint background modeling with deep-learning classification for humans and animals.Source: §6.1 · p. 26

6.2 River Environments

  • River monitoring targets pollution and infrastructure hazards, but moving water, floods, waves, and changing illumination complicate detection; methods used Kalman filtering, space-time spectral modes, and rigid-motion-augmented MOG.Fallen trees can lodge against bridge pillars and accumulate smaller debris.Source: §6.2 · p. 26

6.3 Ocean Environments

  • Maritime surface scenes combine ship traffic, collision avoidance, and intrusion detection with wave, wake, and weather motion; all 34 ChangeDetection.net algorithms tested on Singapore-Marine were ineffective [39] [344].The tested methods produced false positives in water regions or false negatives while suppressing the water background.Source: §6.3 · p. 27

6.4 Submarine Environments

  • Underwater settings comprise swimming pools, fish tanks, and open sea environments.Their backgrounds and targets present distinct challenges, from pool reflections to fish behavior and open-sea visibility and lighting variation.Source: §6.4 · p. 27

6.4.1 Swimming pools

  • Pool systems address ripples, splashes, reflections, and lighting changes using color-space background models, optical flow, or Kalman filtering; Markov random fields and spatiotemporal filtering help with occlusion and nighttime reflections.Private-pool detection also combined mean and CFD models in HSV color space.Source: §6.4.1 · pp. 27–28

6.4.2 Tank environments

  • Fish-tank detection must handle variable illumination and water movement, moving algae, species-specific behavior, different swimming depths, and occlusion by fish or algae [274].Aquaculture tanks may contain many fish of one species, unlike aquariums with multiple species.Source: §6.4.2 · p. 28

6.4.3 Open sea environments

  • Daylong light transitions, typhoons, storms, sea currents, and murky-water clarity challenge open-sea video background subtraction; plants subject to flood-tide and drift can generate false-positive detections.Marine snow consists of small floating particles that strongly reflect light, corrupt images, and complicate static-background modeling.Source: §6.4.3 · pp. 28–29
  • On underwater datasets, SC-GMM outperformed 18 algorithms, while GSM surpassed MOG-EM, ZHGMM, and EA-KDE; ViBe excelled in nearly all Fish4Knowledge videos [179].In pixel-level Fish4Knowledge results, detection rates ranged from 83.2% for APMM to 83.4% for ViBe, despite relatively high false-alarm rates. SC-GMM outperformed 18 algorithms on both Fish4Knowledge and UnderwaterChangeDetection.eu; GSM performed better than the three named models on UnderwaterChangeDetection.eu.Source: §6.4.3 · pp. 29–30

7 Miscellaneous Applications · 7.1 Visual Analysis of Human Activities · 7.2 Optical motion capture

The survey describes background subtraction in human-activity analysis and optical motion capture, where simple indoor setups enable silhouette extraction but practical robustness and resource constraints remain important.

7.1 Visual Analysis of Human Activities

  • In sport and hazardous-activity surveillance, background subtraction supports rapid decisions, performance analysis, and monitoring, with methods ranging from simple silhouettes to dedicated models.The section highlights applications including coach feedback, rowing analysis, and surfer detection.Source: §7.1 · p. 30
  • John et al. report 93.67% accuracy for pose matching in an augmented-reality coaching app using silhouettes from a stationary RGB camera.The camera captures the empty background during initialization; the detected pose is compared with an exemplar at specific intervals.Source: §7.1 · p. 30
  • For indoor rowing, Tamas found that Zivkovic-Heijden GMM and Zivkovic-KDE mark background-projecting shadows as foreground, then developed a fast method to extract athlete body positions.The extracted positions include the head, shoulders, elbows, hands, hips, knees, ankles, and back.Source: §7.1 · p. 30
  • Bastos used original MOG to detect surfers, reporting a false positive rate of 1.77 for a detection rate of 90%, with memory and processing time as drawbacks.Source: §7.1 · p. 30

7.2 Optical motion capture

  • Optical motion capture uses multiple camera silhouettes to build a 3D voxel approximation and track movements onto an avatar; it supports games, virtual clothing, and virtual reality.Silhouettes are extracted in each view, a step also called silhouette detection or extraction.Source: §7.2 · p. 32
  • Chromakeying segments foreground by color against a uniform background, but its restriction to such backgrounds limits it; background subtraction is more suitable for silhouette extraction.Optical capture systems require precise silhouettes and movements from multiple camera viewpoints.Source: §7.2 · p. 32
  • Indoor optical capture can use unimodal background models, with shadows and highlights as key challenges; Wren et al. used a single Gaussian in YUV, while Horprasert et al. [147] suppressed both.Table 10 overviews background models and related methods for optical motion capture and video coding.Source: Table 10 · p. 31, §7.2 · p. 32

7.3 Human-machine interaction

The section surveys real-time, fixed-camera human–machine interactions in games and ludo-applications, where video enables interactive experiences such as Aqu@theque [20][274].

Human-machine interaction

  • Background subtraction locates body pixels for graphical responses in mirrorlike installations and video games, following interaction techniques described by Levin et al. and Warren.Examples include Myron Krueger’s Videoplace and PlayStation Eye-Toy, where participants see their image or silhouette composited into a virtual scene.Source: §7.3 · p. 32
  • Aqu@theque lets aquarium visitors select live-filmed fish on a touch screen to receive educational information and view identified fish in a virtual tank [20][274].Its two original elements are automatic detection and recognition of fish species in a remote tank, and multi-agent behavioral modeling of virtual fish.Source: §7.3 · pp. 32–33

7.4 Vision-based Hand Gesture Recognition

Vision-based hand gesture recognition detects, tracks, and recognizes gestures for applications including human-computer interfaces, behavior studies, sign-language interpretation and learning, and robotics.

Human-Computer Interface

  • For HCI, real-time video background subtraction enables hand gestures as alternatives to keyboards, mice, and joysticks, including augmented screens and computer interfaces.Examples include Park and Hyun and Stergiopoulou et al..Source: §7.4 · p. 33

Behavior Analysis

  • Perrett et al. used background subtraction with SLIC superpixels to detect car occupants’ hands at a center console in both day and night conditions.They chose PBAS because small foreground objects decay into the background model quickly while larger objects persist.Source: §7.4 · p. 33

Sign Language Interpretation and Learning

  • Elsayed et al. detected moving hand areas in real time using skin-color thresholds and YCrCb frame differences, with effective results in indoor and outdoor scenes.The initial background was the first frame without foreground objects.Source: §7.4 · p. 34

Robotics

  • Khaled et al. used a running average to detect hands and the 1$ algorithm for template matching, translating five detected gestures into robot movement commands.Source: §7.4 · p. 34

7.5 Content-based video coding

Content-based video coding separates moving objects from the background so they can be encoded independently, especially when the background remains static. The survey describes background-model-based coding approaches for surveillance and dynamic scenes, including methods that improve rate-distortion performance or reduce bitrate.

7.5 Content-based video coding

  • Video coding separates moving objects from static or dynamic environments so background and video objects can be encoded separately when the background remains unchanged [400].This is particularly relevant to surveillance scenes captured by fixed cameras.Source: §7.5 · p. 34
  • For H.264 coding, Paul et al. [270][271] use MOG- and MD-MOG-based dynamic-background references, while Chakraborty et al. [63][64] develop WNP and SANP alternatives.The MoG-based methods improve rate-distortion performance over H.264 but require high computation time, perform poorly on dynamic videos, and require prior content knowledge; WNP handles dynamic backgrounds without prior knowledge of the video-data distribution, and SANP targets highly dynamic sequences.Source: §7.5 · p. 34
  • For surveillance coding, Zhang et al. [410][411][407][409] and Chen et al. propose background generation, maintenance, prediction, and selective-difference methods; Han et al. compute motion-compensated panorama backgrounds for moving cameras.Zhao et al. propose a background-foreground division-based search algorithm to address HEVC coding limitations.Source: §7.5 · pp. 34–35
  • ALMCM achieves 18.07% bit-rate saving on CIF sequences and 11.88% on SD sequences against background-irrelevant Lagrange multiplier selection.The method adapts the multiplier to the background proportion in a coding unit to improve rate-distortion optimization.Source: §7.5 · p. 35
  • LRSD decomposes surveillance video into background and moving-object components, achieving up to 3 dB PSNR gain over H.264/AVC, especially at relatively low bit rates [70].Guo et al. use a dictionary trained on a small number of observed frames to represent background and motion; their method reduces video size and gains higher PSNR than state-of-the-art codecs.Source: §7.5 · p. 35

7.6 Background substitution

Background substitution, also called background cut or video matting, extracts foreground objects from video and combines them with a new background.

7.6 Background substitution

  • Background subtraction can serve as the first step in background substitution, as in Huang et al..The task extracts the foreground from the input video and combines it with a new background.Source: §7.6 · p. 35

7.7 Carried baggage detection · 7.8 Fire detection · 7.9 OLED defect detection

The three applications use background subtraction within task-specific pipelines: refining carried-baggage contours, screening moving pixels for fire activity, and inspecting OLED images against repetitive-image models.

7.7 Carried baggage detection

  • Tzanidou detects carried baggage using multi-directional gradients and phase congruency.The method targets cast-shadow removal and invariance to gradual illumination changes.Source: §7.7 · p. 36
  • Tzanidou refines foreground contours by classifying edge segments and completing contours with anisotropic diffusion.The method targets closed-contour extraction.Source: §7.7 · p. 36

7.8 Fire detection

  • Several fire detection systems [299][133][122] first use background subtraction to detect moving pixels, then check whether their colors match predefined fire colors.Source: §7.8 · p. 36
  • Toreyin et al. use temporal and spatial wavelet analysis to determine high-frequency activity in fire detection.Source: §7.8 · p. 36

7.9 OLED defect detection

  • For OLED inspection, Jeon and Yoo use a KDE-based background model built from repetitive images around the target area to compute each target pixel’s likelihood.OLED displays have repeating grayscale patterns with significant intensity variations, and defects can affect display quality and life.Source: §7.9 · p. 36
  • OLED’s grayscale repeating patterns and intensity variations make background subtraction applicable to automated defect inspection.OLED defects arise during manufacturing and may affect display quality and life.Source: §7.9 · p. 36

8 Discussion

The discussion identifies camera jitter, illumination changes, and dynamic backgrounds in outdoor scenes as major challenges, then considers prospective models and evaluation resources. It also highlights practical limits in deployment and dataset coverage.

8.1 Solved and unsolved challenges

  • Camera jitter, illumination changes, and dynamic backgrounds in outdoor scenes are the main difficult challenges, and the overview identifies applications needing further investigation.Table 11 groups solved and unsolved challenges by application.Source: §8.1 · p. 36, Table 11 · p. 37

8.2 Prospective solutions

  • Recent fuzzy, RPCA, and deep-learning models are prospective responses; examples include Choquet vehicle detection [230], incPCP traffic counting [352], and fRMC mice detection [15].ReProCS and its variants are described as offering advantages in detection, real-time performance, and memory requirements; online RPCA was also tested for vehicle and person detection.Source: §8.2 · pp. 37–38
  • Deep learning may address unsolved challenges, but its time and memory demands often prevent current use in real applications.The discussion also proposes improvements to MOG, codebook, ViBe, and PBAS; on CDnet 2012, two 2014 MOG improvements achieved significantly better detection than earlier MOG algorithms while remaining usable in applications [318][72].Source: §8.2 · p. 38

8.3 Datasets for Evaluation

  • CDnet 2012 and 2014 are realistic large-scale datasets, but large-scale datasets are absent for animal and insect surveillance and natural environments.Other resources include datasets for RGB-D, infrared, and multispectral video, plus animal-surveillance datasets such as Aqu@theque, Fish4knowledge, Caltech resident intruder mice, Caltech Camera Traps, and eMammal.Source: §8.3 · pp. 38–39

8.4 Libraries for Evaluation

  • BGSLibrary provides more than 43 algorithms, and five of its methods outperformed OpenCV’s three MOG algorithms for highway congestion classification; LRSLibrary collects matrix- and tensor-factorization methods.Some LRSLibrary algorithms can be implemented in C to meet real-time requirements.Source: §8.4 · p. 39

9 Conclusion

The review surveys background subtraction applications and their environment- and object-related challenges. It concludes that foreground masks must be precise and fast, while model choice should match each application’s challenges.

9 Conclusion

  • Moving-object detection is the first step before tracking, recognition, or behavior analysis, so foreground masks must be precise and fast; Varona et al. studied its downstream influence.The review frames speed as a real-time constraint.Source: §9 · p. 40
  • Camera location, environment, and moving-object type create application-specific challenges, favoring dedicated background models or a universal model that selects one for detected challenges.The review notes that environments and objects differ substantially, including in appearance.Source: §9 · p. 40
  • Basic models are sufficiently robust for controlled environments such as indoor optical motion capture.The review presents this as an application-specific model recommendation.Source: §9 · p. 40
  • For traffic surveillance, statistical models provide a suitable framework, but illumination changes and sleeping or beginning foreground objects require specific developments.Source: §9 · p. 40
  • For natural environments, especially maritime and aquatic scenes, more robust methods than top ChangeDetection.net models are needed; recent RPCA and deep-learning models merit consideration.Prasad et al. developed a more robust maritime and submarine method in 2017.Source: §9 · p. 40

Related papers

Loading 1901.03577v1…