PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Neural Network Concepts for Background Subtraction: A Systematic Review and Comparative Evaluation

Thierry Bouwmans, Sajid Javed, Maryam Sultana, Soon Ki Jung

arXiv:1811.05255v1cs.CV

TL;DR

Background subtraction research has rapidly adopted deep neural networks, creating a need to compare their concepts, architectures, and performance across the task. This paper surveys DNN-based background initialization, subtraction, and learned features, then evaluates them on CDnet 2014. The review reports a performance gap favoring supervised DNN methods while identifying unresolved difficulties for moving cameras.

  • Problem

    The field lacked an exhaustive comparative survey explaining DNN approaches for background initialization, subtraction, foreground detection, and learned features.

  • Method

    The paper reviews DNN concepts and methods across background generation, background subtraction, and deep learned features, comparing architectures, challenges, and performance.

  • Results

    Supervised DNN methods show a performance gap over conventional approaches on the CDnet 2014 dataset.

  • Takeaways & Limitations

    DNNs have been applied across background initialization, foreground detection enhancement, ground-truth generation, and deep learned features.

  • Takeaways & Limitations

    Current DNN methods appear to encounter problems with moving-camera PTZ videos, where reported F-Measures are not always high.

Abstract

from arXiv · show

Conventional neural networks show a powerful framework for background subtraction in video acquired by static cameras. Indeed, the well-known SOBS method and its variants based on neural networks were the leader methods on the largescale CDnet 2012 dataset during a long time. Recently, convolutional neural networks which belong to deep learning methods were employed with success for background initialization, foreground detection and deep learned features. Currently, the top current background subtraction methods in CDnet 2014 are based on deep neural networks with a large gap of performance in comparison on the conventional unsupervised approaches based on multi-features or multi-cues strategies. Furthermore, a huge amount of papers was published since 2016 when Braham and Van Droogenbroeck published their first work on CNN applied to background subtraction providing a regular gain of performance. In this context, we provide the first review of deep neural network concepts in background subtraction for novices and experts in order to analyze this success and to provide further directions. For this, we first surveyed the methods used background initialization, background subtraction and deep learned features. Then, we discuss the adequacy of deep neural networks for background subtraction. Finally, experimental results are presented on the CDnet 2014 dataset.

1 Introduction

Background subtraction separates background and foreground in static-camera video, supporting applications such as surveillance, traffic monitoring, and industrial vision. This section traces neural-network approaches toward deep learning and surveys the review’s coverage.

  • Background subtraction supports intelligent surveillance, traffic monitoring, and industrial machine vision in static-camera video.
  • Early neural-network methods modeled backgrounds and detected foregrounds, but RAM-NN required correctly represented background images and lacked background maintenance.
  • ABPNN required specific initial threshold values for each analyzed video, while BNN used three subnetworks and remained complex.
  • Deep neural networks, particularly CNNs, were introduced to alleviate disadvantages of conventional neural-network approaches.
  • Since Braham et al.’s 2016 work, many DNN papers reported strong CDnet 2014 performance and applications spanning initialization, detection enhancement, ground-truth generation, and learned features.
  • The review compares DNN approaches by architecture and performance across background initialization, subtraction, foreground detection, and feature learning.

2 Deep Neural Networks: A Short Overview

This overview introduces DNN architectures, training, theoretical questions, and implementation resources. It also frames the paper’s later comparison of DNN approaches for background/foreground separation.

  • DNN architectures comprise input, hidden, and output layers whose neurons use activation functions to map inputs through nonlinear transformations.
  • Training learns layer weights from large-scale datasets through a cost function and optimization procedure.
  • DNN theory examines architecture expressiveness, non-convex optimization, generalization, fitting, adversarial perturbations, and noisy-label robustness.
  • Architecture and optimization can affect generalization, while different architectures may differ in optimization difficulty.
  • Despite initial theoretical and visualization analyses, DNN understanding remains shallow while successful computer-vision applications show substantial performance gaps.
  • Libraries including Caffe, MatConvNet, CNTK, TensorFlow, Theano, Torch, and DL4J support DNN implementation.
  • The paper surveys DNN approaches for background/foreground separation, comparing their advantages, disadvantages, and CDnet 2014 performance.

3 Background Generation

Background generation initializes a background model, often challenged by foreground contamination and bootstrapping. The reviewed DNN approaches use RBMs, autoencoders, CNNs, U-Nets, and GAN-based context prediction for this task.

  • Background generation initializes the background using frames or models such as temporal averages and medians, but bootstrapping can make initialization difficult.
  • PS-RBM models images through RBM weights and adds a sparsity target to limit weight growth during learning.
  • TARBM incorporates spatial and temporal coherence to generate more stable backgrounds from noisy inputs and adapt to background changes.
  • GRBM accepts real-valued inputs, models pixel-value variance through a constrained Gaussian mixture, and uses the RBM generative paradigm.
  • An autoencoder pipeline first estimates approximate backgrounds and then learns an online-updated background model from them.
  • FCFlowNet estimates background images from a small set of frames containing foreground objects and uses contractive and refinement stages followed by post-processing.
  • BM-Unet maps an input frame to a background image with a probabilistic color heat map and combines color, intensity differences, and optical flow features.
  • DCP combines unsupervised context prediction with semantic inpainting for background initialization; on SBM.net, it achieved an average gray level error of 8.724.

4 Background Subtraction

Deep neural networks support background subtraction through supervised and scene-specific architectures that classify foreground pixels from background information. The reviewed methods span patch-based CNNs, multi-scale designs, encoder-decoder networks, and guided training strategies.

  • Overview: CDnet 2014’s leading methods include three supervised DNN approaches—FgSegNet, BSGAN, and Cascaded CNN—followed by three unsupervised multi-feature or multi-cue methods.The task compares a background image with the current image to label pixels as background or foreground.
  • Convolutional Neural Networks: ConvNet extracts temporal-median background images, generates scene-specific datasets, trains a CNN, and performs patch-based foreground subtraction.Image patches centered on each pixel are extracted from each video frame.
  • Convolutional Neural Networks: ConvNet is limited by high-level representation, scene-specific overfitting, isolated pixel errors, patch-processing cost, and dependence on preprocessing or post-processing.Its scene specificity results from highly redundant training data and requires retraining for other scenes.
  • Multi-scale and Cascaded CNNs: Multi-scale CNNs address size-related limitations by combining outputs generated at different patch scales.The basic 31 × 31-patch model is limited when foreground and background objects have the same size or less.
  • Multi-scale and Cascaded CNNs: Guided Multiscale CNN achieved an F-Measure of 0.7591, exceeding DeepBS and SuBSENSE on CDnet 2014.Reliable pixels from SubSENSE guide training because its initial outputs are not accurate enough to serve directly as ground truth.

4.3 Fully CNNs

Fully convolutional networks reduce reliance on fully connected layers while incorporating multi-scale, residual, atrous-convolution, and shortcut-connected designs. Reported evaluations show strong comparisons, but one study used only six CDnet 2012 sequences.

  • Fully Convolutional Neural Networks: FCNNs replace fully connected layers with convolutional layers to reduce disadvantages associated with fully connected architectures.Cinelli tested LeNet5 and ResNet architectures in this setting.
  • Fully Convolutional Neural Networks: The best Cinelli models were a 32-layer CIFAR-derived dilated network and a pre-trained 34-layer ILSVRC-based dilated model.Cinelli reported visual results without F-measure values.
  • Fully Convolutional Neural Networks: Yang et al.’s FCNN uses shortcut-connected blocks with multiple branches, atrous convolutions, PReLU activations, and CRF refinement.Atrous convolutions expand receptive fields to preserve spatial information.
  • Fully Convolutional Neural Networks: Yang et al.’s method outperformed MOG, Codebook, ViBe, PBAS, and P2M on CDnet 2012.The evaluation compared the method with traditional and recent state-of-the-art background subtraction methods.
  • Fully Convolutional Neural Networks: MV-FCN combines a fully convolutional structure with inception modules and residual networking to process multi-scale information and recover lost features.The inception modules apply filters of different scales to the same input.

4.4 Deep CNNs

Deep CNN approaches combine background initialization, foreground segmentation, reconstruction, and sequence modeling. Their scope varies from background subtraction to object detection, and some evaluations use datasets outside CDnet 2014.

  • Deep CNNs: DeepBS combines RGB average-model background initialization, CNN-based foreground subtraction, and spatial-median post-processing.SuBSENSE first distinguishes foreground and background pixels, after which background pixels initialize the model.
  • Deep CNNs: TS-CNN reconstructs background images in an encoder-decoder stage and uses the reconstruction with the current frame for multi-channel foreground detection.Reconstruction and segmentation losses are jointly optimized.
  • Deep CNNs: ADCNN transfers a generic CNN classifier to surveillance scenes by selecting useful kernels and learning scene context for object-location prediction.The method targets pedestrian and vehicle detection rather than the principle of background subtraction.
  • Deep CNNs: ADCNN was evaluated on the CUHK square, MIT traffic, and PETS 2007 datasets instead of CDnet 2014.This dataset choice limits direct comparison with methods evaluated on CDnet 2014.
  • Deep CNNs: Deep sequence learning uses video sequences, pixel-level semantic features, and convolutional encoder-decoder networks for moving-object detection.The framework uses VGG-16 and can also use GoogLeNet or ResNet50 architectures.

4.5 Structured CNNs

Struct-CNN uses an encoder-decoder architecture to transform RGB frames into pixel-level foreground masks, with superpixel information supporting foreground extraction.

  • Structured CNNs: Struct-CNN extracts a temporal-median RGB background, trains the network, performs subtraction, and extracts foreground using superpixel information.Its architecture resembles VGG16 without fully connected layers.
  • Structured CNNs: The encoder maps 336 × 336 RGB images to 21 × 21 × 512 feature vectors, while the decoder reconstructs 336 × 336 one-channel foreground masks.The decoder uses six deconvolutional and four unpooling layers.

4.6 3D-CNNs

3D-CNN approaches model temporal changes and spatio-temporal information directly, with reported gains over several conventional and CNN-based background-subtraction methods.

  • 3D-CNNs: 3D-CNN tracks temporal changes without a background model during training and handles multiple scenes without scene-specific fine-tuning.This design was reported to outperform ConvNet and deep CNN approaches.
  • 3D-CNNs: 0.9507 average F-Measure was obtained by 3D-CNN on the CDnet 2014 dataset.It also outperformed two illumination-invariant methods on the ESI dataset with extreme and sudden illumination changes.
  • 3D-CNNs: ReMotENet jointly models object appearance and motion using spatial-temporal attention-based 3D ConvNets.Separating spatial and temporal max pooling captures fine-grained temporal information while deeper layers learn representations.
  • 3D-CNNs: 4–8 milliseconds on a GPU was sufficient for ReMotENet to detect relevant motion in a 15-second video.The model was reported to be three to four orders of magnitude faster than object-detection methods and smaller than 1MB.
  • 3D-CNNs: 3D Atrous ConvLSTM combines atrous 3D convolutions with two ConvLSTM networks to capture short- and long-term spatio-temporal information without losing resolution.The end-to-end framework requires no pre- or post-processing and outperformed SuBSENSE, Cascaded CNN, and DeepBS on CDnet 2014.

4.7 CNNs with Different Features

CNN background-subtraction systems use specialized input representations for temporal pixel distributions and depth imagery, including scenes where color information is unavailable.

  • CNNs with Different Features: DPDL replaces intensity inputs with Random Permutation of Temporal Pixels features representing a pixel’s past-observation distribution.The temporal correlation between observations is deliberately not ordered over time.
  • CNNs with Different Features: BGSNet-D targets moving-object detection in depth videos, making it suitable for dark scenes where color information is difficult to obtain.Its design addresses edge noise and missing pixels in captured depth data.
  • CNNs with Different Features: Extended min–max normalization preprocesses depth images before CNN inputs combine an average depth background image with the current depth image.CNNs cannot be applied directly to depth images because captured data contain edge noise and pixel absence.

4.8 Generative Adversarial Networks

Generative adversarial approaches formulate background subtraction through learned foreground mappings or pixel classification, addressing changing illumination and complex backgrounds.

  • Generative Adversarial Networks: BScGAN uses a generator to map the background and current image to a foreground mask, while a discriminator learns the training loss.The discriminator compares predicted and ground-truth outputs while observing the input image and background.
  • Generative Adversarial Networks: BGAN first extracts the background with a median filter, then trains a Bayesian generative adversarial network to classify each pixel.The approach targets sudden and slow illumination changes, non-stationary backgrounds, and ghosts.
  • Generative Adversarial Networks: BPVGAN is a parallel version of the BGAN algorithm proposed in subsequent work.Deep convolutional neural networks are used to construct the Bayesian GAN generator and discriminator.
  • Generative Adversarial Networks: NUMOD is an end-to-end framework that operates in online or batch mode through parametrization by a generative neural network.It decomposes each frame into background, foreground, and illumination changes and generates a background model from a low-dimensional manifold.

5 Deep Learned Features

Deep learned features replace or augment hand-crafted cues for background modeling and foreground detection, with reported robustness and category-dependent performance differences.

  • 5 Deep Learned Features: Hand-crafted color, edge, texture, motion, and depth features address illumination changes, dynamic backgrounds, and camouflage but require an operator to fuse results.Feature selection or fusion is needed to combine outputs from different cues.
  • 5 Deep Learned Features: SDAE learns deep image representations encoding intrinsic scene information, improving robustness of feature description.A binary spatio-temporal background model then captures scene-distribution information in Hamming space for foreground detection.
  • 5 Deep Learned Features: SDAE outperformed MOG, KDE, LBP, and PBAS on the CDnet 2012 dataset.The comparison covers traditional methods and a recent state-of-the-art model.
  • 5 Deep Learned Features: MOG-NeRM globally outperformed MOG with RGB features and CHist on CDnet 2012, but not in intermittentObjectMotion, Low frame rate, Night video, and Thermal categories.NeRM uses ImageNet-trained first-layer StochasticNet responses as features for a MOG background model, with offline one-time formation.
  • 5 Deep Learned Features: MF-Net learns motion patterns from small image patches using three identical CNNs in a triplet network combined with a sample-based background model.A separate Factored 3-Way RBM approach learns global motion from sequential images and selects regions that violate that motion as foreground.

6 Adequacy for the background subtraction task

The review evaluates whether DNN architectures and processing choices fit background subtraction’s practical requirements, while examining how these networks extract and use background features. It identifies common design patterns, application challenges, and unresolved architecture adequacy questions.

  • Architectures and inputs: DNN background-subtraction methods use either the current image alone or both the background and current images as inputs.Some current-image-only methods generate a background image end to end, whereas others avoid computing one explicitly.
  • Architectures and inputs: Most studies adapt established architectures such as LeNet-5, VGG-16, and U-Net rather than designing architectures specifically for background subtraction.Only a few authors proposed fully task-designed architectures.
  • Feature processing: A quasi-similar ConvNet analysis found that early filters perform background subtraction and later layers categorize some background changes without supervised signals.This suggests that feature extraction and integration occur within the network, including automatic discovery of background features.
  • Application requirements: Effective background subtraction must address noise robustness, spatial and temporal coherence, incremental operation, real-time implementation, and video-sequence challenges.The review identifies these as application requirements rather than properties guaranteed by every architecture.
  • Application requirements: Deep learning methods learn deep background and foreground features during training, but pixel-wise DNNs need added spatial and temporal processing for coherence.Handling objects at very different scales is identified as a major challenge for DNN methods.
  • Architecture adequacy: ResNet produced the best results in the few architecture comparisons reported, although these networks were originally designed for image-classification datasets rather than background subtraction.The review therefore highlights the need to assess architecture type and parameters for the specific task and dataset.

7 Experimental results

The evaluation compares conventional, multi-cue, conventional neural, CNN, and GAN-based methods on CDnet 2014 using qualitative visual results and F-measures. Deep learning methods generally produce progressively improved foreground masks, while several difficult categories remain challenging.

  • Dataset and evaluation: CDnet 2014 extends CDnet 2012 with 22 additional camera-captured videos and five categories targeting previously unaddressed challenges.The dataset includes baseline, dynamic backgrounds, camera jitter, shadows, intermittent object motion, thermal, challenging weather, low frame-rate, night video, and PTZ conditions.
  • Dataset challenges: Night videos, low frame-rate sequences, and PTZ videos remain huge challenges, whereas baseline and bad-weather sequences can be handled efficiently by conventional methods.Dynamic backgrounds, thermal video, and camera jitter are described as reachable challenges for top-performing background-subtraction methods.
  • Dataset and evaluation: The comparison includes statistical, multi-cue, conventional neural, CNN, and GAN-based background-subtraction algorithms.The visual evaluation compares MOG, RMOG, SubSENSE, SC-SOBS, AAPSA, five CNN methods, and two GAN methods.
  • Qualitative evaluation: Cascaded CNN outperforms SC-SOBS and AAPSA visually except in the Low-frame Rate and Night Videos categories.FgSegNet and FgSegNet-SFPM also visually outperform DeepBS in Baseline and Thermal categories, while FgSegNet-V2 and GAN methods produce similar visual results.
  • Qualitative evaluation: Foreground masks improve progressively from statistical models to multi-cue models, conventional neural networks, and deep learning models.The qualitative comparison identifies this ordering by mask quality across the presented CDnet 2014 examples.

8 Conclusion

The paper reviews deep neural networks for background generation, subtraction, and learned features, then identifies unresolved architectural, camera-motion, input-feature, and model-family questions.

  • The review covers deep neural networks for background generation, background subtraction, and deep learned features, with experiments on CDnet 2014.
  • The central open question is which deep-network type and architecture best suit initialization, subtraction, and learned features under complex backgrounds.
  • Experiments indicate current deep neural networks encounter problems with moving cameras, while PTZ evaluations are often avoided or have modest F-Measure scores.
  • RGB-D inputs are proposed for challenges such as camouflage, alongside multispectral data and studies of input-feature influence.
  • Applying deep neural networks in the measurement domain with compressive-sensing data is identified as another future direction.
  • Only basic CNNs and GANs have been employed for background subtraction, leaving pyramidal CNNs and several other neural-network families for future investigation.
Loading 1811.05255v1…