PaperlayerView exact revision on arXiv

Source-linked AI summary

BSUV-Net: A Fully-Convolutional Neural Network for Background Subtraction of Unseen Videos

M. Ozan Tezcan, Prakash Ishwar, Janusz Konrad

arXiv:1907.11371v2cs.CV

TL;DR

Supervised background-subtraction methods have often used annotated frames from the test video, leaving their performance on completely unseen videos undocumented. BSUV-Net combines current and multi-timescale background frames with semantic information and illumination-focused augmentation. On CDNet-2014, it outperforms video-agnostic BGS algorithms on seven of nine performance metrics.

  • Problem

    Most top-performing supervised background-subtraction methods rely on annotated frames from test videos, so their performance on completely unseen videos is undocumented.

    Source: §1 · p. 1, §2.2 · p. 3, Selection of training test sets for… · p. 11

  • Method

    BSUV-Net is a supervised fully convolutional network that uses the current frame, two reference backgrounds at different time scales, semantic information, and illumination-change augmentation.

    Source: §1 · pp. 1–2, §3.1 · p. 4

  • Results

    On CDNet-2014, BSUV-Net outperforms state-of-the-art video-agnostic BGS algorithms on 7 out of 9 performance metrics.

    Source: §5 · p. 9

  • Takeaways & Limitations

    The results show potential for deep-learning BGS on unseen videos, and adding SemanticBGS as a post-processing layer can further improve performance.

    Source: §5 · p. 9

    The work did not consider computational complexity, and further work is planned on a shallower implementation for faster inference.

    Source: §5 · p. 9

Abstract

from arXiv · show

Background subtraction is a basic task in computer vision and video processing often applied as a pre-processing step for object tracking, people recognition, etc. Recently, a number of successful background-subtraction algorithms have been proposed, however nearly all of the top-performing ones are supervised. Crucially, their success relies upon the availability of some annotated frames of the test video during training. Consequently, their performance on completely "unseen" videos is undocumented in the literature. In this work, we propose a new, supervised, background-subtraction algorithm for unseen videos (BSUV-Net) based on a fully-convolutional neural network. The input to our network consists of the current frame and two background frames captured at different time scales along with their semantic segmentation maps. In order to reduce the chance of overfitting, we also introduce a new data-augmentation technique which mitigates the impact of illumination difference between the background frames and the current frame. On the CDNet-2014 dataset, BSUV-Net outperforms state-of-the-art algorithms evaluated on unseen videos in terms of several metrics including F-measure, recall and precision.

1 Introduction

Supervised background-subtraction methods had largely been evaluated on videos represented in their training data, leaving performance on unseen videos undocumented. BSUV-Net targets that gap with a fully convolutional network and multiple inputs, and outperforms state-of-the-art algorithms evaluated on unseen videos.

1 Introduction

  • The paper identifies a generalization gap: most supervised methods were tuned to one video or similar videos, leaving unseen-video performance unevaluated.The cited example, FgSegNet, trains on 200 frames from a test video and shows a significant performance drop when applied to an unseen video.Source: §1 · p. 1
  • BSUV-Net trains on frames from different videos than those used for testing, so test-video ground-truth data are absent from training.It uses the current frame, semantic information, and two background frames representing different time scales.Source: §1 · pp. 1–2
  • Its two background references address intermittently static objects and illumination changes, while illumination-focused augmentation further improves resilience to changing light.Semantic information is also provided as an input to improve foreground-boundary segmentation accuracy.Source: §1 · p. 2
  • BSUV-Net outperforms state-of-the-art background-subtraction algorithms evaluated on unseen videos.The authors report extensive experiments on CDNet-2014.Source: §1 · p. 2

2 Related Work

Related work spans background-modeling algorithms, supervised methods with different training and testing regimes, and methods that improve BGS through post-processing. The paper emphasizes that prior supervised approaches generally lack evaluation on wholly unseen videos.

2 Related Work

  • The review groups recent top-performing methods into background modeling, supervised BGS, and post-processing approaches.It does not aim to cover every BGS variant.Source: §2 · p. 2

2.1 BGS by Background Modeling

  • Traditional background models range from per-pixel mean or median values to probabilistic and sample-based models, with later methods adding texture, feedback, or dynamic updates.The review also describes semantic segmentation used to improve a background model.Source: §2.1 · p. 2

2.2 Supervised BGS

  • Video-optimized and video-group-optimized supervised methods use frames from evaluation videos during training; video-agnostic methods train on separate videos for unseen-video evaluation.The paper notes that the top-performing supervised methods to that point belonged to the first two groups.Source: §2.2 · p. 3
  • CNN-based BGS progressed from patch-wise prediction to full-frame and multiscale networks, but the reviewed approaches still used test-video data during training.The review also notes that 3DFR reported results on only 10 of CDNet-2014's 53 videos.Source: §2.2 · p. 3

2.3 Improving BGS Algorithms by Post-Processing

  • Post-processing approaches improve BGS by using semantic segmentation predictions or by combining outputs from multiple algorithms.The review describes a semantic-label grouping approach and methods using genetic programming or a fully convolutional network to fuse BGS outputs.Source: §2.3 · p. 3

3 Proposed Algorithm: BSUV-Net

BSUV-Net predicts foreground in unseen videos using current and two time-scale background frames, each paired with semantic information. Its fully convolutional network uses a relaxed Jaccard loss and illumination augmentation.

3.1 Inputs to BSUV-Net

  • Two background references balance an empty frame useful for intermittently static foreground against a recent frame that better reflects changing illumination but may contain foreground.The empty reference may become stale over time; the recent reference can suppress objects present for fewer than 50 frames in its temporal median.Source: §3.1 · p. 4
  • Current, recent, and empty RGB frames each paired with a foreground probability map form the network’s 12-channel input.Semantic maps are derived from DeepLabv3 class probabilities, with selected object classes grouped as foreground.Source: §3.1 · p. 4

3.2 Network Architecture and Loss Function

  • A residual U-Net encoder-decoder produces pixel-wise foreground probabilities, while relaxed Jaccard loss addresses foreground-background class imbalance.The network uses batch normalization and spatial dropout; its fully convolutional design accepts variable frame sizes, with padding sometimes needed for pooling.Source: §3.2 · pp. 4–5, Figure 1 · p. 5

3.3 Resilience to Illumination Change by Data Augmentation

  • Random global RGB shifts to the empty reference frame during training augment illumination variation and are intended to improve resilience to it.The augmentation addresses illumination differences between an earlier empty background frame and the current frame, which are limited in CDNet-2014.Source: §3.3 · p. 5

4 Experimental Results

On CDNet-2014 unseen videos, BSUV-Net outperforms video-agnostic competitors on almost all metrics, with further gains from SemanticBGS post-processing. The experiments also examine per-category performance, visual results, and the effects of input and background-frame choices.

4.1 Dataset and Evaluation Metrics

  • The evaluation uses 18 video-level train/test splits so every CDNet-2014 video is tested without appearing in that split’s training set.CDNet-2014 contains 53 videos across 11 categories; pixels labeled unknown motion were excluded, and hard-shadow pixels were treated as background.Source: §4.1 · p. 5, §4.2 · p. 6

4.3 Quantitative Results

  • BSUV-Net outperforms video-agnostic competitors on almost all metrics, and SemanticBGS post-processing improves it further and outperforms the available video-agnostic algorithms.Video-optimized methods were omitted from the comparison; post-processing methods were reported separately.Source: §4.3 · p. 6, Table 2 · p. 7
  • BSUV-Net achieves the best per-category F-measure in 5 of 11 categories, with a striking advantage in night scenes.The authors link its night performance to illumination invariance amid local illumination changes.Source: §4.3 · pp. 6–7, Table 3 · p. 7
  • For camera jitter and dynamic backgrounds, median-derived background frames blur the background, leading to false-positive foreground predictions.The authors suggest more advanced background models may improve dynamic-background performance.Source: §4.3 · p. 7, §4.4 · p. 7

4.4 Visual Results

  • Visual comparisons show BSUV-Net handling night illumination changes, shadow regions, and intermittently static objects better than SWCD and WisenetMD in the reported examples.The comparison covers sample frames from eight categories; dynamic-background examples still show missed background pixels around foreground objects.Source: §4.4 · p. 7, Figure 2 · p. 8

4.5 Ablation Study

  • Ablations show that background frames improve over current-frame-only input, while temporal augmentation and FPM channels further improve performance.The final network achieves state-of-the-art results; manual empty-background selection slightly improves precision but significantly decreases recall versus using the video-wide median.Source: §4.5 · pp. 8–9, Table 4 · p. 8, Table 6 · p. 9

5 Conclusions and Future Work

BSUV-Net combines video-agnostic evaluation with a supervised model for unseen videos, outperforming state-of-the-art video-agnostic BGS algorithms on most reported metrics. The authors identify computational complexity and challenging categories as areas for future work.

5 Conclusions and Future Work

  • 7 of 9 performance metrics favor BSUV-Net over state-of-the-art video-agnostic BGS algorithms on CDNet-2014; SemanticBGS post-processing further improves performance.The authors see potential for deep-learning BGS algorithms designed for unseen or unlabeled videos.Source: §5 · p. 9
  • BSUV-Net uses the current frame, two reference frames from different time scales, semantic information for all three, and an illumination-change model.The semantic maps are computed using Deeplabv3 [9].Source: §5 · p. 9
  • Computational complexity was not considered; future work includes temporal augmentation, alternative reference-background models, and a shallower implementation for faster inference.The authors also target improved performance on dynamic-background and camera-jitter categories.Source: §5 · p. 9

Selection of training test sets for evaluation on unseen videos

Because CDNet-2014's supplied frame splits do not evaluate unseen videos, the authors created 18 video-level training/testing splits. Their design aimed to cover relevant scenarios in training, while omitting full leave-k-videos-out cross-validation because of training time.

Selection of training test sets for evaluation on unseen videos

Related papers

Loading 1907.11371v2…