Source-linked AI summary
Learning Multi-scale Features for Foreground Segmentation
Long Ang Lim, Hacer Yalim Keles
TL;DR
Foreground segmentation remains difficult under illumination changes, shadows, dynamic backgrounds, camera motion, camouflage, and subtle regions, while stationary background models adapt poorly to such scenarios. The paper proposes a few-example, end-to-end encoder-decoder network that fuses multi-scale features within an improved FPM and reports outperforming existing state-of-the-art methods across several benchmarks.
Problem
Foreground extraction is challenging in videos with illumination changes, shadows, dynamic backgrounds, camera motion, camouflage, and subtle regions, while stationary background models adapt poorly.
Method
The method improves FPM by fusing multi-scale features and uses global average pooling from encoder features to guide high-level decoder features.
Results
The network outperforms existing state-of-the-art methods in several benchmarks while requiring only a few training examples and no temporal data.
Takeaways & Limitations
Multi-scale feature fusion makes feature pooling more robust to camera motion and can alleviate the need for multi-scale network inputs.
Takeaways & Limitations
Performance is weaker for extremely small foreground objects in dynamic scenes with gradual illumination changes and fails to detect objects completely blended into the background.
Abstract
from arXiv · showhide
Foreground segmentation algorithms aim segmenting moving objects from the background in a robust way under various challenging scenarios. Encoder-decoder type deep neural networks that are used in this domain recently perform impressive segmentation results. In this work, we propose a novel robust encoder-decoder structure neural network that can be trained end-to-end using only a few training examples. The proposed method extends the Feature Pooling Module (FPM) of FgSegNet by introducing features fusions inside this module, which is capable of extracting multi-scale features within images; resulting in a robust feature pooling against camera motion, which can alleviate the need of multi-scale inputs to the network. Our method outperforms all existing state-of-the-art methods in CDnet2014 dataset by an average overall F-Measure of 0.9847. We also evaluate the effectiveness of our method on SBI2015 and UCSD Background Subtraction datasets. The source code of the proposed method is made available at https://github.com/lim-anggun/FgSegNet_v2 .
1 Introduction
Foreground segmentation remains difficult in challenging video conditions, while existing approaches often rely on stationary background models. The paper proposes an encoder-decoder network with multi-scale feature fusion and encoder-guided decoding, designed to achieve accurate segmentation from few examples.
- Foreground extraction is challenging under illumination changes, shadows, dynamic backgrounds, camera motion, camouflage, and subtle regions.
- Stationary background-model approaches are not very effective at adapting to challenging scenarios.
- The method modifies FgSegNet’s Feature Pooling Module to capture wide-range multi-scale information and improve robustness against camera movements.
- The decoder uses global average pooling from the encoder to guide high-level decoder features.
- The network can be trained end-to-end with only a few training examples and without temporal data, while producing highly accurate segmentation results.
- Fusing multi-scale features inside the FPM produces robust feature pooling against camera motion and can reduce the need for multi-scale network inputs.
- The paper reports that the method exceeds state-of-the-art performance on CDnet2014, SBI2015, and UCSD Background Subtraction datasets.
- The authors provide ablation studies and publicly release the source code to facilitate future research.
2 Related Works
Related work spans conventional background modeling and deep-learning approaches for foreground segmentation. The paper builds on multi-scale feature extraction and single-input encoder designs while addressing training efficiency and contextual limitations in earlier methods.
- Foreground segmentation, also called background subtraction, is a major computer-vision task with many proposed approaches.
- Conventional methods commonly construct a background model for each video sequence using statistical, parametric, or nonparametric techniques.
- Patch-wise CNN training can be computationally inefficient, overfit because of redundant pixels, lose higher-context information, and require many training patches.
- Deep-learning methods may instead process whole-resolution images, use temporal data, or combine frames with generated background models.
- Dilated convolution enlarges the network field of view without increasing the number of learned parameters.
- The earlier FPM used parallel dilated-convolution layers on a single-input encoder to provide results comparable to multi-input encoders.
3 The Method
The method revisits FgSegNet’s encoder and Feature Pooling Module, introducing multi-scale feature fusion and decoder guidance from encoder features. The resulting architecture uses M-FPM, Global Average Pooling, and an instance-normalized decoder.
- Encoder: The method retains FgSegNet’s modified VGG-16 encoder, using the first four blocks while removing the fifth block and third max-pooling layer.Dropout layers are inserted after convolutional layers, and the modified encoder is fine-tuned.
- Modified FPM: M-FPM progressively fuses features across dilation rates 4, 8, and 16 to produce wider-receptive-field multi-scale representations.The fused output concatenates five 64-feature groups into F′.
- Modified FPM: M-FPM replaces BatchNormalization with InstanceNormalization and applies SpatialDropout to reduce correlation between pooled feature maps.SpatialDropout drops entire 2D feature maps at rates such as 0.25 when adjacent pixels are strongly correlated.
- Decoder: The decoder contains three 3x3 convolutional layers and one 1x1 projection layer, with sigmoid activation producing the output image-space prediction.The 3x3 layers use 64 feature maps, while the 1x1 layer uses one feature slice.
- Decoder: Two Global Average Pooling coefficient vectors scale decoder features using low-level encoder information before the features are added to the originals and upscaled.The GAP module adds very slight computational cost while improving overall performance.
4 Training Protocol
The models are implemented and trained with a fixed protocol based on the prior work, then evaluated using F-Measure and Percentage of Wrong Classifications. The evaluation maximizes F-Measure and minimizes PWC.
- Implementation: The models are implemented in Keras with a TensorFlow backend and trained on a single NVIDIA GTX 970 GPU.The implementation follows the prior work’s training procedure.
- Optimization: Training uses batch size 1, RMSProp with rho 0.9 and epsilon 1e-08, and an initial learning rate of 1e-4.The learning rate is reduced by a factor of 10 when validation loss stops improving for 5 epochs.
- Optimization: Only the last modified VGG-16 block is fine-tuned while the pretrained coefficients of the original VGG-16 network are retained.Training is capped at 100 epochs with early stopping based on validation loss.
- Metrics: Performance is evaluated primarily with F-Measure and Percentage of Wrong Classifications, maximizing the former and minimizing the latter.F-Measure uses true-positive, false-positive, and false-negative counts; PWC uses all four confusion-matrix counts.
5 Results and Discussion
Experiments across CDnet2014, SBI2015, and UCSD evaluate the proposed architecture under challenging conditions and limited training data. Ablations and comparisons indicate benefits from GAP and M-FPM, while performance remains weaker for tiny or blended foreground objects.
- The evaluation covers CDnet2014, SBI2015, and UCSD Background Subtraction, all containing challenging foreground/background segmentation scenarios.
- Global Average Pooling Experiments: 2.34% points: GAP improves over no GAP in the cameraJitter category.With GAP, the network improves over no GAP in most categories.
- Modified FPM Experiments: M-FPM produces fewer false positives than the original FPM, while the proposed decoder improves over the FgSegNet family decoder in a challenging scene.The authors report that later multi-scale feature fusion makes feature pooling more robust and can mitigate the need for computationally more expensive multi-input features.
- Limited-Training Experiments: 25 frames: the study further reduces training examples by 8x from the 200-frame experiment for CDnet2014 evaluation.The 25- and 200-frame experiments use manually or randomly selected frames across CDnet2014 categories.
- State-of-the-Art Comparisons: 0.43%, 0.77%, 6.38% and 23.89% points: the method improves over FgSegNet S, FgSegNet M, Cascade CNN and DeepBS, respectively, on the Change Detection 2014 Challenge.It is ranked number 1 at the time of submission and significantly improves on camera-motion categories such as PTZ and cameraJitter.
- Limitations: The method performs poorly in LowFrameRate, with an F-Measure of 0.9579, and fails to detect objects blended completely into the background.The reported LowFrameRate difficulty involves extremely small foreground objects in dynamic scenes with gradual illumination changes.
6 Conclusion
The work presents an end-to-end supervised encoder-decoder network designed to learn accurate foreground segmentation from few examples. Its fused multi-scale FPM and simple decoder support robustness to camera motion without temporal data or post-processing.
- The proposed encoder-decoder network is trained end-to-end in a supervised manner using only a few training examples.This reduces the ground-truth labeling burden associated with training data preparation.
- The method is evaluated on SBI2015 and UCSD, with UCSD results reported alongside comparisons to state-of-the-art methods.The UCSD comparison includes the proposed results, FgSegNet S, and FgSegNet M, while Table 6 reports test results at threshold 0.6.
- Fusing multiple-scale features inside the FPM produces robust feature pooling against camera motion and reduces the need for multi-scale network inputs.
- The method uses a simple decoder to improve segmentation performance without requiring temporal data or post-processing.
- The authors conclude that the network outperforms existing state-of-the-art methods across several benchmarks.