Source-linked AI summary
Foreground Segmentation Using a Triplet Convolutional Neural Network for Multiscale Feature Encoding
Long Ang Lim, Hacer Yalim Keles
TL;DR
Moving-object segmentation from video captured by stationary or nonstationary cameras is a crucial computer-vision problem. The paper proposes a supervised triplet encoder-decoder network for multiscale RGB inputs and reports state-of-the-art performance on CDnet2014, including an overall F-Measure of 0.9545 with 50 training frames and first-place challenge ranking.
Problem
Moving-object segmentation from video captured by stationary or nonstationary cameras is a crucial computer-vision problem for surveillance, tracking, recognition, monitoring, motion estimation, and anomaly detection.
Method
The method adapts VGG-16 in a weight-sharing triplet configuration to encode three image scales, combines their embeddings, and uses transposed convolutions to produce pixel-level foreground masks.
Results
0.9545 overall F-Measure across 11 categories with 50 training frames, while the method outperformed current state-of-the-art methods and ranked first in the CDnet2014 challenge evaluation.
Takeaways & Limitations
The network produces high-accuracy masks across challenging scenes, including accurate small-object boundaries, dynamic-background and shadow removal, and robustness to large camera motion without post-processing.
Takeaways & Limitations
Training uses isolated frames and does not consider temporal sequences, which the authors identify as a direction for future redesign.
Abstract
from arXiv · showhide
A common approach for moving objects segmentation in a scene is to perform a background subtraction. Several methods have been proposed in this domain. However, they lack the ability of handling various difficult scenarios such as illumination changes, background or camera motion, camouflage effect, shadow etc. To address these issues, we propose a robust and flexible encoder-decoder type neural network based approach. We adapt a pre-trained convolutional network, i.e. VGG-16 Net, under a triplet framework in the encoder part to embed an image in multiple scales into the feature space and use a transposed convolutional network in the decoder part to learn a mapping from feature space to image space. We train this network end-to-end by using only a few training samples. Our network takes an RGB image in three different scales and produces a foreground segmentation probability mask for the corresponding image. In order to evaluate our model, we entered the Change Detection 2014 Challenge (changedetection.net) and our method outperformed all the existing state-of-the-art methods by an average F-Measure of 0.9770. Our source code will be made publicly available at https://github.com/lim-anggun/FgSegNet.
1 Introduction
Moving-object segmentation is important for video-surveillance and related vision applications, but background subtraction struggles with challenging scene conditions. The paper proposes a triplet-CNN encoder with a transposed-convolutional decoder for robust foreground segmentation.
- Moving-object segmentation supports video surveillance, human tracking, action recognition, traffic monitoring, motion estimation, and anomaly detection.
- Background subtraction classifies moving objects as foreground and non-moving objects as background pixels.
- Robust segmentation must handle illumination changes, shadows, dynamic backgrounds, camera motion, and camouflage effects.
- The proposed encoder-decoder uses a triplet CNN based on adapted pre-trained VGG-16 features and a transposed convolutional decoder.
- The decoder maps multiscale features to a pixel-level foreground probability map, which is thresholded into binary segmentation labels.
- The method significantly improves average F-Measure and average MCC across 11 CDnet2014 categories.
2 Related Works
Prior foreground-segmentation work includes parametric, non-parametric, and deep-learning methods, with trade-offs in adaptability and computational cost. This paper reports scene-specific models trained with few frames that ranked first in the CDnet2014 Challenge.
- Foreground segmentation requires a robust background model to identify moving regions throughout an image sequence.
- Classical background models are sensitive to dynamic background changes, motivating probabilistic methods such as Gaussian Mixture Models.
- ViBe and related non-parametric methods improve robustness to small camera movements, noise, and camouflage-related conditions.
- Deep-learning approaches include computationally expensive patch-based CNNs and image-wise scene-specific networks trained from selected frames.
- 0.9770 overall F-Measure was achieved using scene-specific models trained with 50 and 200 frames, ranking first in the CDnet2014 Challenge.
3 The Method
The method selects scene-specific training frames and uses a triplet, multiscale VGG-16 encoder with a transposed-convolution decoder to produce dense foreground masks.
- Training Examples Selection: Training uses 50 frames for relatively static scenes and 200 frames for complex, dynamic, or camera-moving sequences.Selected frames may contain foreground, background, or both, with more varied scene coverage recommended for difficult sequences.
- Working with imbalanced data: Foreground segmentation faces severe class imbalance because surveillance scenes may contain far more background than foreground pixels.Reported ratios can reach 100:1, 10000:1, or 10000:0, so the method addresses imbalance algorithmically rather than through data-level handling.
- FgSegNet Network Architecture: The encoder processes full-size and downscaled RGB images in three parallel, weight-sharing CNNs to capture contextual information at multiple scales.The input is downscaled by a factor of two, and the three resulting images are embedded before feature fusion.
- TCNN Configuration: The decoder concatenates multiscale features, projects their 1536-channel depth with 1x1 transposed convolutions, and upsamples them into a dense probability mask.Later transposed-convolution blocks enlarge the feature maps to the original input size; the final layer produces a grayscale output.
- Training Details: The training loss excludes unknown non-Region-of-Interest and object-boundary labels, which the authors observed made pixel predictions more confident.The network is trained with binary cross entropy on labeled pixels while deliberately avoiding these unknown regions.
4 Experiments
Experiments evaluate thresholding, metrics, training-frame budgets, category-wise performance, benchmark comparisons, robustness, and runtime on CDnet2014. The method achieves strong overall segmentation, including with few training frames, while low frame rate and severe PTZ blur remain difficult cases.
- Thresholding and metrics: A threshold converts per-pixel probability masks into binary segmentation masks; 0.9 performs best with 200 frames, 0.7 with 50 frames, and 0.8 is fixed for all experiments.No post-processing such as CRF or other graphical models is applied after thresholding.
- Training-frame experiments: 0.9734 overall F-Measure is obtained with 200 training frames, compared with 0.9545 using 50 frames on test frames across 11 categories.The reported values exclude training frames from evaluation.
- Training-frame experiments: 6.5% lower F-Measure occurs in lowFrameRate with 50 frames than with 200, although the 50-frame experiment still reaches 0.9545 overall and outperforms state-of-the-art methods.The lowFrameRate category contains extremely small objects at low frame rates.
- Benchmark comparison: The method outperforms current state-of-the-art methods and ranks first in the CDnet 2014 challenge evaluation, including dataset-wide results with additional frames.A separate comparison uses publicly available frame-level masks and training frames for comparison with the current best method.
- Category-wise results: More than 0.95 F-Measure is obtained in both cameraJitter and PTZ categories, while qualitative results show accurate boundaries and removal of dynamic backgrounds and shadows.The method is also reported as robust to large camera motions, although PTZ remains a lower-performing category in difficult blurred scenes.
- Runtime: Testing 1,500 remaining frames takes around 1.39 minutes, corresponding to about 18 frames per second, and the method is faster than the best previous method in training and segmentation speed.For a 1,700-frame 320x240 sequence with 200 training frames, training takes around 23.7 minutes for 50 epochs.
5 Conclusion
The model is a supervised encoder-decoder network that produces high-accuracy foreground masks across challenging scenes, using multiscale RGB inputs and an adapted triplet VGG-16 encoder. The work also addresses training-data selection and class imbalance, while identifying the absence of temporal learning as future work.
- 5 Conclusion: The model uses a supervised encoder-decoder architecture that processes multiscale raw RGB images and produces high-accuracy segmentation masks without post-processing.VGG-16 is adapted under a triplet configuration, with transposed convolutions upscaling feature maps to image space.
- 5 Conclusion: Training examples are selected to increase scene variety across video sequences, including frames with and without foreground objects.This strategy samples different parts of the video sequence to broaden scene coverage.
- 5 Conclusion: Weight penalization is introduced to address imbalanced foreground-background classifications, especially when foreground pixels are predicted as background.The penalty targets errors that misclassify foreground pixels as background.
- 5 Conclusion: The method is reported as robust across illumination changes, background or camera motion, camouflage, and shadows, for both indoor and outdoor scenes.The authors also report that it outperforms existing methods, including the previous best deep-learning method.
- 5 Conclusion: The model learns from isolated frames rather than temporal sequences, and future work proposes 3D convolutions with different fusion techniques for temporal data.Temporal information is therefore outside the model's current training setup.