PaperlayerView exact revision on arXiv

Source-linked AI summary

Robust Online Matrix Factorization for Dynamic Background Subtraction

Hongwei Yong, Deyu Meng, Wangmeng Zuo, Lei Zhang

arXiv:1705.10000v1cs.CV

TL;DR

Real-time background subtraction must handle changing foregrounds and backgrounds without relying on fixed noise distributions. The proposed online method learns frame-specific foreground mixtures and background knowledge, incorporates affine transformations, and reaches real-time speed while adapting to dynamic video variations.

  • Problem

    Existing online methods use fixed Gaussian- or Laplacian-based loss terms, which do not represent foregrounds with dramatic, multimodal variations across video frames.

  • Method

    The method models each frame's foreground with a mixture of Gaussians regularized by prior foreground/background knowledge, while affine operators accommodate background transformations.

  • Results

    More than 25 FPS enables real-time video processing while the method adapts to dynamic foreground and background variations and affine transformations.

  • Takeaways & Limitations

    The approach provides adaptive online background subtraction for practical videos with changing foregrounds, backgrounds, and transformations.

  • Takeaways & Limitations

    The evaluation focuses on general surveillance videos without supervised foreground/background knowledge and does not compare methods requiring extensive pre-annotations.

Abstract

from arXiv · show

We propose an effective online background subtraction method, which can be robustly applied to practical videos that have variations in both foreground and background. Different from previous methods which often model the foreground as Gaussian or Laplacian distributions, we model the foreground for each frame with a specific mixture of Gaussians (MoG) distribution, which is updated online frame by frame. Particularly, our MoG model in each frame is regularized by the learned foreground/background knowledge in previous frames. This makes our online MoG model highly robust, stable and adaptive to practical foreground and background variations. The proposed model can be formulated as a concise probabilistic MAP model, which can be readily solved by EM algorithm. We further embed an affine transformation operator into the proposed model, which can be automatically adjusted to fit a wide range of video background transformations and make the method more robust to camera movements. With using the sub-sampling technique, the proposed method can be accelerated to execute more than 250 frames per second on average, meeting the requirement of real-time background subtraction for practical video processing tasks. The superiority of the proposed method is substantiated by extensive experiments implemented on synthetic and real videos, as compared with state-of-the-art online and offline background subtraction methods.

1 INTRODUCTION

The paper targets online background subtraction that must handle changing foreground distributions and dynamic backgrounds, including camera jitters. It proposes an adaptive MoG-based method with affine alignment, sub-sampling, and extensions to related alignment tasks.

  • Existing online methods struggle with real videos because camera translation, rotation, scaling, and other changes damage their low-rank background assumptions.
  • Fixed Gaussian or Laplacian noise models cannot represent the dramatic, multimodal foreground variations that occur across video frames.
  • The proposed method models each frame’s foreground as a separate MoG distribution regularized toward parameters learned from previous frames.
  • An automatically fitted affine transformation aligns changing video backgrounds, supporting translation, rotation, scaling, and combinations while improving robustness to camera jitters.
  • Sub-sampling accelerates the method to more than 250 frames per second on average while retaining good accuracy for real-time processing.
  • The method can also extend to image alignment and video stabilization applications.

2 RELATED WORK

Related work progresses from statistical pixel models to low-rank, online, and transformation-aware methods, but existing approaches remain limited by fixed noise assumptions and changing backgrounds. The paper positions its method as addressing both foreground and background variation.

  • Low-rank methods incorporate temporal background similarity and foreground structure such as sparsity and spatial continuity.
  • Traditional background subtraction evolved from mean, median, and histogram models to Gaussian mixtures, but these approaches largely ignore temporal and spatial video structure.
  • The paper focuses on general unsupervised surveillance videos and excludes supervised methods that require large amounts of pre-annotated data.
  • Online methods improve efficiency by updating one frame at a time, yet still inadequately model foreground and background variation in real videos.
  • Transformation-aware batch methods such as RASL address camera jitters by jointly optimizing transformations with low-rank backgrounds and sparse foregrounds.
  • A simple Laplacian loss cannot finely represent complex foreground configurations, motivating the proposed adaptive foreground modeling.

3 ONLINE MOG-LRMF

The paper introduces MoG-LRMF as the modeling basis for representing complex foreground variations within low-rank matrix factorization.

  • MoG-LRMF is introduced as the method most closely related to the paper’s strategy for modeling foreground variations.

3.1 MoG-LRMF Review

The MoG-LRMF review recasts low-rank factorization noise modeling probabilistically. It replaces fixed Gaussian or Laplacian assumptions with a mixture of Gaussians that can approximate more complex noise distributions and be estimated with EM.

  • LRMF represents a data matrix as a low-rank product of basis and coefficient matrices, with optional missing-entry weighting.
  • L2- and L1-norm losses correspond to Gaussian and Laplacian noise assumptions, respectively.
  • These single-distribution assumptions can diverge from real data containing more complicated noise configurations.
  • MoG models are selected because their approximation capability allows the noise term to adapt more flexibly to general distributions.
  • The MoG parameters and latent responsibilities can be estimated with the EM algorithm, followed by weighted L2-LRMF updates for the subspace parameters.

3.2 Online MoG-LRMF: Model

OMoGMF models each incoming frame with a foreground MoG and a background subspace, both regularized by knowledge learned from previous frames. Its probabilistic MAP formulation combines current-frame likelihood with priors that stabilize temporal updates.

  • Probabilistic modeling: OMoGMF fits a separate foreground MoG and background subspace for each newly arriving frame.The foreground parameters and background coefficients/subspace are updated under regularization from earlier frames.
  • Probabilistic modeling: The model is presented as a graphical Bayesian model with conjugate priors for MoG variances and weights and a prior for the background subspace.The full posterior is formed after marginalizing the latent assignment variable.
  • Foreground prior: The foreground regularizer encourages the current MoG distribution to approximate the previously learned distribution through KL-divergence equivalence.The prior parameters encode previous noise knowledge and discourage excessive deviation in the current frame.
  • Objective function: The MAP model combines the current-frame likelihood with regularization terms for foreground noise and background subspace.The likelihood adapts parameters to the current frame, while the regularizers constrain temporal changes.
  • Background prior: The background regularizer uses a Mahalanobis distance to keep each current subspace row near its counterpart from the previous frame.The parameters N_t−1 and ρ control prior strength for the foreground and background updates.

3.3 Online MoG-LRMF: Algorithm

The OMoGMF algorithm solves the online model with alternating EM updates on each new frame. Efficient closed-form updates and limited subspace refinement support fixed-memory, online processing.

  • Online EM: Online EM alternates expectation and maximization steps to update the latent assignments and model parameters for each new frame.The M-step updates MoG parameters, coefficients, and the background subspace.
  • Online M-step: The MoG parameters are updated through a closed-form M-step, while the coefficient vector is obtained from a weighted least-squares problem.The weights depend on the component variances.
  • Subspace update: Each row of the background subspace has a closed-form update that avoids matrix inversions in the stated recursive formulation.Only fixed-size auxiliary matrices need to be retained from the previous update.
  • Subspace update: The subspace is fine-tuned once after recursive EM updates converge with the previous subspace held fixed.The remaining inverse is an r × r matrix, where r is generally much smaller than the data dimensions.
  • Initialization: Initialization uses PCA on a small starting batch, MoG estimation on extracted noise, and auxiliary matrices for subspace learning.The algorithm is summarized as online MoG-LRMF in Algorithm 1.

3.4 Several remarks

The remarks interpret OMoGMF's priors as KL regularization and relate its online updates to a batch formulation under specific parameter settings. They also describe how prior strength controls adaptation over time.

  • Prior interpretation: Conjugate priors for Gaussian and multinomial distributions can be written as equivalent KL-divergence regularizers.The KL form is preferred because it makes the deterministic noise-regularization effect more intuitive.
  • Batch relationship: The online implementation does not retain or recompute all earlier latent variables, coefficients, subspaces, and MoG parameters.This limits direct recovery of the full batch solution from previous frames.
  • Parameter remarks: Using a moderate history scale, such as K=50, balances adaptation to temporal foreground changes against sensitivity to single-frame changes.The experiments set ρ=0.98 so the updated subspace leans slightly toward the current frame.

3.5 Efficiency and Accuracy Amelioration

OMoGMF improves efficiency through sub-sampling and improves foreground accuracy through TV regularization. These choices require balancing sampling rate against accuracy and use MoG variance to set the TV parameter.

  • Efficiency: The method exceeds 250 frames per second on average in Matlab while maintaining good accuracy for real-time processing.The speedup is attributed to embedding sub-sampling into the calculations.
  • Sub-sampling: Sub-sampling updates the model using randomly selected frame entries rather than the full incoming frame.MoG parameters, coefficients, and selected subspace rows are updated from the sampled data.
  • Sub-sampling: The sub-sampling rate creates an efficiency–accuracy trade-off: complex backgrounds require more sampled entries, while evidently low-rank backgrounds can use fewer.The text states that sampling rate is generally inversely proportional to performance.
  • TV-norm regularization: TV minimization imposes spatial smoothness on the extracted foreground to improve foreground-object detection.The method uses TV regularization because foreground objects generally form continuous, spatially smooth regions.
  • TV-norm regularization: The TV weight is set empirically to about 1.5eσ2, where eσ2 is the largest variance among MoG components.The authors report good performance with this setting throughout their experiments.

3.6 Transformed Online MoG-LRMF

The transformed online MoG-LRMF method embeds an automatically optimized geometric transformation into online MoG matrix factorization, aligning changing backgrounds before subspace learning. Iterative updates refine transformation and model parameters, improving robustness to transformed video backgrounds.

  • 3.6 Transformed Online MoG-LRMF: The method embeds affine or projective transformations into online MoG matrix factorization to align changing backgrounds during learning.Transformation parameters are optimized jointly with foreground, background subspace, and noise parameters.
  • 3.6 Transformed Online MoG-LRMF: The nonlinear transformation is optimized through local linearization using the Jacobian, followed by iterative updates of transformation and model parameters.The algorithm updates MoG parameters, v, Δτ, and τ until convergence before updating the background subspace.
  • 3.6 Transformed Online MoG-LRMF: Residuals improve across iterations as t-OMoGMF gradually rectifies the transformation and aligns frames from transformed airport and curtain sequences.The figure illustrates this alignment process on transformed frames from the Li dataset.
  • 3.6 Transformed Online MoG-LRMF: The method can be extended to iterative frame-recursive updating, repeatedly processing initialization frames to obtain a better starting subspace.This extension is called iterative t-OMoGMF, or it-OMoGMF.
  • 3.6 Transformed Online MoG-LRMF: An adaptive activation strategy can select t-OMoGMF when foreground sparsity increases or estimated transformations depart from identity.These conditions are intended to improve adaptation to real scenes.

4 EXPERIMENTS

Experiments evaluate the proposed online methods on Li dataset videos spanning static, illumination-changing, and dynamic backgrounds, using F-measure and processing speed. The methods achieve strong foreground detection, adapt to background changes and missing entries, and retain accuracy under aggressive sub-sampling.

  • 4 EXPERIMENTS: The experiments use nine Li dataset video sequences covering static, illumination-changing, indoor dynamic, and outdoor dynamic backgrounds.Each video contributes 200 training frames, with annotated foreground frames used for evaluation.
  • 4.1 On videos without camera jitter: OMoGMF and OMoGMF+TV achieve better average F-measures than competing methods across the Li dataset.Detected foregrounds are reported as closer to ground truth, consistent with their larger F-measures.
  • 4.1 On videos without camera jitter: Frame-specific MoG noise fitting separates multiple foreground variation layers, with the largest-variance component generally representing the foreground object.This flexible modeling is offered as an explanation for improved foreground extraction.
  • 4.1 On videos without camera jitter: OMoGMF gradually adapts to sudden illumination changes and recovers a proper background when 85% of entries are missing.The missing-entry experiment uses sub-sampling with random subspace initialization.
  • 4.1 On videos without camera jitter: More than 200 FPS is achieved on average with 1% sub-sampling while retaining high F-measure, and OMoGMF's F-measure decreases only slightly from full sampling.Across the evaluated videos, GRASTA shows an evident F-measure drop in 5 of 9 videos, whereas OMoGMF remains accurate at 1% sampling.
  • 4.2 On videos with camera jitter: t-OMoGMF properly aligns video frames and separates foreground from background in jitter videos, with higher average F-measure than competing methods.This result is reported on real videos with camera jitter.

5 CONCLUSION AND DISCUSSION

The proposed online subspace learning method targets practical background subtraction by combining real-time processing with adaptation to foreground, background, and affine variations. Experiments also show its extension to image alignment and video stabilization, while future work includes faster streaming applications.

  • Extension to other tasks: The method can be extended to image alignment and video stabilization, as illustrated by preliminary tests and Fig. 12.
  • Extension to other tasks: On 100 misaligned dummy faces and a 140-frame Gore sequence, it-OMoGMF aligns images or frames and separates physically meaningful noise layers.Largest-variance noise components represent occlusions in the dummy set and expression variations in the Gore video.
  • Concluding remarks: The method reaches more than 25 FPS while adapting to real-time foreground and background variations.It learns frame-specific foreground distributions and background subspaces using previously learned knowledge.
  • Concluding remarks: Affine transformation variables let the method adapt to rotations, transformations, scalings, and distortions in practical videos.
  • Concluding remarks: Experiments on synthetic and real-world videos substantiate the method against state-of-the-art online and offline approaches.
  • Future investigations: Future work includes embedding sub-sampling into t-OMoGMF and it-OMoGMF and applying them to larger real-time video streams.
Loading 1705.10000v1…