Source-linked AI summary
Image Segmentation in Video Sequences: A Probabilistic Approach
Nir Friedman, Stuart Russell
TL;DR
Background subtraction struggles to distinguish shadows from moving objects and can absorb slow-moving vehicles into the background. The paper uses an incremental EM-learned mixture-of-Gaussians model to classify each pixel by appearance; it reports significantly improved moving-object detection, while noting robustness and initialization limits.
Problem
Background subtraction can mistake shadows for moving objects and can have its background image corrupted by slow-moving or stationary vehicles.
Source: §1 · p. 1Method
The paper classifies each pixel using an appearance model conditioned on its class, with mixture-of-Gaussians models learned by an incremental version of EM.
Source: §1 · p. 1Results
The authors report that probabilistic pixel classification with unsupervised learning can significantly improve moving-object detection in video sequences.
Source: §6 · p. 7Takeaways & Limitations
The paper presents EM modeling of a single pixel's appearance over time as a natural probabilistic generalization of a classical deterministic method.
Source: §1 · p. 2Better initialization and model labeling are needed for robustness, and the heuristic approach may not work well in extreme lighting conditions.
Source: §6 · p. 7
Abstract
from arXiv · showhide
"Background subtraction" is an old technique for finding moving objects in a video sequence for example, cars driving on a freeway. The idea is that subtracting the current image from a timeaveraged background image will leave only nonstationary objects. It is, however, a crude approximation to the task of classifying each pixel of the current image; it fails with slow-moving objects and does not distinguish shadows from moving objects. The basic idea of this paper is that we can classify each pixel using a model of how that pixel looks when it is part of different classes. We learn a mixture-of-Gaussians classification model for each pixel using an unsupervised technique- an efficient, incremental version of EM. Unlike the standard image-averaging approach, this automatically updates the mixture component for each class according to likelihood of membership; hence slow-moving objects are handled perfectly. Our approach also identifies and eliminates shadows much more effectively than other techniques such as thresholding. Application of this method as part of the Roadwatch traffic surveillance project is expected to result in significant improvements in vehicle identification and tracking.
1 Introduction
Background subtraction identifies moving pixels by comparing them with a stationary background, but shadows and slow or stationary traffic cause problems. The paper instead classifies each pixel probabilistically by its appearance across classes, using an incremental mixture-of-Gaussians model learned with EM.
1 Introduction
- Background subtraction can mistake moving shadows for parts of vehicles and absorb slow or stationary vehicles into the background model.Earlier Roadwatch work had used background subtraction to locate and track freeway vehicles [Koller et at., 1994].Source: §1 · p. 1
- The paper models each pixel's appearance probabilistically across moving-object, shadow, and background classes, using classification to update models without mixing slow-moving vehicle pixels into the background.A pixel's shadow appearance is described as relatively constant because it is independent of the object casting the shadow.Source: §1 · p. 1
- Pixel appearance conditioned on its hidden type is modeled with a mixture of Gaussians learned by EM, using an incremental version intended to provide real-time performance.The authors report success coping with slow-moving objects and shadows.Source: §1 · p. 1
- Unlike prior EM applications that group similar pixels or motions, the paper uses EM to model one pixel's appearance over time, a probabilistic generalization of a classical deterministic method.The authors describe this use as novel to their knowledge.Source: §1 · pp. 1–2
2 Background subtraction
The section describes background subtraction as maintaining an average background and flagging pixels that differ, with adaptation for changing conditions. It also details failures involving shadows, slow-moving traffic, and pixel-level ambiguity.
2 Background subtraction
- The background is a long-term average, and exponential forgetting weights newer images more while requiring no additional storage.The forgetting time constant is 1/a, and the update is equivalent to using a Kalman filter to track the background image.Source: §2 · p. 2
- Moving pixels can be identified by thresholding their Mahalanobis distance from the background, whose variance can also be computed incrementally.Source: §2 · p. 2
- Under favourable conditions, vehicles are outlined reasonably well, though image noise leaves stray pixels marked as moving.Standard methods can group vehicle pixels and compute and track a smoothed convex hull.Source: §2 · p. 2
- Shadows differ from the background and are identified as moving objects; simple lightening or thresholding can fail because road shadows and markings confound intensity.A shadow on a white road line can be brighter than sunlight on tarmac.Source: §2 · p. 2
- Vehicle parts with approximately the same intensity as the background present a pixel-level detection limitation, and road markings may also be labeled moving.Source: §2 · p. 2
- When traversal time is non-negligible relative to 1/a, slow or stationary objects corrupt the background image and detection can fail completely.Updating only pixels classified as non-moving can still become unstable when erroneous classifications perturb the background.Source: §2 · p. 2
3 Pixel models
Each pixel is modeled as a mixture of road, shadow, and vehicle appearance distributions, with parameters and weights specific to that pixel. The model fits intensity or RGB values and classifies a pixel by its most probable label.
3 Pixel models
- Each pixel’s appearance is modeled as a weighted mixture of road, shadow, and vehicle distributions, with separate parameters and weights for each pixel.The road distribution describes that pixel’s appearance when showing unshadowed road at its physical location.Source: §3 · p. 2
- For one pixel, road and shadow values form fairly well-defined peaks, while vehicle values are widely scattered; fitted three-component Gaussian mixtures approximate the empirical data reasonably well, but imperfectly.The supplied example uses intensity and RGB data for pixel (160,170).Source: §3 · pp. 2–3
- The model uses scalar means and variances for intensity values, and three-dimensional means and 3 × 3 covariance matrices for RGB values.The derivations are identical for the intensity and RGB settings.Source: §3 · p. 3
- A pixel is classified as the label with the highest posterior probability given its observed value.Source: §3 · p. 3
4 Algorithms for learning pixel models
The section formulates learning pixel-specific mixture parameters from image sequences when pixel labels are hidden. It introduces standard EM and sufficient statistics for labeled data, then describes an incremental version suited to real-time implementation.
4 Algorithms for learning pixel models
- An efficient, incremental version of EM is described as suitable for real-time implementation.Source: §4 · p. 3
- Learning seeks mixture-distribution parameters and relative weights that maximize the likelihood of the observed pixel data across an image sequence.The pixel class labels are hidden, so the likelihood is defined with respect to observable pixel values.Source: §4.1 · pp. 3–4
- If pixel labels were annotated, mixture estimation could use sufficient statistics including each class’s pixel count and summed pixel values.Source: §4.1 · p. 4
1 T N,z,-J.LtJ.I.l
The section develops an incremental EM procedure for per-pixel mixture models that avoids storing and reprocessing the full video sequence. It then describes component labeling and reports a cleaner vehicle mask than background subtraction, while noting limits of the current update and labeling schemes.
Incremental EM
- Incremental EM adds each new instance’s expected sufficient statistics without storing or reprocessing earlier data.Standard EM would require storing pixel values from all observed images, which the authors consider impractical for real-time use.Source: §4.2 · p. 5
- The incremental procedure is not guaranteed to increase likelihood at every step, though likelihood increases on average and the process converges to a local maximum with high probability.Source: §4.2 · p. 5
- Because old instances’ contributions are never removed, their sufficient statistics can become outdated and cause long-run problems; exponential forgetting is being explored.Source: §4.2 · p. 5
Sample results
- Figure 4’s vehicle mask is much cleaner than background subtraction, alongside shadow identification and replacement with road values.The authors had not yet completed experiments to establish that this improvement carried over to vehicle detection and tracking.Source: Figure 4 · p. 6, §5 · p. 6
Video processing
- For each new frame, the system updates each pixel’s mixture, heuristically labels components, then classifies pixels; the darkest component is shadow and the highest-variance remaining component is vehicle.The other remaining component is labeled road; this heuristic addresses cases such as white road markings that reverse expected brightness ordering.Source: §5 · pp. 5–6
- The incremental EM variant is described as capable of real-time operation on suitable hardware, with 12 Texas Instruments C40 DSP chips in parallel as the target platform.Source: §5 · p. 6
6 Conclusions and further work
The authors conclude that probabilistic pixel classification with unsupervised learning can significantly improve moving-object detection. They identify robustness of initialization and labeling, along with the limits of purely pixel-based techniques, as further-work concerns.
Conclusions
- The authors report that probabilistic pixel classification with unsupervised learning can significantly improve moving-object detection in video sequences.Source: §6 · pp. 6–7
Further work
- Robustness remains limited by heuristic initialization and labeling, which may fail under extreme lighting; MAP or penalty-based EM is proposed for more flexible priors.The authors expect this change to require little or no additional computational expense and potentially improve robustness.Source: §6 · p. 7
- Purely pixel-based methods may soon reach their limits; temporal and spatial contiguity are currently ignored or handled ad hoc, while spatial Markov networks impose high computational burden.The authors suggest these richer models could support moving cameras and interface with vehicle-behavior belief networks.Source: §6 · p. 7