Source-linked AI summary
It's Moving! A Probabilistic Model for Causal Motion Segmentation in Moving Camera Videos
Pia Bideau, Erik Learned-Miller
TL;DR
Motion segmentation still falls short of human ability to detect moving objects under camouflage, complex backgrounds, and observer motion. The paper derives a likelihood that combines optical-flow angle and magnitude information, then uses it in a segmentation algorithm that achieves state-of-the-art performance across three datasets, including a new camouflaged-animal benchmark.
Problem
Motion-segmentation methods have not matched humans' rapid detection of moving objects in camouflage, complex backgrounds, and moving-camera settings.
Method
The paper derives a probabilistic angle likelihood conditioned on optical-flow magnitude and 3D motion direction, combining this model with motion-segmentation initialization innovations.
Results
The method achieves state-of-the-art performance on three diverse datasets, while removing flow-magnitude conditioning causes performance drops of 7%, 5%, and 22% across the datasets.
Takeaways & Limitations
The results support using flow magnitude to assess the reliability of optical-flow direction and introduce a challenging benchmark for camouflaged-animal motion segmentation.
Takeaways & Limitations
The model assumes observed translational flow is a noisy observation of an unobserved translational motion field, and the ground-truth task uses static-background versus moving-object labels.
Abstract
from arXiv · showhide
The human ability to detect and segment moving objects works in the presence of multiple objects, complex background geometry, motion of the observer, and even camouflage. In addition to all of this, the ability to detect motion is nearly instantaneous. While there has been much recent progress in motion segmentation, it still appears we are far from human capabilities. In this work, we derive from first principles a new likelihood function for assessing the probability of an optical flow vector given the 3D motion direction of an object. This likelihood uses a novel combination of the angle and magnitude of the optical flow to maximize the information about the true motions of objects. Using this new likelihood and several innovations in initialization, we develop a motion segmentation algorithm that beats current state-of-the-art methods by a large margin. We compare to five state-of-the-art methods on two established benchmarks, and a third new data set of camouflaged animals, which we introduce to push motion segmentation to the next level.
1 Introduction
The paper develops a probabilistic motion-segmentation model aimed at matching human motion detection in camouflage, complex backgrounds, and moving-camera videos. Its angle likelihood combines optical-flow angle and magnitude information, and the method achieves state-of-the-art performance across three diverse datasets.
- Motivation: Human observers rapidly detect the moving camouflaged insect despite camera motion, rarity, and complex background geometry.The insect is difficult to identify in a still image but becomes immediately visible when it moves.
- Approach: The paper derives a probabilistic model that captures information about an object's 3D motion in each optical-flow vector.Its conditional angle likelihood is derived from perspective projection and a noisy optical-flow observation model.
- Approach: The angle likelihood addresses ambiguity in inferring 3D motion from optical-flow vectors by weighting image-motion evidence according to flow reliability.Small flow vectors are less informative because modest errors can substantially change their apparent direction.
- Evaluation: The method achieves state-of-the-art performance on the Berkeley Motion Segmentation, Complex Background, and Camouflaged Animal datasets.The Camouflaged Animal Data Set contains nine moving-camera videos featuring animals difficult to see in a single frame.
- Evaluation: The paper introduces the Camouflaged Animal Data Set as a challenging benchmark for motion segmentation in videos containing camouflaged moving animals.The benchmark is designed around objects detectable from motion across frames rather than from a single frame.
2 Related Work
Prior motion-segmentation methods use trajectories, optical-flow models, occlusion cues, or camera-motion estimation, but the paper distinguishes its causal approach from these alternatives. It also emphasizes handling complex background geometry and camera rotation.
- Trajectory-based methods: Trajectory-based methods track points or regions across multiple frames and cluster their motion trajectories into coherent moving objects.Examples use low-dimensional subspaces or pairwise trajectory metrics for segmentation.
- Trajectory-based methods: Trajectory-based methods are non-causal because they require future frames, whereas this method uses flow between two frames and information propagated from previous frames.The paper reports outperforming trajectory-based methods despite this causal design.
- Optical-flow methods: Optical-flow approaches group pixels using motion models, including probabilistic mixtures of rigid components and projective-geometry-based formulations.These methods may assign different motion models to objects according to model-fitting criteria.
- Geometric approaches: The paper does not require scene geometry to be approximated by homographies, enabling motion segmentation in videos with highly complex background geometry.A related canonical-angle-field method handles translational motion but not camera rotation.
- Occlusion-based methods: Occlusion-based methods use depth ordering, independent object motion, or temporary consistency priors to partition videos into layers.These approaches provide another route to disambiguating motion segmentation solutions.
- Camera-motion estimation: Camera-motion estimation from optical flow is closely related to motion segmentation, with several such approaches discussed in supplementary material.The paper treats camera translation and rotation estimation as neighboring work rather than its primary method.
3 Methods
The method removes camera rotation, models motion using a flow-angle likelihood conditioned on flow magnitude, and assigns pixels to motion components through probabilistic segmentation. It combines motion-model priors, camera-motion estimation, posterior propagation, and constrained initialization for moving-camera scenes.
- Camera-motion compensation: For camera translation without rotation, each background flow angle is determined by camera translation, image location, and focal length, not scene depth.The angle is expressed as tθ = arctan(W · y − V · f, W · x − U · f).
- Camera-motion compensation: The optical flow is decomposed by estimating camera rotation and subtracting its rotational component to obtain translational flow.This simplifies assigning flow vectors to motion models because the translational flow angle is independent of unknown scene depth after rotational flow is removed.
- Probabilistic segmentation: Segmentation combines motion-component priors, estimated 3D motion directions, conditioned angle likelihoods, and a prior for a new component to compute posteriors.The posterior is used directly for segmentation, while priors are propagated by flow warping, spatial smoothing, renormalization, and allocation of probability to a new motion component.
- Initialization: Constrained RANSAC initializes background motion by selecting superpixels while forcing three of ten patches to come from image corners.The procedure is designed to robustly estimate camera motion using background-prior-weighted pixels in scenes containing multiple motions.
- Conditional flow likelihood: The angle likelihood p(tθ|Mj, tr) uses flow angle and magnitude because angle carries motion-direction information while magnitude controls angle reliability.Small flow vectors produce less informative directions, whereas larger vectors provide more reliable evidence; translational magnitude alone is not informative for selecting motion under the stated assumptions.
- Conditional flow likelihood: The model derives a conditional angle likelihood from noisy translational flow, using a von Mises distribution centered on the predicted angle.Its concentration increases with the underlying flow magnitude, giving broad angle distributions for small motions and narrow distributions for large motions.
4 Experiments
The experiments define motion segmentation as labeling freely moving objects while excluding static or tethered objects, then evaluate the method against state-of-the-art approaches using balanced metrics across diverse datasets. Dataset inconsistencies required excluding ten BMS-26 videos with incompatible ground truth.
- Evaluation definition: Motion segmentation labels every pixel as static background or moving object, including an entire object when only part moves.
- Evaluation definition: The definition requires segmenting all freely moving objects, excluding tethered and stationary objects.
- Datasets: Ten BMS-26 videos were excluded because their ground truth used inconsistent segmentations, including labeling a foreground wall as moving.
- Metrics and comparison: The evaluation compares the model with five state-of-the-art methods using Matthew’s correlation coefficient and F-measure.
- Results: The method outperforms all other methods by a large margin on all three datasets under both comparison measures.
5 Analysis and Conclusions
Analysis shows that conditioning the angle likelihood on flow magnitude and using constrained RANSAC are important, while competing trajectory methods struggle with foreground depth variation. The authors attribute performance partly to direct use of perspective projection equations.
- Likelihood analysis: Using a constant von Mises concentration instead of flow-magnitude conditioning reduces performance by 7%, 5%, and 22% across the three datasets.
- Initialization analysis: Constrained RANSAC provides consistent gains, with effects ranging from small in some videos to large in others.
- Comparison: Trajectory-based methods often segment rigid foreground regions incorrectly and have difficulty with scenes containing highly variable depth.
- Initialization analysis: Without RANSAC, background translation can fit the car rather than the background, producing an incorrect motion estimate.
- Interpretation: The authors suggest performance may benefit from directly using perspective projection equations rather than projective-geometry approximations.
Supplementary Material
The supplementary material defines motion segmentation criteria, documents benchmark-ground-truth filtering and a new camouflaged-animal dataset, and details the modified Bruss–Horn initialization and error function.
- Motion segmentation definition: Motion segmentation assigns each pixel to static background or moving objects, segments an entire partially moving object, and excludes stationary or previously moving objects.Previously moving objects are treated as tracking rather than motion segmentation.
- Benchmark curation: The authors convert supplied ground truth to binary masks and exclude videos that violate all four motion-segmentation criteria.They report that mislabeled ground truth can substantially alter evaluation, including for marple2.
- Camouflaged Animals Data Set: The new dataset contains nine short YouTube-derived video sequences with accompanying hand-labeled ground truth.Sequences are converted to PNG image sequences for algorithm evaluation.
- Perspective projection: Under translation-only perspective projection, background flow direction depends on image position, camera-motion direction, and focal length, but not scene depth.The supplementary material gives the direction expression and relates it to the Bruss–Horn compatibility calculation.
- Modified Bruss and Horn Error: The Bruss–Horn error can assign similarly small errors to optical-flow vectors pointing with or against the preferred motion direction.This symmetry is problematic when independently moving objects have flow opposite to the background.
- Modified Bruss and Horn Error: The modified error uses the projection-based penalty for flows within 90 degrees of the expected direction and the full vector magnitude for opposing flows.This modification penalizes motion in the opposite direction to the expected flow and supports camera-motion initialization.