PaperlayerView exact revision on arXiv

Source-linked AI summary

Coherent Motion Segmentation in Moving Camera Videos using Optical Flow Orientations

Manjunath Narayana, Allen Hanson, Erik Learned-Miller

arXiv:1511.01619v1cs.CV

TL;DR

Depth-dependent optical flow can split objects with the same real-world motion into separate segments. The paper clusters flow orientations with a probabilistic model that estimates the number of motions, reporting results across diverse videos and unified segmentation of background objects at different depths.

  • Problem

    Objects with the same real-world motion can have different optical flows at different depths, leading to undesirable over-segmentation of background objects.

    Source: §1 · p. 2

  • Method

    The system clusters optical-flow orientations, which are depth-independent under translational camera motion, using predicted orientation fields and a mixture model that estimates the number of motions.

    Source: §1 · p. 2

  • Results

    Results cover all 26 Hopkins videos, SegTrack, and complex-background videos; background objects at different depths can form one segment.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    Using optical-flow orientations avoids depth-dependent over-segmentation and identifies background objects at different depths as one segment.

    Source: §1 · p. 2, §6 · p. 7

    The model represents translation, so camera rotation can cause errors; magnitude information may help when a tracked foreground object disappears from FOF observations.

    Source: §6 · p. 7

Abstract

from arXiv · show

In moving camera videos, motion segmentation is commonly performed using the image plane motion of pixels, or optical flow. However, objects that are at different depths from the camera can exhibit different optical flows even if they share the same real-world motion. This can cause a depth-dependent segmentation of the scene. Our goal is to develop a segmentation algorithm that clusters pixels that have similar real-world motion irrespective of their depth in the scene. Our solution uses optical flow orientations instead of the complete vectors and exploits the well-known property that under camera translation, optical flow orientations are independent of object depth. We introduce a probabilistic model that automatically estimates the number of observed independent motions and results in a labeling that is consistent with real-world motion in the scene. The result of our system is that static objects are correctly identified as one segment, even if they are at different depths. Color features and information from previous frames in the video sequence are used to correct occasional errors due to the orientation-based segmentation. We present results on more than thirty videos from different benchmarks. The system is particularly robust on complex background scenes containing objects at significantly different depths

1. Introduction

The introduction identifies depth-dependent optical flow as a source of over-segmentation and proposes using flow orientations to cluster pixels by real-world motion. A probabilistic model estimates the motions present, and the system reports results across diverse videos, including complex backgrounds.

1. Introduction

  • Objects sharing real-world motion can have different optical flows at different depths, causing background over-segmentation that hinders independently moving-object detection.Source: §1 · pp. 1–2
  • Under translational camera motion, flow orientations do not depend on depth, unlike flow vectors and magnitudes, making them more reliable cues for real-world motion.Source: Figure 1 · p. 1, §1 · p. 2
  • The method matches observed pixel orientations to predicted orientation fields, uses a mixture model to assign motions and pixels, and favors fewer 3-D motions.Source: §1 · p. 2
  • A non-parametric Dirichlet process-like mixture model determines the number of motion components automatically, avoiding a preset cluster count.Source: §1 · p. 2
  • Background objects at different depths can form one segment while coherently moving foreground regions are identified.Source: §1 · p. 2
  • Prior approaches use planar motion compensation or trajectory clustering, with cited methods variously assuming orthographic cameras, offline processing, or affine post-merging.The post-merging step may not work for complex backgrounds, and Gaussian mixtures require choosing the number of components.Source: §1 · p. 2
  • The system uses two frames rather than multi-frame trajectories and operates online without special first-frame initialization; color appearance helps address orientation errors.It can detect objects that are initially at rest and begin moving later.Source: §1 · pp. 2–3
  • Results cover all 26 Hopkins videos, SegTrack, and videos with complex backgrounds, despite the assumption of translational camera motion.The authors describe the results as spanning different scenarios.Source: §1 · p. 3

2. Segmentation using optical flow orientations

The method clusters pixels by optical-flow orientation, which is independent of depth under translational motion, using a probabilistic model that adds components as needed. It supplements the orientation model with parameter refinement and special handling for near-zero motion.

2. Segmentation using optical flow orientations

  • Under translational motion, flow orientations are independent of depth, and the model uses a library of predicted orientation fields to explain observed orientations.The library samples 46 translation values; each pixel’s observed orientation is modeled as Gaussian around the predicted orientation for its assigned motion.Source: §2 · pp. 3–4
  • Starting with one component, Gibbs sampling adds auxiliary motion components as needed to explain the observed orientations.The model samples motion parameters from a finite set and uses the maximum a posteriori pixel assignments.Source: §2 · p. 4

2.1. Choosing α

  • The system selects α by comparing segmentations from α values .0001, .01, and 10, choosing the result with greatest agreement with the candidates’ pixelwise votes.The votes are formed from each candidate’s largest segment and its complement.Source: §2.1 · p. 4

2.2. Gradient descent for largest component

  • After n iterations, gradient descent refines the largest segment’s motion parameters by minimizing average L1 distance to its observed orientations.The refined motion is added to the library to better explain background orientation patterns not covered by the initial motion samples.Source: §2.2 · p. 4

2.3. Handling pixels with near-zero motion

  • Pixels with flow magnitudes below threshold Tf in both directions are marked as zero-motion and excluded from Gibbs sampling.Tf is typically 0.5.Source: §2.3 · p. 4

3. Segmentation comparisons

Compared with spectral clustering of tracked keypoints, FOF segmentation produces dense labels and identifies backgrounds as one segment in examples with large depth disparity.

3. Segmentation comparisons

  • FOF segmentation identifies backgrounds as one segment in large-depth-disparity examples where spectral clustering with a merge step over-segments them.FOF uses two consecutive frames and no post-processing merge, while the spectral-clustering implementation uses three frames.Source: §3 · p. 5
  • FOF segmentation successfully labels the background in most examples and gives a dense image labeling, unlike uncertain spectral-clustering keypoints.The spectral clustering implementation marks some background keypoints uncertain despite its post-processing merge step.Source: §3 · p. 5

4. Appearance modeling

The appearance model combines motion-compensated color histories with FOF segment labels and prior background probabilities to classify pixels. It mixes in a uniform color distribution when background history is unreliable and combines likelihoods with a motion-compensated prior.

4. Appearance modeling

  • The appearance model maintains motion-compensated pixel color histories and uses neighborhood samples with KDE to estimate background and foreground color likelihoods.The FOF mixture model supplies component assignments and probabilities; its largest component is treated as background.Source: §4 · p. 5

4.1. Mixing a uniform distribution component

  • When all previous T frames occlude the background, a uniform color distribution is mixed into the color likelihood, with more weight when background history is unreliable.The uniform component is weighted less when historical pixels are confidently background.Source: §4.1 · pp. 5–6

4.2. Posterior computation

  • The current background posterior combines color likelihoods, FOF segmentation-label likelihoods, and a motion-compensated, smoothed background prior from the previous frame.The paper states that color likelihoods and prior information help recover from FOF segmentation errors.Source: §4.2 · p. 6

5. Results

The system achieved reasonable segmentation accuracy across all three datasets despite dataset-specific challenges, and color and prior information generally improved FOF segmentation. Results cover two existing benchmarks and a new set of videos with complex backgrounds.

5. Results

  • Evaluation covered 26 Hopkins moving-camera videos, SegTrack, and a new complex-background set with forest trees and large occluders.Hopkins ground truth was available for a few frames sampled throughout each video.Source: §5 · p. 6
  • In Hopkins, challenging cases included foreground flow matching camera flow for long periods, foregrounds covering most pixels, or objects labeled moving despite initially stationary frames.The cited videos include cars4, marple6, marple9, and marple11.Source: §5 · pp. 6–7
  • Three SegTrack videos contain multiple moving objects, but tracking-oriented ground truth labels only one primary object as foreground, making the system appear less accurate.The authors retained the original ground-truth labels when reporting results.Source: §5 · p. 7
  • Despite dataset-specific challenges, segmentation remained reasonably accurate across all three datasets, and color and prior information generally improved FOF accuracy.Rotation reduced FOF accuracy in several handheld-camera videos.Source: §5 · pp. 6–7
  • Kwak et al. reported F-measures of .88, .94, and .87 on three Hopkins videos using special first-frame initialization; this method was less accurate on the first two.When its bootstrapped system misses the object in the first frame, it pays a heavy penalty.Source: §5 · p. 7

6. Discussion

The system uses optical-flow orientations to avoid depth-dependent over-segmentation and automatically determine the number of foreground motions. Its translation-only model is prone to errors when the camera rotates, while magnitude information may help when a tracked foreground disappears from FOF observations.

6. Discussion

  • Optical-flow orientations avoid depth-dependent scene over-segmentation, and the system automatically determines the number of foreground motions.The discussion describes results across a wide range of videos, including some with complex backgrounds.Source: §6 · p. 7
  • The translation-only model is prone to error when the camera rotates; explicitly modeling rotation could help address those errors.Source: §6 · p. 7
  • Incorporating magnitude information may improve the model, especially when a tracked foreground object suddenly disappears from FOF observations.Source: §6 · p. 7
Loading 1511.01619v1…