PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Motion Patterns in Videos

Pavel Tokmakov, Karteek Alahari, Cordelia Schmid

arXiv:1612.07217v2cs.CV

TL;DR

Determining independent object motion despite camera motion remains difficult. The paper learns motion patterns with a synthetic-data-trained encoder-decoder network and refines its labels using objectness and CRF cues. It reports state-of-the-art benchmark performance, including a 5.6% IoU advantage on DAVIS.

  • Problem

    Determining whether an object is moving independently of camera motion remains challenging, motivating moving-object segmentation.

  • Method

    MP-Net learns per-pixel independent-motion labels from optical flow using a fully convolutional encoder-decoder trained on synthetic sequences, followed by objectness and CRF refinement.

  • Results

    The method achieves state-of-the-art results on real-world motion-segmentation benchmarks and exceeds previous video-level methods by 5.6% in DAVIS IoU.

  • Takeaways & Limitations

    Learned local motion patterns can support moving-object segmentation while distinguishing object motion from camera motion.

  • Takeaways & Limitations

    Performance depends strongly on optical-flow quality, and limited two-frame temporal information reduces temporal stability relative to video-level inference.

Abstract

from arXiv · show

The problem of determining whether an object is in motion, irrespective of camera motion, is far from being solved. We address this challenging task by learning motion patterns in videos. The core of our approach is a fully convolutional network, which is learned entirely from synthetic video sequences, and their ground-truth optical flow and motion segmentation. This encoder-decoder style architecture first learns a coarse representation of the optical flow field features, and then refines it iteratively to produce motion labels at the original high-resolution. We further improve this labeling with an objectness map and a conditional random field, to account for errors in optical flow, and also to focus on moving "things" rather than "stuff". The output label of each pixel denotes whether it has undergone independent motion, i.e., irrespective of camera motion. We demonstrate the benefits of this learning framework on the moving object segmentation task, where the goal is to segment all objects in motion. Our approach outperforms the top method on the recently released DAVIS benchmark dataset, comprising real-world sequences, by 5.6%. We also evaluate on the Berkeley motion segmentation database, achieving state-of-the-art results.

1. Introduction

The paper addresses the unresolved problem of identifying independently moving objects despite camera motion. It introduces a CNN-based framework that learns motion patterns and achieves strong moving-object segmentation results on real-world benchmarks.

  • Identifying whether an object moves independently of camera motion remains challenging despite extensive prior work on motion estimation and video segmentation.
  • The task is formulated as moving object segmentation using optical flow to distinguish real object motion from camera motion.
  • MP-Net takes optical flow as input and produces per-pixel moving-object scores through a fully convolutional encoder-decoder architecture trained on synthetic data.
  • Objectness cues and a conditional random field refine motion labels to handle flow errors and focus on moving things rather than stuff.
  • 5.6% improvement in IoU over previous video-level methods is achieved on DAVIS, while the method also obtains state-of-the-art results on Berkeley motion segmentation datasets.

2. Related Work

Prior motion and video-segmentation methods rely on geometric, probabilistic, heuristic, or manually supervised approaches. The paper instead positions MP-Net as a learned CNN framework for pixel-level motion-pattern recognition.

  • Motion estimation: Earlier motion-estimation methods use geometry, optical-flow changes, appearance cues, or iterative motion models to estimate moving regions.
  • Motion estimation: Optical-flow orientation models can fail under camera rotation or when object and camera motions are consistent, while some alternatives depend on heuristic initialization.
  • Motion estimation: Scene-flow methods estimate 3D motion using additional information such as stereo disparity or 3D scene models, rather than a CNN-based learning approach.
  • Video object segmentation: Video-object segmentation methods may propagate manual labels, generate object-agnostic over-segments, or classify foreground and background using appearance and auxiliary cues.
  • Related CNN architectures: MP-Net follows encoder-decoder CNN designs used for pixel-level prediction, learning multiscale flow representations and refining them to produce per-pixel labels.

3. Learning Motion Patterns

MP-Net learns motion patterns from optical flow using a multiscale encoder-decoder network trained on synthetic sequences. Its receptive fields provide scene context while decoding restores full-resolution motion labels.

  • MP-Net processes optical flow from consecutive frames and independently predicts per-pixel motion labels for each frame pair.
  • Large receptive fields provide object-and-background context needed to distinguish object motion from camera motion, while full-resolution outputs preserve fine moving structures.
  • The encoder uses convolutions and max-pooling to learn coarse flow representations, while the decoder upsamples and convolves features toward the original resolution.
  • Synthetic training is used because large real-world datasets with ground-truth optical flow and moving-object annotations are unavailable.
  • FlyingThings3D supplies 2250 synthetic sequences with ground-truth flow, augmented with moving-object labels derived from disparity, camera parameters, and scene correspondences.
  • The learned model recognizes consistent local motion patterns in flow fields, enabling more robust motion estimation than techniques that cannot detect such patterns.

4. Detecting Motion Patterns

The approach detects independent object motion in real-world videos by combining MP-Net predictions with objectness cues and segmentation refinement. These components address motion from stuff and inaccuracies in computed optical flow.

  • 4. Detecting Motion Patterns: MP-Net identifies fine details in subtly moving objects when supplied with precise optical flow on synthetic test sequences.Examples include thin structures such as a guitar neck.
  • 4. Detecting Motion Patterns: Real-world videos introduce independently moving stuff, such as water, that is not annotated in standard moving-object benchmarks.The network can estimate this motion, but benchmark evaluation targets objects rather than stuff.
  • 4. Detecting Motion Patterns: Object proposals are aggregated through pixel-level voting to compute an objectness score for each pixel.The score is the number of proposals containing the pixel, normalized by the total number of proposals.
  • 4. Detecting Motion Patterns: Figure 5 presents the processing sequence from video frame and LDOF flow through MP-Net, objectness, initial segmentation, and CRF refinement.Each row displays the corresponding intermediate output on DAVIS examples.
  • 4. Detecting Motion Patterns: The objectness map is combined with MP-Net motion predictions to suppress moving stuff and retain moving objects.In the kiteboarder example, the absence of proposals on water removes the water motion from the segmentation.
  • 4. Detecting Motion Patterns: Computed optical flow can produce imprecise motion segments, especially in low-texture regions, motivating further refinement.The real-world flow is less accurate than the ground-truth flow used on synthetic data.

5. Datasets

The evaluation uses synthetic FlyingThings3D data for training and testing, alongside real-world DAVIS and Berkeley motion-segmentation datasets for evaluation. FlyingThings3D provides automatically derived independent-motion labels from geometric and flow information.

  • 5. Datasets: FlyingThings3D contains 2700 synthetic videos with randomized object trajectories, scenes, and complex camera motion.Each video contains 10 stereo frames, with 2250 training and 450 test videos.
  • 5. Datasets: FlyingThings3D provides ground-truth optical flow, disparity, camera parameters, and object instance masks for every video.Independent-motion labels are derived by comparing corresponding 3D scene points across frames.
  • 5. Datasets: DAVIS is used exclusively for evaluation and contains 50 full HD videos with diverse object and camera motion.The dataset includes challenging cases involving occlusion, motion blur, and appearance changes.
  • 5. Datasets: Table 1 compares input modalities on FlyingThings3D and DAVIS using mean intersection over union scores.Ground-truth flow is used for FT3D evaluation, while LDOF flow is used for DAVIS.

6. Experiments and Results

Experiments show that MP-Net transfers learned motion patterns from synthetic data to real videos, with performance shaped by flow representation, input quality, objectness, and refinement. The approach achieves strong results on DAVIS, BMS-26, and FBMS, while remaining sensitive to optical-flow errors and limited temporal information.

  • Influence of input modalities: GT angle field improves DAVIS IoU by 2.3%, despite decreasing FT3D IoU by 1.4%; it is used in subsequent experiments.Flow orientations are less dependent on camera depth under translational motion, while strong camera rotations remain challenging.
  • Influence of input modalities: Computed-flow training reduces IoU by 9.9% on FT3D and 8.5% on DAVIS, highlighting the importance of high-quality training flow.Adding RGB helps FT3D by 1.7% but hurts DAVIS by 7%, indicating inconsistent appearance features across domains.
  • Evaluation on real videos: Objectness and CRF refinement improve MP-Net results on DAVIS, with CRF particularly helping segmentation around boundaries when using LDOF flow.The combined method achieves IoU 69.7 with LDOF and 68.0 with EpicFlow.
  • Comparison to the state of the art: 5.6% higher IoU and 7% higher F-measure than NLC make the frame-level MP-Net + Objectness + CRF the best unsupervised DAVIS method evaluated.The comparison uses mean-J and mean-F, while temporal stability is not the strongest measure because MP-Net uses only two-frame flow.
  • Comparison to the state of the art: 77.5% F-measure on FBMS exceeds FST, CVOS, and CUT, while MP+Obj + FST is significantly better than most prior BMS-26 methods and on par with PCM.On BMS-26, incorporating MP-Net motion prediction into FST improves the result by 14%.

7. Conclusion

The paper presents a novel approach for learning motion patterns in videos and demonstrates its strength on moving object segmentation. It also identifies future directions for extending the approach.

  • The approach is demonstrated on moving object segmentation, where it outperforms many complex methods relying on engineered features.
  • Future work includes end-to-end video semantic segmentation, memory modules for video object segmentation, and user annotations for ambiguous cases.
Loading 1612.07217v2…