Source-linked AI summary
SegFlow: Joint Learning for Video Object Segmentation and Optical Flow
Jingchun Cheng, Yi-Hsuan Tsai, Shengjin Wang, Ming-Hsuan Yang
TL;DR
Video object segmentation and optical flow are closely related, yet flow can be inaccurate and paired annotations for both tasks may be unavailable. SegFlow jointly learns both tasks through bidirectional feature propagation, iterative offline training, and online fine-tuning, with favorable results on both task benchmarks.
Problem
Video object segmentation needs motion information for temporal consistency, but optical flow is often inaccurate, while datasets containing both segmentation and flow ground truths may be unavailable.
Method
SegFlow is an end-to-end CNN with segmentation and optical-flow branches that exchange features bidirectionally and are trained iteratively offline before object-specific online fine-tuning.
Results
The authors report favorable performance on both video object segmentation and optical flow tasks, with optical flow improving segmentation and segmentation improving flow.
Takeaways & Limitations
Segmentation and optical flow are complementary tasks that can help each other through bidirectional feature propagation in a jointly trained framework.
Takeaways & Limitations
The FlowNetS data layer is specifically designed for FlowNetS, so it cannot be directly applied to SegFlow.
Abstract
from arXiv · showhide
This paper proposes an end-to-end trainable network, SegFlow, for simultaneously predicting pixel-wise object segmentation and optical flow in videos. The proposed SegFlow has two branches where useful information of object segmentation and optical flow is propagated bidirectionally in a unified framework. The segmentation branch is based on a fully convolutional network, which has been proved effective in image segmentation task, and the optical flow branch takes advantage of the FlowNet model. The unified framework is trained iteratively offline to learn a generic notion, and fine-tuned online for specific objects. Extensive experiments on both the video object segmentation and optical flow datasets demonstrate that introducing optical flow improves the performance of segmentation and vice versa, against the state-of-the-art algorithms.
1. Introduction
SegFlow jointly predicts video object segmentation and optical flow because the tasks are closely related but each can be unreliable alone. Its bidirectional, end-to-end framework and iterative training strategy let the two branches exchange information without requiring paired annotations for both tasks.
- Video object segmentation assigns pixel-wise foreground/background labels across a video, typically starting from a known object mask in the first frame.
- Optical flow can improve temporal segmentation smoothness, but inaccurate flow may be incomplete within fast-moving objects.
- SegFlow uses separate segmentation and optical-flow branches whose learned feature representations are propagated bidirectionally to facilitate both tasks.
- The iterative training strategy jointly optimizes both tasks while requiring only one type of ground truth at a time, rather than a dataset containing both annotations.
- The framework is evaluated on video object segmentation and optical-flow datasets, with experiments reporting favorable performance against state-of-the-art algorithms on both tasks.
2. Related Work
Prior work addresses video segmentation and optical flow separately or combines them through computationally expensive inference procedures. SegFlow instead provides a jointly trainable framework with bidirectional feature propagation between segmentation and flow branches.
- Unsupervised Video Object Segmentation: Unsupervised video object segmentation methods use motion, superpixels, or object proposals, with proposal-based approaches requiring heavy computation to generate and associate segments.
- Semi-supervised Video Object Segmentation: Semi-supervised methods track an object mask given in the first frame using superpixels, graphical models, object proposals, or CNN-based offline and online training.
- Optical Flow: Optical-flow approaches include variational and matching-based optimization, while learning-based methods are presented as more efficient alternatives.
- Fusion Methods: Fusion methods combine segmentation and flow but may rely on complicated inference or pre-computed flow, limiting applications or constraining integration.
- Fusion Methods: ObjectFlow jointly iterates segmentation and flow energy optimization online, making it computationally expensive and requiring segmentation results before flow estimation.
- Fusion Methods: SegFlow contrasts with these approaches by using an end-to-end trainable framework that simultaneously predicts pixel-wise foreground segmentation and optical flow.
- Fusion Methods: The proposed architecture contains segmentation and FlowNetS-based flow branches connected during up-sampling through bidirectional multi-scale feature propagation and iterative joint optimization.
3. SegFlow
SegFlow unifies segmentation and optical-flow prediction in two bidirectionally connected branches, combining task-specific representations for joint inference.
- Architecture: SegFlow combines a fully convolutional segmentation branch with a FlowNetS-based optical-flow branch in one unified model.The two branches predict foreground segmentation and flow between video frames.
- Segmentation branch: The segmentation branch adapts ResNet-101 by removing classification layers and fusing multilevel convolutional features during up-sampling.Feature maps from the third through fifth convolution modules are up-sampled and summed for the final prediction.
- Segmentation branch: The segmentation objective uses weighted pixel-wise cross-entropy to address foreground–background pixel imbalance.The weight is computed from the foreground-to-background pixel-number ratio.
- Optical-flow branch: The optical-flow branch uses an encoder–decoder with skip links and is optimized with endpoint error loss on adjacent frames.Using frames t and t+1 aligns flow with segmentation at frame t for later information combination.
- Bi-directional model: SegFlow propagates feature maps bidirectionally across branches at different scales, rather than exchanging only final outputs.Feature-space communication lets the branches share objectness and motion representations during joint optimization.
4. Network Implementation and Training
Training alternates optimization between the two branches because joint annotations are unavailable, then adapts segmentation online to each target object.
- Offline training: The training procedure iteratively updates both branches using only one task’s ground truth at a time.This avoids requiring a dataset containing both segmentation and optical-flow annotations.
- Offline training: After three offline training rounds, segmentation accuracy and optical-flow loss converge on a training subset.The figure reports convergence with respect to training iterations.
- Offline training: The model is initialized from ResNet-101 and FlowNetS weights, with each branch trained while the other branch is frozen.Segmentation uses DAVIS data, while flow uses a separate optical-flow dataset and a smaller learning rate for its EPE loss.
- Offline training: Iterative updates gradually improve optical-flow estimates on DAVIS relative to the initial FlowNetS model.The visual comparison includes the initial model and SegFlow results from the first and second optimization rounds.
- Online training: Online processing fine-tunes the segmentation branch for each sequence using the target object’s mask in the first frame.The flow branch continues providing motion representations but does not update its parameters during this stage.
- Data augmentation: Data augmentation applies one transform throughout a sequence for temporal consistency and synthesizes a next frame by deforming the foreground object.The synthesized frame handles the absence of a next frame when training online from the first test frame.
5. Experimental Results
Experiments evaluate SegFlow on video segmentation and optical-flow benchmarks, using ablations and comparisons with existing methods. Results show benefits from joint feature propagation, iterative training, data augmentation, and bidirectional task interaction.
- 5.1. Dataset and Evaluation Metrics: SegFlow is evaluated on DAVIS for segmentation and on Sintel, KITTI, and Scene Flow datasets for optical flow.DAVIS contains 50 sequences and 3455 annotated frames; the flow datasets include synthesized and real-world scenes.
- 5.1. Dataset and Evaluation Metrics: The evaluation uses region similarity J, contour accuracy F, temporal stability T, and average endpoint error for optical flow.
- 5.2. Ablation Study on Segmentation: 21% Jmean improvement shows that offline training is important for producing better segmentation results.The authors state that the network needs a generic model to discover moving objects before online fine-tuning.
- 5.2. Ablation Study on Segmentation: 7.9% overall Jmean improvement comes from combining online and iterative training, while removing the flow branch reduces Tmean by 5.6%.The joint model also produces smoother temporal results than the segmentation-only variant.
- 5.3. Segmentation Results: More than 10% Jmean improvement over other unsupervised methods is reported on DAVIS, with consistent temporal-accuracy gains from the flow branch.The image-only comparison reports MSK at 69.8 Jmean versus 74.8 for SegFlow; SegFlow uses images as inputs without additional inputs required by some competing methods.
- 5.4. Optical Flow Results: SegFlow consistently improves optical-flow endpoint errors over FlowNetS fine-tuned with the same data and performs favorably against SceneFlowNet.On KITTI, SegFlow without data augmentation outperforms FlowNetS fine-tuned with extensive augmentation, while slight overfitting is observed on Sintel.
- 5.5. Runtime Analysis: SegFlow processes segmentation and optical flow jointly in 0.3 seconds per frame, or 7.9 seconds per frame including online training.The reported runtime is faster than the cited OFL, MSK, and OSVOS comparisons while producing an additional optical-flow result.
6. Concluding Remarks
SegFlow jointly optimizes video object segmentation and optical flow through bidirectional feature propagation. An iterative training strategy relaxes the need for a dataset containing both types of ground truth, and experiments support favorable performance on both tasks.
- SegFlow is an end-to-end trainable network for joint optimization of video object segmentation and optical flow estimation.
- Bidirectional feature propagation improves both segmentation and optical flow within the joint structure.
- Iterative training relaxes the requirement for a large dataset containing both foreground segmentation and optical flow ground truths.
- Extensive ablation studies show favorable performance on both video object segmentation and optical flow tasks.