Source-linked AI summary
Fast and Accurate Online Video Object Segmentation via Tracking Parts
Jingchun Cheng, Yi-Hsuan Tsai, Wei-Chih Hung, Shengjin Wang, Ming-Hsuan Yang
TL;DR
Online video object segmentation requires fast, accurate segmentation of a specified target without access to future frames, while common CNN approaches rely on costly first-frame finetuning. The paper replaces that delay with part-based tracking, ROI segmentation, and similarity-based aggregation, and reports favorable DAVIS accuracy with much faster runtime.
Problem
Online video object segmentation must segment a specified target immediately and accurately, but existing approaches may require costly preprocessing or first-frame finetuning.
Method
The method tracks representative object parts, predicts partial-object masks from their bounding boxes using ROI SegNet, and aggregates them by similarity to the initial object mask.
Results
On DAVIS benchmark datasets, the method performs favorably against state-of-the-art approaches in accuracy while achieving much better runtime performance.
Takeaways & Limitations
The framework provides an online segmentation approach that uses first-frame information without adding much computational load through heavy finetuning.
Abstract
from arXiv · showhide
Online video object segmentation is a challenging task as it entails to process the image sequence timely and accurately. To segment a target object through the video, numerous CNN-based methods have been developed by heavily finetuning on the object mask in the first frame, which is time-consuming for online applications. In this paper, we propose a fast and accurate video object segmentation algorithm that can immediately start the segmentation process once receiving the images. We first utilize a part-based tracking method to deal with challenging factors such as large deformation, occlusion, and cluttered background. Based on the tracked bounding boxes of parts, we construct a region-of-interest segmentation network to generate part masks. Finally, a similarity-based scoring function is adopted to refine these object parts by comparing them to the visual information in the first frame. Our method performs favorably against state-of-the-art algorithms in accuracy on the DAVIS benchmark dataset, while achieving much faster runtime performance.
1. Introduction
Online video object segmentation must localize and segment a specified target immediately while balancing robustness, accuracy, and runtime. The paper addresses this with part-based tracking, ROI segmentation, and similarity-based aggregation without time-consuming target-video finetuning.
- Problem: Online segmentation must separate a specified target at pixel level while processing future frames without access to them.The task is challenging under camera motion, deformation, occlusion, cluttered backgrounds, and strict runtime requirements.
- Problem: Existing methods may require batch processing, optical-flow preparation, augmentation, or costly first-frame finetuning, limiting time-sensitive online use.These methods trade practical responsiveness for preprocessing or adaptation overhead.
- Approach: The proposed method immediately segments a specific object without time-consuming target-video finetuning through part-based tracking, ROI segmentation, and similarity-based aggregation.The three components exploit first-frame information while avoiding heavy finetuning.
- Approach: Part proposals are selected around the initial target, tracked independently, and used to provide temporally consistent regions of interest under deformation, occlusion, and appearance changes.Representative parts are chosen using overlap with the initial mask before tracking in subsequent frames.
- Approach: ROI SegNet predicts partial-object masks from part bounding boxes, while similarity-based aggregation compares tracked parts with the initial object mask to reduce false positives.The aggregation step filters noisy segmentation parts caused by incorrect tracking.
- Results: The framework is reported to perform favorably against state-of-the-art methods on DAVIS while achieving faster runtime performance.Figure 1 includes preprocessing in runtime, averaged across all frames for fair comparison.
2. Related Work
Related work spans unsupervised segmentation, semi-supervised segmentation with an initial mask, and object tracking. The paper emphasizes online constraints created by batch processing, preprocessing, finetuning, and long-term error accumulation.
- Unsupervised Video Object Segmentation: Unsupervised methods discover prominent moving objects without prior target masks but cannot identify a specific instance in multiple-object videos.They commonly use probabilistic, motion, proposal, superpixel, saliency, or optical-flow cues.
- Unsupervised Video Object Segmentation: Unsupervised approaches often process entire videos in batch mode and may generate thousands of candidate regions per frame, increasing computation time.CNN-based methods improve segmentation using hierarchical features and augmented data, but the batch-processing constraint remains relevant to online use.
- Semi-supervised Video Object Segmentation: Semi-supervised methods segment a specific object from an initial mask using tracking, proposals, graphical models, optical flow, or CNNs.Many CNN-based approaches rely heavily on finetuning through the first frame.
- Semi-supervised Video Object Segmentation: Methods that propagate the first-frame mask avoid heavy computation but can accumulate errors over long periods when they use limited first-frame information.The proposed approach instead repeatedly refers to the first frame through part-based tracking and similarity-based aggregation.
- Object Tracking: Object tracking is used as an additional localization cue for video segmentation, with conventional correlation filters and newer deep neural network trackers addressing appearance changes.Tracking supplies localization information that can support subsequent segmentation.
3. Proposed Algorithm
The proposed algorithm combines part-based tracking, ROI segmentation, and similarity-based aggregation to segment video objects online without expensive target-video finetuning.
- Part-based Tracking: Part-based tracking localizes object regions through challenging deformation, fast movement, occlusion, and background noise while maintaining high recall.The method tracks local regions rather than the entire object to retain most object regions for subsequent segmentation.
- Part-based Tracking: Representative parts are selected from first-frame proposals using overlap with the known object box, reducing thousands of proposals to 50–300 parts.Candidates require Sp > 0.7 before non-maximum suppression, and their boxes are tightened within the object mask to reduce background noise.
- ROI Segmentation: ROI SegNet crops and resizes each tracked part, then uses ResNet-101 features and concatenated up-sampled maps for binary part-mask prediction.The network combines feature maps from the last three convolution modules before binary prediction.
- Similarity-based Part Aggregation: Similarity-based aggregation compares current parts with first-frame parts in feature space to suppress false-positive segments without expensive model finetuning.The scoring function combines averaged part-segmentation scores, feature similarity, and a confidence term based on overlap with the initial mask.
- Similarity-based Part Aggregation: 4% improvement in J Mean results from part aggregation compared with averaging part scores alone.The initial object mask helps reject part segmentations that are dissimilar to any first-frame object part.
4. Experimental Results
Experiments on DAVIS 2016 and 2017 show that the proposed method combines strong segmentation accuracy with fast online runtime, without target-specific finetuning. Ablations and tracker evaluations support the roles of part aggregation, tracking, and refinement.
- Tracker Evaluation: 80% recall at 0.5 IoU is achieved by the proposed tracker, compared with around 60% for most other trackers.The evaluation emphasizes high recall because missing part of an object prevents recovery of the corresponding segment.
- Ablation Study: 4% improvement in J Mean results from part aggregation over the score function alone, reducing false part segmentations through comparison with the initial mask.Whole-object tracking combined with segmentation instead caused a 1.1% drop in J Mean, demonstrating unstable tracking.
- DAVIS 2016: Ours-ref achieves 0.824 J Mean at 1.8 seconds per instance per frame, while Ours-part achieves 0.779 J Mean at 0.60 seconds.The reported DAVIS 2016 comparison places the refined method third in J Mean while maintaining fast runtime relative to state-of-the-art methods.
- DAVIS 2017: On DAVIS 2017, the approach achieves similar or better performance than a finetuning-based method without finetuning each instance.The paper notes that finetuning is costly in speed and stored model size when videos contain many objects.
- Runtime Analysis: 0.60 seconds per instance per frame is required without refinement, increasing to 1.8 seconds with CRF and tracker refinements.The 0.60-second runtime includes part-based tracking, ROI segmentation, and part aggregation; parallel Titan X GPUs process hundreds of parts.
5. Concluding Remarks
The method delivers fast, accurate online video object segmentation without time-consuming target-video finetuning. It combines part tracking, ROI segmentation, and similarity-based aggregation while retaining first-frame information.
- Part-based tracking localizes object regions and retains most of the object for subsequent segmentation.The tracker addresses deformation, occlusion, and appearance changes by using local regions surrounding the target.
- An ROI segmentation network outputs partial object segmentations from tracked regions.
- A similarity-based scoring function aggregates object parts into the final segmentation result.The function monitors information from the first frame during aggregation.
- The method achieves fast runtime speed without heavily preprocessing or finetuning on the target video.It exploits CNN-based tracking and segmentation while avoiding the time-consuming finetuning stage.
- The algorithm is applicable to other video analytics tasks requiring fast and accurate online video object segmentation.