RANet: Ranking Attention Network for Fast Video Object Segmentation

Ziqin Wang; Jun Xu; Li Liu; Fan Zhu; Ling Shao

Source-linked AI summary

RANet: Ranking Attention Network for Fast Video Object Segmentation

Ziqin Wang, Jun Xu, Li Liu, Fan Zhu, Ling Shao

arXiv:1908.06647v4Aug 19, 2019cs.CV

TL;DR

Semi-supervised VOS methods often trade accuracy for speed because online learning is costly and offline matching or propagation can suffer mismatching or drifting. RANet combines pixel-level matching, mask propagation, and segmentation in an end-to-end encoder-decoder, with ranking attention selecting useful similarity maps. It reports 85.5% J&F at 33 milliseconds per frame without online learning on DAVIS16, while RANet+ reaches 87.1% J&F with online learning.

Problem
Existing VOS methods struggle to achieve both satisfactory accuracy and speed because online learning is slow, while matching and propagation methods face mismatching or drifting.
Source: §1 · pp. 1–2
Method
RANet integrates pixel-level matching, mask propagation, and segmentation in an end-to-end encoder-decoder with a module that ranks and selects similarity maps.
Source: §1 · p. 2
Results
85.5% J&F at 33 milliseconds per frame is achieved without online learning on DAVIS16, while RANet+ reaches 87.1% J&F with online learning.
Source: §4.2 · pp. 6–7
Takeaways & Limitations
RANet provides a real-time VOS network that combines matching and propagation while reporting state-of-the-art speed and accuracy on DAVIS16/17.
Source: §5 · p. 8

Abstract

from arXiv · show

Despite online learning (OL) techniques have boosted the performance of semi-supervised video object segmentation (VOS) methods, the huge time costs of OL greatly restrict their practicality. Matching based and propagation based methods run at a faster speed by avoiding OL techniques. However, they are limited by sub-optimal accuracy, due to mismatching and drifting problems. In this paper, we develop a real-time yet very accurate Ranking Attention Network (RANet) for VOS. Specifically, to integrate the insights of matching based and propagation based methods, we employ an encoder-decoder framework to learn pixel-level similarity and segmentation in an end-to-end manner. To better utilize the similarity maps, we propose a novel ranking attention module, which automatically ranks and selects these maps for fine-grained VOS performance. Experiments on DAVIS-16 and DAVIS-17 datasets show that our RANet achieves the best speed-accuracy trade-off, e.g., with 33 milliseconds per frame and J&F=85.5% on DAVIS-16. With OL, our RANet reaches J&F=87.1% on DAVIS-16, exceeding state-of-the-art VOS methods. The code can be found at https://github.com/Storife/RANet.

1. Introduction

Semi-supervised VOS must balance accurate segmentation with practical speed. RANet integrates matching and propagation in an end-to-end encoder-decoder and uses ranking attention to improve similarity-map utilization.

Task

Semi-supervised VOS segments objects throughout a video using only the first frame’s annotated mask as a template.The task supports large-scale video processing, editing, video understanding, and object tracking.Source: §1 · p. 1

Motivation

Online-learning methods achieve accurate VOS but require several seconds per frame, while faster matching and propagation methods have sub-optimal accuracy.Matching methods face mismatching under appearance changes, whereas propagation methods can drift under occlusions or fast motion.Source: §1 · p. 1
Existing methods generally cannot provide both satisfactory VOS accuracy and speed, motivating more efficient approaches with a better speed-accuracy trade-off.Both accuracy and speed are described as essential for practical applications.Source: §1 · pp. 1–2

Proposed Approach

RANet uses an encoder-decoder to learn pixel-level matching, mask propagation, and segmentation end-to-end.A Siamese encoder extracts matching features, while a pyramid-like decoder performs propagation and high-resolution segmentation.Source: §1 · p. 2
The Ranking Attention Module ranks and selects dynamic foreground and background similarity maps before decoding them for fine-grained VOS.The module connects the matching encoder with the propagation-based decoder by reorganizing maps according to their importance.Source: §1 · p. 2

Reported Result

85.5% J&F at 30 FPS is reported for RANet on DAVIS16, demonstrating the intended speed-accuracy trade-off.The passage identifies this result as evidence that RANet outperforms previous VOS methods in speed and accuracy.Source: §1 · p. 2

2. Related Works

Prior VOS approaches trade accuracy for speed or speed for accuracy. RANet combines efficient pixel-level matching with propagation and a lightweight decoder to address these limitations.

Online Learning

Online-learning methods fine-tune on the first video frame, achieving effective segmentation but incurring computational expense in practical applications.The related-work discussion describes online learning as effective yet costly.Source: §2 · p. 2

Propagation Methods

Propagation methods use previous-frame information, while RANet also uses a Siamese network but replaces simple feature stacking with pixel-level matching.RANet feeds the previous frame’s mask into its decoder together with selected matching features.Source: §2 · p. 2

RANet Position

RANet’s related-work position is to combine correlation-based matching, mask propagation, and a lightweight decoder for efficient segmentation.The figure text identifies separate encoder, correlation and RAM, and decoder components.Source: §2 · pp. 2–3

Matching Methods

Matching methods are efficient but point-to-point correspondence can produce noisy predictions, motivating decoder-based use of matching results as guidance.The related work contrasts RANet’s richer similarity-map utilization with methods that retain only extreme-value maps.Source: §2 · pp. 2–3

3. Proposed Method

RANet comprises a Siamese encoder, correlation and Ranking Attention Module, and decoder. It filters, scores, ranks, and reshapes similarity maps before combining them with propagated masks for segmentation.

Network Overview

RANet has three parts: feature-extraction encoder, correlation plus RAM integration, and decoder for feature merging and final segmentation.The architecture is illustrated as a sequence of these components.Source: §3.1 · p. 3
The Siamese encoder extracts shared-weight features from the template and current frames for correlation-based matching.Template features are reshaped into a set of pixel-level features for correlation calculation.Source: §3.1 · p. 3, §3.2 · p. 4

Correlation

Correlation produces one similarity map for each pixel-level template feature by matching template features against the current-frame feature map.The template and current-frame features are ℓ2-normalized before correlation.Source: §3.1 · p. 3, §3.2 · p. 4

Propagation and Decoder

The decoder receives selected foreground/background features and the previous frame’s predicted mask, combining matching and propagation for segmentation.The decoder is lightweight and pyramid-like, with feature merging and skip connections.Source: §3.1 · p. 4, §3.4 · p. 5

Ranking Attention Module

RAM first filters similarity maps into foreground and background tensors using the first-frame mask, then learns an importance score for each map.The ranking network combines a two-layer function with channel-wise global max-pooling; larger scores indicate greater importance.Source: §3.2 · p. 4
RAM ranks maps from largest to smallest score and fixes each path at 256 channels by zero-padding or discarding redundant maps.Foreground and background paths are processed separately before channel concatenation.Source: Figure 3 · p. 4, §3.2 · p. 5

Multi-object VOS

For multi-object VOS, RANet shares encoder features and correlation maps across objects while generating object-specific foreground and background masks.This avoids processing every object independently through the shared portions of the network.Source: §3.3 · p. 5

Training

Training uses Adam with an initial learning rate of 10^-5, binary cross-entropy loss, 480 × 864 inputs, and geometric data augmentation.Augmentations include Thin Plate Splines, rotations, scaling, and random cropping.Source: §3.5 · p. 5

4. Experiments

Experiments evaluate RANet on DAVIS16 and DAVIS17 against state-of-the-art methods, then analyze its components, training choices, online learning, and robustness in challenging scenarios.

Comparison to the state of the art

85.5% J &F Mean at 33 milliseconds per frame demonstrates RANet’s offline speed-accuracy result on DAVIS16-val.Without online learning, RANet exceeds the compared offline methods on objective metrics and is faster than all except SiamMask.Source: §4.2 · p. 6
87.1% J &F Mean is achieved by RANet+OL on DAVIS16, outperforming all online-learning VOS methods reported in the comparison.The offline RANet also achieves comparative results with state-of-the-art online-learning methods.Source: §4.2 · p. 7

Validation of the Proposed RANet

RAM improves J Mean by 3.6% over w/o Ranking and 4.4% over Maximum by ranking and selecting similarity maps instead of retaining or maximizing them directly.The ablation attributes the gains to preserving useful similarity information and organizing maps more effectively.Source: §4.3 · p. 7
Removing the correlation layer reduces J Mean to 67.5%, while removing the previous-frame mask lowers J Mean by 4.1 points.These ablations identify correlation and propagated temporal information as important components of RANet.Source: §4.3 · pp. 7–8
Online learning continuously improves J &F Mean as iterations increase, but reduces speed; qualitative results cover appearance changes, fast motion, occlusions, and multiple objects.The visual evaluation spans DAVIS16 and DAVIS17 challenging scenarios.Source: §4.3 · p. 8, §4.4 · p. 8

5. Conclusion

RANet is a real-time VOS network that jointly learns pixel-level feature matching and mask propagation, using ranking attention to exploit similarity features. It achieves state-of-the-art speed and segmentation accuracy on DAVIS16/17, with extensions suggested for tracking and stereo vision.

RANet runs at 30 FPS on a single Titan Xp GPU while targeting accurate VOS.Source: §5 · p. 8
The network end-to-end learns pixel-level feature matching and mask propagation for VOS.Source: §5 · p. 8
The ranking attention module reorganizes similarity features to support fine-grained VOS performance, while matching features guide rather than determine final predictions.Source: §5 · p. 8
The authors report state-of-the-art segmentation accuracy and speed on DAVIS16/17 datasets.Source: §5 · p. 8
The ranking attention module could also be applied to object tracking and stereo vision, while improved propagation or local matching may improve VOS.Source: §5 · p. 8