Source-linked AI summary
Full-Duplex Strategy for Video Object Segmentation
Ge-Peng Ji, Deng-Ping Fan, Keren Fu, Zhe Wu, Jianbing Shen, Ling Shao
TL;DR
Video object segmentation needs a better way to combine appearance and motion than existing simplex strategies provide. FSNet uses bidirectional RCAM and BPM modules to mutually constrain and purify cross-modal features, and reports favourable results across video object segmentation and video salient object detection benchmarks, including a 2.4% Mean-F score margin over MAT on DAVIS16.
Problem
Existing video object segmentation approaches use simplex appearance-motion interactions, limiting mutual feature collaboration and allowing inconsistent cues to accumulate.
Method
FSNet combines bidirectional RCAM feature interaction with BPM purification to extract cross-modal representations and update inconsistent spatial-temporal features.
Results
FSNet achieves favourable performance across five benchmarks and tasks, including a 2.4% Mean-F score margin over MAT on DAVIS16 with less training data.
Takeaways & Limitations
FSNet is presented as a unified framework for robust U-VOS and V-SOD under challenging conditions such as dynamic backgrounds.
Abstract
from arXiv · showhide
Previous video object segmentation approaches mainly focus on using simplex solutions between appearance and motion, limiting feature collaboration efficiency among and across these two cues. In this work, we study a novel and efficient full-duplex strategy network (FSNet) to address this issue, by considering a better mutual restraint scheme between motion and appearance in exploiting the cross-modal features from the fusion and decoding stage. Specifically, we introduce the relational cross-attention module (RCAM) to achieve bidirectional message propagation across embedding sub-spaces. To improve the model's robustness and update the inconsistent features from the spatial-temporal embeddings, we adopt the bidirectional purification module (BPM) after the RCAM. Extensive experiments on five popular benchmarks show that our FSNet is robust to various challenging scenarios (e.g., motion blur, occlusion) and achieves favourable performance against existing cutting-edges both in the video object segmentation and video salient object detection tasks. The project is publicly available at: https://dpfan.net/FSNet.
1 Introduction
Video object segmentation must reconcile appearance and motion cues, but existing direction-independent and simplex strategies overlook their mutual restraint and can accumulate feature drift. FSNet addresses this with bidirectional full-duplex interaction and purification, reporting favourable benchmark performance.
- Appearance and motion features can conflict because they arise from distinct modalities, while simplex strategies guide only one cue with the other.
- Misguided modality-specific cues, such as static shadows or background objects, can corrupt feature propagation and stain segmentation results.
- FSNet introduces bidirectional attention between spatial- and temporal-aware cues to incorporate appearance and motion patterns within one framework.
- RCAM extracts discriminative features from appearance and motion branches, while BPM updates inconsistent spatial-temporal features through bidirectional purification.
- 2.4% higher Mean-F score than MAT on the DAVIS16 leaderboard is reported for FSNet (N=4, CRF), despite using less training data.The comparison is Ours-13K versus MAT-16K.
- FSNet is reported to outperform existing SOTA models on additional challenging DAVSOD19-Normal and DAVSOD19-Difficult tests.
2 Related Works
Related work spans unsupervised video object segmentation and attention-based video analysis, with ongoing difficulty in combining motion and appearance cues. FSNet uses a full-duplex architecture to address limitations of independent, optical-flow-dependent, and simplex fusion strategies.
- Unsupervised VOS: Unsupervised VOS omits first-frame ground-truth masks and is the setting studied in this paper, while semi-supervised VOS is left for future work.
- Unsupervised VOS: Traditional unsupervised VOS methods use handcrafted trajectories, proposals, saliency priors, optical flow, or superpixels, but have limited generalizability.
- Unsupervised VOS: Combining motion and appearance remains a long-standing problem because optical-flow-dependent methods may fail between similar consecutive frames.
- Attention-based VOS: Attention-based VOS methods compute frame saliency and apply spatial-temporal optimization or learn semantic representations through end-to-end detection.
- FSNet architecture: FSNet's architecture uses RCAM to abstract cross-cue representations, cascaded BPMs to re-calibrate inconsistencies, and a decoder to generate predictions.
3 Methodology
FSNet processes appearance frames and optical-flow maps through separate branches, then combines their spatial-temporal representations with bidirectional cross-attention and purification. RCAM produces cross-modal features, while subsequent fusion and decoding yield predictions.
- Input and backbone: Optical flow maps are generated from adjacent frames, and the final frame is discarded so appearance and motion inputs match.The flow generator is FlowNet 2.0, with M_t = H[A_t, A_{t+1}].
- Input and backbone: Matched appearance and motion inputs are processed by two independent ResNet-50 branches to extract modality-specific feature hierarchies.
- Relational Cross-Attention Module: RCAM receives multi-level appearance and motion features and embeds spatial-temporal cross-modal representations through bidirectional channel-wise attention.
- Relational Cross-Attention Module: RCAM uses global average pooling and learned 1×1 convolutions to form sigmoid-bounded attention vectors that modulate features from the other modality.
- Feature fusion: Fused features combine candidate cross-modal features with lower-level fused features through element-wise addition in hierarchy-specific backbone blocks.
- Feature fusion: The implementation uses the top four feature pyramids, with K = 4 denoting the selected feature hierarchies.
3.3 Bidirectional Purification Module
The BPM uses bidirectional attention to combine and recalibrate spatial-temporal features, while cascaded purification and top-down connections produce robust multiscale decoder representations.
- Bidirectional Purification Module: The BPM comprises N cascaded units that improve robustness through bidirectional purification after feature integration.The units use feature allocators to unify representations and reduce computational burden.
- Bidirectional Attention: Motion features enrich fused features through concatenation, while fused features suppress distractors in motion features through multiplication.These two simplex interactions form the module's bidirectional attention scheme.
- Bidirectional Attention: The interlaced decremental connection broadcasts high-level semantic features to low-level features through a top-down pathway.This cross-modal fusion strategy supports robust feature representations across levels.
- Decoder: The final BPM produces two discriminative feature groups with 32 channels for decoder processing.The decoder uses pyramid pooling and U-Net skip connections, retaining global information across scales.
- Decoder: The decoder upsamples and concatenates pyramid features, reduces channels from 64 to 32, and generates predictions with a single-filter convolution and sigmoid.The resulting prediction is produced at frame t.
3.5 Learning Objective
The section specifies the model's loss, prediction formulation, backbone configuration, and implementation settings, including a reported benefit from combining appearance and motion cues.
- Learning Objective: Binary cross-entropy measures dissimilarity between the output prediction and the ground-truth mask.The loss is defined for the prediction map at frame t.
- Backbone and Features: The model uses three standard ResNet-50 backbones for appearance, motion, and merging branches.Each backbone produces four feature hierarchies from conv2_3 through conv5_3.
- Learning Objective: Removing the merging branch causes a 2.5% Sα performance drop on DAVIS16.The passage attributes this to the merging branch's sequential enhancement of spatial-temporal features from RCAMs.
- Implementation: The implementation resizes inputs to 352×352 and trains with multi-scale factors of {0.75, 1, 1.25}.The variant with N=4 BPMs achieves the best performance in the reported experiment.
- Inference: Inference takes 0.08s per frame regardless of flow generation and CRF post-processing.The method resizes each frame and motion map to 352×352 before branch processing.
4.1 Experimental Protocols
The experiments evaluate FSNet across multiple video object segmentation and video salient object detection datasets using a staged training protocol and standard benchmark splits.
- Datasets: The evaluation covers DAVIS16, MCL, FBMS, SegTrack-V2, and DAVSOD19, spanning VOS and V-SOD settings.DAVSOD19 is described as a challenging visual-attention dataset with diverse attributes.
- Training Protocol: Training proceeds in three stages: spatial-branch pretraining, temporal-branch training on optical flow, and joint training from the pretrained subtasks.The supplied passage explicitly describes the first two stages and begins the third.
- Evaluation Protocol: Testing follows standard splits: DAVIS16 validation, FBMS test, DAVSOD19 Easy35, and the complete MCL and SegTrack-V2 sets.The listed sets contain 20, 30, 35, 9, and 13 clips respectively.
4.2 Evaluation Metrics
The evaluation uses region, contour, alignment, structure, error, and precision-recall metrics, with threshold-based measures specified for V-SOD and binary predictions for U-VOS.
- U-VOS Metrics: U-VOS evaluation uses binary prediction maps and reports Mean Region Similarity and Mean Contour Accuracy.Mean Region Similarity is intersection-over-union, while Mean Contour Accuracy uses contour precision and recall.
- V-SOD Metrics: V-SOD prediction maps may be non-binary, so evaluation includes MAE, PR curves, maximum F-measure, enhanced alignment, and structure measures.The supplied metrics cover pixel-wise error, thresholded precision-recall behavior, local-global similarity, and structural similarity.
- Threshold-Based Metrics: PR curves are obtained by thresholding prediction maps over T ∈[0, 255].Maximum F-measure is computed by iterating over these thresholds, with β2=0.3 to emphasize precision.
- Benchmark Reporting: Tab. 1 compares FSNet with 14 unsupervised and seven semi-supervised models on the DAVIS16 validation set.The table distinguishes use of optical flow and CRF post-processing, with best scores marked in bold.
4.3 U-VOS and V-SOD tasks
FSNet achieves strong performance across U-VOS and V-SOD benchmarks, including challenging datasets and scenarios. Its results remain favorable across thresholds and qualitative cases involving fast motion, occlusion, out-of-view objects, and deformation.
- DAVIS16: FSNet outperforms AAAI’20-MAT by 2.4% in Mean-F and 1.0% in Mean-J on DAVIS16, achieving new SOTA performance.
- V-SOD benchmarks: FSNet consistently outperforms 13 SOTA V-SOD models on all metrics across DAVIS16, MCL, and FBMS.For Sα and Fmaxβ, it improves by approximately 2.0% over AAAI’20-PCAS.
- MCL: On MCL, FSNet improves all metrics by 3.0–8.0% over ICCV’19-RCR and CVPR’19-SSAV despite fuzzy boundaries from fast object movements.
- DAVSOD19: On DAVSOD19, FSNet outperforms reported algorithms across all metrics on Easy35 and achieves the best performance on two metrics under Difficult20.It exceeds PCSA by 3.2% in Sα on Easy35 and SSAV by 8.8% in Fmaxβ on Difficult20 without additional training data.
- PR curves: FSNet achieves the best performance under all tested thresholds across six V-SOD datasets, including the three challenging DAVSOD19 subsets.
- Qualitative results: Qualitative results show fine-grained target inference under fast motion, out-of-view conditions, occlusion, and deformation.
4.4 Ablation Study
Ablation studies show that appearance and motion cues benefit from joint use, while RCAM, BPM, and bidirectional interaction each improve the framework. Four BPMs provide the selected performance–complexity trade-off.
- Stimulus selection: Using appearance and motion simultaneously is motivated because motion performs better on DAVIS16 while appearance performs better on MCL.
- RCAM: RCAM consistently outperforms vanilla concatenation-based fusion on DAVIS16 and MCL.The module is associated with mutual correction and attention between modalities.
- BPM: Adding BPM improves performance by 2.0–3.0% over the framework without BPM.The reported improvement is attributed to BPM’s interlaced decremental connection for fusing different signals.
- Number of cascaded BPMs: FSNet performance improves from N=0 to N=4 BPMs, then saturates; more than four BPMs increase model complexity and over-fitting risk.On DAVIS16, Mean-J and Mean-F rise from 76.4 and 76.8 at N=0 to 82.1 and 83.3 at N=4.
4.5 Further Discussion
Further analyses examine final-output selection, CRF post-processing, training-data efficiency, and the distinct roles of RCAM and BPM. The findings support FSNet’s performance with fewer labeled samples and bidirectional module cooperation.
- Final prediction: Choosing the appearance-branch output as the final segmentation result performs slightly better than choosing the motion output or their average.
- Post-processing: FSNet without CRF surpasses MAT in Mean-F, while FSNet with CRF achieves the best Mean-J and Mean-F results.
- Training efficiency: FSNet with CRF surpasses MAT using fewer labeled training images, whereas 3DC-Seg reaches Mean-J=84.3 and Mean-F=84.7 using 158K images.FSNet uses 13K images compared with MAT’s 16K images in the cited comparison.
- Module roles: RCAM and BPM use different interaction patterns: RCAM transmits features point-to-point in the encoder, while BPM supports multi-level feature interaction in the decoder.
- Bidirectional behavior: Bidirectional RCAM and BPM outperform unidirectional settings, and bidirectional purification improves Sα by 2.1% on DAVIS16 and 1.0% on MCL over self-purification.
5 Conclusion
FSNet is a unified framework that leverages mutual constraints between appearance and motion cues for video object segmentation. RCAM abstracts dual-modality features during encoding, while BPM recalibrates inconsistent features during decoding, and experiments validate the architecture across U-VOS and V-SOD.
- FSNet leverages mutual constraints between appearance and motion cues to address video object segmentation.
- RCAM abstracts dual-modality features in the encoding stage, while BPM step-by-step recalibrates inconsistent features in the decoding stage.
- Extensive experiments validate FSNet's functional modules and show it advances state-of-the-art models for both U-VOS and V-SOD tasks.