Source-linked AI summary
Semi-Supervised Video Salient Object Detection Using Pseudo-Labels
Pengxiang Yan, Guanbin Li, Yuan Xie, Zhen Li, Chuan Wang, Tianshui Chen, Liang Lin
TL;DR
Deep video salient object detection relies on large quantities of pixel-wise annotations, while sparse labels provide limited temporal cues. The paper combines a spatial refinement network, a recurrent module, and flow-guided pseudo-labels to train from sparse annotations. It reports outperforming fully supervised state-of-the-art methods and applicability to unsupervised video object segmentation.
Problem
Deep video saliency methods rely on many dense pixel-wise labels, while sparse annotations provide limited temporal cues for temporally consistent saliency maps.
Source: §1 · p. 1Method
The method combines RCRNet and a non-locally enhanced recurrent module with flow-guided pixel-level pseudo-labels generated from sparse annotations.
Source: §1 · p. 2Results
The semi-supervised model with pseudo-labels and sparse manual annotations outperforms existing state-of-the-art fully supervised video salient object detection methods.
Source: §1 · p. 2Takeaways & Limitations
The semi-supervised model is also applicable to unsupervised video object segmentation.
Source: §2.2 · p. 3The approach assumes adjacent video frames differ little enough that motion information can estimate some unannotated-frame labels.
Source: §1 · p. 2
Abstract
from arXiv · showhide
Deep learning-based video salient object detection has recently achieved great success with its performance significantly outperforming any other unsupervised methods. However, existing data-driven approaches heavily rely on a large quantity of pixel-wise annotated video frames to deliver such promising results. In this paper, we address the semi-supervised video salient object detection task using pseudo-labels. Specifically, we present an effective video saliency detector that consists of a spatial refinement network and a spatiotemporal module. Based on the same refinement network and motion information in terms of optical flow, we further propose a novel method for generating pixel-level pseudo-labels from sparsely annotated frames. By utilizing the generated pseudo-labels together with a part of manual annotations, our video saliency detector learns spatial and temporal cues for both contrast inference and coherence enhancement, thus producing accurate saliency maps. Experimental results demonstrate that our proposed semi-supervised method even greatly outperforms all the state-of-the-art fully supervised methods across three public benchmarks of VOS, DAVIS, and FBMS.
1. Introduction
Video salient object detection segments visually distinctive objects with precise contours, but deep-learning methods depend on costly dense pixel annotations and sparse labels provide limited temporal cues. The paper combines a refinement network and recurrent module with flow-guided pseudo-labels to train from sparse annotations, reporting performance above fully supervised state-of-the-art methods.
1. Introduction
- Deep video saliency methods rely on many dense pixel labels, which are time-consuming to create and difficult to keep consistent.Even experienced annotators need several minutes per frame, and saliency subjectivity makes label quality hard to guarantee.Source: §1 · p. 1
- Sparse annotations leave limited temporal cues for consistent video saliency, while motion and appearance changes can attract attention more than object categories.The paper also notes that unsupervised methods based on hand-crafted low-level features can fail in challenging cases.Source: §1 · pp. 1–2
- RCRNet extracts spatial saliency information, and its non-locally enhanced recurrent module models spatiotemporal coherence in the resulting saliency maps.The framework trains with sparse ground-truth labels together with pixel-wise pseudo-labels generated from labeled frames.Source: §1 · p. 2
- The flow-guided pseudo-label generator uses motion information to infer labels for intervals between sparsely annotated frames.The authors motivate this design by observing small differences between adjacent frames in high-sampling-rate videos.Source: §1 · p. 2
- The semi-supervised model trained with generated pseudo-labels and sparse manual annotations outperforms existing state-of-the-art fully supervised video salient object detection methods.The proposed RCRNet+NER is reported to produce more accurate saliency maps under this joint supervision.Source: §1 · p. 2
2. Related Work
Prior video salient object detection methods use deep networks but depend on densely annotated datasets; this work explores reducing that dependence. The paper distinguishes its training-time semi-supervision from video segmentation settings that use labels at inference, and reports applicability to unsupervised video object segmentation.
2.1. Salient Object Detection
- Earlier deep video salient object detection methods used spatiotemporal approaches but relied on densely annotated video datasets.The paper presents its work as the first to explore reducing dense-label dependence for this task.Source: §2.1 · p. 2
2.2. Video Object Segmentation
- Here, semi-supervision reduces training-label dependence, unlike semi-supervised video object segmentation that uses a first-frame mask at inference; the model also applies to unsupervised video object segmentation.The paper identifies unsupervised video object segmentation as the closest related task because neither requires annotations during inference.Source: §2.2 · pp. 2–3
3. Our Approach
The framework combines a residual-connected spatial refinement network, a non-locally enhanced recurrent module, and flow-guided generation of in-between pseudo-labels from sparse annotations. These components support saliency maps with spatiotemporal coherence from dense input frames.
3. Our Approach
- The overall pipeline uses a residual-connected refinement network for spatial features, NER for spatiotemporal coherence, and FGPLG to generate labels between sparse annotations.The generated pseudo-labels and sparse annotations train RCRNet with NER on dense input frames.Source: §3 · p. 3
3.1. Refinement Network with Residual Connection
- The framework uses a spatial feature extractor and pixel-wise classifier, with refinement connections that restore spatial information lost during downsampling.RCRNet uses ResNet-50 features and three cascaded refinement blocks connected to the feature extractor.Source: §3.1 · pp. 3–4
3.2. Non-locally Enhanced Recurrent Module
- The NER module combines non-local blocks, which connect features across video positions, with ConvGRU to model sequential feature evolution.Forward and backward ConvGRU units exchange temporal information, and an additional non-local block further enhances spatiotemporal coherence.Source: §3.2 · p. 4
3.3. Flow-Guided Pseudo-Label Generation Model
- FGPLG estimates an intermediate-frame label by warping endpoint annotations with optical flow, then refining the noisy estimates using a modified RCRNet.Its inputs include the intermediate frame, two warped labels, and optical-flow magnitudes.Source: §3 · p. 3, §3.3 · p. 5
- FGPLG trained on triples sampled at larger interframe intervals can generate dense pseudo-labels of very high quality, demonstrating strong generalization.The experiments select sparse annotations at a fixed interval l.Source: §3.3 · p. 5
4. Experimental Results
RCRNet+NER outperforms prior methods on VOS, DAVIS, and FBMS using approximately 20% of ground-truth labels. The experiments also examine pseudo-label quantities, module contributions, and performance on varied training data.
4.1. Datasets and Evaluation
- Evaluation covers VOS, DAVIS, and FBMS, using precision-recall curves, maximum F-measure, and S-measure.VOS has 200 videos and 116,103 frames, DAVIS has 50 videos and 3,455 annotated frames, and FBMS has 59 videos and 720 sparsely annotated frames.Source: §4.1 · p. 5
4.3. Comparison with State-of-the-Art
- 15.52%, 1.18%, and 4.62% maximum F-measure gains, and 9.41%, 0.68%, and 2.72% S-measure gains, respectively, over prior best methods on VOS, DAVIS, and FBMS.Figure 5 shows RCRNet+NER outperforming other methods on all three datasets using only 20% of ground-truth labels; it also beats fully supervised PDB with the same ResNet-50 backbone.Source: Figure 5 · p. 6, §4.3 · p. 6
4.2. Implementation Details
- Training uses a pretrained RCRNet, FGPLG-generated pseudo-labels, and joint training of RCRNet+NER on clips of four frames.Frames are resized to 448×448; the reported inference speed is approximately 37 ms per frame, or 27 fps.Source: §4.2 · pp. 5–6
4.3. Comparison with State-of-the-Art
- Visual comparisons show RCRNet+NER producing saliency maps closest to ground truth across the displayed cases.The comparison presents the ground truth in the last column.Source: Figure 6 · p. 7
4.4. Sensitivities to Different Amount of Ground Truth and Pseudo-Labels Usage
- On VOS, 20% annotations plus 20% pseudo-labels reached max Fβ = 0.861 and S-measure = 0.874, surpassing training with all ground-truth labels.With 5% annotations and 35% pseudo-labels, performance was comparable; the authors report that pseudo-labels can partly overcome labeling ambiguity and reduce the need for dense manual annotation.Source: Table 2 · p. 7, §4.4 · p. 7
- On VOS, increasing ground-truth usage substantially benefits training when labels are severely insufficient, such as at 5%; denser labels do not always improve results when training data are sufficient.The sensitivity analysis varies the training set by fixed intervals.Source: §4.4 · p. 7, Figure 7 · p. 7
4.5. Ablation Studies
- Adding non-local blocks and DB-ConvGRU improves performance; an extra non-local block adds 0.5% maximum F-measure, and DB-ConvGRU outperforms ConvGRU.The comparison fixes training data at one ground truth and one pseudo-label every five frames.Source: §4.5 · pp. 7–8, Table 3 · p. 8
- All tested pseudo-label-trained variants outperform the no-pseudo-label baseline, while the full flow-guided generator Gd performs best among the tested variants.Adding adjacent ground truth as input slightly improves Gb to Gc; Gd further uses flow-guided motion estimation to exploit adjacent ground truth.Source: §4.5 · p. 8, Table 4 · p. 8
5. Performance on Unsupervised Video Object Segmentation
The paper tests pretrained RCRNet+NER on unsupervised video object segmentation without pre- or post-processing. It outperforms six representative methods on DAVIS and FBMS, supporting its applicability to that task.
5. Performance on Unsupervised Video Object Segmentation
- RCRNet+NER outperforms six representative unsupervised video segmentation methods on both DAVIS and FBMS.The evaluation uses mean Jaccard index on FBMS and mean Jaccard index and mean contour accuracy on DAVIS.Source: §5 · p. 8
- The pretrained model is tested without any pre- or post-processing against FST, SFL, LMP, FSEG, LVO, and PDB.The comparison uses the DAVIS and FBMS datasets.Source: §5 · p. 8
- The authors interpret the results as evidence of spatiotemporal information capture and applicability to unsupervised video segmentation.The paper distinguishes this task's binary object separation from per-pixel saliency probabilities.Source: §5 · p. 8
6. Conclusion
The proposed RCRNet+NER framework uses a few ground truths and flow-guided pseudo-labels to capture spatiotemporal information. It achieves state-of-the-art video salient object detection performance and is also applicable to unsupervised video segmentation.
6. Conclusion
- RCRNet+NER captures spatiotemporal information using a few ground truths and an appropriate number of flow-guided pseudo-labels.The framework is described as accurate and cost-effective.Source: §6 · p. 8
- The proposed method achieves state-of-the-art performance on video salient object detection.Source: §6 · p. 8
- The framework is also applicable to unsupervised video segmentation, and future work will examine keyframe selection instead of interval sampling for ground-truth labels.Source: §6 · p. 8
(Supplemental Materials)
The supplemental materials identify the paper’s authors.
(Supplemental Materials)
- The listed authors are Pengxiang Yan, Guanbin Li, Yuan Xie, Zhen Li, Chuan Wang, Tianshui Chen, and Liang Lin.Source: (Supplemental Materials) · p. 11
1. More Comparison with State-of-the-Art
The proposed RCRNet+NER outperforms the compared salient object detection algorithms across VOS, DAVIS, and FBMS on maximum F-measure, S-measure, and weighted F-measure, while reducing MAE.
1. More Comparison with State-of-the-Art
- Maximum F-measure improves over existing best-performing algorithms by 15.52% on VOS, 1.18% on DAVIS, and 4.62% on FBMS.Source: §1 · p. 11
- S-measure improves over existing best-performing algorithms by 9.41% on VOS, 0.68% on DAVIS, and 2.72% on FBMS.Source: §1 · p. 11
- Weighted F-measure improves over existing best-performing algorithms by 17.04% on VOS, 3.23% on DAVIS, and 3.68% on FBMS.Source: §1 · p. 11
- MAE is reduced over existing best-performing algorithms by 34.67% on VOS, 6.67% on DAVIS, and 5.26% on FBMS.Source: §1 · p. 11
2. Sensitivities to Different Amount of Ground Truth and Pseudo-Labels Usage
Experiments on VOS examine how ground-truth and pseudo-label amounts affect RCRNet+NER. A semi-supervised setting can outperform training with all annotated frames, while very sparse training data yield inferior performance.
2. Sensitivities to Different Amount of Ground Truth and Pseudo-Labels Usage
- The semi-supervised RCRNet+NER model in column “1 / 5” can outperform the model trained with all annotated frames.Source: §2 · p. 11
- With severely deficient training data, such as 5%, models cannot generate temporally consistent saliency maps and performance is inferior.Source: §2 · p. 11
- When training data have enough similar appearances, adding annotation data does not guarantee continuous performance improvement, possibly because of overfitting from label ambiguity.Source: §2 · p. 11
- The authors propose jointly training with an appropriate amount of pseudo-labels and ground truths, giving 20% of each as examples.Source: §2 · p. 11