Source-linked AI summary
SwiftNet: Real-time Video Object Segmentation
Haochen Wang, Xiaolong Jiang, Haibing Ren, Yao Hu, Song Bai
TL;DR
Real-time one-shot VOS must handle object variation without the computational burden of repeatedly modeling redundant frames and pixels. SwiftNet addresses this with Pixel-Adaptive Memory and a Light-Aggregation Encoder, achieving 77.8% J &F at 70 FPS on DAVIS2017 validation.
Problem
Real-time VOS remains challenging because object variation requires sophisticated modeling and matching, while indiscriminate redundancy reduction can degrade accuracy.
Method
SwiftNet uses Pixel-Adaptive Memory to trigger updates on varying frames and process dynamic pixels, together with a Light-Aggregation Encoder for efficient reference encoding.
Results
77.8% J &F at 70 FPS is reported on the DAVIS2017 validation set, with the best overall segmentation accuracy and speed performance.
Takeaways & Limitations
SwiftNet provides a strong and efficient baseline for real-time one-shot VOS.
Abstract
from arXiv · showhide
In this work we present SwiftNet for real-time semisupervised video object segmentation (one-shot VOS), which reports 77.8% J &F and 70 FPS on DAVIS 2017 validation dataset, leading all present solutions in overall accuracy and speed performance. We achieve this by elaborately compressing spatiotemporal redundancy in matching-based VOS via Pixel-Adaptive Memory (PAM). Temporally, PAM adaptively triggers memory updates on frames where objects display noteworthy inter-frame variations. Spatially, PAM selectively performs memory update and match on dynamic pixels while ignoring the static ones, significantly reducing redundant computations wasted on segmentation-irrelevant pixels. To promote efficient reference encoding, light-aggregation encoder is also introduced in SwiftNet deploying reversed sub-pixel. We hope SwiftNet could set a strong and efficient baseline for real-time VOS and facilitate its application in mobile vision. The source code of SwiftNet can be found at https://github.com/haochenheheda/SwiftNet.
1. Introduction
SwiftNet targets the speed–accuracy challenge in real-time one-shot VOS by compressing spatiotemporal redundancy with Pixel-Adaptive Memory and efficient reference encoding. It reports 77.8% J &F at 70 FPS on DAVIS2017 validation.
- Real-time VOS remains difficult because object variation demands sophisticated modeling and matching, while existing accuracy-focused methods sacrifice speed.
- Spatiotemporal redundancy arises from repeatedly modeling static historical frames and processing static pixels that contribute little to segmentation.
- The Light-Aggregation Encoder reduces redundant feature extraction while supporting multi-scale mask–frame aggregation through reversed sub-pixel operations.
- SwiftNet compresses this redundancy with Pixel-Adaptive Memory, using variation-aware temporal triggers and pixel-wise update and matching.
- 77.8% J &F and 70 FPS are achieved on the DAVIS2017 validation set, with the authors reporting the best overall segmentation accuracy and speed performance.
2. Related Work
Related one-shot VOS methods differ in reference modeling and reference–query matching, balancing accuracy against computational cost. Memory-based approaches model object evolution comprehensively but require substantial computation.
- One-shot VOS methods are categorized by how they model references and match reference frames against query frames.
- Two-stage methods generally emphasize accuracy, whereas one-stage methods generally emphasize speed.
- Non-local matching captures all-pairs pixel-wise dependencies but is computationally heavy.
- Fast VOS methods reduce computation through single-frame references, tracked regions, or lightweight similarity and appearance models.
- Memory-based VOS uses all historical frames for object modeling, with STM establishing a major accuracy improvement and later methods modifying its memory or attention mechanisms.
3. SwiftNet
SwiftNet addresses redundancy in matching-based one-shot VOS through Pixel-Adaptive Memory, which adapts temporal updates and spatial pixel selection. Its light-aggregation encoder further reduces reference-encoding cost while preserving multi-scale aggregation.
- SwiftNet: SwiftNet uses Pixel-Adaptive Memory to compress spatiotemporal redundancy in matching-based one-shot VOS.PAM combines variation-aware triggering with pixel-wise memory update and matching.
- Variation-Aware Trigger: PAM triggers memory updates when accumulated inter-frame variation exceeds a threshold instead of using all historical frames indiscriminately.The best reported settings are Pth=200, thf=1, and thm=0.
- Pixel-wise Memory Update: PAM selects highly varying pixels for memory updates, adding their key and value features while omitting less informative pixels.Pixels are ranked by similarity, and the top β percent are selected; β is set to 10% for best performance.
- Pixel-wise Memory Match: Pixel-wise matching reduces the affinity map from RHW ×HWT to RHW ×K, lowering storage and computation without considerable accuracy loss.The matched reference values are concatenated with query values before decoding.
- Light-Aggregation Encoder: The light-aggregation encoder reuses query-encoded feature maps and combines them with mask features through reversed sub-pixel operations for multi-scale aggregation.SwiftNet buffers query features and directly reuses them when a frame is selected for reference encoding.
4. Experiments
Experiments compare SwiftNet with other methods on DAVIS 2016, DAVIS 2017, and YouTube-VOS. The reported comparisons identify SwiftNet as having the best overall segmentation accuracy and inference speed.
- Experiments: SwiftNet is evaluated against other state-of-the-art methods on DAVIS 2016, DAVIS 2017, and YouTube-VOS.Experiments are implemented in PyTorch on one NVIDIA P100 GPU.
- Results: SwiftNet demonstrates the best overall segmentation accuracy among the compared methods.
- Results: SwiftNet demonstrates the best overall inference speed among the compared methods.
4.1. Datasets and Evaluation Metrics
The evaluation uses DAVIS 2016, DAVIS 2017, and YouTube-VOS, measuring segmentation accuracy with J, F, and J &F and speed with FPS.
- DAVIS 2016 contains 50 single-object videos with 3,455 annotated frames, while DAVIS 2017 contains 150 sequences with 10,459 annotated frames and some multiple-object videos.
- DAVIS evaluation reports mean Jaccard J index, mean boundary F score, and mean J &F for segmentation accuracy.
- FPS measures segmentation speed.
- YouTube-VOS contains 4,453 videos with multiple objects; its validation set has 474 sequences spanning 91 classes, including 26 unseen classes.
4.2. Training and Inference
SwiftNet is pretrained on simulated MS-COCO data and then fine-tuned for target datasets, while inference updates memory online as frames are processed.
- Training: SwiftNet is pretrained on simulated MS-COCO data and fine-tuned separately on DAVIS 2017 and YouTube-VOS.
- Training: Training uses 384 × 384 inputs, Adam with an initial learning rate of 1e-5, polynomial scheduling with power 0.9, frozen backbone batch normalization, and batch size 4.
- Inference: At inference, first-frame features and its mask initialize memory M0, which is updated online throughout the video.
- Inference: Each frame uses the previous memory and current image to produce a mask; triggered frames are passed to LAE for memory updates.
4.3. Ablation Study
Ablations show that LAE improves the accuracy–speed trade-off, while PAM compresses spatial and temporal redundancy with limited accuracy loss.
- Light-Aggregation Encoder: LAE surpasses the low-level encoder baseline in both J &F and FPS, with a 19 FPS advantage, and improves J &F by 4.2% over the high-level baseline at comparable FPS.
- Pixel-Adaptive Memory: Pixel-wise memory update and matching increase speed by 30 and 18 FPS under the two temporal strategies, with approximately a 0.4% J &F drop.
- Pixel-Adaptive Memory: Variation-aware triggering increases speed by 17 and 5 FPS under the two spatial strategies, while changing J &F by at most 0.1%.
- Pixel-Adaptive Memory: J &F increases with spatial update ratio β, with β = 10% yielding the best accuracy and larger values providing no significant improvement.
- Pixel-Adaptive Memory: Larger β steadily decreases FPS, whereas variation-aware triggering consistently increases FPS across β values.
4.4. State-of-the-art Comparison
SwiftNet achieves strong accuracy–speed results across DAVIS 2017, DAVIS 2016, and YouTube-VOS, with performance depending on backbone and dataset.
- DAVIS 2017: 70 FPS is achieved by SwiftNet ResNet-18 on DAVIS 2017, while its J &F exceeds the second-fastest SAT-fast by 8.3%.
- DAVIS 2017: 81.1 J &F is achieved by SwiftNet ResNet-50 on DAVIS 2017, while STM reaches 81.8 J &F and runs almost four times slower.
- DAVIS 2016: SwiftNet ResNet-50 on DAVIS 2016 leads STM by 1.1% in J &F and 18.7 FPS, while both SwiftNet backbones outperform other methods in segmentation accuracy.
- YouTube-VOS: SwiftNet ResNet-50 on YouTube-VOS leads the second-best GCNet by 4.6% in overall score G at comparable speed.
- YouTube-VOS: SwiftNet performs stably across seen and unseen YouTube-VOS object classes, indicating favorable generalizability.
5. Conclusion
SwiftNet delivers real-time semi-supervised video object segmentation by compressing spatiotemporal redundancy through Pixel-Adaptive Memory and efficient reference encoding.
- SwiftNet achieves real-time segmentation by compressing spatiotemporal redundancy with Pixel-Adaptive Memory.PAM selects incremental frames and temporally varied pixels for memory updates and matching while ignoring static content.
- SwiftNet reduces temporal redundancy with a variation-aware trigger that selects frames for memory updates and ignores static ones.
- SwiftNet eliminates spatial redundancy through pixel-wise memory updates and matching on temporally varied pixels instead of full-frame operations.
- A light-aggregation encoder accelerates reference encoding while supporting thorough feature processing.