Source-linked AI summary
See More, Know More: Unsupervised Video Object Segmentation with Co-Attention Siamese Networks
Xiankai Lu, Wenguan Wang, Chao Ma, Jianbing Shen, Ling Shao, Fatih Porikli
TL;DR
UVOS must identify primary objects despite complex backgrounds and frame-to-frame appearance changes, while existing deep models often use limited short-term temporal information. COSNet models global frame correlations with gated co-attention, training on same-video pairs and using multiple references at test time. It reports superior performance across three benchmarks, including 75.6% mean J on FBMS, while prior deep UVOS methods remain limited in global temporal coverage.
Problem
UVOS must distinguish primary objects from complex backgrounds without prior target knowledge, despite deformation, occlusion, camera movement, and other frame-level variations.
Method
COSNet uses a unified end-to-end co-attention framework that learns correlations from same-video frame pairs and combines multiple reference frames during segmentation.
Results
COSNet achieves superior performance across three benchmarks, including 75.6% mean J on FBMS and gains of 2.6% J Mean and 4.9% F Mean over PDB on DAVIS16.
Takeaways & Limitations
Global temporal correlation modeling helps COSNet suppress similar target distractions and achieve strong UVOS results using appearance information without optical flow.
Takeaways & Limitations
Recent deep UVOS models generally consider sequential and short-term temporal information, lacking a global view and comprehensive use of inherent video correlations.
Abstract
from arXiv · showhide
We introduce a novel network, called CO-attention Siamese Network (COSNet), to address the unsupervised video object segmentation task from a holistic view. We emphasize the importance of inherent correlation among video frames and incorporate a global co-attention mechanism to improve further the state-of-the-art deep learning based solutions that primarily focus on learning discriminative foreground representations over appearance and motion in short-term temporal segments. The co-attention layers in our network provide efficient and competent stages for capturing global correlations and scene context by jointly computing and appending co-attention responses into a joint feature space. We train COSNet with pairs of video frames, which naturally augments training data and allows increased learning capacity. During the segmentation stage, the co-attention model encodes useful information by processing multiple reference frames together, which is leveraged to infer the frequently reappearing and salient foreground objects better. We propose a unified and end-to-end trainable framework where different co-attention variants can be derived for mining the rich context within videos. Our extensive experiments over three large benchmarks manifest that COSNet outperforms the current alternatives by a large margin.
1. Introduction
UVOS must distinguish primary foreground objects from complex backgrounds without prior object knowledge, while handling substantial appearance changes across frames. COSNet addresses this gap by modeling global frame correlations with co-attention, and reports superior benchmark performance.
- UVOS separates primary foreground objects from backgrounds without manual interaction or prior knowledge about the target object.
- Primary objects are defined by local saliency within individual frames and global consistency across the video sequence.Short clips can leave the foreground ambiguous, whereas sufficiently long sequences reveal frequently appearing objects.
- Object deformation, occlusion, camera movement, out-of-view motion, and environmental variation create major micro-level segmentation challenges.
- Existing deep UVOS models emphasize short-term appearance or motion cues and may not explicitly capture rich relations among distant frames.Optical flow uses a local temporal window, while recurrent processing may fail to explore inter-frame relations explicitly.
- COSNet reformulates UVOS as co-attention, training on same-video frame pairs and using multiple reference frames to produce more discriminative foreground features.Its differentiable gated co-attention emphasizes correlated informative regions and supports global-view inference.
- COSNet reports large performance improvements and superior results over current state-of-the-art methods across three popular benchmarks.
2. Related Work
Research on video object segmentation spans unsupervised and semi-supervised settings, with prior UVOS methods using motion, proposals, saliency, or deep features. Recent deep UVOS models achieve promising results but remain limited in their use of global temporal correlations, motivating co-attention-based modeling.
- Video object segmentation is categorized into unsupervised and semi-supervised settings; this paper focuses on extracting primary objects without manual annotation.
- Early UVOS methods inferred targets from long-term motion trajectories, object proposals, or saliency information.
- Early deep UVOS approaches were limited by insufficient end-to-end learning and heavyweight fully connected architectures.
- Recent deep UVOS models achieve promising results but mainly model sequential and short-term temporal information rather than global video correlations.
- In COSNet training, same-video frame pairs are embedded, correlated through co-attention, and combined before segmentation prediction.
3. Proposed Algorithm
COSNet reformulates unsupervised video object segmentation as global co-attention between frames, combining co-attention-enhanced features with segmentation modules. Its training uses frame pairs, while testing aggregates information from multiple reference frames to exploit global video correlations.
- Co-attention framework: COSNet encodes correlations between video frames through co-attention, helping identify frequently coherent foreground regions.During training, frame pairs from the same video are used for correlation learning; during testing, the model uses co-attention information between the test and reference frames.
- Co-attention mechanisms: Vanilla co-attention computes an affinity matrix between the feature representations of two frames and derives attention summaries from their cross-frame similarities.Feature representations Va and Vb are transformed and compared across spatial locations before producing co-attention-enhanced features.
- Co-attention mechanisms: Symmetric and channel-wise co-attention constrain or simplify the projection matrix to preserve useful embedding properties and reduce channel-wise redundancy.The symmetric variant projects embeddings into an orthogonal common space, while the channel-wise variant applies channel weights before similarity computation.
- Gated co-attention: Gated co-attention learns confidence weights for attention summaries, controlling how much information from each reference frame is preserved despite appearance variation, occlusion, and background noise.The gate uses a logistic sigmoid and updates attention summaries through a channel-wise Hadamard product.
- Network architecture: The final co-attention representation is concatenated with the original feature and passed to a segmentation network to produce the output mask.COSNet comprises feature embedding, co-attention, and segmentation modules in a Siamese architecture.
- Testing pipeline: During testing, COSNet uses multiple reference frames from the same video because frame-by-frame inference does not fully exploit global correlations and can retain noise.Reference frames may be sampled uniformly and combined through averaged predictions or multiple attention summaries.
4. Experiments
Experiments evaluate COSNet on DAVIS16, FBMS, and Youtube-Objects through ablations, reference-frame studies, quantitative benchmarks, and qualitative examples. Results show that global co-attention and multiple reference frames improve segmentation, while COSNet achieves strong performance using appearance information alone.
- Experimental Setup: COSNet is evaluated on DAVIS16, FBMS, and Youtube-Objects using dataset-specific region, boundary, and stability metrics.DAVIS16 uses J, F, and T; FBMS and Youtube-Objects use mean J.
- Diagnostic Experiments: Removing co-attention reduces DAVIS mean J from 80.5 to 71.3, demonstrating the contribution of global correlation modeling.The ablation compares the full COSNet strategy with the DeepLabv3 base feature embedding network.
- Diagnostic Experiments: Global frame sampling outperforms local consecutive sampling, while symmetric co-attention slightly surpasses vanilla co-attention.The study attributes symmetric co-attention’s advantage to an orthogonal constraint that reduces feature redundancy while preserving feature norms.
- Diagnostic Experiments: Mean J improves substantially when reference frames increase from 0 to 1 and continues increasing from 2 to 5, after which performance changes little.The evaluation experiments therefore use N = 5 reference frames.
- Quantitative and Qualitative Results: On DAVIS16, COSNet gains 2.6% in J Mean and 4.9% in F Mean over the second-best PDB method.COSNet outperforms all reported methods across most DAVIS16 metrics.
- Quantitative and Qualitative Results: COSNet achieves 75.6% mean J on FBMS and outperforms compared methods on Youtube-Objects, including videos with deformation, appearance variation, clutter, and multiple moving targets.Unlike many competing methods, COSNet uses only RGB appearance information without optical flow.
5. Conclusion
COSNet treats unsupervised video object segmentation as temporal coherence capture, learning to distinguish primary objects from background across frames. It achieves superior performance on three representative video segmentation datasets and suppresses similar-target distraction without segmentation annotations.
- COSNet estimates primary targets by modeling UVOS as a temporal coherence capturing task.
- Alternating training with salient images and video pairs teaches COSNet both per-frame foreground discrimination and cross-frame temporal correlation.
- COSNet achieved superior performance on three representative video segmentation datasets.
- The method effectively suppresses similar target distraction without annotations during segmentation.
- COSNet is presented as a general sequential-data learning framework extensible to video saliency detection and optical flow estimation.