Source-linked AI summary
Zero-Shot Video Object Segmentation via Attentive Graph Neural Networks
Wenguan Wang, Xiankai Lu, Jianbing Shen, David Crandall, Ling Shao
TL;DR
Zero-shot video object segmentation must identify foreground objects without annotations for testing frames while handling rich relationships across video frames. AGNN addresses this with a fully connected attentive graph and recursive message passing, and the paper reports favorable results across three ZVOS and two IOCS datasets. The findings support using AGNN to capture diverse relations in videos and related images.
Problem
Zero-shot video object segmentation must identify foreground objects without prior knowledge of their identity or annotations for testing frames.
Method
AGNN models frames as nodes in a fully connected graph, uses differentiable attention for frame-pair edges, and recursively passes messages to mine higher-order relations.
Results
AGNN performs favorably against current state-of-the-art methods across three ZVOS and two IOCS datasets.
Takeaways & Limitations
AGNN captures diverse relationships among similar video frames and semantically related images for segmentation tasks.
Takeaways & Limitations
Prior methods are constrained by local or sequential relations under occlusion, scale variation, and appearance changes.
Abstract
from arXiv · showhide
This work proposes a novel attentive graph neural network (AGNN) for zero-shot video object segmentation (ZVOS). The suggested AGNN recasts this task as a process of iterative information fusion over video graphs. Specifically, AGNN builds a fully connected graph to efficiently represent frames as nodes, and relations between arbitrary frame pairs as edges. The underlying pair-wise relations are described by a differentiable attention mechanism. Through parametric message passing, AGNN is able to efficiently capture and mine much richer and higher-order relations between video frames, thus enabling a more complete understanding of video content and more accurate foreground estimation. Experimental results on three video segmentation datasets show that AGNN sets a new state-of-the-art in each case. To further demonstrate the generalizability of our framework, we extend AGNN to an additional task: image object co-segmentation (IOCS). We perform experiments on two famous IOCS datasets and observe again the superiority of our AGNN model. The extensive experiments verify that AGNN is able to learn the underlying semantic/appearance relationships among video frames or related images, and discover the common objects.
1. Introduction
AGNN addresses the difficulty of identifying foreground objects without test-frame annotations by modeling video frames and their relationships globally. Its evaluations cover three ZVOS datasets and extend the framework to image object co-segmentation.
- Motivation: Zero-shot video object segmentation seeks primary objects without prior knowledge of foreground identity or test-frame annotations.The task is motivated by reducing manual effort in video processing and understanding.
- Limitations of prior methods: Existing methods often emphasize local pair-wise or sequential frame relations, overlooking higher-order relationships across correlated frames.This limitation can be especially problematic under occlusion, scale variation, and appearance changes.
- Proposed framework: AGNN represents frames as nodes in a fully connected graph and frame-pair relations as attention-described edges.The differentiable attention mechanism avoids time-consuming optical-flow estimation.
- Proposed framework: Recursive message passing propagates information among nodes to capture higher-order frame relationships and produce foreground estimates from a global view.The framework preserves spatial information for pixel-wise prediction.
- Evaluation and extension: AGNN is evaluated on DAVIS16, Youtube-Objects, and DAVIS17, with reported superior performance over current state-of-the-art methods.The framework is also applied to image object co-segmentation and reports promising results on PASCAL VOC and Internet benchmarks.
- Evaluation and extension: Experiments indicate that AGNN captures relationships among correlated video frames and mines semantics among semantically related static images.The conclusion presents this as an effective solution for video object segmentation and a generalizable framework for related images.
2. Related Work
Prior work includes graph neural networks for structured relationships, handcrafted and deep-learning approaches to video object segmentation, and relational methods for image co-segmentation. AGNN differs by combining spatial preservation, differentiable attention, and iterative high-order relation mining in a unified model.
- 2.1. Graph Neural Networks: Graph neural networks model relationships over structured data through parameterized message passing among nodes.Prior formulations use recurrent or learnable neural-network modules for node states and messages.
- 2.1. Graph Neural Networks: AGNN is a message-passing GNN whose nodes, edges, and message-passing functions are parameterized by neural networks.The paper identifies spatial information preservation and differentiable attention as key differences from conventional fully connected GNNs.
- 2.2. Automatic Video Object Segmentation: Conventional video object segmentation methods use handcrafted cues, heuristic foreground assumptions, trajectories, object proposals, or unsupervised representations.These approaches include objectness, motion, saliency, color, optical flow, and dense point trajectories.
- 2.2. Automatic Video Object Segmentation: Deep-learning video segmentation methods commonly combine appearance and motion with two-stream networks or model temporal dynamics using LSTM variants.These methods support zero-shot solutions through learned video-object features.
- 2.2. Automatic Video Object Segmentation: AGNN provides an end-to-end graph solution that iteratively fuses messages, mines diverse high-order video relations, and uses differentiable attention for frame-pair correlations.The paper presents this combination as distinct from previous automatic video object segmentation methods.
- 2.3. Image Object Co-segmentation: Image object co-segmentation jointly segments common objects in related images using intra-image and inter-image cues.Prior methods use energy functions, scene matching, global appearance, clustering, and manifold ranking.
3. Our Algorithm
AGNN formulates zero-shot video object segmentation as iterative message passing over a fully connected graph of video frames. It combines spatially preserving node embeddings, attention-based edge representations, gated aggregation, recurrent state updates, and readout to model higher-order frame relations.
- Graph formulation: Generic GNN message passing recursively aggregates neighbor information and updates node states for K iterations before mapping final states to node outputs.After k iterations, each state captures relations within the node’s k-hop neighborhood.
- Graph formulation: AGNN represents each video frame as a graph node and every frame relation, including self-connections, as a directed edge in a fully connected graph.Loop-edges connect a node to itself, while line-edges connect different nodes.
- Node and edge representations: AGNN preserves spatial and semantic information by using DeepLabV3-based fully convolutional node embeddings with W × H resolution and C channels.These embeddings serve as the initial representations for video-frame nodes.
- Node and edge representations: Intra-attention models contextual dependencies within each node through loop-edge embeddings, while inter-attention captures bidirectional relations between distinct node embeddings.The inter-attention edge representation reflects the importance of one node’s embedding to another.
- Message aggregation: Edge-weighted neighbor messages are aggregated with a learnable gate that filters irrelevant information from noisy frames caused by camera shifts or out-of-view content.The gate measures message confidence before state updating.
- State update and prediction: ConvGRU updates each node state after message aggregation, and a readout function converts final states into segmentation prediction maps.AGNN performs K propagation iterations to mine higher-order relations and estimate foregrounds from a global view.
- Training: Training batches jointly sample multiple frames, representing their relationships in an N′-node graph and increasing combinations that support data augmentation and generalizability.The reported training setup samples three frames per video and uses K = 3 iterations under computational constraints.
4. Experiments
The experiments evaluate AGNN first on unsupervised video object segmentation, then on image object co-segmentation, and finally through ablation analysis.
- The study evaluates AGNN on unsupervised video object segmentation.
- The study extends AGNN to image object co-segmentation to demonstrate its advantages.
- An ablation study investigates AGNN’s essential components.
4.1. Main Task: ZVOS
AGNN is evaluated on three video object segmentation datasets using alternating static-image and dynamic-video training, with instance proposals for DAVIS17. It outperforms reported methods quantitatively and handles challenging visual changes qualitatively.
- Experimental Setup: AGNN is evaluated on DAVIS16, Youtube-Objects, and DAVIS17 using region, boundary, stability, or mean J metrics.
- Experimental Setup: Training alternates static-image supervision with dynamic-video training, while DAVIS17 inference uses category-agnostic Mask R-CNN proposals.
- Quantitative Performance: 80.7 mean J and 79.1 mean F outperform AGS’s 79.7 mean J and 77.4 mean F on DAVIS16.
- Quantitative Performance: AGNN performs favorably on Youtube-Objects by mean J and significantly outperforms RVOS over most criteria on DAVIS17 test-dev.
- Qualitative Performance: On soapbox and judo, AGNN produces accurate foreground segments despite scale variation, deformation, view changes, similar appearances, and rapid motions.
4.2. Additional Task: IOCS
AGNN is extended to image object co-segmentation, where it uses information from image groups to infer common objects. It achieves superior results on PASCAL VOC and Internet and handles substantial intra-class variation.
- Additional Task: IOCS jointly segments common objects from semantically related images, providing a second setting for AGNN’s high-order relation modeling.
- Experimental Setup: Experiments use PASCAL VOC and Internet, with mean J as the performance measure.
- Experimental Setup: During IOCS inference, AGNN processes image groups so each target image can leverage information from the whole group.
- Quantitative Performance: AGNN achieves the best PASCAL VOC performance because it uses high-order information from multiple images.
- Quantitative Performance: 4.5% performance gain over the second-best method on Internet accompanies AGNN results of 84.0% for Car, 72.6% for Horse, and 76.1% for Airplane.
- Qualitative Results: AGNN successfully detects common objects under background clutter and substantial intra-class appearance variation.
4.3. Ablation Study
The ablation study on DAVIS16 examines AGNN’s overall contribution and its message-passing design. AGNN substantially improves the backbone, while gated aggregation and additional iterations affect performance.
- AGNN improves mean J from 72.2 to 80.6 compared with the DeepLabV3 backbone without AGNN.
- Gated Message Aggregation Strategy: Removing gated message aggregation degrades performance, indicating that channel-wise gates reduce irrelevant-frame influence.
- Message Passing Iterations K: Increasing message-passing iterations from 1 to 3 is evaluated as a factor affecting performance.
5. Conclusion
AGNN captures relationships among video frames to infer common foreground objects and extends this framework to image object co-segmentation. Experiments across three ZVOS and two IOCS datasets show favorable performance against current state-of-the-art methods.
- AGNN uses attention and recursive message passing to capture node similarity and mine higher-order correlations among video frames.
- The framework extends beyond ZVOS to image object co-segmentation, demonstrating its generalizability across related visual tasks.
- Experiments on three ZVOS and two IOCS datasets show AGNN performs favorably against current state-of-the-art methods.