Source-linked AI summary
Video Instance Segmentation using Inter-Frame Communication Transformers
Sukjun Hwang, Miran Heo, Seoung Wug Oh, Seon Joo Kim
TL;DR
VIS per-frame methods are practical but limited by video phenomena, while prior per-clip approaches require substantial computation and memory for frame-to-frame communication. IFC uses memory-token-based inter-frame communication to reduce this overhead, achieving state-of-the-art VIS performance with fast offline and near-online inference.
Problem
Per-frame methods struggle with occlusions and motion blur, while per-clip and full space-time transformer approaches incur heavy computation and memory costs.
Method
IFC decomposes frame processing and inter-frame communication, using concise memory tokens to exchange clip context while instance weights and matching support segmentation and tracking.
Results
IFC achieves the highest AP among compared online, near-online, and offline models while presenting the fastest runtime on YouTube-VIS 2019 validation.
Takeaways & Limitations
IFC provides a per-clip VIS baseline that combines inter-frame information sharing with fast offline and near-online inference.
Takeaways & Limitations
Full-video offline processing remains distinct from near-online use, and the decomposition can fail to achieve high accuracy in the compared large-window setting.
Abstract
from arXiv · showhide
We propose a novel end-to-end solution for video instance segmentation (VIS) based on transformers. Recently, the per-clip pipeline shows superior performance over per-frame methods leveraging richer information from multiple frames. However, previous per-clip models require heavy computation and memory usage to achieve frame-to-frame communications, limiting practicality. In this work, we propose Inter-frame Communication Transformers (IFC), which significantly reduces the overhead for information-passing between frames by efficiently encoding the context within the input clip. Specifically, we propose to utilize concise memory tokens as a mean of conveying information as well as summarizing each frame scene. The features of each frame are enriched and correlated with other frames through exchange of information between the precisely encoded memory tokens. We validate our method on the latest benchmark sets and achieved the state-of-the-art performance (AP 44.6 on YouTube-VIS 2019 val set using the offline inference) while having a considerably fast runtime (89.4 FPS). Our method can also be applied to near-online inference for processing a video in real-time with only a small delay. The code will be made available.
1 Introduction
VIS per-frame methods are practical but struggle with occlusions and motion blur, while per-clip methods handle these challenges at substantially higher computational cost. IFC addresses this efficiency problem by communicating between frames through concise memory tokens and reports fast, accurate inference.
- Per-frame VIS methods are easy to deploy online but have difficulty handling occlusions and motion blur.
- Per-clip methods incorporate multiple frames to address video challenges, but instance- and frame-dependent features increase computation.
- IFC reduces full space-time transformer computation by decomposing processing into Encode-Receive and Gather-Communicate transformers.
- Memory tokens exchange concise clip context between frames without expensive pixel-level space-time attention.
- IFC represents each instance with a unique convolutional weight applied across all frames to capture identity characteristics and support tracking.
- IFC optimizes segmentation and tracking around space-time mask IoU using bipartite matching for predictions and same-identity instances.
- 42.8% AP and 107.1 fps are reported for IFC with ResNet-50, alongside a speed-accuracy balance under near-online inference.
2 Related Work
VIS research moved from per-frame tracking extensions toward per-clip models that exploit multiple frames for accuracy. IFC extends transformer-based video modeling with memory-token communication while preserving frame-wise execution independence for efficiency.
- Per-frame VIS methods add tracking components to image instance segmentation models, while newer algorithms incorporate video characteristics.
- Per-clip models exploit multiple frames to handle motion blur and occlusions, but IFC targets this pipeline with faster predictions.
- Vision transformers use self-attention among input tokens, with classification tokens summarizing information about the overall input.
- IFC adapts memory tokens to video so frames execute independently while interim communications share information across the clip.
- The IFC framework encodes frame and memory tokens, communicates between frames through memory tokens, and decodes outputs into segmentation masks.
3 Method
The proposed per-clip architecture uses Inter-frame Communication Transformers to exchange information through compact memory tokens rather than full space-time attention. It combines frame-wise encoding, cross-frame memory communication, instance-specific convolutional weights, and mask-based matching to reduce computation while preserving clip-level context.
- Model architecture: IFC processes a video clip with a CNN backbone, inter-frame communication encoder, transformer decoder, and mask-generation heads.The backbone embeds frames independently; encoder outputs and object queries produce unique convolutional weights for each clip instance.
- Inter-Frame Communication Encoder: Memory tokens summarize frame scenes and provide the communication channel between otherwise mostly independent frame computations.Encode-Receive exchanges information between frame features and memory tokens, while Gather-Communicate exchanges the memory tokens across frames.
- Inter-Frame Communication Encoder: The encoder alternates Encode-Receive and Gather-Communicate phases so frames receive clip-level information without pixel-level attention across all space-time locations.Gathered memory tokens are grouped by memory index, processed across frames, and redistributed to their originating frames.
- Complexity: Unlike VisTR’s full space-time attention, IFC avoids computation that scales quadratically with the entire clip’s spatiotemporal token count.VisTR has complexity O(C^2(THW) + C(THW)^2), while IFC communicates through a small memory-token set.
- Complexity: With a small memory-token count such as M=8, Gather-Communicate adds negligible computation and Encode-Receive approaches the complexity of independent frame processing.The proposed complexity is O(C^2T(HW + M) + CT(HW + M)^2) for Encode-Receive and O(C^2TM + CT^2M) for Gather-Communicate.
- Instance decoding and matching: The model predicts fixed-size, instance-specific convolutional weights and applies them across all frames, while matching predictions to targets with mask-based similarity.Mask-based matching uses class probability and Dice similarity because box-based matching is weak under occlusion and disappear-and-reappear cases.
4 Experiments
IFC is evaluated on YouTube-VIS benchmarks for accuracy, speed, inference settings, and encoder design. The experiments show strong performance and practical efficiency, while ablations identify how communication, stride, and decomposition affect results.
- Main Results: IFC achieves the highest AP among online, near-online, and offline models on YouTube-VIS 2019 while providing the fastest runtime.Its efficiency is attributed to memory-token decomposition, an instance-agnostic spatial decoder, and GPU-friendly instance matching.
- Main Results: 107.1 FPS is achieved with offline inference, while near-online inference at T = 5 reaches 46.5 FPS with only a small delay.The clip length can be adjusted so the same method supports whole-video offline processing or streaming applications.
- Main Results: IFC achieves the best reported performance on YouTube-VIS 2021.The comparison uses results reported with official implementations for prior methods.
- Ablation Study: Increasing the stride from S = 1 to S = 3 provides approximately 150% speed improvement with only a 0.1% AP drop.Larger strides reduce temporal intersections and computational overhead but make instance matching more difficult.
- Ablation Study: Full self-attention becomes more accurate as clip length increases, but its speed and memory costs are not compensated by marginal improvements.Space-time decomposition maintains comparable speed for large windows but fails to achieve high accuracy, whereas IFC adds only small memory-token computations and benefits from larger windows.
- Ablation Study: Memory tokens improve accuracy over an encoder without communications by passing information between frames, with the default token count set to M = 8.They do not improve single-frame image instance segmentation because mutual frame communication is unavailable.
- Ablation Study: Decomposing memory tokens into groups improves accuracy over inserting all MT tokens together because same-index memories have stronger correspondences.The grouped formulation eases attention construction between corresponding memories.
- Ablation Study: Memory-token attention focuses on difficult instances, including a heavily occluded car, and supports tracking and segmentation under motion blur and appearance changes.The visualization reports successful rabbit tracking and segmentation in a clip with substantial motion blur.
5 Conclusion
The paper proposes IFC, a video instance segmentation network that communicates between frames without full space-time attention. It reports rapid inference and state-of-the-art performance on YouTube-VIS.
- IFC alleviates full space-time attention while building communications between video frames.
- The network achieves rapid inference and sets a new state-of-the-art on the YouTube-VIS dataset.
Broader Impact
IFC is designed to reduce computation while improving performance for video instance segmentation. The authors identify potential relevance for industrial applications requiring both high accuracy and speed.
- IFC targets reduced overall computation while improving performance for video instance segmentation.
- The authors suggest IFC could benefit industrial fields requiring high accuracy and speed, including alert systems.