Source-linked AI summary
Crossover Learning for Fast Online Video Instance Segmentation
Shusheng Yang, Yuxin Fang, Xinggang Wang, Yu Li, Chen Fang, Ying Shan, Bin Feng, Wenyu Liu
TL;DR
Video instance segmentation must handle per-pixel instance labeling in challenging video sequences, making temporal context important. CrossVIS uses crossover learning and a global balanced instance embedding branch; it is reported to lead online VIS methods on three benchmarks while balancing latency and accuracy.
Problem
VIS requires per-pixel instance labeling across video sequences, while video brings challenges such as occlusion, disappearing objects, and many frames to process.
Source: §1 · p. 1Method
CrossVIS uses current-frame instance features to localize the same instances pixel-wise in other frames and adds a global balanced embedding branch for instance association.
Source: §1 · p. 2Results
Takeaways & Limitations
Crossover learning brings cost-free inference improvement, while the lightweight global balanced embedding can stabilize performance.
Source: §5 · p. 9The described crossover-learning setup assumes the same instance exists in both frames being considered.
Source: §3.2 · p. 4
Abstract
from arXiv · showhide
Modeling temporal visual context across frames is critical for video instance segmentation (VIS) and other video understanding tasks. In this paper, we propose a fast online VIS model named CrossVIS. For temporal information modeling in VIS, we present a novel crossover learning scheme that uses the instance feature in the current frame to pixel-wisely localize the same instance in other frames. Different from previous schemes, crossover learning does not require any additional network parameters for feature enhancement. By integrating with the instance segmentation loss, crossover learning enables efficient cross-frame instance-to-pixel relation learning and brings cost-free improvement during inference. Besides, a global balanced instance embedding branch is proposed for more accurate and more stable online instance association. We conduct extensive experiments on three challenging VIS benchmarks, \ie, YouTube-VIS-2019, OVIS, and YouTube-VIS-2021 to evaluate our methods. To our knowledge, CrossVIS achieves state-of-the-art performance among all online VIS methods and shows a decent trade-off between latency and accuracy. Code will be available to facilitate future research.
1. Introduction
CrossVIS targets the challenges of video instance segmentation with crossover learning and a global balanced instance embedding branch. It is reported to perform best among online VIS methods on three benchmarks while balancing latency and accuracy.
1. Introduction
- VIS requires per-pixel instance labeling across video sequences, and real-world use calls for accurate, robust, and fast performance.Source: §1 · p. 1
- VIS remains challenging because video brings many frames to process, heavy occlusion, disappearing objects, and unconventional object-to-camera poses.Source: §1 · p. 1
- Crossover learning uses a current-frame instance feature to pixel-wisely localize the same instance in other frames, without extra feature-enhancement network blocks or inference computation.It is integrated with the instance segmentation loss, learns many-to-many instance-pixel relations, and is end-to-end optimizable.Source: §1 · p. 2
- A global balanced instance embedding branch addresses instance association and yields better, more stable results than pair-wise identity mapping approaches.Source: §1 · p. 2
- CrossVIS is reported to perform best among online VIS methods on YouTube-VIS-2019, OVIS, and YouTube-VIS-2021, with a decent speed-accuracy trade-off.Source: §1 · p. 2
2. Related Work
Related work spans still-image instance segmentation and several approaches to video instance segmentation. CrossVIS builds on a fully convolutional still-image framework and is evaluated on three VIS benchmarks.
Still-image Instance Segmentation
- CrossVIS builds on fully convolutional CondInst, which dynamically generates instance-conditioned mask filters, and extends it to VIS.Source: §2 · p. 2
Video Instance Segmentation (VIS)
- Prior VIS approaches include pair-wise association, task-specific networks, mask propagation, 3D spatiotemporal segmentation, recurrent graph networks, and temporal-context feature refinement.MaskProp is described as accurate but high-latency; VisTR adopts a query-based end-to-end approach.Source: §2 · p. 2
- CrossVIS is evaluated on three VIS benchmarks and shows competitive performance; OVIS and YouTube-VIS-2021 are described as more challenging benchmarks proposed to advance the field.Source: §2 · p. 2
3. Method
CrossVIS combines crossover learning for cross-frame instance representation with global balanced embeddings for online instance association. It generates masks with instance-conditioned dynamic filters and trains detection, segmentation, crossover, and association jointly.
3.1. Mask Generation for Still-image
- A dynamic filter generated for each instance is convolved with mask features to produce its instance mask, using relative coordinates as a localization cue.The MaskHead has three convolutional layers and uses a sigmoid in its final layer for mask prediction.Source: §3.1 · p. 3
3.2. Crossover Learning
- Crossover learning uses an instance’s dynamic filter from one frame with the other frame’s mask features to predict that same instance’s mask in both directions.This cross-frame training encourages representations robust to appearance changes and background clutter, while inference keeps the still-image mask-generation process unchanged.Source: §3.2 · pp. 3–4
3.3. Learning Global Balanced Embeddings for Instance Association
- Pair-wise local embeddings depend on sampled reference frames, which can shift the feature space and make optimization and association unstable.The paper also reports relatively large AP fluctuation with pair-wise embeddings.Source: §3.3 · pp. 4–5
- The global embedding branch replaces sampled-instance embeddings with learnable instance-wise proxies trained as an M-class identity classification problem.For large identity sets, negative-class imbalance and gradients can dominate other subtasks, so the method adopts focal loss.Source: §3.3 · p. 5
3.4. Training and Online Inference
- Detection, segmentation, crossover learning, and instance association are trained jointly in an end-to-end multi-task objective.At test time, CrossVIS processes video frames online, one frame at a time.Source: §3.4 · p. 5
4. Experiments
Experiments evaluate CrossVIS on three VIS benchmarks and compare accuracy and speed; ablations examine its training setup, temporal learning, and instance-association components.
4.1. Dataset
- CrossVIS is evaluated on YouTube-VIS-2019, OVIS, and YouTube-VIS-2021; OVIS emphasizes object occlusion, while YouTube-VIS-2021 augments the 2019 dataset.Results use validation sets; the paper defines AP and AR following [72].Source: §4.1 · pp. 5–6
4.3. Main Results
- 36.3 AP and 39.8 FPS with ResNet-50 under multi-scale training; CrossVIS also achieves the best speed-accuracy trade-off among VIS approaches on YouTube-VIS-2019.Single-scale CrossVIS reaches 34.8 AP with ResNet-50 and 36.6 AP with ResNet-101; the paper reports both as best among online and near-online methods in Table 1.Source: Table 1 · p. 6, §4.3 · pp. 6–7
4.2. Implementation Details
- CrossVIS starts from the corresponding CondInst instance-segmentation model pretrained on COCO train2017 with a 1× schedule, then trains on VIS datasets for a 1× schedule (12 epochs).Frames are resized to 360 × 640 for single-scale training and inference; multi-scale training follows SipMask-VIS.Source: §4.2 · p. 6
4.3. Main Results
- On OVIS, CrossVIS reports 14.9 AP and 18.1 AP with the temporal feature calibration module; on YouTube-VIS-2021, it surpasses MaskTrack R-CNN and SipMask-VIS by a large margin.The OVIS comparison uses the same experimental conditions as the methods investigated in [51].Source: Table 2 · p. 7, §4.3 · p. 7, Table 3 · p. 7
4.4. Ablation Study
- CrossVIS is 0.2 AP_COCO mask lower but 2.9 AP_VIS higher than MaskTrack R-CNN with ResNet-101, and 2.7 AP_VIS higher than CondInst-VIS at the same AP_COCO mask.The authors attribute the VIS improvement mainly to crossover learning and global balanced instance embedding; crossover learning adds no inference parameters, and the embedding branch uses a lightweight fully convolutional design.Source: §4.4 · pp. 7–8
- Crossover learning brings up to 1.4 AP improvement at larger frame intervals and about 1 AP across T = 3 to T = ∞; gains are moderate at small intervals.The reported results are relatively insensitive to the choice of T.Source: §4.4 · p. 8, Table 6 · p. 8
- Global and balanced embedding together improve AP by 1.7, while global embeddings show smaller run-to-run AP variation; crossover learning and the embedding branch together improve AP by 2.7.Individually, crossover learning and global balanced embedding improve AP by 1.0 and 1.4, respectively, and their combined gain is larger than the sum of those individual gains.Source: §4.4 · p. 8, Table 7 · p. 8, Table 8 · p. 8
5. Conclusion
CrossVIS performs best among online VIS methods across three challenging benchmarks while balancing latency and accuracy. The paper attributes cost-free inference improvement to crossover learning and greater stability to its lightweight global balanced embedding.
5. Conclusion
- CrossVIS performs best among online video instance segmentation methods on three challenging benchmarks and offers a decent latency–accuracy trade-off.Source: §5 · p. 8
- Figure 4 compares qualitative VIS results from CrossVIS and MaskTrack R-CNN, with different mask colors marking different instances.Source: Figure 4 · p. 9
- Crossover learning brings cost-free improvement during inference, while the lightweight global balanced embedding helps stabilize performance.Source: §5 · p. 9
Appendix
Qualitative comparisons show CrossVIS segmenting and tracking instances more robustly under occlusion or overlap. Figures 4 and 5 compare its VIS results with MaskTrack R-CNN.
Appendix
- CrossVIS segments and tracks instances more robustly when they are occluded or overlap with one another.Source: Appendix · p. 9
- Figures 4 and 5 present qualitative VIS comparisons between CrossVIS and MaskTrack R-CNN.Source: Appendix · p. 9, Figure 5 · p. 10
- Different color masks in the comparison identify different instances.Source: Figure 5 · p. 10