PaperlayerView exact revision on arXiv

Source-linked AI summary

Video K-Net: A Simple, Strong, and Unified Baseline for Video Segmentation

Xiangtai Li, Wenwei Zhang, Jiangmiao Pang, Kai Chen, Guangliang Cheng, Yunhai Tong, Chen Change Loy

arXiv:2204.04656v2cs.CV

TL;DR

Video panoptic segmentation combines pixel-level segmentation with tracking, while existing video pipelines can be complex. Video K-Net uses K-Net kernels for cross-frame association and kernel interaction, achieving state-of-the-art results across three video panoptic segmentation datasets and improvements on video semantic and instance segmentation.

  • Problem

    Video panoptic segmentation requires segmenting and tracking every pixel across video clips, while existing approaches motivate simpler unified solutions.

    Source: §1 · p. 1

  • Method

    Video K-Net uses learnable K-Net kernels to associate objects across frames, with kernel embeddings, linking, and temporal fusion for segmentation and tracking.

    Source: §1 · p. 2

  • Results

    Video K-Net achieves state-of-the-art results on KITTI-STEP, Cityscapes-VPS, and VIPSeg, boosts video semantic segmentation baselines, and reaches 40.5% AP on VIS with ResNet50, 4.0% above VisTR.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The experiments support Video K-Net as a baseline for unified video segmentation research.

    Source: §1 · p. 2

    Training and inference explore only one frame, a setting chosen mainly for fair comparison with other works.

    Source: §4.4 · p. 9

Abstract

from arXiv · show

This paper presents Video K-Net, a simple, strong, and unified framework for fully end-to-end video panoptic segmentation. The method is built upon K-Net, a method that unifies image segmentation via a group of learnable kernels. We observe that these learnable kernels from K-Net, which encode object appearances and contexts, can naturally associate identical instances across video frames. Motivated by this observation, Video K-Net learns to simultaneously segment and track "things" and "stuff" in a video with simple kernel-based appearance modeling and cross-temporal kernel interaction. Despite the simplicity, it achieves state-of-the-art video panoptic segmentation results on Citscapes-VPS, KITTI-STEP, and VIPSeg without bells and whistles. In particular, on KITTI-STEP, the simple method can boost almost 12\% relative improvements over previous methods. On VIPSeg, Video K-Net boosts almost 15\% relative improvements and results in 39.8 % VPQ. We also validate its generalization on video semantic segmentation, where we boost various baselines by 2\% on the VSPW dataset. Moreover, we extend K-Net into clip-level video framework for video instance segmentation, where we obtain 40.5% mAP for ResNet50 backbone and 54.1% mAP for Swin-base on YouTube-2019 validation set. We hope this simple, yet effective method can serve as a new, flexible baseline in unified video segmentation design. Both code and models are released at https://github.com/lxtGH/Video-K-Net.

1. Introduction

Video K-Net addresses the complexity of video panoptic segmentation by reusing K-Net’s learnable kernels to associate objects across frames. It links and fuses kernels for segmentation and tracking in an end-to-end framework.

1. Introduction

  • Top-down systems handle semantic, instance, and tracking tasks with separate modules, while bottom-up systems still rely on NMS grouping and offline mask tracking.The paper frames both designs as cumbersome pipelines to simplify.Source: §1 · p. 1
  • On Cityscapes-VPS, Video K-Net has higher accuracy with lower model complexity; versus Motion-Deeplab, it gains 10% with a few extra GFLOPS.Figure 2 compares accuracy and GFLOPS at original KITTI-STEP and Cityscapes-VPS image inputs; circle radius represents parameter count.Source: Figure 2 · p. 2
  • K-Net kernels encode appearance and context, allowing Video K-Net to associate identical objects across video without extra tracking queries.The framework uses a modified contrastive loss for kernel association embeddings, links kernels between adjacent frames, and fuses kernels with corresponding features for temporal mask prediction.Source: §1 · p. 2
  • Kernel-level fusion avoids feature-level computation and RoI features, supports online tracking, and adds 2.3% GFLOPs over K-Net.The method formulates video panoptic segmentation as a kernel-linking problem.Source: §1 · p. 2
  • 3.5% STQ on KITTI-STEP and 4% VPQ on Cityscapes-VPS over K-Net; nearly 12% relative improvement over Motion-Deeplab and almost 16% over CLIP-PanoFCN on VIPSeg.The method achieves state-of-the-art results on KITTI-STEP, Cityscapes-VPS, and VIPSeg.Source: §1 · p. 2
  • For video instance segmentation, the clip-level extension reaches 40.5% AP with ResNet50, a 4.0% improvement over VisTR with fewer GFLOPs.The extension adds a temporal kernel fusion module.Source: §1 · p. 2

2. Related Work

Related work describes panoptic segmentation and its video extensions, alongside query-based unification and tracking approaches. Video K-Net builds on kernels to associate instances across frames without complex post-grouping.

2. Related Work

  • Video panoptic segmentation combines panoptic masks with instance tracking IDs across clips; prior work includes short-term Cityscapes-VPS and long-term STEP settings.The cited work uses VPQ for Cityscapes-VPS and STQ for STEP.Source: §2 · p. 3
  • The toy-experiment table covers KITTI-STEP and Cityscapes-VPS using STQ and VPQ; Unitrack uses a ResNet50 appearance model.Source: Table 1 · p. 3
  • Where ViP-DeepLab clusters instance pixels for tracking, Video K-Net links kernels into association embeddings and avoids complex post-grouping.The paper also reviews tracking-by-detection and query-based tracking approaches.Source: §2 · p. 3
  • In the toy video experiment, instances in multiple frames originate from the same K-Net kernel predictions, motivating direct use of K-Net on video.The illustration identifies repeated predictions including Kernel-2 and Kernel-3.Source: Figure 3 · p. 3

3. Method

Video K-Net extends K-Net’s learnable mask kernels to video, using their observed capacity to associate instances across frames as motivation. It adds kernel embeddings, temporal linking, and fusion to support video segmentation and tracking.

3.1. Using K-Net on Video

  • K-Net uses learnable kernels to generate thing or stuff masks, iteratively refining them through feature updates and kernel interactions before mask classification.Source: §3.1 · pp. 3–4
  • Toy video experiments show that instance masks correspond to specific kernels and that several kernels remain linked across frames, motivating kernel-based tracking.Source: §3.1 · p. 4
  • K-Net’s learned kernels already supported tracking association, outperforming Unitrack despite Unitrack’s extra appearance network and motion prediction module.Source: §3.1 · p. 4

3.2. Extending K-Net into Video

  • Kernel association embeddings use mask-IoU assignments and train only on kernels matched to ground-truth masks, avoiding noise from inaccurate unmatched kernels.The positive and negative IoU thresholds are 0.7 and 0.3, respectively.Source: §3.2 · p. 5
  • Video K-Net adds temporal kernel linking and fusion; kernel updating before fusion is essential because direct fusion performs poorly under large motion and scale variation.Source: §3.2 · pp. 4–5

3.3. Video K-Net Architecture

  • The architecture shares key network components across frames, tracks preserved instance masks online, and extends to clip-level video instance segmentation with three kernel-fusion layers.For clip-level video instance segmentation, mean temporal kernels represent objects and kernel indices assign instance IDs across frames.Source: §3.3 · pp. 5–6

4. Experiment

Experiments evaluate Video K-Net across video panoptic, semantic, and instance segmentation tasks, with benchmark comparisons, component ablations, and analysis of efficiency and scope.

4.1. Experiment Setup

  • Experiments cover KITTI-STEP, Cityscapes-VPS, VSPW, VIPSeg, and YouTube-VIS; VPS comparisons report VPQ and STQ, while VSS reports mIoU and mVC.STQ is the geometric mean of Segmentation Quality and Association Quality.Source: §4.1 · p. 6

4.2. Main Results

  • 0.71 STQ and 0.46 VPQ with ResNet50 on KITTI-STEP establish new state-of-the-art results; a stronger backbone adds about 3% STQ and 7% VPQ.Against Motion-Deeplab, Video K-Net improves by 13% on validation and 7% on test.Source: Table 2 · p. 6, §4.2 · p. 6
  • 62.2% VPQ is reported on Cityscapes-VPS after following ViP-Deeplab's pretraining procedure; ResNet50 also outperforms VPSNet by 0.3%.Kernel fusion is not used on this dataset because of its low frame rate.Source: Table 3 · p. 7, §4.2 · p. 7
  • On VSPW, Video K-Net boosts Deeplabv3+ and PSPNet by 2–3% mIoU and 3–4% mVC; on YouTube-VIS, it beats VisTR by 4% with 12 versus 300 training epochs.All methods in the VSPW comparison use the same setting.Source: Table 4 · p. 7, Table 5 · p. 7, §4.2 · p. 7
  • 16.6% VPQ and 14.8% STQ improvements over previous work give the Swin-base model new state-of-the-art results on VIPSeg.With ResNet50, Video K-Net exceeds Clip-PanoFCN by 3% VPQ and 1.6% STQ using 12 training epochs.Source: Table 6 · p. 7, §4.2 · p. 7

4.3. Ablation Study

  • Adding KAE yields 2.8% STQ improvement, kernel linking adds 1.0% AQ, and kernel fusion adds 0.7% STQ and 1.4% SQ.KAE’s gains are especially pronounced on AQ, at about 2.4%; AQ decreases slightly after fusion.Source: §4.3 · p. 7, Table 7 · p. 8
  • Matched-kernel training avoids sample-noise SQ loss; last-stage linking beats early fusion, joint key/reference training helps, and kernel updating matters in fusion.Adding more association samples lowers SQ because many kernels do not generate masks under Hungarian assignment.Source: §4.3 · p. 8

4.4. Visualization and More Analysis

  • Figures 5–6 visualize tracking and segmentation consistency; the method adds 2.3% GFLOPs and 1.8% parameters, but trains and infers on one frame.The authors identify multi-frame kernel use and long-video handling as future work.Source: Figure 5 · p. 8, Figure 6 · p. 8, §4.4 · pp. 8–9

5. Conclusion

Video K-Net is a simple, end-to-end system for video panoptic segmentation that also generalizes to video semantic and instance segmentation.

5. Conclusion

  • Video K-Net achieves state-of-the-art results on Cityscapes-VPS, KITTI-STEP, and VIPSeg.The conclusion reports improvements over previous methods of over 10% on KITTI-STEP and Cityscapes-VPS.Source: §5 · p. 9
  • The system learns kernel association embeddings from thing kernels, then links and fuses kernels to improve tracking and segmentation.Source: §5 · p. 9
  • The method has a simpler pipeline and less computation cost than previous works, and shows generalization on VSPW and competitive results on YouTube-VIS.Source: §5 · p. 9

6. More Experimental Details

The experimental details specify pretraining and dataset-specific training and inference settings for the evaluated video segmentation tasks.

Pretraining

  • COCO pretraining follows K-Net settings for 36 epochs, while Mapillary pretraining follows Panoptic-DeepLab settings for 240 epochs.Mapillary pretraining is used for the largest Swin-base models for fair comparison.Source: §6 · p. 12

Cityscapes-VPS

  • Cityscapes-VPS training uses a nearby reference frame, multi-scale crops, and eight epochs; inference propagates kernel information online from the previous frame.Swin-base training uses 800 × 1600 crops to save memory and computation.Source: §6 · p. 12

KITTI-STEP

  • KITTI-STEP training samples a reference frame from the nearby three, uses 384 × 1248 crops, and runs for 12 epochs.Cityscapes pretraining before STEP training gives the K-Net baseline about 3% STQ gain.Source: §6 · p. 12

VSPW

  • VSPW training samples a frame from the nearest three, and uses the Cityscapes-VPS inference procedure.Source: §6 · p. 12

VIPSeg and YouTubeVIS-2019

  • VIPSeg uses COCO-pretrained K-Net, 720 × 720 crops, and 12 training epochs; YouTubeVIS-2019 jointly trains on five sampled frames for 12 epochs.YouTubeVIS-2019 follows the VisTR training pipeline and uses COCO-pretrained initialization.Source: §6 · p. 12

7. More Visualization Results

The visual examples show Video K-Net segmenting and tracking objects across Cityscapes-VPS and KITTI-STEP clips, while the failure analysis identifies tracking and segmentation errors.

More Visualization on Cityscapes-VPS

  • Video K-Net segments and tracks objects across varied object scales in the Cityscapes-VPS examples, compared with ground truth.The examples use the Swin-base model.Source: §7 · p. 12, Figure 7 · p. 13

More Visualization on KITTI-STEP

Failure Cases Analysis

  • The STEP failure examples include a fast-moving distant car ID switch and two segmentation errors involving car-window color and a rare bike-on-truck case.The authors suggest motion cues may improve the tracking case.Source: §7 · p. 12, Figure 9 · p. 15

Border Impact

  • Because of limited dataset size, the authors do not evaluate robustness to corrupted or extremely long video inputs.Source: §7 · p. 12
Loading 2204.04656v2…