PaperlayerView exact revision on arXiv

Source-linked AI summary

K-Net: Towards Unified Image Segmentation

Wenwei Zhang, Jiangmiao Pang, Kai Chen, Chen Change Loy

arXiv:2106.14855v2cs.CVcs.AI

TL;DR

Existing segmentation tasks use specialized frameworks despite their shared grouping structure. K-Net unifies them with learnable kernels, dynamically updated from image groups and trained through bipartite matching. It reports state-of-the-art panoptic and semantic results and instance performance comparable to Cascade Mask R-CNN with faster inference.

  • Problem

    Semantic, instance, and panoptic segmentation have been addressed with different specialized frameworks despite their underlying connections.

  • Method

    K-Net uses learnable semantic or instance kernels, iteratively updates them from their corresponding pixel groups, and assigns targets with bipartite matching.

  • Results

    K-Net reports new state-of-the-art single-model performance on panoptic and semantic segmentation, while matching Cascade Mask R-CNN's instance performance with 92.2% faster inference.

  • Takeaways & Limitations

    A common kernel-based framework tackles semantic, instance, and panoptic segmentation without bounding boxes or NMS.

Abstract

from arXiv · show

Semantic, instance, and panoptic segmentations have been addressed using different and specialized frameworks despite their underlying connections. This paper presents a unified, simple, and effective framework for these essentially similar tasks. The framework, named K-Net, segments both instances and semantic categories consistently by a group of learnable kernels, where each kernel is responsible for generating a mask for either a potential instance or a stuff class. To remedy the difficulties of distinguishing various instances, we propose a kernel update strategy that enables each kernel dynamic and conditional on its meaningful group in the input image. K-Net can be trained in an end-to-end manner with bipartite matching, and its training and inference are naturally NMS-free and box-free. Without bells and whistles, K-Net surpasses all previous published state-of-the-art single-model results of panoptic segmentation on MS COCO test-dev split and semantic segmentation on ADE20K val split with 55.2% PQ and 54.3% mIoU, respectively. Its instance segmentation performance is also on par with Cascade Mask R-CNN on MS COCO with 60%-90% faster inference speeds. Code and models will be released at https://github.com/ZwwWayne/K-Net/.

1 Introduction

K-Net unifies semantic, instance, and panoptic segmentation through learnable kernels, replacing task-specific components with a common mask-generation framework. Dynamic kernel updates and bipartite matching support instance separation and box-free, NMS-free training and inference.

  • K-Net uses learnable convolutional kernels to generate masks for semantic categories or potential instances, with their combination supporting panoptic segmentation.Kernels convolve image features to produce segmentation predictions.
  • Dynamic kernel updates condition each kernel on its activated image group, improving discrimination among varying instances.The update strategy is applied iteratively to strengthen kernel discriminative ability.
  • Bipartite matching assigns one-to-one learning targets to kernels and handles varying numbers of instances without bounding boxes.The mask-driven approach makes K-Net naturally NMS-free and box-free.
  • 54.6% PQ establishes K-Net as a new state of the art on the reported panoptic benchmark, while semantic segmentation reaches 54.3% mIoU.These results are reported without additional bells and whistles.

2 Related Work

Prior segmentation systems specialize by task, often using detection, grouping, or post-processing components. K-Net instead applies dynamic, learnable kernels to unify mask prediction while avoiding duplicated-mask handling used by several earlier dynamic-kernel methods.

  • Semantic Segmentation: Semantic segmentation commonly treats pixels as a dense classification problem using static kernels for predefined categories.Related approaches vary in architecture, including fully convolutional networks and vision-transformer sequence prediction.
  • Instance Segmentation: Top-down instance methods detect boxes before masks, whereas bottom-up methods group semantically segmented pixels into instances.Bottom-up approaches generally require a grouping process and often perform worse on popular benchmarks.
  • Instance Segmentation: One-stage instance methods avoid detection or embedding generation but still rely on NMS to remove duplicated masks, hindering end-to-end training.Their heuristic post-processing is also unfavorable for real-time applications.
  • Panoptic Segmentation: Panoptic segmentation combines instance and semantic segmentation, with prior systems commonly adding a semantic branch or pixel-grouping strategy to an instance or semantic framework.K-Net instead lets one kernel predict one instance or one semantic category and progressively refines masks and kernels.
  • Dynamic Kernels: Earlier dynamic-kernel methods generate dense-position kernels and require NMS or kernel fusion to remove duplicated objects or masks.K-Net uses learnable kernels updated by their corresponding groups instead of kernels generated from dense grids.

3 Methodology

K-Net unifies semantic, instance, and panoptic segmentation with kernels that represent meaningful pixel groups, then improves them through group-aware updates and interaction. The framework achieves strong cross-task results while remaining box-free and NMS-free, with ablations showing the value of its components and iterative refinement.

  • Unified kernel formulation: K-Net assigns each pixel to either a potential instance or semantic class using a unified set of kernels.Each instance kernel segments at most one object, enabling instance separation and segmentation simultaneously in one pass.
  • Group-aware kernels: The kernel update head assembles group features, adaptively updates kernels according to their pixel groups, and models interactions among kernels.Kernel interaction uses multi-head attention followed by a feed-forward network.
  • Panoptic segmentation: K-Net surpasses prior panoptic segmentation methods by 1.7 and 1.5 PQ on the COCO validation split against box-based and box/NMS-free baselines, respectively.With Swin Transformer, it obtains new state-of-the-art single-model performance without bells and whistles.
  • Instance segmentation: K-Net-N256 matches Cascade Mask R-CNN while achieving 92.2% faster inference at 19.8 versus 10.3 FPS.On COCO test-dev, K-Net also exceeds Cascade Mask R-CNN by 0.7 AP and 6 FPS with 256 instance kernels.
  • Semantic segmentation: K-Net improves semantic segmentation across architectures, including a 6.6 mIoU gain for FCN and a 1.4 mIoU gain over Swin Transformer UperNet.With 640×640 inputs, K-Net with UperNet and Swin Transformer is 0.8 mIoU above the previous single-model state of the art.
  • Ablations: Ablations find adaptive kernel update and kernel interaction necessary for high performance, while performance saturates beyond three or four update stages.More instance kernels improve capacity on complicated images, but results saturate at N = 300, 512, and 768 with 34.9% mAP.
  • Kernel analysis: Learned instance kernels specialize in different image regions and object sizes, indicating that the kernels capture meaningful visual group structure.The analysis averages mask activations from 100 instance kernels over 5000 validation images.

5 Conclusion

K-Net unifies semantic, instance, and panoptic segmentation through learned kernels that iteratively refine image partitions. It achieves state-of-the-art panoptic and semantic results while reporting the fastest inference speed among recent instance-segmentation frameworks.

  • K-Net uses learned static kernels to partition images, then iteratively refines kernels and their group assignments using assembled group features.The framework replaces separate instance-segmentation components with instance kernels that learn to separate objects.
  • K-Net replaces bounding boxes, embedding generation, NMS, kernel fusion, and pixel grouping with instance kernels for instance separation.
  • K-Net provides a unified framework for semantic, instance, and panoptic segmentation through the shared notion of kernels.
  • K-Net obtains new state-of-the-art single-model performance on panoptic and semantic segmentation benchmarks.
  • K-Net surpasses Cascade Mask R-CNN with the fastest inference speed among recent instance-segmentation frameworks.

Appendix

The appendix provides implementation details, benchmark comparisons, kernel-update and failure-case analyses, and a broader-impact discussion.

  • Appendix A: Appendix A provides implementation details for K-Net’s three segmentation tasks.
  • Appendix B: Appendix B provides benchmark details and discusses comparisons between K-Net and other methods.
  • Appendix C: Appendix C analyzes K-Net’s kernel-update results and failure cases.
  • Appendix D: Appendix D discusses K-Net’s broader impact.

A Implementation Details

For semantic segmentation, K-Net is implemented in MMSegmentation and trained with AdamW under a specified 80,000-iteration schedule.

  • K-Net’s semantic-segmentation implementation is based on MMSegmentation.
  • Training uses AdamW with weight decay 0.0005 for 80000 iterations by default.
  • The initial learning rate is 0.0001 and is decayed by 0.1 after 60000 and 72000 iterations.

B.1 Instance Segmentation

The instance-segmentation comparison evaluates K-Net against several established methods using reported accuracy and measured inference speed. Speed is benchmarked on an NVIDIA V100 with pure model inference averaged across repeated runs.

  • Table 2 compares K-Net’s accuracy and inference speed with Mask R-CNN, Cascade Mask R-CNN, SOLO, SOLOv2, and CondInst.Mask R-CNN and Cascade Mask R-CNN were re-implemented with the multi-scale 3× training schedule, while other models used official implementations.
  • Inference speed is measured in frames per second on an NVIDIA V100 GPU without data-loading time.
  • Reported FPS is averaged across three runs, with each run measuring 400 iterations.
  • The speed measurements are presented to verify that K-Net is fast and effective, while implementation-specific optimizations may change them.

B.2 Semantic Segmentation

The section reports semantic-segmentation performance for K-Net under test-time augmentation and larger-scale training settings. These evaluations compare its mIoU with a Swin Transformer state-of-the-art result.

  • 53.3 mIoU is obtained by applying test-time augmentation to the 52.0 mIoU K-Net model.
  • 53.3 mIoU is on par with the current 53.5 mIoU state-of-the-art result.
  • Figure A1 reports PQ for each stage while visualizing mask changes before and after kernel updates.

C Visual Analysis

The visual analysis shows that iterative kernel updates refine mask predictions, while K-Net’s main failure modes involve ambiguous textures and crowded scenes. These cases expose limitations in boundary discrimination and instance capacity.

  • Masks Refined through Kernel Update: Kernel updates progressively refine mask boundaries and fill empty holes in large masks.Static kernels struggle with mask boundaries and may fail to cover the whole image.
  • Masks Refined through Kernel Update: Mask predictions after the second and third update rounds look very similar, indicating limited visible change between those rounds.
  • Failure Cases: Similar texture appearances can cause inaccurate mask boundaries and misclassification of contents.
  • Failure Cases: Crowded scenes challenge K-Net’s ability to recognize and segment all instances with a limited number of instance kernels.

D Broader Impact

The paper presents K-Net as a unified segmentation approach intended to support simpler deployment across several real-world applications. It also states that application safety is not guaranteed because robustness to corrupted images and adversarial attacks was not evaluated.

  • K-Net targets consistent treatment of semantic, instance, and panoptic segmentation through a unified perspective.
  • The authors identify autonomous driving, robotics, and mobile phones as potential real-world application areas.
  • Robustness to corrupted images and adversarial attacks was not evaluated, so application safety is not guaranteed.
Loading 2106.14855v2…