PaperlayerView exact revision on arXiv

Source-linked AI summary

Instances as Queries

Yuxin Fang, Shusheng Yang, Xinggang Wang, Yu Li, Chen Fang, Ying Shan, Bin Feng, Wenyu Liu

arXiv:2105.01928v3cs.CV

TL;DR

QueryInst addresses the open problem of extending query-based detection to instance segmentation with parallel supervision on dynamic mask heads. It uses query correspondences across stages and between mask features and queries, outperforming established non-query methods while maintaining higher speed.

  • Problem

    Extending query-based detection frameworks to instance segmentation remains an open problem despite their comparable object-detection performance.

  • Method

    QueryInst uses parallel dynamic mask heads, exploiting one-to-one correspondences across stages and between mask RoI features and object queries.

  • Results

    QueryInst outperforms previous state-of-the-art non-query instance-segmentation approaches in accuracy and speed, including 48.1 APbox and 42.8 APmask on COCO test-dev, 2 points above HTC and 2.4× faster.

  • Takeaways & Limitations

    Parallel mask supervision improves performance without a corresponding inference-speed decline, while shared queries and multi-head self-attention jointly support detection and segmentation.

  • Takeaways & Limitations

    The vanilla query-based mask design is less effective than Cascade Mask R-CNN, and explicit mask flow causes large training and inference-speed drops.

Abstract

from arXiv · show

Recently, query based object detection frameworks achieve comparable performance with previous state-of-the-art object detectors. However, how to fully leverage such frameworks to perform instance segmentation remains an open problem. In this paper, we present QueryInst (Instances as Queries), a query based instance segmentation method driven by parallel supervision on dynamic mask heads. The key insight of QueryInst is to leverage the intrinsic one-to-one correspondence in object queries across different stages, as well as one-to-one correspondence between mask RoI features and object queries in the same stage. This approach eliminates the explicit multi-stage mask head connection and the proposal distribution inconsistency issues inherent in non-query based multi-stage instance segmentation methods. We conduct extensive experiments on three challenging benchmarks, i.e., COCO, CityScapes, and YouTube-VIS to evaluate the effectiveness of QueryInst in instance segmentation and video instance segmentation (VIS) task. Specifically, using ResNet-101-FPN backbone, QueryInst obtains 48.1 box AP and 42.8 mask AP on COCO test-dev, which is 2 points higher than HTC in terms of both box AP and mask AP, while runs 2.4 times faster. For video instance segmentation, QueryInst achieves the best performance among all online VIS approaches and strikes a decent speed-accuracy trade-off. Code is available at \url{https://github.com/hustvl/QueryInst}.

1. Introduction

QueryInst addresses the difficulty of adapting query-based detection to instance segmentation with parallel dynamic mask heads that exploit query correspondences. It reports improved accuracy and speed over established non-query methods and extends the framework to video instance segmentation.

  • Motivation: QueryInst targets instance segmentation in query-based end-to-end detection, where directly adapting successful non-query mask-generation methods is inefficient.The method is motivated by the need for a framework tailored to query-based detection.
  • Method: Parallel dynamic mask heads transform each mask RoI feature according to its corresponding object query and are trained simultaneously across stages.The design uses one-to-one query correspondences across stages and between mask RoI features and queries within a stage.
  • Results: 48.1 APbox and 42.8 APmask on COCO test-dev are 2 points higher than HTC for both metrics, while QueryInst runs 2.4× faster.These results use a ResNet-101-FPN backbone.
  • Task Jointing: QueryInst jointly supports object detection and instance segmentation through shared queries and multi-head self-attention, enabling communication between the tasks.The architecture also improves object detection performance.
  • Video Instance Segmentation: QueryInst extends to video instance segmentation by adding a vanilla track head and outperforms several existing VIS approaches on YouTube-VIS.The reported comparisons include MaskTrack R-CNN, SipMask-VIS, STEm-Seg, and VisTR.

2. Related Work

Prior work spans query-based set-prediction detectors, conventional object detection architectures, and multiple instance-segmentation paradigms. QueryInst follows R-CNN-based instance segmentation while introducing a query-based framework.

  • Query Based Methods: DETR introduced query-based set prediction for object detection, followed by methods that improve its performance and extend query-based modeling.Related approaches include Deformable DETR, UP-DETR, ACT, TSP, Sparse R-CNN, OneNet, and DeFCN.
  • Object Detection: Object detection research progressed from anchor-based R-CNN methods to anchor-free detectors and more recent query-based set-prediction methods.The passage identifies R-CNN, Fast R-CNN, Faster R-CNN, CenterNet, FCOS, and DETR as representative milestones.
  • Instance Segmentation: Instance segmentation includes R-CNN-based methods such as Mask R-CNN, Cascade Mask R-CNN, and HTC, alongside one-stage approaches including YOLACT, SipMask, CondInst, and SOLO.QueryInst is positioned as a query-based framework following the R-CNN-based line of work.

3. Instances as Queries

QueryInst adapts query-based detection to instance segmentation with parallel dynamic mask heads, exploiting one-to-one query correspondences across stages and between queries and mask RoI features. This design bypasses explicit multi-stage mask connections and proposal-distribution inconsistencies while retaining efficient final-stage inference.

  • Architecture: QueryInst uses a query-based object detector with six query stages and six dynamic mask heads trained in parallel.Sparse R-CNN is the default detector instantiation, with six query stages.
  • Baseline limitation: The vanilla query-based mask design is less effective than Cascade Mask R-CNN, while explicit HTC-style mask flow yields only moderate gains and lowers training and inference speed.The authors attribute part of this limitation to the smaller number of queries, which provides fewer training samples than proposal-based methods.
  • Information flow: One-to-one correspondence across query stages and between mask RoI features and same-stage queries provides per-mask information flow without explicit mask-head connections.Queries act as memory: mask features read query information during the forward pass, while mask gradients write information during backpropagation.
  • Dynamic mask heads: Dynamic mask heads adapt each mask RoI feature using the corresponding transformed object query, establishing query-conditioned mask processing.DynConvmask uses query-generated parameters to transform mask RoI features, followed by a vanilla mask head.
  • Inference: During inference, QueryInst discards the five intermediate dynamic mask heads and uses only final-stage predictions, while queries retain multi-stage mask information.The intermediate heads are used during training but omitted from final mask generation at inference.
  • Design rationale: QueryInst bypasses proposal-distribution inconsistency and explicit multi-stage mask connections, and surpasses Cascade Mask R-CNN and HTC in both accuracy and speed.The same query correspondence that links stages enables the proposed paradigm to avoid those non-query-based issues.

4. Experiments

QueryInst is evaluated on COCO, Cityscapes, and YouTube-VIS, including comparisons, backbone studies, and ablations of its mask-head design. Across these experiments, it reports competitive accuracy, speed, generalization, and video-instance-segmentation performance.

  • COCO Instance Segmentation: COCO comparisons evaluate QueryInst across backbones, augmentations, accuracy metrics, and inference speed against established instance-segmentation methods.The main COCO results are reported on test-dev, with APbox, mask AP, and FPS defined in the table caption.
  • COCO Instance Segmentation: 50.4 box AP and 44.6 mask AP are achieved with a deformable ResNeXt-101-FPN backbone without bells and whistles.The same passage reports the corresponding box and mask AP values together.
  • Cityscapes Instance Segmentation: 39.4 AP on Cityscapes validation and 34.4 AP on its test split surpass several strong baselines, including CondInst variants.QueryInst is reported as achieving leading Cityscapes results without bells and whistles.
  • Video Instance Segmentation: QueryInst-VIS operates in real time at over 30 FPS and improves AP by 4.3 over MaskTrack R-CNN and 2.1 over SipMask-VIS.It also outperforms several established video instance segmentation approaches in both accuracy and speed.
  • Ablation Study: Parallel DynConvmask improves the query-based baseline by 0.7 box AP and 1.9 mask AP while maintaining high FPS.During inference, intermediate parallel mask heads are discarded and only final-stage masks are used.

5. Conclusion

QueryInst is an efficient query-based end-to-end instance segmentation framework using parallel supervision on dynamic mask heads. The paper reports that it surpasses prior non-query-based approaches without sacrificing inference speed.

  • QueryInst uses parallel supervision on dynamic mask heads for query-based end-to-end instance segmentation.
  • The framework outperforms previous state-of-the-art non-query-based instance segmentation approaches.
  • QueryInst improves performance without a corresponding decline in inference speed.

Illustration of 4 Different Query and MSA Configurations

The ablation study finds that sharing both the query and MSA yields the best box and mask AP with the least additional overhead.

  • Shared query and shared MSA achieve the best performance on both box AP and mask AP.
  • The shared query and shared MSA configuration incurs the least additional overhead.

Training Time of QueryInst

Under the same experimental configuration, QueryInst outperforms HTC while requiring less training time.

  • QueryInst outperforms HTC using less training time under the same experimental configuration.

Effects of Different Training Schedules and Data Augmentations to the Mask R-CNN Family

Training schedules and data augmentation affect the Mask R-CNN family differently from QueryInst. Stronger augmentation does not significantly help the Mask R-CNN family under a 3× schedule, whereas QueryInst benefits from it.

  • Effects of Different Training Schedules and Data Augmentations to the Mask R-CNN Family: Stronger data augmentation does not significantly improve Cascade Mask R-CNN and HTC under the 3× training schedule.
  • Effects of Different Training Schedules and Data Augmentations to the Mask R-CNN Family: Mask R-CNN with ResNet-101-FPN converges near optimum under modest augmentation and the 3× schedule, with performance degenerating under longer schedules.
  • Effects of Different Training Schedules and Data Augmentations to the Mask R-CNN Family: Stronger augmentation can improve performance as training time becomes longer, but not under the 3× schedule.
  • Effects of Different Training Schedules and Data Augmentations to the Mask R-CNN Family: Unlike the Mask R-CNN family, QueryInst benefits from stronger data augmentation even under the 3× schedule.
  • Effects of Different Training Schedules and Data Augmentations to the Mask R-CNN Family: The paper leaves further study of these training-schedule and augmentation properties for future work.

Qualitative Results on COCO

Qualitative results are presented on the COCO validation split in Fig. 6.

  • Qualitative results are provided for COCO.The results use the COCO validation split.
  • The qualitative evaluation uses the COCO val split.
  • Figure 6 contains the reported COCO qualitative results.

Qualitative Results on Cityscapes

The paper presents qualitative results on the Cityscapes test split and visualizes how DynConvmask transforms mask features relative to ground-truth instance masks.

  • Qualitative Results on Cityscapes: Qualitative results are shown on the Cityscapes test split.
  • DynConvmask Visualization: Figure 8 studies DynConvmask through additional visualization results.
  • DynConvmask Visualization: Figure 8 compares FPN-extracted mask features, query-enhanced mask features, and ground-truth instance masks across three rows.The first row shows xmask, the second shows xmask∗ enhanced by queries, and the last shows ground-truth masks.
Loading 2105.01928v3…