PaperlayerView exact revision on arXiv

Source-linked AI summary

FastInst: A Simple Query-Based Model for Real-Time Instance Segmentation

Junjie He, Pengyu Li, Yifeng Geng, Xuansong Xie

arXiv:2303.08594v2cs.CV

TL;DR

Query-based models’ superiority on efficient, high-accuracy real-time instance segmentation has not been well demonstrated. FastInst combines activation-guided queries, dual-path updates, and ground-truth mask-guided learning; on COCO it reaches 40.5 AP at 32.5 FPS.

  • Problem

    Query-based models’ superiority on high-accuracy real-time instance segmentation has not been well demonstrated, despite the importance of efficiency for real-world applications.

    Source: §1 · p. 2

  • Method

    FastInst adapts a query-based segmentation framework with IA-guided queries, dual-path feature and query updates, and ground-truth mask-guided learning.

    Source: §1 · p. 2, §3.3 · p. 3, §3.5 · p. 5

  • Results

    40.5 AP at 32.5 FPS on COCO test-dev makes FastInst the only Table 1 algorithm with AP > 40 while maintaining real-time speed.

    Source: §4.2 · p. 6

  • Takeaways & Limitations

    FastInst’s COCO results show query-based models can achieve strong performance while remaining fast, and the authors propose it as a real-time baseline.

    Source: §1 · p. 2, §5 · p. 8

    Like general query-based models, FastInst is not good at small targets.

    Source: §5 · p. 8

Abstract

from arXiv · show

Recent attention in instance segmentation has focused on query-based models. Despite being non-maximum suppression (NMS)-free and end-to-end, the superiority of these models on high-accuracy real-time benchmarks has not been well demonstrated. In this paper, we show the strong potential of query-based models on efficient instance segmentation algorithm designs. We present FastInst, a simple, effective query-based framework for real-time instance segmentation. FastInst can execute at a real-time speed (i.e., 32.5 FPS) while yielding an AP of more than 40 (i.e., 40.5 AP) on COCO test-dev without bells and whistles. Specifically, FastInst follows the meta-architecture of recently introduced Mask2Former. Its key designs include instance activation-guided queries, dual-path update strategy, and ground truth mask-guided learning, which enable us to use lighter pixel decoders, fewer Transformer decoder layers, while achieving better performance. The experiments show that FastInst outperforms most state-of-the-art real-time counterparts, including strong fully convolutional baselines, in both speed and accuracy. Code can be found at https://github.com/junjiehe96/FastInst .

1. Introduction

Query-based instance segmentation has not been shown to match convolutional methods on efficient, high-accuracy real-time benchmarks. FastInst combines three design choices to make query-based segmentation fast and accurate on COCO.

1. Introduction

  • FastInst surpasses most prior real-time instance segmentation methods in speed and accuracy on COCO.Figure 1 compares the models on COCO test-dev; Mask2Former uses a PPM-based FPN pixel decoder for a similar speed-and-accuracy comparison.Source: Figure 1 · p. 1, §1 · p. 2
  • Query-based models’ superiority on fast, efficient instance segmentation remains un demonstrated, while real-time benchmarks are dominated by convolutional models.Mask2Former’s static queries require many decoder layers, its pixel decoder can be heavy, and masked attention may yield suboptimal query updates.Source: §1 · pp. 1–2
  • 40.5 AP at 32.5 FPS on COCO test-dev makes FastInst the only Table 1 algorithm with AP > 40 at real-time speed.The paper also reports a fastest variant with ResNet-50 achieving 35.6 AP at 53.8 FPS on COCO test-dev.Source: §1 · p. 2
  • FastInst follows Mask2Former’s meta-architecture and uses IA-guided queries, dual-path updates, and ground-truth mask-guided learning.These choices target faster query refinement, reduced reliance on a heavy pixel decoder, and improved masked attention.Source: §1 · p. 2

2. Related Work

Prior instance segmentation approaches include region-based, instance activation-based, and query-based methods. FastInst targets the efficiency gap between query-based models and convolutional real-time methods.

2. Related Work

  • Region-based methods use detected boxes and RoI features, but duplicate proposals hurt efficiency.Mask R-CNN predicts each object’s mask with a mask branch; follow-up work improves boxes or masks.Source: §2 · p. 2
  • Instance activation-based methods represent objects with meaningful pixels; FastInst selects high-class-semantic pixels in object regions as queries.This differs from center-activation approaches such as SOLO and weighted pixel combinations such as SparseInst.Source: §2 · p. 2
  • Query-based models perform well, but their computational expense and complex dual-path designs limit real-world efficiency and deployment.FastInst builds its dual-path architecture on plain Transformer layers to improve efficiency.Source: §2 · p. 3

3. Methods

FastInst uses a backbone, pixel decoder, and Transformer decoder to predict instance classes and masks. Its six method components combine lightweight features, selected queries, dual-path updates, and guided training.

3.1. Overall architecture

  • FastInst sends backbone features through a pixel decoder, then uses IA-guided and auxiliary queries with high-resolution pixel features in the Transformer decoder.The decoder updates queries and pixel features and predicts classes and masks at each layer.Source: §3.1 · p. 3

3.2. Lightweight pixel decoder

  • Refined pixel features let FastInst use a lightweight pixel decoder; PPM-FPN enlarges receptive fields for an accuracy-speed trade-off.Unlike prior methods that directly use pixel-decoder feature maps, FastInst produces masks from refined pixel features.Source: §3.2 · p. 3

3.3. Instance activation-guided queries

  • IA-guided queries select high-foreground-probability local-maximum pixel embeddings from E4, while a location cost restricts training matches to pixels inside objects.The queries carry potential-object information at initialization; middle-size E4 is selected as a feature-map trade-off.Source: §3.3 · pp. 3–4

3.4. Dual-path Transformer decoder

  • The dual-path decoder alternates pixel-feature and query updates, co-optimizing both rather than relying on a single-path query update.Pixel updates use cross-attention without pixel self-attention; query updates use masked attention, self-attention, and feedforward layers.Source: §3.4 · p. 4

3.5. Ground truth mask-guided learning

  • Ground-truth mask-guided learning replaces matched queries’ predicted attention masks with matched ground-truth masks and reuses the last-layer matching.Queries without a matched instance use standard cross-attention; the fixed assignment keeps layer predictions consistent and avoids rematching during guided training.Source: §3.5 · p. 5

3.6. Loss function

  • FastInst’s overall loss combines IA-query classification loss, Transformer prediction loss, and ground-truth mask-guided prediction loss.Prediction loss includes mask cross-entropy, dice, and class losses across decoder layers; guided loss uses fixed target assignment and excludes the initial layer.Source: §3.6 · p. 5

4. Experiments

Experiments evaluate FastInst on COCO and ablate its design; the final model outperforms most previous real-time instance segmentation methods in speed and accuracy.

4.1. Implementation details

  • FastInst training uses 50 epochs, batch size 16, AdamW, and a 640-pixel shorter-edge test scale, with the longer edge capped at 864 pixels.FPS is measured on a V100 GPU with batch size 1; FLOPs are averaged over 100 validation images.Source: §4.1 · pp. 5–6

4.2. Main results

  • 32.5 FPS and 40.5 AP make FastInst the only Table 1 method above 40 AP at real-time speed; FastInst-D1 beats SparseInst by 0.9 AP with fewer epochs and less inference time.All Table 1 entries are single-scale COCO test-dev results.Source: Table 1 · p. 6, §4.2 · p. 6

4.3. Ablation studies

  • IA-guided queries outperform zero or learned queries, with a more significant gain when using only one Transformer decoder layer.The queries are selected from feature maps and carry information about potential objects at initialization.Source: §4.3 · p. 7
  • The dual-path strategy performs better than conventional single query updates in the lightweight pixel-decoder setting, while query/pixel update order matters little.The single-path strategies use twice as many Transformer decoder layers for a fair comparison.Source: §4.3 · p. 7, Table 3 · p. 7
  • GT mask-guided learning improves performance by up to 0.5 AP and generalizes across different backbones.The method helps the Transformer decoder learn to update queries for better object embeddings under masked attention.Source: §4.3 · p. 7, Table 4 · p. 7
  • Stronger pixel decoders improve performance but cost more computation, while PPM-FPN offers a speed-accuracy trade-off; segmentation reaches 30.5 AP without a Transformer decoder and saturates around six layers.FastInst is compatible with existing pixel decoders, and few decoder layers are advantageous for real-time use.Source: §4.3 · p. 7, Table 5 · p. 7
  • Increasing IA-guided query count improves segmentation and recall but affects speed; eight auxiliary learnable queries outperform making all queries IA-guided, and E4 balances query-source accuracy and speed.Larger feature maps yield better but limited gains; more than 100 queries slightly degrades performance, while 10 queries achieve 31.2 AP on COCO.Source: §4.3 · p. 8
  • Bipartite matching-based assignment gives a significant gain, location cost reduces matching space and accelerates convergence, and learnable positional embeddings improve inference speed without compromising performance.Figure 3 shows query points under the location-cost loss more concentrated on foreground objects than under the alternative supervision losses.Source: Figure 3 · p. 7, §4.3 · p. 8

5. Conclusion

FastInst combines three efficient components with a query-based segmentation framework to achieve strong COCO performance at fast inference speed. The conclusion also identifies small targets and the training cost of GT mask-guided learning as limitations.

5. Conclusion

  • FastInst combines instance activation-guided queries, a dual-path update strategy, and GT mask-guided learning to achieve strong COCO performance at fast inference speed.The authors describe it as a potential new baseline for real-time instance segmentation.Source: §5 · p. 8
  • Small-target performance remains unsatisfactory, even with stronger pixel decoders or larger feature maps, which add computational burden.Source: §5 · p. 8
  • GT mask-guided learning improves masked attention but increases training costs.Source: §5 · p. 8

Appendix

  • The appendix covers additional ablations, Cityscapes performance, unified segmentation ability, and COCO validation predictions.Source: Appendix · p. 11

A.1. Improvements on original Mask2Former

Ablations on original Mask2Former examine three FastInst components, while a separate DCN study evaluates the location cost. The reported effects vary by component and setting.

A.1. Improvements on original Mask2Former

  • Dual-path updating improves Mask2Former’s efficiency the most, accelerating inference and dramatically reducing model parameters.GT mask-guided learning also performs well; IA-guided queries add little to the original nine-layer decoder, whose learnable queries already decode well, but contribute to the three-layer dual-path decoder.Source: §A.1 · p. 11

A.2. Effect of location cost on deformable convolutional networks

  • The location cost restricts matched pixels to object regions, reducing the matching space and accelerating training convergence.With DCNs, outside-object pixels may be activated through offsets but are poor IA-guided query candidates because they rely on precise offset predictions.Source: §A.2 · p. 11
  • With DCNs, the location cost produces more concentrated and higher-quality IA-guided queries.Source: §A.2 · p. 11

A.3. Effect of local-maximum-first selection strategy

The local-maximum-first strategy selects IA-guided queries in two stages and is especially effective when the query count is small. Without it, multiple selected queries can land on one salient object, reducing recall of other instances.

A.3. Effect of local-maximum-first selection strategy

  • At prediction, the method first selects E4 pixel embeddings that are local maxima in their corresponding class planes, then chooses those with the highest foreground probabilities.Source: §A.3 · p. 11
  • The local-maximum-first selection strategy improves performance, especially when the IA-guided query number is small.The ablation uses FastInst-D1-640 with a ResNet-50 backbone.Source: §A.3 · p. 11, Table III · p. 11
  • Without local-maximum-first selection, two queries can fall on the same salient handbag, hurting recall of other instances.Source: §A.3 · p. 11

A.4. Auxiliary query analysis

The auxiliary-query visualizations show attention to general information, including background edges and class-agnostic foreground objects. Other figures visualize query distributions under different selection or location-cost settings.

A.4. Auxiliary query analysis

  • Auxiliary learnable queries attend to general information such as edges, including background edges, and class-agnostic foreground objects.Figure III shows three of eight auxiliary queries from the last decoder layer of FastInst-D3.Source: §A.4 · p. 11, Figure III · p. 12
  • Figure I compares IA-guided query distributions with and without location cost for a DCN-backed FastInst-D1-640 model.Some queries in the padding area are not shown.Source: Figure I · p. 12
  • Figure II compares IA-guided query distributions without and with local-maximum-first selection, alongside the ground truth, using Na = 10.Source: Figure II · p. 12

B.1. Cityscapes

On Cityscapes, FastInst outperforms Mask2Former by 4.1 AP at similar speed. The paper also reports results for panoptic and semantic segmentation and says FastInst performs better in instance-level segmentation.

B.1. Cityscapes

  • FastInst is evaluated for instance segmentation across eight Cityscapes semantic classes, using full-resolution images at inference.Training uses 90K iterations and crops of 512×1024 pixels.Source: §B.1 · p. 12
  • FastInst outperforms Mask2Former by 4.1 AP on Cityscapes validation with similar speed.The comparison uses the same pixel decoder and training settings.Source: §B.1 · p. 12, Table IV · p. 12

C. Unified Segmentation

  • FastInst is also evaluated on Cityscapes panoptic and semantic segmentation, and the table caption reports better instance-level segmentation than Mask2Former.The models use the same settings as the instance-segmentation experiments.Source: C. Unified Segmentation · p. 12, Table V · p. 12

D. Visualization

The section visualizes FastInst-D3 predictions on COCO validation and highlights duplicate predictions and over-segmentation as typical failures. The figures compare ground truth with predictions at a confidence threshold of 0.5.

D. Visualization

  • 40.1 AP at 32.5 FPS is reported for the visualized FastInst-D3 model on COCO validation.The model uses a ResNet-50-d-DCN backbone and runs on a single V100 GPU.Source: D. Visualization · p. 12, Figure IV · p. 13
  • Figures IV and V place ground truth in the first and third columns and predictions in the second and fourth.Both use a confidence threshold of 0.5.Source: Figure IV · p. 13, Figure V · p. 14
  • Figure VI illustrates duplicate predictions and over-segmentation, alongside a few false positives and false negatives.The examples include a duplicate person prediction and an over-segmented cow.Source: Figure VI · p. 15

Related papers

Loading 2303.08594v2…