Source-linked AI summary
Mask Transfiner for High-Quality Instance Segmentation
Lei Ke, Martin Danelljan, Xia Li, Yu-Wing Tai, Chi-Keung Tang, Fisher Yu
TL;DR
Instance segmentation methods achieve strong detection results, but their masks remain coarse because accurate boundary prediction at high resolution is costly. Mask Transfiner represents sparse error-prone regions with a hierarchical quadtree and refines them using a transformer. Across COCO, Cityscapes, and BDD100K, it reports large performance advantages over two-stage and query-based methods with low computation and memory cost.
Problem
Existing instance segmentation methods produce coarse masks because reduced spatial resolution loses high-frequency boundary information, while high-resolution features are computationally expensive.
Method
Mask Transfiner detects incoherent regions, represents them in a hierarchical quadtree, and jointly refines their nodes with a transformer across multiple scales.
Results
Mask Transfiner achieves large performance advantages across COCO, Cityscapes, and BDD100K for both two-stage and query-based segmentation frameworks.
Takeaways & Limitations
Focusing refinement on sparse incoherent nodes enables high-quality masks with lower computation and memory cost than standard dense or transformer-based processing.
Abstract
from arXiv · showhide
Two-stage and query-based instance segmentation methods have achieved remarkable results. However, their segmented masks are still very coarse. In this paper, we present Mask Transfiner for high-quality and efficient instance segmentation. Instead of operating on regular dense tensors, our Mask Transfiner decomposes and represents the image regions as a quadtree. Our transformer-based approach only processes detected error-prone tree nodes and self-corrects their errors in parallel. While these sparse pixels only constitute a small proportion of the total number, they are critical to the final mask quality. This allows Mask Transfiner to predict highly accurate instance masks, at a low computational cost. Extensive experiments demonstrate that Mask Transfiner outperforms current instance segmentation methods on three popular benchmarks, significantly improving both two-stage and query-based frameworks by a large margin of +3.0 mask AP on COCO and BDD100K, and +6.6 boundary AP on Cityscapes. Our code and trained models will be available at http://vis.xyz/pub/transfiner.
1. Introduction
Mask Transfiner targets the persistent gap between strong object detection and coarse, computationally expensive mask prediction by refining sparse error-prone regions with a quadtree-based transformer. It reports improved segmentation performance while processing only selected high-resolution regions.
- Motivation: Existing instance segmentation methods detect and localize objects effectively, but accurate and efficient mask prediction remains difficult.Recent methods retain a significant gap between bounding-box and segmentation performance, with coarse masks that oversmooth object boundaries.
- Motivation: High-resolution deep features improve mask quality but demand substantial computation and memory.
- Approach: Mask Transfiner detects sparse incoherent regions, builds a hierarchical quadtree across scales, and refines those regions instead of processing all pixels.The incoherent regions are concentrated mainly along object boundaries and in high-frequency areas, yet constitute only a small portion of total pixels.
- Approach: Its transformer refinement network encodes incoherent points, jointly models sparse features across quadtree levels, and predicts their mask labels.The design uses node encoding, sequence encoding, and pixel decoding to process sparse feature points in parallel and model their pixel-wise relations.
- Evidence: Mask Transfiner produces more detailed masks in high-frequency image regions than several competing methods when replacing Mask R-CNN’s default mask head.
- Results: 41.6 APMask on COCO test-dev with ResNet-50 establishes a new state-of-the-art result, while experiments across COCO, Cityscapes, and BDD100K show gains over two-stage and query-based methods.The authors also report efficiency in computation and memory relative to standard transformer usage.
2. Related Work
Prior instance segmentation work includes two-stage, one-stage, kernel-based, and query-based approaches, while Mask Transfiner focuses on correcting sparse mask errors with an end-to-end transformer.
- Existing paradigms: Two-stage methods detect bounding boxes before segmenting each region of interest, whereas one-stage and kernel-based methods remove proposal generation and feature re-pooling for greater efficiency.
- Query-based methods: Query-based methods treat segmentation as set prediction, using object queries to jointly perform classification, detection, and mask regression.
- Query-based methods: Despite their query-based formulation, existing methods still show large detection–segmentation gaps and unsatisfactory mask quality.
- Mask Transfiner: Mask Transfiner differs from prior refinement methods by using an end-to-end transformer with non-local sparse feature points as input queries.Its refinement regions are predicted by a lightweight FCN rather than nondeterministic sampling based on mask scores.
3. Mask Transfiner
Mask Transfiner targets coarse, spatially low-resolution instance masks by detecting sparse error-prone regions and refining them with a quadtree-guided transformer. It combines cascaded incoherence detection, hierarchical mask refinement, and cross-level attention to process only critical nodes.
- Incoherent regions: Incoherent regions capture information lost during downsampling and are concentrated near object boundaries or other high-frequency regions.They are defined by comparing a mask with its downsampled-and-reconstructed version using logical operations.
- Incoherent regions: 43% of wrongly predicted pixels occur in incoherent regions, although those regions occupy only 14% of bounding-box areas.Their coarse-mask accuracy is 56%, and an oracle that fills them with ground-truth labels raises AP from 35.5 to 51.0.
- Incoherent-region detection: Mask Transfiner starts from a coarse mask and multi-scale features, then detects incoherent regions across feature scales with a cascaded lightweight module.Lower-resolution detections are upsampled and fused with neighboring higher-resolution features to guide finer predictions.
- Quadtree mask refinement: The quadtree recursively subdivides detected incoherent nodes into four quadrants and propagates corrected labels from coarse to finer levels.Only identified incoherent points are split and refined, reducing high-resolution computation and memory requirements.
- Transformer refinement: Mask Transfiner’s refinement transformer processes sparse incoherent nodes across three quadtree levels using node encoding, sequence self-attention, and pixel decoding.The sequence encoder supports global spatial and inter-scale reasoning, while the input length remains far smaller than the full image grid.
4. Experiments
Experiments across COCO, Cityscapes, and BDD100K evaluate Mask Transfiner’s components, efficiency, and performance against existing methods. Ablations show that incoherent-region refinement, multi-level processing, and quadtree-based design improve mask quality while retaining computational efficiency.
- Experimental Setup: Mask Transfiner is evaluated on COCO, Cityscapes, and BDD100K using standard mask and boundary-quality metrics.COCO additionally reports AP⋆ with LVIS annotations, while APB focuses on boundary quality.
- Ablation Experiments: 1.8 AP and 0.7 AP gains over full RoIs and detected boundary regions respectively demonstrate the advantage of incoherent-region refinement.The comparison is performed on the COCO validation set.
- Ablation Experiments: 86% recall and 84% accuracy result from lower-level mask guidance, improving detection from 74% recall and 79% accuracy.Recall is emphasized because covering all error-prone regions is more important for subsequent refinement.
- Ablation Experiments: Mask AP⋆ increases from 38.4 to 40.7 as quadtree output sizes grow from 28×28 to 224×224.Performance saturates beyond 112×112, while the three-stage Transfiner runs at 7.1 fps.
- Ablation Experiments: Mask Transfiner’s multi-head attention achieves higher accuracy with lower computation and memory consumption than non-local attention and standard transformers.The comparison examines alternative attention mechanisms on COCO validation data.
5. Conclusion
Mask Transfiner is a high-quality, efficient instance segmentation method that uses hierarchical quadtrees and transformer-based label prediction to improve masks at low computation and memory cost.
- Mask Transfiner detects and decomposes image regions into a hierarchical quadtree, then transforms quadtree points into a query sequence for final label prediction.
- The method produces high-quality masks with low computation and memory cost compared with convolution-based segmentation methods limited by uniform image grids.
- Mask Transfiner improves both two-stage and query-based segmentation frameworks across COCO, Cityscapes, and BDD100K.
- Fully supervised training remains a limitation shared by Mask Transfiner and competing methods.
6. Appendix
The appendix provides implementation details, experiments, qualitative comparisons, speed analysis, and visualizations showing that Mask Transfiner improves fine-grained mask quality while retaining computational efficiency.
- Implementation and Training/Inference Details: Mask Transfiner is implemented in Detectron2 with Faster R-CNN for two-stage segmentation and DETR for query-based segmentation.
- Implementation and Training/Inference Details: Jittering boundary regions improves robustness of incoherent-region detection, whose recall is more important than precision for final performance.
- Implementation and Training/Inference Details: During inference, refined incoherent-node predictions propagate hierarchically from coarse to finer quadtree scales without test-time augmentation.
- Accuracy Comparison: Mask Transfiner achieves 42.2 mask AP on COCO test-dev with ResNet-101 and Faster R-CNN, compared with 39.6 for Mask Scoring R-CNN under the same setting.
- Accuracy Comparison: Mask Transfiner improves the baseline Mask R-CNN by 2.8 mask AP and outperforms PointRend by 0.9 AP on COCO test-dev.
- Inference Speed: Compared with Mask R-CNN, Transfiner incurs about a 35% inference-speed drop but provides a 2.8 mask AP gain and four-times larger output height and width.
- Qualitative Results: Qualitative comparisons across COCO, BDD100K, and Cityscapes show substantially more precise masks, including improved details in high-frequency regions and separated fine structures.
- Increasing Quadtree Refinement Depth: Increasing quadtree depth produces substantially finer detail around object boundaries by preserving more low-level information.