Source-linked AI summary
AdaptIS: Adaptive Instance Selection Network
Konstantin Sofiiuk, Olga Barinova, Anton Konushin
TL;DR
Class-agnostic instance segmentation must separate and accurately mask objects despite occlusion and complex shapes. AdaptIS generates a mask from an image and point proposal using AdaIN-conditioned instance selection, then combines with semantic segmentation for panoptic output. It achieves state-of-the-art accuracy on Cityscapes and Mapillary without COCO pretraining and competitive performance on COCO.
Problem
Instance segmentation must separate and mask each object instance, including strongly overlapping objects that challenge detection-first methods.
Method
AdaptIS uses a point-conditioned instance selection network whose AdaIN parameters are generated from backbone features at the selected point.
Results
AdaptIS achieves state-of-the-art accuracy on Cityscapes and Mapillary without COCO pretraining and competitive performance on COCO.
Takeaways & Limitations
AdaptIS provides a foundation for class-agnostic instance and panoptic segmentation with pixel-accurate masks and direct mask optimization.
Abstract
from arXiv · showhide
We present Adaptive Instance Selection network architecture for class-agnostic instance segmentation. Given an input image and a point $(x, y)$, it generates a mask for the object located at $(x, y)$. The network adapts to the input point with a help of AdaIN layers, thus producing different masks for different objects on the same image. AdaptIS generates pixel-accurate object masks, therefore it accurately segments objects of complex shape or severely occluded ones. AdaptIS can be easily combined with standard semantic segmentation pipeline to perform panoptic segmentation. To illustrate the idea, we perform experiments on a challenging toy problem with difficult occlusions. Then we extensively evaluate the method on panoptic segmentation benchmarks. We obtain state-of-the-art results on Cityscapes and Mapillary even without pretraining on COCO, and show competitive results on a challenging COCO dataset. The source code of the method and the trained models are available at https://github.com/saic-vul/adaptis.
1. Introduction
AdaptIS formulates class-agnostic instance segmentation as generating an object mask from an image and a point proposal. It directly produces pixel-accurate masks, supports objects of any class, and extends to panoptic segmentation with standard semantic segmentation.
- AdaptIS takes an image and point proposal (x, y) and generates a mask for the object located at that position.
- Unlike detection-first and instance-embedding methods, AdaptIS directly optimizes target masks without bounding-box proposals or heuristic post-processing.
- Random points on the same object are designed to produce similar masks, while points between objects can yield a merged mask.
- AdaptIS provides pixel-accurate segmentation for severe occlusions and complex shapes, and can segment objects of different sizes using single-scale features.
- The method pairs with standard semantic segmentation to perform panoptic segmentation and achieves state-of-the-art accuracy on Cityscapes and Mapillary, with competitive COCO performance.
2. Related work
Related methods expose limitations that motivate AdaptIS: detection-first systems struggle with overlapping instances and lose mask detail, while embedding methods require clustering and trail detection-first performance. Panoptic systems commonly combine semantic and instance networks with post-processing.
- Detection-first instance segmentation methods: Detection-first methods can confuse strongly overlapping objects with similar or shared bounding boxes.
- Detection-first instance segmentation methods: ROI pooling reduces object-feature dimensionality, causing information loss and inaccurate masks.
- Instance embedding methods: Instance embedding methods map pixels into an embedding space, bringing same-instance pixels together and separating different instances.
- Instance embedding methods: Embedding-based methods require clustering-style processing and currently show inferior standard-benchmark performance compared with detection-first methods.
- Panoptic segmentation methods: Panoptic segmentation baselines commonly combine semantic and instance networks followed by heuristic post-processing and late fusion.
3. Architecture overview
AdaptIS uses a point-conditioned architecture in which a controller converts local backbone features into AdaIN parameters for mask generation. Relative CoordConv supplies point-relative spatial information, while training directly optimizes masks for sampled object points.
- AdaptIS takes an image and point proposal (x, y), then outputs a mask for the object at that location.
- The lightweight Instance Selection Network is parameterized by AdaIN, allowing its output to adapt to a selected object.
- A controller interpolates the backbone feature at (x, y), processes it with fully connected layers, and supplies AdaIN parameters to the selection network.
- Relative CoordConv encodes coordinates relative to the proposal within radius R, providing a location prior for disambiguating objects.
- Training samples points from objects and uses a pixel-wise loss against corresponding masks, with gradients reaching AdaptIS and the backbone.
- Random point sampling encourages similar outputs for points on the same object while directly optimizing masks rather than embedding distances.
4. Class-agnostic instance segmentation
AdaptIS aggregates independently generated object masks from point proposals, resolving overlaps with a greedy procedure while reusing a single backbone computation. Its design supports occlusion-aware, pixel-accurate segmentation across object sizes.
- Mask aggregation: AdaptIS samples point proposals, generates thresholded object masks, and adds masks whose overlap with already segmented pixels is below 50%.The process terminates when all pixels are segmented or no proposals remain.
- Mask aggregation: The final instance assignment gives each pixel the instance with the highest AdaptIS confidence among segmented objects.
- Efficiency: Only the lightweight AdaptIS head runs repeatedly per image, while the backbone runs once to reduce computation.
- Proposal selection: New proposals are sampled from unknown pixels, avoiding points likely to generate masks overlapping previously segmented objects.
- Panoptic extension: For panoptic segmentation, AdaptIS is paired with a standard semantic branch and jointly trained with the backbone, while point proposals prioritize candidate locations.
5. Panoptic segmentation
The panoptic pipeline combines class-agnostic AdaptIS masks with semantic predictions for stuff and instance labels for things. A dedicated proposal branch prioritizes points, while instance masks receive semantic labels by averaged pixel confidence.
- Network branches: AdaptIS and semantic segmentation branches share a backbone and are trained jointly, while the point proposal branch is trained afterward with the backbone frozen.
- Point proposals: The point proposal branch predicts whether each image location is a good or bad proposal and is trained from proposal IoU against ground-truth masks.The top 20% of sampled proposals by IoU are labeled positive.
- Panoptic formulation: Panoptic segmentation assigns a class label to every pixel and an instance id to each pixel classified as a thing.
- Panoptic inference: Stuff labels are inferred in one semantic pass, then their pixels initialize the segmented map before instance masks are added.
- Panoptic inference: Local maxima of the dense proposal-branch output reduce the candidate set, while instance labels are assigned using average semantic confidence over each mask.
6. Experiments on toy problem
The toy experiment tests AdaptIS on many overlapping, similarly shaped objects that are difficult for detection-first methods. AdaptIS uses random point proposals and is evaluated against Mask R-CNN using average precision at multiple IoU thresholds.
- Toy data: The toy images contain 8 to 22 elongated, similarly colored objects with random positions and orientations in 96×96-pixel images.
- Evaluation: The benchmark compares AdaptIS with a U-Net backbone against Mask R-CNN with an ImageNet-pretrained ResNet-50 backbone.
- Proposal sampling: AdaptIS samples seven random points per iteration and selects the result with the highest average object-mask confidence.
- Evaluation: Average precision is measured at different IoU thresholds to compare the two methods.
- Results: Mask R-CNN struggles when objects have heavily overlapping bounding boxes, whereas point-conditioned AdaptIS can distinguish severely occluded objects.
7. Experiments on standard benchmarks
AdaptIS is evaluated on Cityscapes, Mapillary, and COCO using standard panoptic metrics and a shared DeepLabV3+ backbone. It reaches state-of-the-art accuracy on Cityscapes and improves on prior Mapillary results by more than 3% in PQ, while remaining close to UPSNet on COCO.
- Evaluation setup: The evaluation covers Cityscapes, Mapillary, and COCO validation benchmarks using PQ, PQst, and PQth, with mIoU and AP additionally reported for Cityscapes.
- Cityscapes: AdaptIS achieves state-of-the-art PQ on Cityscapes without COCO pretraining or multiscale testing.
- Mapillary: AdaptIS improves upon the prior state of the art by more than 3% in PQ on Mapillary.
- COCO: On COCO, AdaptIS reports metrics close to those of the recent UPSNet method.
8. Conclusion and future work
AdaptIS introduces a novel network architecture for instance and panoptic segmentation that is conceptually different from mainstream detection-first methods.
- AdaptIS is a novel architecture for instance and panoptic segmentation.
Appendix A. Normalized Focal Loss
The appendix describes Normalized Focal Loss as a modification of Focal Loss designed to focus learning on misclassified pixels while preserving the total gradient scale of binary cross-entropy. It addresses the gradient fading that slows ordinary Focal Loss training.
- AdaptIS formulates point-conditioned object-mask generation as binary semantic segmentation with an end-to-end loss against the ground-truth mask.
- Focal Loss addresses binary cross-entropy's tendency to emphasize pixels that are already correctly classified.
- As prediction accuracy improves, ordinary Focal Loss's total gradient fades, slowing training over iterations.
- Normalized Focal Loss is a Focal Loss modification used in the experiments.
- NFL concentrates on misclassified pixels while matching BCE's total gradient and producing faster convergence and better accuracy.
Appendix B. Ablation studies
Ablations on Cityscapes show that Relative CoordConv and the loss choice affect panoptic quality, with the largest reported drop occurring when Relative CoordConv is removed.
- Approximately 5% drop in PQ follows removal of the Relative CoordConv block.
- Approximately 3% drop in PQ follows replacement of Normalized Focal Loss with simple Focal Loss.
- Replacing Normalized Focal Loss with cross-entropy causes minor PQ degradation, mainly for PQthings.
Appendix C. Analysis of results on COCO
On COCO and Mapillary, the authors reused the Cityscapes architecture without parameter fine-tuning, achieving state-of-the-art Mapillary validation accuracy and competitive COCO validation results. They identify additional training and capacity as possible routes for improving COCO performance, while noting a COCO instance-distribution challenge for AdaptIS.
- The same architecture was used for COCO and Mapillary without fine-tuning any parameters.
- AdaptIS achieved state-of-the-art accuracy on Mapillary validation and competitive results on COCO validation.
- COCO results may improve with more convolutional layers and parameter fine-tuning.
- Most COCO images contain only one instance of the same object, whereas AdaptIS benefits from examples containing multiple instances on one image.