Source-linked AI summary
SSAP: Single-Shot Instance Segmentation With Affinity Pyramid
Naiyu Gao, Yanhu Shan, Yupei Wang, Xin Zhao, Yinan Yu, Ming Yang, Kaiqi Huang
TL;DR
Separate semantic labeling and instance grouping in prior proposal-free methods leave their mutual benefits underused and require multiple modules or passes. SSAP jointly learns semantic classes and a hierarchical pixel-pair affinity pyramid in one shot, then applies cascaded graph partitioning. It achieves 5× speedup and 9% relative AP improvement over non-cascaded partitioning, with state-of-the-art Cityscapes results including 37.3% AP validation, 32.7% AP test, and 61.1% PQ validation.
Problem
Prior proposal-free methods separate semantic labeling from instance grouping, leaving their mutual benefits underused and adding computational costs through multiple modules.
Method
SSAP jointly learns semantic class labels and hierarchical pixel-pair affinities with one backbone, then uses cascaded graph partitioning to generate instances from coarse to fine.
Results
5× speedup and 9% relative improvement on AP are achieved over the non-cascaded approach, alongside state-of-the-art Cityscapes performance.
Takeaways & Limitations
The unified affinity-pyramid and semantic-labeling scheme provides mutual benefits between the two sub-tasks while enabling single-pass proposal-free instance segmentation.
Takeaways & Limitations
Long-range affinities are more difficult to jointly learn with pixel-level semantic class labeling than short-range affinities.
Abstract
from arXiv · showhide
Recently, proposal-free instance segmentation has received increasing attention due to its concise and efficient pipeline. Generally, proposal-free methods generate instance-agnostic semantic segmentation labels and instance-aware features to group pixels into different object instances. However, previous methods mostly employ separate modules for these two sub-tasks and require multiple passes for inference. We argue that treating these two sub-tasks separately is suboptimal. In fact, employing multiple separate modules significantly reduces the potential for application. The mutual benefits between the two complementary sub-tasks are also unexplored. To this end, this work proposes a single-shot proposal-free instance segmentation method that requires only one single pass for prediction. Our method is based on a pixel-pair affinity pyramid, which computes the probability that two pixels belong to the same instance in a hierarchical manner. The affinity pyramid can also be jointly learned with the semantic class labeling and achieve mutual benefits. Moreover, incorporating with the learned affinity pyramid, a novel cascaded graph partition module is presented to sequentially generate instances from coarse to fine. Unlike previous time-consuming graph partition methods, this module achieves $5\times$ speedup and 9% relative improvement on Average-Precision (AP). Our approach achieves state-of-the-art results on the challenging Cityscapes dataset.
1. Introduction
The paper proposes a single-shot, proposal-free instance segmentation method that jointly learns semantic labels and instance-aware affinities, then partitions instances from coarse to fine. This design addresses the computational cost of separate multi-stage modules and achieves state-of-the-art Cityscapes results.
- Motivation: Proposal-free methods typically produce semantic labels and instance-aware features in separate stages, creating additional computational costs for applications.
- Method: The proposed method jointly learns pixel-level semantic classes and pixel-pair affinities with a single backbone network and one forward pass.The affinity represents the probability that two pixels belong to the same instance.
- Method: The affinity pyramid organizes instance-aware pixel-pair relationships hierarchically and is jointly learned with semantic class labeling to exploit mutual benefits.
- Results: 37.3% AP on validation, 32.7% AP on test, and 61.1% PQ on validation establish state-of-the-art performance on Cityscapes.
- Method: The cascaded graph partition module uses the affinity hierarchy to generate instances sequentially from coarse to fine.It constructs a graph from pixels as nodes and affinities as edge scores before partitioning it.
- Results: 5× speedup and 9% relative improvement on AP are achieved over the non-cascaded graph partition approach.
2. Related Work
Prior instance segmentation methods follow proposal-based or proposal-free paradigms. Proposal-free approaches learn instance-aware features and apply grouping procedures, while affinity-based grouping has also been explored through hierarchical clustering.
- Paradigms: Existing instance segmentation approaches are divided into proposal-based and proposal-free paradigms.
- Proposal-Based Methods: Proposal-based methods detect bounding boxes and predict masks within proposals, with representative systems extending detection pipelines through cascaded or combined modules.
- Proposal-Free Methods: Proposal-free methods learn instance-aware features and cluster pixels into instances using boundary-aware energies, embeddings, sequential grouping, or recurrent generation.
- Affinity-Based Methods: Pixel-pair affinity has supported semantic segmentation and instance grouping, including prior work using agglomerative hierarchical clustering for instance-aware affinities.
3. Proposed Approach
The proposed model jointly learns semantic segmentation and a hierarchical pixel-pair affinity pyramid in one unified network, then uses cascaded graph partitioning to generate and refine instances from coarse to fine. The design combines multi-scale affinity cues with semantic information while reducing graph-partition inference cost.
- Unified learning: The unified U-shape network jointly learns semantic segmentation and a multi-range affinity pyramid with a single backbone.Affinity branches operate at feature levels with different resolutions, enabling joint learning of the two tasks.
- Affinity pyramid: Pixel-pair affinity predicts whether two pixels belong to the same instance within local windows, using binary ground truth labels.For each pixel, affinities to neighboring pixels in an r × r window are predicted; same-instance pairs receive label 1 and different-instance pairs receive 0.
- Affinity pyramid: 80% of pixels with all-one affinity labels are randomly dropped, and object-instance pixels receive 3 times the loss during training.These measures address the imbalance caused by mostly interior pixels having all-one ground-truth affinities.
- Affinity pyramid: The affinity pyramid decouples short-range and long-range affinities across resolutions instead of enlarging the affinity window.A 5 × 5 window at 1/64 resolution can connect pixels up to 128 pixels apart, while higher resolutions provide finer detail.
- Cascaded graph partition: Graph partition converts pixel affinities into edge scores and optimizes a partition subject to cycle constraints, but the conventional search-based solver has costly inference.The cascade addresses this cost by partitioning coarse representations first and progressively refining them.
- Cascaded graph partition: The cascaded graph partition module progressively refines instance predictions from low resolution to high resolution using semantic segmentation and affinity cues.Lower-resolution partitioning reduces the number of nodes, while higher-resolution layers refine proposals; segmentation refinement improves errors caused by semantic-segmentation failures.
4. Experiments
Experiments evaluate SSAP on Cityscapes and COCO, analyze joint learning and affinity design, and test cascaded graph partitioning. The method reports strong segmentation results while reducing graph-partition runtime.
- Dataset and metrics: Cityscapes contains 5,000 finely annotated high-resolution images, and the benchmark evaluates eight instance-segmentation classes alongside 11 background classes for semantic segmentation.
- Dataset and metrics: PQ measures both segmentation and recognition quality, with SQ and RQ separately capturing these two components.
- Ablation studies: Joint learning and hierarchical affinity construction outperform learning all affinity-pyramid layers from a single 1/4-resolution feature map.
- Cascaded graph partition: 5× speedup and 9% AP improvement result from initializing cascaded graph partition at 1/16 resolution rather than 1/4 resolution.
- Cascaded graph partition: 2.0% absolute AP improvement is obtained with 1/64-resolution initialization, attributed to lower-resolution proposals reducing disturbing information for prediction.
- Quantitative results: 37.3% AP and 61.1% PQ are achieved on the Cityscapes validation set with ResNet-101, while test-set AP reaches 32.7%.
5. Conclusion
The work presents a single-shot proposal-free instance segmentation method that jointly learns semantic labels and instance differentiation through an affinity pyramid. A cascaded graph partition module improves efficiency and AP, while the method achieves state-of-the-art results on Cityscapes.
- A single-shot proposal-free method jointly learns pixel-level semantic labels and instance differentiation using an affinity pyramid and one backbone network.The affinity pyramid distinguishes instances through pixel-pair affinities and is learned jointly with semantic class labels.
- The approach achieves a new state of the art on the challenging Cityscapes dataset.