Source-linked AI summary
SegSort: Segmentation by Discriminative Sorting of Segments
Jyh-Jing Hwang, Stella X. Yu, Jianbo Shi, Maxwell D. Collins, Tien-Ju Yang, Xiao Zhang, Liang-Chieh Chen
TL;DR
Semantic segmentation is commonly framed as pixel-wise classification, whereas the paper seeks representations that organize pixels into perceptual groups and associate segments across images. SegSort uses end-to-end metric learning with clustering and nearest-segment retrieval, achieving 76% of its supervised counterpart's performance without supervision and improving supervised segmentation over softmax approaches.
Problem
Existing deep learning semantic segmentation methods mainly use pixel-wise classification, while the paper studies segmentation through perceptual groups and cross-image segment association.
Method
SegSort learns pixel-wise embeddings with vMF-based spherical clustering and segment-level maximum-likelihood metric learning, then assigns labels by nearest-segment majority vote.
Results
76% performance of its supervised counterpart is achieved by unsupervised SegSort, while supervised SegSort consistently improves over conventional pixel-wise softmax approaches.
Takeaways & Limitations
SegSort produces more precise boundaries, more consistent region predictions, and interpretable labels explained by retrieved nearest segments.
Takeaways & Limitations
The method assumes pixel-wise normalized embeddings within each segment follow a vMF distribution and uses a constant concentration parameter κ for all distributions.
Abstract
from arXiv · showhide
Almost all existing deep learning approaches for semantic segmentation tackle this task as a pixel-wise classification problem. Yet humans understand a scene not in terms of pixels, but by decomposing it into perceptual groups and structures that are the basic building blocks of recognition. This motivates us to propose an end-to-end pixel-wise metric learning approach that mimics this process. In our approach, the optimal visual representation determines the right segmentation within individual images and associates segments with the same semantic classes across images. The core visual learning problem is therefore to maximize the similarity within segments and minimize the similarity between segments. Given a model trained this way, inference is performed consistently by extracting pixel-wise embeddings and clustering, with the semantic label determined by the majority vote of its nearest neighbors from an annotated set. As a result, we present the SegSort, as a first attempt using deep learning for unsupervised semantic segmentation, achieving $76\%$ performance of its supervised counterpart. When supervision is available, SegSort shows consistent improvements over conventional approaches based on pixel-wise softmax training. Additionally, our approach produces more precise boundaries and consistent region predictions. The proposed SegSort further produces an interpretable result, as each choice of label can be easily understood from the retrieved nearest segments.
1. Introduction
SegSort reframes semantic segmentation as metric learning over perceptual groups, learning embeddings that segment images and associate semantically related segments. It supports both supervised and unsupervised segmentation, with improved region consistency and precise boundaries.
- 1. Introduction: SegSort treats segmentation as perceptual grouping and cross-image association rather than only pixel-wise classification.Pixels map to an embedding space where nearby points indicate shared segments, while clusters across images form semantic concepts.
- 1. Introduction: The method learns pixel embeddings that encourage within-segment similarity and separate different segments through metric learning.Its formulation uses maximum likelihood estimation in a two-stage EM framework with spherical K-Means segmentation and a pixel-to-segment loss.
- 1. Introduction: 76% performance of its supervised counterpart is achieved by unsupervised SegSort for semantic segmentation.The unsupervised system creates pseudo masks from HED-based hierarchical contours and trains the embedding network to separate segments.
- 1. Introduction: SegSort is presented as the first end-to-end trained non-parametric supervised approach, exceeding parametric pixel-wise softmax counterparts.The supervised variant incorporates Neighborhood Components Analysis for semantic labels.
- 1. Introduction: The approach produces more precise boundaries, more consistent region segmentations, and interpretable predictions through retrieved nearest segments.Its contributions are demonstrated on PASCAL VOC 2012 and Cityscapes.
2. Related Works
The related work situates SegSort at the intersection of segmentation, clustering, non-parametric semantic parsing, and metric learning. Its distinction is integrating segmentation and clustering into end-to-end embedding learning rather than using clustering only as preprocessing or regularization.
- 2. Related Works: Segmentation methods cluster local-patch representations using mixture models, mode-finding, or graph partitioning, with superpixels often serving as preprocessing or refinement.These approaches motivate, but do not fully solve, consensus segmentation across information levels.
- 2. Related Works: SegSort integrates image segmentation and clustering into end-to-end embedding learning for semantic segmentation.This targets perceptual organization by combining information from different levels into consensus segmentation.
- 2. Related Works: Earlier non-parametric methods addressed unsupervised grouping through boundaries, superpixels, nearest-neighbor search, or dense deformation fields.Supervised work also learned semantic object exemplars for detection and segmentation.
- 2. Related Works: Unlike a related vMF clustering pipeline, SegSort does not rely on softmax loss to produce the final segmentation.The related method uses contrastive clustering only for feature regularization while retaining softmax prediction.
- 2. Related Works: Metric learning provides precedents for retrieval and recognition, while hypersphere normalization motivates using the vMF distribution.SegSort applies these ideas to pixel and segment representations for semantic segmentation.
3. Method
SegSort separates segmentation into pixel sorting and segment sorting: embeddings are clustered into segments, then optimized so pixels approach their segment prototypes while differing from other segments. Supervised alignment and unsupervised metric learning support semantic grouping and nearest-neighbor inference.
- Framework: SegSort uses a CNN, clustering, and metric learning to generate pixel embeddings, partition them into fine segments, and group segments into semantic clusters.The first two stages are called pixel sorting; the third is segment sorting.
- Pixel Sorting: Pixel sorting assumes normalized embeddings within a segment follow a vMF distribution and uses spherical K-Means to estimate segment assignments.The method alternates assignment and parameter updates in an EM framework.
- Pixel Sorting: In supervised training, segments are partitioned by the ground-truth mask so each contains one semantic label, while false-positive and false-negative fragments provide hard negatives.This alignment makes predicted segments conform to semantic boundaries and regularizes boundary learning.
- Segment Sorting: The vMF loss pulls each pixel embedding toward its own segment prototype and pushes it away from other segment prototypes.The loss is averaged over pixels and does not require ground-truth semantic labels.
- Inference: The framework is non-parametric: segment exemplars are organized by vMF clustering, trained with a nearest-neighbor criterion, and used for consistent inference.During inference, segment sorting is replaced by nearest-neighbor retrieval and majority-label prediction.
4. Experiments
Experiments evaluate SegSort on PASCAL VOC 2012 and Cityscapes in supervised and unsupervised settings, using segmentation architectures and visual comparisons. SegSort improves supervised segmentation, supports interpretable retrieval, and produces unsupervised performance reaching 76% of its supervised counterpart.
- Experimental setup: Experiments use PASCAL VOC 2012 and Cityscapes with DeepLabv3+ and PSPNet segmentation architectures.The architectures are powered by MobileNetV2 and ResNet101, respectively, and pretrained on ImageNet.
- Supervised semantic segmentation: SegSort consistently outperforms Softmax by 1.63–2.43% in mIoU and 4.07–7.97% in mean boundary f-measure.The comparison covers fully supervised semantic segmentation on the reported VOC evaluation sets.
- Supervised semantic segmentation: Visual comparisons show improved thin-structure segmentation and more consistent predictions for regions requiring context.Examples include human and chair legs, motorcycle wheels, and bus trunks.
- Interpretability: Nearest-neighbor segment retrieval makes predictions interpretable by showing semantically relevant retrieved segments and revealing confusion in failure cases.A bike number tag is confused with other motorbike number tags or front lights.
- Supervised semantic segmentation: On Cityscapes, SegSort outperforms Softmax consistently, improving both large-object region consistency and small-object capture.Reported examples include buses and trucks as large objects, and poles and traffic lights as small objects.
- Unsupervised semantic segmentation: Unsupervised SegSort uses HED-owt-ucm pseudo-segmentation masks and achieves 76% performance of its supervised counterpart.The contour detector supplies visual boundaries, while feature learning discovers segments with common features without semantic names.
5. Conclusion
SegSort unifies pixel-level and segment-level sorting in an end-to-end metric-learning framework motivated by perceptual organization. It improves supervised semantic segmentation, extends deep learning to unsupervised semantic segmentation, and makes predictions interpretable through retrieved nearest segments.
- SegSort integrates pixel-level and segment-level sorting through von Mises–Fisher clustering in a unified framework.The approach is trained end to end as a pixel-wise metric-learning model.
- The method consistently improves conventional pixel-wise prediction approaches for supervised semantic segmentation.
- SegSort constitutes a first deep-learning attempt at unsupervised semantic segmentation.
- Retrieved nearest segments explain SegSort predictions, including predictions that are incorrect.
6. Supplementary
The supplementary experiments analyze SegSort’s embeddings, clustering, inference behavior, efficiency, boundaries, and Cityscapes predictions. They report stable performance across several settings, manageable computational overhead, and improved region and category behavior relative to Softmax.
- Embedding visualization: Supervised SegSort visualizes 32-dimensional prototype embeddings with t-SNE, while unsupervised SegSort forms clusters primarily from visual similarities such as color and texture.Unsupervised clusters group wheels and animals separately, with grass and sky placed in other regions.
- Efficiency: SegSort’s inference overhead is 7%-22% in runtime and approximately 1.5% in GPU memory compared with Softmax.The runtime cost is attributed mainly to K-Means rather than kNN.
- Ablation study: 25 clusters produce the highest segmentation performance, slightly exceeding the dataset’s number of categories.
- Ablation study: Embedding dimensions larger than 8 do not drastically change performance, and varying nearest-neighbor counts changes mIoU by only 0.4%.
- Cityscapes results: SegSort improves large-object region consistency and captures small objects more effectively in Cityscapes visual comparisons.Reported examples include buses and trucks among large objects, and poles and traffic lights among small objects.
- Cityscapes results: On the Cityscapes test set, networks trained with SegSort consistently outperform Softmax.
7. Per-category results on VOC
VOC experiments report category-specific gains for SegSort across architectures and backbones. The reported improvements include better fine-structure or small-object detection and consistent gains over Softmax.
- Per-category results: SegSort with DeepLabv3+ / MobileNetV2 captures finer structures such as bikes and motorbikes, while PSPNet / ResNet-101 improves boat and plant detection.
- Backbone comparison: Using DeepLabv3+ with a ResNet-101 backbone, SegSort shows consistent performance improvements over Softmax under matched hyper-parameters.The authors note that the hyper-parameters may not be optimal for this backbone.