Source-linked AI summary
Semantic Instance Segmentation with a Discriminative Loss Function
Bert De Brabandere, Davy Neven, Luc Van Gool
TL;DR
Instance segmentation is difficult to adapt from semantic segmentation because images can contain arbitrarily many instances with permutation-invariant labels. The paper reuses an off-the-shelf architecture with a discriminative loss whose pixel embeddings are clustered into instances, achieving competitive benchmark performance with a simple setup.
Problem
Applying semantic-segmentation architectures and softmax cross-entropy directly to instance segmentation is unclear because images may contain arbitrarily many instances with permutation-invariant labels.
Method
The method retrains an off-the-shelf semantic-segmentation architecture using a discriminative loss that pulls same-instance pixel embeddings together and pushes different-instance means apart for clustering.
Results
The method achieves competitive performance on the Cityscapes and CVPPP benchmarks; on CVPPP it obtains an SBD score of 84.2 versus 84.9 for the state of the art.
Takeaways & Limitations
A simple thresholding post-processing step can convert the network output into discrete instances, and the method can handle complex occlusions.
Takeaways & Limitations
The holistic approach performs worse on datasets with objects in random constellations and diverse settings, where sliding-window detection is more suited.
Abstract
from arXiv · showhide
Semantic instance segmentation remains a challenging task. In this work we propose to tackle the problem with a discriminative loss function, operating at the pixel level, that encourages a convolutional network to produce a representation of the image that can easily be clustered into instances with a simple post-processing step. The loss function encourages the network to map each pixel to a point in feature space so that pixels belonging to the same instance lie close together while different instances are separated by a wide margin. Our approach of combining an off-the-shelf network with a principled loss function inspired by a metric learning objective is conceptually simple and distinct from recent efforts in instance segmentation. In contrast to previous works, our method does not rely on object proposals or recurrent mechanisms. A key contribution of our work is to demonstrate that such a simple setup without bells and whistles is effective and can perform on par with more complex methods. Moreover, we show that it does not suffer from some of the limitations of the popular detect-and-segment approaches. We achieve competitive performance on the Cityscapes and CVPPP leaf segmentation benchmarks.
1. Introduction
Semantic instance segmentation assigns a separate labeled mask to every object, but arbitrary instance counts and permutation-invariant labels complicate direct use of semantic-segmentation losses. The paper proposes replacing the pixel-wise loss while reusing standard architectures and clustering pixel embeddings afterward.
- Task and challenge: Instance segmentation produces a binary mask and semantic label for each individual object, making it harder than semantic segmentation when same-class objects overlap.It is also closely related to object counting and detection.
- Task and challenge: Arbitrary instance counts and permutation-invariant labels complicate applying softmax cross-entropy directly to instance segmentation.An instance label matters only relative to the other instance labels in the image.
- Prior approaches: Proposal-based, recurrent, and representation-clustering pipelines are prominent alternatives, while this method belongs to the clustering category.The authors position their approach as more principled than earlier clustering methods.
- Proposed approach: The discriminative loss maps pixels from the same instance near one another in feature space and separates pixels from different instances.A fast post-processing step clusters the resulting representation into instances.
- Proposed approach: The method plugs an off-the-shelf semantic-segmentation architecture into the framework and retrains it without architectural changes.This treats semantic and instance segmentation consistently while focusing the contribution on the loss function.
- Evaluation: Experiments on two instance-segmentation benchmarks report performance competitive with the state of the art.The paper evaluates Cityscapes and CVPPP leaf segmentation.
2. Related Work
Prior instance-segmentation research uses proposal-based pipelines, recurrent networks, or representations clustered into instances. This paper is closest to the clustering line but uses a less ad-hoc loss and representation design.
- Proposal-based: Proposal-based methods use separate object-proposal and classification stages, often forming multi-stage pipelines.Examples include category-independent region proposals, segmentation proposals, and cascades of networks.
- Recurrent methods: Recurrent methods generate individual instances sequentially with recurrent networks and custom permutation-invariant losses.Some approaches use the Hungarian algorithm to match candidate hypotheses with ground-truth instances.
- Clustering: Clustering methods transform images into representations that are subsequently clustered into discrete instances.Prior representations include segmentation trees, depth ordering, and pixel-wise vectors for instance bounding boxes.
- Clustering: This work also separates representation learning from clustering, but its loss does not impose instance centers, depth ordering, or another specific representation.The authors describe this as less ad hoc than those alternatives.
- Other methods: Other approaches learn watershed energies or combine semantic segmentations with edge maps, with reported shortcomings for occlusions or connected-region extraction.The cited methods differ in how they reason about and separate instances.
- Loss function: The proposed loss is connected to distance-metric learning, discriminative losses, siamese networks, and triplet loss.These works motivate learning feature relationships rather than fixed instance representations.
3. Method
The method learns pixel embeddings that pull same-instance pixels together and push different instances apart, then clusters the embeddings with simple post-processing. Its hinged, center-based loss avoids fixed instance labels and supports a feature-space dimension independent of instance count.
- Pixel embeddings map same-instance pixels close together and different-instance pixels far apart in feature space.
- The loss replaces pairwise penalties between differently labeled embeddings with penalties between their mean embeddings, reducing computation when instances are far fewer than pixels.
- The loss uses intra-cluster attraction, inter-cluster repulsion, and regularization toward the origin.The variance term pulls embeddings toward cluster centers, the distance term separates centers, and the regularization term bounds activations.
- The variance and distance forces are hinged, allowing embeddings within δv of a center and centers beyond 2δd to move freely.This relaxes constraints and permits clusters to occupy local manifolds rather than converging to a single point.
- The combined objective is L = α · Lvar + β · Ldist + γ · Lreg, minimized by stochastic gradient descent with α = β = 1 and γ = 0.001.
- When δd > δv, inference can threshold embeddings within bandwidth b = δv around a cluster center to select that cluster.For permutation-invariant instance labels, an alternative uses δd > 2δv so any embedding can serve as a threshold center.
4. Experiments
Experiments evaluate the method on CVPPP leaf segmentation and Cityscapes instance segmentation, then analyze semantic labels, clustering, and network trade-offs. Results are competitive, while semantic segmentation quality is a major performance bottleneck.
- Datasets: The method is tested on CVPPP leaf segmentation and Cityscapes instance-level semantic labeling benchmarks.The datasets contain a median number of more than 15 instances per image.
- Datasets: CVPPP evaluation uses Symmetric Best Dice for instance segmentation accuracy and Absolute Difference in Count for counting error.The CVPPP A1 subset contains 128 labeled images and 33 test images.
- Results: On CVPPP, the method achieves an SBD score of 84.2, compared with 84.9 for the state of the art.It outperforms all non-deep learning methods and the recurrent instance segmentation method cited as [31].
- Results: On Cityscapes, the method achieves competitive leaderboard results, outperforming all but one unpublished work and performing on par with the MNC-based method SAIS.Evaluation reports AP, AP0.5, AP50m, and AP100m.
- Component analysis: Replacing predicted semantic segmentation with ground truth produces the largest performance increase, while replacing mean-shift clustering has a smaller but measurable effect.The authors attribute the semantic-segmentation influence partly to average precision being averaged over semantic classes.
- Speed-accuracy trade-off: ResNet-38 provides the best accuracy but requires more memory; ENet is much faster than SegNet with almost the same accuracy, and post-processing adds negligible overhead.Increasing resolution beyond 768x384 does not substantially improve accuracy for the tested networks.
5. Conclusion
The paper proposes a discriminative loss for instance segmentation and shows that its clustered output supports competitive benchmark performance and complex-occlusion handling. The approach still relies on pretrained semantic segmentation masks, with joint training left for future work.
- The paper proposes a discriminative loss function for instance segmentation.
- The method clusters network outputs into discrete instances using a simple post-processing thresholding operation tailored to the loss.
- The method handles complex occlusions and achieves competitive performance on the Cityscapes and CVPPP benchmarks.
- The current method uses a pretrained network to produce semantic segmentation masks, while joint instance-and-semantic training is reserved for future work.