Source-linked AI summary
Instance Segmentation by Jointly Optimizing Spatial Embeddings and Clustering Bandwidth
Davy Neven, Bert De Brabandere, Marc Proesmans, Luc Van Gool
TL;DR
Instance segmentation seeks accurate masks at real-time speed, but proposal-based methods are slow and low-resolution while proposal-free methods often sacrifice accuracy. This paper introduces a proposal-free clustering loss that learns object-specific margins and directly optimizes mask intersection-over-union. On Cityscapes, it reports top results at more than 10 fps, including 27.6 Average Precision versus 26.2 for Mask R-CNN.
Problem
Proposal-based methods are accurate but slow and low-resolution, whereas proposal-free methods are faster and high-resolution but often less accurate for real-time instance segmentation.
Method
The method learns object-specific clustering margins while pulling pixels toward instance centers and directly optimizing each mask’s intersection-over-union.
Results
27.6 versus 26.2 Average Precision at more than 10 fps on Cityscapes reports top results surpassing Mask R-CNN in real time.
Takeaways & Limitations
The method is reported as the first proposal-free Cityscapes approach combining real-time execution with high accuracy.
Takeaways & Limitations
Truck, bus, and train classes are highly underrepresented in Cityscapes, negatively affecting test performance on those classes.
Abstract
from arXiv · showhide
Current state-of-the-art instance segmentation methods are not suited for real-time applications like autonomous driving, which require fast execution times at high accuracy. Although the currently dominant proposal-based methods have high accuracy, they are slow and generate masks at a fixed and low resolution. Proposal-free methods, by contrast, can generate masks at high resolution and are often faster, but fail to reach the same accuracy as the proposal-based methods. In this work we propose a new clustering loss function for proposal-free instance segmentation. The loss function pulls the spatial embeddings of pixels belonging to the same instance together and jointly learns an instance-specific clustering bandwidth, maximizing the intersection-over-union of the resulting instance mask. When combined with a fast architecture, the network can perform instance segmentation in real-time while maintaining a high accuracy. We evaluate our method on the challenging Cityscapes benchmark and achieve top results (5\% improvement over Mask R-CNN) at more than 10 fps on 2MP images. Code will be available at https://github.com/davyneven/SpatialEmbeddings .
1. Introduction
Instance segmentation needs accurate, high-resolution masks at real-time speed, but proposal-based and proposal-free approaches each fall short on one of these goals. The paper introduces a proposal-free loss that learns object-specific clustering regions and achieves strong real-time Cityscapes results.
- Instance segmentation locates objects, assigns classes, and generates pixel-perfect masks important for autonomous driving, robotics, and image editing.
- Proposal-based methods such as Mask R-CNN provide accurate masks but produce low-resolution outputs and operate too slowly for real-time applications.
- Proposal-free methods offer high-resolution masks and often faster runtimes, yet generally underperform detect-and-segment methods such as Mask R-CNN.
- The proposed loss pulls pixels toward object centers, optimizes each mask’s intersection-over-union, and learns an object-specific clustering region.
- 27.6 versus 26.2 Average Precision at more than 10 fps shows top Cityscapes results surpassing Mask R-CNN in real time.
2. Related Work
Related work includes proposal-based pipelines that refine detector outputs into masks and dense-prediction methods that associate pixels through embeddings or object-center offsets. The proposed method follows center-pointing while integrating clustering into the loss and directly optimizing mask intersection-over-union.
- Proposal-based methods rely on Faster R-CNN detections and refine object proposals into instance masks, typically at low resolution.
- Dense-prediction approaches generate input-resolution masks using embedding losses that separate pixels from different objects.
- Center-pointing methods assign pixels to object centers through position-relative offset vectors, avoiding spatial-invariance problems in fully convolutional networks.
- Unlike related approaches, this method integrates post-processing clustering into the loss and directly optimizes each instance mask’s intersection-over-union.
- Box2Pix predicts bounding boxes first, then associates pixels with object centers and clusters them efficiently for real-time segmentation.
3. Method
The method treats instance segmentation as pixel assignment through spatial embeddings, learnable instance-specific margins, and seed-based center localization. A Lovasz-hinge objective integrates clustering with mask intersection-over-union optimization, while elliptical margins and learned attraction centers provide extensions.
- Core pipeline: The network learns offset vectors so each pixel’s spatial embedding points toward its object center, treating instance segmentation as pixel assignment.Embeddings combine pixel coordinates with learned offsets; seed maps help locate object centers.
- Core pipeline: Previous centroid-based methods require separate center localization and pixel assignment through density-based clustering, preventing end-to-end optimization.The paper identifies this post-processing step as a source of inferior results.
- Learnable margin: The method replaces fixed-margin assignment with a learnable instance-specific margin, using smaller regions for small objects and larger regions for big objects.This relaxes the loss for pixels farther from large-instance centroids instead of forcing every pixel to point exactly at the centroid.
- Learnable margin: A Gaussian maps embedding-to-centroid distance into foreground probability, assigning a pixel to instance k when φk(ei) > 0.5.The sigma parameter controls the margin: larger sigma produces a larger region, while smaller sigma produces a smaller one.
- Learnable margin: The Lovasz-hinge loss optimizes each instance’s intersection-over-union directly from Gaussian foreground/background maps.The loss jointly optimizes sigma and offset outputs through backpropagation rather than direct supervision of those outputs.
- Loss extensions: The formulation can learn elliptical margins with two-dimensional sigma values and can alternatively learn a more optimal center of attraction from instance embeddings.Elliptical margins may better fit elongated objects such as pedestrians or trains; these modifications are evaluated in ablations.
4. Experiments
Experiments on Cityscapes evaluate the architecture, dataset setup, ablations, benchmark accuracy, and execution speed. The method reaches 27.6 AP at 11 fps while its learnable components improve instance segmentation performance.
- Dataset and setup: The two-branch ERFNet architecture predicts spatial offsets and sigma values in one decoder while producing one seed map per semantic class in the other.ERFNet is a dense-prediction encoder-decoder designed for real-time semantic segmentation.
- Dataset and setup: The evaluation uses Cityscapes fine annotations, which contain 5,000 images at 2048x1024 resolution and span challenging object sizes and scene layouts.Only the fine train set is used, with truck, bus, and train substantially underrepresented.
- Ablation experiments: A learnable instance-specific sigma improves validation performance from 28 AP with fixed sigma to 38.7 AP.The fixed sigma corresponds to a 20-pixel margin and must be selected using the smallest separable object.
- Ablation experiments: Learnable centers of attraction outperform fixed centroids across classes, while two-dimensional sigma produces more accurate elliptical margins than scalar sigma.Elliptical margins can adapt better to rectangular objects such as pedestrians.
- Ablation experiments: The learned margin has a positive correlation with object size, indicating that larger objects receive larger clustering margins.Figure 4 plots one learned margin value for each dataset object against its size.
- Cityscapes results and timing: On Cityscapes, the method obtains 27.6 AP, compared with 26.2 for Mask R-CNN, and reports stronger person, rider, and car results than that baseline.It achieves 27.6 AP at 11 fps; the forward pass takes 65 ms and clustering takes 26 ms at 2MP resolution.
5. Conclusions
The paper proposes a clustering loss that directly optimizes instance-mask intersection-over-union and learns object-specific clustering margins. Applied to a real-time dense-prediction network, it achieves top Cityscapes results at more than 10 fps.
- The proposed clustering loss converts pixel embeddings into foreground/background probabilities and directly optimizes each instance mask’s intersection-over-union.
- The method learns an optimal, object-specific clustering margin for each instance.
- More than 10 fps on Cityscapes demonstrates top benchmark results with a real-time dense-prediction network.