Source-linked AI summary
InstanceCut: from Edges to Instances with MultiCut
Alexander Kirillov, Evgeny Levinkov, Bjoern Andres, Bogdan Savchynskyy, Carsten Rother
TL;DR
Instance segmentation requires assigning pixels to individual objects, a harder refinement of semantic segmentation. InstanceCut combines semantic segmentation and instance-aware boundaries with global MultiCut reasoning, achieving the best published CityScapes result and particularly strong performance for rare classes.
Problem
Instance segmentation must separate individual objects within semantic classes, a harder task than semantic segmentation with importance for detailed scene understanding in autonomous driving and robotics.
Method
InstanceCut combines an instance-agnostic semantic-segmentation CNN, an instance-aware edge-detection CNN, and global MultiCut reasoning to partition images into instances.
Results
The framework achieves the best result among all published methods on the challenging CityScapes dataset and performs particularly well for rare object classes.
Takeaways & Limitations
The approach offers a different modelling paradigm whose instance count arises from global partitioning and whose rare classes require no special treatment.
Takeaways & Limitations
InstanceCut cannot find instances formed by disconnected regions in the image.
Abstract
from arXiv · showhide
This work addresses the task of instance-aware semantic segmentation. Our key motivation is to design a simple method with a new modelling-paradigm, which therefore has a different trade-off between advantages and disadvantages compared to known approaches. Our approach, we term InstanceCut, represents the problem by two output modalities: (i) an instance-agnostic semantic segmentation and (ii) all instance-boundaries. The former is computed from a standard convolutional neural network for semantic segmentation, and the latter is derived from a new instance-aware edge detection model. To reason globally about the optimal partitioning of an image into instances, we combine these two modalities into a novel MultiCut formulation. We evaluate our approach on the challenging CityScapes dataset. Despite the conceptual simplicity of our approach, we achieve the best result among all published methods, and perform particularly well for rare object classes.
1. Introduction
Instance-aware semantic segmentation assigns pixels to individual objects rather than only semantic classes. InstanceCut explores a different paradigm that combines semantic segmentation, instance boundaries, and global partitioning, with strong reported performance despite a connected-component limitation.
- Instance segmentation assigns each semantic-class object separately, supporting applications such as autonomous driving and robotics that require detailed scene understanding.
- InstanceCut addresses variable instance counts, instance-specific labels, pixel-level boundaries, and limited data for rare classes through a different modelling paradigm.
- The method combines an instance-agnostic semantic-segmentation CNN with a new CNN for detecting all instance boundaries.
- InstanceCut cannot find instances formed by disconnected image regions, although the authors report accuracy exceeding all published methods.
- The approach handles rare classes without special treatment and can use any semantic-segmentation method that provides pixel-wise log-probabilities.
- A novel MultiCut formulation reasons globally about the optimal partitioning of an image into instances.
2. Related Work
Related work includes proposal-based, proposal-free, recurrent, clustering, and conditional-random-field approaches to instance segmentation. These methods differ in their dependencies, fixed-instance assumptions, and handling of occlusion or proposal quality.
- Proposal-based methods: Proposal-based methods use detection or proposal generation as a subroutine, but their performance is limited by the quality of the detector or proposal generator.
- Proposal-based methods: Some methods combine detection with foreground/background segmentation, often using end-to-end training and optionally non-maximal suppression.
- Proposal-based methods: Other proposal methods combine learned-scored proposals or generated instance parts into final segmentations.
- Proposal-free methods: Alternative approaches require depth data, use fixed maximum instance counts, filter per-pixel bounding-box predictions, or generate instances sequentially with recurrent networks.
- Proposal-free methods: A proposal-free method predicts semantic segmentation, instance counts, and per-pixel bounding boxes before clustering, making it highly sensitive to the predicted number of instances.
- Proposal-free methods: InstanceCut is also proposal-free, but combines semantic segmentation and object-boundary detection through global reasoning.
3. InstanceCut
InstanceCut combines independent semantic segmentation and instance-aware edge detection with a global image-partitioning formulation. Semantic scores and class-agnostic boundary scores are converted into superpixels and jointly partitioned into labeled instance components.
- 3.1. Overview of the proposed framework: Two independent branches produce per-pixel semantic class scores and class-agnostic instance-aware edge scores from the input image.The semantic branch predicts log-probabilities for each class and pixel, while the edge branch predicts boundary likelihoods.
- 3.1. Overview of the proposed framework: The final partitioning block merges superpixels into connected components and assigns each component a class label, yielding object instances and background.The formulation combines the semantic-labeling and MultiCut subproblems into a joint optimization.
- 3.1. Overview of the proposed framework: Watershed applied to instance-aware edge scores extracts superpixels whose boundaries align with predicted object boundaries.A superpixel representation reduces the optimization problem size; semantic scores are averaged over corresponding pixels.
- 3.3. Instance-Aware Edge Detection: Instance-aware edge detection distinguishes true object boundaries from irrelevant edges by jointly reasoning about contours and semantics.Its network uses intermediate feature maps from a semantic-segmentation FCN, downsamples them to the smallest-map size, concatenates them, and refines perceptron outputs with a context network.
- 3.4. Image Partition: The MultiCut formulation globally reasons about partitioning an image into instances using semantic labels and boundary cuts.Cycle constraints ensure that cut edges define a valid partition, while additional constraints force differently labeled neighboring nodes to be separated.
4. Experiments
The experiments evaluate InstanceCut on CityScapes using qualitative scenes, benchmark metrics, and comparisons with published methods. Performance depends on the semantic segmentation backbone, and the framework achieves the best published test-set result despite its simplicity.
- Dataset: CityScapes provides 5000 street-scene images, eight instance-annotated object classes, and precise object boundaries suited to an edge-based method.The dataset includes 2975 training, 500 validation, and 1525 test images at 1024 × 2048 resolution.
- Qualitative results: Qualitative results visualize input ground-truth instances, instance-aware edge log-probabilities, and the resulting InstanceCut segmentation.In one example, a lamp-post separates a bus and a car, and the method returns two instances.
- Validation results: 14.8 AP and 30.7 AP50% are obtained with Dilation10 on the validation set.
- Validation results: 15.8 AP and 32.4 AP50% are obtained when LRR-4x replaces Dilation10 for semantic segmentation on the validation set.
- Test-set comparison: The method shows the best performance among published methods on the CityScapes test set, with unpublished entries excluded from the comparison.
5. Conclusion
The conclusion presents InstanceCut as an alternative paradigm for instance-aware semantic segmentation. It combines semantic segmentation and instance-aware boundaries through a MultiCut framework and reports the best published CityScapes results.
- InstanceCut represents instance segmentation with instance-agnostic semantic segmentation and instance-aware boundaries.
- Standard FCN networks produce semantic segmentation, while a new architecture produces object boundaries.
- A novel MultiCut framework combines both modalities and reasons globally about instances.
- InstanceCut achieves the best results among published methods on the challenging CityScapes dataset.
Supplementary materials
The supplementary material presents difficult scenes, failure cases, and class-specific CityScapes test-set results. Failures concentrate on small distant objects, heavily occluded people, and occluded instances with disconnected visible parts.
- Difficult scenes: Difficult scenes show cases where InstanceCut predicts most instances correctly.
- Failure cases: Failure cases are attributed to small distant objects, heavily mutually occluded nearby people, and occluded instances with several disconnected visible parts.
- Class-level results: Table 2 reports CityScapes test-set instance-aware semantic segmentation results separately for each semantic class.