Source-linked AI summary
Deep Watershed Transform for Instance Segmentation
Min Bai, Raquel Urtasun
TL;DR
Instance segmentation requires both semantic labeling and assignment of pixels to physical objects, while existing solutions often use complex pipelines. This paper learns a watershed-inspired energy landscape with a feed-forward convolutional network, extracting instances by one energy threshold. It reports more than doubled state-of-the-art performance on Cityscapes, while retaining limitations around occlusion-separated objects, semantic-segmentation errors, and complex scenes.
Problem
Instance segmentation must identify each pixel’s semantic class and physical object instance despite challenging street-scene variation, while existing approaches commonly use complex pipelines.
Method
A feed-forward convolutional network learns a watershed energy landscape whose single-instance basins are separated by equal-height ridges, enabling extraction with one energy-level cut.
Results
More than double the performance of the current state-of-the-art on the challenging Cityscapes Instance Segmentation benchmark.
Takeaways & Limitations
The method provides an end-to-end, fast, accurate, non-iterative approach with runtime constant regardless of the number of object instances.
Takeaways & Limitations
The method does not handle objects split into multiple pieces by occlusion, cannot correct semantic-segmentation errors, and can incorrectly separate or fuse complex scenes.
Abstract
from arXiv · showhide
Most contemporary approaches to instance segmentation use complex pipelines involving conditional random fields, recurrent neural networks, object proposals, or template matching schemes. In our paper, we present a simple yet powerful end-to-end convolutional neural network to tackle this task. Our approach combines intuitions from the classical watershed transform and modern deep learning to produce an energy map of the image where object instances are unambiguously represented as basins in the energy map. We then perform a cut at a single energy level to directly yield connected components corresponding to object instances. Our model more than doubles the performance of the state-of-the-art on the challenging Cityscapes Instance Level Segmentation task.
1. Introduction
Instance segmentation must assign both semantic classes and object instances despite challenging street-scene variation. The paper proposes a simple deep watershed approach that learns an energy landscape for direct instance extraction and reports more than doubled state-of-the-art performance on Cityscapes.
- Instance segmentation assigns each pixel both a semantic class and a physical object instance, unlike semantic segmentation alone.
- Street-scene instance segmentation is difficult because object scale varies widely and appearances are affected by occlusion, specularities, saturation, and motion blur.
- Existing methods commonly rely on complex pipelines involving object proposals, CRFs, RNNs, or template matching.
- The proposed method learns a watershed energy map whose basins represent single instances and whose equal-height ridges enable extraction with one energy-level cut.
- The approach is end-to-end, fast, and non-iterative, with runtime constant regardless of the number of object instances.
- More than double the performance of the current state-of-the-art on the challenging Cityscapes Instance Segmentation benchmark.
2. Related
Related work includes proposal-based systems and deep structured models that combine CNN predictions with global reasoning for instance-level consistency.
- Proposal based: Proposal-based approaches refine object-segment proposals, select them using CNN features and suppression, or process boxes and masks through cascaded networks.
- Proposal based: Some proposal methods explicitly reason about multiple proposals to handle occlusions that split one object into disconnected patches.
- Deep structured models: Deep structured models use CNNs for local instance disambiguation and labeling, followed by CRFs for global instance-label consistency.
3. A Review on the Watershed Transform
The classical watershed transform treats a grayscale image as a topographic surface and partitions it into catchment basins separated by watershed lines. Although intuitive, it can over-segment because small energy perturbations create spurious ridges; the paper motivates learning an alternative energy landscape directly.
- The watershed transform floods a topographic surface from minima while barriers prevent waters from merging, producing catchment basins and watershed boundaries.
- In the conceptual one-dimensional example, the traditional transform produces seven colored components even though the image contains three main components.
- Small perturbations in the energy create spurious ridges, causing the traditional watershed transform to over-segment images.
- Watershed components can be estimated using iterative flooding with dams or alternatives including topological distance, shortest paths, spanning trees, and markers.
- The transform is typically applied to image gradients, but estimating sharp instance-separating gradients is challenging.
4. Deep Watershed Tranform
The method learns a watershed energy landscape with one basin per instance and equal-height dividing ridges, enabling a single-threshold cut without over-segmentation. A direction network and watershed network are pretrained before end-to-end fine-tuning.
- Deep Watershed Transform: The feed-forward network directly learns a watershed energy landscape whose basins represent individual instances and whose ridges share one energy height.This makes the watershed cut a single energy threshold that avoids over-segmentation.
- Deep Watershed Transform: The method uses an intermediate direction task because learning the energy landscape from scratch is complex.The predicted direction map is passed through further layers to produce the final energy.
- Network Inputs: The network uses RGB input gated by semantic segmentation, with the segmentation appended as a fourth channel; it is agnostic to the segmentation algorithm.The paper uses PSPNet semantic segmentation results.
- Direction Network (DN): The Direction Network predicts a normalized distance-transform gradient as a two-channel unit vector pointing away from the nearest instance boundary.Unit-vector targets avoid the numerical discontinuity between equivalent 0° and 360° angles.
- Watershed Transform Network (WTN): The Watershed Network converts the direction map into K = 16 discretized energy values, assigning higher energies to object interiors and emphasizing resolution near zero.Bin 0 represents background or regions within 2 pixels of an instance boundary.
- Network Training: The two subnetworks are pretrained separately and then fine-tuned end-to-end using RGB images, semantic segmentation outputs, and ground-truth distance transforms.Direction pre-training uses weighted mean squared angular error, while watershed pre-training uses modified cross-entropy.
5. Experimental Evaluation
The evaluation tests the approach on Cityscapes using leaderboard metrics, complementary coverage evaluation, ablations, qualitative results, and failure-mode analysis. Results show strong performance, sensitivity to semantic segmentation quality, and specific limitations involving occlusion and semantic-label errors.
- Experimental setup: The evaluation uses Cityscapes instance-level segmentation, including official test and validation results plus validation-set ablations.Cityscapes contains finely annotated street scenes with many instances, occlusions, and varied object scales.
- Metrics: AP evaluates instance rankings across IoU thresholds from 50% to 95%, but can disadvantage methods assigning one instance label per pixel.The authors also report mean weighted coverage, which enforces a single instance label per pixel and is described as more suitable for this approach.
- Comparison to the state-of-the-art: The method achieves a large improvement over the state of the art across all semantic classes without using depth information for training.The cited comparison reports test-set scores and class-specific AP scores in Tables 1 and 2, but no numerical values are supplied here.
- Qualitative analysis: The model retains direction prediction after fine-tuning, while qualitative validation examples show high-quality results using a 50% IoU color-matching threshold.The validation set used for qualitative visualization was not part of training.
- Ablation studies: Better semantic segmentation improves instance-segmentation performance, indicating that the approach benefits from advances in its semantic-segmentation source.The model was evaluated with alternative semantic-segmentation sources, including LRR instead of PSPNet.
- Ablation studies: Oracle instance ordering increases AP by 6.34% without changing the quality of proposed object instances, exposing a limitation of the AP metric.The authors argue that metrics such as mean weighted coverage are needed to evaluate segmentation-based approaches fairly.
- Failure modes: Occlusion can split one object into multiple components, and complex occlusion boundaries can leave neighboring objects insufficiently separated.The authors anticipate that combining the method with top-down reasoning could alleviate these issues; semantic-segmentation errors also cannot be corrected by the method.
6. Conclusion
The paper proposes a watershed-inspired instance segmentation technique that generates a modified energy landscape and directly extracts high-quality object instances. Experiments show more than doubled performance on the challenging Cityscapes Instance Segmentation task.
- The model generates a modified watershed energy landscape from which high-quality object instances are directly extracted.The technique is inspired by the classical watershed transform and uses a deep convolutional neural network with specialized loss functions.
- The experiments more than double state-of-the-art performance on the challenging Cityscapes Instance Segmentation task.
- Predicted and ground-truth instances share a color only when their intersection-over-union exceeds 50%.