PaperlayerView exact revision on arXiv

Source-linked AI summary

Pixelwise Instance Segmentation with a Dynamically Instantiated Network

Anurag Arnab, Philip H. S Torr

arXiv:1704.02386v1cs.CV

TL;DR

Instance segmentation must assign both semantic classes and instance identities at pixel level, beyond the capabilities of semantic segmentation and bounding-box detection. The paper uses an initial semantic segmentation module followed by a dynamically instantiated instance subnetwork with detector cues and an end-to-end CRF. It reports sharper, more accurate segmentations, including substantial high-threshold improvements, while producing outputs without post-processing and improving semantic segmentation during instance-segmentation training.

  • Problem

    Semantic segmentation lacks instance identity, while object detection localizes objects only at a coarse bounding-box level, motivating pixelwise instance segmentation.

  • Method

    An initial semantic segmentation module feeds a dynamically sized instance subnetwork that combines detector outputs and semantic cues through an end-to-end CRF.

  • Results

    The system produces sharper, more accurate instance segmentations and substantial improvements over state-of-the-art methods at high AP r thresholds on Pascal VOC and Semantic Boundaries.

  • Takeaways & Limitations

    The approach naturally produces complete segmentation maps in a single forward pass, without post-processing, while improving semantic segmentation during instance-segmentation training.

  • Takeaways & Limitations

    The object detector is not incorporated into the system’s end-to-end training, and the shape term currently matches only a single shape template per instance.

Abstract

from arXiv · show

Semantic segmentation and object detection research have recently achieved rapid progress. However, the former task has no notion of different instances of the same object, and the latter operates at a coarse, bounding-box level. We propose an Instance Segmentation system that produces a segmentation map where each pixel is assigned an object class and instance identity label. Most approaches adapt object detectors to produce segments instead of boxes. In contrast, our method is based on an initial semantic segmentation module, which feeds into an instance subnetwork. This subnetwork uses the initial category-level segmentation, along with cues from the output of an object detector, within an end-to-end CRF to predict instances. This part of our model is dynamically instantiated to produce a variable number of instances per image. Our end-to-end approach requires no post-processing and considers the image holistically, instead of processing independent proposals. Therefore, unlike some related work, a pixel cannot belong to multiple instances. Furthermore, far more precise segmentations are achieved, as shown by our state-of-the-art results (particularly at high IoU thresholds) on the Pascal VOC and Cityscapes datasets.

1. Introduction

Instance segmentation combines semantic class labeling with pixel-level instance identity, addressing the coarse localization of detection and the instance blindness of semantic segmentation. The proposed end-to-end system reasons over the whole image and achieves especially strong high-threshold results without post-processing.

  • Semantic segmentation labels every pixel by class but cannot distinguish same-class instances, whereas detection localizes instances only with coarse bounding boxes.
  • The proposed network starts with semantic segmentation, then dynamically predicts a variable number of instances using detector cues and a CRF.
  • Unlike proposal-based approaches, the method considers the entire image, attempts to resolve occlusions, and produces a complete segmentation map without post-processing.
  • The system uses object-detector outputs as instance cues while remaining robust to false positives, poor localization, and occlusions.
  • The method produces sharp instance segmentations and improves over state-of-the-art methods at high AP r thresholds on Pascal VOC and Semantic Boundaries.

2. Related Work

Prior instance-segmentation methods commonly refine object or region proposals independently, limiting holistic reasoning and occlusion handling. This work instead combines semantic segmentation and detector outputs in an end-to-end CRF that supports variable instance counts and avoids post-processing.

  • Early methods used per-pixel part classifiers or detector outputs with CRFs and probabilistic depth ordering to model instance structure and occlusions.
  • Detection-based systems classify region or box proposals and refine them into masks, but often rely on independent proposals and additional post-processing.
  • Global methods reason about all instances simultaneously, but prior examples used non-end-to-end graphical-model inference or spectral clustering.
  • The proposed network combines an initial semantic segmentation module with detector outputs and a final CRF to obtain instance segmentation.
  • Compared with earlier related work, the method is end-to-end trained, handles localization errors and occlusions, requires no post-processing, and supports a variable number of instances.

3. Proposed Approach

The proposed network combines semantic segmentation and object-detection cues in a fully differentiable instance CRF. Its dynamic label space and complementary unary and pairwise terms support variable instance counts, localization errors, false positives, and same-class occlusions.

  • The network combines semantic predictions and object-detector outputs to compute unary potentials in a CRF over object instances.
  • The instance subnetwork dynamically uses D detection-based labels plus background, so the label set varies with each image.
  • The Box term uses bounding-box membership, semantic-class probability, and detection confidence to assign pixels to detected instances.
  • Pairwise term: Densely connected pairwise Gaussian potentials encourage appearance and spatial consistency, resolving some same-class occlusions when instances differ in appearance.
  • Global term: The Global term uses semantic predictions across the entire image, helping recover object regions beyond poorly localized detection boxes.It assigns probability mass to an instance label throughout all image pixels and produces more stable training gradients.
  • Shape term: The Shape term matches warped templates to segmentation predictions and combines the selected prior with segmentation unaries to distinguish similar same-class objects.The implementation initializes roughly 250 templates for each of five aspect ratios; only one template is matched per proposed instance.

3.3. Inference of our Dynamic Instance CRF

Inference approximately minimizes the CRF Gibbs energy with differentiable mean-field inference implemented as an unrolled recurrent layer. Dynamic instantiation and shared weights accommodate a variable number of instance labels.

  • Mean-field inference approximately minimizes the Gibbs energy and computes a MAP labeling for the instance CRF.
  • Unrolling mean-field iterations as a recurrent neural network makes CRF inference differentiable and enables end-to-end training.
  • The CRF is dynamically instantiated with a different number of labels for each image because the number of instances varies.
  • The model uses no class-specific weights because instance labels have no fixed semantic meaning, enabling weight sharing across variable-length inputs.

3.4. Loss Function

Training resolves the symmetry of interchangeable instance labels by matching predicted and ground-truth segments before applying segmentation loss. The matching is efficiently computed as bipartite assignment.

  • The matched ground truth is the permutation maximizing IoU with the prediction before loss computation.
  • The prediction may contain a different number of segments from ground truth, and each predicted or true segment carries an instance and semantic class label.
  • After matching, the network uses cross-entropy loss, which performed better than the approximate IoU loss evaluated by the authors.
  • Maximum-weight bipartite matching avoids evaluating every ground-truth permutation, using class-consistent IoU edge weights and dummy nodes for leftovers.
  • Detector-determined prediction ordering remains relatively stable during training, changing mainly when multiple detections overlap one object.

3.5. Network Training

Training first pretrains the semantic segmentation module, then appends the instance subnetwork and fine-tunes the complete system with instance annotations and the specified loss.

  • The semantic segmentation network is first trained with standard cross-entropy loss before the instance subnetwork is appended.The semantic module uses FCN8s with a CRF whose inference is unrolled as an RNN and trained end-to-end.
  • The appended network is fine-tuned using instance-segmentation annotations and only the loss defined for the instance task.

3.6. Discussion

The network produces semantic and instance segmentations in one forward pass without post-processing, while its CRF reasons over the entire image rather than independent proposals. Separating the segmentation network from the object detector provides robustness to detector errors and supports more stable learning.

  • A single forward pass computes both semantic and instance segmentation without patch aggregation, mask-voting, superpixel projection, or spectral clustering.
  • The CRF reasons about the entire image at once instead of processing independent object proposals.
  • The segmentation network remains robust to object-detector errors and poorly localized detections.
  • Using a separately trained object detector avoids shared failure cases and prevents frequent instance-label switching during training, making learning more stable.
  • Mean-field inference occurs within the CRF, but the model is trained with cross-entropy rather than by optimizing the CRF likelihood.

4. Experimental Evaluation

Experiments on VOC, SBD, and Cityscapes evaluate segmentation quality with AP_r and Matching IoU, including ablations of CRF potentials and end-to-end training. The method achieves strongest gains at high IoU thresholds, improves semantic segmentation, and sets a new Cityscapes state of the art.

  • Evaluation Metrics: AP_r averages performance across IoU thresholds from 0.1 to 0.9, while Matching IoU evaluates globally coherent image-level segmentation maps.AP_r uses region overlap rather than bounding-box overlap; Matching IoU matches predicted and ground-truth images before calculating IoU.
  • Ablation Studies: Each unary potential improves instance segmentation in both AP_rvol and Matching IoU, while end-to-end training improves results at all AP_r thresholds.The Global term particularly improves performance over the Box term at the high AP_r threshold of 0.9.
  • Ablation Studies: 2.1% and 2.3% improvements in AP_rvol result from the Global and Shape terms, respectively, compared with 1.3% from the Box term.The Global term uses semantic segmentation over the entire image, while shape priors recover occluded instances.
  • SBD Dataset: 1.5% improvement in AP_r at IoU 0.7 over the previous state of the art and 8.3% improvement in Matching IoU are reported on SBD.Unlike several competing methods, these results use a single forward pass without additional detector-based rescoring.
  • Semantic Segmentation and Cityscapes: 0.9% and 1% semantic-segmentation improvements on VOC and SBD, respectively, follow finetuning for instance segmentation, while the method sets a new Cityscapes state of the art.Cityscapes evaluation uses 2,975 training images and a held-out 1,525-image test set.

5. Conclusion and Future Work

The paper presents an end-to-end instance segmentation approach with intermediate semantic segmentations, naturally varying instance outputs, and no post-processing. Fine-tuning for instance segmentation improves semantic segmentation and yields more accurate, detailed results, especially at high AP r thresholds.

  • The system resembles a semantic segmentation network more than object-detection-derived architectures, producing more accurate and detailed segmentations.
  • The method naturally produces segmentation maps without post-processing and outputs a variable number of instances according to image content.
  • Future work will incorporate an object detector into end-to-end training for joint semantic segmentation, object detection, and instance segmentation.

Appendix

The appendix documents supplementary qualitative and quantitative evaluations on VOC, SBD, Cityscapes, and comparisons with MNC and FCIS. The figures illustrate strengths in boundary precision, false-positive handling, global reasoning, occlusion, and discontinuous objects, alongside failure cases.

  • The supplementary material reports detailed qualitative and quantitative results on VOC and SBD and includes runtime measurements.
  • The appendix compares the method with MNC and FCIS and presents Cityscapes qualitative results.
  • The appendix visualizes VOC and SBD performance across AP r thresholds and reports per-class results at selected thresholds.
  • The method can ignore false-positive detections and associate pixels outside imperfect bounding boxes with the correct detection.
  • Compared with MNC, the method produces precise, boundary-adherent segmentations but can miss instances that MNC identifies.
  • On Cityscapes, the algorithm handles many instances and objects that are discontinuous because of occlusion.

A. Detailed results on the VOC dataset

On VOC, the method’s strongest relative performance appears at high IoU thresholds, where it achieves the highest AP r for most object classes.

  • The method struggles on VOC classes including bicycle, chair, dining table, and potted plant.
  • At IoU threshold 0.9, the method achieves the highest AP r for 16 of the 20 VOC object classes.
  • At IoU threshold 0.7, the method achieves the highest AP r for 15 VOC object classes.
  • At IoU threshold 0.5, the method, MPA 3-scale, and PFN each achieve the highest AP r for 6 categories.

B. Detailed results on the SBD dataset

On SBD, the appendix reports per-class and mean AP r results across available IoU thresholds, while noting weaker high-threshold performance than on VOC. Coarser annotations and training-data limitations constrain precise-boundary evaluation.

  • SBD visualizations show difficulty on chair, dining table, potted plant, and bottle, while bicycle AP r improves relative to VOC.
  • SBD AP r at IoU threshold 0.9 is low compared with VOC, potentially because its annotations are less finely detailed.
  • This reduced pretraining data may hinder the network’s ability to produce precise segmentations.
  • The appendix compares published methods using mean AP r at IoU thresholds 0.7 and 0.5 because other work reported results only at those thresholds.
Loading 1704.02386v1…