Pixelwise Instance Segmentation with a Dynamically Instantiated Network

1. Introduction

• Instance segmentation combines semantic segmentation’s pixel-level class labels with object detection’s ability to distinguish instances, while operating at pixel level.The paper motivates accurate object recognition and localization through applications including autonomous driving, image editing, and robotics.Source: §1 · p. 1
• Proposal-based methods process independent regions, making occlusion handling difficult; overlapping proposals can yield conflicting labels, and detection-based methods cannot recover from false detections.The approaches described either refine box-based proposals into segments or use segment-based proposals.Source: §1 · p. 1
• An initial semantic segmentation module feeds a dynamic network that uses detector cues and a CRF to predict pixelwise class and instance labels without post-processing.The system outputs a variable number of instances per image, considers the entire image to address occlusions, and is robust to false-positive detections and poorly localized boxes.Source: §1 · p. 1, Figure 1 · p. 2
• The paper also evaluates complete segmentation maps with Matching Intersection over Union, since AP and AP r assess individual predictions in isolation.AP is identified as an object-detection metric, and AP r as an instance-segmentation variant [19].Source: §1 · pp. 1–2
• The system achieves substantial improvements over state-of-the-art methods at high AP r thresholds on Pascal VOC and Semantic Boundaries.The network also improves semantic segmentation while being trained for instance segmentation.Source: §1 · p. 2

2. Related Work

• Winn and Shotton used asymmetric CRF pairwise terms to order object parts spatially, while depth-ordered DPM detections with foreground masks to resolve occlusion.Source: §2 · p. 2
• SDS [19] extended R-CNN [16] with proposal-based detection and class-specific masks; its descendants [20] [11] [30] inherit proposal-quality limits and often require post-processing.Dai et al. [12] proposed an end-to-end network for box proposals, masks, and classification; Liu et al. made SDS end-to-end, while [32] iteratively refined proposals.Source: §2 · p. 2
• Detector-free depth-ordering methods reason globally but require ground-truth depth and cap cars at nine; an RNN [45] supports variable outputs only for one category.The proposed system handles a variable number of instances across multiple object classes.Source: §2 · p. 2
• Liang et al. [33] used semantic segmentation and CNN features to predict instance boxes and class counts for spectral clustering; Arnab et al. combined semantics, detector outputs, and a CRF.The additional information predicted by Liang’s network could have come from an object detector; Arnab et al.’s method was not end-to-end and struggled with bounding-box localization errors and occlusion.Source: §2 · pp. 2–3
• Unlike detection-based systems [19] [20] [12], our end-to-end CRF reasons over the whole image, needs no post-processing, and handles variable instance counts plus localization errors and occlusions.The instance unary terms combine detector-box information, initial semantic segmentation, and object shape before the CRF produces the instance map.Source: §2 · p. 3, Figure 2 · p. 3

3. Proposed Approach

• A semantic segmentation module and instance network form a fully differentiable pipeline, given detections, that uses CRF mean-field inference to obtain the MAP instance labelling.The instance CRF unary potentials use the semantic segmentation result and object detector outputs.Source: §3 · p. 3

3.1. Semantic Segmentation subnetwork

• The semantic module uses FCN8s [38] based on VGG [47], followed by recurrent mean-field CRF inference with dense pairwise and higher-order detection potentials.The detection potential encourages consistency between detections and segmentations and recalibrates detection scores; the module outputs per-pixel class probabilities Q.Source: §3.1 · p. 3

3.2. Instance Segmentation subnetwork

• Each pixel is assigned to a detected instance or background, with labels {0, 1, …, D} varying by image and no fixed maximum number of instances.D is the number of detections; the loss handles permutations of instance labels, and joint use of detections and semantic predictions provides robustness to high-scoring false positives.Source: §3.2 · p. 4

3.2.1 Box Term

• The box unary favors assigning a pixel to a detection when it lies within that box, in proportion to its semantic-class probability and detection score.It works well with good initial segmentation, is robust to false-positive detections, and with pairwise terms can resolve same-class occlusions when instances differ in appearance.Source: §3.2.1 · p. 4

3.2.2 Global Term

• The global unary uses semantic confidence without bounding boxes, distributing probability across instances of a class when localization information is unavailable.It assigns probability mass across the image, including object pixels outside a detection box, and makes outputs depend on segmentation predictions everywhere, yielding more stable training gradients across batches.Source: Figure 4 · p. 4, §3.2.2 · p. 5

3.2.3 Shape Term

• The shape term selects a template matching the detected-class segmentation within a box by normalized cross-correlation, then combines it elementwise with segmentation unaries.Templates are warped to the box dimensions with bilinear interpolation; the method matches one template per instance, with matching multiple templates suggested as future work.Source: §3.2.3 · p. 5

3.2.4 Pairwise term

• Densely connected Gaussian pairwise potentials encourage appearance and spatial consistency, with learned weights, and can resolve same-class occlusions using appearance differences.Nearby pixels with similar appearance often belong to the same instance.Source: §3.2.4 · p. 5

3.3. Inference of our Dynamic Instance CRF

• Mean-field inference approximately minimizes the CRF Gibbs energy toward MAP labeling; unrolling it as a recurrent layer enables end-to-end training.Source: §3.3 · p. 5
• The CRF dynamically instantiates labels for each image’s variable instance count, sharing non-class-specific weights, because instance classes lack semantic meaning.Source: §3.3 · pp. 5–6
• During training, ground truth is matched to predictions before loss computation to address label permutations.Source: Figure 6 · p. 6

3.4. Loss Function

• One loss is backpropagated through both segmentation modules, and matching resolves label-permutation symmetry without requiring ground-truth depth maps.Depth ordering in and addresses the symmetry but requires ground-truth depth maps during training.Source: §3.4 · p. 6
• The matched ground truth is the permutation that maximizes IoU with the prediction, accommodating different numbers of predicted and ground-truth segments.Each segment has an instance label and semantic class; the counts may differ.Source: §3.4 · p. 6
• For segmentation training, the method uses the common cross-entropy loss, which the authors found performed better than the approximate IoU loss proposed in [45].Source: §3.4 · p. 6
• A maximum-weight bipartite matching avoids evaluating all ground-truth permutations, using IoU edge weights for matching semantic classes and zero otherwise.Unmatched segments connect to dummy nodes with zero overlap.Source: §3.4 · p. 6
• The detector determines instance ordering and remains static during training, so prediction order changes mainly when detections overlap an object.Source: §3.4 · p. 6

3.5. Network Training

• The semantic segmentation network is FCN8s [38] with a CRF unrolled as an RNN, pretrained using cross-entropy before appending the instance subnetwork and finetuning with instance annotations and only the Sec. 3.4 loss.The CRF-RNN training approach is described in and [2].Source: §3.5 · p. 6
• Semantic pretraining uses learning rate 10^-8, momentum 0.9, and batch size 20; the learning rate is low because the loss is not pixel-normalized, so larger images contribute more loss.The normalized learning rate is approximately 2 × 10^-3.Source: §3.5 · p. 6
• Joint instance-network training lowers the learning rate to 10^-12 and batch size to 1, which gave empirically better results, and clips gradients with ℓ2 norms above 10^9.The clipping threshold was set by observing normal gradient magnitudes; clipping facilitated training the two CRF-RNN inference modules in the complete network, and is common in RNN training [40].Source: §3.5 · p. 6

3.6. Discussion

• The network computes semantic and instance segmentation in one forward pass without post-processing.It avoids patch aggregation, mask voting, superpixel projection, and spectral clustering.Source: §3.6 · p. 6
• Initial semantic segmentation provides robustness to detector errors and poorly localized detections, while the CRF reasons over the entire image rather than independent proposals.The detector is not jointly trained with the network, so the two components do not share the same failure cases.Source: §3.6 · p. 7
• Not jointly training the detector also prevents instance labels from switching often during training, making learning more stable.The network uses mean-field inference in its CRF but optimizes cross-entropy loss rather than the CRF likelihood.Source: §3.6 · p. 7

4.1. Experimental Details

• Semantic pretraining uses Pascal VOC, SBD, and Microsoft COCO [35], followed by instance finetuning on VOC or SBD with separate models for each validation set.Validation images were excluded from pretraining, and the R-FCN detector [13] was trained without images from the instance-segmentation test sets.Source: §4.1 · p. 7

4.2. Evaluation Metrics

• AP_r evaluates predicted regions across IoU thresholds, and AP_r^vol averages results at nine thresholds from 0.1 to 0.9; Matching IoU measures the coherence of the image-wide segmentation map.Unlike AP_r, Matching IoU evaluates a globally coherent segmentation map; the measure was originally proposed in.Source: §4.2 · p. 7

4.3. Effect of Instance Potentials and End-to-End training

• Each unary potential improves AP_r^vol and Matching IoU; end-to-end training improves AP_r at all thresholds, while Global and Shape yield larger AP_r^vol gains than Box.The reported AP_r^vol gains are 1.3% for Box, 2.1% for Global, and 2.3% for Shape; Shape priors can recover occluded, missed instances, and Global uses semantic predictions across the image.Source: §4.3 · pp. 7–8

4.4. Results on VOC Validation Set

• At IoU 0.9, our method improves AP_r over previous state-of-the-art MPA by 6.6% (36% relative) and achieves AP_r^vol 57.5%, with highest AP_r above 0.7.MPA performs better at low IoU thresholds, while our method takes approximately 1.5s per image versus 8.7s for MPA on the same Titan X GPU.Source: Table 2 · p. 8, §4.4 · p. 8

4.5. Results on SBD Dataset

• On SBD, AP_r at IoU 0.7 improves by 1.5% over the previous state of the art [30], AP_r^vol improves by 3.4%, and Matching IoU improves by 8.3% over MNC [12].Unlike [30] [20], our results use a single forward pass without additional detector-based rescoring.Source: Table 3 · p. 8, §4.5 · p. 8

4.6. Improvement in Semantic Segmentation

• Finetuning for instance segmentation improves semantic-segmentation performance by 0.9% on VOC and 1% on SBD.The paper attributes the gains to the relationship between the tasks: instance segmentation is a more specific case of semantic segmentation.Source: §4.6 · p. 8, Table 4 · p. 9

4.7. Results on Cityscapes

• On the Cityscapes test set, our method sets a new state of the art, surpassing concurrent work [21] and the best previous published work by significant margins.The test set contains 1,525 images, and the 500-image validation set was not used for training; evaluation uses the online server.Source: Table 5 · p. 9, §4.7 · p. 9

5. Conclusion and Future Work

• The semantic-segmentation-oriented approach produces more accurate, detailed segmentations, with substantial improvements at high AP r thresholds over methods derived from object-detection networks [12] [20].The approach produces intermediate semantic segmentations.Source: §5 · p. 9
• The system naturally produces segmentation maps without post-processing and outputs a variable number of instances according to the image.Source: §5 · p. 9
• Finetuning for instance segmentation improves the network’s semantic segmentations.Source: §5 · p. 9
• Future work would incorporate an object detector into end-to-end training for joint semantic, object, and instance segmentation, with possible techniques suggested by and [39].Source: §5 · p. 9

Appendix

• VOC and SBD supplements report AP_r results by threshold and class, while the appendix also includes qualitative visualizations and runtime information.VOC results include per-class AP_r at 0.5, 0.7, and 0.9; SBD includes mean AP_r from 0.5 to 0.9 and per-class AP_r at 0.5 and 0.7.Source: Appendix · p. 11
• Examples show the method can ignore false-positive detections and segment pixels outside boxes, while missing detections can cause missed or merged instances.Other failures include incorrect semantic segmentation, a detected bird that was not segmented, and difficulty separating similar occluding cows.Source: Figure 7 · p. 12, Figure 8 · p. 13
• Against MNC [12], our method handles false-positive detections, poorly localized boxes, and overlapping instances more robustly, but can miss small or occluded instances.MNC sometimes produces more precise object-boundary segmentations, but may also create extra instances; our method’s missed instances can stem from inadequate shape priors or missing detections.Source: Figure 9 · p. 14, Appendix · p. 15
• Compared qualitatively with FCIS [31], the method handles false detections and poor box localization, assigns each pixel to one instance, and produces precise segmentations.The comparison uses FCIS results trained on COCO and a model trained on Pascal VOC, whose class set lacks objects such as umbrellas and suitcases.Source: Figure 10 · p. 16
• Cityscapes examples show that the method handles large numbers of instances and non-continuous objects, including a car in the first row occluded by a pole.Source: Figure 11 · p. 17

• The method struggles with bicycle, chair, dining table, and potted plant, classes where semantic segmentation systems also tend to perform below average.For chair, semantic-segmentation IoU was less than half the all-class mean in 16 of the 20 most recent VOC leaderboard submissions at the time of writing.Source: A. Detailed results on the VOC dataset · p. 18
• At IoU 0.9 our method has the highest AP r for 16 of 20 classes, and at 0.7 for 15; at 0.5, it, MPA 3-scale, and PFN [33] each lead in 6 categories.Source: A. Detailed results on the VOC dataset · p. 18

B. Detailed results on the SBD dataset

• Per-class AP r trends resemble VOC, with difficulty on chair, dining table, potted plant, and bottle, but bicycle performance improves under SBD’s coarser wheel labels.VOC often labels individual bicycle spokes, whereas SBD labels the entire wheel as a single circle; the authors suggest this makes fine-detail objects easier to segment.Source: B. Detailed results on the SBD dataset · p. 18, Figure 13 · p. 19
• At IoU 0.9, AP r is low compared with VOC, possibly because SBD’s coarser annotations and larger validation set leave less data for pretraining.The authors also suggest SBD may be unsuitable for measuring AP r at such high thresholds and that its training data may be less suited to their CRF-based system, which delineates sharp boundaries.Source: B. Detailed results on the SBD dataset · p. 18
• The paper compares per-class SBD instance-segmentation results with published methods at IoU 0.7 and 0.5, the only thresholds reported by other work.Source: B. Detailed results on the SBD dataset · p. 18, Table 10 · p. 21, Table 11 · p. 21