Source-linked AI summary
Iterative Instance Segmentation
Ke Li, Bharath Hariharan, Jitendra Malik
TL;DR
Pixel-wise labelling methods often ignore underlying structure, while explicit structural constraints are difficult to specify and may make inference intractable. The paper replaces one-step structured prediction with iterative unconstrained predictions that learn implicit structure from data, achieving state-of-the-art instance-segmentation performance.
Problem
Existing pixel-wise labelling methods often disregard global structure, while manually specifying structural constraints is impractical and inference can become intractable.
Method
The method decomposes structured prediction into multiple unconstrained prediction steps that learn to correct earlier mistakes and implicitly capture output structure.
Results
63.6% mean APr at 50% overlap and 43.3% at 70% overlap establish state-of-the-art performance on instance segmentation.
Takeaways & Limitations
The approach automatically learns priors for shape, region contiguity, and contour smoothness without specifying them a priori.
Takeaways & Limitations
The method is motivated by the need to learn structural constraints automatically because manually engineering abstract shape constraints is neither feasible nor desirable.
Abstract
from arXiv · showhide
Existing methods for pixel-wise labelling tasks generally disregard the underlying structure of labellings, often leading to predictions that are visually implausible. While incorporating structure into the model should improve prediction quality, doing so is challenging - manually specifying the form of structural constraints may be impractical and inference often becomes intractable even if structural constraints are given. We sidestep this problem by reducing structured prediction to a sequence of unconstrained prediction problems and demonstrate that this approach is capable of automatically discovering priors on shape, contiguity of region predictions and smoothness of region contours from data without any a priori specification. On the instance segmentation task, this method outperforms the state-of-the-art, achieving a mean $\mathrm{AP}^{r}$ of 63.6% at 50% overlap and 43.3% at 70% overlap.
1. Introduction
Pixel-wise labelling methods often ignore global structure, making predictions visually implausible while explicit structural modelling is difficult and inference may be intractable. The proposed iterative method learns implicit structure from data and improves instance segmentation.
- Existing pixel-wise labelling methods commonly predict pixels independently, disregarding image structure and producing visually implausible outputs.Superpixels and CRFs enforce local appearance consistency but do not capture global object-level cues such as shape.
- Explicit structural constraints are difficult to formulate for abstract concepts such as shape, and inference can become intractable when higher-order potentials are used.These constraints may also require manual specification of low-energy configurations in advance.
- The method decomposes prediction into multiple steps so later predictions can correct mistakes made earlier.Learning to correct previous mistakes requires the model to capture output structure implicitly.
- 63.6% mean APr at 50% overlap and 43.3% at 70% overlap demonstrate state-of-the-art instance-segmentation performance.The method also learns priors for shape, region contiguity, and contour smoothness without explicitly specifying them.
2. Related Work
Prior work mainly captures local smoothness or uses structured models whose global constraints are difficult to specify and infer. This paper instead uses iterative unconstrained predictions to handle high-dimensional outputs with complex structure.
- Instance-segmentation research commonly segments detected objects, while many methods still predict each pixel independently.This motivates methods that account for dependencies among pixels and object-level structure.
- Earlier approaches enforce local smoothness through image contours, superpixels, or CRFs, but these mechanisms do not represent all global image structure.The related-work discussion distinguishes local appearance cues from broader object-level constraints.
- Traditional pixel-level structured models struggle with global shape constraints because they require higher-order, non-submodular potentials, making exact inference and training intractable.Explicitly encoding relationships such as mutually exclusive object parts is especially difficult.
- The proposed procedure reduces structured prediction to sequential unconstrained prediction problems that correct errors from earlier steps.The prediction process is decomposed into multiple stages using the input image and preceding prediction.
- The method supports high-dimensional pixel-wise prediction with complex structure without requiring a natural distance metric for structural conformity.It extends iterative inference ideas beyond settings with low-dimensional output spaces such as human-joint locations.
3. Method
The method predicts instance masks through an iterative CNN procedure that uses previous heatmaps to learn structural cues such as shape. It is evaluated within a detection-to-segmentation pipeline using region average precision on PASCAL VOC 2012.
- 3.1. Task and Setting: Instance segmentation predicts a mask for each object instance, using detections and heatmaps that are thresholded into final masks.The pipeline runs detection, applies non-maximum suppression, predicts a heatmap for each box, and binarizes it.
- 3.1. Task and Setting: Shape cues help disambiguate adjacent instances whose pixels share the same category but may be grouped incorrectly.The method targets structural constraints that local pixel cues do not capture reliably.
- 3.3. Algorithm: The proposed procedure reduces structured prediction to repeated unconstrained predictions, correcting earlier mistakes while maintaining consistency with the input at each step.Its final prediction is intended to be consistent with both the image and the learned structure.
- 3.3. Algorithm: At test time, the CNN receives the image and the previous heatmap, starting from a constant 1/2 prediction and returning the final iterative output.The same previous-prediction mechanism is used during staged training, where later stages learn from preceding predictions.
- 3.4. Discussion: The multi-step formulation models local corrections rather than directly modelling a potentially complex global structural manifold.The discussion interprets the procedure as learning the gradient of an implicit function whose level set represents the structure.
- 3.2. Segmentation System: The segmentation system modifies a hypercolumn CNN with category-dependent input channels and uses four training stages, with inference typically converging after three steps.The system can additionally project heatmaps to superpixels before producing region predictions.
4. Experiments
Experiments show that iterative refinement improves visual coherence and region accuracy, including on challenging scenes and already strong detections. The method outperforms existing approaches on instance segmentation, reaching mean APr of 63.6% at 50% overlap and 43.3% at 70% overlap.
- Training progression: Across training stages, heatmaps recover missed object parts, suppress adjacent-object regions, and become visually coherent after four stages.The progression indicates that each stage learns local corrections and increasingly respects output structure.
- Benchmark comparison: With rescoring, the proposed method reaches mean APr of 63.6% at 50% overlap and 43.3% at 70% overlap, the best reported instance-segmentation performance.It outperforms the state of the art across the evaluated settings.
- Qualitative comparisons: Compared with the vanilla hypercolumn net, predictions are less noisy, contain fewer holes, and have smoother, more shape-consistent contours.These improvements are visible in heatmap and region comparisons, while superpixel projection mainly benefits the noisier baseline and can sometimes remove details from the proposed predictions.
- Challenging scenes: On challenging scenes containing adjacent people or animals, the proposed method suppresses parts of neighboring objects that the hypercolumn net sometimes includes.The qualitative examples support improved separation of individual instances in crowded configurations.
- Detection-level analysis: Overlap with ground truth improves for 76% of detections, degrades for 15.6%, and remains unchanged for the rest versus the vanilla hypercolumn net.Most detections lie above the comparison diagonal, indicating more accurate region predictions.
- Detection-level analysis: For detections with 75% baseline overlap, the proposed method sometimes raises overlap above 90%, improving already accurate predictions.The reported gains can reach 15% when the vanilla hypercolumn net already achieves reasonably good overlap.
5. Conclusion
The method exploits implicit output structure without manually specifying constraints, learning priors on shape, region contiguity, and contour smoothness for instance segmentation.
- 63.6% mean APr at 50% overlap and 43.3% at 70% overlap demonstrate state-of-the-art instance segmentation performance.
- The method automatically discovers salient output structure from data without a priori specification.
- The learned structure includes priors on shape, region contiguity, and smoothness of region contours.
Supplementary Material
The supplied supplementary-material passage identifies Bharath Hariharan of Facebook AI Research.
- Bharath Hariharan is identified with Facebook AI Research.
6. Per-Category Performance Comparison
The paper reports per-category APr comparisons for the proposed method against state-of-the-art methods on PASCAL VOC 2012 validation data at two overlap thresholds.
- Per-category APr at 50% overlap is reported for the proposed method against the state of the art on PASCAL VOC 2012 validation data.
- Per-category APr at 70% overlap is reported for the proposed method against the state of the art on PASCAL VOC 2012 validation data.
7. Additional Visualizations
The supplementary visualizations compare heatmap and region predictions from the proposed method and the vanilla hypercolumn net on PASCAL VOC 2012 validation images.
- Figures 8, 9, and 10 compare heatmap and region predictions from the proposed method with those of the vanilla hypercolumn net.
- The visual comparisons use images from the PASCAL VOC 2012 validation set.