PaperlayerView exact revision on arXiv

Source-linked AI summary

Instance-Level Salient Object Segmentation

Guanbin Li, Yuan Xie, Liang Lin, Yizhou Yu

arXiv:1704.03604v1cs.CV

TL;DR

Existing salient-object methods detect salient pixels but do not identify individual instances within those regions. This paper introduces a multistage salient instance segmentation framework using multiscale refinement, contour-based proposals, and subset optimization, and reports state-of-the-art performance across public salient-region benchmarks and a new instance-segmentation dataset.

  • Problem

    Prior salient-object methods mainly produce dense saliency maps and are unaware of individual salient-object instances.

  • Method

    The method estimates saliency maps, detects salient-object contours with MSRNet, and identifies instances using MCG proposals and MAP-based subset optimization.

  • Results

    The method achieves state-of-the-art performance on all public salient-region datasets and the new salient instance segmentation dataset.

  • Takeaways & Limitations

    The paper establishes salient instance segmentation as a task and provides a framework and pixelwise annotated database for its evaluation.

  • Takeaways & Limitations

    A prior multiscale saliency method produces maps at one eighth of the input resolution, making accurate contours of small salient instances infeasible.

Abstract

from arXiv · show

Image saliency detection has recently witnessed rapid progress due to deep convolutional neural networks. However, none of the existing methods is able to identify object instances in the detected salient regions. In this paper, we present a salient instance segmentation method that produces a saliency mask with distinct object instance labels for an input image. Our method consists of three steps, estimating saliency map, detecting salient object contours and identifying salient object instances. For the first two steps, we propose a multiscale saliency refinement network, which generates high-quality salient region masks and salient object contours. Once integrated with multiscale combinatorial grouping and a MAP-based subset optimization framework, our method can generate very promising salient object instance segmentation results. To promote further research and evaluation of salient instance segmentation, we also construct a new database of 1000 images and their pixelwise salient instance annotations. Experimental results demonstrate that our proposed method is capable of achieving state-of-the-art performance on all public benchmarks for salient region detection as well as on our new dataset for salient instance segmentation.

1. Introduction

The paper introduces instance-level salient object segmentation to identify individual objects within salient regions, addressing a limitation of saliency methods that produce only dense maps. It combines multiscale saliency refinement, contour detection, proposal optimization, and a new annotated dataset.

  • Motivation: Existing salient object detectors identify salient pixels but remain unaware of individual salient-object instances.
  • Motivation: Instance-level salient object segmentation identifies individual object instances within detected salient regions for more detailed image parsing.The paper motivates this task for image captioning, multilabel recognition, and weakly or unsupervised learning scenarios.
  • Approach: The proposed decomposition predicts a binary saliency map, detects contours of individual salient objects, and identifies salient instances from object proposals.
  • Approach: MSRNet uses parallel streams over scaled inputs and learned attention to fuse multiscale saliency results for region and contour detection.Its refined VGG streams integrate bottom-up and top-down information.
  • Approach: MCG generates salient object proposals from detected contours, after which noisy or overlapping proposals are screened and refined with a fully connected CRF.
  • Contributions: The paper contributes a new challenging dataset for salient instance segmentation and reports benchmark results for salient contour detection and instance segmentation.

2. Related Work

The related work connects salient instance segmentation to salient-region detection, object proposal generation, and instance-aware semantic segmentation. The proposed task combines salient-region detection with instance identification without relying on predefined semantic categories.

  • Salient Region Detection: Deep CNN methods advanced salient-region detection, but prior approaches often use single-scale receptive fields or reduced-resolution outputs that hinder small-instance contour accuracy.
  • Object Proposal Generation: Object proposal methods either rank object windows by objectness or generate proposals by merging segments from multiple segmentation levels.
  • Object Proposal Generation: The paper generates object proposals from salient object contour detection results.
  • Instance-Aware Semantic Segmentation: Instance-aware semantic segmentation unifies object detection and semantic segmentation, whereas salient instance segmentation identifies instances within salient regions without predefined semantic categories.

3. Salient Instance Segmentation

The method uses a four-stage pipeline combining multiscale saliency and contour detection with proposal generation and refinement to produce salient object instances. Its MSRNet integrates shared multiscale streams, top-down refinement, and learned attention before MCG proposals and MAP-based selection.

  • Pipeline: The pipeline performs salient region detection, salient object contour detection, salient instance generation, and salient instance refinement.MSRNet is applied to the first two stages; contour results support proposal generation, followed by subset optimization.
  • Multiscale refinement: MSRNet uses three refined VGG streams with shared parameters and a learned attentional model to fuse information across input scales.The architecture is designed for salient region and contour detection, whose outputs require both low-level cues and high-level semantic information.
  • Refined VGG network: Each refinement module merges top-down and corresponding bottom-up feature maps, optionally up-sampling to increase spatial resolution.The top-down stream contains five stacked refinement modules, and its final output has the original input resolution.
  • Multiscale fusion: Three scaled inputs produce probability maps that are resized and fused using spatially varying, learned attention weights.The fused class map is a pixelwise weighted sum of the scale-specific probability maps.
  • Instance proposal and refinement: MCG generates salient object proposals from four contour maps, while MAP-based subset optimization selects a compact final set of proposals.The method generates 800 proposals per image, retains proposals containing at least 80% salient pixels, and assigns unique instance IDs to remaining proposals.

4. A New Dataset for Salient Object Instances

The authors construct a 1,000-image salient instance dataset with pixelwise instance labels because no suitable datasets existed for this new task. Annotation agreement among three annotators is used to retain high-quality examples.

  • Collection: The dataset begins with 1,388 images selected mostly from existing salient-region datasets to reduce ambiguity in salient-region detection.Sources include ECSSD, DUT-OMRON, HKU-IS, and MSO.
  • Image composition: Two-thirds of the selected images contain multiple occluded salient instances, while the remaining third covers other salient-region configurations.The remaining images contain no salient regions, one salient instance, or multiple non-occluded salient instances.
  • Annotation: Three annotators assigned different instance IDs to salient regions using an interactive segmentation tool.Images were retained only when all annotators divided the salient regions into the same number of instances.
  • Final dataset: The final dataset contains 1,000 images with high-quality pixelwise salient instance labels and salient object contours.The filtering procedure reduces label inconsistency through annotator agreement.

5. Experimental Results

The experiments evaluate MSRNet for salient region detection and salient instance segmentation. MSRNet consistently outperforms existing methods on salient-region benchmarks, while its refinement and multiscale components improve performance and its framework handles connected salient instances.

  • Salient Region Detection: MSRNet is evaluated on six salient-region datasets using precision-recall curves, maximum F-measure, and MAE.The comparison includes eight state-of-the-art methods and excludes results affected by training-test overlap.
  • Salient Region Detection: MSRNet improves maximum F-measure over the best existing method by 1.53%–3.70% and reduces the best MAE by 8.5%–20.4% across six datasets.The reported gains are 1.53%, 1.33%, 3.70%, 1.33%, 2.4%, and 1.8% for maximum F-measure, with corresponding MAE reductions of 10.6%, 20.4%, 13.8%, 8.5%, 13.8%, and 11.1%.
  • Salient Region Detection: MSRNet consistently outperforms other methods in visual saliency comparisons and precision-recall curves across the testing datasets.Its saliency maps are described as closest to ground truth, and the comparison accounts for DHSNet’s training overlap on DUT-OMRON.
  • Network Analysis: The refinement module and attention-based multiscale fusion each outperform a single VGG16 backbone, and together provide complementary benefits.The combined model achieves the best precision-recall curve, average precision, recall, and F-measure on the HKU-IS testing set.
  • Salient Instance Segmentation: The salient instance segmentation evaluation reports contour-detection and instance-segmentation benchmarks, including mAP at IoU thresholds of 0.5 and 0.7.The framework can handle challenging cases where multiple salient object instances are spatially connected.

6. Conclusions

The paper introduces salient instance segmentation and a framework centered on a multiscale saliency refinement network. It also contributes a pixelwise annotated dataset and reports state-of-the-art performance on salient-region and salient-instance benchmarks.

  • Contributions: The paper introduces salient instance segmentation and a framework for solving this new problem.The task is presented as related to salient object detection.
  • Contributions: The framework’s central component is a multiscale saliency refinement network that generates high-quality salient region masks and object contours.These outputs support the framework’s instance-segmentation process.
  • Contributions: The authors construct a new database with pixelwise salient instance annotations to support further research and evaluation.The dataset is part of the paper’s contribution to salient instance segmentation.
  • Results: Experimental results achieve state-of-the-art performance on all public salient-region datasets and on the new salient-instance dataset.The conclusion reports this outcome across both evaluation settings.
Loading 1704.03604v1…