PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Pixels: A Comprehensive Survey from Bottom-up to Semantic Image Segmentation and Cosegmentation

Hongyuan Zhu, Fanman Meng, Jianfei Cai, Shijian Lu

arXiv:1502.00717v1cs.CV

TL;DR

Image segmentation research has advanced across many methods and applications, but existing reviews were sparse and outdated while important challenges remained unresolved. This paper synthesizes 180 publications across major segmentation areas, datasets, and metrics, and proposes design choices and future directions. It also identifies instance-level segmentation as an important remaining limitation.

  • Problem

    Existing segmentation reviews were sparse and outdated, while feature representation, model design, and optimization challenges remained unresolved.

  • Method

    The paper conducts a comprehensive review of 180 publications spanning bottom-up methods, superpixels, interactive segmentation, object proposals, semantic parsing, cosegmentation, datasets, and evaluation metrics.

  • Results

    The survey synthesizes the pros and cons of segmentation methods and identifies design choices and future research directions across the reviewed areas.

  • Takeaways & Limitations

    The review provides shared knowledge and research opportunities for advancing image segmentation toward human perception.

  • Takeaways & Limitations

    Existing methods generally produce one most-likely label per category rather than representing how many same-category instances are present.

Abstract

from arXiv · show

Image segmentation refers to the process to divide an image into nonoverlapping meaningful regions according to human perception, which has become a classic topic since the early ages of computer vision. A lot of research has been conducted and has resulted in many applications. However, while many segmentation algorithms exist, yet there are only a few sparse and outdated summarizations available, an overview of the recent achievements and issues is lacking. We aim to provide a comprehensive review of the recent progress in this field. Covering 180 publications, we give an overview of broad areas of segmentation topics including not only the classic bottom-up approaches, but also the recent development in superpixel, interactive methods, object proposals, semantic image parsing and image cosegmentation. In addition, we also review the existing influential datasets and evaluation metrics. Finally, we suggest some design flavors and research directions for future research in image segmentation.

1. Introduction

Image segmentation seeks to divide images into meaningful, non-overlapping regions, but defining meaningful objects and matching human perception remains difficult. This survey addresses unresolved challenges and the lack of a comprehensive, current synthesis.

  • Perceptual grouping abstracts low-level image primitives into meaningful parts, producing compact representations useful for later indexing, retrieval, and processing.
  • Image segmentation divides a natural image into K non-overlapping meaningful entities, supporting annotation, superpixel processing, scene parsing, body estimation, and object recognition.
  • “Objects” may be things, textures, stuff, or parts of other objects, making bottom-up segmentation challenging and ill-posed.
  • Low-level features capture local properties, remain sensitive to lighting and perspective, and can cause algorithms to over-segment images into trivial regions.
  • The survey responds to unresolved feature, model, and optimization challenges by systematically reviewing recent segmentation work across broad topic areas and identifying future directions.

2. Bottom-up methods

Bottom-up segmentation groups pixels using low-level similarities or global image criteria, through discrete graph-based and clustering methods or continuous variational methods. The surveyed approaches differ in assumptions, representations, optimization, and sensitivity to local versus global information.

  • Bottom-up methods group nearby pixels using local feature homogeneity, while discrete methods treat images as fixed grids and continuous methods treat them continuously.
  • Clustering methods: K-means assigns samples to nearby centers and iteratively updates those centers, working efficiently when cluster counts are known or distributions are spherically symmetric.
  • Clustering methods: Mixture-of-Gaussians models clusters with means, covariances, and mixing weights estimated by expectation maximization before assigning pixels to the most probable cluster.
  • Clustering methods: Mean-shift is non-parametric, automatically determines cluster numbers and modes, and forms segments by grouping pixels with nearby spatial and range-domain convergence points.
  • Discrete methods: Watershed floods morphological surfaces from local minima but can seriously over-segment, while graph-based merging uses relative dissimilarity in a pixel graph to optimize global grouping.
  • Discrete methods: Normalized Cut incorporates global image information by partitioning an affinity graph while encouraging balanced clusters, though finding the exact cut is NP-hard and typically requires approximation.
  • Contour cues: Learned contour affinities combine brightness, color, and texture cues, and gPb combines local and spectral contour signals using learned weights.

3. Superpixel

Superpixels over-segment images into homogeneous regions smaller than objects or parts, offering a more meaningful and efficient representation than individual pixels. Methods trade boundary quality, compactness, and computational speed, with some interfaces also supporting interactive segmentation.

  • Superpixels are intended to be perceptually meaningful homogeneous regions that preserve image structures and convey shape cues beyond pixel-level representation.
  • Representing images with superpixels reduces parameters and computation while enabling larger-support, more discriminative features.
  • Bottom-up algorithms can be adapted for superpixel over-segmentation by tuning cluster numbers, region sizes, kernel sizes, or mode-drifting behavior.
  • Local optimization methods generate faster superpixels by deforming initial regions toward salient contours, using geometric or geodesic flows.
  • Interactive segmentation examples use bounding boxes or scribbles to convey user-provided object information.
  • SEED is reported to run at 30Hz, whereas an entropy-rate graph method takes about 2.5s on a 480x320 image.

4. Interactive methods

Interactive segmentation uses user-provided constraints to guide accurate object localization, iteratively refining results until satisfactory. The section covers interfaces, contour and label-propagation methods, graphical models, priors, and efficiency trade-offs.

  • Overview: Interactive segmentation uses user inputs such as color, contour, or region constraints to produce precise object segmentation for applications including image editing and medical analysis.The user iteratively supplies constraints as the algorithm produces updated segmentation results.
  • Overview: The interactive pipeline alternates between user initialization, algorithmic segmentation, and further user constraints until the result is satisfactory.
  • Contour-based methods: Contour-based methods evolve user-placed contours toward salient object boundaries using local edge costs or global regional and boundary energies.Live-Wire finds shortest paths between seed points, whereas active contours optimize a global energy function.
  • Label propagation based methods: Label propagation begins with user-marked nodes and spreads labels through global or local optimization, with GraphCut and RandomWalk among the widely used global approaches.RandomWalk assigns probabilities based on reaching marked nodes, then thresholds them to form the segmentation.
  • Label propagation based methods: GraphCut models foreground-background labeling with unary and pairwise potentials, while GrabCut uses Gaussian mixture models and bounding-box hard constraints.LazySnap extends GrabCut with superpixels and an interface for adjusting weak or low-contrast boundaries.
  • Models and efficiency: Higher-order boundary-preserving potentials can capture thin details neglected by classical MRF models, while fast cost-volume filtering can approximate slower global optimization.

5. Object Proposals

Object proposals relax precise object segmentation by generating regions likely to cover objects, using class-specific detectors or class-independent and diversified strategies. These methods exploit object cues while facing scalability, speed, and contour-detection constraints.

  • Overview: Object proposal methods generate pools of regions likely to cover objects, separating object-like “things” from background “stuff” instead of deterministically segmenting each object.
  • Class-specific object proposals: Class-specific proposals incorporate detectors such as DPM or Poselets and can refine bounding boxes or use hierarchical regions for segmentation and detection.Their reliance on class-specific detectors limits generality when many object classes must be handled.
  • Class-independent object proposals: Class-independent proposals use appearance or geometric cues to produce general object regions without targeting a predefined class.Their rationale connects general object proposals with salient object detection because objects often differ from backgrounds in appearance or geometry.
  • Class-independent object proposals: CPMC and related methods generate bottom-up regions, apply classifiers to rank them, and use seed sampling strategies such as grid foreground seeds and frame-boundary background seeds.
  • Class-independent object proposals: Geodesic Object Proposal uses superpixels, geodesic distances, and spatially separated seeds, avoiding expensive GrabCut recomputation; learned seed placement yields insignificant performance improvement.
  • Other proposal strategies: Contour-based proposal hierarchies can exceed GrabCut-based accuracy but remain limited by contour-detector speed and accuracy, while diversified methods combine multiple handcrafted strategies without classifier ranking.

6. Semantic Image Parsing

Semantic image parsing assigns pixels or regions to predefined semantic classes to support holistic visual understanding. Modern systems commonly use CRF or MRF formulations, learned features, spatial support, context, and top-down information.

  • Overview: Semantic image parsing divides images into non-overlapping regions corresponding to predefined classes such as cars, grass, and sheep.Its stated ultimate goal is holistic understanding of the visual world.
  • MRF and CRF formulations: Most state-of-the-art parsing systems formulate labeling as inference in an MRF or CRF, providing probabilistic modeling of interactions between output variables and observed features.
  • MRF and CRF formulations: In the CRF formulation, node variables represent pixel or superpixel labels from a predefined set, and maximum-posterior inference is equivalent to minimizing the energy function.The partition function normalizes the distribution but need not be computed for most-likely labeling inference.
  • Parsing pipeline: A typical parsing pipeline extracts local features, trains an appearance model with ground-truth labels, applies the classifier to test pixels, and uses outputs as unary terms.
  • Design components: Feature representations have progressed from pixel-level color and texton cues toward region-, image-level, spatial-layout, and deep convolutional features.
  • Design components: Superpixels provide structure-aware spatial support and reduce model complexity from millions of variables to hundreds or thousands, while context models class co-occurrence.
  • Design components: Combining bottom-up information with top-down detectors supports holistic scene understanding because bottom-up cues capture homogeneous stuff and detectors capture thing classes.

7. Image Cosegmentation

Cosegmentation extracts common foreground objects across multiple images using weak shared-object priors, but varying foregrounds and application settings make modeling, similarity measurement, and optimization difficult. Existing approaches extend single-image models, introduce alternative graph, clustering, and learning strategies, or address multiple-foreground and large-scale scenarios.

  • Cosegmentation extracts common objects from image sets without pixel- or image-level object information, making it suitable for large-scale dataset segmentation.
  • Its central challenges are modeling multiple images, measuring foreground similarity under appearance variation, and extending methods to diverse applications.
  • The survey also identifies emerging settings including multi-class or multiple-foreground cosegmentation, large-scale image cosegmentation, and web image cosegmentation.
  • Existing methods extend single-image segmentation models by combining within-image segmentation with cross-image foreground consistency.
  • MRF-based methods use different consistency terms, including ℓ1, squared ℓ2, reward-based, histogram, and high-order graph-matching formulations, with corresponding approximate or GraphCut-based optimization.
  • Alternative strategies formulate cosegmentation as clustering, graph-based proposal selection, discriminative part learning, or consensus over co-occurring regions.

8. Dataset and Evaluation Metrics

The survey reviews datasets and evaluation metrics spanning bottom-up, interactive, cosegmentation, video, and semantic segmentation tasks. These resources vary substantially in scale, annotation type, realism, and difficulty, while metric design remains challenging because segmentation itself is ill-defined.

  • Dataset development progressed from small, costly manually labeled collections toward large datasets enabled by crowdsourcing platforms.
  • Datasets: BSDS provides 200 training, 100 validation, and 200 testing images, each annotated by at least three subjects, for contour detection and object-agnostic segmentation.
  • Cosegmentation datasets: Cosegmentation benchmarks range from 25 heterogeneous-background image pairs in MSRC to 643 images across 38 groups in iCoseg and 14 groups in FlickrMFC.
  • Video datasets: Video datasets cover binary and multi-class settings with occlusion, deformation, camera motion, lighting variation, and motion blur, including MPI’s 11 videos and around 520 labeled frames.
  • Semantic segmentation datasets: Semantic segmentation resources include CamVid’s 32 classes, MSRC’s 23 classes and 591 images, PASCAL VOC’s 20 classes and 11,530 train/validation images, and COCO’s 328K images with 91 classes.
  • Evaluation metrics: Evaluation uses human-labeled segmentation benchmarks and metrics such as Variation of Information and Intersection-over-Union, but GCE can assign zero error to both pixel-level and whole-image segmentations.

9. Discussions and Future Directions

The review highlights design trade-offs across segmentation methods and proposes future directions involving richer cues, multiple instances, holistic analysis, and learned features.

  • Design flavors: Superpixels reduce training and inference time and support richer features, but may introduce structure errors near boundaries.Pixel-based or hybrid approaches are preferable when boundary appearance matters, while region-based methods suit localization-focused applications.
  • Design flavors: Extrinsic cues from multiple images, videos, and large datasets can add semantic information, but heterogeneous data requires mechanisms for handling noise.Potential sources include multiview imagery, video sequences, depth, and related-image databases.
  • Promising future directions: Future work should exploit internal structures across multiple images, including motion, depth, 3D geometry, and related-image collections.The review identifies these sources as ways to facilitate segmentation beyond single-image analysis.
  • Promising future directions: Most methods do not distinguish multiple same-category instances, motivating combinations of detectors and CRF models and the use of instance-labelled datasets.Microsoft COCO is identified as a dataset that could support progress in this direction.
  • Promising future directions: Holistic systems can combine segmentation with context, geometry, attributes, language, detection, or classification, although inference cost increases.The review expects such systems to outperform monotone analysis in supported examples.
  • Promising future directions: Learned multilayer neural-network features have achieved state-of-the-art performance in several vision tasks and may benefit complex segmentation problems.The review contrasts their potential generalization with task-specific hand-crafted features.

10. Conclusion

The paper reviews recent image-segmentation research across major methodological areas, evaluates design choices, and identifies future directions. Despite substantial progress and commercialization of interactive techniques, robustness and efficiency remain barriers to broadly reliable practical use.

  • Conclusion: The review covers bottom-up, interactive, object-proposal, semantic-parsing, and cosegmentation methods.It presents a broad synthesis of recent image-segmentation developments.
  • Conclusion: The authors discuss the pros and cons of segmentation methods and suggest design choices and future research directions.These recommendations are intended to guide further exploration.
  • Conclusion: Interactive segmentation has been commercialized, but robustness and efficiency limitations still hinder segmentation as a general reliable practical tool.Performance can drop significantly under strong illumination changes, noise corruption, or rain.
Loading 1502.00717v1…