PaperlayerView exact revision on arXiv

Source-linked AI summary

Recurrent Pixel Embedding for Instance Grouping

Shu Kong, Charless Fowlkes

arXiv:1712.08273v1cs.CVcs.LGcs.MM

TL;DR

Instance segmentation needs grouping for a variable number of objects, which standard per-pixel classification does not directly represent. The paper combines spherical pixel embeddings with differentiable recurrent mean-shift grouping, achieving substantial proposal-generation improvements and benefits on other pixel-level tasks.

  • Problem

    Instance segmentation requires structured, permutation-invariant grouping with an unknown number of instances, whereas common pixel-labeling losses decompose over individual labels.

  • Method

    The framework learns unit-length pixel embeddings with pairwise spherical max-margin constraints and groups them using differentiable recurrent mean-shift dynamics.

  • Results

    Object proposal average recall for 10 proposals improves from 0.56 to 0.77, alongside demonstrated benefits for boundary detection and semantic instance segmentation.

  • Takeaways & Limitations

    The framework directly handles variable numbers of instances without heuristic post-processing or instance-size and class-imbalance hyperparameter tuning.

  • Takeaways & Limitations

    Generic bottom-up proposal generation is not informed by top-down semantics, and useful bounds on the recurrent model's Jacobian spectrum were not proved.

Abstract

from arXiv · show

We introduce a differentiable, end-to-end trainable framework for solving pixel-level grouping problems such as instance segmentation consisting of two novel components. First, we regress pixels into a hyper-spherical embedding space so that pixels from the same group have high cosine similarity while those from different groups have similarity below a specified margin. We analyze the choice of embedding dimension and margin, relating them to theoretical results on the problem of distributing points uniformly on the sphere. Second, to group instances, we utilize a variant of mean-shift clustering, implemented as a recurrent neural network parameterized by kernel bandwidth. This recurrent grouping module is differentiable, enjoys convergent dynamics and probabilistic interpretability. Backpropagating the group-weighted loss through this module allows learning to focus on only correcting embedding errors that won't be resolved during subsequent clustering. Our framework, while conceptually simple and theoretically abundant, is also practically effective and computationally efficient. We demonstrate substantial improvements over state-of-the-art instance segmentation for object proposal generation, as well as demonstrating the benefits of grouping loss for classification tasks such as boundary detection and semantic segmentation.

1. Introduction

The paper addresses pixel-level grouping for instance segmentation with a unified architecture that embeds pixels on a hypersphere and recurrently clusters them into instances. It targets variable numbers of permutation-invariant instance labels while improving proposal generation and other pixel-level tasks.

  • Motivation: Pixel-labeling models commonly sum per-pixel classification or regression losses despite structured output spaces.
  • Motivation: Instance segmentation requires a variable number of segments and permutation-invariant instance labels, so fixed k-way classification is unsuitable.
  • Approach: The proposed model assigns unit-length embedding vectors to pixels and uses recurrent mean-shift clustering to produce discrete instance labels.
  • Experiments: The framework is applicable to boundary detection, object proposal generation, and semantic instance segmentation, with grouping loss improving representations even for binary pixel classification tasks.
  • Scope: Bottom-up proposals are less affected by the limitation that generic proposal generation is not informed by top-down semantics.
  • Results: Object proposal average recall for 10 proposals improves from 0.56 to 0.77, while the framework also provides theoretical hyperparameter guidance.

2. Related Work

The related work spans proposal-based and box-free instance segmentation, associative embedding, supervised embedding regression, and iterative clustering methods. The paper distinguishes its approach by directly generating final instances through recurrent mean-shift rather than heuristic decoding or thresholding.

  • Instance segmentation: Prior instance segmentation methods commonly generate proposals or boxes, segment foreground objects, and classify them, while box-free methods avoid some box limitations.
  • Embedding methods: Associative embedding and metric-learning approaches are closely related, but this work integrates recurrent mean-shift to generate final instances directly.
  • Embedding and clustering: Embedding-based clustering work includes normalized cuts, spatial embedding gradients for boundary detection, and learned pairwise similarity followed by embedding.
  • Iterative inference: The recurrent model resembles unrolled iterative algorithms while using fixed parameters intended to provide interpretable convergent dynamics and meaningful gradients.

3. Pairwise Loss for Pixel Embeddings

The paper learns spherical pixel embeddings with a pairwise max-margin loss, weighting pixels to balance instance contributions and analyzing embedding dimension and margin choices theoretically. Sphere-based geometry supplies bounds and practical settings for separating instances.

  • Embedding loss: The framework learns D-dimensional pixel embeddings and measures similarity with a scaled cosine similarity.
  • Embedding loss: Positive pairs are encouraged toward similarity 1, while negative pairs are separated by a margin; pixel weights control individual importance.
  • Embedding loss: The loss has a constant gradient for α = 0.5, limiting the effect of noisy labels near object boundaries.
  • Theory: Theoretical analysis guides pixel weights and the margin hyperparameter, including the role of embedding dimension and instance size.
  • Theory: The total cosine similarity of hyperspherical vectors has a dimension-independent lower bound, unlike Euclidean embeddings.
  • Instance-aware Pixel Weighting: Weighting each pixel by the inverse size of its ground-truth instance makes every instance contribute equally, independent of its size.
  • Instance-aware Pixel Weighting: When D ≥ Q, instance means can occupy orthogonal axes, yielding loss bounds independent of pixel count and embedding dimension.
  • Margin Selection: For three-dimensional embeddings, sphere-packing analysis gives α values of 0.093, 0.274, 0.395, and 0.482 for at most 4, 5, 6, and 7 instances.

4. Recurrent Mean-Shift Grouping

The recurrent mean-shift module converts imperfect pixel embeddings into instance labels through differentiable, recurrent clustering. Its convergent dynamics support end-to-end training, while backpropagation focuses learning on embedding errors that clustering will not correct.

  • Mean Shift Clustering: Mean-shift grouping addresses the problem of generating final instance segmentations from imperfect embeddings without nondifferentiable heuristic post-processing.The module operates recurrently in embedding space to congeal vectors into a small number of instance labels.
  • Mean Shift Clustering: The method uses kernel-density clustering whose modes remain in the embedding space and whose recurrent dynamics depend smoothly on the input embeddings.Mean shift assigns or moves samples toward nearby density modes; the paper uses a von Mises-Fisher kernel for unit-norm embeddings.
  • Mean Shift Clustering: For N pixel embeddings, it forms K = exp(δX^T X), computes degree-normalized affinities, and updates embeddings using the mean-shift direction.The mean shift is M = XKD^-1 − X, followed by movement with step size η.
  • Mean Shift Clustering: The recurrent updates recompute the kernel each iteration and have convergent forward dynamics under general conditions, unlike unconstrained deep recurrent networks.With η = 1 and a Gaussian kernel, the corresponding Gaussian Blurring Mean Shift method has been shown to have cubic convergence under appropriate conditions.
  • Mean Shift Clustering: On synthetic digit images, ten recurrent mean-shift iterations transform initial embeddings into compact, distinct instance labels.The visualization uses a 3-dimensional embedding so the grouping effect can be inspected directly.
  • End-to-end Training: The authors note that the eigenvalues of KD^-1 lie in [0, 1], but do not establish useful corresponding bounds for the Jacobian spectrum.This qualifies the available stability analysis despite the observed convergent behavior.
  • End-to-end Training: Backpropagation through the grouping module emphasizes ambiguous boundary embeddings while reducing updates for errors that later mean-shift iterations can correct.The full system is differentiable, and training accumulates the grouping loss across unrolled iterations, including the initial embedding regression.

5. Experiments

Experiments apply the framework to boundary detection, object proposal generation, semantic segmentation, and instance detection. The embedding-based approach achieves state-of-the-art boundary detection, high-recall proposals with far fewer proposals, and improvements across pixel-labeling tasks.

  • Boundary Detection: The model achieves state-of-the-art boundary detection performance against COB, HED, CEDN, LEP, UCM, ISCRA, NCuts, EGB, and mean shift.The comparison uses standard benchmark precision-recall on BSDS500.
  • Object Proposal Detection: Object proposals achieve high average recall with two orders of magnitude fewer proposals than competing methods.The method also outperforms SharpMask and DeepMask despite training only on PASCAL rather than the larger COCO dataset.
  • Semantic Segmentation: Semantic segmentation combines cross-entropy with the proposed embedding loss to compare models with and without pixel-pair embedding supervision.Performance is evaluated using overall mIoU and mIoU within boundary bands.
  • Semantic Segmentation: The embedding loss improves semantic segmentation by encouraging similar pixel feature vectors within segments.The analysis examines mIoU as a function of distance from ground-truth object boundaries.
  • Semantic Instance Detection: Instance detection is competitive on VOC validation, while mean-shift grouping avoids strong assumptions about object shape or topology.Compared with inst-DML, the method is reported to perform 10x better on the bike category.

6. Conclusion and Future Work

The paper concludes that spherical max-margin embeddings and recurrent mean-shift grouping form an end-to-end framework for variable-number instance recognition. It reports strong instance proposal performance and identifies extensions to other pixel-level domains as future work.

  • Conclusion: The framework combines a pixel-pairwise spherical max-margin embedding loss with recurrent mean-shift grouping.Together, the components support robust recognition of variable numbers of instances without heuristic post-processing or instance-size-specific tuning.
  • Conclusion: The approach substantially outperforms state-of-the-art methods for instance proposal generation when coupled with standard architectures.The conclusion presents the method as simple and amenable to theoretical analysis.
  • Future Work: Future work includes learnable mean-shift variants and applications to surface shape, depth, figure-ground, and multi-task embeddings.

Appendix

The appendix supplies proofs and technical details supporting the paper’s objective and recurrent grouping mechanism. It also explains gradient propagation and provides additional qualitative experiments.

  • Appendix: The appendix proves propositions introduced in the main paper for the objective function and grouping mechanism.
  • Appendix: It details mean-shift computation, gradient calculation, and adaptation of the algorithm for recurrent grouping.
  • Appendix: A toy example illustrates how gradients are back-propagated to the input embedding.
  • Appendix: Additional qualitative results cover boundary detection and instance segmentation.

1. Analysis of Pairwise Loss for Spherical Embedding

The analysis studies how spherical embeddings distribute instance representatives and how this constrains pairwise similarity and the margin parameter. These results provide guidance for setting the embedding loss and pixel weights.

  • Spherical Embedding Analysis: Proposition 1 bounds the total intra-pixel similarity of embedding vectors, including unit vectors on the hypersphere.
  • Spherical Embedding Analysis: For vectors on a 2-sphere, Proposition 2 lower-bounds the similarity of every pair.
  • Spherical Embedding Analysis: Choosing the margin parameter α below the stated pairwise lower bound causes positive loss even for a perfect embedding of n instances.
  • Spherical Embedding Analysis: The analysis treats instance representatives as points on a sphere and seeks to minimize pairwise similarity, corresponding to the Tammes or hard-spheres problem.
  • Spherical Embedding Analysis: Asymptotic results characterize the closest pair distance among optimally distributed spherical points.

2. Details of Recurrent Mean Shift Grouping

The recurrent grouping module adapts mean-shift clustering to spherical embeddings and remains differentiable through iterative updates. Backpropagation emphasizes embedding errors that clustering will not resolve, while GBMS dynamics improve grouping and convergence.

  • Mean-shift grouping: Mean shift identifies density modes by moving each point toward the mean of nearby points within bandwidth b.The bandwidth controls estimator smoothness, and modes satisfy ∇p(x) = 0.
  • Mean-shift grouping: The spherical squared-exponential kernel extends Gaussian mean shift to unit-length embedding vectors.The kernel is interpreted as a von Mises–Fisher distribution for spherical data.
  • Mean-shift grouping: GBMS alternates embedding updates with density re-estimation rather than ascending a fixed density estimate.This variant has convergence-rate guarantees for data initially arranged in compact clusters.
  • Toy example and behavior: After 10 iterations, mean shift sharpens pairwise similarity distributions and drives positive-pair similarity toward 1.The authors report that this makes instance identification easier in the instance-segmentation experiment.
  • Backpropagation: Backpropagation through recurrent mean shift increases gradients for ambiguous instance assignments and decreases them for embeddings clustering will correctly resolve.The grouping module therefore modulates the loss presented to the initial embedding space.
  • Toy example and behavior: With GBMS, data form two compact clusters at ideal label-space locations, whereas updates without GBMS leave visible clusters despite classification-oriented movement.More GBMS loops and losses across loops produce larger corrective gradients, while resolved structure need not be reproduced in the initial embeddings.

3. Additional Boundary Detection Results

On BSDS500, the learned embeddings support boundary detection and encode more than binary boundary membership after logistic-loss fine-tuning. The visualizations show boundary highlighting together with edge orientation and color-distance information.

  • Boundary detection: The model performs binary boundary classification using pixel-pair embedding loss on BSDS500 examples.The authors infer this from consistent colors in the visualized embeddings.
  • Boundary detection: After logistic-loss fine-tuning, boundary embeddings highlight boundary pixels while also capturing edge orientation and distance from conveyed colors.The comparison includes embeddings from different model layers before and after fine-tuning.
  • Boundary detection: Figures 15–18 visualize boundary predictions and three-dimensional embedding representations for randomly selected BSDS500 test images.The embeddings are shown as RGB images before and after fine-tuning.

4. Additional Results on Instance-Level Semantic Segmentation

The PASCAL VOC 2012 visualizations show consistent background embeddings and instance-level embeddings that connect disconnected regions belonging to one object. Additional figures examine embedding trajectories and projected representations across related tasks.

  • Instance-level semantic segmentation: Background pixels receive consistent embeddings, appearing with nearly the same color in the projected visualizations.The paper uses random projections from the learned 64-dimensional hyperspherical embedding space to three dimensions.
  • Instance-level semantic segmentation: The learned embeddings connect disconnected regions that belong to the same instance, including separated horses and a motorbike in the examples.The authors state that this setting poses an unsolved challenge for many other methods.
  • Instance-level semantic segmentation: Figures 19–21 visualize generic and instance-level semantic segmentation using random projections of the embedding vectors.The projected embeddings are displayed alongside semantic-segmentation results used to produce instance-level segmentation.
  • Related visualizations: The embedding visualizations also include circular-boundary structure and boundary-detection representations from related experiments.The paper describes a Mobius effect encoding orientation and distance to a boundary, while boundary embeddings capture edge properties.
Loading 1712.08273v1…