PaperlayerView exact revision on arXiv

Source-linked AI summary

Unsupervised Learning of Image Segmentation Based on Differentiable Feature Clustering

Wonjik Kim, Asako Kanezaki, Masayuki Tanaka

arXiv:2007.09990v1cs.CV

TL;DR

Unsupervised image segmentation must discover meaningful regions while satisfying competing similarity, continuity, and label-diversity criteria. The paper jointly learns CNN feature extraction and differentiable clustering, and extends the method to scribble-guided and unseen-image segmentation. Experiments on benchmark datasets report strong segmentation performance, including better accuracy than existing methods for scribble-guided segmentation while maintaining efficiency.

  • Problem

    Unsupervised segmentation must partition images into an arbitrary number of salient or meaningful regions without prior knowledge.

  • Method

    The method alternates cluster-label prediction and network learning using a CNN with differentiable feature clustering, similarity and spatial continuity losses, and normalized responses before argmax assignment.

  • Results

    The proposed approach was effective on benchmark segmentation datasets, and its scribble-guided extension achieved better accuracy than existing methods while maintaining efficiency.

  • Takeaways & Limitations

    The framework supports completely unsupervised segmentation as well as scribble-guided and unseen-image segmentation using networks pretrained on a few reference images without retraining.

Abstract

from arXiv · show

The usage of convolutional neural networks (CNNs) for unsupervised image segmentation was investigated in this study. In the proposed approach, label prediction and network parameter learning are alternately iterated to meet the following criteria: (a) pixels of similar features should be assigned the same label, (b) spatially continuous pixels should be assigned the same label, and (c) the number of unique labels should be large. Although these criteria are incompatible, the proposed approach minimizes the combination of similarity loss and spatial continuity loss to find a plausible solution of label assignment that balances the aforementioned criteria well. The contributions of this study are four-fold. First, we propose a novel end-to-end network of unsupervised image segmentation that consists of normalization and an argmax function for differentiable clustering. Second, we introduce a spatial continuity loss function that mitigates the limitations of fixed segment boundaries possessed by previous work. Third, we present an extension of the proposed method for segmentation with scribbles as user input, which showed better accuracy than existing methods while maintaining efficiency. Finally, we introduce another extension of the proposed method: unseen image segmentation by using networks pre-trained with a few reference images without re-training the networks. The effectiveness of the proposed approach was examined on several benchmark datasets of image segmentation.

I. INTRODUCTION

The paper frames unsupervised segmentation as partitioning images into meaningful regions without prior knowledge, then proposes a CNN-based method that jointly learns features and clustering while balancing incompatible criteria.

  • Problem: Unsupervised segmentation seeks an arbitrary number of salient or meaningful regions without previous knowledge.The task is more challenging than supervised labeling because the desired number and meaning of regions are not specified.
  • Segmentation criteria: Good segmentation should group similar-feature pixels, preserve spatial continuity, and favor many unique cluster labels.These criteria reflect grouping coherent object regions while separating neighboring regions with dissimilar patterns.
  • Proposed method: The proposed CNN jointly optimizes feature extraction and clustering functions through iterative differentiable cluster-label prediction.This enables end-to-end learning without ground-truth labels.
  • Contributions: A spatial continuity loss replaces fixed superpixel boundaries from the previous method.The study identifies fixed boundaries as a limitation and introduces the loss to mitigate it.
  • Applications: The method is extended to scribble-guided segmentation and unseen-image segmentation using networks pretrained on a few reference images without retraining.The scribble extension reports better accuracy than existing methods while maintaining efficiency.

II. RELATED WORK

The related work covers classical, user-guided, weakly supervised, and deep unsupervised segmentation methods, highlighting limitations involving fixed boundaries and iterative or indirect learning procedures.

  • Classical segmentation: Classical segmentation includes k-means clustering and graph-based segmentation, which assign pixels using feature-centroid proximity or region comparison decisions.The proposed method shares the goal of unsupervised segmentation but is positioned within learning-based approaches.
  • Learning-based unsupervised segmentation: Superpixel-based methods such as MsLRR impose segment boundaries fixed by the superpixel extraction process.This limitation motivates the proposed method’s alternative treatment of spatial continuity.
  • User-guided segmentation: Graph cut supports user-guided segmentation with inputs such as scribbles and bounding boxes, while α-β swap and α-expansion extend graph optimization to multi-label problems.These algorithms repeatedly optimize binary labeling moves for selected labels or label pairs.
  • Weak supervision: Weakly supervised CNN methods typically alternate between generating training targets from weak labels and gradient-based model training.ScribbleSup is cited as an example that propagates scribble labels to other pixels.
  • Deep unsupervised learning: Deep embedded clustering refines clusters using KL divergence and an auxiliary target distribution, whereas the proposed method minimizes softmax loss from estimated clusters.The paper distinguishes its approach as completely unsupervised image segmentation.

III. METHOD

The method alternates cluster-label prediction and CNN parameter learning to jointly optimize feature extraction and clustering for unsupervised segmentation. A normalized response map and argmax assignment produce pseudo-labels used to optimize similarity and spatial continuity.

  • III. METHOD: The method alternates predicting cluster labels with fixed network functions and training feature-extraction and clustering parameters with fixed labels.This converts unsupervised learning into two alternating sub-problems.
  • III. METHOD: The proposed formulation balances feature similarity and spatial continuity, whereas classical clustering and graph-cut combinations operate on fixed feature representations.The fixed-feature restriction can make classical solutions suboptimal when the representation should also be learned.
  • III. METHOD: A CNN jointly learns pixel feature representations and clustering functions so similar-feature pixels receive the same labels.The architecture and loss function address the stated clustering criteria.
  • III. METHOD: The network maps pixel features to q-dimensional response vectors, normalizes them across cluster-space axes, and assigns labels with argmax.The figure describes q = 3 as an illustration and uses the resulting labels as pseudo-targets.
  • III. METHOD: Argmax clustering assigns each pixel to one of q representative axis points, while some clusters may remain empty and the effective label count can range from 1 to q.The representative points are placed at infinite distance on the respective cluster-space axes.

2) Constraint on the number of unique cluster labels:

The method makes the number of active cluster labels adaptive to image content while discouraging collapse into a single segment. It starts with a maximum label count and relies on similarity and continuity constraints to integrate clusters.

  • 2) Constraint on the number of unique cluster labels:: The effective number of labels q′ adapts to image content, with 1 ≤ q′ ≤ q; large q′ indicates oversegmentation and small q′ indicates undersegmentation.The maximum possible label count q is initialized to a large value.
  • 2) Constraint on the number of unique cluster labels:: Similarity and spatial continuity constraints integrate similar or nearby pixels during iterative updates, reducing q′ without an explicit label-count constraint.The reduction emerges from the iterative update process.
  • 2) Constraint on the number of unique cluster labels:: Because grouping criteria alone could yield q′ = 1, the method additionally prefers a large q′ to prevent undersegmentation.The large-label preference is implemented through intra-axis normalization before argmax assignment.
  • 2) Constraint on the number of unique cluster labels:: Intra-axis normalization gives response dimensions zero mean and unit variance, giving each axis an even chance to attain the maximum.It does not guarantee that every cluster index is selected, but it causes many indices to achieve maxima.
  • 2) Constraint on the number of unique cluster labels:: The normalization process consequently gives the system a preference for a large effective number of labels q′.

B. Loss function

The loss function combines feature-similarity and spatial-continuity constraints, with a weighting parameter controlling their balance. For scribble-guided segmentation, an additional weighted scribble-information term is incorporated.

  • B. Loss function: The proposed loss combines a feature-similarity constraint with a spatial-continuity constraint.These terms jointly encode the two pixel-grouping requirements.
  • B. Loss function: The parameter µ balances the feature-similarity and spatial-continuity constraints.
  • B. Loss function: For scribble-guided segmentation, the loss is modified by adding a scribble-information term weighted by ν.The scribble extension retains the feature-similarity and spatial-continuity components.
  • B. Loss function: Each component of the loss function is described in its respective section.

1) Constraint on feature similarity:

Feature similarity is enforced by using argmax-derived cluster labels as pseudo-targets in a cross-entropy loss. Minimizing this loss encourages within-cluster feature similarity and between-cluster feature separation.

  • 1) Constraint on feature similarity:: Argmax applied to the normalized response map produces cluster labels that serve as pseudo-targets for training.
  • 1) Constraint on feature similarity:: A cross-entropy loss between normalized responses and argmax-derived labels enforces the feature-similarity constraint.The loss is computed over response dimensions and pixels using the assigned cluster labels.
  • 1) Constraint on feature similarity:: The objective is to make feature vectors within the same cluster similar and feature vectors from different clusters different.
  • 1) Constraint on feature similarity:: Minimizing the loss updates network weights to extract features that are more efficient for clustering.

2) Constraint on spatial continuity:

The method adds a spatial continuity constraint that favors neighboring pixels receiving the same cluster label. It implements this constraint through horizontal and vertical response-map differences, suppressing excessive labels from complex patterns or textures.

  • Spatial continuity loss favors assigning neighboring pixels the same cluster label.
  • The loss uses the L1-norm of horizontal and vertical differences in the response map as a spatial constraint.
  • Algorithm 1 alternates label assignment and network updates for T iterations, using the continuity-loss weight µ as input.
  • The response-map dimensions are tied to the input image width and height in the continuity-loss formulation.
  • The spatial continuity loss can suppress excessive labels caused by complicated patterns or textures.

3) Constraint on scribbles as user input:

For scribble-guided segmentation, the approach introduces a scribble loss based on partial cross entropy and incorporates scribble labels for marked pixels. The trained network can also be reused on similar target images without retraining.

  • The scribble loss is formulated as partial cross entropy for segmentation with user-provided scribbles.
  • The binary variable u_n identifies scribbled pixels, while s_n specifies each scribbled pixel’s label.
  • Training alternates cluster-label prediction with network-parameter learning while holding the other quantities fixed.
  • Networks pre-trained with one or more similar reference images can be reused for target segmentation without retraining.
  • Batch normalization between the final convolution and argmax layers helps obtain reasonable labels in the proposed CNN.

IV. EXPERIMENTAL RESULTS

Experiments on PASCAL VOC 2012 and BSD500 show strong benchmark performance, smoother segment boundaries, and complementary benefits from continuity loss and batch normalization. The method’s precision-recall evaluation was strongest across the reported cases, while parameter sensitivity depended on the task.

  • Effect of continuity loss: Changing µ altered segmentation detail: µ = 1 produced finer regions, while the preferred setting depended on the desired detail level.
  • Parameter search: µ = 5 was best for unsupervised segmentation, whereas ν = 0.5 was best for segmentation with user input.
  • Unsupervised segmentation: The proposed method produced smoother and more salient segment boundaries than the previous method, with improved performance in Table I.
  • Unsupervised segmentation: The proposed method achieved the best or second-best mIOU scores on PASCAL VOC 2012 and BSD500.
  • Precision-recall evaluation: The proposed method with continuity loss achieved the best average precision scores in every reported Fig. 5 case.
  • Ablation study: Batch normalization consistently and considerably improved performance across datasets, while continuity loss provided solid additional improvement when combined with it.

B. Segmentation with scribbles as user input

With scribble input, the proposed method produced more accurate segmentation than the compared approaches while using a substantially smaller and faster network without pretrained weights.

  • Quantitative and qualitative comparison: The proposed method achieved the best mIOU score among the methods evaluated for segmentation with user input.The comparison included graph cut variants and regularized loss with DeepLab-ResNet-101.
  • Quantitative and qualitative comparison: The proposed method produced more accurate segmented boundaries and was more stable than graph-based methods in the exemplar results.Regularized loss produced relatively rougher object segments, whereas the proposed method yielded more accurate boundaries.
  • Initialization: The proposed method uses random-weight initialization, whereas regularized loss training requires pretrained weights.The compared regularized-loss method failed to train from random states with both DeepLab-ResNet-101 and DeepLab-largeFOV in the reported experiment.

C. Unsupervised segmentation with reference images

Networks trained on a few reference images were used to segment unseen images without retraining, producing acceptable results while greatly reducing per-image segmentation time.

  • Unseen image segmentation: The reference-image experiment trained the proposed method on four BSD500 images, fixed the weights, and then segmented three other images.Reference and test images were selected from different nature scenes.
  • Unseen image segmentation: Segmentation with reference images produced more detailed results and acceptable performance compared with training from scratch on each test image.The reported explanation is that training from scratch integrates clusters under the influence of the continuity loss.
  • Efficiency: Under GPU calculation, reference-image segmentation took under 0.02s per image, compared with approximately 20s when training from scratch.This comparison was reported on a GeForce GTX TITAN X.
  • Single-reference extension: A single reference image produced sufficient test-image segmentation, although test results were less appropriate than the corresponding reference-image result.The experiment reported successful same-label assignment for fishes in one case and differentiated oranges in another.
  • Video extension: The method was extended to video segmentation by training on only part of the frames and using a single frame as a reference for the others.The paper reports that segment matching across frames enabled efficient segmentation without ground truth, with further improvement possible by adding processed images as references.

V. CONCLUSION

The study presents an end-to-end CNN for unsupervised image segmentation that learns features and cluster labels jointly, and demonstrates effectiveness on benchmark datasets and extensions with user input and reference images.

  • Conclusion: The proposed CNN combines convolutional feature extraction with differentiable feature clustering for end-to-end unsupervised image segmentation.Backpropagation updates convolutional filters through the proposed loss and normalized layer responses.
  • Conclusion: Experiments on PASCAL VOC 2012 and BSD500 demonstrated the effectiveness of the proposed method for completely unsupervised segmentation.The conclusion also reports that it outperformed k-means clustering and a graph-based segmentation method.
  • Conclusion: The method was extended to segmentation with scribbles and to utilization of reference images.These applications are identified as extensions based on the proposed segmentation method.
Loading 2007.09990v1…