PaperlayerView exact revision on arXiv

Source-linked AI summary

PiCIE: Unsupervised Semantic Segmentation using Invariance and Equivariance in Clustering

Jang Hyun Cho, Utkarsh Mall, Kavita Bala, Bharath Hariharan

arXiv:2103.17070v1cs.CV

TL;DR

Unsupervised semantic segmentation is difficult because existing clustering methods are suited to homogeneous, object-centric images, while segmentation requires labeling every pixel in complex scenes. PiCIE extends clustering to pixel features and enforces invariance and equivariance across photometric and geometric views. It segments both stuff and things and reports large gains over prior methods on COCO and Cityscapes, while retaining important boundary and semantic-similarity limitations.

  • Problem

    Existing unsupervised clustering is mainly limited to single-class, object-centric images, whereas semantic segmentation must label every pixel in multi-class scenes.

  • Method

    PiCIE jointly learns pixel-level features and clusters while using photometric invariance and geometric equivariance as inductive biases.

  • Results

    +17.5 Acc. and +4.5 mIoU on COCO, with similar advantages of +18 Acc. and +5.3 mIoU on Cityscapes over other baselines.

  • Takeaways & Limitations

    PiCIE can discover image-level concepts and semantically parse images without supervision, including both stuff and things without rigorous tuning or task-specific preprocessing.

  • Takeaways & Limitations

    Unsupervised learning can confuse visually similar but semantically different regions, and boundary precision remains limited without boundary supervision.

Abstract

from arXiv · show

We present a new framework for semantic segmentation without annotations via clustering. Off-the-shelf clustering methods are limited to curated, single-label, and object-centric images yet real-world data are dominantly uncurated, multi-label, and scene-centric. We extend clustering from images to pixels and assign separate cluster membership to different instances within each image. However, solely relying on pixel-wise feature similarity fails to learn high-level semantic concepts and overfits to low-level visual cues. We propose a method to incorporate geometric consistency as an inductive bias to learn invariance and equivariance for photometric and geometric variations. With our novel learning objective, our framework can learn high-level semantic concepts. Our method, PiCIE (Pixel-level feature Clustering using Invariance and Equivariance), is the first method capable of segmenting both things and stuff categories without any hyperparameter tuning or task-specific pre-processing. Our method largely outperforms existing baselines on COCO and Cityscapes with +17.5 Acc. and +4.5 mIoU. We show that PiCIE gives a better initialization for standard supervised training. The code is available at https://github.com/janghyuncho/PiCIE.

1. Introduction

PiCIE addresses unsupervised semantic segmentation in uncurated, multi-label images by combining pixel clustering with photometric invariance and geometric equivariance. It reports finer segmentation of both things and stuff, with strong gains over prior work.

  • Challenge: Unsupervised semantic segmentation must discover concepts and assign accurate labels to every pixel, including both objects and background entities.Image-level recognition can ignore nondistinctive regions, whereas segmentation cannot.
  • Approach: PiCIE learns pixel-level clusters using appearance similarity while enforcing invariance to color changes and equivariance to geometric transformations.These constraints are trained end-to-end in a ConvNet without labels.
  • Results: More than doubling prior-art accuracy, PiCIE substantially outperforms earlier unsupervised semantic-segmentation methods.The paper attributes the improvement to clustering-based learning together with invariance and equivariance objectives.
  • Analysis: The invariance and equivariance objectives connect pixels across scale, pose, and color variation, supporting effective object segmentation.An ablation study reports significant performance improvements from each objective.
  • Conclusion: PiCIE shows that convolutional networks can discover image-level concepts and semantically parse images without supervision.The authors frame this as enabling large-scale discovery from unlabeled, uncurated datasets.

2. Related Work

Prior clustering methods assume semantically homogeneous data points and therefore struggle with scene-centric images. Pixel-level clustering addresses multi-class images, while PiCIE adds cross-view consistency to learn more semantic representations.

  • Image-level clustering: DeepCluster-style methods alternate between clustering feature vectors and training an encoder with cluster assignments as pseudo-labels.These approaches learn cluster-friendly embedding spaces through iterative optimization.
  • Implicit clustering: Mutual-information clustering methods avoid degenerate solutions but effectively encourage uniform cluster distributions, favoring well-balanced datasets.IIC and related methods maximize mutual information between soft assignments from two image versions.
  • Clustering limitations: Off-the-shelf clustering assumes each data point is semantically homogeneous, an assumption violated by multi-class scene-centric images.The resulting formulation is semantic segmentation by clustering pixel-level features.
  • Segmentation without labels: Pixel-level extensions such as IIC and AC remain limited to stuff categories, while PiCIE segments both stuff and things.PiCIE’s in/equivariance loss imposes geometric consistency as an inductive bias for high-level concepts.

3. PiCIE

PiCIE formulates unsupervised segmentation as jointly learned pixel clustering and representation learning. It removes the parametric pixel classifier and uses cross-view losses to enforce photometric invariance and geometric equivariance.

  • Task formulation: PiCIE assigns every pixel to a cluster while jointly learning the feature representation and clustering function.The goal is to discover visual classes and segment unseen images from an uncurated, unlabeled domain.
  • Baseline clustering approach: The baseline alternates between k-means clustering of pixel features and cross-entropy training of a pixel classifier using cluster labels.The embedding produces a feature vector for each pixel, and clustering supplies pseudo-labels.
  • Prototype-based clustering: PiCIE removes the learned parametric pixel classifier and labels pixels by their distance to k-means centroids, or prototypes.The distance function used is cosine distance.
  • Invariance and equivariance: The method adds invariance to photometric transformations and equivariance to geometric transformations so labels remain stable under color changes and warp with image geometry.For geometric transformations, the segmentation of a transformed image should transform correspondingly.
  • Photometric invariance: For photometric consistency, two transformed views are clustered separately and cross-view losses force features and cluster solutions to agree.This encourages identical discovered concepts and assignments across photometric transformations.
  • Geometric equivariance: For geometric consistency, one view uses transformed-image features while the other uses transformed features of the original image, with within-view and cross-view objectives combined.The geometric transformations include random crops and horizontal flips.

4. Experiments

PiCIE is evaluated against clustering baselines on COCO and Cityscapes, including both things and stuff categories. It substantially outperforms prior methods, with geometric consistency especially improving things segmentation and each major component contributing in ablations.

  • Datasets: COCO combines 80 things and 91 stuff categories into 27 evaluation categories, while Cityscapes uses 27 categories across street scenes.The COCO evaluation includes both things and stuff, unlike prior evaluations that considered only stuff.
  • Results: PiCIE largely outperforms baselines, with +17.5 Acc. and 4.5 mIoU on COCO and +18 Acc. and 5.3 mIoU on Cityscapes.The comparisons include modified DeepCluster and IIC adapted for pixel-level segmentation.
  • Qualitative comparison: IIC tends to overfit low-level visual cues, whereas PiCIE’s invariance/equivariance loss supports high-level concepts and segmentation of both things and stuff.The authors report that IIC correctly segments no things categories in Figure 3, while PiCIE performs well on both category types.
  • Things vs stuff: PiCIE improves mainly on things categories by +10 mIoU while maintaining better or compatible stuff performance.The result is attributed to geometric transformation equivariance being effective for objects with distinct shapes and boundaries.
  • Ablation study: Ablations add 5 points from centroid-based classification, 3 points from invariant cross-view learning, and 5.5 points from equivariance learning.With auxiliary over-clustering, the model reaches 49.99 pixel accuracy and 14.36 mIoU.
  • Analysis: Nearest neighbors of correctly predicted segments share high-level semantics, while errors include visually ambiguous snow-versus-sky confusions.The authors identify visual ambiguity as an inherent limitation of unsupervised methods.
  • Representation quality: PiCIE’s learned representations show a minimal gap between unsupervised and linear-classifier performance, suggesting clustering is the major remaining difficulty.The learned representation also provides a better initialization for supervised training.

5. Conclusion

PiCIE introduces unsupervised semantic segmentation through clustering, using geometric consistency to learn invariance and equivariance across photometric and geometric variations.

  • PiCIE incorporates geometric consistency as an inductive bias for learning invariance and equivariance to photometric and geometric variations.
  • Its cross-view loss learns high-level visual concepts needed to segment things categories without supervision.
  • PiCIE works for both stuff and things categories without rigorous hyper-parameter tuning or task-specific pre-processing.

in Clustering – Supplementary Materials

The supplementary material identifies the paper and its authors, institutions, and arXiv publication date.

  • The paper is authored by Jang Hyun Cho, Utkarsh Mall, Kavita Bala, and Bharath Hariharan.
  • The authors are affiliated with the University of Texas at Austin and Cornell University.
  • The paper was published on arXiv as version 1 on 30 March 2021.

S1. More experiment details

The supplementary experiments describe shared architectures, preprocessing, transformations, clustering, training, evaluation, and IIC hyperparameter analysis.

  • Architecture: All methods use Feature Pyramid Networks with ResNet-18, projecting intermediate feature maps to 128 dimensions before upsampling to one-quarter image resolution.
  • IIC setup: IIC experiments use FPN with ResNet-18, an auxiliary over-clustering loss with k = 45, and comparisons involving shallow and deep network variants.
  • Baselines: Modified DeepCluster alternates between computing pixel-level pseudo-labels and training, using mini-batch k-means because storing all dataset feature vectors is infeasible.
  • Preprocessing: Modified DeepCluster and PiCIE use resizing and center-cropping to 320 × 320, while IIC uses preprocessing from its original published implementation.
  • Transformations: Photometric augmentation includes color jitter, grayscale, and Gaussian blur, while geometric augmentation includes random crops and horizontal flips.Color jitter is applied with probability p = 0.8, grayscale with p = 0.2, and Gaussian blur with p = 0.5.
  • Clustering: Cluster centroids are computed with GPU mini-batch k-means using FAISS, with initial centroids from 50 batches and updates every 20 iterations.Preparing pseudo-labels for one COCO epoch takes about 20 minutes, according to the passage.
  • Training: Training uses 10 epochs with ImageNet initialization and 20 epochs from scratch; modified DeepCluster and PiCIE use ADAM with η = 1 × 10^-3.IIC retains its original learning rate η = 1 × 10^-4.
  • Evaluation: Evaluation reports Hungarian-matched pixel accuracy and mean IoU, with separate evaluation procedures for class partitions and stuff-only categories.The stuff-only setting uses image resolution 128 × 128 and K = 15.

S2. More results

Additional results include randomly selected qualitative comparisons and test-time augmentation experiments assessing PiCIE’s learned transformation consistency.

  • Qualitative results: Additional qualitative results are shown for IIC, IIC-res12, modified DeepCluster, and PiCIE.
  • Transformation robustness: PiCIE is evaluated with test-time photometric and geometric transformations to assess its learned invariance and equivariance.
  • Transformation robustness: The test-time transformations include color jitter, Gaussian blur, grayscale, horizontal flips, and random crops.

S3. Analysis

PiCIE analysis identifies visual ambiguity, class co-occurrence, and boundary precision as practical limitations, while test-time augmentation indicates robustness to photometric and geometric transformations.

  • Test-time augmentation: Test-time augmentation evaluates transformations using the same hyperparameters as training and shows robustness to photometric and geometric transformations during inference.The evaluated transformations include brightness, contrast, saturation, hue, grayscale, Gaussian blur, horizontal flip, and random crop.
  • Visual ambiguity: Visual ambiguity causes confusion when semantically different classes share similar appearance, such as snowy ground resembling sky or water.The paper describes this as an inherent limitation of unsupervised learning methods.
  • Co-occurrence: Stuff pixels can subsume foreground classes such as boats or airplanes when those objects consistently co-occur with water or sky.The authors hypothesize that more stand-alone examples or boundary-based contrast could mitigate this effect.
  • Boundary precision: Precise boundaries remain difficult because the method lacks supervision for boundary localization, causing over-confident segmentation around foreground instances.Generic edge detection or iterative refinement such as CRF is suggested as future work outside the project’s scope.
Loading 2103.17070v1…