PaperlayerView exact revision on arXiv

Source-linked AI summary

Self-Supervised Learning of Object Parts for Semantic Segmentation

Adrian Ziegler, Yuki M. Asano

arXiv:2204.13101v2cs.CV

TL;DR

Unsupervised segmentation needs dense representations, but object definitions vary and image-level self-supervision does not directly specify how to learn them. Leopart learns object parts by clustering ViT spatial tokens, using the ViT’s object-localizing attention. It improves results across three semantic-segmentation benchmarks and also advances fully unsupervised segmentation with community-detection overclustering.

  • Problem

    Object definitions vary across segmentation tasks, while image-level self-supervised learning has not addressed the spatially diverse representations needed for unsupervised segmentation.

    Source: §1 · pp. 1–2

  • Method

    Leopart fine-tunes ViT spatial tokens with dense patch clustering to learn object parts that can be grouped into objects under different definitions.

    Source: §1 · p. 2

  • Results

    17%-3%: Leopart surpasses the state of the art on Pascal VOC, COCO-Thing, and COCO-Stuff; its fully unsupervised approach surpasses the state of the art by >6%.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The results support object parts as transferable representations across segmentation benchmarks with different object definitions and as co-occurring parts for fully unsupervised segmentation.

    Source: §5 · pp. 8–9

    Patch-level learning limits segmentation resolution and detection, so the method is unsuitable when fine-grained pixel segmentation or objects smaller than a patch are required.

    Source: §5 · p. 8

Abstract

from arXiv · show

Progress in self-supervised learning has brought strong general image representation learning methods. Yet so far, it has mostly focused on image-level learning. In turn, tasks such as unsupervised image segmentation have not benefited from this trend as they require spatially-diverse representations. However, learning dense representations is challenging, as in the unsupervised context it is not clear how to guide the model to learn representations that correspond to various potential object categories. In this paper, we argue that self-supervised learning of object parts is a solution to this issue. Object parts are generalizable: they are a priori independent of an object definition, but can be grouped to form objects a posteriori. To this end, we leverage the recently proposed Vision Transformer's capability of attending to objects and combine it with a spatially dense clustering task for fine-tuning the spatial tokens. Our method surpasses the state-of-the-art on three semantic segmentation benchmarks by 17%-3%, showing that our representations are versatile under various object definitions. Finally, we extend this to fully unsupervised segmentation - which refrains completely from using label information even at test-time - and demonstrate that a simple method for automatically merging discovered object parts based on community detection yields substantial gains.

1. Introduction

Object definitions vary across segmentation benchmarks, making dense self-supervised learning difficult to guide without labels. The paper learns object-part representations with a ViT, then evaluates their transfer to semantic segmentation and fully unsupervised segmentation.

1. Introduction

  • Object-part representations can be grouped into objects after generic pretraining, offering an alternative to choosing one object definition during training.Object definitions differ in granularity and in what they count as objects, such as stuff and things.Source: §1 · pp. 1–2
  • Human object priors such as saliency and contour detectors can make training effectively semi-supervised and may not generalize.The paper instead stays fully unsupervised and explores dense image-patch clustering.Source: §1 · p. 2
  • Leopart combines a ViT’s object-localizing attention with a dense loss to train spatial tokens for unsupervised segmentation.The contribution is a dense clustering pretext task for semantically rich spatial tokens.Source: §1 · p. 2
  • Leopart outperforms self-supervised ViTs and ResNets in fully unsupervised segmentation and learns transferable dense representations.The paper notes that self-supervised ViTs localize objects but lack category-discriminative spatial embeddings, while supervised ViTs show the reverse pattern.Source: §1 · p. 2
  • 17%-3%: the pretext task surpasses the state of the art on Pascal VOC, COCO-Thing, and COCO-Stuff semantic segmentation.The benchmarks feature different object definitions and granularities.Source: §1 · p. 2
  • >6%: cluster-based foreground extraction and community-detection overclustering surpass the state of the art for fully unsupervised semantic segmentation.This setup refrains from using label information even at test time.Source: §1 · p. 2

2. Related Work

Prior self-supervised methods largely learn image-level representations, while dense methods use spatial objectives or external object priors. Leopart adapts clustering to patch-level teacher–student learning and uses ViT attention to focus on foreground patches.

2. Related Work

  • Image-level self-supervised objectives include instance-level augmentation prediction or discrimination and group-level clustering or teacher-generated targets.The cited approaches include contrastive losses, optimal-transport pseudo-labels, and mutual-information clustering.Source: §2 · p. 2
  • Leopart adapts SwAV’s clustering pretext task and multi-crop strategy to patch-level teacher–student learning.The pipeline produces patch-level cluster predictions and assignment targets from different student and teacher crops, aligning targets and focusing on foreground patches with ViT attention.Source: §2 · p. 2, Figure 2 · p. 3
  • DenseCL establishes spatial correspondences across views, while other dense methods use region proposals, contour detectors, saliency, or object masks as object priors.MaskContrast uses a pretrained saliency estimator to mine positive and negative pixels.Source: §2 · pp. 2–3
  • Unlike a concurrent clustering-plus-contrastive method, Leopart uses one clustering objective for dense settings and does not require an external saliency generator.The cited alternative combines intra-image pixel clustering with an image-level loss, which the passage says is not well suited to generic multi-object images.Source: §2 · p. 3
  • Prior object-part segmentation determines parts given object masks, whereas Leopart uses part representations as an intermediary for semantic segmentation.Another concurrent work merges DINO attention patch keys to detect objects; Leopart fine-tunes DINO spatial tokens for segmentation.Source: §2 · p. 3

3. Method

The method fine-tunes pretrained ViT spatial tokens with dense clustering to learn reusable object-part representations, then uses token clustering for fully unsupervised segmentation.

3. Method

  • Patch embeddings group the same object parts, which can recombine into different objects and are expected to transfer better across datasets.The method uses clustering as an image-patch-level pretext task for intra-image category learning.Source: §3 · p. 3

3.1. Fine-tuning loss for spatial tokens

  • Leopart assigns spatial-token features to balanced prototype clusters and trains the network to predict those assignments using cross-entropy.It fine-tunes pretrained networks and optimizes over spatial features across the image batch.Source: §3.1 · pp. 3–4
  • Global crops provide assignment targets, while local-crop predictions support object-part-to-category reasoning and are aligned over overlapping image regions.The alignment uses crop-intersection boxes and RoI-Align; attention-derived foreground masks weight the training loss.Source: §3.1 · p. 4

3.2.1 Cluster-based Foreground Extraction (CBFE)

  • Fully unsupervised segmentation uses K-means on learned spatial tokens and derives a foreground mask by ranking clusters against ViT attention hints.The attention maps are averaged, smoothed, and thresholded to retain 60% of their mass.Source: §3.2 · p. 4, §3.2.1 · p. 4

3.2.2 Overclustering with community detection (CD)

  • Because label-guided grouping can inflate overclustering results, the method proposes merging clusters without additional supervision.The passage notes that assigning one cluster per pixel would amount to providing full supervision signals.Source: §3.2.2 · p. 4
  • The method links clusters by localized co-occurrence, then uses Infomap communities to merge them to the desired class count for evaluation.Its asymmetric edge weights reflect that parts need not predict one another mutually; Hungarian matching makes evaluation permutation-invariant.Source: §3.2.2 · pp. 4–5

4. Experiments

The experiments assess Leopart through design and training-data ablations, transfer learning across datasets, and fully unsupervised segmentation. The reported results include strong transfer performance and gains from embedding-based foreground extraction and community detection.

4.1. Setup

  • Evaluation uses frozen ViT spatial tokens for linear classification or K-means overclustering, reports mIoU, and averages overclustering across five seeds.The experiments also include FCN fine-tuning.Source: §4 · p. 5, §4.1 · p. 5

4.2. Fine-Tuning Loss Ablations

  • COCO training improves performance by up to 6% over ImageNet-100, while weaker PVOC results support the importance of larger training datasets for ViTs.PVOC training used 500 epochs so models received the same number of iterations.Source: Table 2 · p. 5, §4.2 · p. 6
  • Foreground clustering gains up to 3% over clustering all spatial tokens and up to 10% over clustering mostly background tokens.Attention maps guide clustering toward foreground but are treated as noisy approximations, not as excluding background objects.Source: §4.2 · p. 5
  • Using local crops gives a larger performance gain than adding more crops, and overclustering is more affected, indicating their importance for a semantically structured embedding space.The comparison is four local crops versus two or no local crops.Source: §4.2 · pp. 5–6

4.4. Fully unsupervised semantic segmentation

  • In fully unsupervised PVOC12 segmentation, Leopart exceeds MaskContrast by >6%, using embedding-based foreground extraction and community detection to merge overclusters.Evaluation uses no test-time label information, including the number of final clusters.Source: Table 4 · p. 6, §4.4 · p. 7

4.3. Transfer learning

  • PVOC12 gains exceed 17% (LC) and 5% (overclustering); COCO-Things and COCO-Stuff gains exceed 5%/3% (LC) and 8%/10% (overclustering).The comparisons are against prior self-supervised works.Source: Table 3 · p. 6, §4.3 · p. 6

4.4.1 Performance gain study

  • Leopart improves on DINO by >14%; CBFE improves over DINO attention masks by >9%, foreground-only K-means by >17%, and community detection by >5%.These component gains are reported for PVOC segmentation with K=21.Source: Table 5 · p. 6, §4.4.1 · pp. 7–8

4.3. Transfer learning

  • Fine-tuning a full FCN outperforms prior work, with an around 2% gain from fine-tuning; ViT-B/8 improves FCN performance by around 5%.The paper hypothesizes that Leopart’s embedding space is already close to maximally informative for segmentation under linear transformations.Source: Table 6 · p. 6, §4.3 · p. 7

5. Discussion

The discussion presents object parts as a principled alternative for segmentation across differing object definitions, while noting limits from patch-level maps and hard cluster-to-community assignments. It also stresses monitoring potential societal impacts of self-supervised segmentation.

5. Discussion

  • Patch-level learning limits segmentation resolution and detection, failing on fine-grained pixel tasks or objects smaller than an image patch.Hard cluster-to-community assignments can also misclassify object parts shared across objects when they appear in a specific context.Source: §5 · p. 8
  • Self-supervised segmentation requires rigorous monitoring because unfamiliar segmentations or manifested data biases could cause harm in autonomous driving and virtual reality.The paper notes that these methods can scale to large datasets with little or no human labeling.Source: §5 · p. 8
  • The method improves state of the art on PVOC, COCO-Stuff, and COCO-Thing benchmarks with different object definitions and granularities.The conclusion also describes the learned embedding space as directly usable for fully unsupervised segmentation, where objects can be defined as co-occurring parts.Source: §5 · pp. 8–9

A.1. Implementation details

The appendix documents the model, clustering and augmentation setup, configurations for fully unsupervised experiments, evaluation procedures, and qualitative mask behavior.

A.1. Implementation details

  • The model uses Torch and PyTorch Lightning, Faiss for K-Means, and MapEquation for community detection; training uses ViT-Small (21M parameters) versus ResNet-50’s 23M.The teacher follows an exponential moving average of student weights, with a cosine schedule from 0.9995 to 1 over 50 training epochs.Source: §A.1 · p. 12
  • A three-layer projection head produces 256-dimensional outputs for the L2 bottleneck and prototype matrix, with an 8192-length Sinkhorn-Knopp queue.Training uses temperature 0.1 and Adam with cosine weight decay.Source: §A.1 · p. 12
  • Training aligns local and global crop feature maps to 7x7; 224x224 global and 96x96 local crops must overlap by at least 1% of the original image.Figure 6 shows each global crop receiving N − 1 boxes for the remaining crops’ targets, while each local crop receives Ngc boxes for global-crop targets.Source: §A.1 · p. 12, Figure 6 · p. 12
  • For fully unsupervised PVOC experiments, CBFE uses 200 clusters; community detection uses 150 clusters, 20 communities, a 9% edge threshold, Markov time 2, and 10 seeds.The precision thresholds used for foreground classification are reported in Table 8.Source: §A.1 · p. 12, Table 8 · p. 12
  • Qualitative cluster masks generally capture object shapes but sometimes include excess background and miss small objects.The paper identifies missed small objects as a limitation.Source: Figure 7 · p. 13
  • Evaluation interpolates spatial features before heads or upsamples cluster labels, matches ResNet and ViT resolution, and uses downsampled 100x100 masks for overclustering.Linear heads train for 25 epochs and FCN heads for 30; the FCN uses two convolutional layers, a skip connection, and 2D dropout.Source: §A.1 · p. 13

A.2. Additional Experiments

Additional experiments report gains from larger backbones and varied initializations, compare against DenseCL, assess queue use, and test correspondence on annotated object parts.

A.2. Additional Experiments

  • Up to 6% PVOC transfer gains, around 1% on COCO-Thing and 2% on COCO-Stuff, accompany a larger model’s additional 5.5% state-of-the-art improvement.For fully unsupervised segmentation, ViT-B/8 has a 41.9% relative gain over DINO initialization versus 37.1% for ViT-S/16; foreground extraction also improves by more than 4%.Source: §A.2 · p. 13, Table 9 · p. 14
  • Leopart fine-tuning improves over Moco-v3, MAE, and supervised initializations, including +28% in K=500 performance when starting from MAE.Table 10 presents transfer-learning results across the initializations.Source: §A.2 · p. 14, Table 10 · p. 14
  • On PVOC12, DenseCL with DINO initialization reaches 54% for LC and 17.1% for K=500, while Leopart exceeds it by >15% and >40%, respectively.DenseCL improves on DINO initialization by +3.4% for LC and −0.3% for K=500.Source: §A.2 · p. 14
  • The queue improves clustering performance, and the authors attribute sufficient diversity for equi-partitioned clustering to this mechanism.Table 11 is the queue ablation.Source: §A.2 · p. 14, Table 11 · p. 14
  • On ADE20K street-scene parts, Leopart improves DINO’s parts mIoU by 1.9% at K=500 and 3% at K=1000, supporting increased object-part correspondence.The evaluation uses annotations for 111 part classes across 1983 images; the supervised ViT is not superior to DINO at discovering parts.Source: §A.2 · p. 14, Table 12 · p. 14

A.3. Additional visualizations

The appendix provides visualizations of cluster masks and segmentation maps at several stages of grouping.

A.3. Additional visualizations

  • PVOC12 visualizations show cluster masks and segmentation maps, including additional community-detection results.The paper also provides more cluster masks for PVOC12 validation.Source: §A.3 · p. 14
  • Unmerged foreground clustering results use K=100 to illustrate the segmentation granularity of each object.Source: §A.3 · p. 14
  • Classic overclustering visualizations group clusters into objects using label information.Source: §A.3 · p. 14

A.4.1 PASCAL

The PASCAL appendix specifies head-tuning and evaluation splits, then illustrates part clustering and two ways of merging clusters into objects.

A.4.1 PASCAL

  • Head fine-tuning uses 10,582 trainaug images; evaluation uses 1,449 PVOC12 val images, while unsupervised-method tuning uses 1,464 PVOC12 train images.Evaluation also ignores unlabelled objects and the boundary class.Source: §A.4.1 · p. 14
  • Cars and humans are often split into parts, while birds tend to remain whole; cluster colors are not unique in the K=100 visualization.Source: Figure 8 · p. 15
  • Community detection merges object-part clusters into objects in most shown cases, but hard cluster-to-community assignment misassigns some wheels and merges bicycles with motorcycles.Each cluster can belong to only one object; the examples show a bus wheel assigned to car and bicycles merged with motorcycles.Source: Figure 9 · p. 15
  • Ground-truth merging of 500 clusters produces crisper maps and keeps similar categories separate, though some objects remain only partly segmented after precision-based matching to background.Source: Figure 10 · p. 16

A.5. COCO

The appendix defines separate COCO stuff and thing label setups and evaluation data, then describes an ADE20k street-scene subset with part annotations.

A.5. COCO

  • COCO-Stuff merges 91 fine labels into 15 coarse labels and assigns non-stuff objects “other”; non-stuff objects are ignored during training and evaluation. COCO-Thing merges 80 fine categories into 12 object classes; background is ignored during training and evaluation.Source: §A.5 · p. 16
  • COCO head fine-tuning uses 10% of the data, or 11,829 images, and evaluation uses all 5,000 validation images.Source: §A.5 · p. 17

A.5.1 ADE20k

  • ADE20k contains 3,687 potential part objects, but evaluation is restricted to street scenes with part annotations: 1,983 images and 111 object parts.Source: §A.5.1 · p. 17
  • The ADE20k examples show street-scene images and part masks; dark-green non-part pixels are ignored during feature-space clustering evaluation.Source: Figure 13 · p. 18, §A.5.1 · p. 18

Related papers

Loading 2204.13101v2…