Large-scale Unsupervised Semantic Segmentation

1 INTRODUCTION

• LUSS asks whether semantic segmentation can extend to diverse, large-scale real-world environments, beyond datasets with limited scale or scene diversity.The introduction contrasts restricted benchmarks such as PASCAL VOC and BDD100K with scaling segmentation toward ImageNet.Source: §1 · p. 1
• LUSS must learn category and shape representations together, cluster large-scale data, and assign self-learned labels to pixels without human-annotation supervision.Category features distinguish classes; shape cues such as objectness and boundaries support pixel-level segmentation, and conflicting representations may produce incorrect results.Source: §1 · p. 1
• Learned shape representations may support pretraining for pixel-level downstream tasks, while semi-supervised fine-tuning may support settings with a small amount of labeled data.The introduction names semantic and instance segmentation as downstream examples.Source: §1 · p. 1
• ImageNet-S provides a high-diversity LUSS benchmark with 919 categories, about 1.2 million training images, and nearly 50k pixel-level annotations for evaluation.The annotations include 40k validation images and about 9k training images; the dataset also supports multiple categories per image and multiple evaluation protocols.Source: §1 · p. 2
• The proposed method combines unsupervised representation learning, label generation, and fine-tuning, using pixel-to-pixel alignment, deep-to-shallow supervision, and pixel attention.The authors describe these strategies as enhancing representations and facilitating pixel-level label generation and fine-tuning at large scale.Source: §1 · p. 2

2.1 Unsupervised Segmentation

• Earlier unsupervised segmentation methods often omit category recognition, while unsupervised semantic segmentation methods address categories but largely target small datasets and relatively few, easy categories.The authors also note that large-scale computation and standardized evaluation are often unaddressed, and some methods use supervised prior knowledge.Source: §2.1 · p. 2

2.2 Self-supervised Representation Learning

• Contrastive methods pull positive views together and push negative pairs apart; non-contrastive methods avoid negative pairs, but both lack an explicit category concept in the methods discussed.The authors say this can make them less effective for category-related tasks, where same-category instances need not otherwise share similar representations.Source: §2.2 · p. 2
• Clustering-based self-supervision encourages images toward cluster centers, while pixel-level methods improve spatial representations but may ignore category-related representation ability.The authors characterize clustering as encouraging stronger category-related representations; examples of pixel-level approaches use spatial consistency or matched locations.Source: §2.2 · pp. 2–3

2.3 Weakly Supervised Semantic Segmentation

• Weakly supervised segmentation shares LUSS’s need for shape features, but common modules relying on supervised pretraining or image-level labels are not applicable to LUSS.The authors list affinity prediction, region separation, boundary refinement, joint learning, and sub-category exploration as possible alternative modules.Source: §2.3 · p. 3

3.1.1 Image Annotation.

• Validation/testing masks cover all selected categories, while training annotations label ten images per category and mark other selected objects as ‘ignore’.Images below 1,000 × 1,000 are resized for polygon annotation; the process corrected 296 mislabeled images and found 942 images with missing labels.Source: §3.1.1 · pp. 3–5

3.1 Large-scale LUSS Dataset: ImageNet-S

• ImageNet-S contains 1,183,322 training, 12,419 validation, and 27,423 testing images across 919 categories.Precise pixel masks cover 39,842 validation/testing images and 9,190 training images; ImageNet-S50 and ImageNet-S300 offer smaller-budget subsets.Source: §3.1 · p. 3, §3.1.2 · pp. 5–6

3.1.2 Statistics and distribution.

• Image counts are balanced across most categories, but pixel counts have a long-tail distribution; 8.6% of images contain multiple categories.The pixel imbalance may introduce challenges not considered in image-level representation learning.Source: §3.1.2 · p. 5
• Most objects are relatively small and tend toward image centers, while their boundaries cover almost all image areas.ImageNet-S has position distributions similar to COCO and Open Images, and its distribution map is almost identical to Open Images.Source: §3.1.2 · p. 5
• Across four annotators on 100 images, mask mIoU is 98.7% and boundary mIoU is 92.4% at d = 2%.Smaller objects have lower annotation consistency because boundary regions occupy a larger proportion of their masks.Source: §3.1.2 · p. 5, Table 3 · p. 5

3.2.1 Evaluation Protocols

• Evaluation offers fully unsupervised Hungarian category matching, semi-supervised fine-tuning with about 1% labeled training images, and distance matching via pixel embeddings and k-NN.The semi-supervised protocol avoids matching generated and ground-truth categories.Source: §3.2.1 · p. 6
• The default image-level matching scheme has failed cases when generated categories do not correspond to ground-truth categories.The authors identify this as a limitation of the baseline matching method and expect more effective matching methods.Source: §3.2.1 · p. 6

3.2.2 Evaluation metrics

• Evaluation uses mIoU and boundary mIoU as comprehensive measures, with image-level accuracy and Fβ describing category and shape aspects.Images are evaluated at original resolution.Source: §3.2.2 · p. 6

4.1 Overview

• PASS learns representations, generates pixel-level pseudo-labels through pixel-attention clustering, and fine-tunes the model with those labels before inference.The paper presents this four-step pipeline as one option for LUSS.Source: §4.1 · p. 7

4.2 Unsupervised Representation Learning

• Non-contrastive alignment matches corresponding pixels across views without repelling other positions, aiming to strengthen shape features while preserving category representations.A projection layer limits interference with category representations, and the asymmetric loss aligns overlapped pixel embeddings.Source: §4.2 · pp. 7–8, Figure 8 · p. 7
• Deep-to-shallow supervision uses last-stage embeddings from one view to supervise all stages of another, strengthening low- and mid-level features with high-level guidance.The paper motivates this strategy because direct use of low- and mid-level features leads to sub-optimal representation learning.Source: §4.2 · p. 8, Figure 9 · p. 8

4.3 Pixel-label Generation with Pixel-Attention

• Pixel-attention filters less meaningful regions from pooled features, reducing clustering noise and supporting more accurate object shapes during pixel-label generation.Most channel attention maps highlight semantic regions, though a few highlight background regions.Source: §4.3 · pp. 8–9, Figure 10 · p. 9

4.4 Fine-tuning and Inference

• The segmentation head is fine-tuned with cross-entropy loss against generated pseudo-labels, and inference assigns a label to each pixel.The model is initialized with representation-learning weights before fine-tuning.Source: §4.4 · p. 9

5.1 Implementation Details

• ImageNet-S50 uses ResNet-18, while ImageNet-S300 and ImageNet-S use ResNet-50; all train with batch size 256 for 200 or 100 epochs, respectively.The 200-epoch schedule applies to ImageNet-S50; the 100-epoch schedule applies to ImageNet-S300 and ImageNet-S.Source: §5.1 · p. 9
• SwAV-based representation learning uses LARS with cosine-decayed learning rates; ImageNet-S50 uses multi-crop training, while ImageNet-S300 and ImageNet-S use two 224×224 crops.The SwAV setup also uses queues and freezes clustering prototypes early in training.Source: §5.1 · p. 9
• Pixel-attention is tuned for 20 epochs with the representation model fixed, then the segmentation model is fine-tuned for 20 epochs with cross-entropy supervision.Clustering loss is the default for pixel-attention tuning; segmentation fine-tuning removes representation-learning losses.Source: §5.1 · p. 9

5.2 Comparison with USS Methods

• PASS shows LUSS is achievable on full-scale ImageNet-S; on ImageNet-S50, it outperforms existing USS methods in mIoU, though comparisons are not strictly fair.Saliency-enhanced PASS exceeds MaskContrast in Fβ and mIoU; MaskContrast benefits from saliency maps, and some compared methods use supervised pretraining or other supervision.Source: §5.2 · pp. 9–10, Table 5 · p. 10
• Small objects have worse mask and boundary mIoU than large objects, with smaller size-related gaps for boundary mIoU.The paper attributes the smaller boundary-mIoU gaps to that metric being more robust to object-size changes.Source: §5.2 · p. 10
• Performance scores drop as training data scale grows, and PixelPro performs better on ImageNet-S50 while SwAV performs better on ImageNet-S300 and ImageNet-S.The authors conclude that different data scales favor different representation-learning strategies.Source: §5.2 · pp. 10–11
• Models trained on larger sets perform worse when evaluated on ImageNet-S50 or ImageNet-S300 than models trained on the corresponding smaller sets, though the gaps are described as relatively small.On ImageNet-S50, ImageNet-S50 training gives the best scores across metrics, while ImageNet-S919 training gives the worst.Source: §5.2 · p. 11, Table 6 · p. 11

5.3.1 Representation Learning

• On ImageNet-S300, P2P alignment and D2S supervision improve SwAV by 2.6% and PixelPro by 7.6% in test mIoU.P2P alignment contributes gains of 2.2% over SwAV and 0.5% over clustering-loss-enhanced PixelPro; D2S adds 0.4% and 1.4% over the respective baselines.Source: §5.3.1 · p. 11, Table 7 · p. 11
• P2P alignment raises Fβ across clustering, contrastive, and non-contrastive variants; non-negative P2P also improves mIoU and Img-Acc over the baseline.Adding projection Mp(˜z) improves Img-Acc by reducing interference from pixel-level representations with category representations.Source: §5.3.1 · p. 11, Table 8 · p. 12
• Deep-to-shallow supervision outperforms same-stage supervision in mIoU and Img-Acc, and cross-view supervision is slightly better than same-view supervision.The authors report that same-view supervision has lower training loss but overfits to a sub-optimal solution; optimal supervised stages differ by method.Source: §5.3.1 · pp. 11–12, Table 7 · p. 11, Table 8 · p. 12
• Image-level methods outperform pixel-level methods on LUSS mIoU, image-level accuracy, and Fβ, while clustering methods trade higher image-level accuracy for worse Fβ than contrastive and non-contrastive methods.Image-level supervision also beats most unsupervised methods in mIoU and image-level accuracy but performs worse than most in Fβ; category representation alone does not learn shape features.Source: §5.3.1 · pp. 12–13, Table 9 · p. 12

5.3.2 Label generation and Fine-tuning

• Pixel-attention label generation outperforms image-level and pixel-level alternatives, while image-level generation has better Fβ until the proposed inference strategy is applied.That inference strategy significantly improves Fβ with negligible mIoU change.Source: §5.3.2 · p. 13, Table 10 · p. 13
• On the full ImageNet-S dataset, pixel-level clustering takes about 114 hours, whereas pixel-attention label generation clusters image-level embeddings in the same time as the image-level method.The authors characterize the pixel-level method's full-dataset runtime as unacceptable for real usage.Source: §5.3.2 · p. 13, Table 10 · p. 13
• Unshared attention maps perform better than a shared map; most channels focus on semantic regions, while a few highlight background.The channel maps do not share identical focus.Source: §5.3.2 · p. 13, Table 10 · p. 13
• Fine-tuning after pixel-attention label generation improves test mIoU by 3.3%, indicating that generated pixel-level labels remain noisy.The authors report that fine-tuning further improves semantic segmentation quality.Source: §5.3.2 · p. 13, Table 10 · p. 13

Instance segmentation and object detection

• On ImageNet-S300 pre-training, the proposed method improves SwAV and PixelPro by 1.4% and 1.0% in instance-segmentation mAP, and by 1.7% and 1.2% in object-detection mAP.With full ImageNet-S pre-training, the PixelPro-based method gains 0.6% in instance-segmentation mAP and 0.7% in object-detection mAP.Source: §5.4 · pp. 13–14, Table 11 · p. 14

Semantic segmentation

• On PASCAL VOC semantic segmentation, the method beats SwAV and PixelPro by 1.9% and 2.3% in mIoU with ImageNet-S300 pre-training, and by 2.3% and 2.2% with ImageNet-S pre-training.PixelPro outperforms other image-level methods for semantic segmentation; contrast-based methods outperform clustering and non-contrastive methods among image-level methods.Source: §5.4 · p. 14

Relation between LUSS and transfer learning

• LUSS and downstream tasks favor different representations: SwAV performs well on LUSS but is inferior to some downstream methods, while PixelPro has the reverse pattern.On instance segmentation, MoCov2 gains 1.3% in mAP over SwAV despite a 10% mIoU gap on LUSS; both proposed strategies improve LUSS and downstream performance.Source: §5.4 · p. 14
• Within a representation category, most methods that perform well on LUSS also perform better on downstream tasks, and all tasks benefit from high-quality representations.The section concludes that LUSS and downstream tasks require different representations.Source: §5.4 · p. 14

Pre-trained models

• Replacing supervised ImageNet1k pre-training in SEAM with supervised ImageNet50 lowers test mIoU from 44.5% to 35.8%; MoCo and SwAV pre-training lower it further to 19.1% and 22.3%.The authors report the same reliance on supervised pre-training for SC-CAM and AdvCAM.Source: §5.5 · pp. 14–15, Table 12 · p. 15

Image-level GT labels

• AdvCAM with ground-truth labels improves SEAM test mIoU by 1.7% and 1.8% with ImageNet1k and ImageNet50 pre-training, but gains only 0.4% with generated labels and MoCo.With SwAV pre-training, AdvCAM improves performance by 1.0%; the authors conclude WSSS methods cannot be directly transferred to LUSS without image-level ground-truth labels.Source: §5.5 · p. 15, Table 12 · p. 15

Network architectures

• Replacing ResNet-38 with ResNet-18 in SEAM reduces test mIoU from 49.6% to 44.5%, while the authors say large models make unsupervised LUSS training impracticable at ImageNet-S scale.ImageNet-S datasets are reported as 44 to 800 times larger than PASCAL VOC.Source: §5.5 · p. 15, Table 12 · p. 15

Large-scale semi-supervised semantic segmentation

• With 1% labeled ImageNet-S training images, the proposed method beats SwAV and PixelPro by considerable margins; its PixelPro-based method also surpasses image-level supervised models on ImageNet-S300.In this semi-supervised setting, PixelPro performs similarly to SwAV despite SwAV's large advantage in distance-matching evaluation; the authors conclude pixel-level ground-truth fine-tuning reduces the need for self-learned category representations.Source: §5.6 · p. 16

Evaluate supervised backbone models

• Image-level top-1 accuracy is not always consistent with supervised backbones' ImageNet-S mIoU, indicating category representation need not track shape representation.RF-ConvNeXt achieves high semantic-segmentation performance in the reported evaluation.Source: §5.6 · p. 16, Table 14 · p. 16

Salient object detection with ImageNet-S subset

• ImageNet-Sal selects images with salient objects from ImageNet-S, using manual selection for labeled splits and pre-trained SOD models to help select unlabeled training images.Because selected unlabeled images might lack salient objects, the authors encourage SOD methods to identify such training images themselves.Source: §5.6 · p. 16

• The paper defines LUSS and introduces a high-diversity benchmark with a clear task objective and sufficient evaluation.Source: §6 · p. 16
• Its method learns category and shape representations without human-annotation supervision through enhanced representation learning and pixel-attention-assisted label generation.Source: §6 · p. 16
• Evaluations across multiple protocols reveal LUSS’s potential for pixel-level downstream tasks, while benchmarks of related methods help identify challenges and possible directions.Source: §6 · p. 16

7.1 Evaluation metrics

• mIoU evaluates segmentation-mask quality from pixel labels and includes the ‘other’ category in its category average.Source: §7.1 · pp. 16–17
• Img-Acc evaluates category representation, counting a prediction as correct when its largest-area category belongs to the image’s ground-truth label set.Source: §7.1 · p. 17
• Fβ evaluates category-agnostic shape quality by treating major-category pixels as foreground and ‘other’ pixels as background; β² is set to 0.3.Source: §7.1 · p. 17

7.2 CAM-based Inference

• CAM-based inference normalizes the predicted-category activation map, then assigns labels to regions according to activation values and a threshold.Source: §7.2 · p. 17

7.3 Performance Without ‘other’ Category

• mIoU differences with and without ‘other’ are mostly less than 0.1% on ImageNet-S300/919, while ImageNet-S50 has about a 1% gap.Source: §7.3 · p. 17