Localizing Objects with Self-Supervised Transformers and no Labels

• Object detectors require costly annotated data for high performance, motivating localization of objects without any annotation.Source: §1 · p. 1
• The method uses DINO [13] attention-key correlations between image patches to select a least-correlated seed, expand it with correlated patches, and form a bounding box around a connected mask component.The seed criterion is motivated by the empirical observation that foreground patches are less correlated than background patches.Source: §1 · p. 2
• By operating on one image at a time rather than comparing regions across images, localization has linear complexity with respect to dataset size.Prior collection-level methods use inter-image similarities and face quadratic region-comparison complexity.Source: §1 · p. 2
• Localized boxes train class-agnostic detectors to detect multiple objects, while clustering localized objects enables class-aware detectors trained with predicted locations and cluster IDs.The paper reports that predictions for some clusters correlate well with labelled semantic classes and reach results competitive with weakly supervised detectors [9].Source: §1 · p. 2
• The approach outperforms the state of the art in unsupervised object discovery by a significant margin.Source: §1 · p. 2

Object detection with limited supervision

• Prior localization spans class-agnostic proposals, supervised-network cues, and weak or semi-supervised detection using image-level labels and sometimes box annotations or self-training [9] [23] [37].Some recent approaches instead use automatic supervision from other modalities, such as LiDAR or audio [1]; LOST uses neither annotations nor other modalities, deriving candidates from self-supervised activations, computing pseudo-labels, and training a detector.Source: §2 · pp. 2–3

Object discovery

• Collection-based discovery often selects recurring patterns at high computational cost, with only scaling to large datasets; LOST instead localizes from intra-image similarity in a single image [13].Earlier approaches often emphasize grouping, while others prioritize localization [15].Source: §2 · p. 3
• Earlier methods emphasize grouping or localization [15]; also localizes a single image using scale-invariant features, while mask-learning methods [10] [20] [30] are evaluated only on simple geometric collections.LOST does not use inter-image similarity.Source: §2 · p. 3

Transformers

• Despite nascent vision-transformer research, applications span classification, retrieval, detection, and segmentation [18] [19] [11], and lower texture bias than CNNs motivates localization from their features.Prior work also studies transformer representations from multiple perspectives [13] [8].Source: §2 · p. 3

Self-supervised learning

• SSL learns annotation-free representations through data-derived pretext tasks [26] and can outperform ImageNet pretraining, especially for detection [24] [32] [12] [31] [25]; DINO activations encode object segmentation [13].LOST extracts another kind of transformer feature for object localization.Source: §2 · p. 3

3.1 Transformers for Vision

• LOST uses concatenated patch keys from all heads of the last self-attention layer of a self-supervised DINO transformer [13], rather than relying on CLS-query attention for localization.The paper describes CLS-query self-attention segmentation as the DINO-seg baseline and says it does not fully exploit the transformer features.Source: §3.1 · p. 4

3.2 Finding objects with LOST

• LOST selects an initial seed as the patch with the fewest positive correlations, relying on features where object patches correlate together and negatively with background patches.The method builds a graph whose edges represent positive correlations between patch features and selects the patch with the lowest degree.Source: §3.2 · p. 5
• LOST expands the seed to positively correlated patches among the k lowest-degree patches, then boxes the connected component containing the seed after comparing seed features with all image features.The mask includes patches that positively correlate on average with the seed features; k is typically 100.Source: §3.2 · pp. 5–6

3.3 Towards unsupervised object detection

• Training on LOST pseudo-boxes with one shared foreground label yields a class-agnostic detector that can detect multiple objects per image and, experiments show, localizes better than LOST.The training images have one pseudo-box annotation each, although the detector can output multiple detections.Source: §3.3 · p. 6
• For class-aware detection, LOST crops are encoded with the DINO transformer’s CLS token, clustered by K-means into pseudo-labels, then matched to ground-truth classes with the Hungarian algorithm at evaluation.This provides class labels for LOST’s pseudo-boxes without human supervision.Source: §3.3 · p. 6

4.1 Experimental setup

• The experiments evaluate single-object discovery, class-agnostic detection, and class-aware detection using unlabeled images, with task-specific train and evaluation splits across VOC07, VOC12, and COCO 20k.For class-aware detection, training uses VOC07 and/or VOC12 trainval and evaluation uses VOC07 test; the detector is Faster R-CNN with a DINO-pretrained ResNet50 backbone.Source: §4 · p. 6, §4.1 · p. 7, §4.3 · p. 8

4.2 Application to unsupervised object discovery

• Training a class-agnostic detector on LOST boxes adds 4–7 CorLoc points, and LOST surpasses prior state of the art by at least 10 points on each evaluated dataset.LOST also improves over DINO-seg by 8–17 CorLoc points. The single-object-discovery results compare LOST with prior discovery methods, proposals, and DINO-seg.Source: Table 1 · p. 7, §4.2 · p. 8
• ViT-S/16 features give LOST its best results; transformer representations suit the method better than the tested alternatives, while DINO-pretrained ResNet-50 performs significantly worse.The default features are the last-layer ViT-S keys without the CLS token.Source: §4.1 · p. 6, Table 2 · p. 8, §4.2 · p. 8

4.3 Unsupervised object detection

• For unsupervised class-aware detection, LOST exceeds 50.0% AP for aeroplane, bus, dog, horse, and train, and reaches 72.2% for cat.Some classes outperform weakly supervised WSDDN [9] and PCL, though backbone differences limit comparability; Hungarian matching is used only for evaluation. Training on LOST pseudo-boxes yields higher AP than the pseudo-boxes themselves, while replacing them with rOSD boxes degrades performance.Source: Table 3 · p. 8, §4.3 · p. 9
• LOST + CAD achieves significantly better class-agnostic detection performance than rOSD + CAD, LOD + CAD, Selective Search, and EdgeBoxes.These comparisons use AP@0.5; the supplied passages do not give the table’s individual values.Source: Table 4 · p. 9, §4.3 · p. 9

4.4 Limitations and future work

• LOST can separate non-overlapping same-class instances but is not designed to separate overlapping ones; objects covering most of an image can also cause the seed to fall on background.The paper identifies both as limitations and leaves possible remedies, including matching for instance separation and CLS-token attention for filtering large-object failures, as future work.Source: §4.4 · p. 10

• LOST uses self-supervised pre-trained transformer features to localize objects without labels and outperforms state-of-the-art object-discovery methods by large margins.Source: §5 · p. 10
• LOST’s high-precision boxes serve as pseudo ground truth for training a class-agnostic detector that further improves object-discovery performance.Source: §5 · p. 10
• LOST boxes can train an unsupervised object detector that yields competitive results against weakly supervised counterparts for several classes.Source: §5 · p. 10
• Future work will investigate LOST boxes as high-quality region proposals and self-supervised transformer features for unsupervised object segmentation.Source: §5 · p. 10

A.1 Which transformer features to choose?

• Keys kp as patch features provide the best localization; using queries qp or values vp lowers LOST by at least 11 CorLoc points.The tested alternatives using queries and keys also perform worse than using keys as patch features.Source: §A.1 · p. 16
• The study tests query–key scalar-product similarity, which makes the adjacency matrix symmetric, for seed selection and for expansion and box extraction using queries, keys, or values.These alternatives are reported as worse than using keys as patch features.Source: §A.1 · p. 16
• A further alternative changes the definitions of S and m; the tested alternatives using queries and keys yield worse results than using keys as patch features.The supplied passage does not provide the complete alternative definitions.Source: §A.1 · p. 16

A.2 Importance of the seed expansion step

• Seed expansion enlarges the region of interest to include all object parts rather than only the part localized by the initial seed.The expansion is controlled by k.Source: §A.2 · pp. 16–17
• Without seed expansion (k = 1), localization drops drastically; performance improves as k rises to 100–150, then decreases slightly at 200.Here, k is the maximum number of patches used to construct mask m.Source: §A.2 · p. 17
• With k = 1, boxes can cover only a discriminative object part; increasing k enlarges boxes and can localize objects better, but expansion can still miss whole objects or include multiple objects.The visualizations compare k = 1 and k = 100 and show both types of failure.Source: §A.2 · p. 17

A.3 Analysis of DINO-seg

• Head 4 captures objects well, while the other five heads perform much worse, so the main paper reports DINO-seg using head 4.The comparison evaluates localization accuracy for each of the six available heads.Source: §A.3 · p. 17
• +1 point on both VOC07 and VOC12 over DINO-seg [head 4] for HAIoU, but it still performs significantly worse than LOST.HAIoU selects the box with the highest average IoU overlap with the other five heads’ boxes.Source: §A.3 · p. 17
• Selecting the box corresponding to the head with the biggest connected component (DINO-seg BCC) yields worse results than using head 4.The section reports the two best-performing variants of dynamically selecting one box per image across heads.Source: §A.3 · p. 17

A.4 Impact of the number of clusters on class-aware detection training

• The main class-aware detection experiments assume the exact 20 VOC classes, while this cluster-count study tests performance with only a rough estimate of class count.The study varies the requested number of K-means clusters.Source: §A.4 · pp. 17–18
• The detector achieves good results with 20, 25, 30, and 40 clusters, with noticeable improvement at 30 and 40 clusters.The reported comparison is mean AP across the 20 VOC classes; prior clustering work reports similar findings [36] [1].Source: Table 7 · p. 18, §A.4 · p. 18
• The cluster-count comparison evaluates 20, 25, 30, and 40 clusters using mean AP across the VOC classes.Results use LOST pseudo-boxes for models trained on the VOC07 and VOC12 trainval sets and are measured on VOC07 test.Source: §A.4 · p. 18, Table 7 · p. 18
• When clusters exceed VOC’s 20 classes, Hungarian matching maps only the 20 best-fitting clusters to classes, and unmatched-cluster detections are excluded from per-class AP.Source: §A.4 · p. 18

A.5 Impact of the non-determinism of the K-means clustering

• Across four random-seed runs of LOST + OD†, AP@0.5% has a standard deviation of 0.8, suggesting limited sensitivity to K-means randomness.The experiment uses the model trained on the union of VOC07 and VOC12 trainval sets.Source: §A.5 · p. 18
• The randomness experiment repeats the unsupervised LOST + OD† detection experiment four times using different K-means random seeds.It uses the model trained on the union of VOC07 and VOC12 trainval sets.Source: §A.5 · p. 18

B.1 Results on more datasets used in previous work

• VOC07 noh (3,550 images) and VOC12 noh (7,838 images) are subsets of PASCAL VOC 2007/2012 trainval, excluding images whose only objects are hard or truncated and all boxes annotated hard or truncated.These subsets are also named VOC all in the literature.Source: §B.1 · p. 18
• Table 9 reports multi-object discovery performance in odAP for LOST and baselines.The datasets are described as those used in previous object-discovery works; the supplied passages do not give performance values.Source: §B.1 · p. 18, Table 9 · p. 19

B.2 Multi-object discovery results

• The odAP metric averages AP across predicted-region counts from one to the dataset’s maximum ground-truth objects per image, independent of detections per image.The variants used by LOD are odAP50 (IoU threshold 0.5) and odAP@[50-95] (averaged over 10 equally spaced thresholds from 0.5 to 0.95).Source: §B.2 · p. 19
• LOST + CAD significantly outperforms all previous methods, including LOD [69] + CAD, in multi-object discovery.Because LOST returns one region per image, the comparison uses LOST boxes to train a class-agnostic detector (CAD).Source: §B.2 · p. 19

B.3 Image nearest neighbor retrieval

• For retrieval, LOST box CLS descriptors are compared by cosine similarity, and each image’s top 10 neighbors are evaluated with CorRet.CorRet is the average percentage of retrieved neighbors that contain objects of the same category in the ground-truth image graph.Source: §B.3 · p. 19
• With DINO [13] features, LOST boxes yield a better CorRet score than rOSD and LOD; with VGG16, LOST trails LOD but beats rOSD.Table 10 presents image-neighbor retrieval performance in CorRet.Source: §B.3 · p. 19, Table 10 · p. 20

B.4 Using DINO features

• For a fair comparison with transformer-backed LOST, the authors adapt LOD to transformer features because the original rOSD proposal algorithm does not yield good proposals with them.They instead use edgeboxes and represent proposals with DINO [13] features extracted using ROIPool, evaluating on VOC07 trainval, VOC12 trainval, and COCO20k trainval.Source: §B.4 · pp. 19–20
• Adapting CNN-designed methods to transformer features, and LOST to CNN features, is not direct; many design choices remain, including potentially combining CNN and transformer features.Table 11 presents single-object discovery CorLoc for LOD [69] and LOST with different feature types.Source: §B.4 · p. 20, Table 11 · p. 20

B.5 Using supervised pre-training.

• 16.9% CorLoc with DeiT supervised ImageNet pre-training is significantly worse than LOST’s result with DINO self-supervised pre-training.The authors leave open whether the difference stems from self-supervision or DINO’s more aggressive pre-training augmentation.Source: §B.5 · p. 20
• DINO [13] likewise found significantly worse segmentation performance with its fully supervised model than with its DINO model.The paper notes this as a similar observation, not an explanation of LOST’s CorLoc difference.Source: §B.5 · p. 20
• The authors say it is unclear whether the performance gap reflects the self-supervision loss or DINO’s more aggressive pre-training augmentation.Source: §B.5 · p. 20

C More visualizations (single- and multi-object discovery)

• Figures 6–7 show LOST + CAD successfully localizing multiple objects in images from VOC07 trainval and COCO20k trainval.The appendix presents these among additional qualitative results in Figures 4–9.Source: C More visualizations (single- and… · p. 20
• Figures 8–9 show LOST + OD localization predictions with predicted pseudo-classes on VOC07 and COCO, using a different color for each pseudo-class.Source: C More visualizations (single- and… · p. 20
• In Figure 9, person objects receive three different pseudo-classes, illustrating the difficulty of assigning one class to people in very different positions.Source: C More visualizations (single- and… · p. 20

• LOST pseudo-boxes serve as ground truth for training Faster R-CNN detection models in the unsupervised object-detection experiments.Source: D Training details of the Faster R-CNN… · p. 20
• The detector is Detectron2’s R50-C4 model with a DINO-pretrained ResNet-50 backbone [34], trained using the protocol and most hyperparameters of He et al. [32].Source: D Training details of the Faster R-CNN… · p. 21
• Training uses 16-image batches across 8 GPUs, a 100-step warmup to 0.02, learning-rate drops at 18K and 22K steps, and 24K total steps, except 10K for VOC07 trainval pseudo-boxes.SyncBatchNorm finetunes BatchNorm parameters, an extra BatchNorm layer is added for the RoI head after conv5, and conv1 and conv2 are frozen throughout training.Source: D Training details of the Faster R-CNN… · p. 21