Source-linked AI summary
Localizing Objects with Self-Supervised Transformers and no Labels
Oriane Siméoni, Gilles Puy, Huy V. Vo, Simon Roburin, Spyros Gidaris, Andrei Bursuc, Patrick Pérez, Renaud Marlet, Jean Ponce
TL;DR
The paper addresses object localization without annotations, aiming to reduce reliance on costly labeled data. LOST uses self-supervised vision-transformer patch features on each image to discover objects and generate pseudo-boxes for detector training. It outperforms prior object-discovery methods and further improves performance with a class-agnostic detector, while remaining limited on overlapping instances and images dominated by one object.
Problem
Object localization and discovery without annotations could reduce dependence on costly labeled data, but existing approaches use limited-precision proposals or expensive image-collection exploration.
Method
LOST selects and expands correlated image patches using self-supervised transformer features, then uses the resulting pseudo-boxes to train unsupervised object detectors.
Results
LOST outperforms state-of-the-art object-discovery methods by large margins, while class-agnostic detector training further improves object-discovery performance by 4 to 7 CorLoc points.
Takeaways & Limitations
LOST boxes can serve as pseudo ground truth for class-agnostic detection and support competitive unsupervised detection results for several classes.
Takeaways & Limitations
LOST is not designed to separate overlapping instances, and objects covering most of an image can violate its seed-selection assumption.
Abstract
from arXiv · showhide
Localizing objects in image collections without supervision can help to avoid expensive annotation campaigns. We propose a simple approach to this problem, that leverages the activation features of a vision transformer pre-trained in a self-supervised manner. Our method, LOST, does not require any external object proposal nor any exploration of the image collection; it operates on a single image. Yet, we outperform state-of-the-art object discovery methods by up to 8 CorLoc points on PASCAL VOC 2012. We also show that training a class-agnostic detector on the discovered objects boosts results by another 7 points. Moreover, we show promising results on the unsupervised object discovery task. The code to reproduce our results can be found at https://github.com/valeoai/LOST.
1 Introduction
The paper targets object localization without annotations, motivated by the cost of labeled data and limitations of existing proposal- and collection-based approaches. LOST instead uses self-supervised transformer features and patch correlations within a single image.
- Object detectors require vast amounts of costly annotated data to achieve high performance.
- Unsupervised localization is harder than proposal generation because existing proposals have low precision, while collection-based discovery can require quadratic inter-image region comparisons.
- LOST localizes objects from a single image using self-supervised vision-transformer features and patch similarities, without external proposals or inter-image exploration.
- The method selects a low-correlation seed, expands it with correlated patches, and applies the resulting localization to unsupervised object discovery.
- The authors report a single-image localization method with complexity linear in dataset size, plus class-agnostic and class-aware unsupervised detectors.
2 Related work
Prior work reduces annotation needs through proposals, weak supervision, self-supervision, or other modalities, while object discovery often relies on similarities across image collections. LOST instead extracts candidates and pseudo-labels from self-supervised features without annotations or inter-image similarity.
- Region proposal methods generate many class-agnostic boxes with high recall but low precision, primarily to accelerate later detection searches.
- Weakly-supervised object detection uses image-level labels without bounding boxes, while other approaches obtain automatic supervision from modalities such as LiDAR or audio.
- Object discovery groups images depicting similar objects and localizes those objects, with recent methods emphasizing localization across image collections.
- Collection-based discovery can require expensive inter-image computation, whereas LOST extracts object candidates and pseudo-labels from self-supervised activations without annotations or other modalities.
3 Proposed approach
LOST builds object localizations from patch features extracted by a self-supervised vision transformer. It selects and expands image patches using correlation structure, converts the result into a connected-component box, and uses these pseudo-boxes to train detectors.
- Transformer features: Patch features are embedded from fixed-size image patches, with a learned CLS token appended to the transformer input.
- Transformer features: The method uses transformer patch keys from the last self-attention layer to compute similarities between patches in a single image.
- Input features: LOST takes d-dimensional features from N patches of one image and assumes at least one object is present for localization.
- Initial seed selection: The initial seed is selected as the patch with the fewest positive correlations, based on the assumption that foreground patches correlate differently from background patches.
- Initial seed selection: The patch similarity graph connects patches whose features are positively correlated, and the initial seed is the graph node with the lowest degree.
- Seed expansion: Seed expansion adds patches highly correlated with the initial seed, while selecting candidate patches among those with the lowest degrees.
- Box extraction: LOST compares all image features with the seed set, retains the connected component containing the initial seed, and uses its bounding box as the detected object.
- Unsupervised detection: For class-agnostic detection, all LOST pseudo-boxes receive the same foreground category before training an off-the-shelf detector.
4 Experiments
The experiments evaluate LOST for single-object discovery and unsupervised detection across VOC07, VOC12, and COCO 20k, using transformer and alternative backbone features. LOST substantially improves localization and detection, while its limitations include overlapping-instance separation and failures when an object dominates the image.
- Experimental setup: The experiments cover single-object discovery, class-agnostic detection, and class-aware detection on VOC07, VOC12, and COCO 20k.The evaluation includes transformer, convolutional, and supervised-backbone comparisons.
- Single-object discovery: LOST improves over DINO-seg by 8 to 17 CorLoc points for single-object localization.DINO-seg uses connected components from CLS-token self-attention, whereas LOST uses transformer patch features.
- Single-object discovery: Training a class-agnostic detector on LOST outputs further improves performance by 4 to 7 CorLoc points and surpasses prior state of the art by at least 10 points on each evaluated dataset.The detector returns the highest-scoring box for each image during single-object discovery evaluation.
- Unsupervised object detection: LOST’s unsupervised detector exceeds 50.0% AP@0.5 for aeroplane, bus, dog, horse, and train, reaching 72.2% for cat.Some classes also exceed the AP of weakly-supervised WSDDN and PCL, though backbone differences limit direct comparability.
- Unsupervised object detection: Training on pseudo-boxes yields significantly higher AP than the initial pseudo-boxes, while replacing LOST pseudo-boxes with rOSD boxes degrades detector performance.Class-aware pseudo-labels come from K-means clustering of DINO-ViT-S tokens, with Hungarian matching used only for evaluation.
- Limitations and future work: LOST cannot separate overlapping same-class instances, and objects covering most of the image can cause background seed selection.The latter issue violates the assumption that an individual object covers less area than the background.
5 Conclusion
LOST localizes objects without labels using self-supervised transformer features and outperforms state-of-the-art object discovery methods. Its boxes can support class-agnostic and unsupervised object detectors, while object proposals and segmentation remain future applications.
- LOST localizes objects without any labels by leveraging features from a self-supervised pre-trained transformer.
- High-precision LOST boxes can serve as pseudo ground truth for training a class-agnostic detector that further improves object discovery.
- LOST boxes also support an unsupervised object detector with competitive results against weakly-supervised counterparts for several classes.
- Future work includes using LOST boxes as high-quality region proposals and studying self-supervised transformer features for unsupervised object segmentation.
A Ablation Study
The ablation study tests transformer feature choices and the seed-expansion parameter. Keys provide the strongest localization, while expansion is necessary to grow detections beyond the initially selected object part.
- Transformer keys used as patch features outperform alternatives using queries or values in LOST localization.The tested alternatives also include query-key scalar-product similarities and modified seed or box-extraction similarity matrices.
- Seed expansion enlarges the region of interest to include object parts missed by the initial seed.
- Without seed expansion, k = 1 causes a drastic localization-performance drop; performance improves around k = 100-150 before slightly decreasing at k = 200.
- With k = 1, boxes tend to cover only the most discriminative object part, whereas larger k values produce larger boxes that better localize the object.
A.3 Analysis of DINO-seg
The DINO-seg analysis compares attention heads and heuristics for selecting one box per image. Head 4 is strongest among individual heads, while heuristic selection remains substantially behind LOST.
- Among the six attention heads, head 4 captures objects well, while the other heads achieve much lower localization results.
- DINO-seg HAIoU improves over DINO-seg head 4 by 1 point on both VOC07 and VOC12 but remains significantly worse than LOST.DINO-seg BCC performs worse than the head-4 baseline.
A.4 Impact of the number of clusters on class-aware detection training
The study evaluates class-aware unsupervised detection when the requested K-means cluster count differs from VOC’s 20 classes. Performance remains good across counts and improves noticeably with 30 or 40 clusters.
- The main experiments use 20 K-means clusters because VOC contains 20 object classes, while this analysis also tests rough class-count estimates.
- With more clusters than VOC classes, Hungarian matching maps only the 20 best-fitting clusters to VOC classes and ignores detections from unmatched clusters.
- The unsupervised detector achieves good results across all evaluated cluster counts, with noticeable improvement at 30 and 40 clusters.
- Across four random K-means seeds, AP@0.5 % has a standard deviation of 0.8, indicating fairly low sensitivity to clustering randomness.
B.1 Results on more datasets used in previous work
The paper evaluates its method on datasets used by earlier object-discovery studies, including VOC07 noh and VOC12 noh, and reports multi-region results using odAP.
- Datasets: VOC07 noh and VOC12 noh are subsets of PASCAL VOC 2007 and 2012 trainval sets containing 3,550 and 7,838 images, respectively.The subsets exclude images containing only objects annotated as hard or truncated.
- Evaluation: The evaluated datasets are drawn from benchmarks used in previous object-discovery work.
- Evaluation: Table 9 reports multi-object discovery performance using odAP for the proposed method and baseline approaches.The comparison concerns settings where multiple regions are returned per image.
B.2 Multi-object discovery results
The multi-object discovery evaluation uses LOST boxes to train a class-agnostic detector and compares its image-neighbor retrieval and detection performance with prior methods and feature adaptations.
- Evaluation metric: odAP averages object AP across predicted-region counts from one to the dataset’s maximum number of ground-truth objects per image.The metric does not depend on the number of detections per image.
- Multi-object discovery: LOST + CAD significantly outperforms previous methods, including LOD + CAD, in multi-object discovery.LOST + CAD is a class-agnostic detector trained with LOST boxes.
- Image neighbor retrieval: LOST box descriptors are used to retrieve similar images by comparing CLS descriptors with cosine similarity and selecting the top τ = 10 neighbors.CorRet measures the average percentage of retrieved neighbors that are actual same-category neighbors.
- Image neighbor retrieval: LOST boxes represented by DINO features achieve a better CorRet score than rOSD and LOD, while VGG16 features remain behind LOD but ahead of rOSD.
- Feature adaptation: Adapting LOD to transformer features requires replacing its proposal generation and feature-extraction choices because its original CNN-based proposal algorithm performs poorly with transformer features.The paper uses EdgeBoxes and DINO features extracted with ROIPool for this adaptation.
- Feature adaptation: The authors note that many adaptation choices remain unexplored, including combinations of CNN and transformer features that might improve results further.
B.5 Using supervised pre-training.
The supervised-pretraining experiment tests LOST with a transformer trained fully supervised on ImageNet and finds substantially worse localization than with DINO self-supervision, while qualitative results illustrate successes and pseudo-class failures.
- Supervised pre-training: 16.9% CorLoc is achieved by LOST with a fully supervised DeiT transformer, significantly below the DINO self-supervised model.
- Supervised pre-training: The authors leave unclear whether the performance difference arises from the self-supervision loss or other model properties.
- Qualitative results: LOST + CAD qualitative examples show multiple objects localized in the same image on VOC07 and COCO20k.
- Qualitative results: LOST + OD qualitative results assign pseudo-classes to detections, but person instances in very different positions receive three different pseudo-classes.The examples illustrate difficulty assigning one class to such person instances.
D Training details of the Faster R-CNN detection models
The detection stage trains Faster R-CNN models from LOST pseudo-boxes, using a DINO-pre-trained ResNet-50 implementation and a specified multi-GPU optimization schedule; figures show CAD and OD predictions.
- Training setup: LOST extracts one pseudo-box per unlabeled image, which is then used as ground truth for training off-the-shelf object detectors.
- Training setup: The Faster R-CNN implementation uses Detectron2’s R50-C4 model with a ResNet-50 backbone pre-trained using DINO self-supervision.
- Qualitative localization: Figures 4 and 5 visualize seed-based localization, distinguishing the seed, the seed-only box, and the box obtained from all seeds with k = 100.
- Qualitative detection: Figures 6 and 7 show LOST + CAD predictions on VOC07 and COCO, while Figures 8 and 9 show LOST + OD predictions with color-coded pseudo-classes.