A Survey of Self-Supervised and Few-Shot Object Detection

1 INTRODUCTION

• Dense box and mask annotation is costly and time-consuming, and fine-grained applications may lack pre-labeled datasets, requiring expert annotators.Compared with image classification, these tasks require denser labeling and more annotator training.Source: §1 · p. 1
• Few-shot object detection learns novel classes from few examples but first trains on many labeled base-class examples.The standard approach initializes from ImageNet classification, trains on base classes, then fine-tunes on novel classes.Source: §1 · p. 1
• Classification-oriented self-supervised representations may discard localization information because some learned invariances, such as translation invariance, are undesirable for detection.Higher ImageNet Top-1 accuracy does not necessarily guarantee better object detection performance.Source: §1 · p. 1
• Self-supervised object detection methods may pretrain components beyond the feature extractor, including the RPN and detection head; DETReg [6] pretrains both backbone and detector.The survey identifies DETReg [6] as the then-current state of the art for FSOD on MS COCO.Source: §1 · p. 1
• The survey combines recent few-shot and self-supervised object detection approaches, organizing them with a taxonomy and reviewing concepts, benchmarks, methods, takeaways, and future trends.The authors state that neither combined topic had previously been surveyed.Source: §1 · p. 1

2 RELATED SURVEYS

• Jiao et al. and Zaidi et al. [125] review traditional object detection, including detectors, backbones, datasets, benchmarks, and metrics.They briefly mention few-shot and unsupervised detection as future trends; the survey notes they do not cover DETR or Deformable DETR.Source: §2 · pp. 1–2
• Figure 1 classifies reviewed methods by backbone and detection-head pretraining, and marks whether they were benchmarked on regular detection, few-shot/low-shot detection, and ImageNet classification.DETReg [6] is identified as self-supervised pretraining of the entire architecture and as state of the art for FSOD on MS COCO.Source: Figure 1 · p. 2
• Jing and Tian survey self-supervised visual feature learning across pretext tasks, backbones, and image and video recognition, while this survey focuses on methods designed for detection.Source: §2 · p. 2
• Few-shot classification surveys by Chen et al. and Wang et al. do not explore few-shot object detection.Source: §2 · p. 2
• Khan et al. [56] survey transformer results across vision tasks; this survey reviews transformers as detection backbones and heads, including DETR variants and visual-transformer properties [13].Source: §2 · p. 2

3.1 Key Concepts

• Object detection jointly localizes and recognizes objects; Mask R-CNN additionally predicts an instance mask, while single-stage detectors trade prediction quality for simpler, faster prediction.Source: §3.1 · p. 3
• The backbone extracts image features, and an FPN combines multiple scales through top-down and lateral pathways to add semantic information to lower-resolution layers.Source: §3.1 · p. 3
• Faster R-CNN uses a backbone and optional FPN to produce features, then an RPN proposes boxes and a box head classifies and refines them; NMS filters proposals and final predictions.The survey uses the RPN and box head together as “detection heads.”Source: §3.1 · p. 3
• DETR uses learned queries and transformer encoder-decoder prediction to produce detections without NMS, while its Hungarian matching loss can be difficult to optimize.Deformable DETR attends to learned locations across feature scales and was trained in 10 times fewer epochs than DETR.Source: §3.1 · p. 3

3.2 Datasets and Evaluation Metrics

• PASCAL VOC and MS COCO are widely used in traditional, few-shot, and self-supervised detection; the survey directs readers to Section 4.3 for benchmark use and mAP differences.Source: §3.2 · pp. 3–4

3.1 Key Concepts

• Figure 2 depicts Faster R-CNN with FPN, while Figure 3 depicts DETR’s query-based transformer detector.Source: Figure 2 · p. 4, Figure 3 · p. 4

4 FEW-SHOT OBJECT DETECTION

• Few-shot object detection learns to detect new object categories using only one or a few training examples per class.Source: §4 · p. 4
• The section describes the FSOD framework and its differences from few-shot classification.Source: §4 · p. 4
• The section also covers common datasets, evaluation metrics, and FSOD methods, alongside a taxonomy of popular few-shot and self-supervised detection methods.Source: §4 · p. 4

4.1 FSOD Framework

• FSOD partitions categories into base classes with many training examples and novel classes with only a few examples per class.The backbone is usually pretrained on an image-classification dataset before the FSOD task.Source: §4.1 · p. 4
• Base training uses annotations for base classes, while few-shot finetuning uses a small support set containing base and novel examples.Some methods condition on the support set instead of finetuning.Source: §4.1 · p. 5
• Few-shot evaluation detects base and novel classes jointly, with performance reported separately for each group.Metrics include bAP and nAP variants, with suffixes such as 50 or 75 indicating IoU thresholds.Source: §4.1 · p. 5
• In self-supervised learning, the term base-training may refer to finetuning or training, rather than meta-learning's base-training phase.The support set is typically a fixed subset of training data, while few-shot evaluation uses the full test set.Source: §4.1 · p. 5
• Conditioning-based methods can skip finetuning, while DETReg [6] replaces base training with self-supervised pretraining and reports lower nAP.The survey notes that most reviewed conditioning-based methods still benefit from some form of finetuning.Source: §4.1 · p. 5

4.2 Important Differences with Few-Shot Classification

• Unlike few-shot classification, FSOD images may contain multiple base and novel objects, and training retains images with base objects while filtering novel annotations.Few-shot classification typically assumes one dominant object per image and filters novel images during base training.Source: §4.2 · p. 5
• FSOD finetuning and evaluation jointly detect base and novel classes, whereas few-shot classifiers are typically evaluated only on novel classes.Many FSOD papers nevertheless report only novel-class average precision.Source: §4.2 · p. 5
• Most FSOD methods train on all base classes and finetune once on a fixed support set, rather than using learning-to-learn episodes and query-set loss.Because pre-finetuning weights are not calibrated across episodes, they may not be optimal; hyperparameter tuning can optimize learning rates but not those weights.Source: §4.2 · pp. 5–6
• Kang’s splits use one prespecified fixed support set, which can cause overfitting and overestimate performance, especially in 1-shot object detection; Wang et al. sample multiple support sets (30 seeds) to lower benchmark variance.For PASCAL VOC Split-1, nAP, nAP50 and nAP75 means and variances become stable only after around 20 episodes.Source: §4.2 · p. 6, Figure 5 · p. 6
• FSOD methods commonly train on the union of training and validation data and use the test set for both tuning and evaluation, which overestimates generalization ability.Source: §4.2 · p. 6

4.3 FSOD Datasets

• The dominant FSOD benchmarks are PASCAL VOC, MS COCO, and LVIS, and the survey compares methods on these benchmarks.The discussion also points to benchmark caveats and future best practices.Source: §4.3 · p. 6

4.3.1 PASCAL VOC [28]

• PASCAL VOC splits 20 classes into 15 base and 5 novel classes, with three common splits and novel mAP50 as the main comparison metric.Few-shot finetuning uses fixed support instances, which may overestimate performance.Source: §4.3.1 · p. 6

4.3.2 MS COCO [71]

• MS COCO FSOD uses 20 novel classes and 60 base classes, with novel mAP as its main comparison metric.Evaluation uses val5k; remaining COCO2014 training and validation images are used for base training and few-shot finetuning.Source: §4.3.2 · p. 6

4.3.3 LVIS [41]

• LVIS v0.5 divides 1,230 classes into 776 base and 454 novel classes, making it the largest FSOD benchmark by category count.Its metrics are reported across frequent, common, and rare categories and in aggregate.Source: §4.3.3 · p. 6

4.3.4 Discussion

• On PASCAL VOC, the 20–200-example support set and fixed Kang splits can overfit, especially in 1-shot evaluation, inflating novel AP.TFA-splits average results over 30 randomly sampled support sets, unlike the single fixed Kang-splits support set.Source: §4.3.4 · p. 6
• MS COCO is described as the most reliable benchmark because it has 800–2,400 finetuning boxes and many evaluation boxes for base and novel classes.The survey nevertheless notes that averaging over 30 seeds is less reliable than the 600-episode practice in few-shot classification.Source: §4.3.4 · p. 7
• The current LVIS v0.5 splits omit most novel classes from validation, let base boxes dominate aggregated metrics, and leave evaluated novel classes with only 3.4 boxes per class on average.The survey does not recommend these splits and calls for more balanced splits and evaluation sets.Source: §4.3.4 · p. 7
• ImageNet pretraining can overlap with FSOD novel classes and potentially inflate performance; the survey encourages sanitizing backbone and FSOD datasets for cross-contamination.All but “person” among PASCAL VOC’s 20 categories appear in some form in ImageNet.Source: §4.3.4 · p. 7

• For each class, detections are ranked by confidence and counted as true positives only when IoU exceeds the chosen threshold and the ground-truth object is not already detected.Precision@k uses true positives among the top k detections; recall@k divides those true positives by the total ground-truth boxes.Source: §4.4 · p. 7
• Average precision is the area under the precision–recall curve, with the exact area computation depending on the benchmark.Interpolated precision makes the curve non-increasing by taking the maximum precision at recall levels at least r.Source: §4.4 · p. 7
• Figure 6 plots precision@k and recall@k against the number of boxes considered, then plots precision and interpolated precision against recall.Recall is non-decreasing with k, while precision can fluctuate.Source: §4.4 · p. 7, Figure 6 · p. 8
• PASCAL VOC 2007 commonly approximates AP with 11 interpolated-precision samples at IoU 0.5, while COCO averages mAP over IoU thresholds from 0.5 to 0.95 in 0.05 increments.COCO also commonly reports scores by object size; mAP50 and mAP75 are exact-area metrics at their respective IoU thresholds.Source: §4.4 · p. 8
• FSOD results are often reported separately for base and novel classes as bAP and nAP, with most works emphasizing nAP.Some works also emphasize bAP to avoid catastrophic forgetting.Source: §4.4 · p. 8

4.5.1 Finetuning-only baselines

• Finetuning-only methods train on base classes, then finetune on a support set containing base and novel classes to address class imbalance and reduce novel-class overfitting.TFA uses cosine classification heads and finetunes the last layers while freezing the backbone and RPN; DETReg [6] combines finetuning with self-supervised detection-head pretraining.Source: §4.5.1 · pp. 8–9

4.5.2 Conditioning-based methods

• Conditioning-based methods use annotated support images to condition detection on the classes to recognize, through prototype-based or modulation-based strategies.RepMet classifies proposals by distance to learned representatives; modulation methods use support-derived class weights to modulate query features.Source: §4.5.2 · pp. 9–10
• Most conditioning-based methods are finetuned in practice, and the survey reports that even Attention-FSOD achieves its best performance after finetuning.Some methods can technically condition directly on support examples without finetuning, but many do not report such results.Source: §4.5.2 · p. 10

4.5.3 Add-on methods

• Add-on methods such as TIP can be combined with existing two-stage FSOD methods and use transformation-consistency losses on support weights and detections.TIP also pools features from one view using proposals from another view and computes a detection loss.Source: §4.5.3 · pp. 10–11

5 SELF-SUPERVISED PRETRAINING

• Self-supervised pretraining generates labels from unlabeled data through pretext tasks, and detection-focused methods may pretrain components beyond the backbone, including detection heads.The survey organizes image-level backbone strategies as contrastive, clustering-based, and self-distillative, and distinguishes these from approaches tailored to detection.Source: §5 · p. 11, §5.1 · p. 11

5.1 Image-level Backbone Pretraining

• Image-level pretraining includes global contrastive learning, pixel-level VADeR for dense prediction, and clustering approaches that use assignments as pseudo-labels.VADeR applies InfoNCE to pixel features rather than pooled decoder output and reports encouraging results across structured prediction tasks; SwAV predicts one view’s cluster assignments from another.Source: §5.1 · pp. 11–12

5.1.3 Knowledge Self-distillation (BYOL)

• DINO and related self-distillative methods compare teacher and student representations, and DINO’s unsupervised ViT attention heads can segment different objects.EsViT adds a two-stage architecture and a part-based loss; the survey suggests combining dense pretext tasks with attention-based models may improve transfer to dense tasks.Source: §5.1.3 · p. 12, Figure 7 · p. 12

5.2.1 Untrained Detection Heads

• Detection heads such as the FPN, RPN, and detector heads are often initialized from scratch because their architecture differs from the pretrained backbone.The same issue applies to the encoder and decoder in DETR-style architectures; Pinheiro et al. propose pretraining the FPN.Source: §5.2.1 · p. 12

5.2.2 Task Mismatch

• Classification-oriented pretraining may discard spatial information or introduce invariances that hinder localization, and ImageNet Top-1 accuracy does not necessarily predict detection performance.MoCo reportedly outperforms BYOL and SwAV on detection despite worse classification performance; mAP cannot disentangle classification from localization errors.Source: §5.2.2 · p. 12

5.3 Object Detection Pretraining

• Predictive DETR methods train detection heads to re-predict automatically generated crops, using random crops in UP-DETR and Selective Search proposals in DETReg.UP-DETR conditions decoder queries on crop features, while DETReg treats top Selective Search proposals as foreground objects.Source: §5.3 · p. 13
• Detection-specific methods learn location-aware representations through crop-, window-, or feature-level matching, while SoCo also pretrains the backbone, FPN, and RoI heads without negative examples.InsLoc uses pasted object crops, ReSim aligns windows across views, and DenseCL matches features by cosine similarity; SoCo uses a teacher–student setup.Source: §5.3 · p. 13

5.4 Comparison of Self-Supervised Object Detection Methods

• For MS COCO, many FSOD methods perform better with multi-scale FPN features, while for PASCAL VOC they perform better with single-scale C4 features.The survey cautions that absolute results in its comparison vary with architectures, hyperparameters, schedules, augmentations, and other implementation details.Source: §5.4 · p. 14

6.1 Finetuning is a strong baseline

• Finetuning on novel classes remains common, but its added runtime and hyperparameter tuning may obstruct real-world deployment.The survey hopes to see more competitive finetuning-free methods.Source: §6.1 · p. 14

6.2 Impact of self-supervision for object detection

• Self-supervised detection pretraining brings limited improvements to traditional detection, while its impact appears more significant in few-shot and low-data settings.The survey identifies DETReg [6] as the source of state-of-the-art FSOD results on MS COCO.Source: §6.2 · p. 14

6.3 Using heuristics to generate weak labels

• Heuristics such as Selective Search generate weak labels, and the survey proposes gradually replacing them with learned alternatives as representations improve.One suggested direction is developing differentiable or learnable versions of traditional heuristics.Source: §6.3 · pp. 14–15

6.4 Rise of transformers

• Visual transformers offer both emergent object-segmentation capabilities as backbones and end-to-end detection heads that can avoid NMS.The survey also notes that Pix2Seq formulates detection as language modeling, potentially connecting vision tasks with vision-to-language tasks.Source: §6.4 · p. 15

6.5 Problems with current evaluation procedures

• Table 3 and Table 4 are suitable for general performance orientation, not direct numeric comparisons, because implementations differ in several respects.The cited sources list variation in architectures, features, augmentations, schedules, and image resizing.Source: §6.5 · p. 15

6.5.1 Differences in implementation details

• Similar modulation-based FSOD methods raise the question of whether performance gains come from new ideas or implementation choices.The survey suggests testing older methods in newer frameworks or building a unifying benchmark.Source: §6.5.1 · p. 15

6.5.2 Issues with data splits

• Mixing PASCAL VOC splits and tuning on test sets can undermine reliable generalization estimates.The survey notes that Kang’s and TFA’s splits can yield very different numbers, and that trainval training with test-set tuning can overfit.Source: §6.5.2 · p. 15

6.5.3 Proposed guidelines

• The survey recommends proper train/validation/test splits, lower-variance benchmarks, standardized implementations, and relating new tasks to established ones.For few-shot benchmarks, it recommends averaging confidence intervals across episodes and sampling the few-shot training set rather than fixing instances.Source: §6.5.3 · p. 15

7.1 Weakly-supervised object detection

• Weakly supervised detectors use image-level labels without bounding boxes, often selecting object proposals through multiple-instance learning; point-level labels are another option but require slightly more human effort.The surveyed approaches include proposal selection and refinement, object-count information, class activation maps, and methods for crowded scenes.Source: §7.1 · p. 16

7.2 Self-supervision using other modalities

• Narrated instructional videos can supervise detection through narration–video correlations, while other surveyed work applies self-supervision to lung CT region prediction and 2D-to-3D reconstruction.The narrated-video approach filters clustered proposal boxes before training a detector.Source: §7.2 · p. 16

7.3 Low-data and semi-supervised object detection

• LSOD and SemiOD limit examples across all classes rather than separating base and novel classes; SemiOD generally also uses unlabeled data, as in Class-Rebalancing and SoftTeacher.The survey notes miniCOCO results for several self-supervised methods and says Class-Rebalancing and SoftTeacher also use unlabeled MS COCO images.Source: §7.3 · p. 16

7.4 Few-shot semantic segmentation

• Few-shot semantic segmentation labels pixels rather than identifying individual objects, and weak-label approaches from segmentation remain a relatively unexplored direction for FSOD.Some segmentation studies use only a few labeled pixels per object or image-level labels at meta-testing.Source: §7.4 · p. 16

7.5 Zero-shot object detection

• Zero-shot detection and instance segmentation target novel objects from non-visual descriptions, with methods often projecting visual features into word-embedding space.The surveyed methods also explore vision-language distillation, foreground/background embeddings for the RPN, separate seen/unseen pathways, and ways to combine semantic and visual information.Source: §7.5 · p. 16

8 CONCLUSION

• The survey formalizes the FSOD framework, reviews benchmarks and evaluation metrics, compares few-shot and self-supervised methods, and outlines takeaways and future research directions.It also gives future best-practice recommendations, highlights trends, and points to related tasks.Source: §8 · p. 16