Source-linked AI summary
FSCE: Few-Shot Object Detection via Contrastive Proposal Encoding
Bo Sun, Banghuai Li, Shengcai Cai, Ye Yuan, Chi Zhang
TL;DR
Few-shot object detection suffers from data scarcity, overfitting, and misclassification of accurately localized novel instances as confusable base classes. FSCE adds supervised contrastive proposal encoding to the RoI head, using CPE loss to compact same-category embeddings and separate different categories. It achieves state-of-the-art results across shots and benchmarks, with gains up to +8.8% on PASCAL VOC and +2.7% on COCO.
Problem
Few-shot object detection requires robust representations from very limited data, and novel instances are often localized but misclassified as confusable base classes.
Method
FSCE augments the RoI head with a supervised contrastive branch and CPE loss that pulls same-category proposal embeddings together while pushing different-category embeddings apart.
Results
FSCE achieves state-of-the-art performance across shots and benchmarks, with gains up to +8.8% on PASCAL VOC and +2.7% on COCO.
Takeaways & Limitations
Contrastive proposal encoding is generally applicable to two-stage detectors and can improve few-shot detection without interfering with the training pipeline.
Abstract
from arXiv · showhide
Emerging interests have been brought to recognize previously unseen objects given very few training examples, known as few-shot object detection (FSOD). Recent researches demonstrate that good feature embedding is the key to reach favorable few-shot learning performance. We observe object proposals with different Intersection-of-Union (IoU) scores are analogous to the intra-image augmentation used in contrastive approaches. And we exploit this analogy and incorporate supervised contrastive learning to achieve more robust objects representations in FSOD. We present Few-Shot object detection via Contrastive proposals Encoding (FSCE), a simple yet effective approach to learning contrastive-aware object proposal encodings that facilitate the classification of detected objects. We notice the degradation of average precision (AP) for rare objects mainly comes from misclassifying novel instances as confusable classes. And we ease the misclassification issues by promoting instance level intra-class compactness and inter-class variance via our contrastive proposal encoding loss (CPE loss). Our design outperforms current state-of-the-art works in any shot and all data splits, with up to +8.8% on standard benchmark PASCAL VOC and +2.7% on challenging COCO benchmark. Code is available at: https: //github.com/MegviiDetection/FSCE
1. Introduction
Few-shot object detection is limited by data scarcity and overfitting, with novel instances often localized but misclassified as confusable base classes. FSCE addresses this using supervised contrastive proposal encoding to compact same-category representations and separate different categories, achieving state-of-the-art results across shots and benchmarks.
- Deep detectors require substantial annotated data, making few-shot detection especially vulnerable to overfitting.
- CPE loss reduces same-category embedding variance and increases separation between different-category instances.The design encourages higher similarity for positive proposals from the same category or object than for negative proposals from different categories.
- Novel instances are often localized accurately but misclassified as confusable base classes because classification is the main error source.RPN can generate good foreground proposals for novel instances, while prototype similarities between resembling categories can be high.
- FSCE adds a contrastive branch to the RoI head and jointly optimizes its CPE loss with classification and localization objectives.The branch measures similarity between object proposal encodings and uses proposals with different IoU as contrastive examples.
- FSCE achieves state-of-the-art performance across 1, 2, 3, 5, 10, and 30 shots, with gains up to +8.8% on PASCAL VOC and +2.7% on COCO.
2. Related Work
Related work spans meta-learning, fine-tuning, and contrastive representation learning for limited-data recognition. FSCE builds on these directions while targeting confusable-category misclassification with supervised contrastive learning in few-shot object detection.
- Few-shot learning seeks to recognize new concepts from limited labeled examples, using approaches such as meta-learning, metric learning, and data hallucination.
- Few-shot object detection is more challenging than few-shot classification because it requires both object classification and localization.
- Meta-learning methods transfer knowledge from base classes through episodic or specialized prediction mechanisms, but complex algorithms may overfit and test poorly.
- Fine-tuning methods such as TFA improve few-shot detection, while later work addresses issues including scale bias and positive proposal refinement.
- FSCE applies supervised contrastive learning to few-shot object detection to learn discriminative proposal representations without complexing the model.
3. Method
FSCE adapts two-stage fine-tuning for few-shot detection by restoring useful proposal-learning components and adding contrastive object proposal encoding. Its CPE objective shapes RoI embeddings around proposal consistency and category structure.
- Training strategy: FSCE trains on base classes first, then fine-tunes on balanced novel-plus-base data while freezing the backbone and optimizing detection with CPE loss.The fine-tuning stage uses novel instances and randomly sampled base instances; the RoI feature extractor receives contrastive supervision.
- Training strategy: Jointly fine-tuning FPN and RPN coordinates feature maps for novel objects while avoiding the overfitting risk of broadly adapting the backbone.The method contrasts with TFA’s strategy of freezing most components, including feature structures transferred from base classes.
- Training strategy: The strong baseline rescues novel-object proposals by doubling post-NMS proposal capacity and halving the RoI-head proposal batch size.Novel fine-tuning produces only 1/4 as many positive-anchor proposals as base training, while many discarded RoI samples are backgrounds.
- Contrastive proposal encoding: FSCE adds a contrastive branch parallel to classification and regression, using a 1-layer MLP to encode RoI features for proposal similarity.The branch addresses the fact that post-ReLU RoI features cannot be directly compared and guides contrastive-aware proposal embeddings.
- Contrastive Proposal Encoding loss: CPE loss increases similarity among same-label proposals and separates different-label proposals after feature normalization, forming tighter category clusters.Proposal consistency uses IoU: low-IoU regions may contain irrelevant semantics, while φ=0.7 is selected as a cutoff and g(·) weights proposals by IoU.
- Evaluation: The method evaluates proposal-encoding improvements on PASCAL VOC using nAP50 across three novel splits and compares existing few-shot detection methods under marked protocols.Table 2 distinguishes metalearning methods, averages over ten random seeds, and N-way K-shot meta-testing from the primary evaluation protocol.
4. Experiments
Experiments evaluate FSCE on PASCAL VOC and COCO across few-shot settings, ablate its components and proposal-consistency controls, and inspect its learned representations. FSCE achieves state-of-the-art results while contrastive encodings reduce intra-class variance and improve decision boundaries.
- Experimental setup: FSCE forms an upper envelope for fine-tuning-based methods and memory-inefficient meta-learning approaches across shots and data splits.Experiments follow a consistent few-shot detection construction and evaluation protocol for fair comparison.
- PASCAL VOC results: PASCAL VOC results show FSCE outperforming existing works in every shot and split, with up to +8.8 nAP50 above current SOTA on split 3.FSCE is the first to exceed 50 nAP50 on splits 2 and 3.
- COCO results: COCO results show FSCE setting a new state-of-the-art for all shots, gaining +1.7 nAP and +2.7 nAP75 above current SOTA.The comparison uses the same testing protocol and metrics.
- Strong baseline and components: The modified fine-tuning specification transfers the class-agnostic RPN and RoI head while retaining more low-quality foreground proposals for learning.The design also jointly fine-tunes the FPN top-down convolution and RoI feature extractor.
- Proposal consistency control: Proposal-consistency ablations favor a high-IoU cutoff for 5- and 10-shot settings, but retain and down-weight low-IoU proposals when data sparsity is severe.Linear weighting performs better than exponential decay in low-shot cases.
- Visual inspection and ablation: t-SNE analysis indicates that CPE loss reduces intra-class variance, creates more defined decision boundaries, and rescues missed, low-confidence, and misclassified novel detections.The visualization uses proposal embeddings from randomly selected PASCAL VOC images.
5. Conclusion
FSCE addresses few-shot object detection through contrastive proposal encoding and achieves state-of-the-art results across shots and benchmarks. Its contrastive head has negligible cost and is applicable to two-stage detectors without interfering with the training pipeline.
- FSCE proposes contrastive proposal encoding as a new perspective for solving few-shot object detection.
- FSCE achieves state-of-the-art results in any shot and on both benchmarks, with up to +8.8% on PASCAL VOC and +2.7% on COCO.
- The contrastive proposal encoding head has negligible cost and can be added to any two-stage detector without interfering with the training pipeline.
- FSCE provides a strong baseline comparable to contemporary state-of-the-art methods for future few-shot object detection research.