Source-linked AI summary
Meta-DETR: Image-Level Few-Shot Detection with Inter-Class Correlation Exploitation
Gongjie Zhang, Zhipeng Luo, Kaiwen Cui, Shijian Lu, Eric P. Xing
TL;DR
Few-shot object detection is limited by inaccurate novel-class proposals and by meta-learning methods that overlook correlations among support classes. Meta-DETR uses proposal-free image-level detection with simultaneous multi-class correlational meta-learning, and it reports large-margin state-of-the-art performance across multiple few-shot setups. Its reported failure cases include clustered novel objects, small objects, and less salient objects, motivating boundary-aware and multi-scale extensions.
Problem
Few-shot detection must generalize from abundant base-class annotations to novel objects with very few examples, but existing methods suffer from inaccurate proposals and overlooked inter-class correlations.
Method
Meta-DETR performs pure image-level detection without proposals and uses correlational meta-learning to aggregate multiple support classes simultaneously.
Results
Meta-DETR achieves state-of-the-art performance across multiple few-shot object detection setups, outperforming prior works by large margins.
Takeaways & Limitations
Image-level prediction lifts novel-class proposal constraints, while inter-class correlation reduces misclassification and enhances generalization among similar or related classes.
Takeaways & Limitations
Typical failures involve heavily clustered novel objects, small objects, and less salient objects; the paper proposes object-boundary mechanisms and multi-scale features as future directions.
Abstract
from arXiv · showhide
Few-shot object detection has been extensively investigated by incorporating meta-learning into region-based detection frameworks. Despite its success, the said paradigm is still constrained by several factors, such as (i) low-quality region proposals for novel classes and (ii) negligence of the inter-class correlation among different classes. Such limitations hinder the generalization of base-class knowledge for the detection of novel-class objects. In this work, we design Meta-DETR, which (i) is the first image-level few-shot detector, and (ii) introduces a novel inter-class correlational meta-learning strategy to capture and leverage the correlation among different classes for robust and accurate few-shot object detection. Meta-DETR works entirely at image level without any region proposals, which circumvents the constraint of inaccurate proposals in prevalent few-shot detection frameworks. In addition, the introduced correlational meta-learning enables Meta-DETR to simultaneously attend to multiple support classes within a single feedforward, which allows to capture the inter-class correlation among different classes, thus significantly reducing the misclassification over similar classes and enhancing knowledge generalization to novel classes. Experiments over multiple few-shot object detection benchmarks show that the proposed Meta-DETR outperforms state-of-the-art methods by large margins. The implementation codes are available at https://github.com/ZhangGongjie/Meta-DETR.
1 INTRODUCTION
Few-shot detection seeks transferable knowledge from base classes to recognize novel objects with minimal supervision, but region proposals and class-by-class meta-learning limit generalization. Meta-DETR addresses these constraints with proposal-free image-level detection and simultaneous modeling of multiple support classes.
- Motivation: Few-shot object detection aims to detect novel objects from only a few training samples by transferring knowledge learned from base classes.The task is especially relevant when annotated training samples are scarce or difficult to obtain.
- Limitations: Existing region-based frameworks are sensitive to low-quality proposals, which are difficult to produce for novel classes under limited supervision.The resulting proposal-quality gap obstructs generalization from base classes to novel classes.
- Limitations: Feature-reweighting approaches process one support class at a time and overlook inter-class correlations, limiting discrimination between visually similar classes.Examples include distinguishing cows from horses and learning from related classes.
- Meta-DETR: Meta-DETR performs pure image-level prediction without region proposals, while its correlational meta-learning attends to multiple support classes simultaneously.This design explicitly uses inter-class commonality and uniqueness to support generalization and reduce confusion among similar classes.
- Results: Meta-DETR consistently outperforms state-of-the-art methods by large margins on novel-object detection across extensive experiments.The paper reports this outcome without additional architectural enhancements.
2 RELATED WORK
Few-shot detection builds on transfer-learning and meta-learning paradigms, while Meta-DETR combines meta-learning with DETR-style image-level detection and explicit inter-class correlation modeling.
- Object Detection: Generic object detection jointly performs object localization and classification, but standard detectors depend on abundant annotated samples and degrade sharply in few-shot settings.DETR variants provide pure image-level, end-to-end detection frameworks within generic object detection.
- Few-Shot Object Detection: Few-shot detection methods include transfer learning, which fine-tunes novel concepts, and meta-learning, which learns class-agnostic predictors across auxiliary tasks.The related-work discussion identifies both paradigms and representative methods.
- Meta-DETR: Meta-DETR differs from prior approaches by achieving image-level detection and leveraging correlations among multiple support classes.The paper presents it as the first integration of meta-learning into DETR frameworks and as pioneering explicit inter-class correlation modeling.
3 PRELIMINARIES
Few-shot detection uses base and novel class datasets with very different supervision levels, creating proposal-quality and meta-learning challenges for novel-class generalization.
- Few-Shot Detection Setup: In K-shot detection, each novel class has exactly K annotated objects, while base classes are represented by abundant annotations.The detector learns from disjoint base and novel class sets and targets their union.
- Region-Based Detection: Faster R-CNN-based detectors rely on region proposals whose quality is clearly lower for novel classes under few-shot supervision.This proposal-quality gap hinders transfer of base-class knowledge to novel classes and persists because it is rooted in the region-based framework.
- Meta-Learning: Feature-reweighting meta-learning processes one support class per feedforward, requiring C repeated runs for C classes and overlooking their inter-class correlation.Visually similar classes are correlated in feature space, making this independence problematic for classification.
4 META-DETR
Meta-DETR is an image-level detector built on Deformable DETR that avoids region proposals and aggregates query features with multiple support classes simultaneously. Its CAM performs feature and encoding matching, while target generation and DETR-style optimization support class-agnostic few-shot detection.
- Architecture: Meta-DETR uses Deformable DETR to make image-level predictions without predicted region proposals.This bypasses the constraint of inaccurate novel-class proposals.
- Architecture: CAM aggregates query features with multiple support classes to capture inter-class correlation for class-agnostic prediction.It first embeds query and support features jointly, obtains support prototypes, and matches query features to prototypes and task encodings.
- Correlational Aggregation Module: Feature matching filters query features unrelated to support classes using attention-based matching between query features and support prototypes.The resulting filtered feature map highlights objects belonging to the given support classes.
- Correlational Aggregation Module: Encoding matching assigns pre-defined task encodings to support classes so predictions differentiate classes in a class-agnostic manner.Feature-matching and encoding-matching results are combined before feed-forward prediction.
- Correlational Aggregation Module: A learnable background prototype and zero-valued background encoding resolve ambiguity when query features match none of the support classes.Background occupies many spatial locations in open-set object detection.
- Training Objective: The detector infers a fixed number of object-query predictions, uses bipartite matching for assignment, and optimizes classification and localization losses across decoder layers.Classification uses sigmoid focal loss, while box regression combines ℓ1 and GIoU losses.
- Target Generation: Meta-DETR dynamically maps sampled support-class labels to task-encoding labels and removes annotations for irrelevant objects during target generation.Targets may contain only no-object entries when the query has no objects from the provided support classes.
- Learning and Inference: Training proceeds from abundant base-class training to few-shot fine-tuning with limited novel-class samples, while support prototypes can be precomputed for efficient inference.In K-shot detection, only K object instances are available for each novel category.
5 EXPERIMENTS
Meta-DETR is evaluated on Pascal VOC and MS COCO under standard few-shot detection setups, with repeated support sampling and conventional optimization settings.
- Datasets: Pascal VOC uses 20 classes, three novel/base splits, 1–10 shots, and mAP@0.5 averaged over 10 sampled support datasets.
- Datasets: MS COCO uses 20 shared classes as novel classes and 60 remaining classes as base classes, with 1–30 shots and standard evaluation metrics.
- Implementation: Experiments use ResNet-101 with Deformable DETR architectures in a single-scale implementation for fair comparison.
- Implementation: Training uses AdamW with learning rate 2×10^-4, weight decay 1×10^-4, batch size 32, and benchmark-specific base-training schedules.
5.3 Comparison with State-of-the-Art Methods
Meta-DETR consistently surpasses existing few-shot detectors on Pascal VOC and MS COCO, including comparisons isolating image-level detection and aggregation design.
- Pascal VOC: +4.6% overall mAP over the second-best method is achieved across Pascal VOC setups for novel classes.Results are averaged over multiple randomly sampled support datasets.
- Pascal VOC: Meta-DETR achieves the best base-class and novel-class performance among compared meta-learning methods on Pascal VOC.It also retains competitive base-class performance while detecting novel classes with limited samples.
- MS COCO: Meta-DETR outperforms existing methods under all MS COCO setups by larger margins than on Pascal VOC.The paper attributes this to avoiding inaccurate novel-class proposals and exploiting correlations among more classes.
- Image-level versus region-level detection: The image-level and aggregation comparisons are organized in Table 4, with R denoting region-level and I denoting image-level detection.
5.4 Ablation Studies
Ablations show that Meta-DETR benefits from image-level design choices, simultaneous support-class aggregation, and early correlational aggregation, especially in lower-shot settings.
- CAM impact: When C≥2, CAM improves lower-shot (≤5) performance, with gains of +4.8% mAP at 1-shot and +5.0% mAP at 2-shot.No clear gain is observed at 10-shot; CAM also benefits the region-based FsDetView detector.
- Correlation and class separation: CAM produces better-separated object classes in feature space and reduces misclassification between similar classes such as motorbike/bike and cow/horse.The analysis uses t-SNE visualizations and confusion matrices.
- Number of support classes: As support classes increase from 1 to 10, lower-shot performance first improves and then drops, while 10-shot performance first saturates and then drops.The paper conjectures that the drop at C≥8 reflects limited capacity to differentiate many support classes simultaneously.
- CAM design choices: Applying CAM’s attention modifications verifies the effectiveness of filtering irrelevant query features, applying learned filters, and modeling the background prototype.Table 7 evaluates these three design modifications.
- Qualitative results: Qualitative 10-shot visualizations show effective novel-object detection with highly constrained training samples, while failure cases include false positives and false negatives.White boxes mark true positives; red solid and dashed boxes mark false positives and false negatives, respectively.
- Aggregation timing: Placing CAM at the beginning of the transformer encoder is preferable to late aggregation and highlights the importance of a deep class-agnostic predictor.
5.5 Qualitative Results
Meta-DETR qualitatively detects novel-class objects effectively in 10-shot sample images, with visualizations restricted to novel classes and predictions above a 0.25 confidence threshold.
- Meta-DETR effectively detects novel objects with very limited training samples in qualitative 10-shot evaluations.Only novel-class detections with confidence scores higher than 0.25 are shown.
5.6 Failure Cases and Future Directions
Meta-DETR’s typical failures occur when novel objects are heavily clustered, for small objects, and for less salient objects. The authors propose boundary-focused mechanisms and multi-scale features as future improvements.
- Failure Cases: Heavily clustered novel-object instances are a typical failure case because supervision is limited and object boundaries are difficult to discriminate.
- Failure Cases: Meta-DETR also struggles with detecting small objects and produces false negatives for less salient objects.
- Future Directions: Future work will investigate mechanisms that highlight object boundaries to address some failure cases.
- Future Directions: Because experiments use only single-scale features, the authors propose dual-scale selection for query and support to improve especially small-object detection.
5.7 Extension to Few-Shot Instance Segmentation
Meta-DETR extends its image-level correlational meta-learning framework to few-shot instance segmentation with a mask head. On MS COCO, it outperforms compared methods in 5-shot and 10-shot settings.
- Extension: Meta-DETR’s generic meta-learning framework extends to instance segmentation with simple modifications.The extension adds a mask head while retaining the framework’s inter-class correlational strategy.
- Method: The added mask head uses transformer decoder outputs and encoded image features to predict a binary mask for each object query.
- Experimental Results: Few-shot instance segmentation is evaluated on MS COCO under 5-shot and 10-shot setups, with 20 novel classes and 60 base classes.Segmentation AP is evaluated using mask IoU.
- Experimental Results: Meta-DETR outperforms compared methods by large margins, and comparison with Meta R-CNN supports combining image-level prediction with inter-class correlation.
6 CONCLUSION
Meta-DETR combines pure image-level prediction with inter-class correlational meta-learning for few-shot detection. It achieves state-of-the-art performance across multiple setups and extends to other instance-level few-shot tasks.
- Meta-DETR uses pure image-level prediction to lift constraints from inaccurate novel-class region proposals.
- Its inter-class correlational meta-learning reduces misclassification and enhances generalization among similar or related classes.
- Meta-DETR achieves state-of-the-art performance across multiple few-shot object detection setups, outperforming prior works by large margins.
- The framework can also be easily extended to other instance-level few-shot learning tasks.