Source-linked AI summary
Few-shot Object Detection via Feature Reweighting
Bingyi Kang, Zhuang Liu, Xin Wang, Fisher Yu, Jiashi Feng, Trevor Darrell
TL;DR
Few-shot object detection seeks to detect novel categories despite scarce bounding-box annotations, a setting where conventional CNN detectors can overfit. The paper learns transferable meta features from labeled base classes and adapts them with support-conditioned reweighting in a one-stage detector. Experiments report large-margin gains over baselines across datasets and settings, with faster adaptation to novel classes.
Problem
Few-shot object detection addresses detecting novel objects when categories have only a few annotated examples and conventional CNN detectors require abundant bounding-box data.
Method
The model learns generalizable meta features from base classes and uses a reweighting module to transform novel-class support examples into coefficients that adapt query features.
Results
The detector outperforms competitive baselines by a large margin across multiple datasets and settings, especially with extremely few labels, and adapts faster to novel classes.
Takeaways & Limitations
Support-conditioned feature reweighting transfers detection knowledge from well-labeled base classes to novel classes within the proposed few-shot detection framework.
Takeaways & Limitations
The proposed setting assumes training data contain base classes with sufficient examples alongside novel classes with only a few samples.
Abstract
from arXiv · showhide
Conventional training of a deep CNN based object detector demands a large number of bounding box annotations, which may be unavailable for rare categories. In this work we develop a few-shot object detector that can learn to detect novel objects from only a few annotated examples. Our proposed model leverages fully labeled base classes and quickly adapts to novel classes, using a meta feature learner and a reweighting module within a one-stage detection architecture. The feature learner extracts meta features that are generalizable to detect novel object classes, using training data from base classes with sufficient samples. The reweighting module transforms a few support examples from the novel classes to a global vector that indicates the importance or relevance of meta features for detecting the corresponding objects. These two modules, together with a detection prediction module, are trained end-to-end based on an episodic few-shot learning scheme and a carefully designed loss function. Through extensive experiments we demonstrate that our model outperforms well-established baselines by a large margin for few-shot object detection, on multiple datasets and settings. We also present analysis on various aspects of our proposed model, aiming to provide some inspiration for future few-shot detection works.
1. Introduction
The paper addresses few-shot object detection, where scarce bounding-box annotations make conventional CNN detectors overfit, by adapting detection from base classes to novel classes using few support examples. Its model combines generalizable meta features with support-conditioned reweighting and outperforms baselines across datasets and settings.
- Motivation: Scarce bounding-box annotations can cause CNN detectors to overfit, motivating detection methods that learn novel categories from few examples.The need arises for rare or difficult-to-annotate categories, including endangered animals and medical data.
- Problem: Few-shot object detection must recognize both sufficiently labeled base classes and sparsely sampled novel classes while also localizing objects.The paper notes that detection is harder than few-shot classification because it requires both class prediction and localization.
- Approach: The proposed detector learns generalizable meta features from base classes and uses few novel-class support examples to identify features important for detection.This transfers detection knowledge from base classes to novel classes by adapting the detection prediction network.
- Approach: A lightweight reweighting module converts support images into class-specific coefficients that modulate query-image meta features for novel-object detection.The feature learner, reweighting module, and detection prediction module are trained end-to-end with a two-phase learning scheme and a designed loss.
- Contribution: The work is positioned as an early study of the practically valuable but less explored problem of few-shot object detection.Its analysis also examines learning speed, reweighting vectors, and component contributions.
- Results: The proposed detector outperforms competitive baselines by a large margin, particularly when the number of labels is extremely low, and adapts to novel classes faster.The paper reports evaluation across multiple datasets and settings.
2. Related Work
Prior few-shot learning research largely targets image recognition, using metric, Bayesian, augmentation, or weight-generation approaches. This paper places few-shot adaptation within object detection and presents an architecture combining one-stage detection with support-conditioned feature reweighting.
- General object detection: General object detectors include proposal-based RCNN-family methods and proposal-free methods such as YOLO.The paper situates its detector within the proposal-free detection category.
- Few-shot learning: Few-shot learning studies learning from only a few training examples per class, including Bayesian inference, compositionality, domain adaptation, and semi-supervised label propagation.These approaches address few-shot learning under different assumptions about prior models, domains, or unlabeled data.
- Meta-learning: Meta-learning approaches include metric-learning methods such as Matching, Prototypical, and Relation Networks for recognizing novel images.The cited methods learn similarity, class-specific linear classifiers, or distance metrics.
- Base-and-novel recognition: Other methods address recognition of both base and novel images through hallucination, weight imprinting, or category-specific weight generation.The paper distinguishes these image-classification methods from its focus on object detection.
- This work: The proposed detection architecture combines a meta feature extractor with a reweighting module that maps support samples to class-specific vectors.The one-stage detector predicts objectness, box location, and classification scores, with softmax normalization applied to classification outputs.
3. Approach
The approach defines a base/novel-class few-shot detection setting and introduces feature reweighting within a one-stage detector. It learns meta features and support-conditioned reweighting through episodic training, two-phase fine-tuning, and a calibrated detection loss.
- Few-shot detection setting: The setting provides abundant annotated data for base classes but only a few labeled samples for novel classes, while testing includes both.The model leverages base-class knowledge to detect novel objects.
- Model framework: The detector adds a meta feature learner and reweighting module to a one-stage YOLOv2 framework with a detection prediction module.YOLOv2 directly regresses classification scores and bounding-box coordinates for predefined anchors.
- Feature reweighting: Support image and bounding-box pairs are embedded into class-specific vectors that reweight query-image meta-feature channels before prediction.The reweighting operation is implemented as channel-wise multiplication and highlights features relevant to the target class.
- Training scheme: The model is trained episodically by jointly optimizing the feature learner, reweighting module, and prediction module with detection loss.Base training organizes annotated images into tasks containing support and query sets.
- Training scheme: Two-phase learning first trains on base classes and then fine-tunes on both base and novel classes using balanced k-shot samples.After fine-tuning, averaged reweighting vectors allow inference without support inputs, so the reweighting module can be removed.
- Detection loss: A softmax-calibrated classification loss suppresses less probable class predictions, complementing YOLOv2-style bounding-box and objectness losses.The loss addresses redundant detections produced by binary classification, such as labeling one object as multiple classes.
4. Experiments
Experiments on PASCAL VOC, MS-COCO, and cross-dataset transfer evaluate few-shot detection across novel-class splits, shot counts, adaptation speed, feature behavior, and ablations. The proposed model consistently outperforms baselines, adapts faster, and benefits from deeper-layer reweighting and softmax loss.
- Datasets and settings: The evaluation uses VOC and COCO novel-class splits with 1–10 shots on VOC, 10 or 30 shots on COCO, and a COCO-to-PASCAL transfer setting.VOC uses 15 base and 5 novel classes across three splits; COCO uses 60 base and 20 novel classes.
- Comparison with baselines: The proposed model significantly outperforms baselines on PASCAL VOC, especially in the 1–3-shot regime, with improvements consistent across splits and shot counts.At 5 shots, LSTD(YOLO)-full is 4.3 mAP above YOLO-ft-full on one novel split but 5.1 mAP below it on another.
- Comparison with baselines: On COCO, the model outperforms all baselines at both 10 and 30 shots, while a YOLO baseline trained for the same iterations achieves less than 1% AP.The authors note that few-shot COCO detection remains challenging because of the dataset’s complexity and scale.
- Learning speed: The model converges in 1,200 iterations to higher accuracy, whereas YOLO-ft-full and LSTD(YOLO)-full require 25,000 iterations for full convergence.Figure 3 shows faster convergence for each novel class and on average; Sofa is an exception where the baseline eventually slightly surpasses the model.
- Performance analysis: Reweighting coefficients vary substantially for roughly half the features, while the remaining features are relatively class-insensitive and shared across classes.t-SNE coefficients cluster by class, with visually similar classes tending to lie closer together.
- Ablation studies: Applying reweighting at deeper layers performs better than using earlier layers, while reweighting only half of layer-21 features does not substantially reduce performance.The comparison considers layers 13, 20, and 21, including a half-feature layer-21 variant.
- Ablation studies: Softmax loss significantly outperforms single-binary and multi-binary losses, likely because it suppresses redundant detection results.The binary alternatives differ in whether each iteration uses one class or N classes for binary supervision.
5. Conclusion
The work explores few-shot object detection and introduces a model that rapidly adjusts basic feature contributions for novel classes using few examples. Experiments on realistic benchmark datasets demonstrate effectiveness, alongside analyses of learning speed and design components.
- The work explores the practical and challenging problem of few-shot object detection.
- The model rapidly adjusts contributions of basic features to detect novel classes from a few examples.
- Experiments on realistic benchmark datasets clearly demonstrate the model’s effectiveness.
- The study compares model learning speed and analyzes predicted reweighting vectors and individual design components.
- The authors identify few-shot detection in more complex scenes as an area for further exploration.
Acknolwedgement
The work acknowledges institutional and government support for the research.
- Jiashi Feng was partially supported by NUS IDS, ECRA, and MOE Tier-II grants.
- The work was partly supported by the US DoD, the Berkeley Deep Drive Center, and the BAIR Lab.
Implementation Details
The models use SGD with momentum, weight decay, fixed batch size, staged learning-rate schedules, and multi-scale training. Base training and few-shot fine-tuning use separate iteration budgets and learning rates.
- All models use SGD with momentum 0.9, L2 weight decay 0.0005, and batch size 64.
- Base training runs for 80,000 iterations with stepwise learning-rate changes at iterations 500, 40,000, and 60,000.
- Few-shot fine-tuning uses a constant learning rate of 0.001 for 1500 iterations.
- The training procedure uses multi-scale training.
Additional Ablation Studies
Additional ablations show that averaging reweighting coefficients at test time improves performance without extra inference time, while weight sharing reduces performance. The reported changes quantify the effects of these design choices on base and novel classes.
- Sampling of Examples for Testing: 69.7%/47.2% on base/novel classes falls to 63.9%/45.1% when testing randomly selected support inputs instead of averaging predicted weights.Averaging predicted weights produces an ensembling-like effect without additional inference time.
- Sharing Weights Between Feature Extractor and Reweighting Module: 68.3%/44.8% on base/novel classes results when feature-extractor and reweighting-module weights are shared, down from 69.7%/47.2%.The authors suggest that weight sharing may impose more constraints during optimization.
Complete Results on PASCAL VOC
The PASCAL VOC results report detection performance across base and novel categories for three base/novel splits and varying numbers of novel-category training examples.
- Results are presented for each class and number of shots across three PASCAL VOC base/novel splits.Splits 1, 2, and 3 are reported separately.
- Table 7 reports AP for base and novel categories in the first base/novel split across different novel-category training-example counts.
- Table 8 reports AP for base and novel categories in the second base/novel split across different novel-category training-example counts.
- Table 9 reports AP for base and novel categories in the third base/novel split across different novel-category training-example counts.