Source-linked AI summary
Frustratingly Simple Few-Shot Object Detection
Xin Wang, Thomas E. Huang, Trevor Darrell, Joseph E. Gonzalez, Fisher Yu
TL;DR
Few-shot object detection asks models to recognize and localize novel objects from very few labeled examples, but existing evaluations are unreliable. The paper evaluates a simple two-stage fine-tuning strategy and revised benchmarks, finding strong gains over prior meta-learning methods and new state-of-the-art results.
Problem
Few-shot object detection remains difficult because detection requires both recognizing and localizing novel objects from few labeled bounding boxes, while existing evaluations are statistically unreliable.
Method
The paper trains the full detector on base classes, then freezes feature-extractor parameters and fine-tunes the last layers on a balanced base-and-novel set, while revising benchmark evaluation.
Results
2∼20 points: TFA outperforms prior state-of-the-art meta-learning methods on existing PASCAL VOC and COCO benchmarks, while establishing new states of the art on revised benchmarks.
Takeaways & Limitations
Simple last-layer fine-tuning can provide strong few-shot object-detection performance, including improved rare-class AP with negligible frequent-class precision loss on LVIS.
Takeaways & Limitations
Existing few-shot evaluations have large sample variance and can omit base-class performance, motivating more reliable generalized benchmarks.
Abstract
from arXiv · showhide
Detecting rare objects from a few examples is an emerging problem. Prior works show meta-learning is a promising approach. But, fine-tuning techniques have drawn scant attention. We find that fine-tuning only the last layer of existing detectors on rare classes is crucial to the few-shot object detection task. Such a simple approach outperforms the meta-learning methods by roughly 2~20 points on current benchmarks and sometimes even doubles the accuracy of the prior methods. However, the high variance in the few samples often leads to the unreliability of existing benchmarks. We revise the evaluation protocols by sampling multiple groups of training examples to obtain stable comparisons and build new benchmarks based on three datasets: PASCAL VOC, COCO and LVIS. Again, our fine-tuning approach establishes a new state of the art on the revised benchmarks. The code as well as the pretrained models are available at https://github.com/ucbdrive/few-shot-object-detection.
1. Introduction
Few-shot object detection must recognize and localize novel objects from very few labeled examples, yet existing comparisons are statistically unreliable. The paper studies fine-tuning-based detection, introduces a two-stage approach and revised benchmarks, and reports strong gains over prior methods.
- Motivation: Few-shot object detection requires recognizing object types and localizing targets when only a few labeled bounding boxes are available for novel classes.The localization requirement adds complexity beyond few-shot image classification.
- Method: The paper examines fine-tuning approaches and uses two-stage training: full-detector base training followed by last-layer fine-tuning on a balanced base-and-novel set.Feature-extractor parameters remain fixed during fine-tuning, while instance-level feature normalization is introduced for the box classifier.
- Results: 2∼20 points: TFA outperforms previous state-of-the-art meta-learning methods on existing PASCAL VOC and COCO benchmarks.The reported comparison covers the existing benchmarks rather than the revised evaluation protocol.
- Results: Twice the accuracy: TFA can achieve twice the accuracy of prior sophisticated state-of-the-art approaches in the one-shot setting.This result concerns training with a single novel example.
- Motivation: Existing evaluations have high variance, report novel-class accuracy without base-class retention, and make model comparisons unreliable.The revised protocol addresses both statistical stability and generalized performance reporting.
- Evaluation: The revised benchmarks sample multiple few-shot training groups and report AP for base classes, novel classes, and all classes across PASCAL VOC, COCO, and LVIS.These changes aim to obtain stable accuracy estimates and evaluate knowledge retention.
- Results: ∼4 points: On LVIS, the approach improves average detection precision for rare classes, ∼2 points for common classes, with negligible loss for frequent classes.Rare classes have fewer than 10 images, common classes have 10–100, and frequent classes have more than 100.
2. Related Work
Prior few-shot detection work largely follows meta-learning strategies developed for classification, while fine-tuning has often been treated as an inferior baseline. The paper positions itself as a systematic analysis showing that a restricted fine-tuning design can outperform those approaches.
- Fine-tuning: The paper describes its contribution as the first systematic analysis of fine-tuning-based approaches for few-shot object detection.This claim is stated relative to the authors’ knowledge.
- Meta-learning: Meta-learning seeks task-level knowledge that helps models adapt quickly to new tasks and environments with few labeled examples.Approaches include learning adaptable initializations and generating parameters for novel tasks.
- Few-shot object detection: The literature lacks consensus on evaluation benchmarks and consistent comparisons because of network complexity, implementation details, and evaluation-protocol variance.These concerns parallel broader reliability issues identified in few-shot learning comparisons.
- Metric-learning: Few-shot classification research also uses metric-learning and cosine-similarity classifiers to compare inputs or reduce intra-class variance.These methods are presented as another major line of prior work alongside meta-learning.
- Few-shot object detection: Early few-shot object detection methods attach meta learners to detectors and use support images with bounding-box annotations for feature re-weighting or weight prediction.Examples span single-stage YOLOv2 and two-stage Faster R-CNN detectors.
- Fine-tuning: Unlike prior baselines that fine-tune jointly or update the entire model, the paper fine-tunes only the detector’s last layer on a balanced subset while fixing the remaining model.The authors report that this design substantially improves detection accuracy and suggest base-learned representations transfer to novel classes.
3. Algorithms for Few-Shot Object Detection
Few-shot object detection evaluates detectors on both data-rich base classes and data-scarce novel classes, using average precision for each. The paper contrasts two-stage fine-tuning with meta-learning approaches and highlights evaluation reliability, feature normalization, and memory efficiency.
- Few-shot object detection setting: Few-shot detection uses base classes with many instances and novel classes with only K examples per category.Synthetic PASCAL VOC and COCO splits are balanced K-shot datasets, while LVIS uses frequent, common, and rare categories from its natural long-tail distribution.
- Few-shot object detection setting: Average precision is measured separately on base and novel classes, unlike the N-way-K-shot setting common in few-shot classification.The test set contains both class groups, and the evaluation objective includes detection accuracy for each.
- Two-stage fine-tuning approach: The proposed two-stage fine-tuning trains the detector on base classes, then updates only its box classification and regression layers using a balanced base-and-novel set while freezing the feature extractor.The second stage uses K shots per class, randomly initialized novel-class prediction weights, the same loss, and a smaller learning rate.
- Two-stage fine-tuning approach: Cosine-similarity classification compares normalized proposal features with class weight vectors and empirically reduces novel-class intra-class variance during fine-tuning.The scaling factor α is fixed at 20, and the authors report less base-class accuracy decrease than a fully connected classifier when examples are scarce.
- Meta-learning based approaches: Meta-learning detectors use a meta-learner with support examples to reweight query features or generate class-specific weights, typically through episodic two-stage training.Examples include FSRW and Meta R-CNN for feature re-weighting and MetaDet for weight generation.
- Meta-learning based approaches: The fine-tuning method uses normal batch training and updates only final layers, making it more memory efficient than episodic meta-learning as support-set classes increase.The paper also identifies high variance and missing base-class retention measurements as problems in existing evaluation protocols.
4. Experiments
The experiments compare TFA with prior few-shot detection methods on existing benchmarks and introduce revised evaluations across PASCAL VOC, COCO, and LVIS. TFA consistently improves novel-class detection while largely retaining base-class performance, and repeated sampling stabilizes comparisons.
- Existing benchmarks: TFA outperforms prior methods by about 2∼20 points on PASCAL VOC few-shot detection, with especially large gains in low-shot settings.The reported comparison uses mAP50 across three novel-class sets.
- Existing benchmarks: TFA can double prior performance in one-shot PASCAL VOC cases, while cosine classification helps most at extremely low shots.Cosine and fully connected classifiers are roughly similar at 10-shot.
- Existing benchmarks: TFA preserves base-class accuracy substantially better than full-model fine-tuning, with a reported decrease of less than 2 points on base classes.The approach also outperforms baselines on both base and novel classes in the reported PASCAL VOC comparison.
- Existing benchmarks: On COCO, TFA consistently outperforms baselines across shots, improving novel AP by around 1 point and novel AP75 by around 2.5 points.AP75 uses a stricter matching threshold than AP50.
- Revised benchmark: The revised protocols address unreliable comparisons by sampling multiple training groups, reporting confidence intervals, and evaluating base, novel, and all classes.Existing evaluations had high sample variance and omitted knowledge retention on base classes; means stabilize after around 30 runs.
5. Conclusion
The paper proposes a simple two-stage fine-tuning approach for few-shot object detection and reports strong results on revised benchmarks, including improved rare-class AP on LVIS with negligible frequent-class reduction.
- The two-stage fine-tuning approach outperformed previous meta-learning methods by a large margin on current benchmarks.
- The authors built more reliable benchmarks using revised evaluation protocols.
- On LVIS, the models improved rare-class AP by 4 points with negligible reduction in frequent-class AP.
A. Generalized Object Detection Benchmarks
The revised generalized benchmarks average results across repeated random few-shot samples and report performance separately for base, novel, and all classes. Across PASCAL VOC and COCO, the proposed cosine-based fine-tuning generally outperforms fine-tuning and meta-learning baselines.
- Evaluation protocol: Each benchmark metric reports average AP, AP50, and AP75 for all, base, and novel classes across repeated random training-shot samples.PASCAL VOC uses 30 runs and COCO uses 10 runs, with 95% confidence intervals for each metric.
- PASCAL VOC: TFA w/cos significantly outperforms TFA w/fc in overall AP across most PASCAL VOC splits and shot settings.
- PASCAL VOC: TFA w/cos exceeds TFA w/fc by over 3 points in base-class AP on splits 1 and 3 at 5 and 10 shots.
- PASCAL VOC: Across split 1 and all shots, TFA w/cos consistently exceeds TFA w/fc by over 2 points in novel-class AP75.
- Baseline comparison: The proposed models are usually over 10 points higher in AP than FRCN+ft-full and FSRW across settings.The comparison uses Faster R-CNN for the proposed models and YOLOv2 for FSRW.
- COCO: On COCO, evaluations cover K = 1, 2, 3, 5, 10, and 30 shots, with similar base- and novel-class accuracy patterns but less significant differences.
B. Performance over Multiple Runs
Repeated sampling shows that benchmark estimates stabilize after roughly 30 PASCAL VOC runs and 8 COCO runs. Single-run evaluation can substantially overestimate performance, especially in one-shot settings.
- Repeated sampling: The revised benchmark uses repeated runs with randomly sampled training shots to improve reliability, using 30 runs for PASCAL VOC and 10 for COCO.
- Stability analysis: Cumulative means with 95% confidence intervals are plotted across up to 40 PASCAL VOC and 10 COCO random training-shot groups.
- Stability analysis: After around 30 PASCAL VOC runs and 8 COCO runs, means and variances stabilize sufficiently for reliable method comparisons.
- Single-run bias: In the PASCAL VOC one-shot split-1 case, average AP50 across 40 runs is around 15 points lower than the first-run AP50.
- Single-run bias: First-run accuracies often overestimate actual performance and can produce unreliable comparisons between approaches.
- Stability analysis: PASCAL VOC means and variances become stable after around 30 runs.