Source-linked AI summary
Beyond Max-Margin: Class Margin Equilibrium for Few-shot Object Detection
Bohao Li, Boyu Yang, Chang Liu, Feng Liu, Rongrong Ji, Qixiang Ye
TL;DR
Few-shot detection must classify novel classes using base-class features, but max-margin separation conflicts with representing diverse novel classes. CME decouples localization features, reserves margin space, and applies adversarial feature disturbance; experiments report improvements over one-stage and two-stage baselines, with up to 3.8% average improvement in the cited Pascal VOC comparison.
Problem
Few-shot detection faces a contradiction: class separation favors max-margin features, while novel-class representation favors closer base-class distributions.
Method
CME decouples localization features, uses class-margin loss to reserve novel-class margin space, and disturbs features adversarially to pursue margin equilibrium.
Results
CME improves both one-stage and two-stage few-shot detectors; on Pascal VOC Novel Set 1, its average improvement over compared one-stage detectors is 3.8%.
Takeaways & Limitations
CME provides a plug-and-play approach for alleviating the representation-classification conflict in few-shot detection.
Abstract
from arXiv · showhide
Few-shot object detection has made substantial progressby representing novel class objects using the feature representation learned upon a set of base class objects. However,an implicit contradiction between novel class classification and representation is unfortunately ignored. On the one hand, to achieve accurate novel class classification, the distributions of either two base classes must be far away fromeach other (max-margin). On the other hand, to precisely represent novel classes, the distributions of base classes should be close to each other to reduce the intra-class distance of novel classes (min-margin). In this paper, we propose a class margin equilibrium (CME) approach, with the aim to optimize both feature space partition and novel class reconstruction in a systematic way. CME first converts the few-shot detection problem to the few-shot classification problem by using a fully connected layer to decouple localization features. CME then reserves adequate margin space for novel classes by introducing simple-yet-effective class margin loss during feature learning. Finally, CME pursues margin equilibrium by disturbing the features of novel class instances in an adversarial min-max fashion. Experiments on Pascal VOC and MS-COCO datasets show that CME significantly improves upon two baseline detectors (up to $3\sim 5\%$ in average), achieving state-of-the-art performance. Code is available at https://github.com/Bohao-Lee/CME .
1. Introduction
Few-shot detection relies on base-class features, but representation and classification impose conflicting margin requirements. CME addresses this contradiction through class-margin equilibrium, feature filtering, margin loss, and adversarial feature disturbance.
- Motivation: Few-shot detection learns representations from abundant base classes and adapts them to novel classes with few supervisions.Existing training commonly separates base representation learning from novel-class reconstruction.
- Motivation: Max-margin separation improves classification but increases novel-class diversity, whereas min-margin representation reduces classification separability.These opposing requirements create an unresolved trade-off in the shared feature space.
- Proposed approach: CME converts detection into few-shot classification by filtering localization features that could mislead class margins.A fully connected layer decouples localization features during training.
- Proposed approach: CME implements class-margin equilibrium with a max-margin loss and feature disturbance in an adversarial min-max fashion.The method reserves margin space for novel classes while balancing representation and classification.
- Reported outcome: CME improves upon both one-stage and two-stage baseline detectors with significant performance margins.The contribution statement reports improvements across both detector types without specifying a single aggregate value here.
2. Related Works
Prior few-shot detection builds on one-stage, two-stage, metric-learning, meta-learning, and augmentation methods. Despite this progress, class discriminability, novel/base-class equilibrium, and localization-feature handling remain unresolved concerns.
- CNN-based detection: One-stage detectors offer higher efficiency, while two-stage detectors usually report higher performance but generally require large training datasets.YOLO and SSD exemplify one-stage methods; Faster R-CNN and FPN exemplify two-stage methods.
- Few-shot learning: Few-shot learning methods include metric learning, meta-learning, and data augmentation approaches.These categories respectively compare category membership, accelerate adaptation, or generate additional unseen-class examples.
- Few-shot detection: Prior few-shot detectors use base-class data and meta-learning or sample refinement to adapt prediction networks and enrich object scales.Examples include Meta YOLO, RoI meta-learning for Faster R-CNN, and positive-sample refinement.
- Open issues: Existing methods leave the equilibrium between novel and base classes unresolved and often overlook localization features when treating detection as classification.These issues motivate methods that jointly address class representation and localization.
3. The Proposed Approach
CME trains a few-shot detector by separating localization from classification, enlarging margins among base classes, and disturbing novel-class features during finetuning to pursue margin equilibrium.
- 3.1. Few-shot Detection Framework: CME uses support and query branches to extract prototypes from support images and activate query features for classification and box regression.Support prototypes are formed from masked object features, while query features are combined with class prototypes before prediction.
- 3.1. Few-shot Detection Framework: A fully connected layer filters localization information from convolutional features, converting detection into a pure few-shot classification problem for margin learning.Localization features are class-independent and can perturb class margins, whereas the filtered representation is optimized with the max-margin loss.
- 3.2. Base Training: Class Max-margin: Max-margin learning reduces intra-class variance and increases inter-class distance so the feature space reserves margin space for novel classes.The objective approximates class margins through inter-class and intra-class distances and is combined with detection loss during base training.
- 3.3. Finetuning: Margin Equilibrium: Feature disturbance computes gradient maps during finetuning, truncates high-gradient pixels, and re-samples support masks to disturb finetuned features.The threshold controls the disturbance ratio; experiments set it dynamically so the top 15% highest-gradient pixels are set to zero.
- 3.3. Finetuning: Margin Equilibrium: During finetuning, base and novel classes are jointly processed while re-sampled prototypes occupy class margins and back-propagation maximizes margins between classes.The resulting procedure pursues class margin equilibrium for base classes and embedded novel classes.
4. Experiments
Experiments evaluate CME through ablations, feature-space visualizations, and comparisons with one-stage and two-stage detectors on Pascal VOC and MS COCO. The results show gains from max-margin learning, feature filtering, and adversarial feature disturbance, with improved class separation and detection performance.
- Ablation Study: CME improves over the baseline by 2.6% with max-margin loss, 4.3% with feature filtering, and 5.4% with feature disturbance on Pascal VOC split-1.The ablation evaluates different shot settings on novel classes.
- Ablation Study: Max-margin is effective in 2-, 3-, and 5-shot settings but invalid in the 1-shot setting, where increased class margins hinder novel-class reconstruction.The result supports balancing classification separation with reconstruction under limited data.
- Ablation Study: The feature-filtering module performs best with 512 output channels; 1024 is redundant, while 256 insufficiently represents features.The fully connected layer should use slightly fewer output channels than its 1024 input channels to filter localization-related features.
- Ablation Study: Disturbing base-class prototypes benefits novel-class representation, whereas disturbing novel classes can reduce representation discrimination as available margin space becomes limited.The study attributes this behavior to depressing base-class margins without overly restricting novel-class margin space.
- Ablation Study: Gradient truncation outperforms feature truncation and feature cropping because it implements adversarial min-max margin optimization rather than simple augmentation.This comparison evaluates alternative feature-disturbance strategies.
- Model Analysis: CME reduces overlap between novel Cow and base Cat, Dog, Sheep, and Horse prototypes, improving feature-space separation over the baseline.The t-SNE analysis links margin reservation and equilibrium during finetuning to improved prototype organization.
- Performance Comparison: On Pascal VOC Novel Set 1, CME improves over the one-stage baseline by 0.7% at 1-shot, 7.0% at 2-shot, 2.6% at 3-shot, and 9.8% at 5-shot.The average improvement is 3.8%.
- Performance Comparison: For Pascal VOC Novel Set 1, CME outperforms the compared two-stage detector by 5% at 2-shot and 6% at 5-shot.The average improvement is 1.2% for Novel Set 1, 1.5% for Novel Set 2, and 0.3% for Novel Set 3.
5. Conclusion
CME optimizes feature-space partition and novel-class representation for few-shot object detection. It improves both one-stage and two-stage detectors while addressing the tension between feature representation and classification.
- CME optimizes both feature-space partition and novel-class representation for few-shot object detection.
- During base training, CME preserves adequate margin space for novel classes using a class margin loss.
- During finetuning, CME pursues margin equilibrium by perturbing novel-class instance features adversarially in a min-max fashion.
- CME improves both one-stage and two-stage few-shot detectors as a plug-and-play module.
- CME offers a general approach to feature representation learning and class-margin optimization for few-shot learning.