PaperlayerView exact revision on arXiv

Source-linked AI summary

Meta Faster R-CNN: Towards Accurate Few-Shot Object Detection with Attentive Feature Alignment

Guangxing Han, Shiyuan Huang, Jiawei Ma, Yicheng He, Shih-Fu Chang

arXiv:2104.07719v4cs.CVcs.AIcs.MM

TL;DR

Few-shot object detection must detect novel categories from scarce examples despite poor proposal quality and spatial misalignment. The paper introduces Meta Faster R-CNN, combining coarse-to-fine prototype matching, attentive feature alignment, and separate base-class detection. It achieves state-of-the-art accuracy on multiple FSOD benchmarks, with further fine-tuning.

  • Problem

    Few-shot object detection remains challenging because existing methods generate missed or spatially misaligned proposals for novel classes with scarce training examples.

  • Method

    Meta Faster R-CNN uses Meta-RPN for coarse prototype-based proposal generation, a fine-grained attentive feature-alignment classifier, and separate detection heads for novel and base classes.

  • Results

    With further fine-tuning, the model achieves state-of-the-art accuracy on multiple FSOD benchmarks.

  • Takeaways & Limitations

    Meta-training enables incremental enrollment of few-shot novel classes without training during meta-testing while preserving a dedicated base-class detection pathway.

  • Takeaways & Limitations

    The softmax-based detector is inflexible for adding new classes because it requires fine-tuning a new detector; the few-shot detector is unsuitable for base classes in speed and accuracy.

Abstract

from arXiv · show

Few-shot object detection (FSOD) aims to detect objects using only a few examples. How to adapt state-of-the-art object detectors to the few-shot domain remains challenging. Object proposal is a key ingredient in modern object detectors. However, the quality of proposals generated for few-shot classes using existing methods is far worse than that of many-shot classes, e.g., missing boxes for few-shot classes due to misclassification or inaccurate spatial locations with respect to true objects. To address the noisy proposal problem, we propose a novel meta-learning based FSOD model by jointly optimizing the few-shot proposal generation and fine-grained few-shot proposal classification. To improve proposal generation for few-shot classes, we propose to learn a lightweight metric-learning based prototype matching network, instead of the conventional simple linear object/nonobject classifier, e.g., used in RPN. Our non-linear classifier with the feature fusion network could improve the discriminative prototype matching and the proposal recall for few-shot classes. To improve the fine-grained few-shot proposal classification, we propose a novel attentive feature alignment method to address the spatial misalignment between the noisy proposals and few-shot classes, thus improving the performance of few-shot object detection. Meanwhile we learn a separate Faster R-CNN detection head for many-shot base classes and show strong performance of maintaining base-classes knowledge. Our model achieves state-of-the-art performance on multiple FSOD benchmarks over most of the shots and metrics.

Introduction

Few-shot object detection must generalize from abundant base classes to novel classes with scarce annotations, but existing detectors produce lower-quality proposals for novel classes. Meta Faster R-CNN addresses this with coarse-to-fine prototype matching, attentive feature alignment, and separate detection heads for novel and base classes.

  • Deep-learning object detectors require extensive annotations, which are costly or unavailable for rare categories and can cause overfitting with scarce data.
  • FSOD transfers knowledge from data-rich base classes to novel classes with only a few examples, while remaining efficient and effective is challenging.
  • Existing proposal methods generate substantially poorer novel-class proposals, including missed high-IoU boxes and boxes spatially misaligned with true objects.
  • The model decouples novel- and base-class detection into branches, retaining the Faster R-CNN pipeline for base classes while using a coarse-to-fine detector for novel classes.
  • Meta-RPN uses lightweight nonlinear prototype matching instead of a simple RPN object/nonobject classifier to generate high-recall, category-specific novel-class proposals.
  • The Meta-Classifier performs spatial alignment and foreground-focused matching between noisy proposal features and few-shot class prototypes.
  • Meta-training on base classes enables incremental novel-class enrollment without training during meta-testing, while further fine-tuning achieves state-of-the-art accuracy on multiple FSOD benchmarks.

Related Work

Object detection methods are broadly divided into proposal-based and proposal-free approaches. This work follows the proposal-based family because detection accuracy remains the priority for FSOD.

  • Deep-learning object detectors are broadly grouped into proposal-based and proposal-free methods.
  • The proposed method belongs to the proposal-based family because it prioritizes detection accuracy for few-shot object detection.

Our Approach

Meta Faster R-CNN uses separate coarse-to-fine matching modules for novel-class proposal generation and classification, while retaining a Faster R-CNN head for base classes. Meta-RPN matches anchor features to few-shot class prototypes, and Meta-Classifier aligns proposal and prototype features before foreground-focused similarity scoring.

  • Task Definition: The FSOD task separates disjoint base and novel class sets, with many labeled base examples but only K-shot support images for each novel class.The goal is to detect novel-class objects from few-shot examples while preserving high base-class accuracy.
  • Base-Class Detection: A separate Faster R-CNN head classifies base proposals over base classes and background while predicting bounding-box regression.
  • Novel-Class Proposal Generation: Meta-RPN generates novel-class proposals by matching query-image anchor features with averaged few-shot class prototypes through a lightweight non-linear feature-fusion classifier.The fusion network uses multiplication, subtraction, and concatenation subnetworks rather than a simple linear object/nonobject classifier.
  • Novel-Class Proposal Generation: The Meta-RPN feature-fusion network is convolutional, computationally efficient, and designed to achieve high novel-class proposal recall with few proposals.
  • Novel-Class Proposal Classification: Meta-Classifier uses high-resolution proposal and prototype features, then spatially aligns the prototype to the proposal using dense semantic correspondences.The alignment aggregates prototype features according to an affinity matrix between spatial locations.
  • Novel-Class Proposal Classification: A foreground attention mask emphasizes proposal regions similar to the aligned prototype, suppressing background regions before final non-linear similarity classification.Learnable residual scaling parameters stabilize the feature transformation during training.

Experimental Results

Experiments show that Meta-RPN improves novel-class proposal generation, while attentive feature alignment and feature-fusion choices improve few-shot classification. Across FSOD benchmarks, the method performs strongly in extremely few-shot settings, supports incremental class enrollment, and preserves base-class detection through separate heads.

  • Meta-RPN: Meta-RPN improves novel-class proposal generation over RPN and Attention-RPN through metric-learning-based nonlinear prototype matching, especially after fine-tuning.The comparison evaluates proposal average recall and detection AP using different proposal counts, with 100 proposals per novel class used by default.
  • Meta-Classifier: Attentive feature alignment consistently improves Meta-Classifier performance by addressing spatial misalignment between inaccurate proposals and few-shot class prototypes.Aligning the class prototype to proposal features also preserves proposal structure for precise bounding-box regression.
  • Feature fusion: Combining multiplication, subtraction, and concatenation subnetworks learns complementary feature fusion beyond the individual operations.The concatenation subnetwork alone performs poorly, while combining all three subnetworks eases its training.
  • Meta-learning and fine-tuning: Fine-tuning improves performance in larger-shot settings such as 10/30 shot but can hardly improve 2-shot performance because it is prone to overfitting.This comparison uses ResNet-50 and ResNet-101 variants under meta-training and fine-tuning.
  • Base-class evaluation: Separate detection heads retain base-class detection while avoiding the softmax detector’s need to fine-tune a new classifier when adding classes.Base-class meta-testing remains below the pretrained softmax model’s upper-bound accuracy, although it is much higher than novel-class results.
  • Benchmark results: The model achieves state-of-the-art performance on most shots and metrics across PASCAL VOC and MSCOCO, with meta-learning-only results especially strong for MSCOCO 1/2-shot settings.SRR-FSD and Halluc outperform it on the VOC split1&2 1-shot setting by using linguistic semantic knowledge and image hallucination, respectively.

Conclusion

The model addresses low-quality few-shot proposals with coarse-grained prototype matching for proposal generation and fine-grained attentive feature alignment for classification. Experiments on multiple FSOD benchmarks demonstrate its effectiveness.

  • A lightweight coarse-grained prototype matching network generates proposals for few-shot classes efficiently and effectively.
  • A fine-grained prototype matching network with attentive feature alignment addresses spatial misalignment between noisy proposals and few-shot classes.
  • Experiments on multiple FSOD benchmarks demonstrate the effectiveness of the proposed approach.

Appendix

The appendix provides analyses of attentive feature alignment, running speed, full MSCOCO FSOD results, and implementation details.

  • The supplementary materials analyze attentive feature alignment and provide additional visualization examples.
  • The appendix includes a detailed running speed analysis for each model component.
  • The appendix reports full experimental results on the MSCOCO FSOD benchmarks.
  • Implementation details of the model are provided in the supplementary materials.

Alignment

Attentive feature alignment matches semantically similar proposal and support features spatially, then emphasizes corresponding foreground regions. The module adds negligible parameters and is intended to reduce overfitting risk.

  • Attentive feature alignment addresses spatial misalignment between noisy proposals and few-shot classes using a feature-alignment-based metric-learning method.
  • Dot products measure similarity between proposal and class-prototype CNN grid features, forming the affinity matrix.
  • Row-normalized affinities aggregate prototype grid features into proposal-aligned prototypes and estimate corresponding foreground regions.
  • The attention mask highlights corresponding foreground features identified from the affinity matrix.
  • Similar semantic features are moved to the same position, while higher attention weights are assigned to corresponding foreground regions.
  • The module introduces negligible additional learning parameters and only two newly introduced scaling parameters, γ1 and γ2, for stable training.The remaining module components have no learnable parameters, reducing the risk of overfitting to base classes.
  • Visualization results show that the attention mask roughly localizes foreground objects and the affinity matrix highlights similar semantic regions in proposal and 1-shot support images.

Running Speed Analysis of Our Model

The softmax detector’s running time remains stable across class counts, whereas the few-shot detector’s running time grows with the number of detected classes. The two detectors therefore have complementary efficiency and flexibility trade-offs.

  • The softmax-based detector generates 1000 shared proposals, while the few-shot detector generates 100 proposals per class.The differing proposal counts reflect their distinct detection designs and the reported comparison setup.
  • The softmax-based detector’s running time is stable when detecting different numbers of classes because most operations are shared.Only the final linear classification layer depends on the number of classes.
  • The few-shot detector’s running time is proportional to the number of detected classes because it performs separate detection for each class.This includes class-specific proposal generation and pairwise classification.
  • The few-shot detector is inefficient for base classes considering both running speed and detection accuracy.
  • The softmax detector is inflexible for adding classes, whereas the meta-learning method can enroll new classes without training.The proposed system therefore decouples base- and novel-class detection into two detection heads.

FSOD Benchmarks

The evaluation compares Meta Faster R-CNN with a strong baseline across MSCOCO few-shot settings. Both meta-learning and fine-tuning variants outperform the baseline, while meta-learning is especially effective in low-shot regimes.

  • Full MSCOCO evaluation covers 1/2/3/5/10/30-shot settings for both meta-learning and fine-tuning, compared with Fan et al. (2020).
  • Both proposed variants outperform the strong baseline model of Fan et al. (2020).
  • Meta-learning outperforms Fan et al.'s fine-tuning model in 1-shot PASCAL VOC and 1/2/3-shot MSCOCO settings.
  • Meta-learning performs better than the proposed fine-tuning model in extremely few-shot settings, including 1-shot PASCAL VOC and 1/2/3-shot MSCOCO.
  • Fine-tuning is prone to overfitting with very few samples but can help at larger shot settings.
  • The meta-learning model is presented as more suitable for realistic applications because it adapts efficiently without training and performs effectively in extremely few-shot scenarios.

Implementation Details

The implementation uses Faster R-CNN with separate proposal and detection components for novel and base classes, trained through a three-step framework. Training combines episodic meta-learning, base-class detection learning, and balanced base–novel fine-tuning.

  • Model architecture: Faster R-CNN with ResNet-50/101 extracts image features after res4, while RPN and Meta-RPN generate base- and novel-class proposals, respectively.RoIAlign and res5 extract proposal and few-shot example features before R-CNN classification and bounding-box regression.
  • Three-step training: Meta-learning on base classes trains Meta-RPN and Meta-Classifier episodically for few-shot detection.The first-stage loss combines Meta-RPN and Meta-Classifier losses.
  • Losses: The meta-learning losses use binary cross-entropy for matched versus unmatched examples and smooth L1 loss for bounding-box regression.These components define both Meta-RPN and Meta-Classifier optimization.
  • Optimization: MSCOCO training uses SGD with learning rate 0.002, momentum 0.9, weight decay 0.0001, and batch size 8 across three gradual stages.PASCAL VOC uses the same hyperparameters but halves the training iterations in all stages.
  • Three-step training: A separate base-class detection head uses RPN, an R-CNN classifier, and FPN-based multi-scale features.This stage learns object detection over base classes independently of the few-shot modules.
  • Three-step training: Balanced base–novel fine-tuning uses the first-stage loss while freezing the backbone and base-class detection head.The fine-tuning stage uses both base and novel classes.
Loading 2104.07719v4…